Server-Side Swift from Scratch
Brandon Williams at Swift Summit 2017
Thanks for the great intro. Yeah, I worked at Kickstarter for a long time doing iOS and Android, five and a half years. While I was there, I did a lot of backend work. In fact, still to this day, I think I'm the fourth top contributor to the web repo at Kickstarter. Today, I want to talk about some future directions of Server-Side Swift that I think kind of makes it maybe state-of-the-art, maybe better than most other frameworks out there.
I think it can do things that other frameworks can't really touch. The backend at Kickstarter was Rails and that kind of informed a lot of what I thought a web framework was supposed to be. There's a lot of good things that Rails does, but there's also a lot of not so good things. I steal a lot of ideas directly from Rails. I think we can get a much better Server-Side framework in the long run if we kind of maybe peel back some things and challenge some assumptions that we may have had going into doing Server-Side Swift, and kind of do some things from scratch.
The stuff I'm going to show today is, I think, very exciting. It's some of the coolest stuff I've worked on in a long time and it's going to be very fast. There's a lot to cover. I'm going to be light on the details and just do broad strokes of exciting things, but first things first. This is all a collaboration with my colleague Stephen Celis, so if you ever have any questions you can hit up either one of us. We've been working on this for quite sometime and all building up, because we wanted to do this side project called Point-Free, a Swift and functional programming video series. We wanted to do the website in Server-Side Swift of course, and then we wanted to see what would functional programming and things like that give us in a Server-Side framework. The best part is that, everything is fully open source. If you go to the organization, everything I'm going to show here is there. The entire website, so far of what we have built, is up there. We'll be developing it open source as we go, like payments and everything will be there. Let's get started.
There are a lot of layers to a web framework and I'm only really interested in one of them, so let's, everyone get on the same page about what I mean by a web server and all that jazz. There's a lower level, which is the layer that I'm not really interested in, because I think it's all kind of the same. This layer is responsible for socket connections and TCP and HTTP parsing and SSL and all that stuff.
The ultimate goal is to produce a URL request and you can even think of it as the foundation URLRequest, that can be handed to a higher level framework. It has a URL, a method, an optional post body, some headers, that's pretty much it. All that lower level produces this value. Then we get handed off to the high level, which is the thing I am interested in. It's responsible for interpreting the URLRequest. It picks it apart and figures out what code execution path to take. It fetches data, so it hits Postgres or Redis or Memcached or who knows what else. It renders a view. Once all that data is collected, it now needs to produce JSON, HTML, XML or whatever.
It's ultimate goal is to produce a response and this is a status code, some response headers and some body data, optional body data. It is not unreasonable to say that the web server framework on the level that I want to think about it is just a function from request to response, just a function. In fact, going deep into just thinking of it this way can expose a lot of really wonderful compositions that are hard to see if you overly abstract what we're trying to do here. This is where all my work has been.
My goal is to break up this one single function into as many small composable units as possible. Things like routing and data fetching and middleware and view rendering and all that, needs to be small composable pieces that glue together to make this function. It can all be expressed with pure functions and so we're going to take a lot of inspiration from functional programming.
I'm going to show four components of this function and then we'll dive into those components.
The first is middleware, and this will be the actual atomic unit of composition that all the other ideas will be built off of.
There's routing, which is the thing that picks apart the request to produce a first class value that we can use and a sufficiently advanced router should be able to pick apart anything from the request. It should be able to match on scheme and host and path and query params and method and body and all that jazz.
Then there's data fetching and this is where we go into side effect plan and things get very complicated and difficult, but you can still make a pretty reasonable story out of it.
Then it's time to render the view.
All these pieces can be described with just pure functions and things get really nice.
Let's dive into middleware. Naively we may want to define middleware as a function from request to response, but this isn't going to be useful, because these things don't compose. The output doesn't match the input, so it's just not what we would want to use as an atomic unit of composition. Better would be to create one type that has both request and response in it. The request is, read only, it's just solidified in time, but the response is something that can change with each pass of this function. You just slowly build up the pieces of a response; the status code, the headers and all that.
These things compose, because output is input. You can have lots of these and just compose them together, but better would be to, even better would be to now make these things generic over some parameter. Now you can actually store first class data inside this Conn type, so that as you go you're like building up more and more data of your models and things like that. Then, a great way, like a really advanced way is to make it generic over an additional parameter that won't actually even be used anywhere in the type and this is known as a phantom type. This is the actual type, this is pretty much the exact type we use, and you'll notice that the I doesn't show up anywhere in the type at all.
This allows you to actually encode a kind of state machine in the type system where you are not allowed to do invalid things on certain states. Like I'm doing broad strokes, so I don't want to go too deep into this, but fellow Swift Summit speaker Brandon Kase, who confusingly shares my name, gave a wonderful talk at the Functional Swift Conference in Berlin and I highly recommend you watch that. He also does a Swift Talk episode with Florian and Chris Eidhof.
These are the states that we will travel through with that phantom type perimeter. They represent all the stages to get us from a request coming in to a response coming out. In the first stage, we're only allowed to write the status of the response, in the second stage we're only allowed to write headers, in the third stage we're only allowed to append data to the body. Then finally we close the response, we end it and now we can't do anything with this value except send it off to the browser.
Here's an example of such a middleware. You give it a status code and it will give you back the middleware that transitions you from StatusOpen to HeadersOpen. After you have called this function you are not allowed, enforced by the compiler, to ever write this status again. This fixes a terrible, terrible, anti-pattern in Rails, in which you're allowed to throw in an exception at any layer of the framework, and then that bubbles up to like a 404 or 500 or something like that. It's kind of bonkers that just anywhere in the entire stack of the application you can change the status from what should have been a fully successful, nice response deep in a view- you can all send through an exception and just wipe everything away.
Now we're just like forced to do all of our work, to collect all the data and write the status before we move on to the next stage. Then we would have two pieces of middleware for writing individual headers. One transitions you from HeadersOpen to HeadersOpen, because you can write as many headers as you want. Then once you're done writing headers, and you never need to write another header again, you close the headers. By the time you've finished this stage, you need to write your cookies, write your content type and all that.
Then you have a send which allows you to append data to the body of the response, and that transitions you from BodyOpen to BodyOpen, because you can stream data to the response. Then you end the connection, you go from HeadersOpen, or sorry- that should say BodyOpen, to ResponseEnded and now you're done. Now there's nothing you can do with the response ever again, you're done.
The beautiful thing is, this is just functions, we're just using functions. There's no other types involved, no structs, no classes, no enums, and so we know how to compose these. We don't need additional abstractions to layer these things on top of each other. You just compose. However, if you do it naively, you'll get something like this and it's not the best, because the thing that happens first is on the inner most level. You write the status, write a header, write a header, close the headers, send the data and end. That's not great. That's the downside to function composition in this way. We can fix it quite easily by introducing this arrow operator that takes a function on the left from A to B, a function on the right from B to C and gives you back a function from A to C, just doing function composition.
Now we get to write it in a very nice way. In a way which, it's very clear that the siteMiddleware is writing a status, setting a cookie, setting the content type, closing the headers, sending some data and then ending the whole thing. You can construct a little connection, pipe it through the siteMiddleware and you get a response back. It's just pure functions, there's no mutation or anything anywhere in here. Because it's so simple, just pure functions, no state or abstractions that you need to configure and build up, you could use it anywhere. You could use it with Vapor and Kitura and Perfect as just a small little experiment on one of your routes if you wanted to, and it's nice. Okay. Check, we've got middleware. There's a lot more to say, but just doing broad strokes and that's fun stuff already.
Next is routing. There is a really wonderful story to tell here. Routing is a notoriously tricky and there are a ton of approaches. We are most interested in leveraging as much of the type system as possible to give us really nice guarantees about routing. Let's outline the goals first.
The goal of routing in general is just to take a nebulous URL request and turn it into a first class value. Our goal is to make this as type safe as possible. It maps into an optional A, because not all requests are necessarily going to map into something that we understand. We could always just type gibberish into your URL bar. If the value's nil, you probably go to a 404 page, if not, you'll send it off through the rest of the middleware.
Now type-safe is kind of a term that gets thrown around a lot and isn't well defined, and I know I'm guilty of this. I can at least… I can't provide a good universal definition, but I can give at least like a relative definition in which, we're going to say that, a construction is more type-safe than some other construction if it can catch errors at compile time that the other construction can only catch at runtime.
Most of the routing solutions right now aren't as safe as they could be. Here's an approach that you could take in which you have those router thing and you say, all right when a get comes in, I want to pattern match on the path of the request. I'll say, you have to hit the slash episodes and then slash token in which I want to pluck out that token. If we match all that, we'll get into this closure and I'll be given a dictionary where I can now pluck out the ID.
Not a very type-safe because primers are stringly typed. I can't figure out what type the ID is supposed to be, unless I now actually go look into the code and see how the ID is going to be transformed later. The tokens in the little pattern do not match at all with what we pluck out later, so those things inevitably will get out of sync. Yeah, essentially I can make a lot of changes to this code that will compile just fine, yet will be a runtime error.
Another one where we could maybe layer on a little bit type safety in which the block, when the pattern is matched, the block that's executed now kind of spells out the parameters that expected to pluck out, then we can provide types in here and then somewhere under the hood we need to transform string to these types, but it can be done. The problem with this is, if the number of tokens on the left doesn't match the number of parameters on the right, you can have no choice, but to just crash, because you've produced more tokens or fewer tokens than the closure provided and that's like an invalid thing, so crash or just runtime 500 or something. This isn't ideal either. That's goal one type-safety, that's examples of things that aren't as type-safe as we might hope.
Goal two is invertible. This is something that most routing solutions don't attempt, in which, if we had a first class value, how can we turn it into a request such that then if we routed it, we just got back the first class value we started with. Why would you want to do this? It is very useful for linking within your site, like you have an episode, a little first class value sitting there, and you want to now generate the link that goes over to that episode, you could hard code/episode/42 or whatever. Or you could use the compiler to help write that for you, so that if you ever change URLs, everything just continues matching up.
Rails has a good story here, although it's dynamic and not type-safe, in which every route gets a little helper generated for it. You feed it an episode in this case and you get this path and that's good for using it like inker tags and things like that. It's still 100 possible and actually in fact quite easy to mess this up, but like at least the story is there.
Another goal is self-documentation of the routes. This is very interesting. It would be nice if given an A, produce documentation that shows what kinds of requests route to that A and even maybe describe the types of all the different parameters along the way. Rails gets part of this right in which you can rake routes. You just get a huge list of all the routes that are in your application. Okay, and the amazing thing, I'm here to tell you, is that this is all possible even with the type system in Swift. I think a lot of people would assume that you couldn't do this with types, that you need dynamic languages and it's built on a very old idea known as applicative parsing. It's a deep topic, lots of cool things in it, and I'll just give a quick demo of what it looks like.
Here we have a first class type with a bunch of routes that we're interested in. Like we have a root where you just go to domain/. You have an episodes page of all the episodes with an optional sorting you can do, and you have a particular episode in which the episode, you're allowed to give either the ID of the episode or a human friendly slug of the episode. You want to be able to match both of those things. Then also an optional ref tag that you can maybe use for referrals and analytics and stuff like that.
I want to stress that, I'm specifically not showing the happy path in this slide. This is real world messy data, unhappy path. This is like the kinds of things you need to solve in routing. How do we turn a request into a value that lives in here? All right, so you create a value called a router that describes how we want to parse a request and turn it into a value from the enum. I will 100% sympathize and say that, if I was not familiar with applicative parsing and things like that, this would look like computer burp.
It can be reasonable, you can understand it and if you squint really hard and give me the benefit of the doubt, you can maybe see some patterns in there, like the fact that ... Getting episodes means it's doing a GET, and then we do /episodes and then we pluck out a query param named order and it's optional and it's of type order. Whereas the episode you do /episodes/a path param that is an int or a string with a query param called ref, which is an optional string.
You can learn to manipulate these symbols and use it in anyway I would, and just like you manipulate pluses and multiplications and ands and ors, it's just algebra and don't let anyone tell you you can't do that. Then you use it. You say router.match the incoming request and you just switch on an enum, and you handle all the different cases in your website.
Why would you do this? Why would you deal with something like this? This is currently type checking Swift is happy. I'm going to make a change to it. I am going to take this optional string and I'm going to make it required. I now made it required. Swift is telling me, I expected an either <String, Int> or an optional string, but I got a required string. I don't understand that. So I made a change to the route and Swift stopped me from doing it. I'm not allowed to compile, that's good.
Now I added an additional query parameter, and I didn't make it, yeah, I added an additional query parameter, and now it's saying, I expected either <String, Int> string optional.. but I get some extra stuff here that I don't understand. Then I changed the int or string to be only string and now it's saying, well, I expect either string or int, but I only got string. All the changes I'm making in my router that are not compatible with my routes are compile errors.
Once again this is what it looks like when it's nice and type checked. Then you get all of the cool stuff out of it. I can generate URLs by just giving a first class value and say path two episodes order ascending and I get that URL fragment I can use in an A tag. I can say path two episode with a slag intro to functions with the ref Twitter and I get that URL. It's all done in type-safe matter, no code gen, no boiler plate, good stuff.
I can produce documentation for these routes. I can say the episodes path is /episodes with an order or query param and it's an optional order value. Or, an episode is /episode/A string or an int with the ref tag optional string. It documents itself. It's all built on applicative parsing. Applicative parsing subsumes all ideas that you have ever known about routing. You can do name spaces and nesting, so if you just want to have a bunch of routes in /view one. It can do CRUD resources, so in Rails there's this idea that you can just create a resource where you get all the post get, put, delete and all that stuff, just kind of attached for free. You can do that.
You can do responsive routes if you just want to only match on JSON or XML, and more. I want to stress that, I didn't set out to build any of these features, and in fact I haven't built a lot of these things, but it's all possible, because it's focused on very small composable units that glue together in ways that I couldn't possibly imagine. Okay- check, routing.
Next is data fetching, and this is where we enter into side effect plan, things are very complicated here. Last night and it looks like I did the right thing here, I decided to cut these sections. I'm just going to say, if you want to talk about some, tell someone about it, like hit up Stephen. I didn't tell him I was going to put his face on the slide, so I'm sure he'd enjoy random messages from people. All right, data fetching, check.
Then we get to view rendering and this is where it gets really fun, and this is where the whole Swift tool chain really shines. View rendering, apart from the method will be template languages where you have a file with HTML in it and you get little tokens that you can use to interpolate values and maybe get a little bit of logic constructions in there and things like that. That's like mustache and handle bars and stuff like that.
Templates are highly unsafe and way less expressive than they could be. We built a full HTML DOM library represented in just Swift value types to handle HTML and this is what it could look like to build a very simple document. It's just structs and enums that you piece together into a tree structure. You can transform them, you can map and filter and reduce and do conditionals. Everything that Swift offers, you can do with this. Anything that you're familiar with with arrays and dictionaries and any value type you've ever dealt with, you just get to write functions that mess with these things and transform them. Then you can render it out, and now you have HTML that you can send back to the browser.
Then, so here: why would we do this also? Beyond all the transformation stuff that's a lot of fun, here things are type checking nicely. We can encode into the type system, all the semantic things that HTML expects. For example, I can add a P tag to a list which doesn't make sense, and now the compiler is saying, I got an unexpected node and something that I expected to be the child of a UL right here. I can throw a P tag into a top level HTML, also doesn't make sense, and so I get an appropriate error.
All right, so HTML is only half the story, there's CSS, that can also be lifted to Swift using just simple value types. You don't need SAS and SCSS and all that stuff. All that stuff is just creating a program in language, in adhoc weird programming language to piece together CSS in interesting ways. We have Swift to do that, so if we have lifted it to Swift, we won't need any of that.
Here I've got little fragments of CSS. They're not even tied to selectors. I just have a base font style, a base heading style, and then I can select into my H1 and take the base setting style and add color and padding. I can take H2, add base heading style, add color and padding, and then I can render it out to some CSS, but because it's just value types, I get to use them anywhere I want in a Swift program. I can render it inline into an HTML tag. I could render into a style tag on the HTML page, I could create a route for all the CSS on my site, render it there, put a CDN in front of it, and now it's nice and performant and all that jazz.
I'm going to have to go fast now. Then testing. Testing is so easy, because it's just pure functions. We do snapshot testing, which I think is a lot of times associated with screenshots, but it goes beyond that. You can also snapshot textual information. Here we snapshot the homepage of the Point-Free.co website and now we just gave broad overview of what the resulting HTML looks like, that's actually going to go out to the browser.
Then of course you can do a screenshot test, and here's nine screenshots of the homepage at different responsive breakpoints in different states, generated by Xcode in the IOS simulator. Every time we run tests, every single pixels off, it fails the build. Here's an example of that screenshot. This is the homepage on a phone. When you signed up for, like our pre-launch thing, this is what you get.
You can also even develop a full on webpage in a Swift playground, because there was no site kick connection, there was no web server on the lower level. It was all just pure functions. I can't really talk about this too much.
In conclusion, when thinking about the future of Server-Side Swift, let's take the good ideas from existing frameworks, but nothing more. Like we shouldn't think about what suits Swift and what does not. We should leverage Swift type system to turn runtime errors into compile time errors. We should keep as much of it in Swift. Swift can do like HTML, CSS, keep it all in Swift. You need no build tools, no process, no anything. Look to functional programming for inspiration on how we can solve problems.
The entire picklet of router I showed comes from a paper I read from 2010 called 'Invertible Syntax Descriptions Unifying Parsing and Printing' which is what we needed to do. Lastly, focus on small composable pieces. I never met a pure function I didn't like, because it can just be reused and composed in ways I could have never expected. That's it. Thanks for letting me go very quickly through all that stuff.