Swift on the Server: State of the Union

Chris Bailey at Swift Summit 2017


Video transcript:


So, whilst this is today, I'm going to start by rolling back a year when I was lucky enough to present at Swift Summit last year. So a year ago we had just released Swift 3, which was the first official release of Swift, which had any support for Linux. Up to that point, Linux support had only been available in the development chain. So it was the first release where Linux support was actually there. We already had a number of web frameworks. We had Zewo that was at 0.7. We had Kitura, which had just released 1.0. We had Vapor, which had also just released 1.0. And we had Perfect that was already on 2.0, although they went to 2.0 from 0.8 so they kind of just bypassed 1. We also had IBM Cloud as the Cloud that was declaring that it had done the integration work to allow you to run Swift in a Cloud on Linux machines.

So, whilst it was the first release, we had already achieved a huge amount. But at that point, Swift 3 and Linux support were still very new. We had literally just finished getting dispatch into Linux and we had just finished getting a very early version of URLSession into Linux. Now, since then, of those four frameworks unfortunately Zewo has kind of fallen by the wayside, it's not actually made it's 1.0. But we still have three frameworks left. Of those, Vapor is now just in beta for Vapor 3 and Perfect is on version 3 as well.

So we have three major frameworks, all of which are actually really quite successful now. Kitura itself has doubled the number of downloads in just the last three months and the three of them are kind of duking it out for supremacy, playing something that looks like an advanced game of rock, paper, scissors. Where we have; Kitura drinks water, water soaks paper, paper covers bird- and the competition is actually very good. It's encouraging all of the frameworks to be more innovative. And if, for anybody that watches Big Bang Theory, there is room in this game for a lizard and a Spock.


In addition to the frameworks, we have IBM Cloud still, which now has hundreds of deployments of Swift servers. And we have a second Cloud which just went 1.0 a few days ago from Vapor. And Vapor themselves tell us that they have around about fifty customers running in Vapor Cloud today. We also have the server API Workgroup, so whilst those frameworks are competing on framework capabilities, there's a general understanding that there's low level API's around HTTP networking and security, which are fairly standardized and would really benefit from a lot of collaboration. So we started this effort over the last year and around about a month ago we produced 0.1.0 of the HTTP Server API. This provides a relatively straightforward way of creating Hello World. It doesn't compete with the server frameworks which provide a lot of extra function on top of it, but it's about standardizing those low levels and helping us bring more frameworks into the competition.

Just a few days ago, we raised a pull request to add TLS support, so that's HTTPS, into that server, and that pull request is still open and being discussed actively at the moment. So lots of work has happened in the last year, where we now have three large frameworks, which are being increasingly successful. We're trying to standardize the lower level and we have new frameworks arriving every day. And Brandon's gonna talk about one of those tomorrow.


But not everything's rosy, and that means yes, we need to talk about Foundation on Linux. So Foundation is a collection of around about 1700 APIs. Now, interestingly if you do some searches over GitHub, you will find that of the Objective-C files on GitHub that import any framework or any library, almost 40 percent of those actually import Foundation. So Foundation is used widely throughout the Objective-C ecosystem. And if you do the same analysis for Swift, you find that 90 percent of Swift files that import anything, import Foundation.

Having Foundation support was always considered to be a critical item for Swift on the Server and Swift on Linux. Because without good support for Foundation, being able to port and reuse code that's being used on iOS clients, to the server, just isn't possible. So of those 1700 APIs, on the day that Swift was announced as an open source project, around about 75 percent of those were unimplemented. So a huge amount of unimplemented code, at the point that Swift went open source. Now this was way before Swift 3. At the point that we reached Swift 3 GA a year ago, something like 42-43 percent of those APIs were unimplemented. As I said, we had delivered dispatch, we'd added in URLSession and a lot of critical APIs were there, but we still had over 40 percent of the APIs unimplemented.

Now, in the last year we've had some fairly stellar superstars working on Foundation on Linux, so the open source Foundation project. And these are the top 10 contributors. Now, one of the interesting things is, Philippe and Alex- those two work for Apple. The rest of the work is being done by community members, people outside of Apple. And there's been a huge drive and a huge amount of focus and effort from a range of people.

So if we go through some of them, we have Sho, he has done a huge amount of work, right the way across the codebase doing improvements in several different areas, mostly around improving code hygiene and code quality. We have Simon and John, both of whom are based in London, as indeed is Al Blewitt. And we have kind of like a growing group of people in London that are actually starting to work on these open source projects.

Simon in particular, got involved having met Ian Partridge, one of the IBM team based where I am- in the south of England, at a conference and they started talking and Simon started to get involved in working on the Foundation libraries on Linux. We have Bartek, he's been doing a huge amount of work on JSON encoding and decoding in Codable. We've got Sergey, who's been doing a lot work around URLProtocol and URL work. And we have three people from the IBM team who have been focused on implementing APIs, which are used for server work cases. And a lot of that's been around URLSession, URLProtocol, cookie storage, and so on, allowing you to take libraries which make network connections and use those same libraries on the server.

Just a couple of other people for notable mentions, they aren't in the top 10 committers but they have delivered just as many lines of code as the rest. So, Mamatha and Nethra, they've been working on URLProtocol, amongst other things, and those both also work for the IBM team.

So, in the last year, we started with about 43 percent of the APIs not implemented and we are now today down to about 22 percent. Now, if you think about what's left, a number of the APIs which aren't implemented, there's an argument that they're not that interesting or that they will never be implemented. So there's a lot of stuff around NSCoder, which has just been replaced with Codable. There is EnergyFormatter, which has not been done because it's really not that interesting on a server where it's plugged into the mains. There are stuff around measurement formatting which hasn't been done yet, but it's not difficult work and it can closed out when someone actually needs to do that.


So the big question is, is Foundation on Linux ready? Can I actually rely on it? And the answer is this: today IBM is announcing commercial support for Swift on Linux. That covers Swift, Foundation, and Dispatch. Now, that support is regardless of which server framework you're using, if you want to use Vapor, or Kitura, or Perfect, or another framework, or no framework at all. It also applies regardless of which cloud you happen to be deploying on.

Now, in addition to support for Linux, obviously there's support for the Kitura ecosystem as well. So there's commercial support for Kitura, all of the libraries on the IBM Swift repository, as well as some of the monitoring capabilities that we provide. And again, that's regardless of the cloud in which you choose to deploy it. Whether it is a cloud, whether you're running it on your own hardware.


Okay, so, also if we roll back to last year and the presentation I did here, one of the things that I really talked about was how software interop is hard. And how it's too easy when you have two ends of a connection, with a protocol in between, to do parsing and serialization and decoding the other end that gets things wrong. But what I said was, because we're using Swift, and we have Swift on the client, and Swift on the server, we can share code on both sides and we end up with a much safer solution. And I talked about being able to have a project that you deploy components to the client, components to the server and we kind of work out how to do the protocol in between. But because you're using the same code in both places, you have an inherently type-safe secure system.

So that's what I was saying, but in around April this year there was a conversation between Max and Soroush on Twitter where the question was asked, "You've been doing some server-side work, how much code are we sharing between client and server?" And you could imagine that I expected the answer to be, "Well, lots." But no. Soroush is a very clever guy, so this kind of makes me worried. It makes me think, "If he can't do it, then maybe a year ago that statement that I made was just wrong." So Max asked, "Well, why not? That sounds surprising." And Soroush said, "Well, it's difficult to share because you'd use stuff on the client and then you'd have to have a JSON representation and things change." And he said, "I would love to get to the point that I can share models, and maybe there's some codegen or some protocols that would enable me to do that."

So, this is really why that problem occurs. You start off on your client and you have Swift code and you declare a struct for something like a user profile, and that might have a name, which is a string. It might have a photo that I've stored as data and it might have a date of birth, which is a date. So I start off with strongly typed Swift structures. Then I need to transmit it, and the way to do that over networks, by standard today, is JSON, the JavaScript Object Notation.

Now, JSON doesn't really have types. You can have strings, you can have numbers, you can have booleans, and you can have arrays. But that's it, there's only four. So it's nowhere near as rich a set of capabilities as you get from Swift.

Now, another problem that you get is, so I've got a date of birth, which I've encoded as a date. But there are several different ways of storing dates as strings. It could be year-month-day, it could be day-month-year, it could be month-day-year, depending on which country you're in as to which is the standard.

So, because I'm transferring it as JSON and it's difficult to know how the string might be laid out, it's probably best if I convert that into three separate values. Then I can store them as numbers, and then there's no ambiguity as to which order they are in the string. Then, when it arrives at my Swift server, and I do my JSON decoding, I end up with a dictionary of string Any pairs. And that looks nothing like what I started with. So this is why Soroush was saying it's very difficult to share code.

And the good news is, as you’ve just heard from Priya, Codable is actually changing that and for me it's very much a game changer in the ability to share code between client and server. So, lets look at what you had to do before. So I would get some data and a would use a JSON parser to parse that data and I would end with my JSON type, which is actually a dictionary. The first thing I have to do is start working my way through that dictionary to make sure that the fields I expect actually exist, and if they don't return, an error. So I do that for name, I check to see whether it exists, if it does exist, I have to check it's type, I have to make sure that name is actually a string. And once I've done that I can move onto my next value, which is photo, which I said is data.

So I can check for the photo, and then I can check to see that it's a string. Because JSON doesn't actually implement data, when I take data and I send it to JSON, I have to make it a base64 encoded string. So first I have to check that it's actually a string, then I have to make sure that that string can be base64 encoded back to data. And once I've done that, if I ignore the date of birth case, I can actually create a new profile object from it.

So, I've written a huge amount of boilerplate code in order to take a JSON object and reconstruct the thing I started with on the client. Now, as you've just seen from Priya, when you use Codable with JSON encoders and decoders, this becomes radically simplified. So what I have in here, is reading in some data from my request and I'm requesting it's decoded to a profile type and then, assuming that works, I have success, if it doesn't, I have to take an error and I have to encode that to JSON to send it back down the wire.

Now, I can make this slightly better, I can mean that you don't have to read the data from my request, I can actually do that directly and just request a type comes back from the data. And I can do the same thing the other end, I can send a type and run the JSON encoder under the covers. And that simplifies it again.

So using Codable drastically simplifies what a developer has to do on the server. But this is actually only a small part of what you have to do if you're writing a web API on a server. You actually have to wrap it in all of this. So at the top I have to register my handler with the server, which is called storeProfile. Then, inside storeProfile I have to deal with a web request, so that's an incoming router request. I also get a web response, so I have web requests and web responses that I have to deal with.

And then I have this thing called next, which is kind of like a pseudo-completion handler. And the first thing I have to do, I actually have to check an HTTP header to make sure that the thing coming over the wire says that it's JSON before I can start doing the parsing. And then I have to take that and convert it into my type.

Now, if you're a web programmer, this might make sense. If you're a Swift programmer, it absolutely doesn't. No one writes Swift functions like this, unless you're doing web programming. So in terms of being Swift code, I would probably give this about a 3 out of 10.

So if I was to write an async function in Swift it would look a lot more like this: I would create a store function that takes a profile. So that's a Swift type. And I would have a completion handler that returns, with say another profile and an error, depending on whether it errors. And this is what an async function in Swift should actually look like.


So why can't I write a Swift server and web APIs that just looks like a Swift function? Well, maybe you can. So, Codable Routing is a new set of APIs which we're introducing. This is a Codable route: So I've created a store function that takes a profile parameter, and I have a completion handler where I respond with an optional profile for success and an optional error for failure. So, I've just written a Swift function, and that's gonna be my web API.

So, Swift types, which must implement Codable, that's the only requirement, hence the term Codable Routing, and my error happens to be of type request error, and that's because I need to encode things like HTTP errors into it. But essentially, I'm just writing, what looks like, declaring a Swift function.

Now, that's fine for storing data. Let's say I wanna get some data from a server, so it's the same thing except that what I'm gonna request is an identifier. So, we've extended Int and string to conform to identifier. You can create your own custom identifiers. And that identifier is a particular object that you might have stored on a server.

And I can do delete the same way so I can delete an identified item. And the one bit of web programming you actually have to do is then just register that with your server router. But I've declared functions that I want to be able to call from the client.

So, that takes us back to what we wanted to do, where I have a project and I can declare it, deploy it to the server and I have Swift types that I'm using. So, hopefully I can use that to the client using something that's called a client/server contract. I declare a contract of how the client and the server talk to each other.

Now, for this to work, really where I've declared functions on the server, I wanna also be able to call those functions from the client, using the same form.


So we're also releasing KituraKit. KituraKit is a bespoke client for working with our server. So, you create a client like you might with Alamofire or any other kind of client request. I just instantiate and pointed out where my server is. And then what I do is I make a request that sends those Swift types that implements Codable, so I send a profile and I get a completion handler that has a profile and an error.

So it's a mirror image of what I'm doing on the server. And I can do that for create and I can do it for delete and so on. And all of this is available right now in Kitura 2.


So, that's what you have today, but what could come next? So, now that we're talking about APIs on the client, that talk APIs on the server, and I haven't made you deal with a web request and I haven't made you deal with responses and you haven't had to deal with JSON, JSON doesn't need to be the transport anymore. We can actually convert it to something that maintains those Swift types between the client and the server. So no more having to take data and convert it to base64 encoded strings and back again.

The other advantage of Optimized Transports is JSON contains the name of the field and then the payload. There are transports like Protobuf, and Thrift, which are binary, strongly typed, highly compressed and use about a quarter of the data that you used when you're using JSON. And Tim's gonna talk about this I think later this afternoon, so that's well worth taking a look at.

So, now that you don't have to deal with JSON, I can optimize the transport. I can also do Implicit Security. You don't need to worry about certificate handling and how that data is encoded. We can do it for you, we can do it under the covers because we own both ends of the connection and you just deal with writing a call to a Swift function and receiving that Swift function on the server.

We can also deal with things like User Domains. So, when you're writing an iOS app, there's only one user. That's the user that owns the device that's logging into the app. On the server, you’re in a multi-user domain. You have multiple devices attached to that server. And we can simplify that picture for you by implicitly converting a device into a specific user when it's connected to the server.

And finally, we can do API Protocols. So, wouldn't it be nice if you could, as well as declaring types, which are used on the client and the server, so we currently talked about types that implement Codable and types that implement identifier and an error that's a request error. But those are just the data. Wouldn't it be good if we could actually declare a protocol, which is your API, in the same way that you would when you're doing Swift programming.

So, here's an idea of what that might look like. So, under what we have in Kitura 2 today with Codable handling, I can declare, let's say, a to-do item that implements Codable and it's got some fields. And I can share that on the client and the server and I kind of agree that my client's gonna call the same things that my server can respond to. If I wanted to define that, maybe it would be nice if I could declare a protocol and I'm gonna call that protocol todoStoreAPI and that's going to implement a bunch of functions.

So it will have a Store, it will have a Get, it will have an Update, and it will have an Index and a Delete, because I want to be able to store and get back to-do items. And once I've deployed that, on the server I'm going to be forced to implement them because when you create an instance of a protocol you are forced to implement the functions that the protocol declares.

So on the server I have to implement this. So I provide my user provided implementation. And then I also deploy it onto the client. On the client, I don't need to implement anything because KituraKit can provide a default implementation of each of those functions which makes a remote call to the server.

So I've declared a protocol, which defines my API. I've shared that between the client and the server, and the server's made me implement it, and the client's implicitly gonna call the server when those APIs are called. And then I can just create that connection and I can write my code.

And the other thing that we can do is, because those APIs are identical, if I wanted to I could run them in the same process. So for testing I don't need two processors and I don't need a network connection. I can just get my client to call those APIs directly.


So, all of this is possible and all of this is stuff that we are starting to do and starting to prototype and that we're discussing publicly on the Kitura evolution process. So, that was a taste of what we have today and what could be there in the future. Thank you.


Links:

If you enjoyed this talk, you can find more info: