Let's Code: Storage for Structs

Opinionated Serialization With Generics


So, this is a slightly more accurate title than what is on the menu today. I hope it's not too much of a bait-and-switch. It seems like it's pretty similar. I'm calling this opinionated serialization because a lot of serialization libraries tend to try and do everything for everyone. And in achieving that, they're doing a lot of extra steps that you don't necessarily need to do. So, my goal here is to come up with something that works for 90% of use cases, and we have minimal boilerplate code to set it up. You don't have to do a whole lot of extra stuff to get ready. Let's get started.

Let's talk about serialization techniques.

A little bit about me, my name is Nick O'Neill. I'm a co-founder of a start-up here in San Francisco called Treat. I also do a little library on GitHub called PermissionScope, which gives you nice UI for displaying permissions request in your app, and maybe one more thing by the time we're done with this talk. Let's talk about serialization techniques. It's sort of like JP's topic, but on a slightly higher level. The first thing you probably think of with serialization is NSUserDefaults, which is a really simple way for you to serialize small amounts of data that your users rely on for user defaults really. Just little stuff that they may need to rely on for setting the screens and that sort of thing. And then, if you want to do something that's more complex, you generally rely on NSKeyedArchiver here, and it's okay.

Here's what I always think of when I look at NSKeyedArchiver. I come back to this post by NSHipster probably back in 2013, and it's comparing Core Data and NSKeyedArchiver. It goes through this very long list of features and comparing the two, but in the end, you end up with this. "Do they both persist state?" Absolutely. "Is one of them not a pain in the ass to use?" Yes, and that's NSKeyedArchiver. But of course, this is 2013, we work in Swift now. This to me doesn't look like not a pain in the ass. First of all, your thing here, it has to be NSObject. It's got to be a subclass of NSObject. It's got to be a class. It can't be a struct. It's got to be conformant to NSCoding, which means for every property that it has, it's got to have this convenience initializer which decodes all this data and cast it to a string and then you provide this default value of an empty string. If it doesn't end up casting to a string and then you have to provide the encoder, which just does the whole thing in reverse. And then, every time one of these things changes, you have to go back to every single line in there and add whatever it is you're doing. You're copying pasting, so that means you're making mistakes at the same time.

For Swift, it feels like this is very, very messy. So, lots of boilerplate, you got to be NSObject, no structs, the API's pretty clunky, and then once you do all of that stuff, you still have to make the decision about where you're gonna store this thing and write all the code about how you're bringing it out of the file and how you're putting it back into the file, and where that file is located. It's really a big pain in the ass.

Why do we care so much about structs?

Why do we care so much about structs? Well, if you're familiar with Swift, and I'm sure all of you are, I'm sure you've heard Andy Matuschak's talk about why structs are important, why value types are important, and why you should use them in your app versus reference types. This is related to how you can take something that you understand in your app. You can put some value type somewhere and copy it all over your app, and basically be guaranteed that by changing it somewhere later on, not meant necessarily understanding the method it took to get there, that you're not affecting some other part of your app, because that's been copied all the way along. Structs are also fairly flexible, they're easy to add fields to, they're more structured than dictionaries, you don't have to rely on magic strings to tell you how you're retrieving things out of this dictionary. And you don't have to make these key constants so you can keep track of all these magic strings. The ideal here is a struct archiver that doesn't require a boilerplate and is fairly straightforward to use, maybe a couple of lines of code to use. Now, let's get into some code.

The ideal here is a struct archiver that doesn't require a boilerplate and is fairly straightforward to use, maybe a couple of lines of code to use.

Let's try that. How about presentation large? That would probably be easier. Presentation large. Okay, here's the default, and I'll show you what this app looks like. This is our demo app here. It grabs a list of cities from some sort of API that lives somewhere. Luckily, this one lives on my device, because there's no Wi-Fi. What we expect to do is that we'll tap into one of these cities here and we'll input our favourite and least favourite thing about each one of these cities. Presumably, we upload this to a server and we can compare our favourite and least favourite things with our friends. It's a social network for favourite and least favourite things about cities. I love cheese-steaks. I like to spell them correctly, too.

Alright, we got that. This is our default implementation in this app. Now, we can go back here, and if we go back into Philadelphia, we lost that cheese-steak we put in there. I don't know, maybe we got distracted, we got a text message we had to respond to it. Your app got kicked out of the in-memory stuff even though my 6S tends to keep apps around in memory for way longer than I like to. There are lots of various reasons that this UI has been reset and the user doesn't have the state that they are at when they were filling out this form. And we want them to have this really great experience where they can come in and pick up right where they left off.

...we want them (the user) to have this really great experience where they can come in and pick up right where they left off.

So, we're gonna look at the code for this. The first thing we’re gonna look at is the TableViewController which is pretty straightforward here. We're gonna take just a moment and look at this thing called Unboxer. Unboxer is a piece of code... I wonder if I can switch back to my keynote presentation here too. Yeah, there we go. Unboxer is a piece of code written by this guy John, and it's a really awesome way to take JSON from some sort of a server and get it into a struct. It does what it's supposed to do really well in that you can take forms from JSON that don't necessarily reflect exactly how you would create your struct, and you can sort of manipulate that to get the object that you really want. I originally thought that creating this sort of KeyedArchiver thing for structs was not possible in the current state in Swift, and Unbox really helped me realize how we could do that. So, we're gonna do a very basic version of that now, and if you want to have a more rigorous look at how that would really work, I highly suggest checking out Unbox later on.

Back to the code. We've got this really, really easy request here that grabs some data. It uses Unbox to transfer the JSON which looks, just like that, into our structs. Of course, it translates it into a TableContents. So we've got this top level “items” thing, and items is just an array of this TableCity struct, and this TableCity struct has a name and a state. So, we fill out our table data here, and this is all very boilerplate stuff that you're all familiar with, I'm sure. Then when we get pushed over to our regular ViewController which has our favourite and least favourite things about our cities on it, we can represent this as a struct that tells us the progress of what our state looks like. So, what we wanna do here is we wanna develop something that will save the struct for later on, and then we can unarchive it, and fill it back in if there's any state available with it.

So, what we wanna do here is we wanna develop something that will save the struct for later on, and then we can unarchive it, and fill it back in if there's any state available with it.

Just briefly for planning, what we wanna do here is we wanna take our struct, we're gonna decode it into some sort of intermediate form which is something that feels like JSON, Dictionary is very easy to use. Then we can use the default stuff in Cocoa to write that directly to a file and we could do all sorts of things, we could do some sort of binary data approach like JP suggested earlier. Maybe not exactly like the one he suggested but something close that might be a little bit more efficient but we get all the stuff for free with Cocoa to write something, write this dictionary out to the file system as a plist file for free, no questions asked. So we're gonna use it. Then for unpack, we're gonna do the opposite. We can take the file, we can unarchive it into our JSON-like format, this Dictionary. And then eventually, we're gonna bring that back into a struct, and then use it when it's uncached.

Let's talk through how we're gonna do this. We need something in here that is going to... We're gonna write this struct in, once we fill in some data in the form here. So when textFieldShouldReturn, we're gonna use this summarizeState here which just makes our little detail progress item, our struct. Then we're gonna use this thing Storage.pack to take that struct and archive it with the key as the city name. That makes sense. We only wanna save it for the city that we're currently working on. So now, we have to go through and make this whole thing. I'm gonna propose that we make this thing called Storage, there's a protocol Storable, that's stuff that can be stored. What we wanna do for pack is make this generic function here that takes storable object, we make this JSONWarehouse. Now, as I said before, this could be anything that is binary data later on. For now, it's just JSON, it's easy to do. We're gonna get the JSON to that object and then we're gonna write it to the file system. That's pretty straightforward.

We're gonna get the JSON to that object and then we're gonna write it to the file system. That's pretty straightforward.

What we need to do now, is we need to figure out how to generate these JSON-like data structures from the struct that we're passing in here. The way we do that is by looking at our JSONWarehouse here. Using this toJSON function for a particular Storable object. Now, if you've used generics before, then you understand probably what's going on here and you can sort of see where we're going. But if you haven't, definitely take this as an ideal way generics can be used. I feel like generics are often used in ways that maybe are a little strange. People use generics to use generics. I think that this is truly one of those cases where there's no way that you can get around using generics. This is what generics are for. We are supporting the storable object to go to JSON. When we go through and use the reflecting API to look at that struct and find all of the sub properties in it, we're gonna cast those at some sort of storable default type which represents what we see in the stuff that can be cast right out of a Dictionary into string, float and that sort of thing. We're gonna cast those directly to JSON as well.

We come back up here. We'll declare our StorableDefaultType’s. These are the default types that Swift supports automatically and we will figure out where our errors are in pack. Oh right, we have to declare our DetailProgress as Storable. Here we go. So this is a storable thing here. Let's figure out what else we need to do here. Pretty sure we need to do Storable on the StorableDefaultType and we'll see if that works. It did not work. Dammit Xcode. Okay, we jumped the gun here. We still jumped the gun here. Okay, we're gonna move along and eventually we'll get to the point where we can do this successfully.

Alright, so as we pack all these objects in here, what we wanna do is... Oh, I see what we're doing here. We're missing all the functions in our warehouse and there we're gonna write all these types out, right? So, this is the stuff that I talked about before that you'd have to do if you were writing all of this yourself. You're gonna have to figure out where you're gonna write this stuff to, where you're gonna write this inter-media data storage to, and then we're gonna need to figure out what to call that and how to archive and unarchive that. Right? This is all very basic and we can use it for every type here, but at the same time you don't have to deal with it as an end user. Hey, that was happy.

This is all very basic and we can use it for every type here...

Okay, so we got our city list again. We're gonna go to Philadelphia, we're gonna say we love cheese-steaks. It seemed like we wrote it correctly there. If we come back to Philadelphia here, I didn't un-archive it. Alright, so we haven't done the un-archiving step yet. If our step to write it to a file was pack, our step to come out of it should be unpack. So, we'll come back in here to our ViewController here and we'll figure out where in viewWillAppear. We wanna do Storage.unpack on our city name and get our detailed progress struct back out again. You'll see if we get it out successfully. If it returns something that's not nil, then favourite and least favourite which are strings will just be applied to our text field. So we can pick up exactly where we left off.

Unpack now on our storable object is just the same thing- a generic function that does the reverse. It takes our same JSONWarehouse, it sees that the cache exists, and then it uses this warehouse JSON init function which we declare on Storable to bring that struct back to life as the way it was back before. Of course we need to make sure that our storable type here conforms to the warehouse, and then there's this get function. Now get is one of these things that we use on warehouse to pick through all of the unarchived intermediate form and get out each individual property that is on that struct. The best part about this is if it doesn't exist, it's an optional. So you can provide a default.

So we're gonna implement get here and get is very similar to our pack and unpack.

So if say later on we add a most favourite part to this, all those objects that we already archived will still be unarchivable and we'll still be able to provide a default to them as an empty string or whatever we provide the default. So we're gonna implement get here and get is very similar to our pack and unpack. In that it is a generic function that acts on a StorableDefaultType. Right? So it returns a StorableDefaultType if it exists in this dictionary that we've passed it along. Let's give that a shot. It is not happy. Okay. Oh I see. We did not declare that a warehouse had a get type. We go back to Philadelphia here and our cheese-steak is right where we left it. Of course that's what we wanted.

Now if our least favourite thing here is rocky and then we'll submit that, and of course this is our archived thing again. We need to expire this cache so that we can get rid of it when we don't need it anymore. So luckily that's also simple. We already have this removeCache item, our method on our warehouse here. All we have to do is when we submit, call Storage.expire, and then make sure our storage item tells our warehouse to expire something. Back to Philadelphia, upload, and it's safely expired. Hooray.

Okay, I have a minute left so I'm going to run through a bit of the end here which is, let's say we wanna expand this to different types. We dealt with one struct with a few properties that were fairly simple. If we wanna go back to the Table View Controller here, we can actually cache this whole array of TableCity’s pretty easily. All we have to do is implement this Storable on our TableCity and make sure we conform to the warehouse init method, and then go back and do pack and unpack. You're just gonna have to trust me on this because I don't have a whole a lot of tim e left. For those types that we're dealing with, which is an array of objects rather than just a single storable type. Right? So, let's check if that works. We got our list of cities from the server. We use this little plus button to archive it, and when we come back, it's there immediately. We don't have to wait for the server any more.

So, to do this for a bunch of other types, all we have to do is add those other generic types in, and then, what we end up with is something like this, where we do a toJSON for Storable, we do a toJSON for a default type, we do toJSON for an array of types, an array of storable, and we keep going on until we add all of those things together, and then, we can archive all sorts of things and get some really powerful caching with very, very little code here. So, that's it. Do we have any questions?


Q1: Hi. Really interesting looking library there. Thanks for sharing. I'm curious if you have any... If you could elaborate on your rationale for when you would recommend this approach for persistence to people, like what are the trade-offs you think we are making versus having a more relational database driven storage?

Nick: I think it depends highly on the amount of stuff that you're storing, whether or not it has relational links to each other. Right? If you want to cache something that's coming from a server and it doesn't have anything else associated with it, then why bother with a relational database? Why go through the process of dumping something huge into your app when all you really need is this very, very light struct storage?

Q1 continued: Right. So, things like user preferences you think would be...

Nick: Preferences is great. If you used your NSKeyedArchiver to do your preferences previously, then you could easily use this instead, because it's way less code for you to maintain and think about while you're working on it.

Q1 continued: I just asked because this is a little bit close to the vest for me. We're actually in the process of having a file system base storage model and wanting to move back to core data because we ran into a bunch of gotchas, like it doesn't have indexing, it doesn't have faulting, it doesn't have query ability...

Nick: Yeah, this is definitely not for those sorts of things. This is definitely not for anything that you want to have some more detailed way to pull objects out of and only get certain objects. It's for little stuff just like I showed, either state of a particular page or some user data, briefly caching some stuff that comes in from the server.

Q2: Something I was wondering about when hearing this is, and you've already said this is for small things, I'm thinking about something that might be larger. You might have a bunch of structs that basically are using copy-on-write to not take too much memory, and I'm wondering if one archived those structs in this way, and then unarchived them, you could end up with something that was in fact now much larger than your original.

Nick: Oh, for sure, yeah. There's definitely no protections for unarchiving something that's very, very large. If it was a copy-on-write before and you copied it a bunch of times and then archived it, yeah we've created a bunch of separate copies on the way out, or on the way back to struct land.

Q3: So, this is beautiful, just beautiful. For something very lightweight, I could definitely see this being very useful. I noticed somewhere in the beginning of your slides that you had like... By the end of this talk, I may have something. Would this be it? And if so, when will it be available on Git?

Nick: Soon? Yeah. We're gonna have a thing called Storage. It will be on GitHub soon. Yeah, and it will support structs, default types, embedded structs, arrays. I know that there are other projects like Unbox, the two dictionaries. So, probably dictionary at some point in time. I know there are other projects that even archive enums, which I'm totally unsure of how that actually works at the moment, but I assume that since other people do it, it's possible. So, I'll probably have that too as well, eventually.