Taming Touches and Taps

Adam Bell at Playgrounds Conference, 2017


Video transcript:


Hello. My name is Adam Bell. I'm here to talk to you about touches and taps. For those of you who don't know me, I am a Canadian iOS Engineer. I am currently working on app experience at Facebook. I started out as an iOS jailbreak developer and basically, everything I do, usually involves animation, sounds, or some sort of interactivity. And yes, for those of you that are confused, that is Master Chief wearing a kilt.

So, history lesson. I just want to show you a few videos of some of the things that have changed overtime. Touch has been a very, very, very big piece of history in these past 10 years. I know most of you started as designers or developers, or people who have used your phones. Usually, there's some sort of touch interface that's been available to you all in this sort of period. So we started out ... and we had this sort of primitive stylist input with palm devices. They were palm pilots. They were very slow. You sort of tapped on a thing and then waited a really long time for a thing to happen, but the fact that you had the Internet in your pocket was like, "Oh, my god, this is so cool, cause I can do everything with it." But everything was so slow.

And so the reason why was because this was based around a resistive touch screen on an extremely slow device. Now it was stylus input on tiny, tiny targets. And there was a very large disconnect from user interaction to something happening onscreen. There was basically, no animations whatsoever. The screen would always flicker, frame dropping just left, right, and center, to the point where you'd just click a little magnifying glass to sort of zoom in the photo. It felt like ... it wasn't that great. And so, we moved forward a few years, and we get this thing.

Whoa, it has no keyboard. And then, did it just click. You never clicked a screen before. Is that supposed to happen? Is it supposed to feel so right? It feels like a keyboard, just no keys. What kind of man genius is behind this? Uh, right.


America's largest 3G network.


Right. Blackberry. Yeah, so rest in peace. But that was the Blackberry Storm, and whether you want to make fun of Blackberry all day long, that was really cool because that was different. It sort of had this capacitive, resistive hybrid where, to action on something, you sort of tapped the screen and then sort of force touch it, where the whole screen itself clicks down. And it was a slightly smaller disconnect, because you actually felt that you were doing something. You had this sort of like feedback telling you when you were making an action. There was still very little animation, you had some sort of rotation animation, some little minimal things, but that was basically as far as it got. And then, this thing came out.

Hang on. This is your music, this is your email, this is the web, and this is a call on your iPhone.

Oh, there we come. Basically, this is the reason why we're here. This sort of thing blew away the market, and it did this by many, many reasons.


The first one was its capacitive touchscreen. It was like way ahead of its time. There was multi-touch. It was super fast to do anything. And there was really no disconnect from you manipulating something onscreen. Like, if you did anything with your finger, chances are the screen was going to do something to react to it. You know you pinch to zoom photos in the Web, you tap on links to visit them, acts zoom in when they launch. You had this sort of spatial consistency. And now, we're at this point where this really hasn't progressed much more since then. We've got to this point where, you know, Apple has released like 3D touch, and Microsoft is working on like pre-sensing tactics and things like that. And those are really cool, and those are still working, but it's not there yet. It's nothing fundamental, just yet.

And, what I'm trying to explain is, basically, this mobile era has ushered in a giant touchscreen technology change. And it's been super, super, super fast. And that's only been accelerated by the apps that are actually used on the device. But you might be saying, "Well, why, what's the point? Like why are we making all these changes to do this sort of thing?" And well I mean, you can think of it this way. You don't need a touchscreen phone. A phone is a phone. You call people on it. You can just use a flip phone, right? You don't need fastlane. You can just use code signing, Marin talked about it the whole time. Or you can use fastlane Pools, whatever that is. I mean, you technically don't even really need Swift. You can just use assembly. It's there. It'll work. It'll do the thing. But, when you think about it, they're just improvements to everyday quality of life. What was just a nice thing became the expected thing.

And so, when you apply the same sort of logic to iOS you've got to think about, well what made it so good? It was a phone. It did its thing. But it did all of this with things that were polished to absolute perfection. Interaction directly manipulated content. They always did something and everything was complemented by some sort of animation. Buttons push down. When you open the screen, things would be highlighted. They would zoom in, they would zoom out, things could scroll. It was like totally revolutionary for its time.


And so, let's try and do this sort of thing ourselves. We can try and apply this same sort of polish. So I have this animation here. Well, it's not really an animation. It's a thing that shows a car going from one side of the screen to the other when you click on a button. And so, you might be saying to yourself, "Okay. I can do this. You know, me and my team, we can go through, and we can work, and we can hash this out. It's going to take a really long time. Okay, cool, we got it. Cool." This is what we're going to do. We're going to take the frame, we're going to offset it by about 500 pixels on the side of the screen. Cool, and we're done. Cool. That did exactly what we wanted it to do. Great.

But there's more things we can do here. How can we make this better? Well, what we should be doing is communicating how the car got from one end of the screen to the other, because in real life teleportation is not a thing. I mean, we have to explain what the car did and we can do this through animation. And you guys kind of already know how to do that.


Now, you and your team work together really, really hard. You turn out all this code. And eventually you come up with something like this, where you get a UIView animation. You get a second. You know, you push the car from one side of the screen to the other, it animates, great. And it looks something like this. And this is the expected behavior where it sort of moves and goes, and great. Now you have motion. You're explaining what's happening. So now what?

Well, we sort have to wait for the car to stop moving before we can do anything, and that's kind of lame. We know the car's going to end up somewhere at some point. And, if you really think about it, that's not really that useful for us. When somebody already knows where the car's going to end up, that action is already intended. We know it's going to go somewhere and we know where it's going to end up, so you don't really need to keep all of that there or have it interrupting what you're doing.

And so, if you think about it too, iOS has a lot of this waiting and it can be frustrating. So, if you think about it, if you pop back a UINavigationController. You know, you have a table view and you go back to another table view. You try and scroll, touches are dropped. It doesn't do anything. You know, you need to wait for a photo to open before you can do anything with the photo. If you unlock your phone, you have to wait before you can tap on an icon. The phone is super quick but you're spending a lot of your time waiting for these animations to complete and it can be frustrating. So how can you remove some of this waiting?


Well, this is a paradigm that is called interruptibility. And basically, the way you've got to think about it is there's constant interaction all the time on screen. Your UI should always respond to touches. Any display on an iOS device is always recognizing at somewhere around 60 Hertz. Sometimes it's more, sometimes it's less, but we should use that whenever we can. And with this comes interruptible animations. And interruptible animations ... I just think, "What are you talking about?" Basically it means, that any animation that's occurring on screen you should be able to interrupt it and stop it, and make it do something else. It should be able to be interrupted, cancelled, or ignored. And by ignored, I mean, just because an animation is happening on screen doesn't mean the rest of the UI needs to ignore those sort of touches. They should be able to still respond. The animation shouldn't be blocking any sort of touch input.

And this is how you create something that feels great. And, if you think about it, it's not just phones that have this sort of primitive. You know, video games have had interruptibility for years, and years, and years. If you thought of Street Fighter. Street Fighter engineers actually invented the idea of combos and they did that by mistake. They were never trying to bridge together punches and stuff like that, to do that. But once they realized that their controller input could be interrupted, then they realized, "Oh, wait. Now we have something we can work with. It's a combo system."


And, you go forward ahead in time, we get to something like Super Smash Brothers. You know, there's a lot of people here who like playing Smash Brothers. And there's these professional players that do the most obscene things I've ever seen with this game. I can't do any of this. But, if you stream together combos super quickly, you can actually interrupt the animation frames that the characters onscreen and move around the map way quicker than you normally would.

And even like, shooters. You've played Halo, you've played Destiny, Call of Duty. When you're switching weapons, you don't need to wait for the weapon to finish switching before you can shoot or before you can do anything else. Typically, you can switch weapons, reload, and then throw a grenade all in the same sort of motion, and that's because it doesn't make sense for you to have to want to wait. Because, if you've spent waiting, then you're not actually interacting with the content you're playing with.

But wait. There's more. You might be saying, "Wait. I don't play video games. I don't care about video games." Well, this is actually on your MacBooks as well. If you've used the OSX dock, I'm almost positive everyone in this room has used the OSX dock. It's entirely interruptible. You can put your mouse towards the bottom of the screen and you can click on an icon before it even fully shows onscreen. You can take the mouse in and throw it away, bring it back in. It's the sort of playful experience that makes it look like the dock is actually acting the way it is rather than you're waiting for things. You shouldn't have to wait for things.

And, even Xcode, okay? The editor we love and we hate ... you know, their error dialogues actually work. As soon as a dialogue appears, you can hit the OK button and it will just go away immediately. You don't have to wait for anything. And, I mean, I swear they're just trolling us with this sort of interface where it's really, really great to show us error dialogues that just basically use us and make us feel powerless. What does this even mean, when you try this? What is a person supposed to do when they have this sort of thing? And you're just like, "Great. Cool. Like, I feel great." It's the worst.


So, anyways, what's a significant app? So going back to our app. The car's breaks, we should add those. You know, the car can change directions. We should add that too. So what does this look like? So you go and you think about it, and you're like, "Okay. This car. I animate it." And your just like, "But wait a second, I don't know ... " And you think, who cares. Like it's just a stupid animation in an app. Like nobody really cares if it can be stopped. And now I can test that theory. We'll try it out. We'll see what happens.

So, UIScrollView, is like this thing. Everybody's used it, okay? You know what it is. You can scroll a list. You can tap on an element. You can cancel the scrolling list. You can do whatever you want, and it just behaves the way you'd expect. It behaves as if something is going away and coming back. And it's still unmatched by anything. Like Android has tried to copy it. Windows Phone has tried to copy it. Loads of people have tried to copy it, but it still doesn't feel as right. Everything is fully interruptible in this interface and it feels right.

Now, if we go and change that. What if we just say, okay, anytime there's an animation on the screen we're just going to ignore all your touches. Now this looks totally broken. Could you imagine scrolling through your music library and you see a song you want, and you go and tap the song, and it just scrolls 3 pages away from you, and you're like, great. And you try and bring it back and you're fighting with it. And it's really frustrating. You don't want that. You can't do anything with it. You can't scroll it faster, you can't switch directions, you can't stop it. You just ruined UIScrollView. That's what you did.


Here's another example. The photos app, okay? Photos are presenting. You have this grid of photos behind you now. Point Reyes, some random place in the world, I don't actually know where this is. You want to go and check out those other photos after you've thrown this photo away. So you're basically in the middle of dragging something away and you're about to throw it away, you can see that stuff. Why can't you touch that? There's no reason why I shouldn't be able to touch that, but it's just because the animations themselves are set up to block the interface. You're forced to wait.

And so, without interruptibility, apps just feel laggy, unresponsive, or just completely broken, when you really realize that it's there or when it's missing. And so you start getting these reviews in the App Store, that are like, "You know, I paid $2,000 for this 13-4 phone with like all this ram and I have to wait 3 hours to do the most simplest of things." And you don't want to do this to people. It's not nice. And you don't intend on doing it, but once people start using your app more, and more, and more, they want it to be quicker, they want it to be usable, but they just realize that it's not quite there yet. And that can be frustrating.


So, the way I think about it, it's like 10 years after the iPhone is launched. There's no reason that apps nowadays shouldn't be interruptible in some way, shape, or form. And it's because people want to become quicker. They want to become better at using things. They want their phone to be their sidekick. It's with you all the time. It's doing what you want. And you want it ready for when you want to use it.

Think about it this way. Historically speaking we have horses, and they were our primary method of transportation. And we wanted to go faster. We wanted them to be more reliable. You know, they sort of broke down, they're just not that great, they leave a mess. And so we decided to take 200 of them, put them together, add some metal and we get this thing. And it drives us now from point A to point B. That's progress. We're moving along.

And so, we have this framework. We build the Web on it. It runs. It's kind of slow. And we're like, "We're going to make it faster. So what we're going to do is, we're going to add yellow and give it a new name. And great, we fixed it. Yeah." But you know, iOS has actually been fixing things. They've been making noticeable improvements over time. You know, now in iOS 10, if you unlock your device, it'll actually cue the icon that you tapped as the icons are still animating in. Same with opening a folder. You can cancel it. Safari’s tab view, if you've ever used that on your phone, you can grab all three of the tabs that are open and throw them away all at once. It's very, very interactive. And UIViewController animated transitioning, as much of a nightmare is that API is, their interruptibility is slowly starting to get there. They're actually acknowledging that we can make improvements here. And UIViewProperty Animator is a good thing to see. They're trying to make things interruptible.


And so, let's go back. What do we want to accomplish here? Okay, we want interruption to always be there. We want interaction to always happen. Things should be interrupted, cancelled, or ignored. And never, under any circumstances, any dropped input. We don't want that. Here's the way I think of it. Treat UIViews, or layers, like real world objects. I mean, just because they're moving doesn't mean they can't be stopped, grabbed, or re-thrown. If you have a speeding car and you stick a brick wall in front of it, well, it's probably not going to be speeding anymore. It's going to be a hunk of metal. It applies to a real world example.

Let's try it out. Okay? We've got two things that are speeding. We're going to let them go. And, when they crash into one another ... yeah, that's great, right? No, I'm not going to do that. I'm going to try something else. Okay. This is the sort of interaction that we want to build, where you see ... when I click this car, and I can grab this car and re-throw it in any sort of direction as it's animating. This is a very, very playful UI, and it actually allows this sort of interruptible behavior.

Now, I'm going to switch over to where we're going. And so, I can show you how this works in real life. Well, kind of real life. Anyways here. So, two things to know. You have a green button that means go, and this red button is when you want to be a moron at a red light and just rev your engine to basically annoy everyone around you. Now, I can click this as much as I want and it's still going to be interrupted as I'm doing this. I can also click the go button as much as I want and it's going to shift, and stop, and reverse in place. This is how an interaction should feel. I can also grab the car and throw it, and throw it the other way, and throw it one way. And you can see that with the velocity of the mouse that I'm moving it, it's moving in the exact same rate at which my mouse is moving onscreen.

You can apply this directly to touches. If you take something and you throw it, it should move in the exact same velocity and sort of decay over time. So switch back to Keynote. Right. Now we can go back. Now we can fix this problem. Okay. So we've got an animation running. What are we going to do? We're going to try and stop the animation. And how do we do that? Well, you kind of really can't. UIView animations are actually really limiting. They kind of work, they kind of don't, because you can try these things. You can be like, "Okay. I'm just going to like remove all the animations on the layer." And that's not really good because it ruins everything. You don't want to do that. So you're like, "Okay. Well, I'm going to try and look through and find a certain animation key, and try and remove that." But UIView animations don't really tell you what's animating, so this could change at any point and this could totally break.

You could also think, "Okay. Well, what if I just start another animation for zero time and replace the current one that's running..." No, don't do this. Come on. So, we're like, okay, we're going to put our best heads together. We're all going to work together and we're going to come up with Core Animation. We're going to do this. It's Core Animation. Drop down really low level. Do this thing. And you try it and then, it looks something like this, where you sort of remove the animation after the car runs and it just sort of teleports back. And because what it's doing, it's actually going to the initial frame that you started when you started the animation and that's not what you want. And now you're all probably like, "Ah, I hate this. This isn't fun anymore." And that's because anytime you remove an animation from a layer, you need to reapply the initial value from the presentationLayer to the modelLayer.

Okay, okay. We're going to do this. We're going to set it all up. We're going to do it and fix it. Cool. Now it's all done. Should all this work, you did all this thing, by the way, if you ever thought about it, you should use POP. POP is this great framework. It's a very similar API to Core Animation. Interruptible animations are really a lot easier to do. And it's open source, so you can contribute. And the nice thing is that once you remove an animation, it'll freeze in place. It'll stay exactly where it is.


All right. We're going to switch gears a little bit. So I wrote a little app to sort of show this in real life. And we're going to go through and fix up some of its animations. So this is a camera app that I call Pickment. And it's kind of haphazard and it's not really architected in the best fashion, but it should be good enough to show us what we sort of want to accomplish here.

So, over to Xcode. Cool. So we're going to run this app. And because it's running on a simulator, we're not going to really have a camera live input. If I have enough time, I'll run it on my phone and show you guys how it actually works in real life. So here we go. So we have a camera button, we click the shutter, and it shows a photo. Cool. Now we can share this photo. Oh, sorry. There we go. Cool. So I'm going to throw that away. All right. So now, click the shutter button. Now your photo shows onscreen. Cool. Great. You want to share it, you can do that. If you don't want to, you can throw it away. But the problem is, is that when you do this, this shutter button ... you can't do anything with it. And even if you throw it away, you still can't do anything with it until it's fully off-screen and that animation is terminated. That's not the best experience. So what you can do, the first thing, we have a few things we're going to try out.

So, here we have this darkening view. This is what happens when it presents over top of the camera and sort of darkens everything to give this sort of shadow effect. And what we're going to do is, we're going to initially just allow all touches to go right through it. It shouldn't block anything. There's no reason why it should be there.

And what we're going to do is we're going to hop over to our presentation method. And right now, what we have is, we have a presented animated. So what this does is, it does a standard UIView animation, where you sort of present the photo onscreen over a given time period, it scrolls it into view and that's great. We're actually going to replace this with a slightly different one, where we're actually splitting up the presentation animation and the dimming animation, such that, when you present it onscreen, it's done with a POP animation and we're actually flipping on the darkening views ... user interaction ... to actually block ... Sorry, it's already initially set up to block it and, when we remove it, we're going to undo that sort of blocking behavior. So just dismiss with velocity.

We're also ... what we're going to do, we're going to take the velocity of the current gesture recognizer that it's recognizing, and forward that through to the animation. So, here we go. We have our pan gesture recognizer. So the first thing we have to do is we're going to take all the animations and pause them as soon as the touch comes down, because we want the photo to stop in place. Then, depending on whether or not we scrolled up or we scrolled down enough, we're going to cancel the animation or we're going to bring it back onscreen. So we can do that. And we're going to cut these guys out and run it again.


So now, when we actually present it, we'll still be able to cancel the animation or throw it, and allow it to present onscreen. Or, let's say we mess up the photo we were taking and we wanted to take another photo. We will have that opportunity. So you click it here. You'll see that I can present like tons of them. And that's because we still allowed this sort of user interaction to pass through. We don't want that. We want this to be selective. So we're going to go up here and we're going to disable user interaction on the darkening view. And once, and once only, once we decide to dismiss it, we're going to allow the camera shutter button to be clicked at that point. Because, once we've actually decided that, yeah, this photos going to go off-screen, everything else below it should still work. So we're going to allow that to happen.

So now, once we actually begin this animation ... we can do it. Right. Cool. So now, photos going to present. You can grab it and you can throw it down. And you can still grab it and you can throw it around as much as you want. You can still click this cause I didn't set the code. It's fine. I don't know. Anyways, the point I'm trying to display is like ... these sort of things, even though they're presented here, I can still grab them and present them, and throw them off-screen as they present in. I don't need to wait for this sort of interaction to end before I can throw them off-screen. That's the sort of interruptibility that we want to accomplish. Cool. Go back to the slides. All right.


So, like I said, we sort of want the selective interruptibility. It doesn't necessarily mean that all animations always have to be interruptive all the time. A more accurate description is they should just never really block user interaction when and ever necessary. So in our take away, we made the shutter button always work. I mean, we kind of broke it, but it's always supposed to work. The photo that was captured can be dismissed at any point. This means that, if you ever mess up taking a photo, you can take another one right away without missing that perfect moment. And the shutter button also works immediately, because it was never part of this animation stream to begin with.


So, here's some pro tips. Do not have any sort of navigation blocked by interaction ... animation. You want it to always feel interactive. And, if that's not possible, just selectively disable the interaction. And, if it'll break things, then you should change things to accommodate it. Like, if you can't present multiple photos onscreen, that's probably not a great thing and so you should figure out a nicer solution for how that would work.

And, any translation that you do onscreen should be relative to what the gesture recognizer is. If you put a touch down on the view, anytime you translate it, you should translate it based on where your touch was initially touched down. That way it's always relative, so you can grab it, and throw it and grab it, and throw it again. And you should always see the velocity whenever you can.

And lastly, have fun with it. Treat views as real life objects with weight. If it makes sense to grab and re-throw something, then just do that when you can.


And, in summary, interruptibility requires a little bit more work for you. It can be sometimes frustrating to build, but it empowers the user to use your apps and be more playful, and get good at using them. It leads to a more polished, understandable, and delightful user experience. You know, here's the way I want to think about it. We spent the past 10 years to finding to what makes a great iOS app, so we should spend the next ten making it even better. And I just want to leave you guys with a quote from a video documentary that I saw. It was called, Why CG Sucks (Except it Doesn't). It's by RocketJump Film School.

This: Visual affects artists combine incredible artistry and technical skill, toiling over shots for hours. And, if they do their job well, nobody even notices it, but they still do it. And, if that's not the definition of love for your craft, I don't know what is.

You guys are all super smart and talented people. If you push yourselves to build the next cool thing, you get to be that person that says, "I built this and this feels great." That's what I think is great about building iOS apps, and that's why I love interaction and animation so much. And I think, if you can empower someone to really enjoy using your thing, then I think you did a great job as an engineer, or you did a great job as a designer for coming up with that interaction. Thanks.


Links:

If you enjoyed this talk, you can find more info: