Code signing demystified; certificates, private keys, profiles.

Marin Usalj at Playgrounds Conference, 2017

Video transcript:

Hello everybody. Today we're going to be talking about Codesigning.

I think so, yeah. What do you do? I'm sorry, I was forced to flip the slide. Apparently it's a running joke.

But yes. Codesigning. Woo! Come on, send some enthusiasm. Alright.

Okay, I will start with a quote from Apple. "Like a signature written with ink on paper, a digital signature can be used to identify and authenticate the signer. However, a digital signature is more difficult to forge, and it goes one step further. It can ensure that the signed data has not been altered."

So, why is this important? As a user of iPhone, you get a lot of benefits from using the code that's been signed. When you download, for example, a Facebook app, you're sure that you didn't just download a code that was written by Facebook, but you're sure that nobody touched it in the meantime. Also, it protects developers from unauthorized copying, which is a plus.

Of course, Apple has some of their own benefits of enforcing this. All the aforementioned customer benefits, plus it makes it impossible for programs to download more software and run it, and there is no competition for the App Store on iOS.

Before we dig in, if you're asking yourself, "Why this talk vs just using Fastlane and going home?" Reality is terrible, but in reality things break and things break all the time. Apple breaks stuff, you break stuff, your collaborators break stuff, Felix breaks stuff. It's also important to understand the underlying technology on how this all work. To not waste as many hours as I wasted. I think, if you combine all Apple developers, how many hours everybody wasted, it's, I would say thousands, but ...



Billions. Codesigning on iOS and OS10, it relies on public-key cryptography based on the X.509 standards. OSX Keychain Assistant is used for managing your infrastructure. As you will see, a lot of open source tools and standards are used in combination with some of the Apple's own.

Codesigning has been introduced on iOS from the first day of the App Store. On OSX, I think it was 10.8 with Gatekeeper, it's been getting stricter and stricter in the newer OS releases. It's one of the reasons why you cannot load random extra plugins anymore either, but I think it's a good thing.

There's a lot of pieces involved in this whole process. We won't cover all of it, for sure. We'll cover some of the most important user-facing tools and scenarios. The end goal of this whole thing is that, once you get stuck, you get going faster.

To start off, the whole story starts with the certificate sign request. In public-key infrastructure systems, a CSR is basically messages sent from an applicant to a certificate authority in order to apply for a certificate. I'm not the best programmer, but I'm a terrible artist. Hopefully, this will ... I drew a lot of graphics that might make this easier to all mind map and see how it fits together.

Here you have the developer generating a CSR. You can use the Keychain Assistant on OSX, create a certificate sign request. You input some of your data like your common name, email address, country.

Now, the important part, so remember this slide. Before actually creating a CSR, Keychain will create a private and public keeper. So this is really not obvious, Apple doesn't tell you it's there. It is there. The public key from this keeper goes from the CSR, and then it gets signed. The signature prevents an entity from requesting a bogus certificate of somebody else's public key.

The CSR consists of certification request information, the signature algorithm identifier, and the signature itself. Let's dig in. Okay, so let's see what is a CSR.

If you just open the file, you won't see much because it's a Base64 encoded BK cs 10 or B10. It's mostly used for encoding CSRs for X59 systems. It's expressed with X and 1 so you can use OpenSSL to parse it and see the information inside. For example, in this one you can see my email, common name, county. You can see that the key’s RSA, and the signature is signed with sha256withRSAEncryption.

You can also use OpenSSL to bring up the text contents of it. In this case it's basically the same stuff as you saw before. Subject in all this terminology will be yourself who is signing the code. You have the RSA public key and at the bottom you will see the signature.

Okay, moving on, I will mention a couple things about the key pair my, default is RSA 2048 bit. You public key will be embedded in those CSR, certificates, profiles and it's meant to be shared. The private key is used for actual signing and should not be shared, you should treat it like a secret, break it up, encrypt it. The last thing about this is creating CSR on several computer machines, it can be a problem because the certificate that it generates on one Mac will not be usable on any other Macs unless you have the same private key. So forgetting this is a great way to waste hours. I get angry.

Okay so we did all this work and talk because of a certificate, but what is a certificate? So basically it's a public key combined with some additional information that's signed by the certificate authority, in that case it was Apple, and basically just stating that the information on a certificate is correct. So certificates in practice guarantees that you the developer have built this code, that you're a member of the developer program and Apple have issued you a certificate to do so.

So how do we get a certificate? The same guy creates this CSR, sends to certificate authority, which is Apple. Apple puts some signer data and gets back a certificate. So what is in this certificate? Basically most of the information from the CSR, with some signer data, it has the validity so it's not valid before a certain date, it's not valid after a certain date, and then it has a signature from the authority that seals the whole thing. So if you modify any contents of the certificate this signature would not be valid anymore.

So if you want to look in the certificate, you can also use openSSL, they are usually in DER format so you've got a space for you there. I think default for OpenSLL is BEM. So if you want to be able to parse it specified in format, as you can see there is a bunch of metadata. The important stuff is issuer, subject, that is basically most of your info from the CSR. Validity, so a lot of your problems with codesigning and push notifications regarding the certs would be either the validity has expired or is not correct. You can also use the validity to kind of identify the cert. Then it has the public key, the algorithm for the key and then the same for the, oh sorry, it has some metadata saying like; what is this key used for? In this case you can see it's used for code signing the digital signature and then it has the signature itself.

Okay, moving on. There is one more thing called PKCS #12 and that is an archive file format for storing many cryptography objects in the same file. You will see later why this is important for us but basically it's commonly used in iOS development to bundle certificates with private keys. So remember there was a private key that was generated on the Mac, if you can imagine it, it's that same survey we created, with the other private key that was in key pair with the other public key. So P12 can be encrypted and signed and then if you look inside it's internal storage contents are called SafeBags and they can also be encrypted and sent. So if we take a look, what's in here you can see there's one bag, has some metadata about the cert and then it just contains the cert. Then there's another bag with some metadata about the key and it has the private key.

So a few more concepts, we have Team IDs, Bundle IDs, App IDs, Device IDs, I'll just quickly go over them. 

So Team ID is basically equivalent to a developer account or developer level account, so when you log in and go to like certs in the URL bar you will see that ID. You will see that ID probably in a lot of quotes saying in failures you have later. So the Team ID is related to Bundle ID. 

Bundle ID, you issue one for each one of your apps, you probably see it all the times, it should be in the reverse DNS format. 

Then combined together make an App ID, so App ID, oh sorry before that. We have Device IDs there's not much to say there, each iOS device has a unique ID, it doesn't change, and we will use it later for white listing devices in provision profiles.

So one more important thing in the whole chain is entitlements, and basically what entitlements are saying is, which system resources this app is allowed to use, under what conditions. All these entitlements have a default value which is mostly disabled and whenever you add an entry to an entitlements file you enable them. Some example of these elements are iCloud, push notifications, one controversial one is the get-task-allow, which basically allows external processes to hook on to an internal app. So you always want it enabled when you're developing or debugging and app, you want it disabled when you are shipping the app, Siri etc.

With the entitlements- enable only the ones you're sure you need and this will minimize the damage potential if someone exploits your app. Entitlements are stored in PLIST format, you can also inspect them from alright compiled and signed app. So in this example I already took the compiled app or IPA unpacked and used codesign to get the entitlements out. As you can see this one had a get-task-allow false and has Siri enabled and way more of them that are hidden.

So what ties all of these things that we talked about together is; Provisioning Profiles, and this is where things are complicated. So when you put all this together we get this unique App ID, can run on all these devices with the trust based on the certificate. So in this example we have a Yolo beta app, sorry this is a provisioning profile for the app. Profile has a unique ID, expiration, contains the app identifier and then the cert that we created before and then entitlements that we specified and then it makes a profile. So provisioning profiles, it's not a PLIST, it might look like one with some gibberish on the bottom. It's actually CMS, standing for Cryptographic Message Syntax so you can use security to inspect profiles even on the already built and signed app.

So in this example this is the embedded mobile provision file. If you take a look it has the app name, team name, creation, platform, you can also derive the certificates out of it. So in this case this is a Base64 signed certificate, if you just copy this all out Base64 code, you will get the certificate that was used to sign this. It also contains all the entitlements that we've seen before and then it has some more data like app name, this one is enterprise profile so it will just work on all devices and then it has a unique identifier that you will also see in some codesigning failures.

Okay, now we've got all the concepts done, we can go over signing. So when you sign a code, a signature consists of a seal, digital signature and some code requirements. A seal is just a collection of some checksums or hashes, various parts of the code. The codesigning software encrypts the seal and then creates the digital signature, that usually guarantees the seals integrity. So after the building is done we perform the signing and we sign all the individual components of the app. We also create several digital signatures as you will see. If you have the universal code then each object or for each slice you sign separately and the signature is stored in the binaries, we also sign all the resources so all the sounds, images, nested codes, libraries, you can find them in the _CodeSignature/CodeResources. It's just a PLIST file, if you take a look inside you will see a list of all the resources and then their hash value for each key so which means if you later modify any of the images or sounds or add a resource or something the signature will not be valid anymore, and also it has some of the rules on the bottom.

So we can also verify the signature, let's say you download the app, download the IPA, you can also change the IPA to zip, unzip and it will find the app. For example here I am just verifying the signature of the Lyft app, so if you just run codesign verified. You see the signature's valid and then you just show it in the file, you just echo “wat”, and then you check the signature again and it'll say; the seal resource is missing or invalid and this -vvvv says we added a file that was not in the codesign.

So in practice you'll see a lot of those, you know one guy from your team is like "archiving's broken I cannot do anything" and then that master guy's don't worry I just dig in the provision profile I set him all up, you're good. Then you know 30 minutes later; fix it! And then the other guy's oh I fixed it as well. So by this point what's going even on? And then it goes in circles and three hours later you get this. So it's an enigma what happens when we press the button, don't press it. Basically one of the very common problems is, the one I even mentioned In the beginning, Keychain would create a private key on that guys laptop and all the, so it's crazy that a developer portal lets you download all these provisioning profiles and certs and everything else, which are completely useless, if you don't have this key they're useless. So if you open the Keychain app, make sure that if you look up the certificates make sure the drop down expands down and then you see the key with them. If you don't then you don't have a product key and then they won't work.

So tl;dr; codesign is deterministic, Xcodebuild is not, Xcodebuild ... don't just restart with CA jobs, don't brute force, don't wait five minutes, 10 minutes for it to compile and fail again on the archive. If it fails it probably failed for a reason, use all these tools that I showed you like inspect the profile, see which certificate was used, try to dig in and understand what part is missing. Always know which machine generated the certs and store them in those encrypted .p12 files. Put them in a private repo, somewhere where they're safe, post them on Facebook. Remember that you need a private key. So yeah unpack the .p12 files yourself so pack all this stuff, store it, fetch it on CI, unpack yourself, better than relying on various tools. Use .xconfigs if you can, .xconfigs are just text files that you can store various X configurations, like specifying the profile, specifying the certs instead of just relying on, storing them in the ppx project 'cause you can slip, like GitHub can hide some of the diffs and you can slip changes you don't want.

Alright my name is Marin, you can find me on GitHub @supermarin or on the interwebs, I work for Lyft, you can request rides in a couple tabs in San Francisco, or sorry in America, on your watch, on your phone, with Siri, maps. We are hiring, come talk to me if you are interested. I would like to say thanks to Lyft and Playgrounds for inviting me to be here and that would be the end of my talk.


If you enjoyed this talk, you can find more info: