Recently, my company has begun working on integrating the various services and products we offer. Since most of our customers purchase or use multiple services, it only makes sense that the data they have in one should be available in all (I know, sometimes the obvious takes a while). To do this, we're using .Net web services to allow this to happen. In order to protect these services, we're using code signing certificates and access gateways that verify these certs.
A problem we've run into is that most of our internal users in these environments connect via Citrix Xenapp servers. In order to limit the management of certs in this environment, the code signing certs are added to the local machine, rather than individual, stores. This allows any user of these machines to connect to the services without our having to make sure the certs are installed in everyone's profiles, etc. We had a release that was going live today that served as integration between two services, but found they couldn't talk to each other. It turns out that access to certs was the issue.
When installing these certs, we simply installed them by double-clicking them and following the import wizard. We'd tell the wizard to install them into the local machine store and then used the winhttpcertcfg utility that comes with Windows to grant only specific groups access to these certs. When put into place, though, on our test systems we found that no users could use the service. When we checked permissions on these certs, the users definitely had been granted access to them, but it just wasn't working.
When we had users log into machines that had the cert and asked them to run the command:
winhttpcertcfg -l -c LOCAL_MACHINE\TrustedPeople -s certname
They would get the following error:
Error: Access was not successfully obtained for the private key. This
can only be done by the user who installed the certificate.
When an admin, namely myself, ran the same command we'd see that the users had been explicitly granted access. This made no sense whatsoever, and in googling the error I found I wasn't alone in my confusion. During this search, I came across a utility that's put out by Microsoft called the FindPrivateKey tool. After using this spiffy little tool, I found that the cert was actually installed in MY profile. It was showing as part of the local machine store, but was in C:\Documents & Settings\me\Application Data\Microsoft\Crypto\RSA. It was actually supposed to reside in C:\Documents & Settings\All Users\Application Data\Microsoft\Crypto\RSA. Well, that would explain why it worked fine for me, now we just needed to find out why it was going there.
When fighting with a problem, I often find it's easiest when I've hit a roadblock to try and rethink the issue. Importing the cert via the wizard wasn't generating any errors, but it also wasn't putting it into the right place. All of the tools were giving me error messages or information that lead to dead ends. On a whim, I decided to change my first step and instead of running the wizard by double-click it, I'd trying importing the cert directly into the Certificates MMC. Now, a quick word of instruction: you can't just do Start -> Run -> certmgr.msc as this will just take you to your personal store. You have to launch MMC, add the Certificates snapin and choose Computer Account, Local Machine in order to get to the store you want.
When I did the import this way, I got a completely different error:
An internal error occurred. This can be either the user profile is not accessible or the private key that you are importing might require a cryptographic service provider that is not installed on your system.
Aha! I love different errors, especially when they lead me to the solution. So, after reading this article, I checked out the permissions on the RSA folder in All Users and sure enough they were different. Oddly enough, they seemed less restrictive than what's listed in the article, but I decided to set them as written to test. And, it worked! When I installed a cert this time, it went into All Users. If I ran the winhttpcertcfg command above, it would show that the users listed in the article were the ones who had access to the cert. In other words, all winhttpcertcfg really appears to do is set NTFS permissions on files. Ah, obscurity!
Okay, solution's now in hand, the day's halfway over, release is tonight and I have 75 servers that all need to have their configuration updated...whatever does one do? You break out the time honored skill of writing batch files, of course! I compiled all of the files I would need to make this work right into a folder. When running scripts across multiple servers, consistency is key. In this environment, I've striven for as much as I can, but these some servers are cycled on an annual basis as we retire old hardware. Consistency's an ideal, but not implemented perfectly. Compiling what I need and pushing it out to each server as part of the process really ensures it'll work.
The first batch file, setupfixes.cmd:
rd /s /q \\%1\l$\certs
I run this by issuing a for loop:
for /f %x in (serverlist.txt) do setupfixes.cmd %x
serverlist.txt contains simply a list of servers, one per line. As the loop iterates through, it simply replaces %x with the server name. This batch creates a new folder on the remote machine, copies all of the necessary files into it, runs a second batch using psexec from Sysinternals and then deletes the folder and all files since it contains our certs that we want to restrict access to. The second batch, permfixes.cmd:
cscript //h:cscript
L:\certs\xcacls.vbs "T:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA" /G everyone:;E1234589B /G Administrators:F /G SYSTEM:F /T /S /I Remove /SPEC B
winhttpcertcfg -g -i L:\certs\cert.pfx -c LOCAL_MACHINE\TrustedPeople -a Administrators -p password
winhttpcertcfg -g -c LOCAL_MACHINE\TrustedPeople -s cert -a "domain\client users"
winhttpcertcfg -r -c LOCAL_MACHINE\TrustedPeople -s cert -a everyone
winhttpcertcfg -g -c LOCAL_MACHINE\TrustedPeople -s cert -a Administrator
winhttpcertcfg -r -c LOCAL_MACHINE\TrustedPeople -s cert -a Administrators
I'm using xcacls.vbs because the permissions from the article are more granular than what can be set by cacls. Again, our build process specifies we set the scripting host to cscript, but that doesn't guarantee everyone's done that. Easy enough to ensure in advance, though. I set the permissions on the RSA folder to match what MS says is the default. I'm honestly not all that happy about these are they're fairly open, but I needed to get this out to production. I push these perm changes down the tree and tell it to remove inheriting permissions from higher levels. Finally, we import the cert, grant access to the users we want to have it, remove admins and everyone, and add the local admin.
Why remove admins, but add the administrator? We want to restrict access to this cert to a limited set of users. Admins like me shouldn't be able to use it as it would give them escalated privliges in the environment. But, we might need to do something with it at a later date, so we leave a back door. Since login as local admins is an audited event, we'll be notified of any potential security breach.
I've tried to include as much detail in this solution as possible because I had a hard time finding it myself. My hope is that when someone googles either "Error: Access was not successfully obtained for the private key. This can only be done by the user who installed the certificate" or "An internal error occurred. This can be either the user profile is not accessible or the private key that you are importing might require a cryptographic service provider that is not installed on your system." they'll come here and be able to fix their issues. I've put the errors in twice to make sure it gets indexed well. :)
