Monthly Archives: October 2009

You are browsing the site archives by month.

SSL and Certificates - Part 3 of 3

Part 1 and Part 2 of this series covered the basic cryptographic concepts behind SSL certificates, and looked at how an SSL certificate is constructed and how it is validated. This installment will discuss what different kinds of certificates exist, some things to watch out for, and two big takeaways that will save you time, money, and aggravation.

Traditionally, an SSL server certificate, such as the Wells Fargo Bank certificate that we discussed in Part 2, were issued for the Fully Qualified Domain Name (“FQDN”) of the server the certificate is intended to secure. The certificate we discussed was specifically issued to “” This is called the “Common Name” (abbreviated as “CN”), and is specified in the “Subject” field of the certificate:

The Common Name Field

That means that if there were other DNS entries, such as “” or “” that happened to resolve to the IP address of the same physical server, and you pointed your browser at one of those, you would get a certificate error – because the certificate was issued to “,” not to one of those other entries, and the browser won’t be happy unless the host name in the address bar exactly matches the Common Name listed in the Subject field.

In recent years, a couple of new kinds of certificates have been introduced. One is the Multiple Domain, or “UCC” (Unified Communications Certificate) certificate. [Note: Yes, I realize that saying “UCC certificate” is inherently redundant – like saying “PIN number.” If, by chance, my former English professor is reading this, I apologize. But I’m going to do it anyway.] A UCC certificate contains an extra field called the “Subject Alternative Names” field, which can be used to list multiple subdomains that the certificate can be used to secure. For example, a UCC certificate could be used to secure “,” “,” “,” and so forth, provided that all of those subdomains are explicitly listed in the Subject Alternative Names field. That means that you must specify what subdomains you want listed when you purchase the certificate, and if you want to add or delete one, the certificate must be regenerated by the issuer (which will generally cost you more money).

In addition to the Subject Alternative Names field, a UCC certificate still has a “Common Name” listed in the “Subject” field. However, according to the X.509 certificate standard, if the Subject Alternative Names field is present, the client browser is supposed to ignore the contents of the Common Name field (although not all of them do). Therefore, if the common name is “,” but that common name is not repeated as one of the Subject Alternative Names, a browser that strictly adhered to the standard would end up with a certificate error if it tried to connect to “” This interaction between Common Name and Subject Alternative Names has some implications for mobile devices that we’ll come back to in a bit.

The other new kind of certificate is the “Wildcard” certificate. A Wildcard certificate could be issued for, say, “*,” and used to secure any and all first level subdomains. (E.g., is a first level subdomain; is not a first level subdomain, and could not be secured with a Wildcard certificate.) A Wildcard certificate does not contain a Subject Alternative Names field – instead, the Wildcard (“*”) is actually listed as the Common Name in the Subject field.

If you are running a browser that was released anytime since 2003, it should support Subject Alternative Names, and probably Wildcard certificates as well. In that case, your browser will be happy if one of the following three conditions is true:

  1. The host name in the address bar of the browser exactly matches the Common Name of the certificate. (Unless the cert is a UCC cert, in which case the browser is supposed to ignore the Common Name.)
  2. The Common Name is a Wildcard, and the host name in the address bar matches the Wildcard.
  3. The cert is a UCC cert, and the host name in the address bar exactly matches one of the names listed in the Subject Alternative Names field.

However, if you are using a browser on a mobile device of some kind, it’s a different story. Windows Mobile 6.x devices support both Subject Alternative Names and Wildcards. Windows Mobile 5 devices support Subject Alternative Names, but do not support Wildcards. If you’re not running a Windows Mobile 5 or 6 device, you’re going to have to check with the vendor of your mobile device. Some support both Subject Alternative Names and Wildcards, some only support one of them, some support neither.

So what? Well, if you’re trying to use a mobile device to synchronize e-mail with an Exchange Server, it’s usually done by pointing the device at the same URL that you’re using for Outlook Web Access (“OWA”). If you’re using a Wildcard certificate to secure your OWA site, and your mobile device doesn’t support Wildcards, you’re out of luck – it’s simply not going to work. However, if you’re using a UCC certificate to secure your OWA site, and the URL of the OWA site is also the Common Name of that UCC certificate, your mobile device will be happy even if it doesn’t support UCC certificates…because it will simply look in the Common Name field and find a match.

So here’s big takeaway #1: If you’re going to use a UCC certificate to secure multiple URLs, and one of those URLs happens to be the URL you’re going to use to synch email to mobile devices, make sure that URL is the Common Name of the certificate in addition to being listed as one of the Subject Alternative Names.

Another common “gotcha” involving SSL and mobile devices involves intermediate certificates. Remember that “chain of trust” discussion from the second post in this series? It is increasingly common to find that the certificate you have purchased to secure your Web site is not chained directly to the CA’s trusted root. Instead, there is at least one intermediate certificate in the chain between the trusted root and the certificate you purchased. This isn’t a problem for “big Windows,” because the browser is smart enough to sense that the certificate the server is presenting is not chained directly to the trusted root that it knows about, and to request the intermediate certificate(s) so it can validate the complete chain of trust. Mobile devices, including Windows Mobile devices, are not that smart.

Mobile devices depend on the server to present the entire certificate chain, including any intermediate certificates, at the time of connection. And the server won’t do that unless all of the intermediate certificates are present in that server’s own local computer certificate store. Installing the certificates into IIS for use in securing the OWA Web site does not automatically put them in the local computer certificate store – you must explicitly import them.

But why purchase a commercial certificate at all? Can’t you be your own Certificate Authority if you’re running a Windows Active Directory Domain? Yes, you can…if you don’t care about supporting connections from any PCs other than ones that have been joined to the domain, and you don’t care about supporting mobile devices. For example, when you set up a Windows Small Business Server, the wizard that configures that server for OWA automatically secures it with a self-issued certificate. That’s not a problem for any PC or laptop that has been joined to your SBS domain, because the very act of joining a computer to a domain inserts the domain’s own self-issued root certificate into the computer’s trusted root certificates store. But if you then try to connect from your home PC, or your mother-in-law’s PC, or any other PC that isn’t a member of your domain, you get a certificate error. At least with a PC, you have the opportunity to override the error and connect to the Web site anyway…but a mobile device will simply fail to connect, while typically giving you very little information about what the problem is.

You may or may not be able to manually import a certificate into the trusted root certificate store of your mobile device. Some mobile operators give their subscribers that level of “management access” to their mobile devices and some don’t. Some mobile operators provide special certificate installation utilities for their smart phones, some don’t. Sometimes there are workarounds, sometimes there aren’t. To our knowledge, there is no definitive list available of which mobile devices have their certificate stores locked down and which don’t. So the question is: How much is your time worth? The first time you (or we, on your behalf) spend a half day trying to make a mobile device work with an SSL certificate that wasn’t built into the phone, you will have spent more money – not to mention the time and aggravation – than it would have cost to go to a public CA and purchase a certificate that’s already supported.

So big takeaway #2 is: If you’re going to synch e-mail to mobile devices, do yourself a favor and decide in advance what mobile devices you’re going to support, then buy an SSL certificate from a public CA whose trusted root is already supported by those mobile devices. You’ll save money in the long run, and probably keep your blood pressure lower as well.

For more information on certificates and mobile devices, including a list of the trusted root certificates that ship with Windows Mobile 5 and Windows Mobile 6 devices, download the Moose Logic Technical Bulletin entitled Recommended Best Practices for Exchange Synchronization with Mobile Devices.

Which App Streaming Is Best?

For quite some time now, Citrix has had the ability to stream applications on demand, either to XenApp servers, or to desktop/laptop PCs. If you own current versions of XenApp, you can use it. Microsoft also has an application streaming product called App-V, which it evolved from its acquisition of Softricity a few years back. They recently announced that they were going to discontinue the App-V for Terminal Services licenses, and just bundle the rights into what is now (in Windows 2008 R2) called the Remote Desktop Services (“RDS”) CAL. So if you own Server 2008 TS CALs or 2008 R2 RDS CALs, you’ve got the rights to use App-V to stream apps to your Remote Desktop Servers a.k.a. Terminal Servers.

Not wanting to be left out of the application streaming game, VMware went shopping a while back, and bought ThinApp. They maintain that ThinApp is better - or at least safer - because it runs exclusively in user mode, whereas both App-V and Citrix App Streaming require the explicit installation of an agent that contains kernel components.

So what’s the real story? Which application streaming technology should you use? Which is really best? As is so often the case with IT, the answer is a resounding, “It depends.” It’s sometime frustrating, but the fact is that we work in an industry where there is often no single “right way” to do something. But today I ran across a blog entry over in the Citrix Community Blog area that did such a great job of delving into the differences that I thought it was worth linking to here.

Check it out and let us know what you think.

How to Ruin Your Weekend and Other Hazards of Mis-Configured HA

NOTE: This was originally posted in October, 2009, and may not be a problem any more with current versions of XenServer, as some of the more recent comments would tend to verify - but we will keep the post active for historical purposes. (added by blog administrator, March 16, 2012)

The Level 1 HA (High Availability) feature that comes with Citrix Essentials for XenServer may be one of the best ways to crash your whole virtual infrastructure if you don’t understand how it works and don’t design in an appropriate level of redundancy. This of course will lead to hours of down time, unhappy management, possible data loss, and lots of extra work for you (most likely on a weekend).

The basics -
HA is designed to monitor the XenServer virtualization environment. When HA is enabled, the administrator can specify which virtual machines (VMs) need to be automatically restarted if the host server they’re running on should fail. If there is a failure of a host server, HA should then automatically restart its designated guest VMs on another host in the XenServer “resource pool.” Note that the HA function does not “live migrate” the guest VMs, because when a host fails the VMs on that host also fail. Rather, it selects another host server and restarts the VMs on that host. For all of this to happen correctly, Citrix’s HA requires two things to be true at all times:

  1. Each XenServer must be able to communicate with its peers in the pool.
  2. Each XenServer in the pool requires access at all times to the HA heartbeat disk, which is shared by all the XenServers in the pool.

If either of these two items is not true for any given XenServer in the pool, that server will “fence.” The short definition of “fencing” is that the XenServer suspects – although it’s not absolutely sure – that it is experiencing some kind of failure, so to protect against possible data corruption it shuts itself down – essentially sacrificing itself to protect the data – until a human comes along and sorts things out. If the fenced server is in a correctly configured HA pool, guest VMs that were configured for HA restart will be restarted on a surviving XenServer.

Considerations -
So… you have two XenServers all set up and all your VMs configured just the way you like them, and you decide to turn on HA. Everything appears to be working until one of the hosts suffers a failure and goes off line. (Murphy’s Law says this will happen on a Saturday evening right before your BBQ party is starting.) With HA enabled, you would expect, based on the whole “High Availability” concept, that everything would be OK. Critical VMs should get restarted on the other host and you should be able to deal with the failed host on Monday.

Oh, but wait, remember HA rule #1? The XenServer host that is still running suddenly does not have any peers to talk to. It no longer knows whether or not it’s healthy so, in the interest of protecting your data from corruption, it does what it’s designed to do – it fences, and now both of your XenServers are down. They may try to reboot, but you are now in an endless loop of fencing, and to get it resolved, you’re going to have to know how to use the “xe host-emergency-ha-disable force=true” command to resolve your problems. (And if you don’t understand that last sentence, you’re in for a long weekend.)

This results in a situation that we in IT refer to as “not good,” with a chance of “career altering,” and you’re going to miss your BBQ party.

Here’s another scenario that will spoil your party: What if both XenServers are actually healthy, and all the virtual servers are up and functioning, but the network link for the management communications between the XenServers fails? Again, each XenServer would think it was stranded from the pool and fence itself in an attempt to correct the issue. With both servers fencing, this would again create an endless loop of server fencing. In essence, one server would start to come back online and would still not see the other XenServer and would fence again, and so on, and so on.

So for those reasons a two-XenServer pool cannot successfully run HA! Just don’t do it - even though you can configure HA on a two-server pool the result can be disastrous and ruin your weekend…not to mention your next performance review.

Well, what about HA in a three node XenServer pool? Based upon the previously described scenarios, you now have a valid “pool,” in which HA will function. So you configure and enable HA, and when you test the HA functionality by killing one of the XenServers, everything works like it is supposed to. The guest VMs are restarted on the surviving XenServer hosts and you’re happy that everything is working correctly.

But here is another “gotcha!” If you have only one Ethernet interface per XenServer assigned to management, and they’re all plugged into one switch, what happens if the management link fails because a NIC fails – or even worse, the switch fails? If it’s just a NIC in one server, then that XenServer will fence – not too bad but still not what you want. If you were using a different set of NICs (as you always should) for the guest VMs to communicate with the rest of the world, then the guests on that server were probably up and working just fine until the server fenced. Sure, the critical ones will restart on the remaining servers, but you’ve lost a third of the resources in your pool unnecessarily.

Now let’s consider what would happen if the switch should fail and you had only single management ports on each XenServer all plugged into just that one switch. If this happens, it may be time to dust off the old resume, because you have just lost your entire XenServer pool. Why? Because when the switch went down, all the XenServers lost communication with one another, and each assumed that, because it was suddenly isolated from the pool, it must be experiencing some kind of failure. Therefore the whole pool fenced.

Conclusions -
Citrix’s HA does not work in a two host pool, period. With a pool of three or more XenServers you’ll be OK if you design the infrastructure correctly so that there is no single point of failure in your peer communications. How? Simply by bonding together two NICs, dedicating them to the management communication function, and then splitting the bonded pairs between two separate Ethernet switches. That way you’re protected against both a NIC failure and a switch failure.

But you’re not out of the woods yet! Don’t forget HA rule #2 – servers need to see the HA heartbeat disk. This is equally important, and you must consider the topology of that side of the network (iSCSI, Fiber, etc.) and be sure it is also redundant. And if you’re using iSCSI multi-pathing (e.g., with a pair of mirrored DataCore iSCSI SAN nodes), be sure to manually bump up the HA timeout interval so that if one of the SAN nodes should fail, the multi-pathing function has time to fail over to the other node before the XenServers all conclude that the HA heartbeat disk is gone – otherwise, again, they will all fence. Our testing indicates that a two minute timeout appears to have an adequate margin of safety. The default setting of one minute (oops - the default is actually 30 seconds) is definitely too short. Unfortunately, this setting does not appear to be persistent, so if you turn HA off and then back on, you’ll need to manually reset the timeout interval again. (This is probably a job for Workflow Studio, but we just haven’t had time to work through the process yet.)

NO Single Points of Failure
HA will do a fine job of protecting you, if you build the network correctly. So make sure you’ve built in enough redundancy that you have no single point of failure, and enjoy your BBQ.

P.S.: If you can’t justify more than two XenServers, but you still have one or more critical guests that need to be highly available, there is a solution: Marathon Technologies’ everRun VM. But that’s another post for another day.

SSL and Certificates - Part 2 of 3

In Part 1, we discussed basic cryptography, and worked our way up to symmetrical encryption systems such as AES, which accepts key lengths as long as 256 bits. We also discussed why key length was important to a cryptosystem, and alluded to the fact that there are also asymmetrical systems. An asymmetrical system uses a key pair, such that anything that is encrypted with one key can only be decrypted with the other. The math behind such a system is way beyond the scope of this humble blog, so we ask that you simply take our word for it that such systems exist.

In most systems that make use of key pairs, one of the keys is made public, and the other is kept secret. This is generically called a Public Key Infrastructure, or PKI. Consider the following use cases:

  • If you know my public key, you can use it to encrypt a message that only I can decrypt - assuming, of course, that I’ve kept my private key truly private.
  • I can encrypt something using my private key that can then be decrypted by anyone who has my public key. And since the public key is, well, public, that means pretty much anyone.

The first use case has obvious benefits. But what good is encrypting a message that anyone could theoretically decrypt? Well, if you know my public key – and you have some way of knowing that it’s really mine – then you can be pretty sure that any message that you successfully decrypt with it must have been sent by me (again, assuming that I’ve kept my private key safe). That makes for a pretty good digital signature.

So, the question becomes this: to use some kind of PKI, how can I securely transmit my public key to the people who might want to communicate with me (or authenticate my communication with them), in such a way that they are confident that it’s really my public key? One way, certainly, to get it to you would be to physically hand it to you on some kind of storage medium – which could be a USB flash drive, a CD, or even a piece of paper. If that’s not feasible, perhaps I could give it to someone you trust, who could then give it to you and vouch for its authenticity. That, by the way, is the concept behind “PGP” (which stands for “Pretty Good Privacy”) – you establish a circle of trust, and when Bob sends you Jane’s public key, it’s up to you to decide how much you really trust Bob.

That might be acceptable for exchanging secure e-mail with your friends, but it’s probably not good enough if we’re talking about securing access to your on-line banking system. So how do you know, when you point your browser at, say,, that the server you end up talking to, which is asking you for stuff like your social security number and password, is really a server that belongs to Wells Fargo Bank? Enter the SSL Certificate.

The next time you’re on your favorite banking or shopping site, click on the little padlock symbol (exactly where that symbol is will depend on what browser and version you’re running…IE8 displays it right at the end of the address field where the URL is displayed). I’m going to stick with Wells Fargo Bank for now, and break down what you should see if you choose “View Certificate.”

First, you’ll see something like this:

SSL Certificate Screen Capture

It tells you briefly the purpose for which the certificate was issued (to ensure the identity of a remote computer). It tells you that the certificate was issued to, and that it is valid through June 5, 2010. But how do you know you can trust it? Where did the certificate come from?

Well, if you click on the “Certification Path” tab, it will tell you where it came from:

SSL Chain of Trust

This shows the “chain of trust,” and tells you that the “root” of that chain is the VeriSign Class 3 Public Primary Certificate Authority (“CA”). VeriSign is one of the public CAs that companies like Wells Fargo can go to and purchase certificates for purposes such as this. VeriSign is, in fact, probably the best known (and most expensive). But how do we really know that the certificate is valid? Just because it says it came from VeriSign, how do we know it really did?

Hold that thought, and let’s click on the “Details” tab:

Certificate Details Tab

Now we’re getting into the nitty-gritty of the certificate. Note the two fields called “Signature algorithm” and “Signature hash algorithm.” Remember those - we’ll come back to them later. For now, scroll down on the upper portion of the certificate, and highlight “Public key:”

The Public Key

You can now see the actual public key for this certificate displayed in the lower part of the window. Basically, the server is trying to tell you, “Here’s my public key. Use it to encrypt anything that you want to securely send to me.” But, still, how do we know it’s real? How do we know it hasn’t been tampered with?

Scroll down to the bottom, and you’ll see a reference to something called a “thumbprint:”

The Thumbprint

You will also see what algorithm was used to generate the thumbprint - in this case, it’s an algorithm called “sha1.” You probably don’t know what the “sha1” algorithm is, but your PC does. It’s used to generate a “hash value” on the contents of the certificate (remember, to your PC the entire certificate is just one long binary number). This is a one-way computation - in other words, it is not possible to look at the hash value, even knowing the algorithm, and work backwards to determine the original value that was used to generate the hash.

So the certificate is signed by running the sha1 algorithm on its contents, and then encrypting the results using the private key of the next-higher certificate in the chain of trust. That encrypted result is transmitted with the certificate as the “thumbprint.” Your computer then takes the contents of the certificate, runs the sha1 algorithm on it, and compares the results with the transmitted thumbprint, which it decrypts using the public key of the next-higher certificate in the chain of trust. If the hash values exactly match, you can be confident that the certificate hasn’t been tampered with – because it would be, for all practical purposes, impossible to tamper with the contents of the certificate without altering the thumbprint value.

Now, back to the basic question of trust. Each certificate in the chain of trust is, in this fashion, digitally signed using the private key of the certificate above it in the chain. You can, in fact, click on any certificate in the chain, click the button labeled “View Certificate,” and examine the details of that certificate, just as we examined the details in the example above. Ultimately, we find ourselves at the VeriSign “root” certificate, and the ultimate question becomes how do we really know that the public key presented in that root certificate, and which we are supposed to use to validate the signature of the next lower certificate in the chain, is valid - since we’ve now come to the end of the chain, and have no higher authority to use to validate the signature of the root?

The answer (which may surprise you) is that the manufacturer of your browser software made a deal with VeriSign to build their root certificate into the browser. If you’re running Internet Explorer, go to the “Tools” menu, and choose “Internet Options.” Click on the “Content” tab, and then click the “Certificates” button. Then click the tab that says “Trusted Root Certification Authorities,” and scroll down. Guess what?

Trusted Root Certificates

There it is. Ever notice that from time to time Microsoft’s update service pushes out “root certificate updates?” Now you know what that’s all about - they’re either adding certificates to the trusted root certificate store, or replacing ones that are about to expire.

So, now that we know we can trust the certificate that the Wells Fargo Web server presented to us, what do we do with it? Well, we’ve learned two things from the certificate: First, we know that the server we’re communicating with is indeed what it claims to be – part of the server farm; second, we know that server’s public key. Since we know its public key, we know that we can send information to it securely. However, we don’t want to use the public/private key pair for our entire session, because it so happens that an asymmetrical encryption scheme requires more processing effort that a symmetrical scheme – it would be preferable to use a symmetrical encryption scheme. But we don’t want to just use the same symmetrical key every time, because one of the basic precepts of cryptography is that the more encrypted data you have to work with, the easier it is to break the encryption. Therefore, your PC will use the server’s public key to securely negotiate a session key that will be used to symmetrically encrypt and decrypt just this banking session. Tomorrow, or next week, when you log on again, your PC will negotiate a totally different session key.

In the next post, we’ll talk more about the different kinds of certificates, what they’re used for, and some of the pitfalls of using certificates to secure communications in your own networks.

Time To Switch Virtualization Platforms?

Have you been considering moving from VMware ESX or vSphere to either Citrix® XenServer™ or Microsoft® Windows® Server 2008 Hyper-V™ - but been concerned about exactly how to go about it? Knowing what tools to use to make the migration go smoothly is often a major concern. Also, what kind of support can you get during the transition? And structured training on a new platform is not inexpensive, either. Now Citrix is trying to eliminate these obstacles with a new promotion that runs through March 31, 2010.

On October, 14, 2009, Citrix announced a new program called Project “Open Door”. Customers who switch existing VMware servers to XenServer or Hyper-V, and add Citrix Essentials™ for advanced virtualization management, will receive additional technical support, training, and conversion tools from Citrix at no additional cost.

The Project Open Door promotion will be effective worldwide from October 1 – March 31, 2010. Customers who decommission five or more VMware vSphere 4 or VI3 servers and replace them with XenServer or Hyper-V plus the Citrix Essentials solution, receive the following:

  • A free five incident support pack (5 by 8 hours) for every five servers converted
  • A voucher for six hours of online training for every five servers converted
  • Free migration tools for seamlessly transferring virtual machines from VMware to XenServer or Hyper-V

Check out for more information on the program. If you’re seriously considering making the switch, this just might be the time to do it.