Ran into the same problem. Had VMWare Server 2 installed on an XP home for testing purpose. Was astonished it worked on XP home (remember VMWare Server 1 did not install on XP home). The XP home machine was accessed via VNC. Some problems occurred (VMWare Server 2 Web interface was not accessible from outside, only from the machine itself). Location of problem source difficult (firewall settings ok, ports open etc). I decided to uninstall VMWare Server 2 and reinstall it with proper DNS credentials etc for the web interface. Reinstallation subsequently failed due to the SSL keys issue. I've redownloaded the installation package from vmware, in vain. I've removed VIX, in vain too. VMWare Server still can't generate the SSL keys. There's no file openssl.* on the system. So, boogey, what's the problem creating ssl keys? The machine even has an url consisting of a fully qualified domain name (www..) and is available to the web. So why would the VMware Server 2 installer not be able to generate SSL keys?
Somehow I have the feeling the first installation must have left a registry entry that prevents the installer to generate SSL keys. Does anybody by any chance know which entry? Or does anybody have a clue what causes the ssl issue?