Cannot open Virtual Machine Console

Original document located at vmware KB:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=749640

Details

• 
  
  When you try to connect to a virtual machine console from VirtualCenter, you see one or more of these errors:
  • Error connecting: Login (username/password) incorrect
  • Error connecting: Host address lookup for server <SERVER> failed: The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for Do you want to try again?
  • Error connecting: cannot connect to host <host>: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. Do you want to try again?
  • Error connecting: You need execute access in order to connect with the VMware console. Access denied for config file.
  • Unable to connect to MKS: failed to connect to server IP:903. For more information, see
    Troubleshooting the firewall policy on an ESX Server (1003634).ESX 4.0 hosts lose network connectivity when multiple service console interfaces are configured on subnets that use DHCP IP addresses (1010828).
• You cannot open a remote console to a virtual machine.
• Virtual machine console is black (blank).
• The VMware Infrastructure (VI) Client console tab session may time out or disconnect while in use.
• Migration of virtual machines using vMotion fails.
• This issue may affect a single ESX  host. If the virtual machines are moved to another ESX host, you may be able to connect to the console without error.
• This issue may occur if you try to connect to the console using VMware Infrastructure (VI) Client connected directly to the ESX host or to vCenter Server.

Solution

If your network is configured such that a firewall exists between the ESX host and the client running the workstation running VI Client, you might not be able to open a virtual machine console. To connect to a virtual machine console from VI Client, port 903 needs to be open in any firewall between the the workstation running VI Client and the ESX host. This
applies even if VI Client is connected to VirtualCenter and not directly to ESX host.


Note: Before performing the steps in this article:

• For more information on restarting the Management agents, see Restarting the Management agents on an ESX Server (1003490).
• For more information on editing configuration files, see Editing configuration files in VMware ESX (1017022).

To troubleshoot this issue:
(Just issuing step 1 worked for me !)

1. 
   
   Log in to the VirtualCenter Server directly through Terminal Services or a Remote KVM and attempt a connection from VI Client from this system. If this method works, the firewall is likely preventing the console from working. Configure your firewall to allow communications on port 903 between the ESX host and the workstation running VI Client.
   
   If port 903 is not open or cannot be opened in your environment, enable the vmauthd proxy. This forces remote console communication to be sent on port 902 on the Service Console, instead of 903.
   
   Note: By enabling this setting there may be degradation in performance communicating to the ESX host service console, if remote consoles are heavily utilized.
   
   To enable the proxy:
   1. Log in to the ESX host's service console as root.
   2. Open /etc/vmware/config with a text editor. 
   3. Add the following line:
      vmauthd.server.alwaysProxy = "TRUE"
   4. Issue the following command to restart xinetd:
      service xinetd restart
2. Verify the ESX firewall policy.  For more information, see
3. Verify that the ESX host and the workstation running VI Client are correctly synced to an NTP service. This is required to satisfy SSL handshaking between VI Client and ESX. For more information, see Verifying time synchronization across environment (1003736).
4. 
   
   DNS problems are a common cause of virtual machine console problems. Verify name resolution in your environment. For more information, see:
   1. 
      

• Identifying issues with and setting up name resolution on ESX Server (1003735)
• Configuring name resolution for VMware VirtualCenter (1003713)

• After verifying DNS, open a command prompt on the VI Client machine and perform the following:
  ipconfig /flushdns
  ipconfig /registerdns
• Verify /var partition is not full.
• Verify that the permissions for the virtual machine's .vmx file are set correctly. To set the permissions, run the command:
  chmod 755 </full/path/to/virtual machine.vmx>
• If your ESX host has more than one service console configured, verify that they are not on the same network.
• Check if the Service Console IP is routing traffic to the workstation running the vCenter. For more information on configuring the Service Console Gateway, see Changing the IP address, default gateway, and hostname of the Service Console in ESX (4309499).