Upgrade to Vspere ESX 4.1 failed, ESX host boot stops and drops into Troubleshooting mode.

Recently, I've got this error whe upgrading an ESX host from version 4.0 to 4.1 upd 01, using the upgrade manager. I was looking to the console, after the ESX host reboot and:



ESX host boot stops at the error: VSD mount/Bin/SH:cant access TTY job control turned off

I was in panic.



Rebooting the host failed, giving me the same error.


I didn't want to re-install the host, because I had some Production Virtual Machines in the local storage, and wanted to solve it quick, as my maintenance window was almost over.




On the ESX console, you see the error:



VSD mount/bin/sh:can't access TTY; job control turned off.



The ESX host does not boot and drops into Troubleshooting (busy box) mode.



Looking into the /var/log/messages file, there was something similar to:



sysboot: Getting '/boot/cosvdmk' parameter from esx.conf


sysboot: COS VMDK Specified in esx.conf: /vmfs/volumes/4b27ec62-93ec3816-0475-00215aaf882a/esxconsole-4b27e9e3-20ee-69d7-ae11-00215aaf882a/esxconsole.vmdk


sysboot: 66.vsd-mount returned critical failure


sysboot: Executing 'chvt 1'






How to solve it:






This issue occurs if an ESX host cannot identify the esxconsole.vmdk file in which the service console resides.






1. Go to the console of the ESX host. After the error message, ESX drops into Troubleshooting (busy box) mode.


2. Find the .vmdk for the service console by running the command:






grep "/boot/cosvmdk" /etc/vmware/esx.conf






The output is similar to:






/boot/cosvmdk = "/






For example:






/boot/cosvmdk = "/vmfs/volumes/4a14d968-88bf7161-700f-00145ef48f76/esxconsole-4a14d906-2f96-7956-7284-00145ef48f74/esxconsole.vmdk"






3. Make note of the and the


4. Verify that the files exist by running the command:






ls -al /vmfs/volumes//






Where and is from the output of step 2.






The output is similar to:






total 7906560


drwxr-xr-x 1 root root 840 May 21 00:45 .


drwxr-xr-t 1 root root 2660 Oct 21 09:10 ..


-rw------- 1 root root 8095006720 Oct 26 15:37 esxconsole-flat.vmdk


-rw------- 1 root root 475 May 21 00:32 esxconsole.vmdk


drwxr-xr-x 1 root root 980 May 21 00:45 logs






5. Ensure that both the esxconsole-flat.vmdk and the esxconsole.vmdk files exist.






In my case, the esxconsole.vmdk file did not exist.






I troubleshoot using a modification to the following: http://kb.vmware.com/kb/1002511






All the step in this section are important !!!






a. Identify and record the exact size of the -flat file using a command similar to:



# ls -l esxconsole-flat.vmdk


-rw------- 1 root root 4294967296 Oct 11 12:30 esxconsole-flat.vmdk


Write down this number (4294967296) it could be different in your case, it is usually 8GB !



b. Rename the file, as all the valuable data is in this file !


# mv esxconsole-flat.vmdk esxconsole-flat_ORIG.vmdk






c. Use the vmkfstools command to create a new virtual disk:



# vmkfstools -c 4294967296 -a buslogic esxconsole.vmdk



Remember to use buslogic as the bus type.



d. Rename the new blank created disk file esxconsole-flat.vmdk


# mv esxconsole-flat.vmdk esxconsole-flat_BLANK.vmdk



e. Rename the old console disk to the new file


# mv esxconsole-flat_ORIG.vmdk esxconsole-flat.vmdk






Type exit, or reboot the ESX server.






You will be able to boot the server normally.






Later, you can delete the file esxconsole-flat_BLANK.vmdk






Problem solved !










Note:


If the esxconsole-flat file does not exist in step 5, you must re-install the ESX host to recreate the service console.








Original VMware KBs:






http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012874


http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002511

/*.vmdk
values in the output.
/esxconsole.vmdk"