How to configure the right amount of swap space in Linux

http://www.cyberciti.biz/tips/linux-swap-space.html

Linux and other Unix-like operating systems use the term "swap" to describe both the act of moving memory pages between RAM and disk, and the region of a disk the pages are stored on. It is common to use a whole partition of a hard disk for swapping. However, with the 2.6 Linux kernel, swap files are just as fast as swap partitions. Now, many admins (both Windows and Linux/UNIX) follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. Let us say I've 32GB RAM, should I set swap space to 64 GB? Is 64 GB of swap space really required? How big should your Linux / UNIX swap space be?


Old dumb memory managers


I think the '2x swap space' rule came from Old Solaris and Windows admins. Also, earlier memory mangers were very badly designed. There were not very smart. Today, we have very smart and intelligent memory manager for both Linux and UNIX.


Nonsense rule: Twice the size of your main system RAM for Servers

According to OpenBSD FAQ:


Many people follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. This rule is nonsense. On a modern system, that's a LOT of swap, most people prefer that their systems never swap. You don't want your system to ever run out of RAM+swap, but you usually would rather have enough RAM in the system so it doesn't need to swap.


Select right size for your setup

Here is my rule for normal server (Web / Mail etc):

Swap space == Equal RAM size (if RAM < 2GB)

Swap space == 2GB size (if RAM > 2GB)

Many Oracle experts recommend something as follows for heavy duty Oracle server with fast storage such as RAID 10:


Swap space == Equal RAM size (if RAM < 8GB)

Swap space == 0.50 times the size of RAM (if RAM > 8GB)

However, newer servers can have 


Red Hat Recommendation

Red hat recommends setting as follows for RHEL 5:



The reality is the amount of swap space a system needs is not really a function of the amount of RAM it has but rather the memory workload that is running on that system. A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. In this case the system will simply lock the anonymous and system V shared memory into RAM and use the remaining RAM for caching file system data so when memory is exhausted the kernel only reclaims pagecache memory.



Considering that 1) At installation time when configuring the swap space there is no easy way to predetermine the memory a workload will require, and 2) The more RAM a system has the less swap space it typically needs, a better swap space



Systems with 4GB of ram or less require a minimum of 2GB of swap space

Systems with 4GB to 16GB of ram require a minimum of 4GB of swap space

Systems with 16GB to 64GB of ram require a minimum of 8GB of swap space

Systems with 64GB to 256GB of ram require a minimum of 16GB of swap space

Swap will just keep running servers...

Swap space will just keep operation running for a while on heavy duty servers by swapping process. You can always find out swap space utilization using any one of the following command:

cat /proc/swaps

swapon -s

free -m

top



See how to find out disk I/O and related information under Linux. In the end, you need to add more RAM, adjust software (like controlling Apache workers or using lighttpd web server to save RAM) or use some sort of load balancing.



Also, refer Linux kernel documentation for /proc/sys/vm/swappiness. With this you can fine tune swap space.



A note about Desktop and Laptop

If you are going to suspend to disk, then you need swap space more than actual RAM. For example, my laptop has 1GB RAM and swap is setup to 2GB. This only applies to Laptop or desktop but not to servers.



Kernel hackers need more swap space

If you are a kernel hacker (debugging and fixing kernel issues) and generating core dumps, you need twice the RAM swap space.



Conclusion

If Linux kernel is going to use more than 2GB swap space at a time, all users will feel the heat. Either, you get more RAM (recommend) and move to faster storage to improve disk I/O. There are no rules, each setup and configuration is unique. Adjust values as per your requirements. Select amount of swap that is right for you.

Setting up tftp server on Redhat Linux and allow write access to tftpboot

A TFTP Server is often used by Cisco devices to manage their file transfers. Here is a quick guide on how to setup a TFTP server on Centos/Redhat.

First obtain the RPM. In Centos, use yum to search and install tftp-server.

yum search tftp

yum install tftp-server

After it's installed, the RPM creates a new folder named 'tftpboot' in the root partition. You must change the access rights for it be read and write accessible.

chmod 777 /tftpboot

The tftp-server is controlled by xinetd which is a Linux daemon that handles certain lightweight network applications. Edit the file name 'tftp' located in /etc/xinetd.d/ . The TFTP service is defaulted to off so change the disable option to 'no'

# default: off

# description: The tftp server serves files using the trivial file transfer \

#       protocol.  The tftp protocol is often used to boot diskless \

#       workstations, download configuration files to network-aware printers, \

#       and to start the installation process for some operating systems.

service tftp

{

        socket_type             = dgram

        protocol                = udp

        wait                    = yes

        user                    = root

        server                  = /usr/sbin/in.tftpd

        server_args             = -s -c /tftpboot

        disable                 = no

        per_source              = 11

        cps                     = 100 2

        flags                   = IPv4

}

Make sure the "disable = yes" option is set to "no" to enable the service.

TFTP server also comes default read only. In order to enable write access the -c argument needs to be added to the server_args.

server_args = -s -c /tftpboot

Ensure you restart the xinetd service when making changes to this file.

service xinetd restart

You may verify the server is started by verifying that it's listening on UDP port 69.

netstat -nap |grep :69

The output should look like this:

udp        0      0 0.0.0.0:69                  0.0.0.0:*                               19141/xinetd

If you have a firewall, make sure that UDP port 69 is opened.

----

End of posting.

For more, visit http://tipstricksandmore.blogspot.com

Reliable and unexpensive web hosting and domain names at http://www.netlinecorp.com

How to kill the process running on specific port in Linux

I would like to stop the process which is running on port 8080 in a Linux server, but I don't know the process that owns it.






'lsof -w -n -i tcp:8080'


or


'fuser -n tcp 8080'


or


'netstat -anp | grep :8080[[:blank:]]'






should show you the process ID.






If there is no (init) script to use to shut down the offending service (since TCP/8080 means proxy) to kill you can by PID 'kill -9 pidnumber'.

How to send email from the Linux command line

The Linux command line can be very powerful once you know how to use it. You can parse data, monitor processes, and do a lot of other useful and cool things using it. There often comes a need to generate a report and mail it out. It could be as simple a requirement as a notification that the day’s backup went through fine, or did not. I’ll help you get started with sending mails from the Linux command line and in shell scripts. We will also cover sending attachments from the command line. We will begin with the “mail” command.



MAIL


First run a quick test to make sure the “sendmail” application is installed and working correctly. Execute the following command, replacing “you@youremailid.com” with your e-mail address.






# mail -s “Hello world” you@youremailid.com






Hit the return key and you will come to a new line. Enter the text “This is a test from my server”. Follow up the text by hitting the return key again. Then hit the key combination of Control+D to continue. The command prompt will ask you if you want to mark a copy of the mail to any other address, hit Control+D again. Check your mailbox. This command will send out a mail to the email id mentioned with the subject, “Hello world”.






To add content to the body of the mail while running the command you can use the following options. If you want to add text on your own:






# echo “This will go into the body of the mail.”
mail -s “Hello world” you@youremailid.com






And if you want mail to read the content from a file:






# mail -s “Hello world” you@youremailid.com < /home/calvin/application.log






Some other useful options in the mail command are:






-s subject (The subject of the mail)


-c email-address (Mark a copy to this “email-address”, or CC)


-b email-address (Mark a blind carbon copy to this “email-address”, or BCC)






Here’s how you might use these options:






# echo “Welcome to the world of Calvin n Hobbes”
mail -s “Hello world” calvin@cnh.com -c hobbes@cnh.com -b susie.derkins@cnh.com






MUTT


One of major drawbacks of using the mail command is that it does not support the sending of attachments. mutt, on the other hand, does support it. I’ve found this feature particularly useful for scripts that generate non-textual reports or backups which are relatively small in size which I’d like to backup elsewhere. Of course, mutt allows you to do a lot more than just send attachments. It is a much more complete command line mail client than the “mail” command. Right now we’ll just explore the basic stuff we might need often. Here’s how you would attach a file to a mail:






# echo “Sending an attachment.”
mutt -a backup.zip -s “attachment” calvin@cnh.com






This command will send a mail to calvin@cnh.com with the subject (-s) “attachment”, the body text “Sending an attachment.”, containing the attachment (-a) backup.zip. Like with the mail command you can use the “-c” option to mark a copy to another mail id.






SENDING MAIL FROM A SHELL SCRIPT


Now, with the basics covered you can send mails from your shell scripts. Here’s a simple shell script that gives you a reading of the usage of space on your partitions and mails the data to you.






#!/bin/bash


df -h
mail -s “disk space report” calvin@cnh.com






Save these lines in a file on your Linux server and run it. You should receive a mail containing the results of the command. If, however, you need to send more data than just this you will need to write the data to a text file and enter it into the mail body while composing the mail. Here’s and example of a shell script that gets the disk usage as well as the memory usage, writes the data into a temporary file, and then enters it all into the body of the mail being sent out:






#!/bin/bash


df -h > /tmp/mail_report.log


free -m >> /tmp/mail_report.log


mail -s “disk and RAM report” calvin@cnh.com < /tmp/mail_report.log






Now here’s a more complicated problem. You have to take a backup of a few files and mail then out. First the directory to be mailed out is archived. Then it is sent as an email attachment using mutt. Here’s a script to do just that:






#!/bin/bash


tar -zcf /tmp/backup.tar.gz /home/calvin/files


echo
mutt -a /tmp/backup.tar.gz -s “daily backup of data” calvin@cnh.com






The echo at the start of the last line adds a blank into the body of the mail being set out.






This should get you started with sending mails form the Linux command line and from shell scripts. Read up the “man page” for both mail and mutt for more options.

How to export ipod notes to your PC computer for free?

There are few applications in the market offering this feature, but if you want to do it for free and easily, take note of how to do it:

1) You will need a PC computer with iTunes and Microsoft Outlook. Of course, you also need your ipod and the cable connected to the PC.
2) You must install Microsoft Outlook (not Outlook Express), and set it up as the default email client. You can do it temporarily if you want, and then change it back later to your favorite email client.
This is because itunes uses Microsoft Outlook as the default application to synchronize the Notes.
Go to iTunes, select your ipod under Devices, and from the tabs on top, select Info. In the section "Other" tick the option "Sync notes with: Outlook"
Do it, sync your ipod.
3) You will find all your notes in Outlook, that is the yellow sticky notes icon under "Tasks" on the left menu. Or just use the menu: Go to -> Notes.
You can start copying and pasting the contents of your notes, but if you have many of them, it is a better idea to export them to notepad, however, there is no such functionality in Outlook :(
Don't worry, go to the next step.
4) Open outlook, Alt+F11 to open VBA editor, put the following code in the right blank area, while on the vba editor, click the save button.

Sub exportNotes()
On Error GoTo huboerr
myfolder = "c:\notes\" 'change this to your folder path but folder path has to end with a backlash
Set myNote = Application.GetNamespace("MAPI").GetDefaultFolder(olFolderNotes)
For ibi = 1 To myNote.Items.Count
fname = myNote.Items(ibi).Subject
myNote.Items(ibi).SaveAs myfolder & fname & ".txt", 0

Next
Exit Sub
huboerr:
MsgBox fname
Resume Next

End Sub


Then all you need to do is to change the first line of code to the path of export folder
While in vba editor, when hit F5, all the notes in the notes folder will be exported to the folder you specified, the notes name will be the same as you have in your Outlook notes and in your ipod . You will probably have to edit the note names in Outlook before exporting them, as you may name them using some special characters (*, %, etc.) that are not accepted by Windows...

I hope it saved you some time re-typing the notes in your PC, and some bucks if you were thinking about buying an application for doing that.

Enjoy !

How do I find discovered SAN targets and LUNs on a Linux host with Emulex HBA cards?

The sys file system (sysfs) provides details on what devices have been discovered.

Change directory to /sys/class/scsi_host. Perform an ls:

[tam@tsdownload scsi_host]$ ls
host0

Each SCSI host adapter and those that look like SCSI host adapters to Linux are assigned a host number by Linux. You may see more than one hostX (where x is a number). In the example above, only one SCSI host bus adapter is found, host0. If there are more than one, perform an ls command on each:


[tam@tsdownload scsi_host]$ ls host0
[tam@tsdownload scsi_host]$ ls host0
board_online lpfc_discovery_min_wait node_name
cmd_per_lun lpfc_discovery_wait_limit nport_evt_cnt
ctlreg lpfc_drvr_version num_discovered_ports
device lpfc_fcp_bind_method option_rom_version
disc_adisc lpfc_fcp_class outfcpio
disc_map lpfc_fdmi_on portfcid
disc_npr lpfc_hba_queue_depth port_name
disc_plogi lpfc_link_speed portnum
disc_prli lpfc_linkup_wait_limit port_type
disc_reglgn lpfc_log_verbose proc_name
disc_unmap lpfc_lun_queue_depth programtype
disc_unused lpfc_max_luns scan
events lpfc_nodev_tmo serialnum
fabric_name lpfc_scan_down sg_tablesize
fwrev lpfc_topology speed
hdw lpfc_use_adisc state
host_busy management_version unchecked_isa_dma
info mbox unique_id
issue_lip modeldesc
lpfc_ack0 modelname


Any items listed with lpfc are Emulex adapters. You can check the model, firmware version, and driver version:

[tam@tsdownload host0]$ cat info
Emulex LP10000 2Gb PCI-X Fibre Channel Adapter on PCI bus 05 device 20 irq 209

[tam@tsdownload host0]$ cat lpfc_drvr_version
Emulex LightPulse Fibre Channel SCSI driver 8.0.16.27

[tam@tsdownload host0]$ cat fwrev
1.91A5 (T2D1.91A5)

Change directory to the host directory and then the device directory:

[tam@tsdownload scsi_host]$ cd host0
{tam@tsdownload host0]$ device
[tam@tsdownload device]$ ls
detach_state power target0:0:0 target0:0:1

Each target and LUN is listed as targetX:Y:Z where:
•X is the bus number
•Y is the target number
•Z is the LUN number.
These are the devices discovered by the host bus adapter and Linux SCSI midlayer.

To determine what type of devices were found, change directory to the targetX:Y:Z directory and then the X:X:X:X directory below it. You can cat the model, vendor, and other files to learn more about the devices:
[tam@tsdownload device]$ cd 0:target0:0:0
[tam@tsdownload target0:0:0]$ cd 0:0:0:0
[tam@tsdownload 0:0:0:0]$ ls
block detach_state dump power rescan scsi_level timeout vendor
delete device_blocked model queue_depth rev state type

[tam@tsdownload 0:0:0:0]$ cat model
ST336854FC

[tam@tsdownload 0:0:0:0]$ cat vendor
SEAGATE

[tam@tsdownload 0:0:0:0]$ cat state
running

The 0:0:0:0 directory represents host bus adapter 0, bus 0, target 0, LUN 0.


Original document (official Emulex web site):
http://www.emulex.com/knowledge/search/viewArticle.jsp?docId=68

Upgrade to Vspere ESX 4.1 failed, ESX host boot stops and drops into Troubleshooting mode.

Recently, I've got this error whe upgrading an ESX host from version 4.0 to 4.1 upd 01, using the upgrade manager. I was looking to the console, after the ESX host reboot and:



ESX host boot stops at the error: VSD mount/Bin/SH:cant access TTY job control turned off

I was in panic.



Rebooting the host failed, giving me the same error.


I didn't want to re-install the host, because I had some Production Virtual Machines in the local storage, and wanted to solve it quick, as my maintenance window was almost over.




On the ESX console, you see the error:



VSD mount/bin/sh:can't access TTY; job control turned off.



The ESX host does not boot and drops into Troubleshooting (busy box) mode.



Looking into the /var/log/messages file, there was something similar to:



sysboot: Getting '/boot/cosvdmk' parameter from esx.conf


sysboot: COS VMDK Specified in esx.conf: /vmfs/volumes/4b27ec62-93ec3816-0475-00215aaf882a/esxconsole-4b27e9e3-20ee-69d7-ae11-00215aaf882a/esxconsole.vmdk


sysboot: 66.vsd-mount returned critical failure


sysboot: Executing 'chvt 1'






How to solve it:






This issue occurs if an ESX host cannot identify the esxconsole.vmdk file in which the service console resides.






1. Go to the console of the ESX host. After the error message, ESX drops into Troubleshooting (busy box) mode.


2. Find the .vmdk for the service console by running the command:






grep "/boot/cosvmdk" /etc/vmware/esx.conf






The output is similar to:






/boot/cosvmdk = "/






For example:






/boot/cosvmdk = "/vmfs/volumes/4a14d968-88bf7161-700f-00145ef48f76/esxconsole-4a14d906-2f96-7956-7284-00145ef48f74/esxconsole.vmdk"






3. Make note of the and the


4. Verify that the files exist by running the command:






ls -al /vmfs/volumes//






Where and is from the output of step 2.






The output is similar to:






total 7906560


drwxr-xr-x 1 root root 840 May 21 00:45 .


drwxr-xr-t 1 root root 2660 Oct 21 09:10 ..


-rw------- 1 root root 8095006720 Oct 26 15:37 esxconsole-flat.vmdk


-rw------- 1 root root 475 May 21 00:32 esxconsole.vmdk


drwxr-xr-x 1 root root 980 May 21 00:45 logs






5. Ensure that both the esxconsole-flat.vmdk and the esxconsole.vmdk files exist.






In my case, the esxconsole.vmdk file did not exist.






I troubleshoot using a modification to the following: http://kb.vmware.com/kb/1002511






All the step in this section are important !!!






a. Identify and record the exact size of the -flat file using a command similar to:



# ls -l esxconsole-flat.vmdk


-rw------- 1 root root 4294967296 Oct 11 12:30 esxconsole-flat.vmdk


Write down this number (4294967296) it could be different in your case, it is usually 8GB !



b. Rename the file, as all the valuable data is in this file !


# mv esxconsole-flat.vmdk esxconsole-flat_ORIG.vmdk






c. Use the vmkfstools command to create a new virtual disk:



# vmkfstools -c 4294967296 -a buslogic esxconsole.vmdk



Remember to use buslogic as the bus type.



d. Rename the new blank created disk file esxconsole-flat.vmdk


# mv esxconsole-flat.vmdk esxconsole-flat_BLANK.vmdk



e. Rename the old console disk to the new file


# mv esxconsole-flat_ORIG.vmdk esxconsole-flat.vmdk






Type exit, or reboot the ESX server.






You will be able to boot the server normally.






Later, you can delete the file esxconsole-flat_BLANK.vmdk






Problem solved !










Note:


If the esxconsole-flat file does not exist in step 5, you must re-install the ESX host to recreate the service console.








Original VMware KBs:






http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012874


http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002511

/*.vmdk
values in the output.
/esxconsole.vmdk"

Vsphere ESX NIC numbering

VMware had an issue with the nics port numbering. If you have on board nics, and some PCIe nics, the numbering is not correct. With HP server and Broadband and Intel adapter it happens.

 

It pretty much assigns them in whatever order it finds them - you can see this article (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002315) for a rundown on the process. Now if you are really dead set on changing them it can be done. You would have to:

 

You can change the numbering by changing the esx.conf

 

1. go into the console (tech support mode on ESXi)

2. cd /etc/vmware

3. backup your esx.conf

                eg. cp esx.conf esx.conf.bak

4. edit esx.conf and look for your vmnic entries.. will look something like

     /device/000:001:00.0/vmkname = "vmnic0"

     /device/000:001:00.1/vmkname = "vmnic1"

     /device/000:002:00.0/vmkname = "vmnic2"

     /device/000:002:00.1/vmkname = "vmnic3"

5. Order as desired by editing the string vmnicX

6. Look for mac address assignment to vmnicX and change them accordingly, change the lines for the mac "xx:xx:xx:xx:xx:xx" and virtualmac "yy:yy:yy:yy:yy:yy" to be consistent. Don't change just the "vmnicX", leave everything in the same place.

 

                /net/pnic/child[000X]/mac/ = "xx:xx:xx:xx:xx:xx"

                /net/pnic/child[000X]/name/ = "vmnicX"

                /net/pnic/child[000X]/virtualmac/ = "yy:yy:yy:yy:yy:yy"

 

7. Save the edited file and reboot

 

You may have to readjust your nic bindings in networking in the Virtual Center when done.

 

For many implementations, the numbering is not an issue, but for a correct design (using nic port to a specific Switch port) and validate the cabling the right numbering is important. The use of host profiles is standardized, including the nic names, which makes things easier to maintain with less efforts.

 

Example: We use vmnic0 and vmnic5 for service console and vmkernel for iscsi, and vmnic1 and vmnic4 for Virtual Machines Data, if the numbering is not correct, you can use the same nic/port for High Availability, if you have a nic problem, both nics/ports can be the some physical port, and you have no vmkernel, or Service Console

 

This is just an example what can happen if you have your infrastructure very well documented, and the correct design for High Availability on all connections.

 

I hope It is useful for you as it was for me !

VMware vcenter server migration from 4.0 to 4.1

VMwAre Vcenter server is used for maintaining the esx servers from a centralized place with user permission and cluster creation and the licensing issues.

New version 4.1 MUST be installed on a Windows 64 bit OS computer.

I migrated it from an old server to a new one, so I had to proceed with a migration.

While migrating from 4.0 to 4.1, we need to follow these steps for getting old database converted to new format, as it is different from 4.0 to 4.1.

 

Before migration, the following services of vcenter should be stopped

VMware VirtualCenter Server

VMware VirtualCenter Management Webservices

VMware vCenter Update Manager Service

 

Mount the vcenter ISO image or load the bunt DVD in the old vcenter server machine. Open the cd and copy the Datamigration folder to C:\Datamigration

Extract all the file in the same folder.

IMPORTANT: Verify that you have at least 8 GB of available space, as many configuration files will be copied to C: drive.

 

 

Navigate to C:\Datamigration\ and run backup.bat , the script will backup all configuration settings from your recent vCenter installation and database. (Software builds, used ports, database information, server names and VUM Settings/Patches etc.).. once finished all the important data is stored in C:\Datamigration\Data\
If there are any errors while making the backup… check this and fix before you go further!
Now you need to copy the C:\Databasemigration\ folder to your new planned vCenter 4.1 server. You can do this by share, copy this on USB stick or whatever.

Migrate the database and install vCenter 4.1:
- Login to the new vCenter server;
- Mount the new VMware vCenter Server 4.1 and modules
Open CMD:

- Navigate to C:\Data migration\ and run install.bat. The script will check if the correct data is present..

If you used a different name for your new vCenter server press Yes.. to continue

Then you must select the vCenter 4.1 and VUM installation source. In my case the mounted ISO has drive letter F:\

The configuration will be validated, correct ports numbers, vCenter serial number etc.. all good..

.. The vCenter Installation is started

I already installed a Microsoft SQL 2005 Enterprise server (local) and created a 32 bit ODBC connection, THEN select "vCenter41" DSN

If needed, give the correct credentials

 

You can modify this later, please check: http://kb.vmware.com/kb/1001046 for more information

 

 

- I changed the installation path to D:\Program Files\VMware\Infrastructure\ .. default location is C:\..\.

 

 

 

 

 

Oke, the installation was not so difficult. After clicking finish the install.bat will continue with the VMware Update Manager database migration and installation.

VMware Update Manager – Database migration and installation software:

As you can see "vCenter Server migration installed completed successfully" great

The install.bat script will continue and check again if the selected VUM ports are available and if the copied data migratable is for the new VUM database location

 

 

 

 

 

 For VMware Update Manager you need to create a 64-Bit DSN. You can create this here: C:\WINDOWS\SysWOW64\odbcad32.exe .. oke, thats done.. select the VUM (already created before installation) database.

 

 

 

 

The installation and patch download location is by default C:\.. I prefer to install this data on a separate disk.. so I changed the installation path to D:\…\.. 

 

 

 

Now you can connect your vCenter server and manage your hosts and VM's.. sometimes you can't!

Tip before connect the vCenter:

- Reboot your new vCenter 4.1 server;

- On the ESX host: Restart your management services: service mgmt-vmware restart

- On the ESX host: Use a text editor to view/change the IP address inside the xxx.xxx.xxx.xxx tags the following file:/etc/opt/vmware/vpxa/vpxa.cfg

Finally result:

 

 

 

Successful migration process of vCenter 4.0 to new 4.1 server.

How to fix VMware error fsck.ext3 Unable to resolve UUID with no data loss

It happened to me after a power failure, that damaged a server of our vmware farm. The resolution is presented below:

ESX 4.0 host fails to boot after power operation with the error: fsck.ext3: Unable to resolve UUID

Symptoms

• After power-cycling or rebooting an ESX 4.x server, the following error message is produced during boot:
  

  fsck.ext3: Unable to resolve 'UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa'
  

  
  And
  

  *** An error occurred during the file system check.
  *** Dropping you to a shell; the system will reboot 
  *** when you leave the shell.
  

  
   

• After encountering this error, you are unable to boot into ESX or Troubleshooting mode.
  
• The unresolvable EXT file systems or partitions most commonly later appear to have mount points such as /var, /opt and /tmp.
  

Resolution

 

This issue occurs when the boot-time file system check utility (FSCK) for EXT-3 file systems cannot resolve a file system (by UUID) defined in /etc/fstab.

Issues that can result in this may include:

• The default roll-back option is left enabled when a subsequent upgrade is being performed.
  
• The device not present during system boot.
  
• The unresolvable EXT file systems appear to reside on disks/devices that are initialized later during system boot (e.g. the last LUN).
  

Note: If you are experiencing an outage with virtual machines down, consider resolving the situation in a timely manner through the reinstallation of VMware ESX. Troubleshooting may take more time than a reinstallation, which is in the order of approximately 20 minutes.

Otherwise refer to instructions below for submission of information to VMware Technical Support for technical analysis.

Further troubleshooting is available in the shell:

• Confirm the UUIDs which were not resolvable, and remain so, by running fsck again without additional arguments. Information similar to the following is displayed:
  

  
  # fsck
  
  fsck 1.39 (29-May-2006)
  e2fsck 1.39 (29-May-2006)
  esx-root: clean, 32953/641280 files, 414801/1281175 blocks
  e2fsck 1.39 (29-May-2006)
  /dev/sdt1: clean, 35/140832 files, 25323/281596 blocks
  fsck.ext3: Unable to resolve 'UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa'
  e2fsck 1.39 (29-May-2006)
  /dev/sdt6: clean, 31/250368 files, 27851/500220 blocks
  e2fsck 1.39 (29-May-2006)
  /dev/sdt7: clean, 22/250368 files, 16815/500220 blocks
  

  
   

• Record the UUID or UUIDs which failed to resolve. You may take a screen shot of your System Management Interface, take a picture, or write the values down.
  
• Confirm these same values in the /etc/fstab file.
  

  # cat /etc/fstab
  
  UUID=79815890-f11c-4907-80fe-d1cd6bf061f8 /        ext3    defaults                  1 1
  UUID=45460133-027b-40b6-8b4d-e52aaf4c417f /boot    ext3    defaults                  1 2
  None                    /dev/pts                   devpts  defaults                  0 0
  /dev/cdrom              /mnt/cdrom                 udf,iso9660 noauto,owner,kudzu,ro 0 0
  /dev/fd0                /mnt/floppy                auto    noauto,owner,kudzu        0 0
  None                    /proc                      proc    defaults                  0 0
  None                    /sys                       sysfs   defaults                  0 0
  UUID=34d192db-17eb-442e-9613-c5c24c6fa9fa
  /var/log ext3    defaults,errors=panic     1 2
  UUID=e32ec5f4-d795-414a-8d73-a2bb3ea86342 swap     swap    defaults                  0 0
  
  Note: Highlighted in blue is the mount point for the respective unresolvable UUID, in red.
  

• Verify what UUIDs the system is currently aware of by running the following command:
  

  # ls -l /dev/disk/by-uuid
  
  total 0
  lrwxrwxrwx 1 root root 10 Nov  9 14:36 45460133-027b-40b6-8b4d-e52aaf4c417f -> ../../sdm1
  lrwxrwxrwx 1 root root 10 Nov  9 14:36 e32ec5f4-d795-414a-8d73-a2bb3ea86342 -> ../../sdr1
  lrwxrwxrwx 1 root root 10 Nov  9 14:36 34d192db-17eb-442e-9613-c5c24c6fa9fa -> ../../sdr2
  lrwxrwxrwx 1 root root 10 Nov  9 14:36 79815890-f11c-4907-80fe-d1cd6bf061f8 -> ../../sdr5
  
  Notes:
  

  • This output reveals the UUID-to-partition relationship for all discovered EXT partitions in the system. Affected mount points or content can be associated using the previous step.
    
  • It is possible in some environments that none of the known partitions reported by listing /dev/disk/by-uuid match the unresolved UUID. This is correctable; for additional instructions, proceed to the following sections and correct the content of the /etc/fstab file.
    

  
   

Solution

VMware is currently investigating further for a full root-cause and solution. Workarounds are available below.

If you are able to reproduce this issue while maintaining production via alternate servers, contact VMware Technical Support after completing the following:

1. Log into the terminal of the affected ESX server.
   
2. Remount the root partition in read-write mode:
   

   # mount / -o remount,rw
   

   
    

3. Configure Serial Line Logging per the section Configuring the Service Console for VMware ESX 3.x and 4.x in KB article: Enabling serial-line logging for an ESX and ESXi host (1003900).
   
4. Reboot the ESX server and log the results via your listening serial terminal.
   
5. Contact VMware Technical Support and file a Support Request. For additional information, see Filing a Support Request (1021619).
   

Workarounds

Both recommended workarounds involve the modification of the /etc/fstab file. You may either:

• Generate a new UUID for the affected file system(s) and update /etc/fstab to match the new value(s).
  
• Update /etc/fstab to incorporate the correct UUID from the file system.
  

Applying a new UUID

Apply a new UUID to the EXT-3 file systems which fail to resolve and update the /etc/fstab file.

1. Run tune2fs against each Linux partition on the suspected disk device. For example:
   

   # tune2fs -l /dev/sdr2 | grep UUID
   Filesystem UUID:          34d192db-17eb-442e-9613-c5c24c6fa9fa
   
   # tune2fs -U random /dev/sdr2
   tune2fs 1.39 (29-May-2006)
   
   # tune2fs -l /dev/sdr2 | grep UUID
   Filesystem UUID:          25a18c70-ffcb-4b15-9d2d-1cfab1754d86
   

2. Update /etc/fstab with the updated UUID. From earlier steps, /dev/sdr2 partition was determined to be the /var/log mount point:
   

   
    

   1. Remount the root partition in read-write mode:
      

      # mount / -o remount,rw
      

      
       

   2. Open the /etc/fstab file for re-writing. For more information, see Editing configuration files in VMware ESX (1017022).
      
   3. Search for, and change, the original UUID to the newly-generated UUID from earlier steps, above.
      
   4. Save the file and remount the root partition in read-only mode:
      

      # mount / -o remount,ro
      

   5. Reboot the server using shutdown -r now.
      

 

You can read the full document at (check the "mount" syntax):

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1017162&sliceId=1&docTypeID=DT_KB_1_1&dialogID=127160699&stateId=0%200%20138435051