Wednesday, 10 February 2010

SOLVED: unable to access file unspecified filename since it is locked

SOLVED: unable to access file since it is locked

This error message is so generic, and appeared to me at least in two different situations.

First time, trying to remove a snapshot of a VM, and second time, trying to power on a VM.

This is the second time now where I cannot power on the VDR virtual machine.

The message "Unable to access file since it is locked appears in the Recent tasks panel.




Since the filename is unspecified, it makes it hard to figure out what the issue is.
Why did it happen? Well, first I have to tell you that something went wrong in a backup process of one of my VMs, specifically the Virtual Center Server (in my case, this is a VM). It could be ANY VM, but in this case is this one. The last time, the same issue happened with a different VM, but anyway, the problem is the same.



I figured out that the job failed, and It couldn't remove the snapshot. (VDR appliance creates a snapshot, and then copies the contents to the destination; after that, the snapshot is removed). But something went wrong, and the snapshot was not removed. It caused all future jobs also failed. I tried manually creating a snapshot for the failed VM, and then remove it, to force it to "Delete All" the unused snapshots, but the procedure failed, giving me the same error: "unable to access file since it is locked" ... mmmm, who is locking and which file ?????

I tried also moving the VM to another ESX in the cluster; restarted vmware management services; restarted the VM; restarted the ESX host itself, but no luck.

Ok, you got it, VDR is locking it .... but which file ???? I searched hours in google and Vmware KB, but nothing ... I opened a ticked with VMware. It took one day for them to call me back, just to acknowledge the ticket. It took another 3 days to have an email from them asking me for uploading the log files (I already did it at the time I opened the ticket !).It took another 3 days for them to call me, but I was busy and I couldn't work with them, after three more days, I called them again, but they told me they will call me back ... GRRR. I hate VMware support procedures and times.

Don't worry, I solved it by myself, and let me tell you how:

1) Shutdown the VDR appliance. It will free up the locked files in your VMs that were not sucessfuly backed up.

2) Create manually a snapshot in every VM with the problem, then "Delete all" snapshots will work !, you won't get that error message again.

3) "Try" to power on the VDR (VMware Data Recovery) appliance... Oh, no, the same message again! And now, I cannot power up the Virtual Machine !

4) I found the VDR "mounts" the hard disks of the VMs it is creating the backups, so, go to the VDR, and in commands "Edit settings".

By default, the VDR has only one hard disk, but mine shows three: those two extra hard disks corresponds to the Virtual Machine the backup failed!

Look at the hard disk description path for the first hard disk drive, and disk mode independent checkbox is not checked.



Look at the extra hard disks added to the VDR, see the hard disk description path (it corresponds to the VM that was in progress of backup). It also has the independent checkbox checked.





Select one by one the extra hard disks and click "Remove". Be very careful here, selecting just remove, and DO NOT dele files from disk.





Also, confirm that you selected the right hard disks ! If you make a mistake, just hit Cancel and do it again.

5) Verify that you have the single and right hard disk in place.



6) Power on the (VMware Data Recovery) appliance. Problem solved !



39 comments:

  1. You just saved me a lot of time, thanks for sharing this!

    ReplyDelete
  2. This is great!!! There is nothing on the VMWare communities about this at all. I was starting to get very worried :) Thanks so much for the great write up!!

    ReplyDelete
  3. This is one of the best troubleshooting guides I've seen. It fit my problem exactly AND fixed it! Thanks for taking the time to document this issue. :)

    --Nicole

    ReplyDelete
  4. Yeahhhhhhhhhh!!! Thanks a lot you save me :)

    ReplyDelete
  5. Thanks Very much - this has saved me a load of time today and possibly a client too...

    ReplyDelete
  6. You saved my day, Ed!!!! Thank you so very much ;-)

    ReplyDelete
  7. Thanks, good issue and great paper !

    ReplyDelete
  8. ED,

    This is really a nice stuff. it really saved my day. i got into this problem and searched for a solution in google and first got into this blog and it worked like a miracle.
    I am very pleased. Thanks buddy.

    With Best Wishes,
    Srinivas Kolla.

    ReplyDelete
  9. Man thanks for putting this out there, saved me big time!

    ReplyDelete
  10. Thank you very much. You saved me.

    ReplyDelete
  11. I had deleted an empty SAN volume that was mounted in a VM, and that's when my problems started. VMware tech support found that the machine had a second .vhd hard drive which was somehow connected to the deleted volume. I'm not sure how or why it was initially configured (I didnt' set it up), but removing the un-needed drive from the VM fixed the issue for me.

    ReplyDelete
  12. Thank You! last time this happened to me, we just ghosted the vm because we couldnt figure out what had it locked, after scouring blogs and vmware we were looking for locked files by processes through dmesg and all kinds of stuff.. now that i found this i was able to fix the issue in 5 mins!

    ReplyDelete
  13. w00t! VMware's kb articles had me headed in the wrong direction. Thanks !

    ReplyDelete
  14. Thanks for sharing, saved me a lot of time.

    ReplyDelete
  15. Seriously, thank you. I have used this guide a dozen times over the last 10months or so.

    Epic win.

    ReplyDelete
  16. WOW Thanks very much! This guide is ace :0)

    ReplyDelete
  17. hey.. thanks GOD finally i found this blog.

    i ever had similliar problem, i try same steps that you describe above and DONE. everythings working well

    now im facing the same problem, but at different VMs.. this VMs more critical, there i sno excuse for down time.. if im delete the Extra vDisk at VDR, i there restart needed or not at the VDR Appliance or the VMs that the vDisk Locked.

    Thanks & Regards

    ReplyDelete
  18. We're using PHD for backups. Your solution helped me. Thank you

    ReplyDelete
  19. Unable to access file since it is locked


    In my case, i have ESXi 4 standard edition i can't failover my virtual servers as well. So, the only thing which i did is Powered Off virtual servers and restart ESX machine and error resolved.

    ReplyDelete
  20. Yep...I had only 1 HDD...deleted it (not remove from disk) and readded existing. It created a new HDD and new SCSI controller. Booted right up! Thanks.

    ReplyDelete
  21. Thanks man, you saved my ass...

    ReplyDelete
  22. thanks a million man!

    ReplyDelete
  23. Major props for this write-up. I have come to the conclusion that VMware Data Recovery, at least in the 1.x series, is not production-ready. And nobody from VMware support seems to know anything about it. Will try again with VDR after our 5.0u1 upgrade next week I suppose.

    ReplyDelete
  24. Using veeam 5 with esx 4.1 and worked a treat, THANK YOU SO MUCH

    ReplyDelete
  25. Thanks for the article - it pointed me in the right direction, Acronis also causes this problem.

    ReplyDelete
  26. Thanks man, you saved my ass too :)

    ReplyDelete
  27. Had the same problem on our VCenter VM. But it still had the file in use after shutting down the VMware Data Recovery machine. So i set DRS to manual and vmotioned all the guests to other hosts except the VCenter and VDR machine (which were on this host). After that i shutdown the VCenter server. Connected with the VI-Client directly to the host put it in maintenance mode and rebooted it. After the reboot. I got it out of maintenance mode and removed the snapshot (be sure to take it out of maintenance mode first). Everything was fine afterwards, started the VCenter server and opened a beer.

    ReplyDelete
  28. Hello, I'm having the same issue when consolidation is needed. I'm not showing any other hard drives under the settings of the VM. Still says the file is locked. I even shut down the backup software VM. Still says file is locked. I have read they I would have to shut down the VM itself that needs consolidation. But doing this every time a backup my fail and the VM needs consolidation isn't practical. Does anyone have any recommendations? Thank you

    ReplyDelete
  29. Hello, I'm having the same issue when consolidation is needed. I'm not showing any other hard drives under the settings of the VM. Still says the file is locked. I even shut down the backup software VM. Still says file is locked. I have read they I would have to shut down the VM itself that needs consolidation. But doing this every time a backup my fail and the VM needs consolidation isn't practical. Does anyone have any recommendations? Thank you

    ReplyDelete
  30. restrhttp://www.veeam.com/kb1213

    ReplyDelete
  31. u r a star .... 100% works for me

    ReplyDelete
  32. Good stuff.
    Different backup product here, Vizioncore vRanger, same problem. I didn't even think to look there until I found your post.
    Found vRanger had mounted 3 of my VM's disks. Now my snapshot consolidation completed successfully and I can go back to mbralign'ing my vmdks.

    Thanks!

    ReplyDelete
  33. I´ve done all this but I still get...

    /sbin/fsck.xfs: LABEL = data12 does not exist

    FSK faild for at least one filesystem

    Now what?? Please help!

    ReplyDelete
  34. same happened to me. It works the way mentioned here . Very big thanks.

    ReplyDelete
  35. I was able to resolve the issue with a storage vMotion. Kinda annoying, but it worked. Hopefully it doesn't happen again tonight when backups run...

    ReplyDelete
  36. Thank you, saved me from a HUGE headache.

    ReplyDelete
  37. Perfect solution for a big problem. you saved the day

    ReplyDelete