You know those issues that just sit in the background for a while until you can’t take it anymore and have to work them out. You have to fix them. You just have to make them stop?!? (Maybe not, heh). This problem was one of those ones.
For a number of our VMware guests, when being a good guy and installing VMware Tools, this error would come up.
Unable to install VMware Tools. An error occurred while trying to access image file "/usr/lib/vmware/isoimages/solaris.iso" needed to install VMware Tools: 2 (No such file or directory). If your product shipped with the VMware Tools package, reinstall VMware ESX, then try again to install the VMware Tools package in the virtual machine.
The required VMware Tools ISO image does not exist or is inaccessible.
Seems like a bit of an overreaction to try to get you to reinstall ESXi.
Now I use the very cool repo/unmanaged solution for the majority of our guests, since they run RHEL 6. If you don’t know about this check it out here. If you use RHEL kickstart, it is trivial to make new guests come out with tools installed and the tools repo enabled. Anyway, enough gushing, just check it out if you use RHEL on VMware.
I should note before getting into this, that I am running ESXi from a USB key. This problem seems to be related to the backend media that ESXi is installed on, so if you’re on different media then your mileage may vary. I’d be interested to hear from anyone who sees this error on a non usb platform.
Following the path given in the error on a good system (/usr/lib/vmware/isoimages/) leads you to a symlink called /productLocker/vmtools/ , which in turn is a symlink to /locker/packages/4.1.0/ . No idea why this has to be so complicated, but there’s probably a reason.
After some more digging (tracking back through the innumerable symlinks), it seemed like the partition below was missing.
On a bad system
This is from a good system, notice the last partition
The mount point for this partition seems to be
Which was missing too. /vmfs/volumes/Hypervisor3 is a symlink back to the volume id of the partition. In the end, I just couldn’t figure out a pattern. Some hosts were fine and had the /productLocker symlink, and a bunch didn’t have it. Consequently, I have no idea how this happened to us, or how to prevent it, but let’s just get on and fix it. The solution seems clear, let’s just make a new partition, format it, and dump the files from a working system on there.
Partitioning the disk as FAT16 was easy (fdisk ftw), but formatting the disk was proving difficult. ESXi doesn’t seem to contain a working version of mkfs.vfat on it, and vmkfstools won’t do FAT16. (Also tried a mkfs.vfat binary from RHEL 5; didn’t work either). Of course, I could’ve pulled the usb key out, and fixed it up on a linux box, but I found an easier way.
To summarize, you create a scratch disk directory for the host inside some of your shared storage, and then point the host to it. After rebooting the host all the symlinks point to the new scratch location on your datastore. For complete steps on how to do it read this. VMware also recommends this as a best practice since your ESXi scratch disk area is now not on a limited ram disk.
Ok. Great. Now ESXi has a place to put VMware Tools. But how do you get the Tools files on there? I found the easiest way to do this was to apply a VMware Tools fix from VUM (VMware Update Manager). If you don’t use VUM I’m sure you could manually download the fix and apply it on your host using esxupdate. The latest VMware Tools fix is here. And that’s it – problem solved!
Graham
Thanks for this – will go and read again, but wondered whether you’d already blogged about (a) the repo/unmanaged solution and (b) running ESXi from a USB key ?
Again, thanks
Hey Dave,
Nice to hear from you. I blogged once about how to add a package to an arbitrary repo but I’ve never mentioned running a VMware tools repo before. I’ve have blogged about running ESXi on a USB stick, but this is really a workaround for a bug that crops up when you run on a USB stick.
I’ll try to be a bit clearer in the future. It’s hard to get the balance between too much background detail and “just the answer” , and I definitely tend towards the former!
Cheers,
Graham.
Graham – I’m pretty sure we’re seeing the same issue. And I’m pretty sure there’s a private VMware KB article on it. I’ve raised a case to try and get its contents.
[Internal] IBM Embedded ESXi system is missing partition 8 which prevents VMware Tools images from being mounted (#1038851)
We’re running ESXi 4.1 (U2) on x3650 & x3850’s
Hi James – wow that sounds exactly like it. That’s interesting, I didn’t know it was IBM hardware specific (but it seems to be?). Don’t know if I included it, but I’m seeing these on IBM HS22 blades. We didn’t get preinstalled USB keys or anything either, we just installed ESXi manually on each one.
Curious if you built your machine using the iso install or dd’ing to the image file to your usb key.
Not sure why but I could build my machine using the 4.0 iso but could not w/ 4.1 so I dd to a USB key the 4.1 version. I suspect it is due to the IDE Hard Drive I use, if I use a SATA HD it will build. After I dd the image, I restored the settings using vicfg-cfgbackup
Another issue could be that this is a Gigabyte Motherboard and at one point I attempted to fix the partition order to get it to boot of the USB key. I fixed it by place this key on a linux box and going into expert mode and fixing the partition order.
Just a follow up on my last post…
I pulled the USB key and placed it in my linux box, ran fdisk, went to expert mode, and fixed the partitioning and now vmtools works.
My situation is probably different from you all, in that this is a white box with a Gigabyte motherboard (Gigabyte Motherboards do not play well with ESXi USB keys), I dd’d the image after I bunzip’d the installer from vmware, and this a restore from backup.
Hi Lunakoa,
I used the 4.1 iso install and pointed it at the USB keys each time. Your fdisk solution sounds like it would’ve worked for me too.
Cheers,
Graham.
I have the same issue on one of my IBM x3650’s that I’m booting from external USB stick with ESXi 4.1. This is the first of 4 that I have done that has this issue. The 3650’s are booting images from SAN. Can you step me through the best way to fix this without causing disruption to the 3 Guests running on this machine?
Sorry limited knowledge with Linux… 😉
@Graham – I don’t see missing partitions as you have shown in this blog. I do see however missing Hypervisor1 through 3 on the effected machine (I’m not getting the same sort of output from fdisk -l, so it’s most likely just me…)
Hi Barry,
I think if I had to keep the guests going (and Vmotion to another host isn’t an option) I would download the VMware tools iso file for your guest os from http://packages.vmware.com/tools/esx/4.1u2/index.html, copy it to your datastore, and then attach the iso file to the CD drive in the guest and install as normal. Then at least you will have VMware Tools installed in your guest VMs.
As to fixing the root cause, I think some more investigation is in order. Maybe have a look at esxcfg-info -s? You get way to much info, but if you redirect it to a file and have a look through it for ‘usb’ maybe it’ll yield some insights. Feel free to post it here if you want. Maybe compare it with one of the working hosts that you have? If you also try vmkfstools -P /vmfs/volumes/Hypervisor1 on one of the working hosts, it’ll print out where the Hypervisor1 partition actually lives on the usb disk. Maybe that partition doesn’t exist on the bad machine?
Check out this post too -> http://vm-help.com/esx/esx3i/check_system_partitions.php , maybe you have a bad partition in there somewhere?
Hope some of that helps!
Graham.
Thank you Graham. Excellent info!
Hey Graham – just as a follow up
VMware and IBM both suggested that this was a due to a batch of faulty IBM USB keys. On the faulty keys, re-imaging the keys made no difference – productLocker (p8) would never appear. I did not try your manual re-creation method but I presume it would have worked.
IBM sent us 8 (blank) replacement keys, we re-imaged them and they’ve all been good since.
We’re both Sydney based so maybe an AUST/APAC thing?