Installing RHEL 5 using the VMware Paravirtualized SCSI driver (pvscsi)

VMware ESX 4.0 offers a number of different SCSI disk controllers. One of the newer and more interesting ones is the paravirtualized scsi adapter. As with any type of paravirtualized  driver the paravirtualized scsi driver promises to be faster and use less CPU cycles than the other storage drivers, and since I/O performance is so important to overall system performance in virtualized environments, I thought it was worth a look. I don’t have any solid numbers about how it performs with WebSphere Portal, but once I do you will be the first to know.

The new driver has a number of drawbacks, most notably it is not supported as a boot device. VMware expect you to configure your OS on one ‘normal’ drive and then install a secondary drive with the paravirtualized adaptor. As well as being a pain from a management standpoint , it also ignores the speed increases that you would derive from having the OS on this faster device.

The method below will allow you to install Red Hat Enterprise 5.4 on your primary boot device using the pvscsi driver, and includes steps about how to automate the process with Redhat’s excellent kickstart system. If you are just interested in doing this on one system, I would follow these steps instead . My steps are good if you want to automate this procedure over many machines, but it’s probably overkill if you just want to try pvscsi on one machine. Hopefully these steps will be outdated soon when the pvscsi module makes it into the regular mainline kernel, which looks like it might happen in 2.6.33!

1. Create a new VM from the vSphere Console in the usual way.

2. Click edit settings on the new VM and select the SCSI controller 0 device, then up on the right hand side pane, click Change SCSI Controller Type and select VMware Paravirtual.

pvscsi-esx

Now we must create boot media. I use iso images and the virtual cdrom device in VMware, but you could modify these steps and use them with PXE booting or USB booting quite easily. The process here is to add the pvscsi kernel module to the boot disk, so the RHEL installer can see the VMware disk.

3. Grab the prebuilt boot.iso from the images/ subdirectory and unextract it. You can do this by running:

mount -o loop boot.iso /mnt/boot

cd /mnt/boot

mkdir ~/boot

cp -R * ~/boot

cd ~/boot

You should now have an isolinux subdirectory in this directory.

4. Extract initrd.img . This is the initial ram disk which contains the boot files that the installer requires to run. Create a temp directory and extract it into it.

mkdir tmp
cd tmp
gzip -dc ../initrd.img | cpio -ivd

5. Now we need to extract (again!) the modules archive, so we can add pvscsi to it.

cd modules
gzip -dc modules.cgz | cpio -ivd

This will make a new subdirectory which will have the same name as your kernel version. Mine looks like this :
2.6.18-164.el5/x86_64
If you are running a 32 bit kernel or a different version of Redhat, yours will look slightly different.

6. Extract the VMware tools rpm and copy the precompiled driver into the modules directory. Make another temp directory and copy the Vmware Tools rpm into it. You should be able to get the VMware tools rpm from your ESX host or from the VMware website.

rpm2cpio VMwareTools-4.0.0-latest.i386.rpm | cpio -idmv
find . -name pvscsi.o | grep 2.6.18-164.el5 (this will return the path to where the driver is - the one below is my path)
cp usr/lib/vmware-tools/modules/binary/bld-2.6.18-8.el5-x86_64smp-RHEL5/objects/pvscsi.o [temp dir from step 4]/modules/2.6.18-164.el5/x86_64/pvscsi.o
cp usr/lib/vmware-tools/modules/binary/bld-2.6.18-8.el5-x86_64smp-RHEL5/objects/pvscsi.o [temp dir from step 4]/modules/2.6.18-164.el5/x86_64/pvscsi.ko

Then delete the temporary VMware tools directory.

7. Now we need to edit a bunch of text files that are in the modules subdirectory so Redhat can find the new driver.

At the bottom of module-info add:

pvscsi
        scsi
        "VMware PVSCSI driver"

At the bottom of modules.alias add:

alias pci:v000015ADd000007C0sv*sd*bc*sc*i* pvscsi

At the bottom of modules.dep add:

pvscsi: scsi_mod

In pci.ids, search for ’15ad’ . You will see an entry for a few VMware devices. To this list add:

07c0  PVSCSI SCSI Controller

The whole stanza will look something like this:

15ad  VMware
        0405  SVGA II Adapter
        0710  SVGA Adapter
        07c0  PVSCSI SCSI Controller

8. Now we need to compress everything. First the modules.cgz (and still in the modules directory) :

find 2.6.18-164.el5 | cpio -H newc -ov | gzip -9 -c - > modules.cgz
rm -rf 2.6.18-164.el5

9. Compress the initrd.img

cd ..
find . | cpio -H newc -ov | gzip -9 -c - > initrd.img

10. Everything is compressed up again, now to make a boot cd.

cd ..
mkisofs -o pvscsi.iso -V pvscsi -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -T .

11. Copy pvscsi.iso to your ESX datastore and click edit settings on your paravirtualized guest vm. Set the cdrom to boot from pvscsi.iso .

This is enough to get the installation image to boot correctly with the pvscsi driver, but what will happen next is that the server will reboot using the regular kernel, and it won’t have the pvscsi driver available to it, so the machine will not reboot. We need to install and configure the VMware tools rpm after the RHEL install so when the machine reboots it will have access to the pvscsi driver file.

If you don’t want to use kickstart to do this, and the end of the install, drop into a shell by typing ALT-CTRL-F2 and download the VMware tools rpm from a local webserver, or you could copy it to the guest VM using scp or something. For example :

rpm -Uvh http://yourwebserver/vmwaretools.rpm
vmware-config-tools.pl --default

Reboot and you should be good. Read on if you would like to automate this procedure.

12.The easiest way to create a kickstart file is to run through a normal install and then to copy the ~/anaconda-ks.cfg file from the server, which will have all your settings saved in it. Or look at my example. . All you should need to do here is change the url, key, hostname, rootpw parameters and edit the url to the vmware tools rpm to something valid for your environment and you should be good to go.

Stay tuned for performance numbers to show if any of this is even worth it! 🙂

This entry was posted in howto and tagged , , , , . Bookmark the permalink.

13 Responses to Installing RHEL 5 using the VMware Paravirtualized SCSI driver (pvscsi)

  1. JeremyinNC says:

    Nice. On the Windows side there are massive gains if you’re VMs are doing a lot of IO. I ran an interesting test the other day – I took a Server 2008R2 x64 machine (laptop, I’m lazy and desktops are heavy but stay with me) and exported an NFS share using the built in NFS server. Then I mounted it as a volume on our ESX host, moved a VM to it and ran SQLIO tests *on the VM*. I got 2.6k IOPS (SSD disk).

    Then I ran the exact same test locally on the laptop. 2.7k IOPS. Obviously it’s limited by the laptop’s crappy controller but it also tells me that even on a relatively high IO VM you’re not paying a lot for the hypervisor.

    This is using NFS as the data protocol obviously (dual 1gig NICs to the storage switch from the ESX host, single 1gig on the laptop but that didn’t appear to be a bottleneck).

    Trying to get our Ubuntu VMs to boot off the paravirtual devices is not fun 🙁

  2. Is available this iso somewhere to download?

    tia

  3. Graham says:

    Hey Maurizio,

    I’m not sure if Redhat Enterprise is available to download, but the free version of it, CentOS is here . Keep in mind that this is just the boot cd, you will need to expose the rest of the install tree on a NFS/FTP/WWW server so your guest vm can get at it.

    Cheers,

    Graham.

  4. Sebastian says:

    Graham, it seems that whenever you upgrade the kernel you lose the paravirtualized driver, don’t you?

  5. Graham says:

    Hey Sebastian,

    Yeah that is to be expected. You need to relink the pvscsi kernel modules back to the new kernel that you’ve installed. Running the vmware-config-tools.pl script should do it for you.

    Cheers,

    Graham.

  6. Lucien Hercaud says:

    Hello
    I replaced isolinux/initrd.img onto the RedHat DVD with the one modified as described above. With this the installation goes “finger in the nose”.

    As an extra, I created (onto the same DVD) a directory where I copied the VMwareGuestAdditions RPM.
    Last, boot the same DVD in “linux rescue” (F2) mode and say you want the FileSystems mounted from the freshly installed OS, then mount the DVD onto /mnt/sysimage/cdrom (mkdir first), chroot as advised and “rpm -U VMwareGuestAdditions” followed by the run of the *.pl command (as advised) will make your disk bootable on to of the paravirtual SCSI driver.

  7. Graham says:

    Hi Lucien,

    I’m sorry the technique didn’t work for you. Thanks for replying with your method.

    Which version of RHEL were you using? I haven’t tried it with 6 yet.

    It is easier to do it your way if you’re doing it once, but I was trying to automate it so I could run it every time I build a new VM. I think your method could probably be adapted to be run completely hands off too.

  8. Jarrett Lee says:

    RHEL6 has the pvscsi module built-in. No need to do any of this for RHEL6. I can only assume that CentOS 6 will be the same way when they finally get around to doing a release.

  9. Graham says:

    Hey Jarrett,

    Great, I thought it might. Thanks very much for letting us know. I’m looking forward to CentOS 6 too 🙂

    Cheers,

    Graham.

  10. Lucien Hercaud says:

    ReHi

    Actually all worked very well and I’m very gratefull to the original post.

    I used the same method to add the vmxnet and vmxnet3 modules so that a network
    installation can go on. The boot.ISO is less than 20M. All the rest of the magic now happens in kickstart (%post) and the vmware tools get installed on the on-disk image.

    Regards

  11. Graham says:

    Hi Lucien,

    Thanks for writing back, glad it helped out. From what I’ve heard VMXNET 3 is always recommended, while PVSCSI is a bit more mixed. On my VMs their network card type is often marked as ‘flexible’ which I don’t really understand – should go back at investigate this a bit more. I do the vmware tools install in the kickstart as well (I love the –default option) .

    Have a good one,

    Graham.

  12. RomanT says:

    Mr G,

    Checkout vmware kb on the various types. As of 4.1, just go for VMXNET if you have VMtools installed. Although problem will be within the vmmaker ‘director’ – you’re sucking packages down pre-vmtools. Could always change it after…

    Anyway – I’ll leave you with that 🙂

    Cheers.,

    R

  13. Graham says:

    Thanks for the link! Since I do the VMware tools install during the kickstart phase, it looks like ‘Flexible’ will give me an Vlance adaptor during the kickstart stage and then a VMXNET adaptor once the machine reboots.

    I suspect that RHEL6 supports VMXNET3 out of the box, so I will try change my templated kickstart file to default to that on creation, so the ‘sucking packages’ part will go through VMXNET3. Paravirtualized goodness 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *