This problem has been annoying me for ages, and I’m excited to finally have an answer to it. Every now and again, users would complain that performance had dropped through the floor. I’ve got pretty used assuming that bad VM performance = bad storage performance , and so jump onto the N Series / filer straight away.

is a good start when investigating – it’ll show you how many NFS packets are passing through the filer, and the CPU, Network and Disk utilization levels needed to service the requests . The Disk Util column is interesting too – it isn’t an average or anything, it’s the busiest disk in the filer. And since we’ve got 50-ish disks in the filer.

This is what a normal sysstat output looks like. (At least for me).

And this is one when all hell is breaking loose:

See the NFS column? It’s not like the CPU is busy because it is servicing NFS requests. It’s not being overtaxed by the VMs – it’s something internal to the filer. Even the Disk Util isn’t very high. What’s going on?

In our particular environment, our VMs are really disposable, since the build of WCM that goes into them is obsolete the next day. At any time the filer might be half filled with switched off VMs. Eventually, the filer fills up and you have to delete all the obsolete VMs. This is what has caused the high CPU – deleting a bunch of VMs. Each are about 25 gigs and I must have deleted around 200 or something. The deleting process itself is quite quick, but it spiked the CPU for about an hour. An agonizing hour!

At the very least, it’s great to know what was going on. I’ll open a case with IBM support and report back (if I can). In the meantime better write a delete queue to try not to tax it as badly.

We don’t really give you a ‘scriptable’ way to do this, which would be good if you needed to do it many many times. The manual steps are described in the infocenter topic for Portal 7.0 here (step 10).

This is easily remedied with a quick ConfigEngine script:

Simply copy this to a new xml file under PortalServer/wcm/prereq.wcm/config/includes, and then on each node in your cluster, run

Alternatively, if you don’t want to copy the script to each node, you can set the ServerName and NodeName variables on the command line too and just invoke the task as many times as you have cluster nodes.

If you’d like to create your BoMC iso using Ubuntu , you might’ve noticed that BoMC doesn’t support it.

I’m on 10.04 and found that using the RHEL5 binary worked. At the time of writing, the name of the file is ibm_utl_bomc_3.00_rhel5_i386.bin . I tried the RHEL6 one first and it complained about a missing libssl library (libssl.so.10) so I would guess if you are on a newer version of Ubuntu, you should use this one instead.

You might see this error when executing it, and it will cause the downloads to fail

Simply execute this to create an Redhat release file at /etc/redhat-release , and this will make BoMC work.

Then simply execute it like this, put in your machine details, and out comes the iso:

This particular time, this error cropped up when trying to run Eclipse Memory Analyzer over a X11 Forwarding session using SSH. Eclipse would throw this error and fail to start.

To fix it, run

If you don’t have dbus-uuidgen , it’s in the dbus package, which can be installed by issuing yum install dbus (of course!).

Happy weekend!

For a number of our VMware guests, when being a good guy and installing VMware Tools, this error would come up.

Seems like a bit of an overreaction to try to get you to reinstall ESXi.

Now I use the very cool repo/unmanaged solution for the majority of our guests, since they run RHEL 6. If you don’t know about this check it out here. If you use RHEL kickstart, it is trivial to make new guests come out with tools installed and the tools repo enabled. Anyway, enough gushing, just check it out if you use RHEL on VMware.

I should note before getting into this, that I am running ESXi from a USB key. This problem seems to be related to the backend media that ESXi is installed on, so if you’re on different media then your mileage may vary. I’d be interested to hear from anyone who sees this error on a non usb platform.

Following the path given in the error on a good system (/usr/lib/vmware/isoimages/) leads you to a symlink called /productLocker/vmtools/ , which in turn is a symlink to /locker/packages/4.1.0/ . No idea why this has to be so complicated, but there’s probably a reason.

After some more digging (tracking back through the innumerable symlinks), it seemed like the partition below was missing.

On a bad system

This is from a good system, notice the last partition

The mount point for this partition seems to be

Which was missing too. /vmfs/volumes/Hypervisor3 is a symlink back to the volume id of the partition. In the end, I just couldn’t figure out a pattern. Some hosts were fine and had the /productLocker symlink, and a bunch didn’t have it. Consequently, I have no idea how this happened to us, or how to prevent it, but let’s just get on and fix it. The solution seems clear, let’s just make a new partition, format it, and dump the files from a working system on there.

Partitioning the disk as FAT16 was easy (fdisk ftw), but formatting the disk was proving difficult. ESXi doesn’t seem to contain a working version of mkfs.vfat on it, and vmkfstools won’t do FAT16. (Also tried a mkfs.vfat binary from RHEL 5; didn’t work either). Of course, I could’ve pulled the usb key out, and fixed it up on a linux box, but I found an easier way.

To summarize, you create a scratch disk directory for the host inside some of your shared storage, and then point the host to it. After rebooting the host all the symlinks point to the new scratch location on your datastore. For complete steps on how to do it read this. VMware also recommends this as a best practice since your ESXi scratch disk area is now not on a limited ram disk.

Ok. Great. Now ESXi has a place to put VMware Tools. But how do you get the Tools files on there? I found the easiest way to do this was to apply a VMware Tools fix from VUM (VMware Update Manager). If you don’t use VUM I’m sure you could manually download the fix and apply it on your host using esxupdate. The latest VMware Tools fix is here. And that’s it – problem solved!

Boom!

#Original Post#
We’re working on a boot from SAN ‘modernization’ project at the moment. Which is another way to say we’re getting rid of it. It’s way too complicated! And if the SAN goes down, all the boot partitions go with it. This just introduces an extra dependency that can completely nullify any fault tolerant or HA strategies that you may have in place in the vSphere layer. No prizes for guessing who this has happened to recently!

What to do instead? The new HS22V blade systems we are using have a tiny little usb port on the motherboard where you can install a USB key. Load ESXi on it, and you’re golden.

So I need to make a list of which hosts need fixing up and which are booting from usb already or are using local storage. We’d started this process recently but want to step it up now. But which ESXi machines in the farm are booting from what ? How can you tell?

With fdisk of course – simply enable Remote Tech Support mode (see here for details) and ssh into your host.

Run

and look for the ‘*’ in the boot column. That’s the partition that ESXi is booting from. If you see a device called /dev/disks/mpx.vmhba32:C0:T0:L0 , you’re booting from usb. If the device is something more like /dev/disks/naa.600508e000000000194a56b4310b4804 you are booting from SAN or a local disk.

Here’s the fdisk output for my usb stick.

I was asked by a customer how to uninstall Process Server, while keeping your Portal Server intact. I couldn’t find any documentation or steps to do this, so I figured it out for them.
“Your mileage may vary” – give this a try on a test box first, before you try on a production box. I didn’t test clusters either, although I don’t see how they would present a problem.

These steps are valid for 6.0 .

Unfortunately, the Process Server uninstaller doesn’t do a great job of cleaning up after itself, so we need to go around after it and clean up the mess.

1. Why not confirm you have Process Server installed first? Try:

If you get back something with this in it:

Bingo! You have Process server.

2. Run the config task bpe-unconfig . This config task will uninstall Process Server related components from your Portal Server (like the Task Container Enterprise Apps) .

3.

Stop all AppServer processes and then run the Process server uninstaller, which you will find at :

Take care to make sure this box below is unticked, so your whole AppServer isn’t uninstalled.

4. After the uninstaller is completed, delete the following jars in AppServer/lib . They are Process Server jars, and the Portal Server (or server1) will not start until you do this. As you can see there are tons of them. I got the list from seeing what was on the Process Server install iso. This list worked for me in my testing (which consisted of starting Portal and the Admin Console and making sure the UI still worked), but I can’t vouch for any custom code that you are running. As always, try this out in a test environment first, and take a backup before messing around with AppServer/lib . This seems to be a superset of what was installed on my system too – so don’t be alarmed if some of the jars in this list are not present on your system.

If you don’t delete these jars, you will get the following error message after issuing startServer.sh :

5. It’s also a good idea to clean out wp_profile/logs/* as there is a cache of jar file paths and their associated classes (the preload cache) that will need to be regenerated after deleting those jars.

6. The only thing the bpe-unconfig task doesn’t seem to do is delete the BPE shared library. To do this, start server1, and log in to the admin console. Navigate to Environment -> Shared Libraries and highlight the BPE library and click delete.

7. You’re done, start Portal and test everything works!

I also wrote a little script that does some checks and then does all this for you (except for step 6 which I couldn’t be inspired enough to write a jacl script for). It’s fairly quick and dirty, but if nothing else it will give you an idea of what has to happen to remove Process Server. You can get it here. Just rename the suffix after you download it. Understandably, WordPress would not yet me save a shell script up to my server.

When you create your VM in vSphere, simply select the “Custom” option, rather than the “Typical” one as custom allows you to pick your disk type. When the disk type radio button comes up select PVSCSI – and that’s it. When you run the Ubuntu install, the installation process will see the disk as normal, and there it is, you’re done.

I’m not sure exactly why this this works in Ubuntu – whether the pvscsi driver has been added to a later version of the kernel, or if Ubuntu/Debian has patched the kernel to include support for it.

While we are on the subject of paravirtualized drivers, the vmxnet3 driver also works out of the box in the Ubuntu 10.04 install.

I’ll give it a try on RHEL6 soon, I haven’t tried it yet and am curious if it works in the same way.

There are some caveats about pvscsi.

There are some very good reasons not to use it – first it’s not supported by VMware on the boot disk as described above, and secondly it should not be used in a low throughput environment.

Update: 11th Feb, 2011
The gotchas around pvscsi has changed – according to this KB article , the issue Scott identified with pvscsi in non intensive I/O workloads has gone away in vSphere 4.1, so . Thanks to at Matt Liebowitz at thelowercasew.com for spotting this. You can read his post on it here. . Matt asserts that the boot disk restriction is gone in 4.1 also, but it still unfortunately applies on Linux guests, according to this other KB article. I would suspect this is because it has been so tricky to configure. With it being included out of the box in Ubuntu Server, maybe they will lift this restriction soon? Personally, since I am running a dev and test lab, I think I will switch over to it exclusively anyway.

To recap, this error happens because DB2 can’t understand the more modern and secure sha512 password cipher and must use the older md5 cipher . About 6 months ago I wrote a post about how to workaround the problem on Ubuntu and other modern Linux distributions.

To make a long story short, this problem goes away with DB2 9.5 fixpack 5 , and DB2 9.7 . It is still present in DB2 9.1 fixpack 9, which is the most recent version of the DB2 9.1 series at the moment, so you will need to perform the steps in the linked post above if you need to be on this version. Otherwise, it would be much simpler and more secure to go to the latest fixpack of 9.5 or 9.7.

One of the great things about how DB2 releases fixpacks is that they are installable. What I mean by this is that you don’t have to install the first version of the software and apply patches to it, you can just apply the patch at the outset.

When testing out DB2 9.7 fixpack 2 on Linux, I hit a weird error:

It was a bit more like a major error. The DB2 installer could not create the instance user and so could not create the instance. Which means the system is pretty much useless. The following turned up in the db2setup.log:

What was interesting was the installer would breeze through creating the groups, but baulk on creating the users. The problem turned out to be a mismatch between the algorithm used to encrypt and read back the passwords in the response file. Something changed in the algorithm in DB2 9.7 FP2 and I was using an old response file from the GA version of the code . Both versions are incompatible with each other. You can clearly see the differences below.

9.7 GA, 9.7 FP1 (also works with 9.5) .

9.7 FP2

Obviously security has been increased in FP2 because the password hash is so much longer. Please note that it’s not forwards compatible either. What I mean by this is that the newer version of the response file will not work with older versions of the code either.

To conclude, the solution in this case is simple. Rerun the installer that comes with DB2 9.7 FP2 and generate a new response file for use with this version. I looked through the DB2 APARs list trying to find the fix that changed the encryption algorithm, but couldn’t see it.