If you’d like to create your BoMC iso using Ubuntu , you might’ve noticed that BoMC doesn’t support it.

I’m on 10.04 and found that using the RHEL5 binary worked. At the time of writing, the name of the file is ibm_utl_bomc_3.00_rhel5_i386.bin . I tried the RHEL6 one first and it complained about a missing libssl library (libssl.so.10) so I would guess if you are on a newer version of Ubuntu, you should use this one instead.

You might see this error when executing it, and it will cause the downloads to fail

Simply execute this to create an Redhat release file at /etc/redhat-release , and this will make BoMC work.

Then simply execute it like this, put in your machine details, and out comes the iso:

I was asked by a customer how to uninstall Process Server, while keeping your Portal Server intact. I couldn’t find any documentation or steps to do this, so I figured it out for them.
“Your mileage may vary” – give this a try on a test box first, before you try on a production box. I didn’t test clusters either, although I don’t see how they would present a problem.

These steps are valid for 6.0 .

Unfortunately, the Process Server uninstaller doesn’t do a great job of cleaning up after itself, so we need to go around after it and clean up the mess.

1. Why not confirm you have Process Server installed first? Try:

If you get back something with this in it:

Bingo! You have Process server.

2. Run the config task bpe-unconfig . This config task will uninstall Process Server related components from your Portal Server (like the Task Container Enterprise Apps) .

3.

Stop all AppServer processes and then run the Process server uninstaller, which you will find at :

Take care to make sure this box below is unticked, so your whole AppServer isn’t uninstalled.

4. After the uninstaller is completed, delete the following jars in AppServer/lib . They are Process Server jars, and the Portal Server (or server1) will not start until you do this. As you can see there are tons of them. I got the list from seeing what was on the Process Server install iso. This list worked for me in my testing (which consisted of starting Portal and the Admin Console and making sure the UI still worked), but I can’t vouch for any custom code that you are running. As always, try this out in a test environment first, and take a backup before messing around with AppServer/lib . This seems to be a superset of what was installed on my system too – so don’t be alarmed if some of the jars in this list are not present on your system.

If you don’t delete these jars, you will get the following error message after issuing startServer.sh :

5. It’s also a good idea to clean out wp_profile/logs/* as there is a cache of jar file paths and their associated classes (the preload cache) that will need to be regenerated after deleting those jars.

6. The only thing the bpe-unconfig task doesn’t seem to do is delete the BPE shared library. To do this, start server1, and log in to the admin console. Navigate to Environment -> Shared Libraries and highlight the BPE library and click delete.

7. You’re done, start Portal and test everything works!

I also wrote a little script that does some checks and then does all this for you (except for step 6 which I couldn’t be inspired enough to write a jacl script for). It’s fairly quick and dirty, but if nothing else it will give you an idea of what has to happen to remove Process Server. You can get it here. Just rename the suffix after you download it. Understandably, WordPress would not yet me save a shell script up to my server.

When you create your VM in vSphere, simply select the “Custom” option, rather than the “Typical” one as custom allows you to pick your disk type. When the disk type radio button comes up select PVSCSI – and that’s it. When you run the Ubuntu install, the installation process will see the disk as normal, and there it is, you’re done.

I’m not sure exactly why this this works in Ubuntu – whether the pvscsi driver has been added to a later version of the kernel, or if Ubuntu/Debian has patched the kernel to include support for it.

While we are on the subject of paravirtualized drivers, the vmxnet3 driver also works out of the box in the Ubuntu 10.04 install.

I’ll give it a try on RHEL6 soon, I haven’t tried it yet and am curious if it works in the same way.

There are some caveats about pvscsi.

There are some very good reasons not to use it – first it’s not supported by VMware on the boot disk as described above, and secondly it should not be used in a low throughput environment.

Update: 11th Feb, 2011
The gotchas around pvscsi has changed – according to this KB article , the issue Scott identified with pvscsi in non intensive I/O workloads has gone away in vSphere 4.1, so . Thanks to at Matt Liebowitz at thelowercasew.com for spotting this. You can read his post on it here. . Matt asserts that the boot disk restriction is gone in 4.1 also, but it still unfortunately applies on Linux guests, according to this other KB article. I would suspect this is because it has been so tricky to configure. With it being included out of the box in Ubuntu Server, maybe they will lift this restriction soon? Personally, since I am running a dev and test lab, I think I will switch over to it exclusively anyway.

Recently, someone sent me a Windows DB2 backup for me to look at, and I wanted to move it to a Linux machine.

Here’s some steps about how you would do this.

1. On the source machine run:

where is the name of the database you want to move.

2. Copy the database.sql to your destination machine.

3. Create a database on the destination machine. You can use the same name, or use a different name and edit first connect statement in the database.sql file to point to the new name.

4. If you look at the database.sql, you will notice that the CREATE TABLESPACE commands will have paths in them. This will probably present a problem, since the paths are unlikely to exist on your destination machine. There are a number of ways you could deal with this, but the easiest in my opinion is to just use DB2′s automatic storage feature and let DB2 worry about it. To show you what I mean by this, here is a tablespace definition from my source database script, straight from db2look.

You can see that a Unix/Linux DB2 install would puke on the path in there. Using automatic storage command this becomes:

Much simpler, but you will need to change these tablespace definitions by hand.

5. After fixing up the tablespace paths, try executing:

I had a problem on my system, doing a Windows -> Linux restore. The database.sql had Windows line endings in it, and so when running it on Linux the command just hung. Running dos2unix database.sql fixed it. (This problem was the actual point of this post, but it seemed hard to explain by itself, so…. here we are).

6. Now you should have a skeleton database on the destination, with all the tables there just ready to receive data. To copy the actual data, we’ll be using db2move.
On the source machine, create a new folder and change into it, and then run:

7. This should fill the folder up with .ixf and .msg files . Copy the entire folder to your destination machine.

8. On the destination machine change to the export folder you just copied over and run:

9. db2load temporarily suspends the referential integrity constraints of DB2. This is done so it doesn’t matter which order you load tables in. But since you have suspended these checks while the data is being loaded in you will need to go and make db2 check each table, to make sure that any referential integrity constraints in the database are valid. If you don’t, you get back a message like this when accessing any tables that you have loaded:

This means – “You need to check this table, to make sure it is valid” .

To do this, you can query the system catalog table to get back a list of all the tables that need checking, and with a little awk, generate a script to run against the database. There’s probably a prettier way to do this, but it works.

Now you’re done. If you run a simple select statement against one of the tables you have loaded into the database, DB2 should return with the data. It’s worth noting that this method will work if changing versions of DB2, and will also cover moving from 32 bit versions of DB2 to 64 bit.

But my Was Password was correct. I had set it myself and just used it to log into the admin console.

The key is the “MethodDelegat” part of the log message. This column is the source of the log message. “Delegate method” made me think of bundled applications that use the RunAsRole . The pznscheduler application runs on a schedule, and sure enough also has a RunAsRole associated with it.

So what has happened, is that the WAS password has changed but the password is also saved with the pznscheduler application, and that saved password hasn’t been changed. Quite simply, there is a mismatch. This can happen on 6.1 and 7.0, and may also be related to the Portal Admin Password – it just depends which user is mapped as the RunAsRole . There are more applications that use the RunAsRole for Portal 6.0 .

For 6.0 here are the default out of the box applications you need to worry about when changing the password. LWP_TAI, pznscheduler, LWP_CAI, LWP_Security_Ext.

Of course, you may have other custom applications you need to worry about. The LWP apps use the Portal Admin user by default, so that’s why I didn’t have to change them.

For 6.1 you should check pznscheduler and CatalogHandler and for 7.0 the pznscheduler application is the only one you need to worry about.

So to fix it up, if you log into the WAS Admin Console and look for the pznscheduler app in the list of Enterprise Applications, you will see the “Map RunAs roles to users” link like in the picture below.

Then in the next dialog, reenter the new password.

This dialog can be a bit funny so you may have to play around with it a bit. Restart Portal and the error should be gone!

One of the problems we had getting it to work was the NVRAM IV card . Being such an old machine, the battery backed cache on there was getting on a bit. Messages like this would come up in the console when booting:

In this situation, the machine sits in suspended animation for up to 10 hours until the battery charges to a certain level. It’s like being stuck in the bios. My battery is so old that it won’t hold enough charge, so I’m SOL.

You can override this charging phase by pressing ctrl-c when the filer is booting, however it will turn itself off after 24 hours, lest you should get complacent and think that your data is safe in the event of a power outage. NetApp as a company seem to be really serious about preventing data outage (mad respect yo)!

I think it would be cool to replace the battery in there, but I haven’t been able to find anywhere that will send me one for a reasonable price. Reading through the lines of the NetApp support site seams to suggest that the battery is only bundled with a NVRAM card, and that you can’t get them separately. I’ve cracked open the plastic seal on the battery and it looks like it is just a few camera batteries wired together and attached to a random circuit board. Haven’t been able to find any suitable parts yet to build my own, but if I do, you dear reader will be the first to know.

Until then I will disable the automatic shutdown, which seems to be undocumented, for obvious reasons. At the NetApp console, go :

It’ll periodically spit out little passive aggressive missives like this:

But just ignore them – it should keep on trucking!

I’ve noticed that Fedora has dropped the compat-libstdc++-296 and compat-libstdc++-33, which we require for DB2. The compat packages are in still the ‘Everything’ directory in the Fedora 12 mirror sites, so we can download them and add them to our repository. This approach would work with any RHEL/CentOS/Fedora type distro.

1. Extract the CD/DVD to a folder.
2. Copy the packages you would like to add to the Packages subdirectory
3. Cd to the top level directory of the extracted files (you should see directories like Packages, repodata, images as subdirectories )
4. Run this command :

This will update the package metadata in the repodata directory to include the new packages that you have added.
5. Add your new packages to the %packages section of the kickstart file – anaconda will now find the packages. For example, this is what mine says:

This package selection is really minimal and enables us to build a Fedora machine from nothing in about 5 minutes!

1. SSH keys. If you have a heavily locked down machine with ssh key logins, you cannot use the online converter.

2. Network. If you can’t hit the ‘source’ machine with ssh, you can’t convert it. Maybe it’s on another network, or has a firewall running on it.

3. So since all that didn’t work, you’re forced to install the converter on the same machine you are converting. Except you don’t want to do that because you will have to fill out lots of forms because you are installing a new piece of software on a production machine.

This is a fail safe method of converting a physical Linux server into a virtual one. I converted my physical machine to an ESX 4.0 host, but this method would probably work for other versions of VMware as well.

1. Power down the server.

2. Boot up the server using a linux live cd . (I like Slax)

3. At the command line, run:

Where server is a (linux) server with about twice as much disk space as you have, and /dev/sda is the address of the disk you would like to convert. You could probably do more that one disk using this howto, but I only did one.

4. Install Qemu on the Linux server. You should be able to find it in your yum repo, or apt-get repo.

5. Run this to convert the raw dd image to a vmdk.

6. Copy the sda.vmdk to your datastore, using scp if you have ssh enabled on your VMware host, or using the Datastore browser if you don’t. Make sure to create a folder for the new machine.

7. Create a new VM in the vSphere Console, using values similar to the physical machine you are replacing. Make sure to select Custom at the first screen, not typical.

8. When it comes time to create a new disk, point vSphere at your newly converted vmdk file.

9. Boot up the server, and you’re done!

The new driver has a number of drawbacks, most notably it is not supported as a boot device. VMware expect you to configure your OS on one ‘normal’ drive and then install a secondary drive with the paravirtualized adaptor. As well as being a pain from a management standpoint , it also ignores the speed increases that you would derive from having the OS on this faster device.

The method below will allow you to install Red Hat Enterprise 5.4 on your primary boot device using the pvscsi driver, and includes steps about how to automate the process with Redhat’s excellent kickstart system. If you are just interested in doing this on one system, I would follow these steps instead . My steps are good if you want to automate this procedure over many machines, but it’s probably overkill if you just want to try pvscsi on one machine. Hopefully these steps will be outdated soon when the pvscsi module makes it into the regular mainline kernel, which looks like it might happen in 2.6.33!

1. Create a new VM from the vSphere Console in the usual way.

2. Click edit settings on the new VM and select the SCSI controller 0 device, then up on the right hand side pane, click Change SCSI Controller Type and select VMware Paravirtual.

Now we must create boot media. I use iso images and the virtual cdrom device in VMware, but you could modify these steps and use them with PXE booting or USB booting quite easily. The process here is to add the pvscsi kernel module to the boot disk, so the RHEL installer can see the VMware disk.

3. Grab the prebuilt boot.iso from the images/ subdirectory and unextract it. You can do this by running:

You should now have an isolinux subdirectory in this directory.

4. Extract initrd.img . This is the initial ram disk which contains the boot files that the installer requires to run. Create a temp directory and extract it into it.

5. Now we need to extract (again!) the modules archive, so we can add pvscsi to it.

This will make a new subdirectory which will have the same name as your kernel version. Mine looks like this :
2.6.18-164.el5/x86_64
If you are running a 32 bit kernel or a different version of Redhat, yours will look slightly different.

6. Extract the VMware tools rpm and copy the precompiled driver into the modules directory. Make another temp directory and copy the Vmware Tools rpm into it. You should be able to get the VMware tools rpm from your ESX host or from the VMware website.

Then delete the temporary VMware tools directory.

7. Now we need to edit a bunch of text files that are in the modules subdirectory so Redhat can find the new driver.

At the bottom of module-info add:

At the bottom of modules.alias add:

At the bottom of modules.dep add:

In pci.ids, search for ’15ad’ . You will see an entry for a few VMware devices. To this list add:

The whole stanza will look something like this:

8. Now we need to compress everything. First the modules.cgz (and still in the modules directory) :

9. Compress the initrd.img

10. Everything is compressed up again, now to make a boot cd.

11. Copy pvscsi.iso to your ESX datastore and click edit settings on your paravirtualized guest vm. Set the cdrom to boot from pvscsi.iso .

This is enough to get the installation image to boot correctly with the pvscsi driver, but what will happen next is that the server will reboot using the regular kernel, and it won’t have the pvscsi driver available to it, so the machine will not reboot. We need to install and configure the VMware tools rpm after the RHEL install so when the machine reboots it will have access to the pvscsi driver file.

If you don’t want to use kickstart to do this, and the end of the install, drop into a shell by typing ALT-CTRL-F2 and download the VMware tools rpm from a local webserver, or you could copy it to the guest VM using scp or something. For example :

Reboot and you should be good. Read on if you would like to automate this procedure.

12.The easiest way to create a kickstart file is to run through a normal install and then to copy the ~/anaconda-ks.cfg file from the server, which will have all your settings saved in it. Or look at my example. . All you should need to do here is change the url, key, hostname, rootpw parameters and edit the url to the vmware tools rpm to something valid for your environment and you should be good to go.

Stay tuned for performance numbers to show if any of this is even worth it!

Working in System Test, you tend to see problems crop up that can only be recreated after a certain amount of time. A memory leak, for example, might not appear to cause any problems until well into a test run. And it’s tough luck if you haven’t enabled the correct settings to obtain diagnostic information to debug the problem. You need to stop the server, set the required parameters (verbose gc, Xdump parameters, whatever) and rerun the test to reproduce the problem, which may take days.

Today at work a situation arose where a run had failed and the server was hung. You could send a kill -3 (a sigquit) at the process it would spit out a javacore file which is useful if you have some sort of thread contention or deadlocks. However the developer working on this problem wanted a heap dump, something which is difficult to generate if the server is not responding.

(Incidently, if the server is responding, you can use wsadmin to generate a heap dump, which is way easier than the method described below).

I ended up stumbling upon a neat way to generate a heap dump when the server is not responding.

1. Run

Gcore is in the gdb package and forces the system to dump a binary core file. It should be about as large as the memory space that the java process was using when you ran it. Gcore is a Linux specific command; gdb proper should work on other unices, gcore is more of a convenience command. On Windows, use Dr Watson. The IBM support site has some excellent technotes around this sort of thing. I usually just google “Mustgather websphere hang” or something and they’ll often come up.

2. Run

to put all these nifty java tools in your path.

3. Run

This converts the system dump (core dump) into a useable format. It also produces an xml file that lists each active method and class taken at the time of the dump .

4. Run

Where “big number” is a memory size bigger than the initial core file in bytes.
jdmpview gives you a command line menu of ways you can poke and prod the dump file. And it is from here where you can construct a java heap dump from a system dump. Simply type heapdump at this prompt and it will create a phd (portable heapdump format) file which you can use in one of the many analysis tools in IBM Support Assistant, or else send it on to your friendly neighbourhood support professional. A word of warning. With a large system dump file the final ‘heapdump’ step seems to be quite slow. Still, it is better than waiting for days to reproduce the initial problem!