One of the problems we had getting it to work was the NVRAM IV card . Being such an old machine, the battery backed cache on there was getting on a bit. Messages like this would come up in the console when booting:

In this situation, the machine sits in suspended animation for up to 10 hours until the battery charges to a certain level. It’s like being stuck in the bios. My battery is so old that it won’t hold enough charge, so I’m SOL.

You can override this charging phase by pressing ctrl-c when the filer is booting, however it will turn itself off after 24 hours, lest you should get complacent and think that your data is safe in the event of a power outage. NetApp as a company seem to be really serious about preventing data outage (mad respect yo)!

I think it would be cool to replace the battery in there, but I haven’t been able to find anywhere that will send me one for a reasonable price. Reading through the lines of the NetApp support site seams to suggest that the battery is only bundled with a NVRAM card, and that you can’t get them separately. I’ve cracked open the plastic seal on the battery and it looks like it is just a few camera batteries wired together and attached to a random circuit board. Haven’t been able to find any suitable parts yet to build my own, but if I do, you dear reader will be the first to know.

Until then I will disable the automatic shutdown, which seems to be undocumented, for obvious reasons. At the NetApp console, go :

It’ll periodically spit out little passive aggressive missives like this:

But just ignore them – it should keep on trucking!

I’ve noticed that Fedora has dropped the compat-libstdc++-296 and compat-libstdc++-33, which we require for DB2. The compat packages are in still the ‘Everything’ directory in the Fedora 12 mirror sites, so we can download them and add them to our repository. This approach would work with any RHEL/CentOS/Fedora type distro.

1. Extract the CD/DVD to a folder.
2. Copy the packages you would like to add to the Packages subdirectory
3. Cd to the top level directory of the extracted files (you should see directories like Packages, repodata, images as subdirectories )
4. Run this command :

This will update the package metadata in the repodata directory to include the new packages that you have added.
5. Add your new packages to the %packages section of the kickstart file – anaconda will now find the packages. For example, this is what mine says:

This package selection is really minimal and enables us to build a Fedora machine from nothing in about 5 minutes!

1. SSH keys. If you have a heavily locked down machine with ssh key logins, you cannot use the online converter.

2. Network. If you can’t hit the ‘source’ machine with ssh, you can’t convert it. Maybe it’s on another network, or has a firewall running on it.

3. So since all that didn’t work, you’re forced to install the converter on the same machine you are converting. Except you don’t want to do that because you will have to fill out lots of forms because you are installing a new piece of software on a production machine.

This is a fail safe method of converting a physical Linux server into a virtual one. I converted my physical machine to an ESX 4.0 host, but this method would probably work for other versions of VMware as well.

1. Power down the server.

2. Boot up the server using a linux live cd . (I like Slax)

3. At the command line, run:

Where server is a (linux) server with about twice as much disk space as you have, and /dev/sda is the address of the disk you would like to convert. You could probably do more that one disk using this howto, but I only did one.

4. Install Qemu on the Linux server. You should be able to find it in your yum repo, or apt-get repo.

5. Run this to convert the raw dd image to a vmdk.

6. Copy the sda.vmdk to your datastore, using scp if you have ssh enabled on your VMware host, or using the Datastore browser if you don’t. Make sure to create a folder for the new machine.

7. Create a new VM in the vSphere Console, using values similar to the physical machine you are replacing. Make sure to select Custom at the first screen, not typical.

8. When it comes time to create a new disk, point vSphere at your newly converted vmdk file.

9. Boot up the server, and you’re done!

The new driver has a number of drawbacks, most notably it is not supported as a boot device. VMware expect you to configure your OS on one ‘normal’ drive and then install a secondary drive with the paravirtualized adaptor. As well as being a pain from a management standpoint , it also ignores the speed increases that you would derive from having the OS on this faster device.

The method below will allow you to install Red Hat Enterprise 5.4 on your primary boot device using the pvscsi driver, and includes steps about how to automate the process with Redhat’s excellent kickstart system. If you are just interested in doing this on one system, I would follow these steps instead . My steps are good if you want to automate this procedure over many machines, but it’s probably overkill if you just want to try pvscsi on one machine. Hopefully these steps will be outdated soon when the pvscsi module makes it into the regular mainline kernel, which looks like it might happen in 2.6.33!

1. Create a new VM from the vSphere Console in the usual way.

2. Click edit settings on the new VM and select the SCSI controller 0 device, then up on the right hand side pane, click Change SCSI Controller Type and select VMware Paravirtual.

Now we must create boot media. I use iso images and the virtual cdrom device in VMware, but you could modify these steps and use them with PXE booting or USB booting quite easily. The process here is to add the pvscsi kernel module to the boot disk, so the RHEL installer can see the VMware disk.

3. Grab the prebuilt boot.iso from the images/ subdirectory and unextract it. You can do this by running:

You should now have an isolinux subdirectory in this directory.

4. Extract initrd.img . This is the initial ram disk which contains the boot files that the installer requires to run. Create a temp directory and extract it into it.

5. Now we need to extract (again!) the modules archive, so we can add pvscsi to it.

This will make a new subdirectory which will have the same name as your kernel version. Mine looks like this :
2.6.18-164.el5/x86_64
If you are running a 32 bit kernel or a different version of Redhat, yours will look slightly different.

6. Extract the VMware tools rpm and copy the precompiled driver into the modules directory. Make another temp directory and copy the Vmware Tools rpm into it. You should be able to get the VMware tools rpm from your ESX host or from the VMware website.

Then delete the temporary VMware tools directory.

7. Now we need to edit a bunch of text files that are in the modules subdirectory so Redhat can find the new driver.

At the bottom of module-info add:

At the bottom of modules.alias add:

At the bottom of modules.dep add:

In pci.ids, search for ’15ad’ . You will see an entry for a few VMware devices. To this list add:

The whole stanza will look something like this:

8. Now we need to compress everything. First the modules.cgz (and still in the modules directory) :

9. Compress the initrd.img

10. Everything is compressed up again, now to make a boot cd.

11. Copy pvscsi.iso to your ESX datastore and click edit settings on your paravirtualized guest vm. Set the cdrom to boot from pvscsi.iso .

This is enough to get the installation image to boot correctly with the pvscsi driver, but what will happen next is that the server will reboot using the regular kernel, and it won’t have the pvscsi driver available to it, so the machine will not reboot. We need to install and configure the VMware tools rpm after the RHEL install so when the machine reboots it will have access to the pvscsi driver file.

If you don’t want to use kickstart to do this, and the end of the install, drop into a shell by typing ALT-CTRL-F2 and download the VMware tools rpm from a local webserver, or you could copy it to the guest VM using scp or something. For example :

Reboot and you should be good. Read on if you would like to automate this procedure.

12.The easiest way to create a kickstart file is to run through a normal install and then to copy the ~/anaconda-ks.cfg file from the server, which will have all your settings saved in it. Or look at my example. . All you should need to do here is change the url, key, hostname, rootpw parameters and edit the url to the vmware tools rpm to something valid for your environment and you should be good to go.

Stay tuned for performance numbers to show if any of this is even worth it!

Working in System Test, you tend to see problems crop up that can only be recreated after a certain amount of time. A memory leak, for example, might not appear to cause any problems until well into a test run. And it’s tough luck if you haven’t enabled the correct settings to obtain diagnostic information to debug the problem. You need to stop the server, set the required parameters (verbose gc, Xdump parameters, whatever) and rerun the test to reproduce the problem, which may take days.

Today at work a situation arose where a run had failed and the server was hung. You could send a kill -3 (a sigquit) at the process it would spit out a javacore file which is useful if you have some sort of thread contention or deadlocks. However the developer working on this problem wanted a heap dump, something which is difficult to generate if the server is not responding.

(Incidently, if the server is responding, you can use wsadmin to generate a heap dump, which is way easier than the method described below).

I ended up stumbling upon a neat way to generate a heap dump when the server is not responding.

1. Run

Gcore is in the gdb package and forces the system to dump a binary core file. It should be about as large as the memory space that the java process was using when you ran it. Gcore is a Linux specific command; gdb proper should work on other unices, gcore is more of a convenience command. On Windows, use Dr Watson. The IBM support site has some excellent technotes around this sort of thing. I usually just google “Mustgather websphere hang” or something and they’ll often come up.

2. Run

to put all these nifty java tools in your path.

3. Run

This converts the system dump (core dump) into a useable format. It also produces an xml file that lists each active method and class taken at the time of the dump .

4. Run

Where “big number” is a memory size bigger than the initial core file in bytes.
jdmpview gives you a command line menu of ways you can poke and prod the dump file. And it is from here where you can construct a java heap dump from a system dump. Simply type heapdump at this prompt and it will create a phd (portable heapdump format) file which you can use in one of the many analysis tools in IBM Support Assistant, or else send it on to your friendly neighbourhood support professional. A word of warning. With a large system dump file the final ‘heapdump’ step seems to be quite slow. Still, it is better than waiting for days to reproduce the initial problem!

1. Install libstdc++5 from here. Apparently this has been dropped from karmic (9.10) but you can use 9.04 packages instead and it will work.

2. Make sure you can ping yourself on the fully qualified hostname that you will be using in the install. This is a good general tip for installing Portal. If the machine doesn’t respond, add an alias in your /etc/hosts file, like this:

Also think about disabling any firewalls . The install process needs to be able to connect to certain ports running on your machine.
If you let the install run all the way through, and then re enable your firewall after, debugging and adding new firewall exceptions should be way easier.

3. Drop this file in /etc . It will make AppServer think that you are running RHEL
5, which is a supported OS.

4. Unlink /bin/sh and link it to /bin/bash, like this:

Here’s a more detailed post about why you need to do this, if you are interested.

5. Don’t use sudo ./install.sh, but rather execute sudo su and then run ./install.sh . The installer executes a ConfigEngine.sh task late in the install, and if you use sudo to execute install.sh, the installer will attempt to run the ConfigEngine task as your logged in user, not as root, and the task will fail.

I think you should also be able to run the install as a non root user, I just haven’t tried it yet, I will post back when I have.

These steps also work with 6101.

Another guy in the office mentioned that he had converted his work laptop into a VM, so he could work at home on his nice quick desktop machine. I run Ubuntu 9.04 on the laptop and Windows 7 at home.

The T60 has a pretty hectic partitioning scheme, LVM, ext4, encrypted partitions, the whole works. This procedure handled it all. It makes a ‘bit for bit’ copy of the drive, and I knew it should work, but am happy it did anyway.

Here’s how to do it:

1. Boot up using a live cd. I used an Ubuntu 9.04 disk.
2. Grab a external drive bigger that the entire internal drive. The T60 hard drive is only 100GB, so this wasn’t hard.
3. Use fdisk -l to figure out which drive is which. My internal drive was /dev/sda and the external was /dev/sdb
4. Mount the external drive, if it isn’t mounted. Let’s say it’s mounted at /media/external
5. Run

If you don’t have an external drive, you could use netcat or scp or something, and send the drive image over the wire instead. It’ll just be much slower.

6. Reboot the laptop, you don’t need it anymore.
7. Copy drive.img to the host machine. I’m running Virtual Box 3.06 on Windows 7.
8. Run this command to convert the raw disk image to a Virtual Box drive image file.

9. Create a new Virtual Box Guest in the usual way, and point it to the vdi file. I used a SATA controller since my laptop has one, but not sure if this matters or not.
10. Boot up your new guest machine.

Hey presto, it’s the same! Now I don’t have to carry my laptop home anymore. It seems to run a bit slower than the native machine, but that’s to be expected.

This would probably work for a Windows physical machine too, although I think you’d want to add an extra step at the beginning. On the Windows laptop, set the IDE controller to use the Microsoft driver, rather than your default driver before you take the image. That way, when the guest OS boots up, it will have access to a ‘lowest common denominator’ disk driver.

Luckily enough, there’s an easy way to do this. The infocenters for WebSphere Application Server and WebSphere Portal are available in Eclipse Help System form, which is pretty much the same way they are presented over the web.

You can download the infocenter in Eclipse Help System format from this page for WAS and this page for Portal.

If you don’t have Eclipse already, download it from here. Any one of the versions should do. Unzip Eclipse and then copy the infocenter zips into the eclipse/plugins directory. Then unzip the infocenter zips in the plugins directory.

Now start Eclipse, and pick any old workspace. Click Help -> Help Contents, and the Infocenter zips that you copied in should be along side any other help files that come with Eclipse originally.

Now you have a local copy, make sure to update it since the infocenter is updated monthly.

When you edit the user’s properties this nasty error can appear if your Portal server is connected to an LDAP.

Error entering mail address into Self Care Portlet

Btw, this is a 6.1 or Portal.Next beta specific error, it should work fine on 6.0.

Here’s the full text of the error:

The reason this happens is that the portlet ( the self care portlet in this case)  is wired up to write the email address you entered in the form to a VMM attribute called ibm-primaryEmail . If your ldap schema doesn’t have a user attribute in it called ibm-primaryEmail , then you’re going to get an error when you try and write something to it.

Just to check it out, let’s look at the LDAP schema on this server (which is IBM Tivoli Directory Server 6.0)  . I’m using the awesome and free Apache Directory Studio to investigate the LDAP schema here.  Once the connection to the ldap is defined, go LDAP -> Open Schema Browser , and select the tab attribute types.

TDS ldap schema

Ok, so we have an attribute type ‘drink, favouriteDrink’ ;o) , but no ibm-primaryEmail . No matter, there is a ‘mail’ attribute there. We can make Portal use that to save email related attributes.

Open up wkplc.properties and find the section entitled LDAP Attribute Configuration (it’s near the bottom) . Here’s my completed one:

Cool, now run the task :

If you are using a federated ldap setup, edit the corresponding federated properties instead, and then run the following task:

. Restart the server and try the form again. It should correctly save the email attribute for the user and you can get on with sending email through Portal. Just for kicks, lets look at what that task did. It just edits the wimconfig file, which defines how VMM interacts . Open wimconfig.xml (wp_profile/config/cells/<cellname>/wim/config/wimconfig.xml) and search for ibm-primaryEmail.

Here is the part that does the mapping:

So the task is really just a (welcome) convenience, all it does it edit the xml file for you. Anyone who has tried to set up multirealms on 6.0 would be grateful for that!

After getting Portal installed on this Ubuntu machine, you’re probably going to want to transfer the default Derby database to something more robust like DB2. So you edit wkplc_comp.properties and wkplc_dbtype.properties, and start to run:

And you get this in the ConfigTrace.log

Hmm, ok, I thought db2 was working. A good habit when debugging these things is to take the piece that ConfigEngine is trying run and run it independently. So right now I want ConfigEngine to create an empty db2 database that I can run database-transfer against. Try this:

And that comes back successfully. However, that command sequence is not an accurate representation of what ConfigEngine is actually doing. We’re running ConfigEngine as root. But the ConfigEngine script is using the “user db2inst1 using ” modifiers on the end of the database create command. So how about this?

Ah ha, a failure. In the first example, DB2 already trusts the user that we’re logged is as (db2inst1), so it doesn’t need to go back to the operating system and authenticate it. In the second example, we are logged in as root, so db2 needs to go to the operating system and authenticate the user. Ubuntu uses the tried and true passwd + shadow file combo to store usernames and their associated passwords. The trouble is since Ubuntu 8.10, it uses the newer and more secure SHA512 hashing function to store the passwords, and DB2 doesn’t understand SHA512. So the workaround is to change the hashing function in use on the machine, reset the password and then we should be able to use the “user db2inst1 using ” type commands again.

Open /etc/pam.d/common-password in a text editor and change this line:

to

Then run passwd db2inst1 and put the same or a new password. If you look at the shadow file , the hash will change from something like this:
SHA512

MD5

(The unencrypted password here is ‘password’ – don’t get too excited!) .

Now try running ConfigEngine.sh create-database again. It should work. Make sure you change the system /etc/pam.d/common-password back to sha512, as you want the rest of your users to use this hashing function as it is more secure than md5sum . Hopefully DB2 should address this in a fixpack.