How to add additional packages to a Fedora repo

It’s time to give Fedora 12 a whirl on our servers. In the lab we are massive users of anaconda’s kickstart system for doing unattended installs. This means that everything has to be perfect, otherwise the install will hang. For instance, if a package that you have defined in the kickstart isn’t there, anaconda (the name for Fedora’s installer) will hang.

I’ve noticed that Fedora has dropped the compat-libstdc++-296 and compat-libstdc++-33, which we require for DB2. The compat packages are in still the ‘Everything’ directory in the Fedora 12 mirror sites, so we can download them and add them to our repository. This approach would work with any RHEL/CentOS/Fedora type distro.

1. Extract the CD/DVD to a folder.
2. Copy the packages you would like to add to the Packages subdirectory
3. Cd to the top level directory of the extracted files (you should see directories like Packages, repodata, images as subdirectories )
4. Run this command :

createrepo --update -g repodata/comps.xml .

This will update the package metadata in the repodata directory to include the new packages that you have added.
5. Add your new packages to the %packages section of the kickstart file – anaconda will now find the packages. For example, this is what mine says:

%packages --nobase


This package selection is really minimal and enables us to build a Fedora machine from nothing in about 5 minutes!

Posted in howto | Tagged , , , , , , | Leave a comment

Vmware player on RHEL 5: How to get rid of Clearlooks error

Quick tip: if you get this error when starting vmplayer from the command line on RHEL 5:

/usr/share/themes/Clearlooks/gtk-2.0/gtkrc:59: error: unexpected identifier `animation', expected character `}'

Just open the file /usr/share/themes/Clearlooks/gtk-2.0/gtkrc and comment out line 59, like so:

#animation = FALSE

This doesn’t seem to affect functionality, but if you’re one of those people who wants their logs to be clean, the fix is easy.

Posted in Uncategorized | Leave a comment

Offline Physical 2 Virtual conversion (p2v) using free tools

I’m a bit confused about VMware’s Converter and when you can use it. Which offering is free? Which is paid? From the last time I looked, I think you need the enterprise product to do an offline conversion. And call me old school, but online conversion just seems like voodoo to me. I was in the office this weekend, trying to convert an important physical machine to a virtual one and it failed for all sorts of reasons:

1. SSH keys. If you have a heavily locked down machine with ssh key logins, you cannot use the online converter.

2. Network. If you can’t hit the ‘source’ machine with ssh, you can’t convert it. Maybe it’s on another network, or has a firewall running on it.

3. So since all that didn’t work, you’re forced to install the converter on the same machine you are converting. Except you don’t want to do that because you will have to fill out lots of forms because you are installing a new piece of software on a production machine.

This is a fail safe method of converting a physical Linux server into a virtual one. I converted my physical machine to an ESX 4.0 host, but this method would probably work for other versions of VMware as well.

1. Power down the server.

2. Boot up the server using a linux live cd . (I like Slax)

3. At the command line, run:

dd if=/dev/sda | ssh user@server 'dd of=sda.img'

Where server is a (linux) server with about twice as much disk space as you have, and /dev/sda is the address of the disk you would like to convert. You could probably do more that one disk using this howto, but I only did one.

4. Install Qemu on the Linux server. You should be able to find it in your yum repo, or apt-get repo.

5. Run this to convert the raw dd image to a vmdk.

qemu-img convert sda.img -O vmdk sda.vmdk

6. Copy the sda.vmdk to your datastore, using scp if you have ssh enabled on your VMware host, or using the Datastore browser if you don’t. Make sure to create a folder for the new machine.

7. Create a new VM in the vSphere Console, using values similar to the physical machine you are replacing. Make sure to select Custom at the first screen, not typical.

8. When it comes time to create a new disk, point vSphere at your newly converted vmdk file.

9. Boot up the server, and you’re done!

Posted in howto | Tagged , | Leave a comment

Installing RHEL 5 using the VMware Paravirtualized SCSI driver (pvscsi)

VMware ESX 4.0 offers a number of different SCSI disk controllers. One of the newer and more interesting ones is the paravirtualized scsi adapter. As with any type of paravirtualized  driver the paravirtualized scsi driver promises to be faster and use less CPU cycles than the other storage drivers, and since I/O performance is so important to overall system performance in virtualized environments, I thought it was worth a look. I don’t have any solid numbers about how it performs with WebSphere Portal, but once I do you will be the first to know.

The new driver has a number of drawbacks, most notably it is not supported as a boot device. VMware expect you to configure your OS on one ‘normal’ drive and then install a secondary drive with the paravirtualized adaptor. As well as being a pain from a management standpoint , it also ignores the speed increases that you would derive from having the OS on this faster device.

The method below will allow you to install Red Hat Enterprise 5.4 on your primary boot device using the pvscsi driver, and includes steps about how to automate the process with Redhat’s excellent kickstart system. If you are just interested in doing this on one system, I would follow these steps instead . My steps are good if you want to automate this procedure over many machines, but it’s probably overkill if you just want to try pvscsi on one machine. Hopefully these steps will be outdated soon when the pvscsi module makes it into the regular mainline kernel, which looks like it might happen in 2.6.33!

1. Create a new VM from the vSphere Console in the usual way.

2. Click edit settings on the new VM and select the SCSI controller 0 device, then up on the right hand side pane, click Change SCSI Controller Type and select VMware Paravirtual.


Now we must create boot media. I use iso images and the virtual cdrom device in VMware, but you could modify these steps and use them with PXE booting or USB booting quite easily. The process here is to add the pvscsi kernel module to the boot disk, so the RHEL installer can see the VMware disk.

3. Grab the prebuilt boot.iso from the images/ subdirectory and unextract it. You can do this by running:

mount -o loop boot.iso /mnt/boot

cd /mnt/boot

mkdir ~/boot

cp -R * ~/boot

cd ~/boot

You should now have an isolinux subdirectory in this directory.

4. Extract initrd.img . This is the initial ram disk which contains the boot files that the installer requires to run. Create a temp directory and extract it into it.

mkdir tmp
cd tmp
gzip -dc ../initrd.img | cpio -ivd

5. Now we need to extract (again!) the modules archive, so we can add pvscsi to it.

cd modules
gzip -dc modules.cgz | cpio -ivd

This will make a new subdirectory which will have the same name as your kernel version. Mine looks like this :
If you are running a 32 bit kernel or a different version of Redhat, yours will look slightly different.

6. Extract the VMware tools rpm and copy the precompiled driver into the modules directory. Make another temp directory and copy the Vmware Tools rpm into it. You should be able to get the VMware tools rpm from your ESX host or from the VMware website.

rpm2cpio VMwareTools-4.0.0-latest.i386.rpm | cpio -idmv
find . -name pvscsi.o | grep 2.6.18-164.el5 (this will return the path to where the driver is - the one below is my path)
cp usr/lib/vmware-tools/modules/binary/bld-2.6.18-8.el5-x86_64smp-RHEL5/objects/pvscsi.o [temp dir from step 4]/modules/2.6.18-164.el5/x86_64/pvscsi.o
cp usr/lib/vmware-tools/modules/binary/bld-2.6.18-8.el5-x86_64smp-RHEL5/objects/pvscsi.o [temp dir from step 4]/modules/2.6.18-164.el5/x86_64/pvscsi.ko

Then delete the temporary VMware tools directory.

7. Now we need to edit a bunch of text files that are in the modules subdirectory so Redhat can find the new driver.

At the bottom of module-info add:

"VMware PVSCSI driver"

At the bottom of modules.alias add:

alias pci:v000015ADd000007C0sv*sd*bc*sc*i* pvscsi

At the bottom of modules.dep add:

pvscsi: scsi_mod

In pci.ids, search for ’15ad’ . You will see an entry for a few VMware devices. To this list add:

07c0 PVSCSI SCSI Controller

The whole stanza will look something like this:

15ad VMware
0405 SVGA II Adapter
0710 SVGA Adapter
07c0 PVSCSI SCSI Controller

8. Now we need to compress everything. First the modules.cgz (and still in the modules directory) :

find 2.6.18-164.el5 | cpio -H newc -ov | gzip -9 -c - > modules.cgz
rm -rf 2.6.18-164.el5

9. Compress the initrd.img

cd ..
find . | cpio -H newc -ov | gzip -9 -c - > initrd.img

10. Everything is compressed up again, now to make a boot cd.

cd ..
mkisofs -o pvscsi.iso -V pvscsi -b isolinux/isolinux.bin -c isolinux/ -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -T .

11. Copy pvscsi.iso to your ESX datastore and click edit settings on your paravirtualized guest vm. Set the cdrom to boot from pvscsi.iso .

This is enough to get the installation image to boot correctly with the pvscsi driver, but what will happen next is that the server will reboot using the regular kernel, and it won’t have the pvscsi driver available to it, so the machine will not reboot. We need to install and configure the VMware tools rpm after the RHEL install so when the machine reboots it will have access to the pvscsi driver file.

If you don’t want to use kickstart to do this, and the end of the install, drop into a shell by typing ALT-CTRL-F2 and download the VMware tools rpm from a local webserver, or you could copy it to the guest VM using scp or something. For example :

rpm -Uvh http://yourwebserver/vmwaretools.rpm --default

Reboot and you should be good. Read on if you would like to automate this procedure.

12.The easiest way to create a kickstart file is to run through a normal install and then to copy the ~/anaconda-ks.cfg file from the server, which will have all your settings saved in it. Or look at my example. . All you should need to do here is change the url, key, hostname, rootpw parameters and edit the url to the vmware tools rpm to something valid for your environment and you should be good to go.

Stay tuned for performance numbers to show if any of this is even worth it! 🙂

Posted in howto | Tagged , , , , | 13 Comments

Generating a java heap dump when WebSphere AppServer is not responding

This one is going to be esoteric.

Working in System Test, you tend to see problems crop up that can only be recreated after a certain amount of time. A memory leak, for example, might not appear to cause any problems until well into a test run. And it’s tough luck if you haven’t enabled the correct settings to obtain diagnostic information to debug the problem. You need to stop the server, set the required parameters (verbose gc, Xdump parameters, whatever) and rerun the test to reproduce the problem, which may take days.

Today at work a situation arose where a run had failed and the server was hung. You could send a kill -3 (a sigquit) at the process it would spit out a javacore file which is useful if you have some sort of thread contention or deadlocks. However the developer working on this problem wanted a heap dump, something which is difficult to generate if the server is not responding.

(Incidently, if the server is responding, you can use wsadmin to generate a heap dump, which is way easier than the method described below).

I ended up stumbling upon a neat way to generate a heap dump when the server is not responding.

1. Run gcore -o
Gcore is in the gdb package and forces the system to dump a binary core file. It should be about as large as the memory space that the java process was using when you ran it. Gcore is a Linux specific command; gdb proper should work on other unices, gcore is more of a convenience command. On Windows, use Dr Watson. The IBM support site has some excellent technotes around this sort of thing. I usually just google “Mustgather websphere hang” or something and they’ll often come up.

2. Run . /bin/ to put all these nifty java tools in your path.

3. Run jextract -nozip
This converts the system dump (core dump) into a useable format. It also produces an xml file that lists each active method and class taken at the time of the dump .

4. Run
jdmpview -core -xml .xml -J-Xmx

Where “big number” is a memory size bigger than the initial core file in bytes.
jdmpview gives you a command line menu of ways you can poke and prod the dump file. And it is from here where you can construct a java heap dump from a system dump. Simply type heapdump at this prompt and it will create a phd (portable heapdump format) file which you can use in one of the many analysis tools in IBM Support Assistant, or else send it on to your friendly neighbourhood support professional. 🙂 A word of warning. With a large system dump file the final ‘heapdump’ step seems to be quite slow. Still, it is better than waiting for days to reproduce the initial problem!

Posted in howto | 1 Comment

CNTR0019E: Initialization failed due to invalid property “supportedMemberTypes”.

Any problem where WMM (WebSphere Member Manager) doesn’t start properly is a deal breaker for Portal. Once WMM fails, every other component will fail, and you’ll be faced with a nasty 404 message.

After installing and on a machine here I got this error message, and a 404 when I tried to hit portal. These fixes update the version of WMM to the (current) latest version. Here’s what was in my SystemOut.log

[12/14/09 17:31:09:618 EST] 0000002a SystemOut O WMM Implementation Version: WMM5.6_PK83731 (April 1 2009)
[12/14/09 17:31:09:618 EST] 0000002a WSMM Message E init() Initialization failed due to invalid property “supportedMemberTypes”.
[12/14/09 17:31:09:653 EST] 0000002a WSMM Message E ejbCreate() Initialization failed due to invalid property “supportedMemberTypes”.
[12/14/09 17:31:09:655 EST] 0000002a ExceptionUtil E CNTR0019E: EJB threw an unexpected (non-declared) exception during invocation of method “getConfigurationData”. Exception data: ; nested exception is:
at Source)

I’m not sure what was updated in WMM for this one, but the solution is simple: install WAS to fix it.

Posted in tip, Uncategorized | Tagged , , , | Leave a comment

5 Steps to installing Portal 6.1.5 on Ubuntu 9.10

Another new version of Portal, another new version of Ubuntu . Here’s how to get them to play nicely together. Most of these steps are needed for any recent version of Ubuntu. Obligatory disclaimer, Ubuntu is a non supported platform, so don’t run anything important on this it. IBM support won’t help you!

1. Install libstdc++5 from here. Apparently this has been dropped from karmic (9.10) but you can use 9.04 packages instead and it will work.

2. Make sure you can ping yourself on the fully qualified hostname that you will be using in the install. This is a good general tip for installing Portal. If the machine doesn’t respond, add an alias in your /etc/hosts file, like this: localhost yourhostname

Also think about disabling any firewalls . The install process needs to be able to connect to certain ports running on your machine.
If you let the install run all the way through, and then re enable your firewall after, debugging and adding new firewall exceptions should be way easier.

3. Drop this file in /etc . It will make AppServer think that you are running RHEL
5, which is a supported OS.

4. Unlink /bin/sh and link it to /bin/bash, like this:

cd /bin
unlink sh
ln -s /bin/bash sh

Here’s a more detailed post about why you need to do this, if you are interested.

5. Don’t use sudo ./, but rather execute sudo su and then run ./ . The installer executes a task late in the install, and if you use sudo to execute, the installer will attempt to run the ConfigEngine task as your logged in user, not as root, and the task will fail.

I think you should also be able to run the install as a non root user, I just haven’t tried it yet, I will post back when I have.

These steps also work with 6101.

Posted in howto | Tagged , , , | 2 Comments

The security certificate “America Online Root Certification Authority 2” has a public key of length greater than 2048 bit.

I’m doing a bit of work for a client and they run this crazy Citrix Metaframe thing, which I’ve heard of but never used before. It’s like a remote access tool / website wrapped in java applets and special clients and all sorts of other whizbangerry.

But I had a problem connecting to it. It would load a java applet and then Java would die when initializing with the following error. I’m running the latest Sun Java 6 u16.

A local security certificate could not be loaded. (error code: 7)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at Source)
at Source)
at Source)
at Source)
at Source)
at Source)
Caused by: The SSL cryptography library failed. The security certificate "America Online Root Certification Authority 2" has a public key of length greater than 2048 bit.

It’s about identical to this unsolved post at Ubuntu launchpad too. The certificates for a simple client JRE are stored in the cacerts file which lives in jre/lib/security/cacerts . It looks as if the root certificate for AOL is too long or recently updated or something and it’s not playing nicely with MetaFrame. So somehow we have to ditch that root cert. It would only really be a problem if we are unlucky enough to have our certificate signed by that root CA.

I assumed that MetaFrame worked on older jres and that the problem is that I am using a brand new one. So luckily Sun keep an archive of old JDKs and JREs here. So it is quite simple, download an install an old JRE (I got 5u1) and rip the cacerts file out of there and dump it into your new JRE’s directory and try it again. Worked like a charm for me. You probably don’t want to do this permanently (I guess they updated the cacerts file for a reason?) but if you really need to log into MetaFrame, it’ll do.

Posted in random | Tagged , , , , | 1 Comment

Creating a Virtualbox guest machine from a physical machine

I’ve got a Lenovo T60 , which I run Notes and Sametime on. It’s a nice machine but a pain to have to carry home every day if I want to work at home.

Another guy in the office mentioned that he had converted his work laptop into a VM, so he could work at home on his nice quick desktop machine. I run Ubuntu 9.04 on the laptop and Windows 7 at home.

The T60 has a pretty hectic partitioning scheme, LVM, ext4, encrypted partitions, the whole works. This procedure handled it all. It makes a ‘bit for bit’ copy of the drive, and I knew it should work, but am happy it did anyway.

Here’s how to do it:

1. Boot up using a live cd. I used an Ubuntu 9.04 disk.
2. Grab a external drive bigger that the entire internal drive. The T60 hard drive is only 100GB, so this wasn’t hard.
3. Use fdisk -l to figure out which drive is which. My internal drive was /dev/sda and the external was /dev/sdb
4. Mount the external drive, if it isn’t mounted. Let’s say it’s mounted at /media/external
5. Run dd if=/dev/sda of=/media/external/drive.img bs=4096

If you don’t have an external drive, you could use netcat or scp or something, and send the drive image over the wire instead. It’ll just be much slower.

6. Reboot the laptop, you don’t need it anymore.
7. Copy drive.img to the host machine. I’m running Virtual Box 3.06 on Windows 7.
8. Run this command to convert the raw disk image to a Virtual Box drive image file.

C:\Program Files\Sun\VirtualBox>VBoxManage.exe convertfromraw \
c:\drive.img c:\users\\.VirtualBox\Machines\laptop\drive.vdi

9. Create a new Virtual Box Guest in the usual way, and point it to the vdi file. I used a SATA controller since my laptop has one, but not sure if this matters or not.
10. Boot up your new guest machine.

Hey presto, it’s the same! Now I don’t have to carry my laptop home anymore. It seems to run a bit slower than the native machine, but that’s to be expected.

This would probably work for a Windows physical machine too, although I think you’d want to add an extra step at the beginning. On the Windows laptop, set the IDE controller to use the Microsoft driver, rather than your default driver before you take the image. That way, when the guest OS boots up, it will have access to a ‘lowest common denominator’ disk driver.

Posted in howto | Tagged , , | 1 Comment

Oracle runstats oneliner (ok, two liner)

Use this to generate a runstats script for your Oracle system. Only works for 6.1 . Also very ugly 🙂

export PROFILE_PATH=/opt/WebSphere/wp_profile
cat ${PROFILE_PATH}/ConfigEngine/properties/ | grep DbUser | grep -v source | grep -v '#' | grep -v DBA | awk -F = '{print "execute dbms_stats.gather_schema_stats(ownname=>'"'"'" $2 "'\'', cascade=> TRUE);"}' > reorg.sql

Now copy this file to your oracle system and run this:

sqlplus / as sysdba @reorg.sql

Pow! All done.

Posted in Uncategorized | Tagged , , | Leave a comment