Offline Physical 2 Virtual conversion (p2v) using free tools

I’m a bit confused about VMware’s Converter and when you can use it. Which offering is free? Which is paid? From the last time I looked, I think you need the enterprise product to do an offline conversion. And call me old school, but online conversion just seems like voodoo to me. I was in the office this weekend, trying to convert an important physical machine to a virtual one and it failed for all sorts of reasons:

1. SSH keys. If you have a heavily locked down machine with ssh key logins, you cannot use the online converter.

2. Network. If you can’t hit the ’source’ machine with ssh, you can’t convert it. Maybe it’s on another network, or has a firewall running on it.

3. So since all that didn’t work, you’re forced to install the converter on the same machine you are converting. Except you don’t want to do that because you will have to fill out lots of forms because you are installing a new piece of software on a production machine.

This is a fail safe method of converting a physical Linux server into a virtual one. I converted my physical machine to an ESX 4.0 host, but this method would probably work for other versions of VMware as well.

1. Power down the server.

2. Boot up the server using a linux live cd . (I like Slax)

3. At the command line, run:

Where server is a (linux) server with about twice as much disk space as you have, and /dev/sda is the address of the disk you would like to convert. You could probably do more that one disk using this howto, but I only did one.

4. Install Qemu on the Linux server. You should be able to find it in your yum repo, or apt-get repo.

5. Run this to convert the raw dd image to a vmdk.

6. Copy the sda.vmdk to your datastore, using scp if you have ssh enabled on your VMware host, or using the Datastore browser if you don’t. Make sure to create a folder for the new machine.

7. Create a new VM in the vSphere Console, using values similar to the physical machine you are replacing. Make sure to select Custom at the first screen, not typical.

8. When it comes time to create a new disk, point vSphere at your newly converted vmdk file.

9. Boot up the server, and you’re done!

Installing RHEL 5 using the VMware Paravirtualized SCSI driver (pvscsi)

VMware ESX 4.0 offers a number of different SCSI disk controllers. One of the newer and more interesting ones is the paravirtualized scsi adapter. As with any type of paravirtualized  driver the paravirtualized scsi driver promises to be faster and use less CPU cycles than the other storage drivers, and since I/O performance is so important to overall system performance in virtualized environments, I thought it was worth a look. I don’t have any solid numbers about how it performs with WebSphere Portal, but once I do you will be the first to know.

The new driver has a number of drawbacks, most notably it is not supported as a boot device. VMware expect you to configure your OS on one ‘normal’ drive and then install a secondary drive with the paravirtualized adaptor. As well as being a pain from a management standpoint , it also ignores the speed increases that you would derive from having the OS on this faster device.

The method below will allow you to install Red Hat Enterprise 5.4 on your primary boot device using the pvscsi driver, and includes steps about how to automate the process with Redhat’s excellent kickstart system. If you are just interested in doing this on one system, I would follow these steps instead . My steps are good if you want to automate this procedure over many machines, but it’s probably overkill if you just want to try pvscsi on one machine. Hopefully these steps will be outdated soon when the pvscsi module makes it into the regular mainline kernel, which looks like it might happen in 2.6.33!

1. Create a new VM from the vSphere Console in the usual way.

2. Click edit settings on the new VM and select the SCSI controller 0 device, then up on the right hand side pane, click Change SCSI Controller Type and select VMware Paravirtual.

Now we must create boot media. I use iso images and the virtual cdrom device in VMware, but you could modify these steps and use them with PXE booting or USB booting quite easily. The process here is to add the pvscsi kernel module to the boot disk, so the RHEL installer can see the VMware disk.

3. Grab the prebuilt boot.iso from the images/ subdirectory and unextract it. You can do this by running:

You should now have an isolinux subdirectory in this directory.

4. Extract initrd.img . This is the initial ram disk which contains the boot files that the installer requires to run. Create a temp directory and extract it into it.

5. Now we need to extract (again!) the modules archive, so we can add pvscsi to it.

This will make a new subdirectory which will have the same name as your kernel version. Mine looks like this :
2.6.18-164.el5/x86_64
If you are running a 32 bit kernel or a different version of Redhat, yours will look slightly different.

6. Extract the VMware tools rpm and copy the precompiled driver into the modules directory. Make another temp directory and copy the Vmware Tools rpm into it. You should be able to get the VMware tools rpm from your ESX host or from the VMware website.

Then delete the temporary VMware tools directory.

7. Now we need to edit a bunch of text files that are in the modules subdirectory so Redhat can find the new driver.

At the bottom of module-info add:

At the bottom of modules.alias add:

At the bottom of modules.dep add:

In pci.ids, search for ‘15ad’ . You will see an entry for a few VMware devices. To this list add:

The whole stanza will look something like this:

8. Now we need to compress everything. First the modules.cgz (and still in the modules directory) :

9. Compress the initrd.img

10. Everything is compressed up again, now to make a boot cd.

11. Copy pvscsi.iso to your ESX datastore and click edit settings on your paravirtualized guest vm. Set the cdrom to boot from pvscsi.iso .

This is enough to get the installation image to boot correctly with the pvscsi driver, but what will happen next is that the server will reboot using the regular kernel, and it won’t have the pvscsi driver available to it, so the machine will not reboot. We need to install and configure the VMware tools rpm after the RHEL install so when the machine reboots it will have access to the pvscsi driver file.

If you don’t want to use kickstart to do this, and the end of the install, drop into a shell by typing ALT-CTRL-F2 and download the VMware tools rpm from a local webserver, or you could copy it to the guest VM using scp or something. For example :

Reboot and you should be good. Read on if you would like to automate this procedure.

12.The easiest way to create a kickstart file is to run through a normal install and then to copy the ~/anaconda-ks.cfg file from the server, which will have all your settings saved in it. Or look at my example. . All you should need to do here is change the url, key, hostname, rootpw parameters and edit the url to the vmware tools rpm to something valid for your environment and you should be good to go.

Stay tuned for performance numbers to show if any of this is even worth it!

Generating a java heap dump when WebSphere AppServer is not responding

This one is going to be esoteric.

Working in System Test, you tend to see problems crop up that can only be recreated after a certain amount of time. A memory leak, for example, might not appear to cause any problems until well into a test run. And it’s tough luck if you haven’t enabled the correct settings to obtain diagnostic information to debug the problem. You need to stop the server, set the required parameters (verbose gc, Xdump parameters, whatever) and rerun the test to reproduce the problem, which may take days.

Today at work a situation arose where a run had failed and the server was hung. You could send a kill -3 (a sigquit) at the process it would spit out a javacore file which is useful if you have some sort of thread contention or deadlocks. However the developer working on this problem wanted a heap dump, something which is difficult to generate if the server is not responding.

(Incidently, if the server is responding, you can use wsadmin to generate a heap dump, which is way easier than the method described below).

I ended up stumbling upon a neat way to generate a heap dump when the server is not responding.

1. Run

Gcore is in the gdb package and forces the system to dump a binary core file. It should be about as large as the memory space that the java process was using when you ran it. Gcore is a Linux specific command; gdb proper should work on other unices, gcore is more of a convenience command. On Windows, use Dr Watson. The IBM support site has some excellent technotes around this sort of thing. I usually just google “Mustgather websphere hang” or something and they’ll often come up.

2. Run

to put all these nifty java tools in your path.

3. Run

This converts the system dump (core dump) into a useable format. It also produces an xml file that lists each active method and class taken at the time of the dump .

4. Run

Where “big number” is a memory size bigger than the initial core file in bytes.
jdmpview gives you a command line menu of ways you can poke and prod the dump file. And it is from here where you can construct a java heap dump from a system dump. Simply type heapdump at this prompt and it will create a phd (portable heapdump format) file which you can use in one of the many analysis tools in IBM Support Assistant, or else send it on to your friendly neighbourhood support professional. A word of warning. With a large system dump file the final ‘heapdump’ step seems to be quite slow. Still, it is better than waiting for days to reproduce the initial problem!

5 Steps to installing Portal 6.1.5 on Ubuntu 9.10

Another new version of Portal, another new version of Ubuntu . Here’s how to get them to play nicely together. Most of these steps are needed for any recent version of Ubuntu. Obligatory disclaimer, Ubuntu is a non supported platform, so don’t run anything important on this it. IBM support won’t help you!

1. Install libstdc++5 from here. Apparently this has been dropped from karmic (9.10) but you can use 9.04 packages instead and it will work.

2. Make sure you can ping yourself on the fully qualified hostname that you will be using in the install. This is a good general tip for installing Portal. If the machine doesn’t respond, add an alias in your /etc/hosts file, like this:

Also think about disabling any firewalls . The install process needs to be able to connect to certain ports running on your machine.
If you let the install run all the way through, and then re enable your firewall after, debugging and adding new firewall exceptions should be way easier.

3. Drop this file in /etc . It will make AppServer think that you are running RHEL
5, which is a supported OS.

4. Unlink /bin/sh and link it to /bin/bash, like this:

Here’s a more detailed post about why you need to do this, if you are interested.

5. Don’t use sudo ./install.sh, but rather execute sudo su and then run ./install.sh . The installer executes a ConfigEngine.sh task late in the install, and if you use sudo to execute install.sh, the installer will attempt to run the ConfigEngine task as your logged in user, not as root, and the task will fail.

I think you should also be able to run the install as a non root user, I just haven’t tried it yet, I will post back when I have.

These steps also work with 6101.

Creating a Virtualbox guest machine from a physical machine

I’ve got a Lenovo T60 , which I run Notes and Sametime on. It’s a nice machine but a pain to have to carry home every day if I want to work at home.

Another guy in the office mentioned that he had converted his work laptop into a VM, so he could work at home on his nice quick desktop machine. I run Ubuntu 9.04 on the laptop and Windows 7 at home.

The T60 has a pretty hectic partitioning scheme, LVM, ext4, encrypted partitions, the whole works. This procedure handled it all. It makes a ‘bit for bit’ copy of the drive, and I knew it should work, but am happy it did anyway.

Here’s how to do it:

1. Boot up using a live cd. I used an Ubuntu 9.04 disk.
2. Grab a external drive bigger that the entire internal drive. The T60 hard drive is only 100GB, so this wasn’t hard.
3. Use fdisk -l to figure out which drive is which. My internal drive was /dev/sda and the external was /dev/sdb
4. Mount the external drive, if it isn’t mounted. Let’s say it’s mounted at /media/external
5. Run

If you don’t have an external drive, you could use netcat or scp or something, and send the drive image over the wire instead. It’ll just be much slower.

6. Reboot the laptop, you don’t need it anymore.
7. Copy drive.img to the host machine. I’m running Virtual Box 3.06 on Windows 7.
8. Run this command to convert the raw disk image to a Virtual Box drive image file.

9. Create a new Virtual Box Guest in the usual way, and point it to the vdi file. I used a SATA controller since my laptop has one, but not sure if this matters or not.
10. Boot up your new guest machine.

Hey presto, it’s the same! Now I don’t have to carry my laptop home anymore. It seems to run a bit slower than the native machine, but that’s to be expected.

This would probably work for a Windows physical machine too, although I think you’d want to add an extra step at the beginning. On the Windows laptop, set the IDE controller to use the Microsoft driver, rather than your default driver before you take the image. That way, when the guest OS boots up, it will have access to a ‘lowest common denominator’ disk driver.

How to get your own local copy of the Infocenter

If I’m going out to meet a customer, I’ll always make sure to have a local copy of the infocenter on hand in case I have to look something up.

Luckily enough, there’s an easy way to do this. The infocenters for WebSphere Application Server and WebSphere Portal are available in Eclipse Help System form, which is pretty much the same way they are presented over the web.

You can download the infocenter in Eclipse Help System format from this page for WAS and this page for Portal.

If you don’t have Eclipse already, download it from here. Any one of the versions should do. Unzip Eclipse and then copy the infocenter zips into the eclipse/plugins directory. Then unzip the infocenter zips in the plugins directory.

Now start Eclipse, and pick any old workspace. Click Help -> Help Contents, and the Infocenter zips that you copied in should be along side any other help files that come with Eclipse originally.

Now you have a local copy, make sure to update it since the infocenter is updated monthly.

How to solve CWWIM4520E on Portal 6.1

You may have seen this error if you tried the steps in “Configuring WCM email actions with a local SMTP server”.

When you edit the user’s properties this nasty error can appear if your Portal server is connected to an LDAP.

Error entering mail address into Self Care Portlet

Btw, this is a 6.1 or Portal.Next beta specific error, it should work fine on 6.0.

Here’s the full text of the error:

The reason this happens is that the portlet ( the self care portlet in this case)  is wired up to write the email address you entered in the form to a VMM attribute called ibm-primaryEmail . If your ldap schema doesn’t have a user attribute in it called ibm-primaryEmail , then you’re going to get an error when you try and write something to it.

Just to check it out, let’s look at the LDAP schema on this server (which is IBM Tivoli Directory Server 6.0)  . I’m using the awesome and free Apache Directory Studio to investigate the LDAP schema here.  Once the connection to the ldap is defined, go LDAP -> Open Schema Browser , and select the tab attribute types.

TDS ldap schema

Ok, so we have an attribute type ‘drink, favouriteDrink’ ;o) , but no ibm-primaryEmail . No matter, there is a ‘mail’ attribute there. We can make Portal use that to save email related attributes.

Open up wkplc.properties and find the section entitled LDAP Attribute Configuration (it’s near the bottom) . Here’s my completed one:

Cool, now run the task :

If you are using a federated ldap setup, edit the corresponding federated properties instead, and then run the following task:

. Restart the server and try the form again. It should correctly save the email attribute for the user and you can get on with sending email through Portal. Just for kicks, lets look at what that task did. It just edits the wimconfig file, which defines how VMM interacts . Open wimconfig.xml (wp_profile/config/cells/<cellname>/wim/config/wimconfig.xml) and search for ibm-primaryEmail.

Here is the part that does the mapping:

So the task is really just a (welcome) convenience, all it does it edit the xml file for you. Anyone who has tried to set up multirealms on 6.0 would be grateful for that!

ERRORCODE=-4214, SQLSTATE=28000 from DB2 on Ubuntu

Two posts in one day, wow. It’s all part of our special series: how to install and configure WebSphere Portal 6.1 on Ubuntu. This isn’t a Portal only issue, rather it’s a DB2+Ubuntu issue.

After getting Portal installed on this Ubuntu machine, you’re probably going to want to transfer the default Derby database to something more robust like DB2. So you edit wkplc_comp.properties and wkplc_dbtype.properties, and start to run:

And you get this in the ConfigTrace.log

Hmm, ok, I thought db2 was working. A good habit when debugging these things is to take the piece that ConfigEngine is trying run and run it independently. So right now I want ConfigEngine to create an empty db2 database that I can run database-transfer against. Try this:

And that comes back successfully. However, that command sequence is not an accurate representation of what ConfigEngine is actually doing. We’re running ConfigEngine as root. But the ConfigEngine script is using the “user db2inst1 using ” modifiers on the end of the database create command. So how about this?

Ah ha, a failure. In the first example, DB2 already trusts the user that we’re logged is as (db2inst1), so it doesn’t need to go back to the operating system and authenticate it. In the second example, we are logged in as root, so db2 needs to go to the operating system and authenticate the user. Ubuntu uses the tried and true passwd + shadow file combo to store usernames and their associated passwords. The trouble is since Ubuntu 8.10, it uses the newer and more secure SHA512 hashing function to store the passwords, and DB2 doesn’t understand SHA512. So the workaround is to change the hashing function in use on the machine, reset the password and then we should be able to use the “user db2inst1 using ” type commands again.

Open /etc/pam.d/common-password in a text editor and change this line:

to

Then run passwd db2inst1 and put the same or a new password. If you look at the shadow file , the hash will change from something like this:
SHA512

MD5

(The unencrypted password here is ‘password’ – don’t get too excited!) .

Now try running ConfigEngine.sh create-database again. It should work. Make sure you change the system /etc/pam.d/common-password back to sha512, as you want the rest of your users to use this hashing function as it is more secure than md5sum . Hopefully DB2 should address this in a fixpack.

Getting WebSphere Portal to install on Ubuntu

Ok, ok, I know Ubuntu isn’t supported by IBM, but if you’re a developer, and you run Ubuntu on your laptop and would like to run Portal on there too, here’s how you do it. This should work for Debian too . I guess this isn’t really a Portal only problem, rather it’s a general Application Server + Ubuntu issue. I’ve only tried this with 6.1, but it is probably an issue with 6.0 as well, since it uses similar profile creation code.

For some reason Ubuntu, ships a shell called ‘dash’ as their default, rather than good old bash. (Well, they have their reasons.) This presents a problem for App Server, as all the scripts in AppServer/bin use #!/bin/sh (the Bourne shell) as their command interpreter, which on a Redhat machine is symlinked to /bin/bash . The reason why AppServer uses the Bourne shell is that it needs to work on AIX and Solaris too, and you can’t be sure that you have bash on one of those machines.

If you try to call wsadmin.sh on an Ubuntu system you get this error :

Now, when you’re installing Portal, it installs AppServer, and then calls the manageprofile.sh script to create the default profile for Portal “wp_profile” . Manageprofile.sh calls a bunch of ant scripts that needs to use wsadmin.sh (you can see where I’m going here….) . You might notice in AppServer/log/wp_profile_create.log messages like this :

It actually hangs for 20 minutes waiting for wsadmin to start!

So how do you fix it? Simply, before you install it, unlink /bin/sh -> /bin/dash , and relink so it reads /bin/sh -> /bin/bash

Carefully:

Now the install should run properly.

Configuring WCM email actions with a local SMTP server

Wow, here goes, a post about WCM!

We often need to set up simple email system to test WCM’s email action. This feature is used in workflow. The system can be configured to email a user when they need to approve content to move to the next workflow stage, for example.

I like to set up a local smtp/pop3 server on the same machine that Portal is running on, then we don’t have to worry about DNS or MX records or anything nasty like that and can focus on the WCM features. This could be useful for you if you need to run a demo or something all on the same machine.

I did these steps on a Redhat 5 machine. My server’s fully qualified hostname is narrabeen.ilwwcm.au.ibm.com , for those following along at home.

First install postfix and dovecot . They are simple and secure and do just what we want. If you are familiar with another MTA, like sendmail, just use that instead.

or for the non yum users out there

Note that you should remove sendmail after you install postfix otherwise rpm will complain about dependency issues.

Dovecot works completely out of the box, just run :

And test it from your machine by trying to telnet to port 110 on the server.

Cool, lets make some local users that we can send mail to.

This creates local mailboxes on the server that we will address later as : <user>@<fullyqualifieddomainname> . In my case these addresses are user1@narrabeen.ilwwcm.au.ibm.com and adminuser@narrabeen.ilwwcm.au.ibm.com . Please note that WCM will not forward to user@localhost type addresses, so you will need to address your mail to a ‘proper’ domain.

Now the WCM part. Open your WCMConfigService.properties file. I am using 6.1.0.1 at the moment, but this is valid for any version of WCM. The config file named differently in 5.1 (connect.cfg), and located somewhere else (under Portal Home instead of in the profile) in 6.0, but the properties are quite similar. On 6.1 this is located in [PROFILE_PATH]/PortalServer/wcm/shared/app/config/wcmservices/WCMConfigService.properties
We are concerned with this paragraph:

Change mail.yourserver.com to your mail server , and the default addresses to adminuser@yourmailserver.com . The defaultfromaddress will be the user that WCM will send the mail as. You should make this a real address so if the user replies to the email, it won’t go bouncing around your email system for days. Restart Portal to make those changes take effect .

Now we need to link users in Portal to the email addresses we have created. Load up Portal, log in as your admin user and go to Administration -> Users and Groups .

Create a new user and enter the email address.

Entering Admin email address

Edit your admin user too, and assign it the email address admin@localhost, which you created earlier.

Create a new library and assign the user manager access to the library. Go to the authoring portlet, click configure and select the library you just created. Logout and log in as the new user.

Now you’ll have to create all the basic stuff before you can create content. Start with the email action and enable all the options, so it will send an email to every account involved in the process. Now add the email action to the ‘entry’ section of the first workflow stage and add the workflow stage to a workflow. Create a presentation template, authoring template, site, site area and finally a piece of content. It should be in the draft stage now. Since the email action is in the entry section, it will be fired every time we create new content, which is good for testing the email action.

Email action on the first workflow stage

Configure up your POP3 client of choice and point it to your Portal/email server using the admin username and password (since the admin user will need to approve the content). I am using the excellent Thunderbird for this purpose. If everything has gone well, we should see an email sitting in Thunderbird.

Ding, new content to be approved!

As you can see, the email that appears is from wcmadmin@narrabeen.ilwwcm.au.ibm.com , which is the value I used for “defaultfromaddress” in the WCMConfigService.properties file.

What if it doesn’t work ? Email is like a series of tubes (with apologies to Ted Stevens) . You need to check that the email is being generated by WCM, and then if WCM is sending it to the right place, and *then* if postfix is sending it to dovecot. SystemOut doesn’t log emails that are sent out so it can be difficult to debug. This trace string is your friend if you’re not sure where to start. It should log WCM’s attempt to send the mail to postfix.

To use the trace string, go to Administration -> Portal Analysis -> Enable Tracing and copy the trace string into the box. A trace.log file should appear in [PROFILE_ROOT]/logs/WebSphere_Portal/

If WCM is sending the email properly, you will see something like this in the trace.log :

If the mail is still not turning up in the mailbox, we need to turn our attention to /var/log/maillog . By default both dovecot and postfix will log to this file.

I had this problem when trying to set this up. Postfix does a number of checks to figure out whether to relay the mail to another mail server, or to keep it on the local server and route it to a local user. We want it to do the latter.
We need to change the value mydestination so it matches the mail server address we are using (everything to the right of the @ symbol in the email address) . Additionally we should change mynetworks_style from subnet (the default) to host, as we don’t want postfix to accept mail from any other host in our subnet or not.
If your postfix server is on a different machine to portal, you’d obviously skip this step.

To make these changes take effect run:

Now create a new piece of content. Tailing /var/log/maillog helps to make sure the mail is correctly hitting postfix.

Jackpot!

Another useful command to use when debugging postfix is postconf -n , which will print all the parameters that are currently in effect for the server.

Well, hope that made some sense, send me a message and let me know how you went.