Generating a java heap dump when WebSphere AppServer is not responding

This one is going to be esoteric.

Working in System Test, you tend to see problems crop up that can only be recreated after a certain amount of time. A memory leak, for example, might not appear to cause any problems until well into a test run. And it’s tough luck if you haven’t enabled the correct settings to obtain diagnostic information to debug the problem. You need to stop the server, set the required parameters (verbose gc, Xdump parameters, whatever) and rerun the test to reproduce the problem, which may take days.

Today at work a situation arose where a run had failed and the server was hung. You could send a kill -3 (a sigquit) at the process it would spit out a javacore file which is useful if you have some sort of thread contention or deadlocks. However the developer working on this problem wanted a heap dump, something which is difficult to generate if the server is not responding.

(Incidently, if the server is responding, you can use wsadmin to generate a heap dump, which is way easier than the method described below).

I ended up stumbling upon a neat way to generate a heap dump when the server is not responding.

1. Run

gcore -o <corefile> <pid>

Gcore is in the gdb package and forces the system to dump a binary core file. It should be about as large as the memory space that the java process was using when you ran it. Gcore is a Linux specific command; gdb proper should work on other unices, gcore is more of a convenience command. On Windows, use Dr Watson. The IBM support site has some excellent technotes around this sort of thing. I usually just google “Mustgather websphere hang” or something and they’ll often come up.

2. Run

. <wp_profile>/bin/setupCmdLine.sh

to put all these nifty java tools in your path.

3. Run

 jextract <coredumpname> -nozip

This converts the system dump (core dump) into a useable format. It also produces an xml file that lists each active method and class taken at the time of the dump .

4. Run

jdmpview -core <corefilename> -xml <corefilename>.xml -J-Xmx<big number>

Where “big number” is a memory size bigger than the initial core file in bytes.
jdmpview gives you a command line menu of ways you can poke and prod the dump file. And it is from here where you can construct a java heap dump from a system dump. Simply type heapdump at this prompt and it will create a phd (portable heapdump format) file which you can use in one of the many analysis tools in IBM Support Assistant, or else send it on to your friendly neighbourhood support professional. 🙂 A word of warning. With a large system dump file the final ‘heapdump’ step seems to be quite slow. Still, it is better than waiting for days to reproduce the initial problem!

This entry was posted in howto. Bookmark the permalink.

One Response to Generating a java heap dump when WebSphere AppServer is not responding

Leave a Reply

Your email address will not be published. Required fields are marked *