JVM crashes with Sun JVM 1.5.0_6

[kevinr]

If you are running Sun JVM 1.5.0_6 (Windows or Linux) you may see the Alfresco server crash with the following report:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x40520cd1, pid=31769, tid=2309221296 # # Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode) # Problematic frame:
# V  [libjvm.so+0x3c9cd1]
#

—————  T H R E A D  —————

Current thread (0x89602788):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=31780]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000000

Registers:
EAX=0x00000000, EBX=0x407027f0, ECX=0x89a3cec0, EDX=0x89dafc44 ESP=0x89a3ce34, EBP=0x89a3ce58, ESI=0x89db336c, EDI=0x89a3cec0 EIP=0x40520cd1, CR2=0x00000000, EFLAGS=0x00010292

Top of Stack: (sp=0x89a3ce34)
0x89a3ce34:   89a3d470 89a3d480 40520c16 8a8f8ca8
0x89a3ce44:   8a8f8cc8 89a3ce78 407027f0 89db336c
0x89a3ce54:   89db3328 89a3ce88 40520d75 89db3a60
0x89a3ce64:   89dafc44 89db3a60 89a3cec0 89db3ac4
0x89a3ce74:   89db4168 89db3328 407027f0 89db3a60
0x89a3ce84:   00000000 89a3cee8 40521258 89db3a60
0x89a3ce94:   89db336c 89db3a60 89a3cec0 ffffffff
0x89a3cea4:   89db4704 89a3cee8 405a5bf1 89db4168 

Instructions: (pc=0x40520cd1)
0x40520cc1:   10 eb de 31 c0 eb 8e 8b 42 04 83 ec 0c 8b 40 08
0x40520cd1:   8b 08 50 ff 51 28 83 c4 10 e9 48 ff ff ff 90 55 

Stack: [0x899be000,0x89a3f000),  sp=0x89a3ce34,  free space=507k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V  [libjvm.so+0x3c9cd1] V  [libjvm.so+0x3c9d75] V  [libjvm.so+0x3ca258] V  [libjvm.so+0x4511fd] V  [libjvm.so+0x450e90] V  [libjvm.so+0x1a11c0] V  [libjvm.so+0x19e432] V  [libjvm.so+0x1474b3] V  [libjvm.so+0x1a6929] V  [libjvm.so+0x1a6281] V  [libjvm.so+0x4c8366] V  [libjvm.so+0x4c2ba3] V  [libjvm.so+0x424338] C  [libpthread.so.0+0x5cb7]

[b]Current CompileTask:
opto:2497      org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object; (99 bytes)[/b]
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This has been identified as a JVM compiler (HotSpot) bug in that particular version of the JVM.

You can either:
1. Revert to Sun JVM 1.5.0_5 (which does not appear to have the problem)
2. Remove the method from JIT compliation:

This is done by creating a ".hotspot_compiler" file where java is called.
See the attached file which excludes the offending method.
For tomcat, this file would go in the TOMCAT_HOME/bin directory.
For use from Alfresco source, it would go in the project directory: e.g. \alfresco-src\root\projects\repository

To confirm this file is being picked up:
You should see the following as the JVM start up:

CompilerOracle: exclude org/apache/lucene/index/IndexReader$1 doBody‍

When an attempt to compile this method is made, it will be rejected and you will see:

### Excluding compile:  org.apache.lucene.index.IndexReader$1::doBody

If you want to see all compilation progress you can use  XX:+PrintCompilation

The file should contain the following line:

exclude  org/apache/lucene/index/IndexReader$1 doBody‍

It is possible the JVMs differ in how they find this file:

http://sunsolve.sun.com/searchproxy/document.do?assetkey=1-9-73893-1

[kevinr]

Things have changed since then…!

[antoine]

Hi,
I tried this an had the same error with 1.5.0_09 on a fresh CentOS 4.4 install yesterday. There were also filesystem problems (old scsi drives) which have since completely hosed the entire installation, but I definitely got this with CentOS 4.4, sun java 1.5.0_09 and the trial enterprise jboss 1.3.1 - all downloaded and installed yesterday…
Cheers
Antoine

[prototribe]

Antione,

What i've found is that for highly loaded environments its a combination of java, and Redhat's thread safe libraries (tls) that seem to be causing the crash. What we've done is disable tls. It forces java to spawn new processes to handle requests, but avoids the complexity of individual threads in a single process. Which is also known to cause problems with Xen server virtualization. We can now do file operations (like copy 5 GB across machines with no issues).

This does in theory slow down the system a bit, but in our estimation is worth it for the increased stability.

It also may be possible to use a fixed version of tls (see the Xen mailinglist for information about that), but so far all we've done is disable tls.

mv /lib/tls /lib/tls.disabled
mv /usr/lib/tls /usr/lib/tls.disabled

reboot