topic Re: Access Share website locks cpu at 100% in Alfresco Archive

Access Share website locks cpu at 100%

brazen — Tue, 24 Aug 2010 13:49:55 GMT

When I access the share website, I can browse around for a bit, but if I try to view a document, then then every Alfresco server locks up and top show java maxing out the cpu at 100%. I left this overnight and the cpu never did go down. I have to restart tomcat to make it go away and to be able to

Re: Access Share website locks cpu at 100%

dranakan — Tue, 24 Aug 2010 14:15:28 GMT

Hello,

You can use HotTread to see what Alfresco (java) is doing and post here the results…

HotTread : http://weblogs.java.net/blog/brucechapman/archive/2008/03/hot_threads.html)
More help to execute the command : http://forums.alfresco.com/fr/viewtopic.php?f=8&t=3786#p17060

Good luck

Re: Access Share website locks cpu at 100%

kevinr — Tue, 24 Aug 2010 14:55:54 GMT

Sounds like a problem with the setup of one of the 3rd party progs that are used for document conversion i.e. PDF2SWF or similar, Alfresco doesn't tend to do that otherwise!! Viewing a massive pdf/doc or similar for the first time could cause that…?

CTRL+Break (on Windows - don't know the equivalent on linux…) to get a thread-dump and see what is running - post back what you find…

Re: Access Share website locks cpu at 100%

brazen — Tue, 24 Aug 2010 17:35:34 GMT

I'll check out HotTread.

This was not viewing a massive document. I was trying to view a text document that I had just created with one line of text.

I happened to notice this in the alfresco-global.properties file:

#share.context=share

#share.host=${localname}

#share.port=8080

#share.protocol=http‍‍‍‍‍‍‍

I had not uncommented these, and I'm guessing I should? Is there any other configuration that I would have to do to use Alfresco Share?

Re: Access Share website locks cpu at 100%

kevinr — Tue, 24 Aug 2010 18:48:41 GMT

OK. If those settings were wrong you would not be able to get past the login page, so I don't think its that. I still think it could be the Flash previewer (which calls the repository to generate a thumbnail of the doc via PDFBox->PDF2SWF so one of those libraries/external programs could be locking up for some reason.

Kev

Re: Access Share website locks cpu at 100%

brazen — Tue, 24 Aug 2010 20:42:36 GMT

I'm trying to run hotthread, but all I get is this:

brazen@alfrescoserver:/$ sudo java -classpath /usr/lib/jvm/java-6-sun-1.6.0.20/lib/tools.jar:HotThread.jar hotthread.Main 10856
Exception in thread "main" java.io.IOException: well-known file is not secure
        at sun.tools.attach.LinuxVirtualMachine.checkPermissions(Native Method)
        at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:93)
        at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:46)
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:195)
        at hotthread.Main.main(Main.java:52)‍‍‍‍‍‍‍

10856 is the pid of the java process from the output of top.

Re: Access Share website locks cpu at 100%

dranakan — Wed, 25 Aug 2010 06:06:52 GMT

Use the user running java (Alfresco)

Sample of use :

[alfresco@localhost Alfresco3]$ ps aux | grep java

alfresco  3410 79.6 53.4 1516052 1247352 ?     Sl   Aug26 1985:22 /usr/bin/java -Xms512m -Xmx1024m -XX:MaxPermSize=256m -server -Duser.language=fr -Duser.region=FR -XX:CompileCommand=exclude,org/apache/lucene/index/IndexReader$1,doBody -XX:CompileCommand=exclude,org/alfresco/repo/search/impl/lucene/index/IndexInfo$Merger,mergeIndexes -XX:CompileCommand=exclude,org/alfresco/repo/search/impl/lucene/index/IndexInfo$Merger,mergeDeletions -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/Alfresco3/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/Alfresco3/tomcat/endorsed -classpath :/opt/Alfresco3/tomcat/bin/bootstrap.jar -Dcatalina.base=/opt/Alfresco3/tomcat -Dcatalina.home=/opt/Alfresco3/tomcat -Djava.io.tmpdir=/opt/Alfresco3/tomcat/temp org.apache.catalina.startup.Bootstrap start
alfresco 13890  0.0  0.0   3912   672 pts/1    S+   08:43   0:00 grep java

[alfresco@localhost Alfresco3]$ /usr/java/jdk1.6.0_13/bin/jstack 3410 > jstack_alfresco.log‍‍‍‍‍‍

Re: Access Share website locks cpu at 100%

shmee — Wed, 25 Aug 2010 21:02:03 GMT

I have the exact same problem. I also run Ubuntu Server 10.04, and Tomcat 6, but for me it just happens when I go to the Document Library. It says it is looking for documents and then it just maxes out my server.

I guess there could be a problem with SWFTools or something, but my KTree install (Yes I know it is PHP vs Java/Tomcat) which uses SWFTools for document preview works just fine.

UPDATE:

I found out my problem!!! I needed to have OpenOffice running as a Server, and in order to do that I had run the command:

nohup soffice -invisible -accept='socket,host=localhost,port=8100;urp' -display :0 &‍

Re: Access Share website locks cpu at 100%

kevinr — Thu, 26 Aug 2010 08:50:57 GMT

Excellent thanks for posting that - does that fix the issue for anyone else?

Re: Access Share website locks cpu at 100%

brazen — Fri, 27 Aug 2010 17:23:24 GMT

Use the user running java (Alfresco)

Ok, here is the process info:

brazen@alfrescoserver:~$ ps aux | grep java
tomcat6   3535  7.0 31.2 2700692 971792 pts/1  Sl   11:24   1:47 /usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128M -XX:+UseConcMarkSweepGC -server -Xss1024K -Xms1G -Xmx2G -XX:MaxPermSize=128M -XX:NewSize=512m -XX:CMSInitiatingOccupancyFraction=80 -Djava.net.preferIPv4Stack=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start‍‍

I figured out that I had to run it as tomcat6 AND I had to copy the HotThread.jar to a place that the tomcat6 user could read it (I was trying to run it in my own home directory). So I copied it to /tmp and switched to the /tmp directory:

sudo -u tomcat6 java -classpath /usr/lib/jvm/java-6-sun/lib/tools.jar:HotThread.jar hotthread.Main 3535‍

I ran it after a fresh boot and it did give sensible output:

31.0% CPU Usage by Thread 'RMI TCP Connection(53)-127.0.1.1'
  10/10 snapshots sharing following 33 elements
    sun.management.ThreadImpl.getThreadInfo0(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:145)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
    com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
    com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
    com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
    com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
    javax.management.StandardMBean.invoke(StandardMBean.java:391)
    com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
    com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
    javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
    javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
    javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
    javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
    javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
    sun.reflect.GeneratedMethodAccessor399.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    sun.rmi.transport.Transport$1.run(Transport.java:159)
    java.security.AccessController.doPrivileged(Native Method)
    sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:619)

 0.0% CPU Usage by Thread 'main'
  10/10 snapshots sharing following 13 elements
    java.net.PlainSocketImpl.socketAccept(Native Method)
    java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
    java.net.ServerSocket.implAccept(ServerSocket.java:453)
    java.net.ServerSocket.accept(ServerSocket.java:421)
    org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
    org.apache.catalina.startup.Catalina.await(Catalina.java:657)
    org.apache.catalina.startup.Catalina.start(Catalina.java:617)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

 0.0% CPU Usage by Thread 'Reference Handler'
  10/10 snapshots sharing following 3 elements
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This time though, the site locked up and pegged out the cpu as soon as I tried logging in to Alfresco Share. I tried running HotThread again, but this time HotThread doesn't seem to run correctly:

Exception in thread "main" java.rmi.UnmarshalException: error unmarshalling return; nested exception is: 
        java.io.EOFException
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
        at hotthread.Main.main(Main.java:64)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2554)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1297)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155)
        … 6 more‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So I tried the jstack command you mention and got this rediculous amount of output:

2010-08-27 12:18:04
Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):

"JMX server connection timeout 201" daemon prio=10 tid=0x0000000041ee3800 nid=0x22df in Object.wait() [0x00007f69a6fee000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
   - locked <0x00007f69dd6307f0> (a [I)
   at java.lang.Thread.run(Thread.java:619)

"http-8080-9" daemon prio=10 tid=0x00000000424ba000 nid=0x67a2 in Object.wait() [0x00007f69a6068000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00007f69dbc667e0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:619)

"http-8080-8" daemon prio=10 tid=0x0000000042779000 nid=0x67a1 in Object.wait() [0x00007f699e5e4000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00007f69dbc665d0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:619)

"http-8080-5" daemon prio=10 tid=0x0000000041e77000 nid=0x667e in Object.wait() [0x00007f699dad3000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00007f69dbc65cb8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:619)

"http-8080-3" daemon prio=10 tid=0x000000004275f000 nid=0x667c in Object.wait() [0x00007f699dcd5000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00007f69dbc65988> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:619)

"http-8080-2" daemon prio=10 tid=0x0000000041da0800 nid=0x667b in Object.wait() [0x00007f699ddd6000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00007f69dbc65778> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x0000000042699800 nid=0x635c runnable [0x00007f69a6ceb000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69dbc54ae8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"Attach Listener" daemon prio=10 tid=0x0000000042320000 nid=0x635b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"http-8080-Acceptor-0" daemon prio=10 tid=0x00007f69a0271000 nid=0x1520 runnable [0x00007f69ac3de000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d34a6330> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
   at java.lang.Thread.run(Thread.java:619)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f69a1329000 nid=0x151f waiting on condition [0x00007f69a636b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
   at java.lang.Thread.run(Thread.java:619)

"Thread-72" daemon prio=10 tid=0x0000000041dcb000 nid=0x12e5 in Object.wait() [0x00007f699e2e1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69db88bae8> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at com.sun.star.lib.util.AsynchronousFinalizer$1.run(AsynchronousFinalizer.java:89)
   - locked <0x00007f69db88bae8> (a java.util.LinkedList)

"MessageDispatcher" prio=10 tid=0x0000000042830800 nid=0x1243 runnable [0x00007f699e3e2000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007f69db3c22b8> (a java.io.BufferedInputStream)
   at com.sun.star.lib.connections.socket.SocketConnection.read(SocketConnection.java:158)
   at com.sun.star.lib.uno.bridges.java_remote.XConnectionInputStream_Adapter.read(XConnectionInputStream_Adapter.java:59)
   at java.io.DataInputStream.readInt(DataInputStream.java:370)
   at com.sun.star.lib.uno.protocols.urp.urp.readBlock(urp.java:349)
   at com.sun.star.lib.uno.protocols.urp.urp.readMessage(urp.java:89)
   at com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge$MessageDispatcher.run(java_remote_bridge.java:122)

"Thread-69" daemon prio=10 tid=0x000000004266b800 nid=0x1137 runnable [0x00007f699e6e5000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007f69db3c22e8> (a java.io.BufferedInputStream)
   at com.sun.jndi.ldap.Connection.run(Connection.java:807)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f69a10a3800 nid=0x1135 runnable [0x00007f699e8e7000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69db3ce9a0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"CIFS_IdleSessionReaper_NIO" daemon prio=10 tid=0x00007f69a81d5800 nid=0x1134 waiting on condition [0x00007f699e9e8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.smb.server.nio.NIOCifsConnectionsHandler$IdleSessionReaper.run(NIOCifsConnectionsHandler.java:158)
   at java.lang.Thread.run(Thread.java:619)

"CIFSConnectionsHandler" prio=10 tid=0x00007f69a8d27000 nid=0x1133 runnable [0x00007f699eae9000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
   - locked <0x00007f69db3c8ed0> (a sun.nio.ch.Util$1)
   - locked <0x00007f69db3c8eb8> (a java.util.Collections$UnmodifiableSet)
   - locked <0x00007f69db3d3b08> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
   at org.alfresco.jlan.smb.server.nio.NIOCifsConnectionsHandler.run(NIOCifsConnectionsHandler.java:404)
   at java.lang.Thread.run(Thread.java:619)

"CIFSRequestHandler_1" prio=10 tid=0x00007f69a8b96000 nid=0x1132 in Object.wait() [0x00007f699ebea000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69db3d6870> (a org.alfresco.jlan.server.SrvSessionQueue)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.SrvSessionQueue.waitWhileEmpty(SrvSessionQueue.java:110)
   - locked <0x00007f69db3d6870> (a org.alfresco.jlan.server.SrvSessionQueue)
   at org.alfresco.jlan.smb.server.nio.CIFSRequestHandler.run(CIFSRequestHandler.java:244)
   at java.lang.Thread.run(Thread.java:619)

"FTP Server" prio=10 tid=0x00007f69a0dc1800 nid=0x111f runnable [0x00007f699f0ef000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69db3c2650> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.alfresco.jlan.ftp.FTPServer.run(FTPServer.java:587)
   at java.lang.Thread.run(Thread.java:619)

"NetBIOSRefresh" daemon prio=10 tid=0x00007f69a10a2800 nid=0x1100 waiting on condition [0x00007f699eceb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer$NetBIOSNameRefresh.run(NetBIOSNameServer.java:578)

"NetBIOSRequest" daemon prio=10 tid=0x00007f69a015e000 nid=0x10ff in Object.wait() [0x00007f699edec000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d81d7ca8> (a java.util.Vector)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer$NetBIOSRequestHandler.run(NetBIOSNameServer.java:215)
   - locked <0x00007f69d81d7ca8> (a java.util.Vector)

"CIFS Server" prio=10 tid=0x00007f69a015d000 nid=0x10f5 waiting on condition [0x00007f699eeed000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.smb.server.SMBServer.run(SMBServer.java:490)
   at java.lang.Thread.run(Thread.java:619)

"NetBIOS Name Server" daemon prio=10 tid=0x00007f69a0585800 nid=0x10f4 runnable [0x00007f699efee000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainDatagramSocketImpl.receive0(Native Method)
   - locked <0x00007f69d81d7bf8> (a java.net.PlainDatagramSocketImpl)
   at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
   - locked <0x00007f69d81d7bf8> (a java.net.PlainDatagramSocketImpl)
   at java.net.DatagramSocket.receive(DatagramSocket.java:725)
   - locked <0x00007f69d81d8cd0> (a java.net.DatagramPacket)
   - locked <0x00007f69d81d7ba8> (a java.net.DatagramSocket)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer.run(NetBIOSNameServer.java:1613)
   at java.lang.Thread.run(Thread.java:619)

"PassthruReaper" daemon prio=10 tid=0x00007f69a0214800 nid=0x10ef waiting on condition [0x00007f699f1f0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.repo.security.authentication.ntlm.NTLMAuthenticationComponentImpl$PassthruReaperThread.run(NTLMAuthenticationComponentImpl.java:166)

"PassthruOfflineChecker" daemon prio=10 tid=0x00007f69a026b800 nid=0x10e6 waiting on condition [0x00007f699f2f1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.auth.passthru.PassthruServers$PassthruOfflineChecker.run(PassthruServers.java:122)

"NodeMonitor_Alfresco" daemon prio=10 tid=0x00007f69a0269800 nid=0x10df in Object.wait() [0x00007f699f3f2000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d80a43d8> (a org.alfresco.filesys.repo.NodeEventQueue)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.filesys.repo.NodeEventQueue.waitWhileEmpty(NodeEventQueue.java:111)
   - locked <0x00007f69d80a43d8> (a org.alfresco.filesys.repo.NodeEventQueue)
   at org.alfresco.filesys.repo.NodeEventQueue.removeEvent(NodeEventQueue.java:78)
   - locked <0x00007f69d80a43d8> (a org.alfresco.filesys.repo.NodeEventQueue)
   at org.alfresco.filesys.repo.NodeMonitor.run(NodeMonitor.java:525)
   at java.lang.Thread.run(Thread.java:619)

"FileStateReaper" daemon prio=10 tid=0x00007f69a066a800 nid=0x10de waiting on condition [0x00007f699f4f3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.filesys.cache.FileStateReaper.run(FileStateReaper.java:153)
   at java.lang.Thread.run(Thread.java:619)

"ContentQuotaManagerChecker" daemon prio=10 tid=0x00007f69a0c74800 nid=0x10dd waiting on condition [0x00007f699f5f4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.filesys.repo.ContentQuotaManager.run(ContentQuotaManager.java:423)
   at java.lang.Thread.run(Thread.java:619)

"TimedRequestProcessor" daemon prio=10 tid=0x00007f69a0c6d800 nid=0x10dc waiting on condition [0x00007f699f6f5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$TimedRequestProcessor.run(ThreadRequestPool.java:229)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker25" daemon prio=10 tid=0x00007f69a07bf000 nid=0x10db in Object.wait() [0x00007f699f7f6000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker24" daemon prio=10 tid=0x00007f69a07bd000 nid=0x10da in Object.wait() [0x00007f699f8f7000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker23" daemon prio=10 tid=0x00007f69a07bb000 nid=0x10d9 in Object.wait() [0x00007f699f9f8000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker22" daemon prio=10 tid=0x00007f69a0dba800 nid=0x10d8 in Object.wait() [0x00007f699faf9000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker21" daemon prio=10 tid=0x00007f69a0db8800 nid=0x10d7 in Object.wait() [0x00007f699fbfa000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker20" daemon prio=10 tid=0x00007f69a0db6800 nid=0x10d6 in Object.wait() [0x00007f699fcfb000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker19" daemon prio=10 tid=0x00007f69a0db5000 nid=0x10d5 in Object.wait() [0x00007f699fdfc000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker18" daemon prio=10 tid=0x00007f69a1258000 nid=0x10d4 in Object.wait() [0x00007f699fefd000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker17" daemon prio=10 tid=0x00007f69a1256000 nid=0x10d3 in Object.wait() [0x00007f699fffe000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker16" daemon prio=10 tid=0x00007f69a1254800 nid=0x10d2 in Object.wait() [0x00007f69a4142000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker15" daemon prio=10 tid=0x00007f69a1252800 nid=0x10d1 in Object.wait() [0x00007f69a4243000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker14" daemon prio=10 tid=0x00007f69a0066800 nid=0x10d0 in Object.wait() [0x00007f69a4344000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker13" daemon prio=10 tid=0x00007f69a0064800 nid=0x10cf in Object.wait() [0x00007f69a4445000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker12" daemon prio=10 tid=0x00007f69a0062800 nid=0x10ce in Object.wait() [0x00007f69a4546000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker11" daemon prio=10 tid=0x00007f69a0173000 nid=0x10cd in Object.wait() [0x00007f69a4647000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker10" daemon prio=10 tid=0x00007f69a0c54000 nid=0x10cc in Object.wait() [0x00007f69a4748000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker9" daemon prio=10 tid=0x00007f69a05a6000 nid=0x10cb in Object.wait() [0x00007f69a4849000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker8" daemon prio=10 tid=0x00007f69a05a5800 nid=0x10ca in Object.wait() [0x00007f69a494a000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker7" daemon prio=10 tid=0x00007f69a0083000 nid=0x10c9 in Object.wait() [0x00007f69a4a4b000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker6" daemon prio=10 tid=0x00007f69a0082000 nid=0x10c8 in Object.wait() [0x00007f69a4b4c000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker5" daemon prio=10 tid=0x00007f69a16c6000 nid=0x10c7 in Object.wait() [0x00007f69a4c4d000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker4" daemon prio=10 tid=0x00007f69a04ac000 nid=0x10c6 in Object.wait() [0x00007f69a4d4e000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker3" daemon prio=10 tid=0x00007f69a0aa7800 nid=0x10c5 in Object.wait() [0x00007f69a4e4f000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker2" daemon prio=10 tid=0x00007f69a12b1000 nid=0x10c4 in Object.wait() [0x00007f69a4f50000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"AlfJLANWorker1" daemon prio=10 tid=0x00007f69a0cfc800 nid=0x10c3 in Object.wait() [0x00007f69a5051000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d8039ea0> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:112)
   - locked <0x00007f69d8039ea0> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:619)

"indexThread10" daemon prio=10 tid=0x00007f69a12c9800 nid=0x1082 waiting on condition [0x00007f69a5152000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread9" daemon prio=10 tid=0x0000000041e5a000 nid=0x1081 waiting on condition [0x00007f69a5253000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread8" daemon prio=10 tid=0x00007f69a0f09000 nid=0x1080 waiting on condition [0x00007f69a5354000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread7" daemon prio=10 tid=0x00007f69a021c000 nid=0x107f waiting on condition [0x00007f69a5455000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread6" daemon prio=10 tid=0x00000000426a2800 nid=0x107e waiting on condition [0x00007f69a5556000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread5" daemon prio=10 tid=0x00007f69a011f800 nid=0x107d waiting on condition [0x00007f69a5657000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread4" daemon prio=10 tid=0x00007f69a034a000 nid=0x107c waiting on condition [0x00007f69a5758000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"Thread-22" daemon prio=10 tid=0x00007f69a0276000 nid=0x105e runnable [0x00007f69a5a5b000]
   java.lang.Thread.State: RUNNABLE
   at java.io.FileInputStream.readBytes(Native Method)
   at java.io.FileInputStream.read(FileInputStream.java:199)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007f69d7c17cc8> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:90)
   at org.alfresco.util.exec.RuntimeExec$InputStreamReaderThread.run(RuntimeExec.java:736)
   - locked <0x00007f69d7bb3fe8> (a org.alfresco.util.exec.RuntimeExec$InputStreamReaderThread)

"Thread-21" daemon prio=10 tid=0x00007f69a0752800 nid=0x105d runnable [0x00007f69a595a000]
   java.lang.Thread.State: RUNNABLE
   at java.io.FileInputStream.readBytes(Native Method)
   at java.io.FileInputStream.read(FileInputStream.java:199)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007f69d7c14f48> (a java.io.BufferedInputStream)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00007f69d7c17468> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:90)
   at org.alfresco.util.exec.RuntimeExec$InputStreamReaderThread.run(RuntimeExec.java:736)
   - locked <0x00007f69d7bb3528> (a org.alfresco.util.exec.RuntimeExec$InputStreamReaderThread)

"process reaper" daemon prio=10 tid=0x00007f69a074f000 nid=0x105b runnable [0x00007f69a5859000]
   java.lang.Thread.State: RUNNABLE
   at java.lang.UNIXProcess.waitForProcessExit(Native Method)
   at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
   at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)

"indexThread3" daemon prio=10 tid=0x00000000428fc800 nid=0x103a waiting on condition [0x00007f69a5b5c000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread2" daemon prio=10 tid=0x000000004240d800 nid=0x1039 waiting on condition [0x00007f69a5c5d000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"indexThread1" daemon prio=10 tid=0x00000000428bd000 nid=0x1038 waiting on condition [0x00007f69a5d5e000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40057a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"Timer-0" daemon prio=10 tid=0x000000004290a000 nid=0x1029 in Object.wait() [0x00007f69a5f67000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00007f69d79e0840> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"RMI TCP Accept-50506" daemon prio=10 tid=0x0000000041eda800 nid=0xf93 runnable [0x00007f69a626a000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59d7e00> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50505" daemon prio=10 tid=0x0000000042856000 nid=0xf81 runnable [0x00007f69a65e4000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59e2e38> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50504" daemon prio=10 tid=0x000000004206f800 nid=0xf7e runnable [0x00007f69a68e7000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59e2ef0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50503" daemon prio=10 tid=0x0000000042769800 nid=0xf7b runnable [0x00007f69a6bea000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59e2fa8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50502" daemon prio=10 tid=0x0000000042345000 nid=0xf78 runnable [0x00007f69a6eed000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59e1fd8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50501" daemon prio=10 tid=0x00000000425ea000 nid=0xf76 runnable [0x00007f69a70ef000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d59e3060> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"Thread-13" daemon prio=10 tid=0x0000000042616800 nid=0xf75 in Object.wait() [0x00007f69a71f0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.alfresco.repo.avm.AVMRemoteTransportService.run(AVMRemoteTransportService.java:169)
   - locked <0x00007f69d59ede98> (a org.alfresco.repo.avm.AVMRemoteTransportService)
   at java.lang.Thread.run(Thread.java:619)

"DefaultScheduler_Worker-10" prio=10 tid=0x00007f69a0677800 nid=0xf5e in Object.wait() [0x00007f69a73f2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa510> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-9" prio=10 tid=0x00007f69a0542000 nid=0xf5d in Object.wait() [0x00007f69a74f3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa5d8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-8" prio=10 tid=0x00007f69a1477800 nid=0xf5c in Object.wait() [0x00007f69a75f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa6a0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-7" prio=10 tid=0x00007f69a1091800 nid=0xf5b in Object.wait() [0x00007f69a76f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa768> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-6" prio=10 tid=0x00007f69a00c9000 nid=0xf5a in Object.wait() [0x00007f69a77f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa830> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-5" prio=10 tid=0x00007f69a00e7800 nid=0xf59 in Object.wait() [0x00007f69a78f7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa8f8> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-4" prio=10 tid=0x00007f69a0dc6000 nid=0xf58 in Object.wait() [0x00007f69a79f8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fa9c0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-3" prio=10 tid=0x0000000042434800 nid=0xf57 in Object.wait() [0x00007f69a7af9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59faa88> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-2" prio=10 tid=0x0000000042066800 nid=0xf56 in Object.wait() [0x00007f69a7bfa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fab50> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"DefaultScheduler_Worker-1" prio=10 tid=0x00000000421f5800 nid=0xf55 in Object.wait() [0x00007f69a7cfb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:523)
   - locked <0x00007f69d59fac18> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x0000000042166800 nid=0xeda in Object.wait() [0x00007f69a7dfc000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d40cb9e8> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:485)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x00007f69d40cb9e8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"RMI Scheduler(0)" daemon prio=10 tid=0x00007f69a10a6800 nid=0xec6 waiting on condition [0x00007f69ac2dd000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00007f69d40cbc08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
   at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)

"RMI Reaper" prio=10 tid=0x00007f69a0ac5800 nid=0xec4 in Object.wait() [0x00007f69ac4df000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d40cd018> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00007f69d40cd018> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50508" daemon prio=10 tid=0x00007f69a0ce1800 nid=0xec3 runnable [0x00007f69ac5e0000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d40cc5b0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-50500" daemon prio=10 tid=0x00007f69a00ea000 nid=0xec2 runnable [0x00007f69ac6e1000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69d40cbee8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:619)

"Store org.alfresco.cache.ticketsCache Spool Thread" daemon prio=10 tid=0x0000000042442000 nid=0xead waiting on condition [0x00007f69ac7e2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)
   at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:74)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

"net.sf.ehcache.CacheManager@38002f54" daemon prio=10 tid=0x00000000420b7800 nid=0xeac in Object.wait() [0x00007f69ac8e3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d40cd448> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00007f69d40cd448> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"net.sf.ehcache.CacheManager@5fe940a6" daemon prio=10 tid=0x000000004232f000 nid=0xeab in Object.wait() [0x00007f69ace08000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d40cdc98> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00007f69d40cdc98> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"GC Daemon" daemon prio=10 tid=0x0000000041e2a000 nid=0xe10 in Object.wait() [0x00007f69ad6ca000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00007f69d34a0600> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x00007f69d34a0600> (a sun.misc.GC$LatencyLock)

"Low Memory Detector" daemon prio=10 tid=0x00007f69a8028800 nid=0xdf5 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007f69a8026000 nid=0xdf4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00007f69a8024000 nid=0xdf3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f69a8021800 nid=0xdf2 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007f69a801f800 nid=0xdf1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000041d87800 nid=0xdd7 in Object.wait() [0x00007f69adf4f000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00007f69d3300a70> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000041d85800 nid=0xdd6 in Object.wait() [0x00007f69ae050000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x00007f69d3300900> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000041cd3800 nid=0xdd1 runnable [0x00007f6a3f4a5000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00007f69dc745ca0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
   at org.apache.catalina.startup.Catalina.await(Catalina.java:657)
   at org.apache.catalina.startup.Catalina.start(Catalina.java:617)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

"VM Thread" prio=10 tid=0x0000000041d81000 nid=0xdd5 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000041ce2800 nid=0xdd2 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000041ce4000 nid=0xdd3 runnable 

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000041d37800 nid=0xdd4 runnable 
"VM Periodic Task Thread" prio=10 tid=0x00007f69a802c000 nid=0xdf6 waiting on condition 

JNI global references: 801‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Access Share website locks cpu at 100%

brazen — Fri, 27 Aug 2010 21:34:39 GMT

The problem actually seems to be rather sporadic as to when it happens. Sometimes I can log in to share, sometimes I can't. Sometimes I can get all the way to creating a new text document, sometimes I can't. Sometimes it happens when I click to view a document, and sometimes it actually shows the page for the preview and then stops/maxes-out. I can NEVER get so far as to actually show the preview. The farthest I ever get is to show the document preview page but then it says something like "generating preview" and stays there. I've let it set for an hour just to see, and no change.

I even tried uninstalling all openoffice packages from Ubuntu and install the debs downloaded from the openoffice website, but no change.

Re: Access Share website locks cpu at 100%

brazen — Mon, 30 Aug 2010 21:47:56 GMT

Ok I fixed the problem by change the JAVA_OPTS in /etc/default/tomcat6, but I'm afraid my JAVA_OPTS may still not be optimal. Keep in mind, I'm on a VMWARE ESX Server virtual machine with 3G of RAM and 4 cpus assigned to this virtual machine.

Here is how I originally had my JAVA_OPTS, as suggested by the Alfresco installation guide for a machine with more than 2G of ram:


JAVA_OPTS="${JAVA_OPTS} -XX:+UseConcMarkSweepGC"
JAVA_OPTS="$JAVA_OPTS -server -Xss1024K -Xms1G -Xmx2G -XX:MaxPermSize=128M -XX:NewSize=512m -XX:CMSInitiatingOccupancyFraction=80"
‍‍‍‍

I was looking through, the tomcat bundle and got these JAVA_OPTS from it's configuration. So I commented out all JAVA_OPTS in /etc/default/tomcat6, and put this in:


JAVA_OPTS="-Xms512m -Xmx1024m -Xss1024k -XX:MaxPermSize=256m -XX:NewSize=256m -server"
‍‍‍

The only possibly offending option I can see is using the CMS garbage collecter, but I've seen numerous places suggest using this. Or maybe including $JAVA_OPTS is pulling in something bad. Any thoughts?

Re: Access Share website locks cpu at 100%

brazen — Mon, 30 Aug 2010 23:32:22 GMT

Through meticulously changing each java_opts settings, one at a time, and then testing… I've found the offending settings to be -XX:MaxPermSize. If I set it to 128m, which is what it should be set at according to the jvm tuning wiki page and the installation guide, then I have the problem with share. If I set it to 256m, then it works fine.

Has anybody else experienced this? Does it work fine for other people with -XX:MaxPermSize=128m ? Would there be any advantage to setting it to something even higher, such as 512m?

My java_opts line in /etc/default/tomcat6 ends up being:


JAVA_OPTS="${JAVA_OPTS} -Xms1G -Xmx2G -Xss1024k -XX:MaxPermSize=256m -XX:NewSize=512m -server -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80"
‍‍‍

Re: Access Share website locks cpu at 100%

kevinr — Tue, 31 Aug 2010 12:02:21 GMT

Yes this is a problem with many large webapps, it is not an Alfresco problem as such, the JVM needs a lot of space to load in all the classes for a large app plus large J2EE container (TomCat is small, but JBoss is bigger) - so 256MB is probably needed - especially on 64bit machines!

Cheers,

Kev