cancel
Showing results for 
Search instead for 
Did you mean: 

alfresco4 cpu usage

sharifu
Confirmed Champ
Confirmed Champ
I have set up alfresco 4 on centos 64bit. allocated 8gb ram with 2 cpu's
the server is dedicated jus tfor alfresco
Over the past month the cpu usage has increased by 25%

Can some one help me pinpoint why?


VM summary from jconsole



VM Summary
Thursday, 2 August 2012 11:53:07 o'clock BST

Connection name: 
controlRole@service:jmx:rmi:///jndi/rmi://hector.woking:50500/alfresco/jmxrmi
Virtual Machine: 
Java HotSpot(TM) 64-Bit Server VM version 17.1-b03
Vendor: 
Sun Microsystems Inc.
Name: 
26378@hector
Uptime: 
62 days 18 hours 35 minutes
Process CPU time: 
22 days 8 hours 29 minutes
JIT compiler: 
HotSpot 64-Bit Server Compiler
Total compile time: 
8 minutes

Live threads: 
    334
Peak: 
    399
Daemon threads: 
    299
Total threads started: 
159,604
Current classes loaded: 
32,843
Total classes loaded: 
38,693
Total classes unloaded: 
 5,850



Current heap size: 
1,823,115 kbytes
Maximum heap size: 
3,040,896 kbytes
Committed memory: 
3,040,896 kbytes
Pending finalization: 
0 objects
Garbage collector: 
Name = 'ParNew', Collections = 211,823, Total time spent = 6 hours 32 minutes
Garbage collector: 
Name = 'ConcurrentMarkSweep', Collections = 547, Total time spent = 1 hour 9 minutes

Operating System: 
Linux 2.6.18-194.17.1.el5
Architecture: 
amd64
Number of processors: 
2
Committed virtual memory: 
5,571,744 kbytes
Total physical memory: 
8,245,320 kbytes
Free physical memory: 
  395,204 kbytes
Total swap space: 
8,385,920 kbytes
Free swap space: 
8,385,816 kbytes

VM arguments: 
-Djava.util.logging.config.file=/opt/alfresco-4.0.d/tomcat/conf/logging.properties -Xms2G -Xmx3G -XX:MaxPermSize=512M -Xss1024k -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=80 -XX:NewSize=1G -Dalfresco.home=/opt/alfresco-4.0.d -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=172.16.0.21 -Dsun.security.ssl.allowUnsafeRenegotiation=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/alfresco-4.0.d/tomcat/endorsed -Dcatalina.base=/opt/alfresco-4.0.d/tomcat -Dcatalina.home=/opt/alfresco-4.0.d/tomcat -Djava.io.tmpdir=/opt/alfresco-4.0.d/tomcat/temp
Class path: 
/opt/alfresco-4.0.d/tomcat/bin/bootstrap.jar
Library path: 
/opt/alfresco-4.0.d/java/jre/lib/amd64/server:/opt/alfresco-4.0.d/java/jre/lib/amd64:/opt/alfresco-4.0.d/java/jre/../lib/amd64:/usr/local/apr/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Boot class path: 
/opt/alfresco-4.0.d/tomcat/endorsed/serializer.jar:/opt/alfresco-4.0.d/tomcat/endorsed/xalan.jar:/opt/alfresco-4.0.d/java/jre/lib/resources.jar:/opt/alfresco-4.0.d/java/jre/lib/rt.jar:/opt/alfresco-4.0.d/java/jre/lib/sunrsasign.jar:/opt/alfresco-4.0.d/java/jre/lib/jsse.jar:/opt/alfresco-4.0.d/java/jre/lib/jce.jar:/opt/alfresco-4.0.d/java/jre/lib/charsets.jar:/opt/alfresco-4.0.d/java/jre/classes
4 REPLIES 4

afaust
Legendary Innovator
Legendary Innovator
Hello,

it is difficult / near impossible to answer that based on the information at hand. I suggest you take some measurements on what's going on in your VM by regularily running jstack. This gives you output that may be helpful in determining what operations are being executed at times of high CPU usage. You may also use Alfresco auditing for collection of semantic data to see what use cases are executed by users and have some reference data for analysis (i.e. for a move, investigate the scope via node browser, e.g. how many nodes are affected etc.).

Regards
Axel

sharifu
Confirmed Champ
Confirmed Champ
Im not sure how to do this. can you explain a bit futher

sharifu
Confirmed Champ
Confirmed Champ

2012-08-03 10:36:20
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

"Attach Listener" daemon prio=10 tid=0x0000000056363800 nid=0x5696 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"RMI TCP Connection(55102)-172.16.0.49" daemon prio=10 tid=0x00002aabcc5d9800 nid=0x541c runnable [0x00000000552c3000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   - locked <0x00002aab5b077cf0> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:66)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Connection(54874)-172.16.0.49" daemon prio=10 tid=0x00002aabcc01c800 nid=0x45b9 runnable [0x000000005d169000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   - locked <0x00002aab44428ff8> (a java.io.BufferedInputStream)
   at java.io.FilterInputStream.read(FilterInputStream.java:66)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

"Thread-127980" daemon prio=10 tid=0x00002aabc80b2800 nid=0x3ffa runnable [0x00000000532a3000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
   - locked <0x00002aab33afbe38> (a java.io.BufferedInputStream)
   at com.sun.jndi.ldap.Connection.run(Connection.java:808)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Connection(54307)-172.16.0.49" daemon prio=10 tid=0x00002aabcc0e2800 nid=0x2654 in Object.wait() [0x000000005a141000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf2376528> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
   at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417)
   - locked <0x00002aaaf2376528> (a com.sun.jmx.remote.internal.ArrayNotificationBuffer)
   at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209)
   at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258)
   at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227)
   at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1233)
   at sun.reflect.GeneratedMethodAccessor6174.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
   at sun.rmi.transport.Transport$1.run(Transport.java:159)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

"JMX server connection timeout 159603" daemon prio=10 tid=0x0000000056e17000 nid=0x2653 in Object.wait() [0x0000000052798000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aab2e5a32b0> (a [I)
   at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
   - locked <0x00002aab2e5a32b0> (a [I)
   at java.lang.Thread.run(Thread.java:662)

   …
   …
   …
   .
   .
   .

   
"http-8080-1" daemon prio=10 tid=0x0000000057dba800 nid=0x67d5 in Object.wait() [0x00000000446c1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaff4cccb0> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
   - locked <0x00002aaaff4cccb0> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
   at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
   at java.lang.Thread.run(Thread.java:662)

"http-8443-4" daemon prio=10 tid=0x000000005706f800 nid=0x67d4 runnable [0x0000000040857000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
   at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
   at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
   - locked <0x00002aab3108c378> (a com.mysql.jdbc.util.ReadAheadInputStream)
   at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2499)
   at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2952)
   at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2941)
   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3489)
   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2562)
   - locked <0x00002aab2fc6f008> (a java.lang.Object)
   at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1643)
   - locked <0x00002aab2fc6f008> (a java.lang.Object)
   at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)
   at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.commit(PoolingDataSource.java:211)
   at org.hibernate.transaction.JDBCTransaction.commitAndResetAutoCommit(JDBCTransaction.java:139)
   at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:115)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   - locked <0x00002aaac651fcf0> (a org.alfresco.util.transaction.SpringAwareUserTransaction)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
   at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:372)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
   at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:662)

"http-8443-3" daemon prio=10 tid=0x0000000056779800 nid=0x67ca runnable [0x0000000048f08000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
   at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
   at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
   - locked <0x00002aab2dceb310> (a com.mysql.jdbc.util.ReadAheadInputStream)
   at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2499)
   at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2952)
   at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2941)
   at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3489)
   at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2113)
   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2562)
   - locked <0x00002aab2d4e0998> (a java.lang.Object)
   at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4956)
   - locked <0x00002aab2d4e0998> (a java.lang.Object)
   at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
   at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
   at org.apache.commons.dbcp.PoolableConnectionFactory.activateObject(PoolableConnectionFactory.java:704)
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
   at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
   at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:50)
   at $Proxy5.setReadOnly(Unknown Source)
   at org.springframework.jdbc.datasource.DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:155)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:508)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:335)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:273)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.begin(SpringAwareUserTransaction.java:411)
   - locked <0x00002aaac7c93460> (a org.alfresco.util.transaction.SpringAwareUserTransaction)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:377)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
   at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:372)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
   at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:662)

"http-8443-2" daemon prio=10 tid=0x00000000563e0800 nid=0x67c9 in Object.wait() [0x0000000048e08000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafdf580f8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00002aaafdf580f8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:662)

"http-8443-1" daemon prio=10 tid=0x00000000563df800 nid=0x67c8 in Object.wait() [0x0000000048d07000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafdf5ae58> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
   - locked <0x00002aaafdf5ae58> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
   at java.lang.Thread.run(Thread.java:662)

"http-8443-Acceptor-0" daemon prio=10 tid=0x0000000056e4d800 nid=0x67c7 runnable [0x0000000048c06000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaee42da00> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:152)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
   at java.lang.Thread.run(Thread.java:662)

"ajp-8009-Acceptor-0" daemon prio=10 tid=0x0000000057e3f800 nid=0x67c6 runnable [0x0000000048b05000]
   java.lang.Thread.State: RUNNABLE
   at org.apache.tomcat.jni.Socket.accept(Native Method)
   at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
   at java.lang.Thread.run(Thread.java:662)

"ajp-8009-CometPoller-0" daemon prio=10 tid=0x0000000056b92000 nid=0x67c5 in Object.wait() [0x0000000048a04000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf5e89268> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
   - locked <0x00002aaaf5e89268> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Thread.run(Thread.java:662)

"ajp-8009-Poller-0" daemon prio=10 tid=0x00000000560a9800 nid=0x67c4 in Object.wait() [0x0000000048903000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf5e892b0> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
   - locked <0x00002aaaf5e892b0> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Thread.run(Thread.java:662)

"http-8080-Acceptor-0" daemon prio=10 tid=0x00000000560a9000 nid=0x67c3 runnable [0x0000000048802000]
   java.lang.Thread.State: RUNNABLE
   at org.apache.tomcat.jni.Socket.accept(Native Method)
   at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
   at java.lang.Thread.run(Thread.java:662)

"http-8080-Sendfile-0" daemon prio=10 tid=0x00000000565f3800 nid=0x67c2 in Object.wait() [0x0000000048701000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf5e88fe0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1805)
   - locked <0x00002aaaf5e88fe0> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
   at java.lang.Thread.run(Thread.java:662)

"http-8080-CometPoller-0" daemon prio=10 tid=0x0000000056e65800 nid=0x67c1 in Object.wait() [0x00000000476f1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafdebcf20> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
   - locked <0x00002aaafdebcf20> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Thread.run(Thread.java:662)

"http-8080-Poller-0" daemon prio=10 tid=0x00000000566cc000 nid=0x67c0 in Object.wait() [0x000000004102a000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf5e89028> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Object.wait(Object.java:485)
   at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
   - locked <0x00002aaaf5e89028> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
   at java.lang.Thread.run(Thread.java:662)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x000000005640e800 nid=0x67bf waiting on condition [0x00000000430ab000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
   at java.lang.Thread.run(Thread.java:662)

"JobLockService1" prio=10 tid=0x000000005702f000 nid=0x67be waiting on condition [0x00000000484ff000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf298c408> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"AlfrescoJbpmJobExecutor:172.16.0.21:1" daemon prio=10 tid=0x00002aab98d40800 nid=0x67bd in Object.wait() [0x00000000481fc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf9e02cc0> (a org.alfresco.repo.workflow.jbpm.AlfrescoJobExecutor)
   at org.jbpm.job.executor.JobExecutorThread.run(JobExecutorThread.java:68)
   - locked <0x00002aaaf9e02cc0> (a org.alfresco.repo.workflow.jbpm.AlfrescoJobExecutor)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x00002aaba09f3800 nid=0x67b9 in Object.wait() [0x0000000048600000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf2af9b00> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaf2af9b00> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

"RMI RenewClean-[172.16.0.21:46963]" daemon prio=10 tid=0x0000000056101800 nid=0x67b1 in Object.wait() [0x00000000480fb000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa126320> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaafa126320> (a java.lang.ref.ReferenceQueue$Lock)
   at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-0" daemon prio=10 tid=0x00002aaba1223000 nid=0x67b0 runnable [0x0000000047ffa000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaafa072a80> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:662)

"CIFS_IdleSessionReaper_NIO" daemon prio=10 tid=0x00002aab98a9a000 nid=0x67af waiting on condition [0x0000000047ef9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.smb.server.nio.NIOCifsConnectionsHandler$IdleSessionReaper.run(NIOCifsConnectionsHandler.java:158)
   at java.lang.Thread.run(Thread.java:662)

"CIFSConnectionsHandler" prio=10 tid=0x00002aab98a98800 nid=0x67ae runnable [0x0000000047df8000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
   - locked <0x00002aaafa189028> (a sun.nio.ch.Util$1)
   - locked <0x00002aaafa189040> (a java.util.Collections$UnmodifiableSet)
   - locked <0x00002aaaf9e02d38> (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
   at org.alfresco.jlan.smb.server.nio.NIOCifsConnectionsHandler.run(NIOCifsConnectionsHandler.java:404)
   at java.lang.Thread.run(Thread.java:662)

"CIFSRequestHandler_1" prio=10 tid=0x00002aab9908e800 nid=0x67ad in Object.wait() [0x0000000047cf7000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa189058> (a org.alfresco.jlan.server.SrvSessionQueue)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.SrvSessionQueue.waitWhileEmpty(SrvSessionQueue.java:110)
   - locked <0x00002aaafa189058> (a org.alfresco.jlan.server.SrvSessionQueue)
   at org.alfresco.jlan.smb.server.nio.CIFSRequestHandler.run(CIFSRequestHandler.java:255)
   at java.lang.Thread.run(Thread.java:662)

"FTP Server" prio=10 tid=0x00002aaba0f08800 nid=0x67ac runnable [0x0000000047bf6000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaf6884468> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.alfresco.jlan.ftp.FTPServer.run(FTPServer.java:585)
   at java.lang.Thread.run(Thread.java:662)

"NetBIOSRefresh" daemon prio=10 tid=0x0000000056e97000 nid=0x67ab waiting on condition [0x0000000047af5000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer$NetBIOSNameRefresh.run(NetBIOSNameServer.java:578)

"NetBIOSRequest" daemon prio=10 tid=0x0000000056e96000 nid=0x67aa in Object.wait() [0x00000000479f4000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa18ce68> (a java.util.Vector)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer$NetBIOSRequestHandler.run(NetBIOSNameServer.java:215)
   - locked <0x00002aaafa18ce68> (a java.util.Vector)

"CIFS Server" prio=10 tid=0x00002aaba0ed9000 nid=0x67a9 waiting on condition [0x00000000478f3000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.smb.server.SMBServer.run(SMBServer.java:493)
   at java.lang.Thread.run(Thread.java:662)

"NetBIOS Name Server" daemon prio=10 tid=0x00002aaba0ed8800 nid=0x67a8 runnable [0x00000000477f2000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainDatagramSocketImpl.receive0(Native Method)
   - locked <0x00002aaafa188480> (a java.net.PlainDatagramSocketImpl)
   at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)
   - locked <0x00002aaafa188480> (a java.net.PlainDatagramSocketImpl)
   at java.net.DatagramSocket.receive(DatagramSocket.java:725)
   - locked <0x00002aaaf9ec6570> (a java.net.DatagramPacket)
   - locked <0x00002aaafa134180> (a java.net.DatagramSocket)
   at org.alfresco.jlan.netbios.server.NetBIOSNameServer.run(NetBIOSNameServer.java:1613)
   at java.lang.Thread.run(Thread.java:662)

"PassthruReaper" daemon prio=10 tid=0x00002aaba0ccc800 nid=0x67a1 waiting on condition [0x00000000475f0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.repo.security.authentication.ntlm.NTLMAuthenticationComponentImpl$PassthruReaperThread.run(NTLMAuthenticationComponentImpl.java:166)

"PassthruOfflineChecker" daemon prio=10 tid=0x00002aaba066a000 nid=0x67a0 waiting on condition [0x00000000474ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.auth.passthru.PassthruServers$PassthruOfflineChecker.run(PassthruServers.java:122)

"NodeMonitor_Alfresco" daemon prio=10 tid=0x00002aaba0f3d000 nid=0x679f in Object.wait() [0x00000000473ee000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa126860> (a org.alfresco.filesys.repo.NodeEventQueue)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.filesys.repo.NodeEventQueue.waitWhileEmpty(NodeEventQueue.java:111)
   - locked <0x00002aaafa126860> (a org.alfresco.filesys.repo.NodeEventQueue)
   at org.alfresco.filesys.repo.NodeEventQueue.removeEvent(NodeEventQueue.java:78)
   - locked <0x00002aaafa126860> (a org.alfresco.filesys.repo.NodeEventQueue)
   at org.alfresco.filesys.repo.NodeMonitor.run(NodeMonitor.java:524)
   at java.lang.Thread.run(Thread.java:662)

"Notify_Alfresco" daemon prio=10 tid=0x00002aaba0e48800 nid=0x679e in Object.wait() [0x00000000472ed000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa126938> (a org.alfresco.jlan.smb.server.notify.NotifyChangeEventList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.smb.server.notify.NotifyChangeHandler.run(NotifyChangeHandler.java:1008)
   - locked <0x00002aaafa126938> (a org.alfresco.jlan.smb.server.notify.NotifyChangeEventList)
   at java.lang.Thread.run(Thread.java:662)

"ContentQuotaManagerChecker" daemon prio=10 tid=0x00002aaba0ca5800 nid=0x679d waiting on condition [0x00000000471ec000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.filesys.repo.ContentQuotaManager.run(ContentQuotaManager.java:465)
   at java.lang.Thread.run(Thread.java:662)

"FileStateReaper" daemon prio=10 tid=0x00002aaba108a000 nid=0x679c waiting on condition [0x00000000470eb000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.filesys.cache.FileStateReaper.run(FileStateReaper.java:161)
   at java.lang.Thread.run(Thread.java:662)

"TimedRequestProcessor" daemon prio=10 tid=0x00002aaba08ae800 nid=0x679b waiting on condition [0x0000000046fea000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$TimedRequestProcessor.run(ThreadRequestPool.java:255)
   at java.lang.Thread.run(Thread.java:662)

"AlfJLANWorker25" daemon prio=10 tid=0x00002aaba0b01000 nid=0x679a in Object.wait() [0x0000000046ee9000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa164b40> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:106)
   - locked <0x00002aaafa164b40> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:662)

…….
……
…..

"AlfJLANWorker1" daemon prio=10 tid=0x00002aaba11ab000 nid=0x6782 in Object.wait() [0x00000000456d1000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa164b40> (a java.util.LinkedList)
   at java.lang.Object.wait(Object.java:485)
   at org.alfresco.jlan.server.thread.ThreadRequestQueue.removeRequest(ThreadRequestQueue.java:106)
   - locked <0x00002aaafa164b40> (a java.util.LinkedList)
   at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:127)
   at java.lang.Thread.run(Thread.java:662)

"Timer-7" prio=10 tid=0x00002aaba0be5000 nid=0x6781 in Object.wait() [0x00000000455d0000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaafa108ae8> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:485)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x00002aaafa108ae8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"indexThread10" daemon prio=10 tid=0x00002aaba1150000 nid=0x6780 waiting on condition [0x00000000454cf000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf628f710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)
…..
…..
…..

"indexThread4" daemon prio=10 tid=0x00002aaba01de000 nid=0x677a waiting on condition [0x0000000044fca000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf628f710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"Timer-6" prio=10 tid=0x00002aaba091d000 nid=0x6778 in Object.wait() [0x0000000044dc8000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf0163b70> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:485)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x00002aaaf0163b70> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)
….


..
.

"Timer-2" prio=10 tid=0x00002aaba11e9000 nid=0x6762 in Object.wait() [0x00000000449c4000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf426aaa0> (a java.util.TaskQueue)
   at java.lang.Object.wait(Object.java:485)
   at java.util.TimerThread.mainLoop(Timer.java:483)
   - locked <0x00002aaaf426aaa0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"indexThread3" daemon prio=10 tid=0x0000000056ee9800 nid=0x6761 waiting on condition [0x00000000448c3000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf628f710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"indexThread2" daemon prio=10 tid=0x00002aaba1253800 nid=0x6760 waiting on condition [0x0000000040dcb000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf628f710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"indexThread1" daemon prio=10 tid=0x00002aaba096c000 nid=0x675f waiting on condition [0x0000000040cca000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaf628f710> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"IndexInfo Cleaner Deamon" daemon prio=10 tid=0x00002aaba119e000 nid=0x675e in Object.wait() [0x00000000447c2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf23444f8> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00002aaaf23444f8> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"JobAcquisitionThread" prio=10 tid=0x00002aaba0ea5000 nid=0x675d waiting on condition [0x000000004030a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at org.activiti.engine.impl.jobexecutor.JobAcquisitionThread.run(JobAcquisitionThread.java:102)

"RMI RenewClean-[172.16.0.21:35074,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" daemon prio=10 tid=0x000000005747b800 nid=0x675b in Object.wait() [0x00000000445c0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf1d749c0> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaf1d749c0> (a java.lang.ref.ReferenceQueue$Lock)
   at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-0" daemon prio=10 tid=0x00002aaba0733000 nid=0x675a runnable [0x00000000444bf000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaf1d0cda8> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:662)

"Thread-7" daemon prio=10 tid=0x00002aaba09c8800 nid=0x6758 in Object.wait() [0x0000000040960000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf1ce9278> (a org.alfresco.repo.avm.AVMRemoteTransportService)
   at org.alfresco.repo.avm.AVMRemoteTransportService.run(AVMRemoteTransportService.java:169)
   - locked <0x00002aaaf1ce9278> (a org.alfresco.repo.avm.AVMRemoteTransportService)
   at java.lang.Thread.run(Thread.java:662)

"DefaultScheduler_QuartzSchedulerThread" prio=10 tid=0x00002aaba0683000 nid=0x6756 in Object.wait() [0x00000000443be000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236eab8> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:296)
   - locked <0x00002aaaf236eab8> (a java.lang.Object)

"DefaultScheduler_Worker-10" prio=10 tid=0x00002aaba0e81000 nid=0x6755 in Object.wait() [0x00000000442bd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236d4d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaf236d4d0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)





"DefaultScheduler_Worker-1" prio=10 tid=0x00002aaba0a20800 nid=0x674c in Object.wait() [0x00000000439b4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236dc20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaf236dc20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"Timer-1" daemon prio=10 tid=0x00002aaba0c05800 nid=0x674b in Object.wait() [0x00000000438b3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236ddb0> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00002aaaf236ddb0> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"SolrWatcherScheduler_QuartzSchedulerThread" prio=10 tid=0x00002aaba07de800 nid=0x674a in Object.wait() [0x00000000437b2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236df20> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:411)
   - locked <0x00002aaaf236df20> (a java.lang.Object)

"SolrWatcherScheduler_Worker-3" prio=10 tid=0x00002aaba07e4800 nid=0x6749 in Object.wait() [0x00000000436b1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236ead0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaf236ead0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"SolrWatcherScheduler_Worker-2" prio=10 tid=0x00002aaba069b800 nid=0x6748 in Object.wait() [0x00000000435b0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236eba0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaf236eba0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"SolrWatcherScheduler_Worker-1" prio=10 tid=0x00002aaba0d1f800 nid=0x6747 in Object.wait() [0x0000000040209000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf236ec70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaf236ec70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"RMI RenewClean-[172.16.0.21:50508]" daemon prio=10 tid=0x0000000056397000 nid=0x673e in Object.wait() [0x000000004122c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf237d540> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaf237d540> (a java.lang.ref.ReferenceQueue$Lock)
   at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
   at java.lang.Thread.run(Thread.java:662)

"RMI Scheduler(0)" daemon prio=10 tid=0x00002aaba089c800 nid=0x673d waiting on condition [0x000000004112b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaee282158> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
   at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"RMI Reaper" prio=10 tid=0x00002aaba071f800 nid=0x673b in Object.wait() [0x0000000042faa000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee281ab8> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaee281ab8> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-50508" daemon prio=10 tid=0x00002aaba092d800 nid=0x673a runnable [0x0000000042ea9000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaf2386fe0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-50500" daemon prio=10 tid=0x00002aaba0d43000 nid=0x6739 runnable [0x0000000040f29000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaf237a480> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:662)

"Store org.alfresco.cache.tagscopeSummaryCache Spool Thread" daemon prio=10 tid=0x00002aaba02c8800 nid=0x672c waiting on condition [0x0000000042ca7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)
   at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:74)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

"Store org.alfresco.cache.ticketsCache Spool Thread" daemon prio=10 tid=0x00002aaba0910800 nid=0x672b waiting on condition [0x0000000042ba6000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)
   at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:74)
   at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)

"net.sf.ehcache.CacheManager@70181c87" daemon prio=10 tid=0x00002aaba0740000 nid=0x672a in Object.wait() [0x0000000042aa5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaf01af070> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00002aaaf01af070> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x00002aab98fb8000 nid=0x6729 in Object.wait() [0x00000000428a3000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaef471828> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaef471828> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

"pool-3-thread-1" prio=10 tid=0x00002aaba0439800 nid=0x6728 waiting on condition [0x0000000041571000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaef41b0b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x00002aaba0104000 nid=0x6725 waiting on condition [0x00000000429a4000]
   java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00002aaaeec3d860> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:662)

"Timer-0" daemon prio=10 tid=0x00002aaba04c3800 nid=0x6723 in Object.wait() [0x00000000427a2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee560928> (a java.util.TaskQueue)
   at java.util.TimerThread.mainLoop(Timer.java:509)
   - locked <0x00002aaaee560928> (a java.util.TaskQueue)
   at java.util.TimerThread.run(Timer.java:462)

"SolrTrackerScheduler_QuartzSchedulerThread" prio=10 tid=0x00002aaba04c2800 nid=0x6722 in Object.wait() [0x00000000426a1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee55fac8> (a java.lang.Object)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:296)
   - locked <0x00002aaaee55fac8> (a java.lang.Object)

"SolrTrackerScheduler_Worker-3" prio=10 tid=0x00002aaba04c0800 nid=0x6721 in Object.wait() [0x00000000425a0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee55db48> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
   - locked <0x00002aaaee55db48> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"SolrTrackerScheduler_Worker-2" prio=10 tid=0x00002aaba04bf800 nid=0x6720 runnable [0x0000000040bc9000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
   at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
   - locked <0x00002aaac6e124b8> (a java.lang.Object)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
   at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
   - locked <0x00002aaac6e12620> (a com.sun.net.ssl.internal.ssl.AppInputStream)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   - locked <0x00002aaac6e241a8> (a java.io.BufferedInputStream)
   at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
   at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
   at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
   at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
   at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:110)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:86)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:307)
   at org.alfresco.solr.client.SOLRAPIClient.getAclChangeSets(SOLRAPIClient.java:149)
   at org.alfresco.solr.tracker.CoreTracker.getSomeAclChangeSets(CoreTracker.java:1545)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1265)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:491)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

"SolrTrackerScheduler_Worker-1" prio=10 tid=0x00002aaba04bd800 nid=0x671f runnable [0x0000000040ac8000]
   java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:129)
   at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
   at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
   - locked <0x00002aaac57aef10> (a java.lang.Object)
   at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
   at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
   - locked <0x00002aaac57af078> (a com.sun.net.ssl.internal.ssl.AppInputStream)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
   - locked <0x00002aaac57c0c00> (a java.io.BufferedInputStream)
   at org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:250)
   at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)
   at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)
   at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:196)
   at org.apache.commons.httpclient.ChunkedInputStream.exhaustInputStream(ChunkedInputStream.java:369)
   at org.apache.commons.httpclient.ChunkedInputStream.close(ChunkedInputStream.java:346)
   at java.io.FilterInputStream.close(FilterInputStream.java:155)
   at org.apache.commons.httpclient.AutoCloseInputStream.notifyWatcher(AutoCloseInputStream.java:194)
   at org.apache.commons.httpclient.AutoCloseInputStream.close(AutoCloseInputStream.java:158)
   at org.apache.commons.httpclient.HttpMethodBase.releaseConnection(HttpMethodBase.java:1181)
   at org.alfresco.httpclient.HttpMethodResponse.release(HttpMethodResponse.java:43)
   at org.alfresco.solr.client.SOLRAPIClient.getAclChangeSets(SOLRAPIClient.java:163)
   at org.alfresco.solr.tracker.CoreTracker.getSomeAclChangeSets(CoreTracker.java:1545)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1265)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:491)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

"GC Daemon" daemon prio=10 tid=0x00002aaba0058800 nid=0x671a in Object.wait() [0x0000000041a5c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee3db6f0> (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:100)
   - locked <0x00002aaaee3db6f0> (a sun.misc.GC$LatencyLock)

"RMI TCP Accept-0" daemon prio=10 tid=0x00002aab9804d000 nid=0x6718 runnable [0x000000004195b000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaaee282ca0> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
   at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
   at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x0000000055ea9800 nid=0x6717 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x0000000055ea7000 nid=0x6716 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0000000055ea2000 nid=0x6715 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0000000055ea0000 nid=0x6714 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000055e9d800 nid=0x6713 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000055e79000 nid=0x6712 in Object.wait() [0x0000000041e99000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee201030> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
   - locked <0x00002aaaee201030> (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
   at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000055e77000 nid=0x6711 in Object.wait() [0x0000000041d98000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on <0x00002aaaee200eb8> (a java.lang.ref.Reference$Lock)
   at java.lang.Object.wait(Object.java:485)
   at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
   - locked <0x00002aaaee200eb8> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000055db9000 nid=0x670c runnable [0x0000000041b95000]
   java.lang.Thread.State: RUNNABLE
   at java.net.PlainSocketImpl.socketAccept(Native Method)
   at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
   - locked <0x00002aaafb6e8410> (a java.net.SocksSocketImpl)
   at java.net.ServerSocket.implAccept(ServerSocket.java:453)
   at java.net.ServerSocket.accept(ServerSocket.java:421)
   at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
   at org.apache.catalina.startup.Catalina.await(Catalina.java:662)
   at org.apache.catalina.startup.Catalina.start(Catalina.java:614)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

"VM Thread" prio=10 tid=0x0000000055e73000 nid=0x6710 runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000055dc7800 nid=0x670d runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000055dc9800 nid=0x670e runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000055e28000 nid=0x670f runnable
"VM Periodic Task Thread" prio=10 tid=0x00002aab9805a800 nid=0x6719 waiting on condition

JNI global references: 2071


What do i do with the jstack?

afaust
Legendary Innovator
Legendary Innovator
Hello,

it depends on your technical knowledge of Java and Alfresco. Usually, you would run jstack every few seconds while the load is high and sample the output to derive common patterns, i.e. identify long running operations, massively parallel operations, etc., stuff that may cause your load.
Unfortunately, if you're not familiar with this type of analysis, the output will not be of much help to you, and I don't think anyone in the forum will have the time and want to go through more than 2-3 of the jstack outputs.

As an easier tool, you may want to try out a profile like JProfiler or YourKit, which can monitor your application for a while and produce some aggregated reports. You would still need to have some understanding of Java and Alfresco to make sense of it, but you generally end up with better / more condensed data and some graphical tools to help you make sense. Profiling on the other hand can incur sever performance penalties when done too intensively, so it should not be used on a production system for more than hotspot monitoring for a short period of time.

Regards
Axel