Alfresco randomly crashes every couple hours

[slooney]

I have a default install of Alfresco 4.2.c Community on redhat running a postgres database. It crashes every couple hours with the following messages in the catalina log.

2013-04-22 13:10:30,108  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-7] Agent: null
2013-04-22 13:10:30,108  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-7] Invoking Web Script org/alfresco/repository/solr/transactions.get (format json, style: argument, default: json)
2013-04-22 13:10:30,108  DEBUG [extensions.webscripts.AbstractWebScript] [http-bio-8443-exec-3] Rendered template org/alfresco/repository/solr/transactions.get.json in 0.337374ms
2013-04-22 13:10:30,109  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-3] Web Script org/alfresco/repository/solr/transactions.get executed in 3.542079ms
2013-04-22 13:10:30,109  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-3] Processed script url (GET) /api/solr/transactions in 3.801744ms
2013-04-22 13:10:30,110  DEBUG [webscripts.servlet.WebScriptServletResponse] [http-bio-8443-exec-7] Cache - set response header Cache-Control: no-cache
2013-04-22 13:10:30,110  DEBUG [webscripts.servlet.WebScriptServletResponse] [http-bio-8443-exec-7] Cache - set response header Pragma: no-cache
2013-04-22 13:10:30,110  DEBUG [extensions.webscripts.DeclarativeWebScript] [http-bio-8443-exec-7] Rendering response: content type=application/json, status=200
2013-04-22 13:10:30,110  DEBUG [extensions.webscripts.DeclarativeWebScript] [http-bio-8443-exec-7] Rendering template 'org/alfresco/repository/solr/transactions.get.json'
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractWebScript] [http-bio-8443-exec-7] Rendered template org/alfresco/repository/solr/transactions.get.json in 0.287676ms
2013-04-22 13:10:30,111  DEBUG [webscripts.servlet.WebScriptServlet] [http-bio-8443-exec-4] Processing request (GET) https://localhost/alfresco/service/api/solr/transactions?fromCommitTime=1366657651620&toCommitTime=1...
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-4] (Runtime=ServletRuntime, Container=Repository) Processing script url (GET) /api/solr/transactions
2013-04-22 13:10:30,111  DEBUG [webscripts.servlet.WebScriptServletRequest] [http-bio-8443-exec-4] Content Type: null
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-4] Agent: null
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-4] Invoking Web Script org/alfresco/repository/solr/transactions.get (format json, style: argument, default: json)
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-7] Web Script org/alfresco/repository/solr/transactions.get executed in 3.328746ms
2013-04-22 13:10:30,111  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-7] Processed script url (GET) /api/solr/transactions in 3.619028ms
2013-04-22 13:10:30,113  DEBUG [webscripts.servlet.WebScriptServletResponse] [http-bio-8443-exec-4] Cache - set response header Cache-Control: no-cache
2013-04-22 13:10:30,113  DEBUG [webscripts.servlet.WebScriptServletResponse] [http-bio-8443-exec-4] Cache - set response header Pragma: no-cache
2013-04-22 13:10:30,113  DEBUG [extensions.webscripts.DeclarativeWebScript] [http-bio-8443-exec-4] Rendering response: content type=application/json, status=200
2013-04-22 13:10:30,113  DEBUG [extensions.webscripts.DeclarativeWebScript] [http-bio-8443-exec-4] Rendering template 'org/alfresco/repository/solr/transactions.get.json'
2013-04-22 13:10:30,114  DEBUG [extensions.webscripts.AbstractWebScript] [http-bio-8443-exec-4] Rendered template org/alfresco/repository/solr/transactions.get.json in 0.273216ms
2013-04-22 13:10:30,114  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-4] Web Script org/alfresco/repository/solr/transactions.get executed in 3.011041ms
2013-04-22 13:10:30,114  DEBUG [extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-4] Processed script url (GET) /api/solr/transactions in 3.189685ms
2013-04-22 13:10:31,650  ERROR [alfresco.util.SocketOpenOfficeConnection] [MessageDispatcher] disconnected unexpectedly
Apr 22, 2013 1:10:31 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-apr-8080"]
Apr 22, 2013 1:10:31 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-apr-8009"]
Apr 22, 2013 1:10:31 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8443"]
Apr 22, 2013 1:10:31 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [org.springframework.extensions.surf.CssImageDataHandler$1] (value [org.springframework.extensions.surf.CssImageDataHandler$1@7c677ed1]) and a value of type [byte[]] (value [[B@6eb70b25]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@333ce2ec]) and a value of type [org.alfresco.web.scripts.SlingshotRemoteClient] (value [org.alfresco.web.scripts.SlingshotRemoteClient@62560e3b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [org.springframework.extensions.webscripts.connector.RemoteClient$1] (value [org.springframework.extensions.webscripts.connector.RemoteClient$1@7fc075ec]) and a value of type [org.apache.commons.httpclient.HttpClient] (value [org.apache.commons.httpclient.HttpClient@42fc5a82]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [org.springframework.extensions.webscripts.servlet.mvc.ResourceController$1] (value [org.springframework.extensions.webscripts.servlet.mvc.ResourceController$1@459a2f29]) and a value of type [byte[]] (value [[B@d8e5072]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper$1] (value [org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper$1@7a437351]) and a value of type [freemarker.template.DefaultObjectWrapper] (value [freemarker.template.DefaultObjectWrapper@6248098c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 22, 2013 1:10:31 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/share] created a ThreadLocal with key of type [org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper$1] (value [org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper$1@676af8bc]) and a value of type [freemarker.template.DefaultObjectWrapper] (value [freemarker.template.DefaultObjectWrapper@1ff033af]) but failed to..

Tomcat is not running when I look at the server after I can not access it. It runs perfectly for a couple hours or more and will restart just fine once I see that its down.

The funny thing is, this server ran fine for several months (un-used but running) and then one day a couple weeks ago when I started trying to test the server for my manager, it started crashing. All of my testing was done via the webpages, adding sites etc….no console changes or anything.

At first glance I thought the JVM might need more memory so I upped it to 8gb… And the behavior is the same…
# ps -ef | grep java
root     28064     1 37 10:51 pts/1    00:02:05 /opt/alfresco-4.2.c/java/bin/java -Djava.util.logging.config.file=/opt/a
lfresco-4.2.c/tomcat/conf/logging.properties -Xmx8G -XX:MaxPermSize=256m -Djava.util.logging.manager=org.apache.juli.Cla
ssLoaderLogManager -Djava.endorsed.dirs=/opt/alfresco-4.2.c/tomcat/endorsed -classpath /opt/alfresco-4.2.c/tomcat/bin/bo
otstrap.jar:/opt/alfresco-4.2.c/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/alfresco-4.2.c/tomcat -Dcatalina.home=/o
pt/alfresco-4.2.c/tomcat -Djava.io.tmpdir=/opt/alfresco-4.2.c/tomcat/temp org.apache.catalina.startup.Bootstrap start
root     28276 27968  0 10:56 pts/1    00:00:00 grep java

I have enabled debug logging in postgres to see if I can gather anything of value from there. Any help would be greatly appreciated. I love the tool but in this condition it is far from production ready….
I can provide logs/configs if any are relevant or needed


thanks,

Stephan

[andy]

This just looks like normal tomcat shutdown. Alfresco does not usually cause the container to shutdown.
Is there anything else in any of the tomcat logs?
Is there anythings else that can take out the process automatically on this machine?
Is there any pattern to the shutdown times in the logs?

Andy