Alfresco crashing during working hours

[hostman]

So, we are now experiencing this problem - Alfresco is very slow during working hours and finally crashes. Our setup is following:
Virtual server for applications (alfresco and share): 4 x 2.40 GHz, 9.5 GB RAM;
Database server: 4 x 2.40 GHz, 5GB RAM;
OS: Centos 6;
Alfresco version: Alfresco 4.2.e + Alvex Enterprise 2.1.2 plugin;
Users: <500;
Active users: 10-20;

<strong>Problem we are having</strong>
Alfresco documentation says, that with such server parameters we should be able to run server with 500 users or 50 concurrent.
During working hours, load average of server increases to ~30, CPU runs 100% and system becomes very slow. After running slow system crashed and starts throwing errors. I attach screenshot of server load:
[img=640x480]https://cdn-forums.alfresco.com/sites/forums/files/load_average.png[/img]

Those spikes in load average looks strange for me. When I am using the system when there is no load, I see 10-40% CPU utilization for 1 or 2 seconds after each click I make (home page, repository, my tasks, site page). If I press my completed tasks, there is 40% CPU utilization for about 2-3 seconds (I have only 135 tasks completed, other business users has a lot more tasks completed).
Some quotes from logs:

2016-02-11 09:17:16,038  INFO  [webscripts.connector.RemoteClient] [http-apr-8080-exec-314] Error status 408 Read timed out
 java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:150)
   at java.net.SocketInputStream.read(SocketInputStream.java:121)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
   at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
   at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
   at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
   at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
   at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:1054)
   at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:829)
   at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:577)
   at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:511)
   at org.springframework.extensions.webscripts.connector.HttpConnector.call(HttpConnector.java:70)
   at org.springframework.extensions.webscripts.RequestCachingConnector.call(RequestCachingConnector.java:90)
   at org.springframework.extensions.webscripts.connector.AbstractConnector.call(AbstractConnector.java:116)
   at org.springframework.extensions.webscripts.connector.AuthenticatingConnector.call(AuthenticatingConnector.java:92)
   at org.alfresco.web.scripts.UserPreferences.getValue(UserPreferences.java:66)
   at sun.reflect.GeneratedMethodAccessor1018.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
   at org.mozilla.javascript.JavaMembers.get(JavaMembers.java:117)
   at org.mozilla.javascript.NativeJavaObject.get(NativeJavaObject.java:113)
   at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:1544)
   at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1375)
   at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1364)
   at org.mozilla.javascript.gen.c5._c32(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1859)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108)
   at org.mozilla.javascript.gen.c5._c0(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1862)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
   at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.gen.c5.exec(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:318)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:192)
   at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:1305)
   at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
   at org.springframework.extensions.webscripts.PresentationContainer.executeScript(PresentationContainer.java:70)
   at org.springframework.extensions.webscripts.LocalWebScriptRuntimeContainer.executeScript(LocalWebScriptRuntimeContainer.java:236)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
   at org.springframework.extensions.webscripts.WebScriptProcessor.executeBody(WebScriptProcessor.java:310)
   at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
   at org.springframework.extensions.surf.render.RenderService.process(RenderService.java:599)
   at org.springframework.extensions.surf.render.RenderService.renderSubComponent(RenderService.java:505)
   at org.springframework.extensions.surf.render.RenderService.processComponent(RenderService.java:419)
   at org.springframework.extensions.surf.render.RenderService.renderComponent(RenderService.java:942)
   at org.springframework.extensions.surf.render.RenderService.renderRegionComponents(RenderService.java:900)
   at org.springframework.extensions.surf.render.bean.RegionRenderer.body(RegionRenderer.java:89)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.RenderService.renderRegion(RenderService.java:851)
   at org.springframework.extensions.directives.RegionDirectiveData.render(RegionDirectiveData.java:91)
   at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
   at freemarker.core.Environment.visit(Environment.java:274)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment$3.render(Environment.java:246)
   at org.springframework.extensions.surf.extensibility.impl.DefaultExtensibilityDirectiveData.render(DefaultExtensibilityDirectiveData.java:119)
   at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
   at freemarker.core.Environment.visit(Environment.java:274)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment.visit(Environment.java:406)
   at freemarker.core.BodyInstruction.accept(BodyInstruction.java:93)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Macro$Context.runMacro(Macro.java:172)
   at freemarker.core.Environment.visit(Environment.java:614)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:106)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment.process(Environment.java:199)
   at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.process(FTLTemplateProcessor.java:171)
   at org.springframework.extensions.webscripts.WebTemplateProcessor.executeBody(WebTemplateProcessor.java:438)
   at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
   at org.springframework.extensions.surf.render.RenderService.processTemplate(RenderService.java:721)
   at org.springframework.extensions.surf.render.bean.TemplateInstanceRenderer.body(TemplateInstanceRenderer.java:140)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.bean.PageRenderer.body(PageRenderer.java:85)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.RenderService.renderPage(RenderService.java:762)
   at org.springframework.extensions.surf.mvc.PageView.dispatchPage(PageView.java:411)
   at org.springframework.extensions.surf.mvc.PageView.renderView(PageView.java:306)
   at org.springframework.extensions.surf.mvc.AbstractWebFrameworkView.renderMergedOutputModel(AbstractWebFrameworkView.java:316)
   at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
   at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
   at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
   at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
   at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:74)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:168)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.CSRFFilter.doFilter(CSRFFilter.java:313)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:378)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1810)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:724)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

2016-02-11 09:22:34,322  ERROR [extensions.webscripts.AbstractRuntime] [http-apr-8080-exec-325] Exception from executeScript - redirecting to status template error: 01110055 Failed to execute script 'classpath*:alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js': 01110054 TypeError: Cannot read property "data" from null (jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37)
 org.springframework.extensions.webscripts.WebScriptException: 01110055 Failed to execute script 'classpath*:alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js': 01110054 TypeError: Cannot read property "data" from null (jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:200)
   at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:1305)
   at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
   at org.springframework.extensions.webscripts.PresentationContainer.executeScript(PresentationContainer.java:70)
   at org.springframework.extensions.webscripts.LocalWebScriptRuntimeContainer.executeScript(LocalWebScriptRuntimeContainer.java:236)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:378)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
   at org.springframework.extensions.webscripts.WebScriptProcessor.executeBody(WebScriptProcessor.java:310)
   at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
   at org.springframework.extensions.surf.render.RenderService.process(RenderService.java:599)
   at org.springframework.extensions.surf.render.RenderService.renderSubComponent(RenderService.java:505)
   at org.springframework.extensions.surf.render.RenderService.processComponent(RenderService.java:419)
   at org.springframework.extensions.surf.render.RenderService.renderComponent(RenderService.java:942)
   at org.springframework.extensions.surf.render.RenderService.renderRegionComponents(RenderService.java:900)
   at org.springframework.extensions.surf.render.bean.RegionRenderer.body(RegionRenderer.java:89)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.RenderService.renderRegion(RenderService.java:851)
   at org.springframework.extensions.directives.RegionDirectiveData.render(RegionDirectiveData.java:91)
   at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
   at freemarker.core.Environment.visit(Environment.java:274)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment$3.render(Environment.java:246)
   at org.springframework.extensions.surf.extensibility.impl.DefaultExtensibilityDirectiveData.render(DefaultExtensibilityDirectiveData.java:119)
   at org.springframework.extensions.surf.extensibility.impl.ExtensibilityModelImpl.merge(ExtensibilityModelImpl.java:408)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.merge(AbstractExtensibilityDirective.java:169)
   at org.springframework.extensions.surf.extensibility.impl.AbstractExtensibilityDirective.execute(AbstractExtensibilityDirective.java:137)
   at freemarker.core.Environment.visit(Environment.java:274)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:126)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment.visit(Environment.java:406)
   at freemarker.core.BodyInstruction.accept(BodyInstruction.java:93)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Macro$Context.runMacro(Macro.java:172)
   at freemarker.core.Environment.visit(Environment.java:614)
   at freemarker.core.UnifiedCall.accept(UnifiedCall.java:106)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.MixedContent.accept(MixedContent.java:92)
   at freemarker.core.Environment.visit(Environment.java:221)
   at freemarker.core.Environment.process(Environment.java:199)
   at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.process(FTLTemplateProcessor.java:171)
   at org.springframework.extensions.webscripts.WebTemplateProcessor.executeBody(WebTemplateProcessor.java:438)
   at org.springframework.extensions.surf.render.AbstractProcessor.execute(AbstractProcessor.java:57)
   at org.springframework.extensions.surf.render.RenderService.processTemplate(RenderService.java:721)
   at org.springframework.extensions.surf.render.bean.TemplateInstanceRenderer.body(TemplateInstanceRenderer.java:140)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.bean.PageRenderer.body(PageRenderer.java:85)
   at org.springframework.extensions.surf.render.AbstractRenderer.render(AbstractRenderer.java:77)
   at org.springframework.extensions.surf.render.RenderService.renderPage(RenderService.java:762)
   at org.springframework.extensions.surf.mvc.PageView.dispatchPage(PageView.java:411)
   at org.springframework.extensions.surf.mvc.PageView.renderView(PageView.java:306)
   at org.springframework.extensions.surf.mvc.AbstractWebFrameworkView.renderMergedOutputModel(AbstractWebFrameworkView.java:316)
   at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
   at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
   at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
   at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
   at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
   at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
   at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:74)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:168)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.CSRFFilter.doFilter(CSRFFilter.java:313)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:378)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
   at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
   at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1810)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:724)
Caused by: org.springframework.extensions.webscripts.WebScriptException: 01110054 TypeError: Cannot read property "data" from null (jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:328)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScript(JSScriptProcessor.java:192)
   … 106 more
Caused by: org.mozilla.javascript.EcmaError: TypeError: Cannot read property "data" from null (jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js#37)
   at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3350)
   at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3340)
   at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3356)
   at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3375)
   at org.mozilla.javascript.ScriptRuntime.undefReadError(ScriptRuntime.java:3388)
   at org.mozilla.javascript.ScriptRuntime.getObjectProp(ScriptRuntime.java:1362)
   at org.mozilla.javascript.gen.c5._c2(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:37)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108)
   at org.mozilla.javascript.gen.c5._c3(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:41)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108)
   at org.mozilla.javascript.gen.c5._c33(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:1871)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:108)
   at org.mozilla.javascript.gen.c5._c0(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js:2093)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
   at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
   at org.mozilla.javascript.gen.c5.call(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.mozilla.javascript.gen.c5.exec(jar:file:/opt/alfresco/tomcat/webapps/share/WEB-INF/lib/alvex-share-4.2.d-generic-ui-ee-share-1.16.jar!/alfresco/web-extension/site-webscripts/org/alfresco/share/header/share-header.get.js)
   at org.springframework.extensions.webscripts.processor.JSScriptProcessor.executeScriptImpl(JSScriptProcessor.java:318)
   … 107 more
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

After Alfresco restart everything works fine until again load average reaches value of ~30.

<strong>Solutions we have tried</strong>

We have tried several solutions, which were not very successful:
1. Different servers for alfresco application and database (worked for couple months);
2. Tuning JVM, but no results;
Current JVM settings:

JAVA_OPTS="-XX:MaxPermSize=256m -Xss1024K -Xms1G -Xmx8G -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true -server"‍

Do you have any other ideas which would be worth trying? Maybe you have any ideas how to identify which user actions are causing these spikes in load average?

[hostman]

I have been playing with parameters in ctl.sh for ages. None of the customizations gives good results. My /alfrescoinstall/tomcat/scripts/ctl.sh at the moment has:

…
export JAVA_OPTS="-XX:MaxPermSize=256m -Xss1024K -Xms7G -Xmx7G -XX:+DisableExplicitGC -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true -server"
…
‍‍‍‍‍

I didn't change anything in /alfrescoinstall/tomcat/bin/setenv.sh and it has:

JAVA_HOME=/opt/alfresco/java
JRE_HOME=/opt/alfresco/java
JAVA_OPTS="$JAVA_OPTS "
export JAVA_HOME
export JRE_HOME
export JAVA_OPTS
‍‍‍‍‍‍‍‍

[eswbitto]

Try putting your parameter here too and then restart alfresco completely. Any java changes (or anything for that matter) require a restart of the service.


JAVA_HOME=/opt/alfrescoinstall/java
JRE_HOME=$JAVA_HOME
JAVA_OPTS="-XX:-DisableExplicitGC -XX:ReservedCodeCacheSize=128m $JAVA_OPTS "
JAVA_OPTS="-XX:MaxPermSize=256M -Xms128M -Xmx4096M $JAVA_OPTS " # java-memory-settings
export JAVA_HOME
export JRE_HOME
export JAVA_OPTS

[douglascrp]

It seems you are using Alvex Enterprise.
I'm presuming this based on the log messages.

Why don't you try to contact them?

The error seems to be related with a header extension.
I couldn't find anything odd by looking at the community version's source though.