topic Re: alfresco 5.0.b restart taking long in Alfresco Archive

alfresco 5.0.b restart taking long

sharifu — Tue, 15 Sep 2015 11:59:53 GMT

After an upgrade from 4.2.x to 5.0.x community the restart is taking forever. In the logs I see java out of heap space etc. my settings as followsJAVA_HOME=/opt/alfresco/javaJRE_HOME=$JAVA_HOMEJAVA_OPTS="-XX:+DisableExplicitGC -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco -Dcom.sun.manageme

Re: alfresco 5.0.b restart taking long

afaust — Sat, 19 Sep 2015 19:11:45 GMT

Hello,

how did you (in your last 4.2 environment) arrive at those settings?

Even if you deploy Repository, Share and SOLR on the same Tomcat instance should you never need more than 512M, unless something is seriously wrong with customizations / addons in the system.

Setting ReservedCodeCacheSize is something that I have never heard/read of being relevant from anyone I worked with in the Alfresco community, and 2G also seem like an unreasonable amount. I have worked on Enterprise installations with up to 70.000 users, up to 20 distinct addon modules/business apps or up to 25 million documents, and the default code cache for JIT always sufficed.

Setting Xss to 512m means that every thread can have an associated stack of up to 512 MiB. By default, a Tomcat instance uses up to 200 HTTP connector threads and combined with background threads, this means Alfresco would use up to 250 threads. 250 * 512 MiB = 128 GiB of allowed stack - do you have that much RAM?
A Xss setting of 1024k is something I have seen floating around in Alfresco-related blogs and documentation, and even that is being quite generous. I am typically using 256k stacks (or not even setting any value at all, i.e. using the default). A Xss of 1024k or higher is only ever needed when there are badly written customizations / addons, e.g. with insane amounts of recursion or partially / non-working exit conditions to those recursions.

I am not saying those settings are "wrong" in any way, but I am honestly shocked by those values. It would really be interesting to know how you arrived at those settings. Maybe I am just being ignorant about some specific issues I have never heard of…

Some questions regarding to your problem:
Is there a reason you are not using Alfresco 5.0.d? There might have been issues in 5.0.b that have been fixed since then.
Also, how big is your SOLR index? Did you do a full index rebuild when switching to 5.0 (I have seen people trying to use the index from 4.2)? How many documents do you have in your repository, and how large are they in average?
Did you change anything in the SOLR configuration files related to the suggester (solrconfig.xml)?

Regards
Axel

Re: alfresco 5.0.b restart taking long

sharifu — Mon, 21 Sep 2015 11:49:00 GMT

Thos java settings I have been using since 3.4d community and it has been working well ever since. I took the default settings since 3.4d community and have been incrementing each option untill the server felt fast enough when browsing the share. I have 48GB RAM

I have been having problems upgrading to 5.0.d community so i am trying to do it in steps. from 4.2.f it was not working so i went to 5.0.b. I am in process of upgrading to 5.0.c community.

size of repository:

147M   alfresco.sql
189G   contentstore
4.0K   contentstore.deleted
30G   contentstore.tar.gz
60K   keystore
1.8M   oouser
43G   solr4
69G   solr4Backup
29G   solr4Backup.tar.gz

I have many files over thousands. average file can be between 2-5MB

the only thing I have changed in solr4 is alfresco.socketTimeout in solcore.properties. I have set this to 160000 as all previous version i get a tracking communication tmeout. I have set this for achive-spacestore and worspace-spacestore.

I also apply following fixes
https://issues.alfresco.com/jira/browse/MNT-12884
https://forums.alfresco.com/forum/installation-upgrades-configuration-integration/installation-upgrades/rhel-apr-based-apache

Re: alfresco 5.0.b restart taking long

sharifu — Mon, 21 Sep 2015 15:04:00 GMT

I have tried removing the Xss option and it is still taking too long to restart. below is the error from log

WARNING: [SetContextPropertiesRule]{Context} Setting property 'debug' to '0' did not find a matching property.
Sep 21, 2015 3:28:09 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
java.net.SocketTimeoutException: Read timed out
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(Unknown Source)
   at java.net.SocketInputStream.read(Unknown Source)
   at sun.security.ssl.InputRecord.readFully(Unknown Source)
   at sun.security.ssl.InputRecord.read(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
   at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
   at sun.security.ssl.AppOutputStream.write(Unknown Source)
   at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
   at java.io.BufferedOutputStream.flush(Unknown Source)
   at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
   at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
   at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:135)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:111)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:371)
   at org.alfresco.solr.client.SOLRAPIClient.getTextContent(SOLRAPIClient.java:1028)
   at org.alfresco.solr.SolrInformationServer.addContentPropertyToDocUsingAlfrescoRepository(SolrInformationServer.java:2185)
   at org.alfresco.solr.SolrInformationServer.addContentToCachedDoc(SolrInformationServer.java:2172)
   at org.alfresco.solr.SolrInformationServer.updateContentToIndexAndCache(SolrInformationServer.java:2129)
   at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:117)
   at org.alfresco.solr.tracker.AbstractWorkerRunnable.run(AbstractWorkerRunnable.java:40)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)
2015-09-21 15:43:20,538 ERROR [solr.core.SolrCore] [searcherExecutor-6-thread-1] null:java.lang.OutOfMemoryError: Java heap space
   at org.apache.lucene.util.packed.Packed64.<init>(Packed64.java:73)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1034)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
   at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
   at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
   at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:97)
   at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
   at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:197)
   at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:289)
   at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
   at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup.build(WFSTCompletionLookup.java:121)
   at org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)
   at org.apache.solr.spelling.suggest.SolrSuggester.build(SolrSuggester.java:143)
   at org.apache.solr.spelling.suggest.SolrSuggester.reload(SolrSuggester.java:170)
   at org.apache.solr.handler.component.SuggestComponent$SuggesterListener.newSearcher(SuggestComponent.java:466)
   at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1734)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

Sep 21, 2015 3:43:20 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /opt/alfresco-5.0.b/tomcat/webapps/alfresco.war

Re: alfresco 5.0.b restart taking long

afaust — Tue, 22 Sep 2015 18:04:41 GMT

Ok - so your system can still be considered small / normal sized.

Looking at the source code of SOLR, it appears that the error is caused by Java not being able to allocate a large long[] array in the available memory. It may be that your memory is sufficient globally, but the memory area where the array is to be allocated is not large enough to hold it.

Can you retry with the following additional JVM parameters?
-XX:NewRatio=1 -XXurvivorRatio=2

This should increase the size of the memory areas where new objects / arrays are created in and decrease the area for long-lived objects (all relative to each other and the overall size of the heap, Xms/Xmx). (see https://blogs.oracle.com/jonthecollector/entry/the_second_most_important_gc)
Since SOLR can create a lot of short-lived objects, it is beneficial to allow some breathing room for this (without choking of long-lived caches of course). I am using the above settings in most of the customer installation, unless the customer has already switched to a different memory management system / GC algorithm. We currently prefer G1GC over ConcMarkSweepGC because it does not have any kind of static relation between old/new/survivor memory areas and can better adapt to different system load scenarios.

Let us know if those different ratios help.

Regards
Axel

Re: alfresco 5.0.b restart taking long

sharifu — Wed, 30 Sep 2015 13:39:00 GMT

I havetried using the following and it is still taking over 20min to start up


export JAVA_OPTS="-server -XX:+DisableExplicitGC -Djava.awt.headless=true -Xms512M -Xmx16G -XX:MaxPermSize=256M -Dalfresco.home=/opt/alfresco -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dsun.security.ssl.allowUnsafeRenegotiation=true -XX:ReservedCodeCacheSize=128m -XX:NewRatio=1 -XX:SurvivorRatio=2"
‍‍‍



2015-09-30 08:36:25,197  ERROR [solr.core.SolrCore] [searcherExecutor-7-thread-1] null:java.lang.OutOfMemoryError: Java heap space
   at org.apache.lucene.util.packed.Packed64.<init>(Packed64.java:73)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1034)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
   at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
   at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
   at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:97)
   at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
   at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:197)
   at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:289)
   at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
   at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup.build(WFSTCompletionLookup.java:121)
   at org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)
   at org.apache.solr.spelling.suggest.SolrSuggester.build(SolrSuggester.java:143)
   at org.apache.solr.spelling.suggest.SolrSuggester.reload(SolrSuggester.java:170)
   at org.apache.solr.handler.component.SuggestComponent$SuggesterListener.newSearcher(SuggestComponent.java:466)
   at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1734)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

 2015-09-30 08:44:01,857  ERROR [solr.core.SolrCore] [searcherExecutor-6-thread-1] null:java.lang.OutOfMemoryError: Java heap space
   at org.apache.lucene.util.packed.Packed64.<init>(Packed64.java:73)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1034)
   at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:1001)
   at org.apache.lucene.util.packed.GrowableWriter.<init>(GrowableWriter.java:46)
   at org.apache.lucene.util.packed.PagedGrowableWriter.newMutable(PagedGrowableWriter.java:58)
   at org.apache.lucene.util.packed.AbstractPagedMutable.fillPages(AbstractPagedMutable.java:56)
   at org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:52)
   at org.apache.lucene.util.packed.PagedGrowableWriter.<init>(PagedGrowableWriter.java:45)
   at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:164)
   at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:197)
   at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:289)
   at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
   at org.apache.lucene.search.suggest.fst.WFSTCompletionLookup.build(WFSTCompletionLookup.java:121)
   at org.apache.lucene.search.suggest.Lookup.build(Lookup.java:190)
   at org.apache.solr.spelling.suggest.SolrSuggester.build(SolrSuggester.java:143)
   at org.apache.solr.spelling.suggest.SolrSuggester.reload(SolrSuggester.java:170)
   at org.apache.solr.handler.component.SuggestComponent$SuggesterListener.newSearcher(SuggestComponent.java:466)
   at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1734)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
   at java.lang.Thread.run(Unknown Source)

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: alfresco 5.0.b restart taking long

afaust — Thu, 01 Oct 2015 08:02:04 GMT

Can you please do a test with a deactivated suggester component to check if the delay / long duration (until the OOM) is only due to the suggester? Simply open the solrconfig.xml for solrHome/workspace-SpacesStore/conf and comment out the searchComponent "suggest" and the request handler "/suggest".

Apart from that, did you already monitor CPU / disk usage during the startup phase? Disk usage (transfer and wait times) can be interesting to determine if excessive or slow IO are an issue. Obviously, the OOM error is the final straw, but it could be of interest what happens leading up to that.

Regards
Axel

Re: alfresco 5.0.b restart taking long

sharifu — Tue, 06 Oct 2015 13:57:38 GMT

I am testing 5.0.d now with default settings with high Xmx and it seems to restart as normal now. once in production I shall let you know if this is causing issue still.