topic Alfresco opening same Lucene file over and over and never closing it. in Alfresco Archive https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282339#M235469 <HTML><HEAD></HEAD><BODY><SPAN>I recently upgraded the company website from 3.3g to 4.2f. We were having an issue getting search results beyond 1000 and it looked like upgrading would solve it. The good news is that we can query beyond 1000 and can delete the archived files all at once. The bad news is that the number of open files keeps increasing minute by minute, the vast majority of which are the same Lucene file:</SPAN><BR /><SPAN>java 27701 root 256r REG 8,1 17133704 2165528 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/1e50ad66-f781-4886-9bbc-b8d288e604aa/_0.cfs</SPAN><BR /><BR /><SPAN>I was getting a "too many open files" error shortly after install. I bumped the system setting up to over 10,000 and it worked for 3-4 days. This time it gave me no error, it just became unresponsive.</SPAN><BR /><BR /><SPAN>Any help figuring out what to look at would be greatly appreciated. My JSON strings also seem to get encoded twice somehow after the upgrade; either that or forward slashes started getting escaped and are not decoded later.</SPAN><BR /><BR /><SPAN>I only see two other issues going on: Head and Broken pipe. Neither seem to have much affect on things.</SPAN><BR /><BR /><SPAN>Head error:</SPAN><BR /><SPAN>ERROR: org.springframework.extensions.webscripts.AbstractRuntime - Exception from executeScript - redirecting to status template error: 00280428 Script url /rss does not support the method HEAD org.springframework.extensions.webscripts.WebScriptException: 00280428 Script url /rss does not support the method HEAD</SPAN><BR /><BR /><SPAN>Broken pipe:</SPAN><BR /><SPAN>WARN : org.springframework.extensions.webscripts.connector.RemoteClient - Exception during close() of HTTP API connection</SPAN><BR /><SPAN>ClientAbortException:&nbsp; java.net.SocketException: Broken pipe</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:359)</SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:309)</SPAN><BR /><BR /><SPAN>Alfresco Global Properties:</SPAN><BR /><SPAN>###############################</SPAN><BR /><SPAN>## Common Alfresco Properties #</SPAN><BR /><SPAN>###############################</SPAN><BR /><BR /><SPAN>dir.root=/opt/alfresco-4.2.f/alf_data</SPAN><BR /><BR /><SPAN>alfresco.context=alfresco</SPAN><BR /><SPAN>alfresco.host=127.0.0.1</SPAN><BR /><SPAN>alfresco.port=80</SPAN><BR /><SPAN>alfresco.protocol=http</SPAN><BR /><BR /><SPAN>share.context=share</SPAN><BR /><SPAN>share.host=127.0.0.1</SPAN><BR /><SPAN>share.port=80</SPAN><BR /><SPAN>share.protocol=http</SPAN><BR /><BR /><SPAN>### database connection properties ###</SPAN><BR /><SPAN>db.driver=org.gjt.mm.mysql.Driver</SPAN><BR /><SPAN>db.username=alfresco</SPAN><BR /><SPAN>db.password=&lt;OMITTED&gt;</SPAN><BR /><SPAN>db.name=alfresco</SPAN><BR /><SPAN>db.url=jdbc:mysql://localhost/alfresco?useUnicode=yes&amp;characterEncoding=UTF-8</SPAN><BR /><SPAN>db.pool.initial=10</SPAN><BR /><SPAN>db.pool.max=350</SPAN><BR /><SPAN>hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect</SPAN><BR /><BR /><SPAN>### FTP Server Configuration ###</SPAN><BR /><SPAN>ftp.enabled=true</SPAN><BR /><SPAN>ftp.port=1121</SPAN><BR /><BR /><SPAN>### RMI service ports ###</SPAN><BR /><SPAN>alfresco.rmi.services.port=50500</SPAN><BR /><SPAN>avm.rmi.service.port=0</SPAN><BR /><SPAN>avmsync.rmi.service.port=0</SPAN><BR /><SPAN>attribute.rmi.service.port=0</SPAN><BR /><SPAN>authentication.rmi.service.port=0</SPAN><BR /><SPAN>repo.rmi.service.port=0</SPAN><BR /><SPAN>action.rmi.service.port=0</SPAN><BR /><SPAN>deployment.rmi.service.port=0</SPAN><BR /><BR /><SPAN>### External executable locations ###</SPAN><BR /><SPAN>ooo.exe=/opt/alfresco-4.2.f/libreoffice/program/soffice.bin</SPAN><BR /><SPAN>ooo.enabled=true</SPAN><BR /><SPAN>ooo.port=8100</SPAN><BR /><SPAN>img.root=/opt/alfresco-4.2.f/common</SPAN><BR /><SPAN>img.dyn=${img.root}/lib</SPAN><BR /><SPAN>img.exe=${img.root}/bin/convert</SPAN><BR /><SPAN>swf.exe=/opt/alfresco-4.2.f/common/bin/pdf2swf</SPAN><BR /><SPAN>swf.languagedir=/opt/alfresco-4.2.f/common/japanese</SPAN><BR /><BR /><SPAN>jodconverter.enabled=false</SPAN><BR /><SPAN>#jodconverter.officeHome=/opt/alfresco-4.2.f/libreoffice</SPAN><BR /><SPAN>#jodconverter.portNumbers=8100</SPAN><BR /><BR /><SPAN>### Initial admin password ###</SPAN><BR /><SPAN>alfresco_user_store.adminpassword=&lt;OMITTED&gt;</SPAN><BR /><BR /><SPAN>### E-mail site invitation setting ###</SPAN><BR /><SPAN>notification.email.siteinvite=false</SPAN><BR /><BR /><SPAN>### License location ###</SPAN><BR /><SPAN>dir.license.external=/opt/alfresco-4.2.f</SPAN><BR /><BR /><SPAN>### Solr indexing ###</SPAN><BR /><SPAN>#index.subsystem.name=solr</SPAN><BR /><SPAN>dir.keystore=${dir.root}/keystore</SPAN><BR /><SPAN>#solr.port.ssl=8443</SPAN><BR /><BR /><SPAN>### Lucene indexing ###</SPAN><BR /><BR /><SPAN># The subsystem type value. Value is either solr or lucene.</SPAN><BR /><SPAN>index.subsystem.name=lucene</SPAN><BR /><BR /><SPAN># The location for lucene index files</SPAN><BR /><SPAN>dir.indexes=/opt/alfresco-4.2.f/alf_data/lucene-indexes</SPAN><BR /><BR /><SPAN># The location for index backups</SPAN><BR /><SPAN>dir.indexes.backup=/opt/alfresco-4.2.f/alf_data/backup-lucene-indexes</SPAN><BR /><BR /><SPAN># Index recovery types; value can be NONE, VALIDATE, AUTO, FULL.</SPAN><BR /><SPAN>index.recovery.mode=AUTO</SPAN><BR /><BR /><SPAN># Lucene</SPAN><BR /><SPAN>lucene.indexer.cacheEnabled=true</SPAN><BR /><SPAN>lucene.indexer.maxDocIdCacheSize=1000000</SPAN><BR /><SPAN>lucene.indexer.maxDocumentCacheSize=10000</SPAN><BR /><SPAN>lucene.indexer.maxParentCacheSize=1000000</SPAN><BR /><SPAN>#lucene.query.maxClauses=1000000</SPAN><BR /><BR /><SPAN># The maximum number of search results to perform permission checks against</SPAN><BR /><SPAN>system.acl.maxPermissionChecks=10000</SPAN><BR /><BR /><SPAN># The maximum time spent pruning results</SPAN><BR /><SPAN>system.acl.maxPermissionCheckTimeMillis=100000</SPAN><BR /><BR /><BR /><SPAN>### BPM Engine ###</SPAN><BR /><SPAN>system.workflow.engine.jbpm.enabled=false</SPAN><BR /><BR /><SPAN>#</SPAN><BR /><SPAN># Outbound Email Configuration</SPAN><BR /><SPAN>#————-</SPAN><BR /><SPAN>mail.host=localhost</SPAN><BR /><SPAN>mail.port=25</SPAN><BR /><SPAN>mail.username=</SPAN><BR /><SPAN>mail.password=</SPAN><BR /><SPAN>mail.encoding=UTF-8</SPAN><BR /><SPAN><A class="jive-link-email-small" href="mailto:mail.from.default=alfresco@aosmd.com" rel="nofollow noopener noreferrer">mail.from.default=alfresco@aosmd.com</A></SPAN><BR /><SPAN>mail.smtp.auth=false[/monospace]</SPAN><BR /><BR /><SPAN>&lt;em&gt;Java process:&lt;/em&gt;</SPAN><BR /><SPAN>[monospace]/opt/alfresco-4.2.f/java/bin/java -Djava.util.logging.config.file=/opt/alfresco-4.2.f/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:MaxPermSize=4G -Xms1G -Xmx4G -XX:-DisableExplicitGC -XX:ReservedCodeCacheSize=128m -XX:+DisableExplicitGC -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco-4.2.f -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true -XX:ReservedCodeCacheSize=128m -Djava.endorsed.dirs=/opt/alfresco-4.2.f/tomcat/endorsed -classpath /opt/alfresco-4.2.f/tomcat/bin/bootstrap.jar:/opt/alfresco-4.2.f/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/alfresco-4.2.f/tomcat -Dcatalina.home=/opt/alfresco-4.2.f/tomcat -Djava.io.tmpdir=/opt/alfresco-4.2.f/tomcat/temp org.apache.catalina.startup.Bootstrap start</SPAN><BR /><BR /><SPAN>I'm not sure where else to look for clues to what is wrong, but if I don't fix it soon, we will revert back to 3.3g.</SPAN></BODY></HTML> Wed, 27 Jan 2016 20:28:23 GMT markd_paulson 2016-01-27T20:28:23Z Alfresco opening same Lucene file over and over and never closing it. https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282339#M235469 I recently upgraded the company website from 3.3g to 4.2f. We were having an issue getting search results beyond 1000 and it looked like upgrading would solve it. The good news is that we can query beyond 1000 and can delete the archived files all at once. The bad news is that the number of open fil Wed, 27 Jan 2016 20:28:23 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282339#M235469 markd_paulson 2016-01-27T20:28:23Z Re: Alfresco opening same Lucene file over and over and never closing it. https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282340#M235470 <HTML><HEAD></HEAD><BODY><SPAN>Since it was the same Lucene index file that was being reopened and never closed, I thought the indexes might be fouled up.</SPAN><BR /><SPAN>1) I shutdown Alfresco.</SPAN><BR /><SPAN>2) I changed alfresco-global.properties: index.recovery.mode=FULL</SPAN><BR /><SPAN>3) I deleted /opt/alfresco/alf_data/lucene-indexes directory.</SPAN><BR /><SPAN>4) I restarted Alfresco.</SPAN><BR /><SPAN>5) I changed alfresco-global.properties index.recovery.mode back to AUTO, since I don't want it to re-index on any subsequent restarts of Alfresco.</SPAN><BR /><BR /><SPAN>I don't have conclusive proof of what happened to foul up the index, but I have a theory. So, take it with a grain of salt:</SPAN><BR /><SPAN>I'm assuming that when I first started after the upgrade that, when the number of open files reached the system max per process (was set at 1024), something in the index got askew. So, even after I raised the linux system limit for open files, the index needed fixing.</SPAN></BODY></HTML> Fri, 29 Jan 2016 18:33:24 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282340#M235470 markd_paulson 2016-01-29T18:33:24Z Re: Alfresco opening same Lucene file over and over and never closing it. https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282341#M235471 <HTML><HEAD></HEAD><BODY><SPAN>It ran fine for a while, but eventually it started grabbing the same file over and over again without closing it:</SPAN><BR /><SPAN>[root@www alfresco-4.2.f]# lsof -p 4117 | grep cfs | more</SPAN><BR /><SPAN>java&nbsp;&nbsp;&nbsp; 4117 root&nbsp;&nbsp; 52u&nbsp;&nbsp; REG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8,1&nbsp;&nbsp; 27407689&nbsp;&nbsp; 7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs</SPAN><BR /><SPAN>java&nbsp;&nbsp;&nbsp; 4117 root&nbsp; 142u&nbsp;&nbsp; REG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8,1&nbsp;&nbsp; 27407689&nbsp;&nbsp; 7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs</SPAN><BR /><SPAN>java&nbsp;&nbsp;&nbsp; 4117 root&nbsp; 143u&nbsp;&nbsp; REG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8,1&nbsp;&nbsp; 27407689&nbsp;&nbsp; 7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs</SPAN><BR /><SPAN>java&nbsp;&nbsp;&nbsp; 4117 root&nbsp; 144r&nbsp;&nbsp; REG&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 8,1&nbsp;&nbsp; 27407689&nbsp;&nbsp; 7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs</SPAN><BR /><BR /><SPAN>The listing just continues like this and increases as time goes on.</SPAN><BR /><BR /><SPAN>I haven't been able to reproduce on the test server, but have tried all options I can think of.</SPAN></BODY></HTML> Fri, 29 Jan 2016 21:59:07 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282341#M235471 markd_paulson 2016-01-29T21:59:07Z Re: Alfresco opening same Lucene file over and over and never closing it. https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282342#M235472 <HTML><HEAD></HEAD><BODY><SPAN>Hey Mark.</SPAN><BR /><BR /><SPAN>Please, check this page </SPAN><A href="https://wiki.alfresco.com/wiki/Too_many_open_files" rel="nofollow noopener noreferrer">https://wiki.alfresco.com/wiki/Too_many_open_files</A><BR /><BR /><SPAN>You have to configure your server to be able to handle them.</SPAN></BODY></HTML> Tue, 16 Feb 2016 17:58:02 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-opening-same-lucene-file-over-and-over-and-never/m-p/282342#M235472 douglascrp 2016-02-16T17:58:02Z