Alfresco opening same Lucene file over and over and never closing it.

[markd_paulson]

I recently upgraded the company website from 3.3g to 4.2f. We were having an issue getting search results beyond 1000 and it looked like upgrading would solve it. The good news is that we can query beyond 1000 and can delete the archived files all at once. The bad news is that the number of open files keeps increasing minute by minute, the vast majority of which are the same Lucene file:
java 27701 root 256r REG 8,1 17133704 2165528 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/1e50ad66-f781-4886-9bbc-b8d288e604aa/_0.cfs

I was getting a "too many open files" error shortly after install. I bumped the system setting up to over 10,000 and it worked for 3-4 days. This time it gave me no error, it just became unresponsive.

Any help figuring out what to look at would be greatly appreciated. My JSON strings also seem to get encoded twice somehow after the upgrade; either that or forward slashes started getting escaped and are not decoded later.

I only see two other issues going on: Head and Broken pipe. Neither seem to have much affect on things.

Head error:
ERROR: org.springframework.extensions.webscripts.AbstractRuntime - Exception from executeScript - redirecting to status template error: 00280428 Script url /rss does not support the method HEAD org.springframework.extensions.webscripts.WebScriptException: 00280428 Script url /rss does not support the method HEAD

Broken pipe:
WARN : org.springframework.extensions.webscripts.connector.RemoteClient - Exception during close() of HTTP API connection
ClientAbortException:  java.net.SocketException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:359)
        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:309)

Alfresco Global Properties:
###############################
## Common Alfresco Properties #
###############################

dir.root=/opt/alfresco-4.2.f/alf_data

alfresco.context=alfresco
alfresco.host=127.0.0.1
alfresco.port=80
alfresco.protocol=http

share.context=share
share.host=127.0.0.1
share.port=80
share.protocol=http

### database connection properties ###
db.driver=org.gjt.mm.mysql.Driver
db.username=alfresco
db.password=<OMITTED>
db.name=alfresco
db.url=jdbc:mysql://localhost/alfresco?useUnicode=yes&characterEncoding=UTF-8
db.pool.initial=10
db.pool.max=350
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect

### FTP Server Configuration ###
ftp.enabled=true
ftp.port=1121

### RMI service ports ###
alfresco.rmi.services.port=50500
avm.rmi.service.port=0
avmsync.rmi.service.port=0
attribute.rmi.service.port=0
authentication.rmi.service.port=0
repo.rmi.service.port=0
action.rmi.service.port=0
deployment.rmi.service.port=0

### External executable locations ###
ooo.exe=/opt/alfresco-4.2.f/libreoffice/program/soffice.bin
ooo.enabled=true
ooo.port=8100
img.root=/opt/alfresco-4.2.f/common
img.dyn=${img.root}/lib
img.exe=${img.root}/bin/convert
swf.exe=/opt/alfresco-4.2.f/common/bin/pdf2swf
swf.languagedir=/opt/alfresco-4.2.f/common/japanese

jodconverter.enabled=false
#jodconverter.officeHome=/opt/alfresco-4.2.f/libreoffice
#jodconverter.portNumbers=8100

### Initial admin password ###
alfresco_user_store.adminpassword=<OMITTED>

### E-mail site invitation setting ###
notification.email.siteinvite=false

### License location ###
dir.license.external=/opt/alfresco-4.2.f

### Solr indexing ###
#index.subsystem.name=solr
dir.keystore=${dir.root}/keystore
#solr.port.ssl=8443

### Lucene indexing ###

# The subsystem type value. Value is either solr or lucene.
index.subsystem.name=lucene

# The location for lucene index files
dir.indexes=/opt/alfresco-4.2.f/alf_data/lucene-indexes

# The location for index backups
dir.indexes.backup=/opt/alfresco-4.2.f/alf_data/backup-lucene-indexes

# Index recovery types; value can be NONE, VALIDATE, AUTO, FULL.
index.recovery.mode=AUTO

# Lucene
lucene.indexer.cacheEnabled=true
lucene.indexer.maxDocIdCacheSize=1000000
lucene.indexer.maxDocumentCacheSize=10000
lucene.indexer.maxParentCacheSize=1000000
#lucene.query.maxClauses=1000000

# The maximum number of search results to perform permission checks against
system.acl.maxPermissionChecks=10000

# The maximum time spent pruning results
system.acl.maxPermissionCheckTimeMillis=100000


### BPM Engine ###
system.workflow.engine.jbpm.enabled=false

#
# Outbound Email Configuration
#————-
mail.host=localhost
mail.port=25
mail.username=
mail.password=
mail.encoding=UTF-8
mail.from.default=alfresco@aosmd.com
mail.smtp.auth=false[/monospace]

<em>Java process:</em>
[monospace]/opt/alfresco-4.2.f/java/bin/java -Djava.util.logging.config.file=/opt/alfresco-4.2.f/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:MaxPermSize=4G -Xms1G -Xmx4G -XX:-DisableExplicitGC -XX:ReservedCodeCacheSize=128m -XX:+DisableExplicitGC -Djava.awt.headless=true -Dalfresco.home=/opt/alfresco-4.2.f -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true -XX:ReservedCodeCacheSize=128m -Djava.endorsed.dirs=/opt/alfresco-4.2.f/tomcat/endorsed -classpath /opt/alfresco-4.2.f/tomcat/bin/bootstrap.jar:/opt/alfresco-4.2.f/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/alfresco-4.2.f/tomcat -Dcatalina.home=/opt/alfresco-4.2.f/tomcat -Djava.io.tmpdir=/opt/alfresco-4.2.f/tomcat/temp org.apache.catalina.startup.Bootstrap start

I'm not sure where else to look for clues to what is wrong, but if I don't fix it soon, we will revert back to 3.3g.

[markd_paulson]

Since it was the same Lucene index file that was being reopened and never closed, I thought the indexes might be fouled up.
1) I shutdown Alfresco.
2) I changed alfresco-global.properties: index.recovery.mode=FULL
3) I deleted /opt/alfresco/alf_data/lucene-indexes directory.
4) I restarted Alfresco.
5) I changed alfresco-global.properties index.recovery.mode back to AUTO, since I don't want it to re-index on any subsequent restarts of Alfresco.

I don't have conclusive proof of what happened to foul up the index, but I have a theory. So, take it with a grain of salt:
I'm assuming that when I first started after the upgrade that, when the number of open files reached the system max per process (was set at 1024), something in the index got askew. So, even after I raised the linux system limit for open files, the index needed fixing.

[markd_paulson]

It ran fine for a while, but eventually it started grabbing the same file over and over again without closing it:
[root@www alfresco-4.2.f]# lsof -p 4117 | grep cfs | more
java    4117 root   52u   REG                8,1   27407689   7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs
java    4117 root  142u   REG                8,1   27407689   7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs
java    4117 root  143u   REG                8,1   27407689   7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs
java    4117 root  144r   REG                8,1   27407689   7242146 /opt/alfresco-4.2.f/alf_data/lucene-indexes/workspace/SpacesStore/6a3eb45a-befa-44dc-8e85-3008e832236c/_0.cfs

The listing just continues like this and increases as time goes on.

I haven't been able to reproduce on the test server, but have tried all options I can think of.

[douglascrp]

Hey Mark.

Please, check this page https://wiki.alfresco.com/wiki/Too_many_open_files

You have to configure your server to be able to handle them.