Solr Index Size Increase After Upgrade to Alfresco 6.2.0 and Alfresco Search services 1.4.1

brahmaiah926 — Tue, 01 Jun 2021 05:24:19 GMT

We have upgraded the lower environment recently from Alfresco 5.2 to 6.2 and solr4 to Alfresco search services 1.4.1. We have re-indexed solr after upgrade.

Now we have observed that the new index size with Search services 1.4.1 is around 55 GB where as in our same old environment the size was less than 10 GB. Adding both old and new configurations below. Kindly check and let us know what is the reason for additional index size.
Old Solr4 Configuration:
#
# solrcore.properties - used in solrconfig.xml
#
# data is in ${data.dir.root}/${data.dir.store}






data.dir.root=/var/lib/solr4/index
data.dir.store=workspace/SpacesStore
enable.alfresco.tracking=true

#
# Alfresco version
#
alfresco.version=5.2.0 (r@scm-revision@-b@build-number@)


#
# Properties loaded during alfresco tracking
#

alfresco.host=localhost
alfresco.port=8080
alfresco.port.ssl=8443
alfresco.baseUrl=/alfresco
alfresco.cron=0/15 * * * * ? *
alfresco.stores=workspace://SpacesStore
alfresco.index.transformContent=false
#alfresco.ignore.datatype.1=d:content
alfresco.lag=1000
alfresco.hole.retention=3600000
# alfresco.hole.check.after is not used yet
# It will reduce the hole checking load 
alfresco.hole.check.after=300000
alfresco.batch.count=1000
alfresco.recordUnindexedNodes=true

# encryption

# none, https
alfresco.secureComms=https

# ssl
alfresco.encryption.ssl.keystore.type=JCEKS
alfresco.encryption.ssl.keystore.provider=
alfresco.encryption.ssl.keystore.location=ssl.repo.client.keystore
alfresco.encryption.ssl.keystore.passwordFileLocation=ssl-keystore-passwords.properties
alfresco.encryption.ssl.truststore.type=JCEKS
alfresco.encryption.ssl.truststore.provider=
alfresco.encryption.ssl.truststore.location=ssl.repo.client.truststore
alfresco.encryption.ssl.truststore.passwordFileLocation=ssl-truststore-passwords.properties

# Tracking

alfresco.corePoolSize=20
alfresco.maximumPoolSize=-1
alfresco.keepAliveTime=120
alfresco.threadPriority=5
alfresco.threadDaemon=true
alfresco.workQueueSize=-1

# HTTP Client

alfresco.maxTotalConnections=200
alfresco.maxHostConnections=200
alfresco.socketTimeout=360000

# SOLR caching

solr.filterCache.size=256
solr.filterCache.initialSize=128
solr.queryResultCache.size=1024
solr.queryResultCache.initialSize=1024
solr.documentCache.size=1024
solr.documentCache.initialSize=1024
solr.queryResultMaxDocsCached=2048

solr.authorityCache.size=128
solr.authorityCache.initialSize=64
solr.pathCache.size=256
solr.pathCache.initialSize=128

solr.ownerCache.size=128
solr.ownerCache.initialSize=64

solr.readerCache.size=128
solr.readerCache.initialSize=64

solr.deniedCache.size=128
solr.deniedCache.initialSize=64

# SOLR

solr.maxBooleanClauses=10000

# Batch fetch

alfresco.transactionDocsBatchSize=500
alfresco.nodeBatchSize=100
alfresco.changeSetAclsBatchSize=500
alfresco.aclBatchSize=100
alfresco.contentReadBatchSize=4000
alfresco.contentUpdateBatchSize=1000

# Warming

solr.filterCache.autowarmCount=32
solr.authorityCache.autowarmCount=4
solr.pathCache.autowarmCount=32
solr.deniedCache.autowarmCount=0
solr.readerCache.autowarmCount=0
solr.ownerCache.autowarmCount=0
solr.queryResultCache.autowarmCount=4
solr.documentCache.autowarmCount=512

solr.queryResultWindowSize=512


#
# TODO
#
# cross language support
# locale expansion
# logging check report ....
#
# 

alfresco.doPermissionChecks=true

# Evaluate permissions as a PostFilter. Setting this to false will evaluate permissions as a Query.

alfresco.postfilter=true

#
# Metadata pulling control
#
alfresco.metadata.skipDescendantDocsForSpecificTypes=false
alfresco.metadata.ignore.datatype.0=cm:person
alfresco.metadata.ignore.datatype.1=app:configurations
alfresco.metadata.skipDescendantDocsForSpecificAspects=false
#alfresco.metadata.ignore.aspect.0=

alfresco.topTermSpanRewriteLimit=1000

#
# Suggestions
#
solr.suggester.enabled=false
# -1 to disable suggester build throttling
solr.suggester.minSecsBetweenBuilds=3600

#
# Limit the maximum text size of transformed content sent to the index - in bytes
#
alfresco.contentStreamLimit=10000000
solr.query.maximumResultsFromUnlimitedQuery=100000

--------------------------------------------------------------------
New Alfresco Search Services 1.4.1 Solr Configuration:
#Mon Apr 20 11:43:19 UTC 2020
solr.authorityCache.initialSize=64
alfresco.encryption.ssl.truststore.location=ssl.repo.client.truststore
solr.suggester.minSecsBetweenBuilds=3600
solr.filterCache.size=256
alfresco.batch.count=5000
alfresco.contentReadBatchSize=100
alfresco.corePoolSize=8
data.dir.root=/opt/alfresco-search-services/solrhome/
alfresco.encryption.ssl.keystore.provider=
alfresco.nodeBatchSize=100
alfresco.template=rerank
solr.pathCache.initialSize=128
alfresco.encryption.ssl.truststore.type=JCEKS
alfresco.host=xxx.amazonaws.com
alfresco.lag=1000
alfresco.maxTotalConnections=200
alfresco.encryption.ssl.keystore.location=ssl.repo.client.keystore
alfresco.encryption.ssl.truststore.provider=
alfresco.topTermSpanRewriteLimit=1000
alfresco.port.ssl=8443
alfresco.contentStreamLimit=10000000
solr.filterCache.initialSize=128
alfresco.changeSetAclsBatchSize=500
solr.ownerCache.initialSize=64
solr.suggester.enabled=false
alfresco.cron=0/10 * * * * ? *
alfresco.commitInterval=2000
data.dir.store=alfresco
alfresco.encryption.ssl.truststore.passwordFileLocation=ssl-truststore-passwords.properties
solr.queryResultCache.initialSize=1024
solr.readerCache.autowarmCount=0
alfresco.threadDaemon=true
alfresco.newSearcherInterval=3000
solr.pathCache.size=256
alfresco.recordUnindexedNodes=false
alfresco.doPermissionChecks=true
solr.authorityCache.autowarmCount=4
solr.ownerCache.size=128
alfresco.metadata.skipDescendantDocsForSpecificTypes=false
alfresco.port=8080
alfresco.keepAliveTime=120
solr.documentCache.autowarmCount=512
solr.queryResultCache.size=1024
enable.alfresco.tracking=true
alfresco.workQueueSize=-1
solr.ownerCache.autowarmCount=0
solr.documentCache.size=1024
alfresco.hole.retention=3600000
alfresco.contentUpdateBatchSize=1000
alfresco.encryption.ssl.keystore.type=JCEKS
solr.queryResultMaxDocsCached=2048
alfresco.threadPriority=5
alfresco.baseUrl=/alfresco
solr.deniedCache.initialSize=64
solr.pathCache.autowarmCount=32
alfresco.socketTimeout=360000
solr.authorityCache.size=128
solr.readerCache.size=128
solr.filterCache.autowarmCount=32
alfresco.postfilter=true
alfresco.secureComms=https
solr.readerCache.initialSize=64
solr.maxBooleanClauses=10000
alfresco.metadata.ignore.datatype.1=app\:configurations
alfresco.metadata.ignore.datatype.0=cm\:person
alfresco.stores=workspace\://SpacesStore
solr.deniedCache.size=128
alfresco.aclBatchSize=100
solr.queryResultWindowSize=512
alfresco.hole.check.after=300000
solr.documentCache.initialSize=1024
shard.method=DB_ID
alfresco.metadata.skipDescendantDocsForSpecificAspects=false
alfresco.maxHostConnections=200
solr.deniedCache.autowarmCount=0
alfresco.encryption.ssl.keystore.passwordFileLocation=ssl-keystore-passwords.properties
alfresco.maximumPoolSize=-1
solr.queryResultCache.autowarmCount=4
alfresco.transactionDocsBatchSize=500

I can seen in the new environment that the index size of alfresco core is increasing drastically without adding any files into Alfresco repository.

The index size was around 50 GB on thursday which has became 97 GB now(Monday).

topic Solr Index Size Increase After Upgrade to Alfresco 6.2.0 and Alfresco Search services 1.4.1 in Alfresco Forum

Solr Index Size Increase After Upgrade to Alfresco 6.2.0 and Alfresco Search services 1.4.1