topic Re: Alfresc indexing slow due to transformation in Alfresco Archive

Alfresc indexing slow due to transformation

dmorozov — Thu, 12 May 2011 22:14:24 GMT

Hello,I have been fighting last week with Alfresco going terribly slow because (I think) of Tika transformations happening in background.Please provide an advice how to solve this issue.We have Alfresco 3.4.d installed on Ubuntu 64 bit server.RAM: 16GCPU: 4JVM settings: -Djava.awt.headless=true -ser

Re: Alfresc indexing slow due to transformation

dmorozov — Thu, 12 May 2011 22:19:45 GMT

By the way we have enough free space on hard drive so this is not an issue.

Additionally I tried to run Alfresco with 2G max heap size configured and saw exactly the same issue.
After some time (5-15 min) memory was pumped up to max but it was still handling all http requests properly.
And eventually it slowed down and hanged. CPU utilization showed with JConsole was 100% and it caused by GC called too often trying to release memory.

We have small amount of users working with Alfresco (10-20 maximum).
We have reporting tool traversing JCR tree to generate report but JProfiler didn't show any memory leaks related that.

Re: Alfresc indexing slow due to transformation

vmm — Fri, 13 May 2011 10:10:55 GMT

Hi,

we have the same problem with 3.4d.

For us the problem is caused when Alfresco tries to index a .xlsx file that have one million rows.

the threaddump:

"schedulerFactory_Worker-4" - Thread t@30
   java.lang.Thread.State: RUNNABLE
   at java.lang.ClassLoader.defineClass1(Native Method)
   at java.lang.ClassLoader.defineClassCond(Unknown Source)
   at java.lang.ClassLoader.defineClass(Unknown Source)
   at java.security.SecureClassLoader.defineClass(Unknown Source)
   at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2733)
   - locked org.apache.catalina.loader.WebappClassLoader@1dec1dd
   at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1124)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1612)
   - locked org.apache.catalina.loader.WebappClassLoader@1dec1dd
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Unknown Source)
   at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.getJavaImplClass(SchemaTypeImpl.java:1709)
   at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.getJavaImplConstructor(SchemaTypeImpl.java:1725)
   at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createUnattachedNode(SchemaTypeImpl.java:1853)
   at org.apache.xmlbeans.impl.schema.SchemaTypeImpl.createElementType(SchemaTypeImpl.java:1021)
   at org.apache.xmlbeans.impl.values.XmlObjectBase.create_element_user(XmlObjectBase.java:893)
   at org.apache.xmlbeans.impl.store.Xobj.getUser(Xobj.java:1657)
   at org.apache.xmlbeans.impl.store.Xobj.find_element_user(Xobj.java:2062)
   at org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTRowImpl.getCArray(Unknown Source)
   - locked org.apache.xmlbeans.impl.store.Locale@3f3b50
   at org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTRowImpl$1CList.get(Unknown Source)
   at org.openxmlformats.schemas.spreadsheetml.x2006.main.impl.CTRowImpl$1CList.get(Unknown Source)
   at java.util.AbstractList$Itr.next(Unknown Source)
   at org.apache.poi.xssf.usermodel.XSSFRow.<init>(XSSFRow.java:66)
   at org.apache.poi.xssf.usermodel.XSSFSheet.initRows(XSSFSheet.java:178)
   at org.apache.poi.xssf.usermodel.XSSFSheet.read(XSSFSheet.java:147)
   at org.apache.poi.xssf.usermodel.XSSFSheet.onDocumentRead(XSSFSheet.java:134)
   at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:234)
   at org.apache.poi.POIXMLDocument.load(POIXMLDocument.java:190)
   at org.apache.poi.xssf.usermodel.XSSFWorkbook.<init>(XSSFWorkbook.java:182)
   at org.apache.poi.xssf.extractor.XSSFExcelExtractor.<init>(XSSFExcelExtractor.java:56)
   at org.apache.poi.extractor.ExtractorFactory.createExtractor(ExtractorFactory.java:172)
   at org.apache.poi.extractor.ExtractorFactory.createExtractor(ExtractorFactory.java:152)
   at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:65)
   at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:68)
   at org.alfresco.repo.content.TikaOfficeDetectParser.parse(TikaOfficeDetectParser.java:78)
   at org.alfresco.repo.content.transform.TikaPoweredContentTransformer.transformInternal(TikaPoweredContentTransformer.java:185)
   at org.alfresco.repo.content.transform.AbstractContentTransformer2.transform(AbstractContentTransformer2.java:161)
   at org.alfresco.repo.content.transform.AbstractContentTransformer2.transform(AbstractContentTransformer2.java:137)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.indexProperty(ADMLuceneIndexerImpl.java:944)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:620)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:585)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1580)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:217)
   at sun.reflect.GeneratedMethodAccessor406.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
   at java.lang.reflect.Method.invoke(Unknown Source)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy82.index(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)

Re: Alfresc indexing slow due to transformation

loftux — Fri, 13 May 2011 10:39:17 GMT

I'm not aware of a way to prevent indexing of a specific file, but if you know what file it is, put a password on the file. That way transformation will fail (rather than halting the whole server). If needed, write the password in the description field so that those who needs to open can do so.

It would just be a workaround, and not a terribly good one, just thought that it may be worth having as an option.

Another would be to disable content transformations for all excel files until properly resolved. Have a look in content-services-context.xml to find out what beans to change/disable. Maybe comment out the bean <bean id="transformer.Poi", that way I think it will user OpenOffice instead. But you have to try this out.

Re: Alfresc indexing slow due to transformation

mrogers — Fri, 13 May 2011 12:49:34 GMT

I've had a chat with our resident Tika Guru who pointed out that this sounds like TIKA-521 which was fixed late last year.

So you may like to try the latest version of Tika.

Re: Alfresc indexing slow due to transformation

vmm — Mon, 16 May 2011 12:37:26 GMT

We tried with tika 0.9 and had the same problem.

For the moment we have comment the mapping for xlsx and xls files in mimetype-map.xml, so Alfresco stores excel files like binarys.

Re: Alfresc indexing slow due to transformation

mrogers — Mon, 16 May 2011 13:29:37 GMT

You need version 1.0 or above…

Re: Alfresc indexing slow due to transformation

dmorozov — Fri, 20 May 2011 18:40:31 GMT

Thank you very much it helped.

So here is solution for others:
1. Checkout Tika sources trunk (google for it)
2. Build.

Notes: It will create tika 1.0 snapshot version.
Only one issue I got with compilation is missed jdom artifact in tika-parsers submodule. Just add this dependency into tika-parsers/pom.xml and build.

Additionally make sure that you will update all dependent libraries. For example it brings new versions of Apache POI (3.8-beta2) and PDFBox (1.5) libraries.
In my case I just created empty web application with maven and put tika-core and tika-parsers as dependencies. Maven will collect all required libs for you.
Then you will just need to make sure that in your Alfresco you have correct versions. Add extra libraries resolved by maven just in case.

After patching Tika I able to run my Alfresco with 2G heap size configured and average memory usage is about 600M with jumps up to 2G while documents re-indexing.

Re: Alfresc indexing slow due to transformation

promethius — Tue, 07 Feb 2012 19:35:01 GMT

I am having the exact same issue, Could you give me any step by step instructions on how you fixed this. Thanks.

Re: Alfresc indexing slow due to transformation

mrogers — Tue, 07 Feb 2012 20:44:07 GMT

Just upgrade to 4.0.

Re: Alfresc indexing slow due to transformation

ebogaard — Mon, 12 Mar 2012 11:36:35 GMT

We have the same problem with 3.4d, and an upgrade to 4.0 isn't possible yet.
Because the worker hogs so much cpu, other problems arise.
Is there any way to sum up all the new .JARs that need to be installed for the indexing to work properly again and stop it locking up?