topic PDF FullText Indexing Alfresco indexer ignores blanks/spaces in Alfresco Archive

PDF FullText Indexing Alfresco indexer ignores blanks/spaces

gsx — Mon, 01 Jun 2009 09:58:49 GMT

Hi all,I have a question about the PDF indexing functionality in Alfresco. I have many pdf files that have been created by using a document scanner (Fujitsu ScanSnap S510) which is simply scanning documents, doing OCR and creating PDF files. The software suite of Fujitsu is using ABBYY Tools for thi

Re: PDF FullText Indexing Alfresco indexer ignores blanks/spaces

gsx — Mon, 01 Jun 2009 10:50:19 GMT

I was searching for the cause of this problem for a long time. I think I finally know where the issue is coming from. After finding out that Alfresco is using PDFBox to extract the text from PDF files I've searched in the bug reports of PDFBox and found many entries for exactly the same problem. As I don't believe that this issue will be resolved soon (the issue is reported since 3 years now…) I would like to switch to a different transformation/extraction tool and rebuild the index in Alfresco from scratch.

Is this approach possible somehow? Does someone know good alternatives to PDFBox that are working together with Alfresco?

BTW: you can find one of the reported issues in PDFBox here: http://sourceforge.net/tracker/index.php?func=detail&aid=1922502&group_id=78314&atid=552832

You can test this behaviour with following PDF document: http://sourceforge.net/tracker/download.php?group_id=78314&atid=552832&file_id=271548&aid=1922502

I would appreaciate any ideas. Thanks!

Re: PDF FullText Indexing Alfresco indexer ignores blanks/spaces

jitse — Wed, 15 Jul 2009 09:37:34 GMT

Hi, you could change the pdfbox transformer (doing the conversion to plain text) to pdftotext (fromhttp://www.foolabs.com/xpdf/).
You can activate this transformer by http://wiki.alfresco.com/wiki/Content_Transformations following the wiki.
You should add a configuration somewhat like this (depending on where you place your totext-tool)


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
        <!– disable standard pdfbox text transformer –>
        <bean id="transformer.PdfBox" class="java.lang.String"/>
        <!– has the above injected, is newly created below –>
    <bean id="transformer.complex.OpenOffice.PdfBox" class="java.lang.String"/>
 
        <!– pdftotext command line binary –>
        <bean id="transformer.PdfToTextTool"
                class="org.alfresco.repo.content.transform.RuntimeExecutableContentTransformer"
                parent="baseContentTransformer">
                <property name="transformCommand">
                        <bean name="transformer.pdftotext.Command"
                                class="org.alfresco.util.exec.RuntimeExec">
                                <property name="commandMap">
                                        <map>
                                                <entry key="Linux.*">
                                                        <!–<value>${catalina.base}/webapps/alfresco/WEB-INF/bin/pdftotext-linux -enc UTF-8 ${options} ${source} ${target}</value>–>
                                                        <value>/usr/bin/pdftotext -enc UTF-8 ${options} ${source} ${target}</value>
                                                </entry>
                                                <entry key="Windows.*">
                                                        <value>${catalina.base}/webapps/alfresco/WEB-INF/bin/pdftotext-win32.exe -enc UTF-8 ${options} ${source} ${target}</value>
                                                </entry>
                                        </map>
                                </property>
                                <property name="defaultProperties">
                                        <props>
                                                <prop key="options"></prop>
                                        </props>
                                </property>
                        </bean>
                </property>
                <property name="explicitTransformations">
                        <list>
                                <!–<bean
                                        class="org.alfresco.repo.content.transform.ContentTransformerRegistry$TransformationKey">
                                        <constructor-arg>
                                                <value>application/pdf</value>
                                        </constructor-arg>
                                        <constructor-arg>
                                                <value>text/plain</value>
                                        </constructor-arg>
                                </bean>–>
                     <bean class="org.alfresco.repo.content.transform.ExplictTransformationDetails" >
                         <property name="sourceMimetype"><value>application/pdf</value></property>
                         <property name="targetMimetype"><value>text/plain</value></property>
                     </bean>
                        </list>
                </property>
        </bean>
 
   <!– replaces bean transformer.complex.OpenOffice.PdfBox –>
   <bean id="transformer.complex.OpenOffice.PdfToTextTool"
        class="org.alfresco.repo.content.transform.ComplexContentTransformer"
        parent="baseContentTransformer" >
      <property name="transformers">
         <list>
            <ref bean="transformer.OpenOffice" />
            <ref bean="transformer.PdfToTextTool" />
         </list>
      </property>
      <property name="intermediateMimetypes">
         <list>
            <value>application/pdf</value>
         </list>
      </property>
   </bean>
</beans>
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

A german guy http://thinkalfresco.blogspot.com/2009/03/speeding-up-pdf-indexing-alfresco-hack.html did some nice comparison which led me to it, although I myself was experiencing problems with the full-text indexing of pdf's in the AVM(staging sandbox) which I have yet to resolve.

I hope this helps.

Jitse

Re: PDF FullText Indexing Alfresco indexer ignores blanks/spaces

_sax — Fri, 17 Jul 2009 13:26:41 GMT

Just for the record
With 3.2 this doesn't seem to work, as the appropriate class is missing.
http://forums.alfresco.com/en/viewtopic.php?f=10&t=19404

Re: PDF FullText Indexing Alfresco indexer ignores blanks/spaces

mwildam — Wed, 05 Aug 2009 13:12:59 GMT

Experiencing the same (Missing RuntimeExecutableContentTransformer) I have filed a bug: https://issues.alfresco.com/jira/browse/ALFCOM-3288