AIX + Websphere installation (Lucene Alfresco stack problem)

[jmgab]

Hy,

we are trying to deploy Alfresco 1.4 on AIX (5.3) for Websphere (6.1).

Unfortunately, we encounter the following exception (the cause is in bold) :

[21/02/07 15:51:41:384 GMT] 00000042 WebApp        E   Exception caught while initializing context
org.alfresco.error.AlfrescoRuntimeException: Bootstrap failed
…
com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:213)
   at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:163)
   at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1469)
Caused by: java.nio.BufferUnderflowException
   at java.nio.Buffer.nextGetIndex(Buffer.java:419)
   at java.nio.DirectByteBuffer.getLong(DirectByteBuffer.java:783)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.setStatusFromFile(IndexInfo.java:1489)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.setStatusFromFile(IndexInfo.java:1298)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.access$32(IndexInfo.java:1294)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo$2.doWork(IndexInfo.java:348)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:1742)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.<init>(IndexInfo.java:344)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getIndexInfo(IndexInfo.java:243)
   at org.alfresco.repo.search.impl.lucene.LuceneBase2.initialise(LuceneBase2.java:97)
   at org.alfresco.repo.search.impl.lucene.LuceneIndexerImpl2.getUpdateIndexer(LuceneIndexerImpl2.java:477)
   at org.alfresco.repo.search.impl.lucene.LuceneIndexerAndSearcherFactory2.createIndexer(LuceneIndexerAndSearcherFactory2.java:352)
   at org.alfresco.repo.search.impl.lucene.LuceneIndexerAndSearcherFactory2.getThreadLocalIndexer(LuceneIndexerAndSearcherFactory2.java:309)
   at org.alfresco.repo.search.impl.lucene.LuceneIndexerAndSearcherFactory2.getIndexer(LuceneIndexerAndSearcherFactory2.java:293)
   at org.alfresco.repo.search.impl.lucene.LuceneIndexerAndSearcherFactory2.getIndexer(LuceneIndexerAndSearcherFactory2.java:72)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:615)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:213)
   at $Proxy17.getIndexer(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.updateNode(IndexerComponent.java:51)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:615)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:335)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:165)
   at $Proxy57.updateNode(Unknown Source)
   at org.alfresco.repo.node.index.NodeIndexer.onUpdateNode(NodeIndexer.java:96)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:615)
   at org.alfresco.repo.policy.JavaBehaviour$JavaMethodInvocationHandler.invoke(JavaBehaviour.java:243)
   at $Proxy60.onUpdateNode(Unknown Source)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:615)
   at org.alfresco.repo.policy.PolicyFactory$MultiHandler.invoke(PolicyFactory.java:243)
   at org.alfresco.repo.policy.$Proxy76.onUpdateNode(Unknown Source)
   at org.alfresco.repo.node.AbstractNodeServiceImpl.invokeOnUpdateNode(AbstractNodeServiceImpl.java:267)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.setProperties(DbNodeServiceImpl.java:889)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.addAspect(DbNodeServiceImpl.java:589)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.createStore(DbNodeServiceImpl.java:193)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:615)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:213)
   at $Proxy13.createStore(Unknown Source)
   at org.alfresco.repo.importer.ImporterBootstrap.bootstrap(ImporterBootstrap.java:358)
   … 100 more

now, we don't know if it comes from the AIX file management system or from the IBM Jdk.

Anybody can give us some advices ?

For instance, it is possible to avoir Lucene index file locking ? (if it may be the effective cause …)

thanks a lot

JM

ps : it's working on Windows XP for the same Websphere (6.1).

[kevinr]

That's a strange error - do you have more than one Alfresco instance using the same indexes? As it looks like index corruption…

Thanks,

Kevin

[jmgab]

ok … I apologize for the mail … we have deleted the database and the lucene-index directory and now it works again.

May be somebody has changed something on the directory … ?

Thank you very much for your reply.

JM

[jmgab]

In fact … the error occurs back again 😞

… and there is only one instance of Alfresco using the data :!:

The only difference (between when it works and when it doesn't work) is that we have changed the repository.jar library.

To be clear :

step 0 : a BufferUnderflowException appears (we don't know why)

step 1 : we use a repository.jar I've modified to put some "System.out" call (to see the content of the IndexInfo file with standard IO classes) and we reset all data (the database is cleaned and the index files are removed)
… and it worked again (cf. my previous post)

step 2 : now we are trying to install the "original" repository.jar (when I say "original" … it is not true since we have had to change the Xerces import of ISO9075.java)
… and it doesn't work no more (same BufferUnderflowException).

As all of this is not clear, we are going to investigate more deeply and may be I will provide you more contextual information about the phenomenon.

thanks a lot

JM

[andy]

Hi

I would go with 2.0. The IBM JDK issues should be fixed along with a few index issues. This error is not one we have seen before.

Can you check if you have zero length IndexInfo files anywhere?

This file is written twice (also the Backup version) - if one copy is corrupt due to a forced shutdown - the other should be used. May be this is not the case under some circumstance.

Are your indexes on a local disk?

Regards

Andy

[jmgab]

Ok,

we will try the new alfresco version next monday.

Indeed, we have zero length IndexInfo files … but also zero length BackUpIndexInfo files.
And the indexes are on a local disk but outside of the websphere repository.

Just for your information, I've put the code between "//>>" and "//<<" within the following method :

    private void setStatusFromFile(FileChannel channel) throws IOException
    {
       // >>
       try {
          File indexInfoFile = new File(this.indexDirectory, INDEX_INFO);
           BufferedReader in = new BufferedReader(new FileReader(indexInfoFile));
           int ch;
           while ((ch = in.read()) != -1) {
              System.out.println(">>file>>"+ ch); // SOP
           }
       // <<

… original and unchanged Alfresco code with NIO access …

       // >>
       in.close();
       } catch (IOException e) {e.printStackTrace();}   
       // << 
    }

and it seems to work (but we have to push test forward) … but I really don't know why … ?? I will investigate on the difference between NIO (I've never used) and IO this week end.

to follow …

JM

ps : thank you again for your help

[jmgab]

OK,

we have found a "fix" (?) for Alfresco 1.4

we have changed in the IndexInfo class source the value of the constant "useNIOMemoryMapping"

now we have : useNIOMemoryMapping = false;

it works now … but we are expecting some side effects.

Do you know what this change may imply ?

thanks a lot

JM

[andy]

Hi

Originally we tested both. NIO was faster (about twice the speed) so we went with that. There should be no major side effects.

Without NIO you can not be sure the data has been written to disk, the write buffer is manged explicitly (and not left to the java stuff) and is is easy to set the info file to the correct length 🙂 (With nio there is some gumpf from the largest size …)

This option has not been tested anywhere near as much.

Regards

Andy

[andy]

Hi

I have now managed to reproduce this issue with alfresco 2.1.dev websphere 6.1 and the embedded 1.5 JDK.

It would appear that the index info file is not being written correctly to disk with nio.

The same code with the Sun JDK works OK.

Andy

[andy]

Hi

I have resolved this issue by applying the patches to upgrade to WAS 6.1.0.7 and then upgrading the embedded JRE to 1.5.0 SR4.

What does "java -version" report?
I think you want to see SR3 in there somewhere.

I was seeing the error when:

1) starting with a clean DB and alf_data
2) start the app server (which bootstraps)
3) App runs OK
4) shut down app server
5) restart  -> nio error in the log

I have now done this and then stopped and restarted the app server repeatedly and all seems to be fine. There were nio related fixes described for the patch but none quite matched the problem. However, it appears fixed.

Andy