Job DEFAULT.ftsIndexerJobDetail threw an unhandled Exception

[bhavikp]

Hi,

When the scheduler run at 3AM daily OOTB, it will throw exception with below stack trace. I am using alfresco 3.1.0 EE.

Stack Trace:
——————–

03:06:23,711 ERROR [org.quartz.core.JobRunShell] Job DEFAULT.ftsIndexerJobDetail threw an unhandled Exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
   at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
   at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
   at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
   at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
   at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:510)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy39.index(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:52)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: java.sql.SQLException: Io exception: Got minus one from a read call
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
   at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
   at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
   at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
   at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
   at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
   at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
   at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:974)
   at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
   … 14 more
03:06:23,827 ERROR [org.quartz.core.ErrorLogger] Job (DEFAULT.ftsIndexerJobDetail threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:541)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy39.index(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:52)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
   … 1 more
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
   at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
   at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
   at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
   at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
   at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:510)
   … 9 more
Caused by: java.sql.SQLException: Io exception: Got minus one from a read call
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
   at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
   at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
   at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
   at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
   at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
   at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
   at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:974)
   at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
   … 14 more
03:06:57,984 ERROR [org.alfresco.repo.avm.OrphanReaper] Garbage collector error
org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Cannot open connection
Caused by: org.hibernate.exception.GenericJDBCException: Cannot open connection
   at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
   at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
   at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
   at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
   at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
   at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:510)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:219)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.begin(SpringAwareUserTransaction.java:415)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:311)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:229)
   at org.alfresco.repo.avm.OrphanReaper.doBatch(OrphanReaper.java:349)
   at org.alfresco.repo.avm.OrphanReaper.execute(OrphanReaper.java:59)
   at org.alfresco.repo.avm.OrphanReaperJob.execute(OrphanReaperJob.java:44)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: java.sql.SQLException: Io exception: Got minus one from a read call
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
   at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
   at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:441)
   at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
   at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
   at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
   at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
   at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
   at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:974)
   at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
   at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
   … 16 more

Can any one help why this happen?

Regards,
Bhavik

[bhavikp]

Hi,

This issue has been resolved.

Steps:

1. Stop Alfresco Server

2. Delete everything under <ALFRESCO_HOME>/alf_data/lucene-indexes (Take a backup of lucene-indexes folder in case of any failure)

3. Edit the following property file:
    <ALRESCO_HOME>/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/repository.properties
    Set the property index.recovery.mode as follows:
    # The index recovery mode (NONE, VALIDATE, AUTO, FULL)
    index.recovery.mode=FULL

4. Start Alfresco Server
   
Watch the log file, alfresco will rebuild the lucene index.
You may also want to set index.recovery.modeback to VALIDATE.

Regards,
Bhavik.