Deployment hangs on 'in progress' state

[diegovilar]

Hello there.

I've been having problems trying to deploy to FSR. I have this scenario: 3 FSRs, 2 of them on the local network and another one to which I deploy through a WAN (let's call this one the distant, slow FSR). Deploying a small content succeeds on all of them. So far so good. The problem is that whenever I attempt to deploy a not so small content to the distant, slow FSR (not big also, say 15 or 20 megabytes), the deployment never finishes, Alfresco Explorer shows "in progress" forever and nothing else happens, even though it works just fine on the other 2 FSRs (the ones on the LAN). Checking the depdata directory on the FSR machine, I can see that all of the files have been uploaded and are there sitting and waiting to be moved to the target directory. Checking the logs, I get the following stacktrace regarding the action:

10:41:29,248 ERROR [org.alfresco.repo.avm.actions.AVMDeploySnapshotAction] org.hibernate.exception.LockAcquisitionException: could not execute query
10:41:29,326 ERROR [org.hibernate.transaction.JDBCTransaction] JDBC rollback failed
java.sql.SQLException: Conexão Fechada
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
   at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:208)
   at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:1133)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:265)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:288)
   at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
   at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:581)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:745)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:722)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:325)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.completeTransactionAfterThrowing(SpringAwareUserTransaction.java:519)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:487)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:292)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:166)
   at org.alfresco.repo.action.AsynchronousActionExecutionQueueImpl$ActionExecutionWrapper.run(AsynchronousActionExecutionQueueImpl.java:267)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)
10:41:29,326 ERROR [org.alfresco.util.transaction.SpringAwareUserTransaction] Application exception overridden by rollback exception
java.lang.Exception
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:487)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:292)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:166)
   at org.alfresco.repo.action.AsynchronousActionExecutionQueueImpl$ActionExecutionWrapper.run(AsynchronousActionExecutionQueueImpl.java:267)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)
10:41:29,326 ERROR [org.alfresco.repo.action.AsynchronousActionExecutionQueueImpl] Failed to execute asynchronous action: Action[ id=240bc0fb-5b43-11de-bf16-e94486ade990, node=null ]
org.springframework.orm.hibernate3.HibernateJdbcException: JDBC exception on Hibernate data access; nested exception is org.hibernate.exception.GenericJDBCException: Cannot release connection
Caused by: org.hibernate.exception.GenericJDBCException: Cannot release connection
   at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
   at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
   at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
   at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:449)
   at org.hibernate.jdbc.ConnectionManager.aggressiveRelease(ConnectionManager.java:400)
   at org.hibernate.jdbc.ConnectionManager.afterTransaction(ConnectionManager.java:287)
   at org.hibernate.jdbc.JDBCContext.afterTransactionCompletion(JDBCContext.java:221)
   at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:174)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:581)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:745)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:722)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:325)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.completeTransactionAfterThrowing(SpringAwareUserTransaction.java:519)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:487)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:292)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:166)
   at org.alfresco.repo.action.AsynchronousActionExecutionQueueImpl$ActionExecutionWrapper.run(AsynchronousActionExecutionQueueImpl.java:267)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Already closed.
   at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:77)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)
   at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:95)
   at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:445)
   … 16 more
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

By the DB errors on the stack, I figured it could be related to the database connection timing out after an idle period (since it's a bigger deployment to a distant FSR), so I changed oracle's idle timeout to 60min and tested it again with a ~10min deployment only to incur on the same problem.

Does anyone have any idea of what might be happening and how to fix it?

Thanks in advance,
Diego

[mrogers]

Alfresco 2.9 is not a version I'm familiar with.    But I can see from the stack trace that you provided that the problem is in createSnapshot on the authoring environment which is suffering from some sort of timeout or deadlock.

The easiest solution is probably to upgrade to a more recent version of Alfresco.     

But if that's not an option then you may be able to
a) Snapshot the store before attempting to deploy in parallel and pass this snapshotID into deployment rather than -1 (latest version).
b) Control deployment such that only one deployment is done at a time.  

A step forward would be to check that deployment to your slow FSR works if you don't have interference from your two fast FSRs.