topic Deadlock in cluster during AcquireJobsCmd in Alfresco Archive https://connect.hyland.com/t5/alfresco-archive/deadlock-in-cluster-during-acquirejobscmd/m-p/158337#M112450 <HTML><HEAD></HEAD><BODY><SPAN>I have a cluster of 2 Activiti 5.14 instances, sharing the same PostgreSQL 9.1 database. I have a process with timers.</SPAN><BR /><SPAN>From time to time I experience a deadlock during AcquireJobsCmd execution:</SPAN><BR /><BR /><PRE class="language-none line-numbers"><CODE><BR />2013-12-17 12:00:01,083 Thread-96&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ERROR [&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; impl.interceptor.CommandContext] [&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; close] [140] : Error while closing command context: org.apache.ibatis.exceptions.PersistenceException: <BR />### Error updating database.&nbsp; Cause: org.postgresql.util.PSQLException: BŁĄD: deadlock detected<BR />&nbsp; Szczegóły: Proces 17382 oczekuje na ShareLock na transakcja 1398139; zablokowany przez 21060.<BR />Proces 21060 oczekuje na ShareLock na transakcja 1398140; zablokowany przez 17382.<BR />&nbsp; Wskazówka: Przejrzyj dziennik serwera by znaleźć szczegóły zapytania.<BR />### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.updateTimer-Inline<BR />### The error occurred while setting parameters<BR />### SQL: update ACT_RU_JOB&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SET REV_ =&nbsp; ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_EXP_TIME_ = ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_OWNER_ = ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RETRIES_ = ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_STACK_ID_ = ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_MSG_ = ?,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DUEDATE_ = ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; where ID_= ?&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; and REV_ = ?<BR />### Cause: org.postgresql.util.PSQLException: BŁĄD: wykryto zakleszczenie<BR />&nbsp; Szczegóły: Proces 17382 oczekuje na ShareLock na transakcja 1398139; zablokowany przez 21060.<BR />Proces 21060 oczekuje na ShareLock na transakcja 1398140; zablokowany przez 17382.<BR />&nbsp; Wskazówka: Przejrzyj dziennik serwera by znaleźć szczegóły zapytania.<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:147)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:560)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:444)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:170)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:117)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)<BR />&nbsp;&nbsp;&nbsp;at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)<BR />&nbsp;&nbsp;&nbsp;at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:59)<BR />&nbsp;&nbsp;&nbsp;at java.lang.Thread.run(Thread.java:662)<BR />Caused by: org.postgresql.util.PSQLException: BŁĄD: wykryto zakleszczenie<BR />&nbsp; Szczegóły: Proces 17382 oczekuje na ShareLock na transakcja 1398139; zablokowany przez 21060.<BR />Proces 21060 oczekuje na ShareLock na transakcja 1398140; zablokowany przez 17382.<BR />&nbsp; Wskazówka: Przejrzyj dziennik serwera by znaleźć szczegóły zapytania.<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:381)<BR />&nbsp;&nbsp;&nbsp;at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)<BR />&nbsp;&nbsp;&nbsp;at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<BR />&nbsp;&nbsp;&nbsp;at java.lang.reflect.Method.invoke(Method.java:597)<BR />&nbsp;&nbsp;&nbsp;at org.postgresql.ds.jdbc23.AbstractJdbc23PooledConnection$StatementHandler.invoke(AbstractJdbc23PooledConnection.java:455)<BR />&nbsp;&nbsp;&nbsp;at com.sun.proxy.$Proxy57.execute(Unknown Source)<BR />&nbsp;&nbsp;&nbsp;at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:297)<BR />&nbsp;&nbsp;&nbsp;at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:404)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:41)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:66)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:45)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:108)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75)<BR />&nbsp;&nbsp;&nbsp;at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)<BR />&nbsp;&nbsp;&nbsp;… 13 more<BR /><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><BR /><SPAN>From PostgreSQL log I can see statements&nbsp; that caused this deadlock:</SPAN><BR /><PRE class="language-none line-numbers"><CODE><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Proces 17382: update ACT_RU_JOB<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SET REV_ =&nbsp; $1,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_EXP_TIME_ = $2,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_OWNER_ = $3,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RETRIES_ = $4,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_STACK_ID_ = $5,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_MSG_ = $6,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DUEDATE_ = $7 <BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; where ID_= $8<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; and REV_ = $9<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Proces 21060: update ACT_RU_JOB<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SET REV_ =&nbsp; $1,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_EXP_TIME_ = $2,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; LOCK_OWNER_ = $3,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; RETRIES_ = $4,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_STACK_ID_ = $5,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; EXCEPTION_MSG_ = $6,<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; DUEDATE_ = $7 <BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; where ID_= $8<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; and REV_ = $9<BR /><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><SPAN>(though I cant see the history of these transactions)</SPAN><BR /><BR /><SPAN>I suspect that there is a flaw in AcquireJobsCmd class. It selects a set of jobs ready to run and then updates one by one these jobs. But the select doesnt specify "order by" so the order of rows and the order of updates is not guarantied, so i.e. if we have 2 jobs: A and B, then first instance of activiti may get rows in order A,B and the second instance B,A. First instance updates (locks) row A, second instance updates row B. Then first instance tries to update row A and the second instance tries to update row B. And there is a deadlock.</SPAN><BR /><BR /><SPAN>Is this a bug or I am missing something?</SPAN><BR /><BR /></BODY></HTML> Wed, 18 Dec 2013 14:31:26 GMT tomasz_lewandow 2013-12-18T14:31:26Z Deadlock in cluster during AcquireJobsCmd https://connect.hyland.com/t5/alfresco-archive/deadlock-in-cluster-during-acquirejobscmd/m-p/158337#M112450 I have a cluster of 2 Activiti 5.14 instances, sharing the same PostgreSQL 9.1 database. I have a process with timers.From time to time I experience a deadlock during AcquireJobsCmd execution:2013-12-17 12:00:01,083 Thread-96&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ERROR [&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; impl.interceptor.CommandContext] [&nbsp;&nbsp;&nbsp; Wed, 18 Dec 2013 14:31:26 GMT https://connect.hyland.com/t5/alfresco-archive/deadlock-in-cluster-during-acquirejobscmd/m-p/158337#M112450 tomasz_lewandow 2013-12-18T14:31:26Z Re: Deadlock in cluster during AcquireJobsCmd https://connect.hyland.com/t5/alfresco-archive/deadlock-in-cluster-during-acquirejobscmd/m-p/158338#M112451 <HTML><HEAD></HEAD><BODY><SPAN>Thanks for posting this. We had a team call on it and found out that you indeed found a bug in the job executor.</SPAN><BR /><BR /><SPAN>The corresponding Jira issue is </SPAN><A href="http://jira.codehaus.org/browse/ACT-1879#comment-337677" rel="nofollow noopener noreferrer">http://jira.codehaus.org/browse/ACT-1879#comment-337677</A><BR /><SPAN>It's fixed at </SPAN><A href="https://github.com/Activiti/Activiti/commit/1ea692de8bd2626fd2836ed31d160a036c3f4aa8" rel="nofollow noopener noreferrer">https://github.com/Activiti/Activiti/commit/1ea692de8bd2626fd2836ed31d160a036c3f4aa8</A><BR /><BR /><SPAN>This will be part of the next release. However, you can apply the same simple fix as in the git commit (set the maxJobsPerAcquisition to one).</SPAN><BR /><BR /><SPAN>Thank you very much!</SPAN></BODY></HTML> Thu, 19 Dec 2013 10:32:14 GMT https://connect.hyland.com/t5/alfresco-archive/deadlock-in-cluster-during-acquirejobscmd/m-p/158338#M112451 jbarrez 2013-12-19T10:32:14Z