JobEntity was updated by another transaction concurrently

[tomekzaremba]

Hi,

during stress test of our application most of process instances failed because of exception:

org.activiti.engine.ActivitiOptimisticLockingException: JobEntity [id=1301291] was updated by another transaction concurrently
   at org.activiti.engine.impl.db.DbSqlSession$CheckedDeleteOperation.execute(DbSqlSession.java:297)
   at org.activiti.engine.impl.db.DbSqlSession.flushRegularDeletes(DbSqlSession.java:924)
   at org.activiti.engine.impl.db.DbSqlSession.flushDeletes(DbSqlSession.java:890)
   at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:615)
   at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:212)
   at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:138)
   at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)
   at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
   at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
   at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
   at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
   at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
   at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
   at pl.play.executor.ComLoggingExecuteAsyncRunnable.executeJob(ComLoggingExecuteAsyncRunnable.java:34)
   at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.run(ExecuteAsyncRunnable.java:50)
   at pl.play.executor.ComLoggingExecuteAsyncRunnable.run(ComLoggingExecuteAsyncRunnable.java:25)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Retries counter for jobs met -1 and about 80% of started process instances freezes.
All jobs that failed are exclusive and are starting async serviceTasks.

Our architecture contains two Activiti based applications connected to same database. Both are using SpringAsyncExecutor to obtain new jobs. Most of the configuration is default for Activiti framework.

What are we doing wrong? Is Activiti not able to handle our use case to use async executor in multiple engines to process started instances?

Any help will be appreciated.

[tomekzaremba]

I see that in jobEntity neither lockOwner nor lockTime are set for job. Why this is not default behavior? Maybe I set this off?

[tomekzaremba]

I see in user guide:

{code}
The Async executor will execute the asynchronous job directly without first polling the database. In case the asynchronous job is an exclusive job, the Async executor will first lock the process instance execution and if that succeeds execute the job and unlock the process instance execution again. If the process instance lock fails, it will retry.
{code}

My jobs are both asynchronous and exclusive. When I debug and stops on serviceTask my jobEntity still does not have lockOwner and lockTime. Is this correct or is it Activiti bug? 

[tomekzaremba]

My fault - jobs are correctly locked / unlocked. Maybe my retry policy is causing errors in multi-engine environment (R3/PT10S).

Do you have any hints, how can I configure my cluster to handle processing without changing retry policy? Maybe async executor is not suitable solution for me? Maybe ActivitiOptimisticLockException should be handled besides default serviceTask policy - maybe it is proper to change due_date adjusting and don't decrement retries counter?

[tomekzaremba]

Any hints? How cluster setup should be handled properly with async executor?

[jbarrez]

Ok, quite a bit of text to read.

The optimistic locking exception os to be expected in a clustered setup, the stalling of course not.
This seems to trigger some vague memory in me. I think we've worked on that (https://github.com/Activiti/Activiti/commit/e59d08629cfe514d7ee23eacd619a6b7bfb509e6) … so it might be good to build a snapshot version and give it a go.

[vsk106]

Hi jbarrez,
we are using Alfresco Activiti Enterprise BPM Suite v1.4.5 and we are facing same issue without any cluster for async Service tasks.
Could you please confirm issue has been fixed in v1.4.5 or not ?