topic Re: Deadlock when having multiple executors in Alfresco Archive

Deadlock when having multiple executors

dan_ — Thu, 10 Mar 2016 00:00:01 GMT

I am writing a simple application to run 8 executors, each with a max thread pool of 2. I am using Activiti 5.19.0.1.The total number of process instances is 10k. It takes 10 to 16 minutes to complete the 10k processes on a mac pro with 16G. The database used is Postgres.All tasks in the process are

Re: Deadlock when having multiple executors

dan_ — Thu, 10 Mar 2016 00:01:41 GMT

I am sorry, I could not attach the doc file since only text files can be uploaded.

Re: Deadlock when having multiple executors

dan_ — Thu, 10 Mar 2016 04:35:49 GMT

Attached the pg_log that shows the deadlocks

Re: Deadlock when having multiple executors

dan_ — Mon, 14 Mar 2016 20:58:45 GMT

I have fixed the issue I described above which occurs in highly concurrent environments.

It seems that the engine context reuse delays commits on the database side which leads to deadlocks with a lot of commits waiting on the database side and not visible from the application side.

I think the retry count should be unlimited or at least have a high threshold so that a job which has not executed due to unavailable lock, can have the chance to be retried till it is successfully executed.

My fix does:

1- set by default this.contextReusePossible to false in the default constructor of the CommandConfig class
2- use select for update nowait on to update process instance lock time in the Execution.xml file
3- set the number of retries to 10 in the JobEntity class.

I am happy to contribute the attached patch. I am eager to have your feedback.

Thanks

Re: Deadlock when having multiple executors

jwestra — Tue, 15 Mar 2016 05:53:23 GMT

Have you been able to repeat on Activiti 6? Just curious, since they have a different persistence model/approach.

Re: Deadlock when having multiple executors

jbarrez — Tue, 15 Mar 2016 11:01:47 GMT

@dan_: thanks for loking into this so detailed.

So remarks
- the contextReusePossible general switch to false is not generally applicable, as it breaks many other use case (service call in java delegates being one). I wonder, why was that needed in the context of this?
- The default retries to 10 is a configurable option anyway
- But the sql change is interesting! … but it's probably postgres only, right? Why is this query better in the locking than the default one?

Re: Deadlock when having multiple executors

dan_ — Tue, 15 Mar 2016 15:47:42 GMT

Not yet, but am going to try it soon.

Re: Deadlock when having multiple executors

dan_ — Tue, 15 Mar 2016 19:21:03 GMT

Thanks for taking the time to give feedback.

For the why contextReusePossible is set to false. Actually, when running the application (attached to initial post) which starts 10k instances, I systematically reach a point where roughly around 9500 instances terminates, but the engine blocks, the threads are waiting except one or two. I suspected a deadlock, which is usually the case. However, I was surprised to see bunches of commits and update processes set idle and active on the database side, not able proceed due to deadlocks (server status view). In contrast, on the application side only have one or two thread still in runnable state.

I started looking into the code where the commits are done. In the CommandContextInterceptor class, the call to context.close() does the call to commit. However, I found out that if the contextReusePossible is true the commit is not done " if (!contextReused) { context.close();}". It seems to me that explains the bunch of idle commit process on the database side. I wanted the commits to be done as soon as the job is executed. Consequently, switching the contextReusePossible to false by default does the job for me and the my application managed to terminate all processes without blocking of deadlocks.

What, I understand from you that setting contextReusePossible to false by default breaks service calls, you mean developer provided service tasks, right? If this is the case, I can conclude that if an applicaiton having no service tasks, it is will behave correctly right? Or you mean that it breaks Activiti services? In this case, I am eager to know your input on why my application succeeded to terminate all process instances.

Concerning retries. I am wondering whether a retried job gets executed the number of times it is retried. By getting executed, I mean the execute method of the developer DelegateExecution implementation is invoked with every job retry. If this is the case, it might create side-effects, right? Consider the case when a job sends an email, it will sent the email the number of times the job is getting retried right? I understand that, such a situation is tolerable in the case of transactional data change as the transaction is rolled back, nevertheless it requires careful job design to mitigate side-effects.

When I read the code for ExecuteAsyncRunnable, I understand that a job is retried, if an OptimisticLockException is thrown or the developer provide implementation of the DelegateExecution throws and an exception itself right? But when I run my application, I find that the DelegateExecution is executed with every retrial. There is something I do not understand here. Can you please enlighten me on that.

For SQL change "select … for update nowait", it is available with all major databases: Oracle, SQL server, Postgres. In case a lock cannot be obtained immediately, it will raise an error and return right after without blocking which is very interesting to use in the case of Activiti job locking for execution (less resource consumption, more CPU cycles, more interestingly reduces deadlocks). BTW, I noticed huge performance gains 2 to 3 times improvement, when I applied my changes and running the same application attached to the initial post.

Thanks!

Re: Deadlock when having multiple executors

dan_ — Wed, 16 Mar 2016 05:14:06 GMT

What I meant by performance improvement is process completed throughput per second.

Re: Deadlock when having multiple executors

dan_ — Sun, 20 Mar 2016 03:11:56 GMT

Any feedback. partly based a feedback, I need to decide whether to use the engine or not. I am afraid, this is flaw in the design of how jobs are selected for execution (multiple threads can execute same job). Additionally, the mechanism of job retrials is not clear whether it may cause side effects. Say a job does increase the salary by 5%. If the job is retried, the salary would have been increased twice which is not acceptable from a business standpoint.

Thank you!

Re: Deadlock when having multiple executors

dan_ — Sun, 20 Mar 2016 03:14:29 GMT

Tried Activiti 6 rc2, same issue is experienced.

Re: Deadlock when having multiple executors

jbarrez — Tue, 22 Mar 2016 18:30:53 GMT

Hi dan_,

I have this topic high on my todo list to
1) verify the deadlocks in your original posts
2) read more about what the select for update nowait impact is and how it reacts on different databases

"Say a job does increase the salary by 5%. If the job is retried, the salary would have been increased twice which is not acceptable from a business standpoint."

No, if the increase in salary is done in a non-transactional way, then you are correct. it's always important in Activiti to make that logic transactional.

"I am afraid, this is flaw in the design of how jobs are selected for execution (multiple threads can execute same job). "

I must disagree with this: multiple threads won't execute the same job. When being retried, yes. But the actual execution of logic _and_ continuation of the process instance: no.

I'll post back when we've looked closer into the 'nowait' semantics and trying to reproduce the deadlocking.

Many thanks for the great comments with loads of information!

Re: Deadlock when having multiple executors

dan_ — Tue, 22 Mar 2016 19:02:15 GMT

Hi Jerome,

Thanks for clarifying the job retrial mechanism. In light of what you have explained, do you think it make sense to set a limit on the number of job retrials. I think it should be unlimited. Since a job should be retried until success. For there came my confusion.

Thanks,

Dan

Re: Deadlock when having multiple executors

jbarrez — Tue, 22 Mar 2016 19:16:29 GMT

The idea of a finite number of retries, is to avoid that an already dead system doesn't get hammered forever … and that it goes into some sort of 'dead-letter-queue' thing (like for messaging) where an admin can manually retry the job.

Of course, you can set it to integer.Max if you like.

Alternatively, to have full control, you can inject your own implementation of org.activiti.engine.impl.jobexecutor.FailedJobCommandFactory (which by default creates a RetryJobCmd that does the default behavior) and do whatever you please.

Re: Deadlock when having multiple executors

jbarrez — Thu, 24 Mar 2016 15:15:55 GMT

Hi dan_,

When using 5.19.0.1, I'm seeing:

Exception in thread "pool-1-thread-6" org.activiti.engine.ActivitiOptimisticLockingException: Could not lock process instance
at org.activiti.engine.impl.persistence.entity.ExecutionEntityManager.updateProcessInstanceLockTime(ExecutionEntityManager.java:208)
at org.activiti.engine.impl.cmd.LockExclusiveJobCmd.execute(LockExclusiveJobCmd.java:55)

Note sure if it's the same thing as you see … BUT when I run on 5.19.0.2 (and master, 5.20-SNAPSHOT), I'm not seeing any exception anymore, and all process instances get completed. Can you give that a go on your side too?

Re: Deadlock when having multiple executors

dan_ — Fri, 25 Mar 2016 20:52:21 GMT

Thanks Joram,

I did test it on 5.19.0.2 for 10k, it work with no problem, but I observed significant throughput decrease. I will repeat the test for 100k, 1M process and will reflect on that.

Thank you for taking this issue seriously.

Dan

Re: Deadlock when having multiple executors

jbarrez — Tue, 29 Mar 2016 07:52:38 GMT

Thanks dan_ for the update. A throughput decrease is not wanted, but maybe it's a side effect of the fix we made. Can't recall the exact details, but there was definitely something broken - the queue size was not honored properly, which made that jobs were executed even if the queue was full. So, definitely have a go at playing with queue size parameters too, as it will have a serious impact in that release!

Re: Deadlock when having multiple executors

dan_ — Tue, 29 Mar 2016 16:16:11 GMT

Thanks Joram for the information,

In turn, I want to report that we were able to run successfully 1_000_000 process on 4 different nodes, it took 9 hours. The best of all, is that no deadlock were detected.

Thanks for the effort you expended on fixing issues.

Re: Deadlock when having multiple executors

jbarrez — Mon, 04 Apr 2016 13:54:27 GMT

dan_, that is very awesome to hear! Thanks for posting back!

Re: Deadlock when having multiple executors

remibantos — Fri, 03 Jun 2016 13:57:53 GMT

Hi,

I had the same issue using Activiti 5.20 deployed on Wildfly with SQL Server when enabling async continuations for some service tasks in my BP.
Note: My set-up of Activiti might be a little bit atypical as I use both activiti-cdi and activiti-rest-api.

Changing contextReuseProssible value to false fixed it for me too. Could you explain what exactly the goal of this property is and what does the context reuse mean exactly for jobs executor?

Thanks,
Rémi