Suspended Process Definition but starting process Instances. Urgent !!!

[sharat]

Hello,

We are running Activiti 5.14 and we see this weird issue of Suspended process definition but still executing the process instances.
We are running 5.14 two engines pointing to Oracle DB. We never saw this issue until now.

Any suggestions ?

Thanks

[martin_grofcik]

Hi,

according to:
org.activiti.engine.test.api.repository.ProcessDefinitionSuspensionTest#testSuspendMultipleProcessDefinitionsByKeyit should not be possible to start process instance based on suspended process definition.
Could you create jUnit test for this case?
http://www.jorambarrez.be/blog/2012/09/24/how-to-write-a-unit-test/

Regards
Martin

[sharat]

Martin,
Sure that should not be the case and it never happened to us since more than an year we started using Activiti but happened now. in our production. I don't know if I can replicate the issue.

FYI.. the process instance had not been not started manually, The procdef was schedule to run daily and suspended after couple runs. But started to execute even in suspended state.

I will create the unit test and respond.

[jspinney]

We have the same 5.14 issue in our production environment and have duplicated in our staging environment on a daily run basis with only a start, wait 1 minute, and end node in the workflow.   The workflow ran for a week and was then suspended.  The next scheduled instance was reflected in the job table as expected, however the engine would not execute for two days and then execute one day.  The pattern would alter as to the number of skipped days but there is always an instance run when the deployment is in suspended state at some point.

[jspinney]

A member of my team found the following potential cause for consideration.
http://www.jorambarrez.be/blog/2012/12/20/tweaking-process-definition-cache/
The only possiblity of instance running after suspend is that theprocess def is picked from cache and cache is not refreshed

[jspinney]

We have duplicated the issue in a two node environment where the suspension of the deployment was processed by node 2 but the process definition cache in node 1 was not refreshed.  As the next instance was scheduled and was processed by node 2 the suspend was recognized and not executed.  The next instance was scheduled as expected.   However when the instance was processed on node 1 the cached process definition was not in the suspended state. 
We then stopped node 2 and the cache on node 1 was refreshed and the scheduled instance was not executed due to the suspended state.
We repeated the series of events and when the described conditions existed, restarted node 1 forcing a restart and the suspended deployment was reflected in the cache on both nodes.  
Could you recommend an approach to dynamically force a cache refresh on all nodes in the cluster when a deployment is suspended?

[martin_grofcik]

http://www.jorambarrez.be/blog/2013/01/03/distributed-cache-activiti/

[jspinney]

Thank you Martin.
John