cancel
Showing results for 
Search instead for 
Did you mean: 

Thread hangs during RuntimeService.startProcessInstanceByKey

michalg
Champ in-the-making
Champ in-the-making
Hello,

I would like ask about a problem with Activiti, which starts to be quite serious. But first some background:
we are using Activiti as embedded Spring bean in our application. Application is deployed on several tomcat nodes.
Engine itself has standard configuration, only jobExecutor lock has been decreased:

  <bean id="configuredJobExecutor" class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
    <property name="lockTimeInMillis" value="120000"/> <!– 2 minutes –>
  </bean>

  <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
    <!– Database configurations –>
    <property name="databaseSchemaUpdate" value="true" />
    <property name="dataSource" ref="activitiDataSource" />
    <property name="transactionManager" ref="activitiTransactionManager" />
    <property name="jobExecutor" ref="configuredJobExecutor" />
    <property name="jobExecutorActivate" value="true" />
    <property name="deploymentResources" value="classpath*:/bpmn20/*.bpmn20.xml" />
  </bean>

  <bean id="processEngine" class="org.activiti.spring.ProcessEngineFactoryBean">
    <property name="processEngineConfiguration" ref="processEngineConfiguration" />
  </bean>
 
  <bean id="repositoryService" factory-bean="processEngine" factory-method="getRepositoryService" />
  <bean id="runtimeService" factory-bean="processEngine" factory-method="getRuntimeService" />
  <bean id="taskService" factory-bean="processEngine" factory-method="getTaskService" />
  <bean id="historyService" factory-bean="processEngine" factory-method="getHistoryService" />
  <bean id="managementService" factory-bean="processEngine" factory-method="getManagementService" />

We first started with Activiti version 5.9, but later upgraded to version 5.13 and lately to 5.14.
With 5.13 we observed problem with process instances not being started by runtime service (5.14 release contains lots of bug fixes, but unfortunately it didn't fix this one). When calling:
[java]
runtimeService.startProcessInstanceByKey(…)
[/java]
thread hangs - function does not return, no exception is thrown. To fix this we need to restart the application, which restarts the Activiti engine. After restart it works fine, but after random ammount of time (sometimes after 5 to 7 days, other times after 15 hours) problem occurs again and another restart is needed.

I've enabled debug logs on Activiti but they provide very little information for me. When problems occurs logs look like this:

DEBUG o.a.e.i.interceptor.LogInterceptor - — starting StartProcessInstanceCmd ——————————————————–
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ooo Using Connection [Transaction-aware proxy for target Connection  from DataSource
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ==>  Preparing: select * from ACT_RE_PROCDEF where KEY_ = ? and VERSION_ = (select max(VERSION_)
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ==> Parameters: default_terminate(String), default_terminate(String)
DEBUG o.a.e.i.interceptor.LogInterceptor -                                                                                                    
DEBUG o.a.e.i.interceptor.LogInterceptor - — starting StartProcessInstanceCmd ——————————————————–
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ooo Using Connection [Transaction-aware proxy for target Connection  from DataSource
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ==>  Preparing: select * from ACT_RE_PROCDEF where KEY_ = ? and VERSION_ = (select max(VERSION_)
DEBUG o.a.e.i.p.e.P.selectLatestProcessDefinitionByKey - ==> Parameters: default_activate(String), default_activate(String)
DEBUG o.a.e.i.interceptor.LogInterceptor -

and after a while, it just logs:

[http-bio-8080-exec-2257] DEBUG o.a.e.i.interceptor.LogInterceptor -                                                                                              
[http-bio-8080-exec-2257] DEBUG o.a.e.i.interceptor.LogInterceptor - — starting StartProcessInstanceCmd ——————————————————–
[http-bio-8080-exec-2256] DEBUG o.a.e.i.interceptor.LogInterceptor -                                                                                              
[http-bio-8080-exec-2256] DEBUG o.a.e.i.interceptor.LogInterceptor - — starting StartProcessInstanceCmd ——————————————————–
[http-bio-8080-exec-2261] DEBUG o.a.e.i.interceptor.LogInterceptor -                                                                                              
[http-bio-8080-exec-2261] DEBUG o.a.e.i.interceptor.LogInterceptor - — starting StartProcessInstanceCmd ——————————————————–
[http-bio-8080-exec-2266] DEBUG o.a.e.i.interceptor.LogInterceptor - 

This problem is really annoying - at the moment we are forced to monitor application and restart the server every couple of days. There is no error thrown when this happens, thread just hangs when executing Activiti code.

Did anyone else had the same problem?
If needed I can provide more Activiti logs and information.

Thank you for any help
4 REPLIES 4

frederikherema1
Star Contributor
Star Contributor
What datasource are you using? If that are all the logs you get, it seems like it never gets any response from the "selectLatestProcessDefinitionByKey - ==>  Preparing: select * from ACT_RE_PROCDEF where KEY_ = ? and VERSION_ = (select max(VERSION_)". Make sure you use a connection-pool that checks connections and evicts/recreates broken ones…

Do you have any details on the process-definition that is being used? Does it contain any asynchronous tasks or timers?
Also, does your activiti-engine use the job-executor a lot? In very high-load on the job-executor, the following condition can occur:

- All job-executor threads are occupied.
- The job-aquisition-thread looks for jobs to execute. In case the thread-pool is fully dried up, a special "strategy" is used to determine what to do with that job. The default (this is configurable) is that the calling thread runs the job instead. The calling thread, in this case, is the job-aquisition thread. This blocks any additional jobs from being executed as long as that job is being executed.
- When a new job is added (eg. using startProcessInstanceByKey), the job-executor is hinted at the end of the transaction commit. This hint will notify the aquisition-thread. if this is blocked, the current thread will wait for the aquisition-thread to complete. In case you have long-running jobs, this can cause a long wait-time. In case you have never-ending jobs (due to error), these jobs won't be retried because the thread that is responsible for actually re-running is occupied itself.

I'm suspecting the datasource is the culprit, just mentioning the job-executor scenario as additional info…

Hi frederikheremans,
I was wondering how to configure the default behavior of the calling thread running the job instead that you mentioned as configurable?

michalg
Champ in-the-making
Champ in-the-making
We are using postgresql and pgpool. Used configuration should manage connections, but we have observed some problems with it and are working on configuration fixes. So if this is the case, that fix should also help with this problem.

There is no very high-load on the job-executor at the moment, so job executor scenario is less probable and first task in launched process is async, so execution should return immediately.

We will observe situation after modifying datasource configuration and confirm if that was the case.

Thank you for useful hints Smiley Happy

frederikherema1
Star Contributor
Star Contributor
Great, keep us posted on the progress.