Jobexecutor does not cope with simultaneousness

[sazzadul]

Hi,

I wanted to share an experience with you and hopefully find a solution for it too.

It seems like Jobexecutor simply goes into hibernation when it gets to much to do.

Here is the scenario…
I have a process with a few servicetasks(async) and a user task with timerboundaryevent which was a duedate say 15 minutes ahead.
Now when I start a number of processes (say 100) they all enter the user task and waits for the duedate to expire.
So far so good, but as soon as the duedate expires the jobexecutor kicks in and it begins to work, I can see in the RU_JOB table that it updates the lock column for a few rows but thats basically it. The jobexecutor hereafter doesn't do a single thing, it totally stops. What is even worse it stops processing the new processes as well.

I have tried to increase the maxpoolsize, waitTimeInMillis without any luck. I have even tried a customjobexecutor(as suggested in this http://forums.activiti.org/en/viewtopic.php?f=6&t=3523&hilit=jobexecutor) which uses commonj but the result is same.

I am using jetty with Oracle, have also tried postgres and mssql2008 the result is same.
There is no exception whatsoever being thrown from activiti so I have no clue where the problem lies.

Is there any bug in the jobexecutor ? I hope I have made myself clear enough to reproduce this error.

Thanks.
Sazzadul

Jobexecutor does not seem to tackle simultaneousness at all.

[meyerd]

Hi sazzadul,

Sounds strange.

Do you see any exceptions - either in the log or in the database?

[bheurter]

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processes and starts remaining async tasks.  1 times on 3, It hangs.  No explanations, no messages.

Any solutions ?

[sazzadul]

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processed and starts remaining async tasks.  1 times on 3, It hangs.  No explanations, no messages.

Any solutions ?

I would recommend to run only one instance of the JobExecutor and check the db configuration as it may have major impact. Here is the configuration I am using and it's working like a charm. I have tested with 500 simultenouse tasks with no issues. I would also recommend use of an another id generator for instance the
StrongUuidGenerator which comes with Activiti. Let me know if this configuration helps.


    <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
       <property name="driverClassName" value="${jdbc.driver}" />
    <property name="url" value="${jdbc.url}" />
    <property name="username" value="${jdbc.username}" />
    <property name="password" value="${jdbc.password}" />
    </bean>

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitManager">
            <bean class="org.springframework.orm.jpa.persistenceunit.DefaultPersistenceUnitManager"
                  p:defaultDataSource-ref="dataSource"/>
        </property>
        <property name="jpaProperties">
            <props>
                <prop key="hibernate.dialect">${hibernate.dialect}</prop>
                <prop key="hibernate.hbm2ddl.auto">update</prop>
            </props>
        </property>
        <property name="jpaVendorAdapter">
            <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
                <property name="showSql" value="false"/>
            </bean>
        </property>
    </bean>

  <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
      <property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>

<bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
  <property name="databaseType" value="oracle" />
  <property name="dataSource" ref="dataSource" />
  <property name="transactionManager" ref="transactionManager" />
  <property name="databaseSchemaUpdate" value="true" />
  <property name="jobExecutorActivate" value="true" />
  <property name="history" value="activity" />
  <property name="idGenerator">
   <bean class="org.activiti.engine.impl.persistence.StrongUuidGenerator" />
  </property>
  <property name="idBlockSize" value="2000" />
</bean>


[demotdi]

Hi,

I am facing an issue with job executor. Sometimes the jobs update the lock exp time and owner in the table and the job executor does not respond. It doesn't throw any exceptions. The application hangs and I have to restart it.

Is there a way to see the exception and clear out the locks when the application is running? I would appreciate any help.

Thanks.

[trademak]

Hi,

Did you try to run Activiti with a lower log level? Because the job executor then logs a message every couple of seconds that it's querying the database.
It's really strange if the job executor doesn't respond. Can it be that the job is executed 3 times and failed every time?
After 3 times, the job is not retried anymore by default.

Best regards,

[mmaker1234]

Hello Activiti developers,

We also suffered by shutting JobExecutor on Activiti Engine 5.11.

The environment is:
- A modified job retry command that actually does not retry;
- A process definition with asynchronous ServiceTask activities.
- Engine configuration, which allows only one thread in the pool:  <bean id="processEngineConfiguration" class="org.activiti.engine.impl.cfg.JtaProcessEngineConfiguration">
    …
    <property name="jobExecutorActivate" value="true" />
    <property name="jobExecutor">
      <bean class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
        <property name="corePoolSize" value="1"/>
        <property name="maxPoolSize" value="1"/>
      </bean>
    </property>     
    …

My observations are the following:
- We have about ten instances of the process definition;
- We have a bug in the service task, which throws an exception
- The job exececutor creates a new thread after each failure:

pool-1-thread-1
pool-1-thread-2
pool-1-thread-3
pool-1-thread-4
pool-1-thread-5

- There are no further traces from the JobExecutor.

It seems the JobExecutor "refuses" to serve after the fifth failure. As you can see above we have more than five process instances, i.e. there are more "failures" to proceed.

We survived the problem by suspending these faulty processes, that is why I claim that there is no other reason to look for. As the processes are suspended, we can easily resume them and see if they will break again the JobExecutor, if you need such a proof (the server is a test one and its manipulation is somewhat difficult).

I hope this information could help you to track the problem.

[trademak]

Hi,

Would it be possible to create a unit test that shows the issue with the job executor?

Best regards,

[mmaker1234]

Hello Tijs,

Here is the unit test: https://docs.google.com/file/d/0B0LoSQX-9OwXRVVVNzhzTHJDWUE/edit?usp=sharing.

I hope it is understandable but I'm ready to answer any questions you might have regarding the test case.

[stevennguyen]

Hi Tijs & mmaker1234,

Is it's true that "The job exececutor creates a new thread after each failure" ?

We're facing the same issue wit Job Executor.  We have 16 manage servers in 4 boxes dedicate for Job executors.  If one hang and all Job Executors hang.   I couldn't find which jobs cause Jo executors to hang. I've check the lower level log and found not exception.

Below are the end if the log:

013-08-13 18:23:12,222 insertTimer DEBUG -  - ==> Parameters: 93061860(String), null, null, true(Boolean), 93061859(String), 85901776(String), null, 3(Integer), null, null, 2013-08-13 18:24:12.146(Timestamp), R5/2013-08-13T18:23:12/PT1M(String), timer-intermediate-transition(String), timerintermediatecatchevent1(String)
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG -  - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG -  - ==>  Preparing: insert into ACT_HI_ACTINST ( ID_, PROC_DEF_ID_, PROC_INST_ID_, EXECUTION_ID_, ACT_ID_, TASK_ID_, CALL_PROC_INST_ID_, ACT_NAME_, ACT_TYPE_, ASSIGNEE_, START_TIME_, END_TIME_, DURATION_ ) values ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG -  - ==> Parameters: 93061861(String), ProvAddItem:1774:85456592(String), 85901776(String), 93061859(String), timerintermediatecatchevent1(String), null, null, TimerCatchEvent(String), intermediateTimer(String), null, 2013-08-13 18:23:12.147(Timestamp), null, null
2013-08-13 18:23:12,227 updateByteArray DEBUG -  - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,227 updateByteArray DEBUG -  - ==>  Preparing: update ACT_GE_BYTEARRAY set REV_ = ?, BYTES_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,227 updateByteArray DEBUG -  - ==> Parameters: 2(Integer), java.io.ByteArrayInputStream@7ceaeb21(ByteArrayInputStream), 85901780(String), 1(Integer)
2013-08-13 18:23:12,241 updateExecution DEBUG -  - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,241 updateExecution DEBUG -  - ==>  Preparing: update ACT_RU_EXECUTION set REV_ = ?, PROC_DEF_ID_ = ?, ACT_ID_ = ?, IS_ACTIVE_ = ?, IS_CONCURRENT_ = ?, IS_SCOPE_ = ?, IS_EVENT_SCOPE_ = ?, PARENT_ID_ = ?, SUPER_EXEC_ = ?, SUSPENSION_STATE_ = ?, CACHED_ENT_STATE_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,241 updateExecution DEBUG -  - ==> Parameters: 4(Integer), ProvAddItem:1774:85456592(String), null, false(Boolean), false(Boolean), true(Boolean), false(Boolean), null, null, 1(Integer), 0(Integer), 85901776(String), 3(Integer)
2013-08-13 18:23:12,243 deleteMessage DEBUG -  - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,243 deleteMessage DEBUG -  - ==>  Preparing: delete from ACT_RU_JOB where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,243 deleteMessage DEBUG -  - ==> Parameters: 93037099(String), 2(Integer)