topic Re: Jobexecutor does not cope with simultaneousness in Alfresco Archive

Jobexecutor does not cope with simultaneousness

sazzadul — Wed, 28 Mar 2012 22:50:51 GMT

Hi,I wanted to share an experience with you and hopefully find a solution for it too.It seems like Jobexecutor simply goes into hibernation when it gets to much to do. Here is the scenario…I have a process with a few servicetasks(async) and a user task with timerboundaryevent which was a duedate say

Re: Jobexecutor does not cope with simultaneousness

meyerd — Thu, 29 Mar 2012 07:56:36 GMT

Hi sazzadul,

Sounds strange.

Do you see any exceptions - either in the log or in the database?

Re: Jobexecutor does not cope with simultaneousness

bheurter — Thu, 26 Apr 2012 08:03:38 GMT

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processes and starts remaining async tasks. 1 times on 3, It hangs. No explanations, no messages.

Any solutions ?

Re: Jobexecutor does not cope with simultaneousness

sazzadul — Thu, 26 Apr 2012 11:04:16 GMT

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processed and starts remaining async tasks. 1 times on 3, It hangs. No explanations, no messages.

Any solutions ?

I would recommend to run only one instance of the JobExecutor and check the db configuration as it may have major impact. Here is the configuration I am using and it's working like a charm. I have tested with 500 simultenouse tasks with no issues. I would also recommend use of an another id generator for instance the
StrongUuidGenerator which comes with Activiti. Let me know if this configuration helps.


    <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
       <property name="driverClassName" value="${jdbc.driver}" />
    <property name="url" value="${jdbc.url}" />
    <property name="username" value="${jdbc.username}" />
    <property name="password" value="${jdbc.password}" />
    </bean>

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="persistenceUnitManager">
            <bean class="org.springframework.orm.jpa.persistenceunit.DefaultPersistenceUnitManager"
                  p:defaultDataSource-ref="dataSource"/>
        </property>
        <property name="jpaProperties">
            <props>
                <prop key="hibernate.dialect">${hibernate.dialect}</prop>
                <prop key="hibernate.hbm2ddl.auto">update</prop>
            </props>
        </property>
        <property name="jpaVendorAdapter">
            <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
                <property name="showSql" value="false"/>
            </bean>
        </property>
    </bean>
 
  <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
      <property name="entityManagerFactory" ref="entityManagerFactory"/>
 </bean>

 <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
  <property name="databaseType" value="oracle" />
  <property name="dataSource" ref="dataSource" />
  <property name="transactionManager" ref="transactionManager" />
  <property name="databaseSchemaUpdate" value="true" />
  <property name="jobExecutorActivate" value="true" />
  <property name="history" value="activity" />
  <property name="idGenerator">
   <bean class="org.activiti.engine.impl.persistence.StrongUuidGenerator" />
  </property>
  <property name="idBlockSize" value="2000" />
 </bean>

Re: Jobexecutor does not cope with simultaneousness

demotdi — Fri, 04 May 2012 17:30:56 GMT

Hi,

I am facing an issue with job executor. Sometimes the jobs update the lock exp time and owner in the table and the job executor does not respond. It doesn't throw any exceptions. The application hangs and I have to restart it.

Is there a way to see the exception and clear out the locks when the application is running? I would appreciate any help.

Thanks.

Re: Jobexecutor does not cope with simultaneousness

trademak — Mon, 07 May 2012 08:03:08 GMT

Hi,

Did you try to run Activiti with a lower log level? Because the job executor then logs a message every couple of seconds that it's querying the database.
It's really strange if the job executor doesn't respond. Can it be that the job is executed 3 times and failed every time?
After 3 times, the job is not retried anymore by default.

Best regards,

Re: Jobexecutor does not cope with simultaneousness

mmaker1234 — Mon, 25 Mar 2013 15:19:10 GMT

Hello Activiti developers,

We also suffered by shutting JobExecutor on Activiti Engine 5.11.

The environment is:
- A modified job retry command that actually does not retry;
- A process definition with asynchronous ServiceTask activities.
- Engine configuration, which allows only one thread in the pool:

  <bean id="processEngineConfiguration" class="org.activiti.engine.impl.cfg.JtaProcessEngineConfiguration">
    …
    <property name="jobExecutorActivate" value="true" />
    <property name="jobExecutor">
      <bean class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
        <property name="corePoolSize" value="1"/>
        <property name="maxPoolSize" value="1"/>
      </bean>
    </property>      
    …

My observations are the following:
- We have about ten instances of the process definition;
- We have a bug in the service task, which throws an exception
- The job exececutor creates a new thread after each failure:

pool-1-thread-1
pool-1-thread-2
pool-1-thread-3
pool-1-thread-4
pool-1-thread-5

- There are no further traces from the JobExecutor.

It seems the JobExecutor "refuses" to serve after the fifth failure. As you can see above we have more than five process instances, i.e. there are more "failures" to proceed.

We survived the problem by suspending these faulty processes, that is why I claim that there is no other reason to look for. As the processes are suspended, we can easily resume them and see if they will break again the JobExecutor, if you need such a proof (the server is a test one and its manipulation is somewhat difficult).

I hope this information could help you to track the problem.

Re: Jobexecutor does not cope with simultaneousness

trademak — Mon, 25 Mar 2013 16:41:38 GMT

Hi,

Would it be possible to create a unit test that shows the issue with the job executor?

Best regards,

Re: Jobexecutor does not cope with simultaneousness

mmaker1234 — Tue, 26 Mar 2013 14:23:31 GMT

Hello Tijs,

Here is the unit test: https://docs.google.com/file/d/0B0LoSQX-9OwXRVVVNzhzTHJDWUE/edit?usp=sharing.

I hope it is understandable but I'm ready to answer any questions you might have regarding the test case.

Re: Jobexecutor does not cope with simultaneousness

stevennguyen — Thu, 15 Aug 2013 03:21:47 GMT

Hi Tijs & mmaker1234,

Is it's true that "The job exececutor creates a new thread after each failure" ?

We're facing the same issue wit Job Executor. We have 16 manage servers in 4 boxes dedicate for Job executors. If one hang and all Job Executors hang. I couldn't find which jobs cause Jo executors to hang. I've check the lower level log and found not exception.

Below are the end if the log:

013-08-13 18:23:12,222 insertTimer DEBUG - - ==> Parameters: 93061860(String), null, null, true(Boolean), 93061859(String), 85901776(String), null, 3(Integer), null, null, 2013-08-13 18:24:12.146(Timestamp), R5/2013-08-13T18:23:12/PT1M(String), timer-intermediate-transition(String), timerintermediatecatchevent1(String)
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ==> Preparing: insert into ACT_HI_ACTINST ( ID_, PROC_DEF_ID_, PROC_INST_ID_, EXECUTION_ID_, ACT_ID_, TASK_ID_, CALL_PROC_INST_ID_, ACT_NAME_, ACT_TYPE_, ASSIGNEE_, START_TIME_, END_TIME_, DURATION_ ) values ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ==> Parameters: 93061861(String), ProvAddItem:1774:85456592(String), 85901776(String), 93061859(String), timerintermediatecatchevent1(String), null, null, TimerCatchEvent(String), intermediateTimer(String), null, 2013-08-13 18:23:12.147(Timestamp), null, null
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ==> Preparing: update ACT_GE_BYTEARRAY set REV_ = ?, BYTES_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ==> Parameters: 2(Integer), java.io.ByteArrayInputStream@7ceaeb21(ByteArrayInputStream), 85901780(String), 1(Integer)
2013-08-13 18:23:12,241 updateExecution DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,241 updateExecution DEBUG - - ==> Preparing: update ACT_RU_EXECUTION set REV_ = ?, PROC_DEF_ID_ = ?, ACT_ID_ = ?, IS_ACTIVE_ = ?, IS_CONCURRENT_ = ?, IS_SCOPE_ = ?, IS_EVENT_SCOPE_ = ?, PARENT_ID_ = ?, SUPER_EXEC_ = ?, SUSPENSION_STATE_ = ?, CACHED_ENT_STATE_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,241 updateExecution DEBUG - - ==> Parameters: 4(Integer), ProvAddItem:1774:85456592(String), null, false(Boolean), false(Boolean), true(Boolean), false(Boolean), null, null, 1(Integer), 0(Integer), 85901776(String), 3(Integer)
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ==> Preparing: delete from ACT_RU_JOB where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ==> Parameters: 93037099(String), 2(Integer)

Re: Jobexecutor does not cope with simultaneousness

trademak — Thu, 15 Aug 2013 07:54:02 GMT

Hi,

The job executor is a thread pool and polls for jobs to be processed.
When a job fails it is placed back in the jobs table and the retry count is increased by 1.
When a job executor polls for jobs again the failed job will be processed again until the same job has been retried 3 times, then the job is placed in deadlock mode and it will be be retried again.

What's the exact behavior you are experiencing? How many jobs are there in the ACT_RU_JOB table and what are the job executor threads doing?

Best regards,

Re: Jobexecutor does not cope with simultaneousness

stevennguyen — Sat, 17 Aug 2013 12:03:22 GMT

Hi Tijs,

We have serveral job executors. When job executor hang, we restart it. After it pickup a few jobs, it hang again. All job executors will be hung within a few minutes. The number of jobs can be from 200 to 20,000 when we have the issue. We did a threaddump, but not able to find job executor thread.

I've seperated out different type of jobs, by trucate ACT_RU_JOB , manually insert jobs back and have only jobexecutors running to process them. Certain type of job, job executors can proccess 20K within 10 mins without an issue. We found that the job that causes job executor to hang, it requires external webservice call and take longer to process.

I see the last printout from the lob with Server error. Could this causes the job executor crash or hung?
Last printout from Job Executor:
Aug 16, 2013 8:20:47 PM org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable run
INFO: JobExecutor[org.activiti.engine.impl.jobexecutor.DefaultJobExecutor] starting to acquire jobs
INFO: Error while closing command context
org.activiti.engine.JobNotFoundException: No job found with id '94593698'.
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:59)
        at org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:60)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
Aug 16, 2013 8:21:56 PM org.activiti.engine.impl.interceptor.CommandContext close
SEVERE: Error while closing command context
org.activiti.engine.ActivitiOptimisticLockingException: HistoricVariableInstanceEntity[103316416] was updated by another transaction concurrently
        at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:652)
        at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:460)
        at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:167)
        at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:114)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:69)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)

Last few lines in the log:

Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.framework.policy.advertisementimpl.AdvertisementHelperImpl registerExtension
WARNING: Registering oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry extension failed; java.lang.ClassNotFoundException: oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry
Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.spi.WLSProvider createServiceDelegate
WARNING: Could not read WSDL Definition from URL wsdlDocumentLocation: 2 counts of InaccessibleWSDLException.

Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.framework.policy.advertisementimpl.AdvertisementHelperImpl registerExtension
WARNING: Registering oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry extension failed; java.lang.ClassNotFoundException: oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry
Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.spi.WLSServiceDelegate addWsdlDefinitionFeature
SEVERE: Failed to create WsdlDefinitionFeature for wsdl location: zip:/apps/opt/weblogic/config/BPS/servers/fduscpw8_bps01/tmp/_WL_user/CMPRealtime/shqoub/APP-INF/lib/ITWBAgent.jar!/META-INF/wsdl/heartbeat.wsdl, error: com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException, message: 2 counts of InaccessibleWSDLException.

Re: Jobexecutor does not cope with simultaneousness

trademak — Wed, 21 Aug 2013 07:25:14 GMT

Hi,

From the last part of your log I see error messages for executing the web service call. This would mean that the job is retried 3 times and fails 3 times. After that the job will not be processes again. If this happens for every job, eventually the job executor will stop because all jobs are in failed state. Could that be the case?

Best regards,

Re: Jobexecutor does not cope with simultaneousness

stevennguyen — Wed, 11 Sep 2013 14:33:08 GMT

Tijs,

Found the root of this issue. We're using Weblogic version 11g and it's using older version of joda jar than Activiti's joda jar. In some scenario, we're getting "ERROR" and it's being not catched. This cause AcquireJobsRunnable to die. Below is the Error that I caught and print in AcquireJobsRunnable.

org.joda.time.DateTime.parse(Ljava/lang/StringLorg/joda/time/DateTime;
Sep 2, 2013 9:32:54 PM org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable run
INFO: 232513==>org.activiti.engine.impl.calendar.DurationHelper.parseDate(DurationHelper.java:119)
org.activiti.engine.impl.calendar.DurationHelper.<init>(DurationHelper.java:64)
org.activiti.engine.impl.calendar.CycleBusinessCalendar.resolveDuedate(CycleBusinessCalendar.java:30)
org.activiti.engine.impl.persistence.entity.TimerEntity.calculateRepeat(TimerEntity.java:90)
org.activiti.engine.impl.persistence.entity.TimerEntity.execute(TimerEntity.java:72)
org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:71)
org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:60)
org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:38)
org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:50)
org.activiti.engine.impl.jobexecutor.CallerRunsRejectedJobsHandler.jobsRejected(CallerRunsRejectedJobsHandler.java:26)
org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:67)
java.lang.Thread.run(Thread.java:662)

Re: Jobexecutor does not cope with simultaneousness

trademak — Thu, 12 Sep 2013 06:42:49 GMT

Okay, but are you saying that there are new jobs for the job executor to process and it will not execute the new ones anymore, because of this error?

Best regards,

Re: Jobexecutor does not cope with simultaneousness

mmaker1234 — Tue, 17 Sep 2013 08:00:25 GMT

Hello Tijs,

At least in our case "<i>…there are new jobs for the job executor to process and it will not execute the new ones anymore, because of this error</i>" is exactly what we observed. I suppose Steven Nguyen observes the same.

Re: Jobexecutor does not cope with simultaneousness

trademak — Tue, 17 Sep 2013 09:25:25 GMT

Okay thanks, we'll try to reproduce it.

Best regards,

Re: Jobexecutor does not cope with simultaneousness

pollop — Wed, 25 Sep 2013 13:45:01 GMT

- Wrong post -

Re: Jobexecutor does not cope with simultaneousness

frederikherema1 — Wed, 02 Oct 2013 07:49:03 GMT

This has been fixed in b10dfc9ab6170641030beb9e84151ed4a949ef03 on master, will be part of the 5.14 release

Re: Jobexecutor does not cope with simultaneousness

mmaker1234 — Thu, 03 Oct 2013 08:33:58 GMT

Thank you very much, Frederik! 🙂

I'll check when 5.14 is released.