JobExecutor.waitTimeInMillis being ignored

[jsilva]

Hello, everyone.

I'm trying to change the default value for the JobExecutor's waitTimeInMillis property, but my jobs are being retried immediately.

I'm using Activiti 5.15.1 and this is the relevant configuration:

<bean id="processEngineConfiguration" class="org.activiti.engine.impl.cfg.StandaloneProcessEngineConfiguration">
    <!– … –>
    
    <property name="jobExecutor">
       <bean id="jobExecutor" class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
          <property name="waitTimeInMillis" value="60000" />
       </bean>
    </property>
  </bean>
‍‍‍‍‍‍‍‍‍‍‍

I've found that the reason why this happens comes from the code block at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable:68:

if (jobsAcquired < maxJobsPerAcquisition) {
    // …
} else {
    millisToWait = 0;
}
‍‍‍‍‍‍‍

So I have three questions:

1. Could someone please explain to me the reason why someone would want a job to execute immediately when the maximum number of jobs has been acquired?

2. What would you suggest to wait before retrying a job? Somehow overriding a FailedJobCommandFactory?

Thanks for your time.

[sathish1]

Can u confirm if the jobExecutorActivate parameter is set to true?? My understanding the best practice is set to false when you are doing unit testing and any of the parameters including the waitTimeInMillis will be ignored. More from the user-guide here <link>http://www.activiti.org/userguide/#jobExecutorConfiguration</link> Cheers.

[martin_grofcik]

Hi,

1. Could someone please explain to me the reason why someone would want a job to execute immediately when the maximum number of jobs has been acquired?

In the case when maxJobsPerAcquisition were acquired we can expect that there are still another jobs to execute. that why we should check once again.

2. What would you suggest to wait before retrying a job?

It depends on your requirements and activiti implementation I would not recommend to set it <=1000 ms for activiti <= 5.15.1.
5.16 should be able to handle wait time < 1000 (https://github.com/Activiti/Activiti/pull/312). But again I would not recommend to use value < 500ms. (I does not make sense to set this value <1000ms for long running business processes). Another limitation is DB implementation of timestamp precision (e.g. older MYSQL).

Somehow overriding a FailedJobCommandFactory?

I do not understand the question.

Regards
Martin

[jsilva]

Thanks for the quick replies.

@sathish: The job executor is, indeed, on.

@martin.grofcik

In the case when maxJobsPerAcquisition were acquired we can expect that there are still another jobs to execute.

OK, so you want to retry the jobs as fast as possible in order to get to the other jobs, is that it? I can get that, but in some cases it increases the likelihood of the job failing yet again.

It depends on your requirements and activiti implementation I would not recommend to set it <=1000 ms for activiti <= 5.15.1.

5.16 should be able to handle wait time < 1000 (https://github.com/Activiti/Activiti/pull/312). But again I would not recommend to use value < 500ms. (I does not make sense to set this value <1000ms for long running business processes). Another limitation is DB implementation of timestamp precision (e.g. older MYSQL).

As mentioned in the first post, I'm trying to set mine to 60000, so none of that should be an issue. My issue is that since maxJobsPerAcquisition is 1, AcquireJobsRunnable never waits for the 60000 ms like I want it to – it retries the job immediately.

Somehow overriding a FailedJobCommandFactory?

I had an idea to create a Command that would somehow force the job to wait before retrying a failed job. Yes, I understand that this would block one of the JobExecutor's threads. I'm looking for a better way, though. Any ideas?

[martin_grofcik]

Hi,

Could you  share your process definition?
(Even better is whole jUnit test)

(One cause for immediate timer execution could be 312 pull request)

Regards
Martin

[jsilva]

At a first glance, it doesn't seem related to that pull request. Here's a unit test with the behavior:

http://speedy.sh/m7kCn/immediate-retry-unit-test.zip

[martin_grofcik]

Hi jsilva,

Thanks for your jUnit test. It helps me a lot to understand the problem.


Issue description:

1. Timer definition is <timeDuration>PT1S</timeDuration> it means timer should be fired 1sec after process deployment.
2. job executor wait time is 60sec. Usually timer is scheduled before jobexecutor run and job executor has to be notified about this event.
3. job executor notification is done (5.16-SNAPSHOT - it differs little bit from 5.15.1 (but logic is the same))

  public void pokeJobExecutor(JobEntity job) {
    JobExecutor jobExecutor = Context.getProcessEngineConfiguration().getJobExecutor();
    int waitTimeInMillis = jobExecutor.getWaitTimeInMillis();
    if (job.getDuedate().getTime() < (Context.getProcessEngineConfiguration().getClock().getCurrentTime().getTime()+waitTimeInMillis)) {
      hintJobExecutor(job);
    }
  }
and hint jobExecutor

protected void hintJobExecutor(JobEntity job) { 
    JobExecutor jobExecutor = Context.getProcessEngineConfiguration().getJobExecutor();
    JobExecutorContext jobExecutorContext = Context.getJobExecutorContext();
    TransactionListener transactionListener = null;
    if (job.isExclusive()
            && jobExecutorContext != null
            && jobExecutorContext.isExecutingExclusiveJob()) {
      // lock job & add to the queue of the current processor
      Date currentTime = Context.getProcessEngineConfiguration().getClock().getCurrentTime();
      job.setLockExpirationTime(new Date(currentTime.getTime() + jobExecutor.getLockTimeInMillis()));
      job.setLockOwner(jobExecutor.getLockOwner());
      transactionListener = new ExclusiveJobAddedNotification(job.getId());     
    } else {
      // notify job executor:     
      transactionListener = new MessageAddedNotification(jobExecutor);
    }

    Context.getCommandContext()
      .getTransactionContext()
      .addTransactionListener(TransactionState.COMMITTED, transactionListener);
      .addTransactionListener(TransactionState.COMMITTED, new JobAddedNotification(jobExecutor));
    }

ExclusiveJobAddedNotification adds job directly to the queue without checking the due time. There was fix proposed in pull request #312.

Regards
Martin

[trademak]

Hi,

We've done some refactoring to the job executor so that it takes into account the wait time when a job fails. The default wait time is set to 10 seconds. But you can override this in the ProcessEngineConfiguration with the defaultFailedJobWaitTime and asyncFailedJobWaitTime properties.

Best regards,

[sridhar_b]

Hi Core dev,

I'm facing the same issue. I'm using Activiti 5.12.

I have enabled the DefaultJobExecutor.  Also, added the property "waitTimeInMillis" to the bean and set its value to 10min.

With the above change, I see the Retry is not functioning correctly. I have a timer with delay of 5min between each failed Job. Once the job is failed, it retries immediately and doesn't wait for the timer.
Please help me in fixing this.
Thanks in advance.

Please see my configurations below.

<property name="jobExecutor">
<bean class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
<property name="queueSize" value="3"/>
<property name="corePoolSize" value="3"/>
<property name="maxPoolSize" value="10"/>
<property name="maxJobsPerAcquisition" value="3"/>
<property name="waitTimeInMillis" value="600000"/>
</bean>
</property>

[jsilva]

In my case, migrating to 5.16 solved the issue. A year-late thanks to Tijs and company!