Jobexecutor tuning

[sazzadul]

Hi,

     We are currently running activiti(version 5.10) in production with 3 instances of jobexecutor and with following configuration, database is oracle but the jobexecutor is not scaling upto our needs.
We have to create thousands of process instancess per hour.

<bean id="myJobExecutor" class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
      <property name="queueSize" value="30" />
      <property name="corePoolSize" value="30" />
      <property name="maxPoolSize" value="100" />
      <property name="maxJobsPerAcquisition" value="30" />
      <property name="waitTimeInMillis" value="1000" />
         <property name="lockTimeInMillis" value="180000"/>
     </bean>

   <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
         <property name="dataSource" ref="dataSource" />
         <property name="transactionManager" ref="transactionManager" />
         <property name="databaseSchemaUpdate" value="true" />
         <property name="jobExecutorActivate" value="true" />
         <property name="jobExecutor" ref="myJobExecutor" />
        <property name="history" value="activity" />
        <property name="idGenerator">
         <bean class="org.activiti.engine.impl.persistence.StrongUuidGenerator" />
        </property>
        <property name="idBlockSize" value="2000" />
   </bean>

I am not seeing anything abnormal in the output log neither any memory usage issue. Is the configuration good enough ? Is there anything to tune in the database side like for instance adding indexes ?

It looks like jobexecutor suddenly becomes very slow.

Any help will be greatly appriciated

[trademak]

Hi,

You can certainly optimize it by adding indexes. Oracle has tools to give you advise on that, right?
Do you know for sure that the issue lies with the job executor?
Are you starting synchronous or asynchronous processes? Does the job wait before the process is completed or do you have an async task in between?

Best regards,

[sazzadul]

Thank you for your reply.

All our processes consist of many subtasks and each subtask is asynchronous process(javadelegate) which simply call webservice function(the webservice call is very fast so the problem doesn't lie here). Other than subtasks we havn't anything else.

I am not sure if its database related, I should perhaps ask our dba to do an analyse.

It seems like the jobexecutor is very slow to move steps in a process. Is there any waiting in between the steps ?

Hi,

You can certainly optimize it by adding indexes. Oracle has tools to give you advise on that, right?
Do you know for sure that the issue lies with the job executor?
Are you starting synchronous or asynchronous processes? Does the job wait before the process is completed or do you have an async task in between?

Best regards,

[trademak]

Hi,

You have set the max acquisition to 30. This means that if there are 30 jobs in the database it will retrieve all these 30 jobs, and the other job executors do the same.

What's causing the issue is that the jobs are running in exclusive mode by default. This means that the jobs are executed in sequence rather than in parallel.
So even if the web services are really fast, it piles up. What you can do to solve this is to add an attribute of exclusive="false" to the async task.
For more details, see here:

http://activiti.org/userguide/index.html#exclusiveJobs

Best regards,

[sazzadul]

Thanks for your sugggestion but  I wonder if it really would help me. All the steps in our process is dependent on each other which means stepA must finish before stepB can start. So do you think using exclusive=false on stepA and stepB will solve my problem ?

[trademak]

Hi,

I don't know how you implemented your process definition of course.
But if step A is an async service task and you have defined step B sequentially as another async service task with a sequence flow between step A and B it's just fine to set exclusive to false.
But after rethinking this a bit more, this will not solve a lot. Non-exclusive jobs are only more performant when you have multiple jobs running for the same process instance.
It would do however no harm to set it to exclusive false anyway.

Let's go back to your configuration again. You have a core pool size of 30 and max pool size of 100, which means that 30 simultaneous threads are always kept and a maximum of 100 threads are executed.
Is your environment able to execute 100 thread simultaneously in a performant way. Do you see CPU pile up to 100% for example?

In addition you have defined the acquisition size of 30. This means that every time 30 jobs are executed by the same thread. Since you have defined a lot of simultaneous threads, I would say it would be better to decrease this size to for example the default size of 3. Then you would make a lot more use of the thread pool.

Best regards,

[sazzadul]

Hi,

I have now adjusted the parameter values for jobexecutor after suggestion from you which looks like:

<bean id="myJobExecutor" class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor">
    <property name="queueSize" value="30" />
    <property name="corePoolSize" value="30" />
    <property name="maxPoolSize" value="50" />
    <property name="maxJobsPerAcquisition" value="3" />
    <property name="waitTimeInMillis" value="1000" />
       <property name="lockTimeInMillis" value="60000"/>
   </bean>

and only running one instance of jobexecutor since I found out that running several instances didn't have any improvements.

I am still facing with performance issues, whenever the load increaces(say 400 process instances per hour) the jobexecutor becomes slower and slower.

Under minor load(say 50 prosess instances per hour) its working as expected.

I haven't tried with the exclusive flagg. I have a lot of subprocesses and each process calls atleast 15-20 subprocess and wonder if this might causing the problem ??

What is a little strange that under minor load everthing works perfectly. Under load the process takes upto several minutes(5-10 minutes) to complete whereas it only taks 10 seconds under minor load.

[trademak]

Hi,

I really need more information from you to be able to analyze this further.
Do you see that the job table is exploding with jobs?
How do you know that the job executor is the bottleneck?
Do you see CPU go to 100%?
Are all these sub process calls done in sequence with an async attribute? Can you attach your process definition?

Best regards,