Jobexecutor does not cope with simultaneousness

[sazzadul]

Hi,

I wanted to share an experience with you and hopefully find a solution for it too.

It seems like Jobexecutor simply goes into hibernation when it gets to much to do.

Here is the scenario…
I have a process with a few servicetasks(async) and a user task with timerboundaryevent which was a duedate say 15 minutes ahead.
Now when I start a number of processes (say 100) they all enter the user task and waits for the duedate to expire.
So far so good, but as soon as the duedate expires the jobexecutor kicks in and it begins to work, I can see in the RU_JOB table that it updates the lock column for a few rows but thats basically it. The jobexecutor hereafter doesn't do a single thing, it totally stops. What is even worse it stops processing the new processes as well.

I have tried to increase the maxpoolsize, waitTimeInMillis without any luck. I have even tried a customjobexecutor(as suggested in this http://forums.activiti.org/en/viewtopic.php?f=6&t=3523&hilit=jobexecutor) which uses commonj but the result is same.

I am using jetty with Oracle, have also tried postgres and mssql2008 the result is same.
There is no exception whatsoever being thrown from activiti so I have no clue where the problem lies.

Is there any bug in the jobexecutor ? I hope I have made myself clear enough to reproduce this error.

Thanks.
Sazzadul

Jobexecutor does not seem to tackle simultaneousness at all.

[houssam]

still facing the same issue with 5.14 version, any suggestion?

[trademak]

Could you explain what exactly you are facing? Please explain with some background about how you are using jobs etc.

Best regards,

[erny18031]

Hi.
Just a guess: Is there are remote possibility that the job executor threads are not returned to the thread pool after job execution errors, or that the returned threads are not functional anymore?

Regards.

[trademak]

It's implemented in a way that this shouldn't happen, but there can always be a bug of course.
That's why I'm asking for more information to be able to dive into the exact issue in more details.

Best regards,

[huco]

Hello.

We are experiencing similar issues with job executor. We are running 10 service tasks at most at the same time and sometimes the transition from completed script task (which precedes service task) to service task takes more than an hour and a half. We are using activiti 5.12 in our web app and after the script task was completed, the following exception was thrown:
<code>
org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:62)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "Thread-156" java.lang.NoClassDefFoundError: org/activiti/engine/impl/jobexecutor/ExecuteJobsRunnable
at org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:62)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
… 3 more
</code>

Hour and a half later service task started it's execution and completed successfully. We are not using any timers in our process.

Data recored in  act_hi_actinst for that particular case:
<code>
"Script Task";"scriptTask";"";"2014-01-22 00:53:34.307";"2014-01-22 00:53:34.318";11
"Unlock";"serviceTask";"";"2014-01-22 02:33:38.991";"2014-01-22 02:33:39.048";57
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:39.048";"2014-01-22 02:33:39.049";1
"Receive Task";"receiveTask";"";"2014-01-22 02:33:39.049";"2014-01-22 02:33:41.951";2902
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:41.951";"2014-01-22 02:33:41.952";1
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:41.952";"2014-01-22 02:33:41.952";0
"End";"endEvent";"";"2014-01-22 02:33:41.952";"2014-01-22 02:33:41.952";0
</code>

In a few cases the transition never happend and service task never actually started. Such service tasks are present in act_ru_execution table but nothing is happening.
<code>
"servicetask3";t;f;t;f;1;4
</code>

Any idea what's causing these issues?

Best regards

[frederikherema1]

That's really strange… In case there is a "finished" entry in ACT_HI_ACTINST, the service-task did run and the transaction was comitted. How do you know the service-task wasn't run?

Seems like the error occurs when jobs are still being executed, but the server is going down.

[huco]

Hi.

Right at the start of service task we are printing progress lines to logger and catalina and nothing gets recorded.

We've upgraded to 5.14. version but it looks like the problem is still present. When time came to start the Service Task, the following exception was thrown:
<code>
org.activiti.engine.ActivitiException: Job 166561 failed
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:83)
        at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:37)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.activiti.engine.ActivitiException: couldn't instantiate class com.activiti.UnlockServiceTask
        at org.activiti.engine.impl.util.ReflectUtil.instantiate(ReflectUtil.java:132)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.instantiateDelegate(ClassDelegate.java:163)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.getActivityBehaviorInstance(ClassDelegate.java:136)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.execute(ClassDelegate.java:113)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:45)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:52)
        at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:36)
        at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:79)
        at org.activiti.engine.impl.persistence.entity.MessageEntity.execute(MessageEntity.java:29)
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:71)
        … 9 more
Caused by: org.activiti.engine.ActivitiClassLoadingException: Class not found: com.activiti.UnlockServiceTask
        at org.activiti.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:82)
        at org.activiti.engine.impl.util.ReflectUtil.instantiate(ReflectUtil.java:129)
        … 31 more
Caused by: java.lang.ClassNotFoundException: com.activiti.UnlockServiceTask
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.activiti.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:63)
        … 32 more
</code>
There were 80 other process instances which successfully completed that service task.

Any clue why this one failed?

Thanks

[jbarrez]

Absolutely no clue … if 80 instances can find the class … there is no reason why the 81st should fail :s