Activiti on Websphere and Spring AsyncJobExcutor

[ggr]

Context : Make long story short, it has been decided in my company to use Activiti, we have strong knowledge in EMC Workflow Engine. We have deployed it in a hurry in production on IBM Websphere Application Server.
Not enough load/perfomance tests, the inevitable happens : the server crashed.

After some investigations, blog posts & forums readings we found a solutions.

[h2]1. Work Manager et Timer Scheduler[/h2]

Managing your own thread on a WAS is never a good idea. Even if the new class ManagedAsyncJobExecutor use a threads pool of the container this is not suitable for the WAS.
It use work manager - a pool of threads - and managed it by itself.

I re-wrote the existing DefaultAsyncJobExecutor using the spring class org.springframework.core.task.AsyncTaskExecutor, it works like the java java.util.concurrent.ExecutorService.
This interface is mainly implemented by two classes

• one for a Servlet Container : org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor
• one for a Applicaton Container : org.springframework.scheduling.commonj.WorkManagerTaskExecutor (needs the jndi name of the work manager)

   
The AbstractAsyncJobExecutor starts two threads two retrieve jobs to treat. Regarding the code, this is just a runnable snippet that is scheduled.

I rewrite the two classes AcquireAsyncJobsDueRunnable & AcquireTimerJobsRunnable, the common code (99%) has been mutualized in AbstractAcquireJobsRunnable.
I use the time scheduler to avoid Thread.sleep(). I use the Spring implementation because of the WAS, but a standard java one does exist java.util.concurrent.ScheduledExecutorService.
I think this is better to use this than a thread.

The philosophy is still the same :

• The Async executor starts schedule of execution of the AcquireAsyncJobsDueRunnable & AcquireTimerJobsRunnable
• When the scheduler is executed, two tasks are executed by the AsyncTaskExecutor - one for each runnable
• They check the database to retrieve jobs and put runnable task for each job in the AsyncTaskExecutor.
• if there is no more job to treat or if the queue of the AsyncTaskExecutor is full, the AbstractAcquireJobsRunnable schedule its next execution.

I'm a little bit annoyed with the term AsyncJobExecutor because it does more than just executing job. It rerieves jobs to treat and then only executes them.
It is more a AsyncJobExecutorManager or something like that.

One more strange thing is that AcquireAsyncJobsDueCmd constructor requires a AsyncJobExecutor object. In fact, it only needs lockTimeInMillis and maxJobsPerAcquistions params.

Globally, the rewriting of the classes is more OOP / Spring compliant.

[h2]2. Transaction Manager[/h2]

We use the inappropriate transactionManager : org.springframework.jdbc.datasource.DataSourceTransactionManager.
For WAS, you have to use : org.springframework.transaction.jta.WebSphereUowTransactionManager


[h2]3. Websphere Application Server Config[/h2]

Define a workmanager :

• name : workmanager/listener
• Work request queue size　: 0
• Work request queue full action　: FAIL
• Alarm threads : 15
• Minimum threads : 10
• Max threads : 200
• Threads priority : 5

Define a scheduler :

• name : timer/listener

[h2]4. Activiti 6 and message queue based async executor[/h2]

The mechanism explained below is applicable for Activiti 6 OOTB.

Following this post http://www.jorambarrez.be/blog/2016/07/20/message-queue-async-executor-benchmark/, we can adapt the mechanism like this based on the producer / consumer pattern.

• Use a scheduler to schedule tasks to retrieve jobs
• job are put in a queue (java, jms, whatever.)
• use a second scheduler to read this queue and put the job in the workmanager

I dont know if it will be more efficient, but it a regulate the load of the executor

Feel Free to ask more details of the solutions.

Thanks for providing such a great product.


FIles in attachment, rename it to codE.zip

[jbarrez]

Hi GGR, thanks for the detailed post, really interesting!

1. Wouldn't it be easier to use the org.activiti.spring.SpringAsyncExecutor here? It seems you're using Spring anyway. This SpringAsyncExecutor expects a taskExecutor to be injected (which maps to what you have). or is this how you are using it already?

2. and 3. thanks for posting that. useful for many others.

4. I'm not sure yet how this would regulate the load of the executor better/differently?
Currently, the thread based solution uses a blockinqueue, when it's full the acquiring will also be throttled.
In general, the current implementations works pretty much like you describe (except a second scheduler is not used, it's passed to the queue itself).

So am I right that maybe you're looking for different/interfaces here?

> Thanks for providing such a great product.

Many thanks!

[ggr]

HI,

1. Of course, it will be easier, but you cant retrieve the threadpoolexecutor of a WAS.
We are using a WAS because of the organization policy, but I am not convinced that this is safer/robust/scalable than a tomcat.

4. Two point for this :
Considering the use of Thread.sleep() / scheduling of task
If you a have only a limited number of threads to work with - 10, 3 will be allocated to retrieve the jobs and 7 will process the jobs.
If the queue is full, the 3 retrieving job will be sleeping instead of processing jobs.

Regulation of the load

I do not know - because I do not have testing enough - if the regulation will be better.
I have studied your "queue based async executor" of v6, and I think this a really good solution.
Indeed, we plan to use it : we want to split the execution of job by process to mitigate the risk.
Architecture will be something like :
* one oracle DB
* one server to retrieve jobs
* As many queues/JMS as processes definition
* at least one job executor per queues.

The only things that bother me, is that there is not enough abstraction, and the standard implementation & the jms executor one does not share the same interfaces/abstract class.
On the other hand, I understand the legacy and the need that the engine can be started with the less configuration possible.