clustering with async tasks and the async executor

[amalinky]

Greetings! I'm using Activiti 5.17 with SpringBoot, using an external postgres DB. I have a bunch of established simple workflows that are all some number of Java service tasks in a row, like startEvent -> serviceTask1 -> serviceTask2 -> theEnd. These are submitted by an external scheduler that calls the rest API. So right now the entire workflow always executes on one node and in one transaction.

I want to migrate this to a clustered environment, looking to gain scale and robustness. If a workflow contains 3 java service tasks, I want them to be able to execute on three different nodes. IE - node1 executes task1 and commits transaction to the db. node2 acquires job from db, executes task2, commits transaction to db, and so on.

I've got a test environment with two identical instances of my springboot application, both pointing at the same postgres db. I made a test workflow that calls a very simple java service task 5 times, adding activiti:async="true" to each service task with the intent of forcing a transaction boundary and allowing other instances to pick up the job of the next task. The task just reads and increments an execution variable so I can verify that they're not stepping on one another.

To test, I'm submitting 100 workflows to ONE of my two nodes. The BPMN looks like this:

<process id="AsyncPOCWF" name="Async POC WF">

        <serviceTask id="task-1" name="Task 1" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-2" name="Task 2" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-3" name="Task 3" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-4" name="Task 4" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-5" name="Task 5" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />

        <startEvent id="theStart"/>
        <sequenceFlow sourceRef="theStart" targetRef="task-1"/>
        <sequenceFlow sourceRef="task-1" targetRef="task-2"/>
        <sequenceFlow sourceRef="task-2" targetRef="task-3"/>
        <sequenceFlow sourceRef="task-3" targetRef="task-4"/>
        <sequenceFlow sourceRef="task-4" targetRef="task-5"/>
        <sequenceFlow sourceRef="task-5" targetRef="theEnd"/>
        <endEvent id="theEnd"/>
</process>

And my success scenario is if the service tasks begin executing on both nodes, not just the one where I submit the workflow.

This WORKS - but only if the async executor is disabled - with these options:
                    SpringProcessEngineConfiguration.setJobExecutorActivate(true);
                    SpringProcessEngineConfiguration.setAsyncExecutorActivate(false);
                    SpringProcessEngineConfiguration.setAsyncExecutorEnabled(false);

If I disable async executor (and enable jobexecutor), all the execution happens on only one of the two nodes.
                    SpringProcessEngineConfiguration.setJobExecutorActivate(false);
                    SpringProcessEngineConfiguration.setAsyncExecutorActivate(true);
                    SpringProcessEngineConfiguration.setAsyncExecutorEnabled(true);

My async executor options are:

asyncExecutor.setAsyncJobLockTimeInMillis(86400000);
asyncExecutor.setCorePoolSize(2);
asyncExecutor.setMaxPoolSize(3);
asyncExecutor.setMaxAsyncJobsDuePerAcquisition(1);
asyncExecutor.setDefaultAsyncJobAcquireWaitTimeInMillis(10000);                    asyncExecutor.setDefaultTimerJobAcquireWaitTimeInMillis(10000);


My question is - why does this happen? Are there other config switches I can throw on the async executor that will help? My hunch was that the async executor was "too performant" for the second node to have any remaining work slices that the second node can acquire, so I played around with moving these sliders up and down to try and "slow down" the first node - no such luck.

Many thanks in advance!

[amalinky]

My XML got eaten

Here's the test BPMN referred to above (hopefully):

<process id="AsyncPOCWF" name="Async POC WF">

        <serviceTask id="task-1" name="Task 1" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-2" name="Task 2" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-3" name="Task 3" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-4" name="Task 4" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />
        <serviceTask id="task-5" name="Task 5" activiti:class="com.blahblah.AsyncPOCTest" activiti:async="true" />

        <startEvent id="theStart"/>
        <sequenceFlow sourceRef="theStart" targetRef="task-1"/>
        <sequenceFlow sourceRef="task-1" targetRef="task-2"/>
        <sequenceFlow sourceRef="task-2" targetRef="task-3"/>
        <sequenceFlow sourceRef="task-3" targetRef="task-4"/>
        <sequenceFlow sourceRef="task-4" targetRef="task-5"/>
        <sequenceFlow sourceRef="task-5" targetRef="theEnd"/>
        <endEvent id="theEnd"/>
</process>

[jbarrez]

The 'problem' is that the Async executor will continue to take on work on the same, unless its queue is full …

so if you give it more work … it'll not be able to cope. And I think the default is 100 indeed 😉
Or you can make the queue size smaller to force that behaviour.