Performance: Tests and Results (oh oh?!)

[luecking]

I've done some performance testing.

:idea: SZENARIO:
===========
The pure Activiti runtime engine used by a stand-alone application, no server, the in-memory-db, only one thread and a simple flow.
And i have a comparison to our own "old" runtime engine (C++ program), which stores it's own configuration in the windows registry.
The test happens on the same machine (Core2Duo 2GHz, 2GB)

I took a similar flow.
start = start event
end = end event
step = service script with a delegate class, which does nothing.
callAction
subTask
OR = Exclusive gateway

Because of initialization, I do not measure the first run.
I do 1000 runs to get a realistic value.
The results are in ms for _one_ run!

:!: RESULTS:
===========
(1)
start > step > OR > step > OR > step > [callAction: start > step > step > end] > [subTask: start > step > step > end] > [subTask: start > step > step > end] > step > end

Activiti: 130 ms

Based on that, I tried to figure out some details.

(2) Replacing the callAction by subTask.

start > step > OR > step > OR > step > [subTask: start > step > step > end] > [subTask: start > step > step > end] > [subTask: start > step > step > end] > step > end

Activiti: 120 ms


(3) Replacing the subflows by the steps directly.

start > step > OR > step > OR > step > step > step > step > step > step > step > step > end

Activiti: 26 ms


:!: CONCLUSION:
===========
CallActiviti & SubTask are very expensive (to expensive?)
It seems that every instance of a subTask will stored (serialized?) although it is not necessary in some scenarious.

Detailed results:
Step: 2 ms
Subflow: 30 ms
CallAction 40 ms

For the comparison.
Our own "old" runtime needs for scenario number (2):   0,33 ms      


:?: QUESTIONS:
===========
Are there any concepts, ideas, plans, … to optimize the performance of the runtime-engine?
Is it possible to return only the result of a subTask, without storing the whole instance of a subTask?

I'm looking forward for your ideas and remarks.
Thank you for reading and participating  

regards Oliver

[trademak]

Hi,

It's very difficult to respond to this comparison because you don't mention any details about your own C++ engine.
Does it for example keep a history of all executed steps?
Did you try to turn off the history storage?

You mention you tested a subTask, but what did you do exactly?
You are right that a sub task is stored separately in the database. But it's also meant for adhoc workflow.

So to summarize, it's very difficult to respond to your performance testing.

Best regards,

[luecking]

Hi,
thank you for your response… it's a good start for a discussion.

Here are some responses.

Does it for example keep a history of all executed steps?

No.
It is not necessary, because we later use external hardware, which works with own hardware related internal status information.
They will overwrite all flow related status information, so it will not be necessary to use this feature.

Did you try to turn off the history storage?

Is it possible?
Do you read about it or do you have examples?

You mention you tested a subTask, but what did you do exactly?

I defined a subTask inside the diagram and the subTask uses one start/end-event and two service-tasks with empty delegate classes.

You are right that a sub task is stored separately in the database. But it's also meant for adhoc workflow.

What do you mean with "adhoc" workflow in this context?

So to summarize, it's very difficult to respond to your performance testing.

Yes I now, but somewhere I had to start 😉


Just a remark:
I do not expect that the Activiti runtime engine will be faster than our specific opimized runtime engine.
But with a factor of 80 up to 400 the difference is much to big and therefore I'm looking for reasons and solutions.
Our whole process takes round about 30 seconds. Currently our engine produces with 100 steps nearly 33 ms.
If we use Activiti we need between 2600 ms and 13000ms. So the flow takes instead of 30sec now between 33 up to 43 seconds.

We would like to use Activiti because of the Designer, the BPMN standard and probably other possibilities (based on BPMN) to extend our workflow logic. Therefore we are able to accept a decrease of performance, but not in such a high number!


A base solution might be to overwrite the ExportMarshaller, so that he can include external callActions directly into one BPMN.xml file or to unpack all subTasks directly into the BPMN.xml, so that the exported file doesn't contain any subTasks.


thank you for your questions!

regards Oliver

[trademak]

Hi,

I know it's a bit blunt, but did you take the time to read the user guide or my Activiti in Action book?
Turning off the history storage saves quite a lot of performance and how you can do this is described in the user guide.

What you call a subTask is exactly an embedded sub process. Did you have sub processes in your own c++ engine as well? There's some additional process instance storage related to sub processes, so you'll always see a decrease in performance when using them.

If you want to go purely for performance then yes it would be a good idea to get rid of the call activities and embedded sub processes,

Best regards,

[luecking]

Hi,

…did you take the time to read the user guide…

Yes, but it is a lot to read and it is nearly impossible to understand everything in the first 5 minutes 😉
Is it possible to add a chapter regarding performance?
Possible content:
- History Service
- SubFlows/callActions
- a rough technical overview how the internal technology is working to understand how to solve performance issues in general.

The idea is, that depending on the necessary needs of a project, everyone can compare the increase of performance with the benefit of functionality.

I know that usual BPMN workflows (mixed with user tasks) do not have performance issues.
A colleague of mine was testing activiti server-side and they decided to move to an own specific flow engine, also because of performance doing many processes in parallel (main problem was also the subtask).
I try to use it inside a single threaded client application with limited hardware ressources.

It seems that we both have problems with the performance outside the user-interactions.

Turning off the history storage saves quite a lot of performance

Right, we do not need any history. Just pure running through the flow and forget everything… only the final result counts.
I have done some additional testing.

In case (1) and (2) the benefit of switching off the history will be round about 20-40 ms.
In case (3) the benefit will be  2-4 ms.

We are getting faster… step by step 😉

What you call a subTask is exactly an embedded sub process. Did you have sub processes in your own c++ engine as well?

Yes, we have sub processes in our C++ engine. They are designed in various diagrams (like callAction), but at least during init they will be stored at one integrated storage.
Keep in mind that we do not have to store already finished subflows. So our subflows do not have a performance difference between a number of steps inside or outside a subflow.

There's some additional process instance storage related to sub processes

right… that's I'm looking forward to … 😉
I just need one return value of a single step, I do need the storage of the whole instance (serialization?).
Do you know if optimizations at this point are possible?

so you'll always see a decrease in performance when using them

As I wrote before: I can live with a performance decrease, regarding the benefit of activiti, but it must not be to big 😉

If you want to go purely for performance then yes it would be a good idea to get rid of the call activities and embedded sub processes,

Is it possibly a good (optional) extension to the default designer exporter to include the content of external callActions directly into the calling flow (but not as an emebedded subTask!) ?

Thank you for interest!

Best regards,
Oliver