Activiti performance testing results, XA trans question

[bwestrich]

Hello Activiti users,

We've been trying out Activiti for the past few months and think it works great in general.

We've just finished our performance testing of it as well (with an Oracle database), and it performed reasonably well when using Oracle's non-XA driver. For example, in one of our tests we were able to perform around 500 transitions a second). 

Before testing, we followed some of the suggestions in this excellent forum post and its replies that was started by romanoff:
http://forums.activiti.org/en/viewtopic.php?f=6&t=1523

For example, we bumped up the id block size as follows:
<property name="idBlockSize" value="10000" />

And we also added a few indexes to the database……..
create index IDX_ACT_HI_ACTINST on ACT_HI_ACTINST(EXECUTION_ID_, ACT_ID_);
create index IDX_ACT_HI_PROCINST on ACT_HI_PROCINST(SUPER_PROCESS_INSTANCE_ID_);

I don't recall off the top of my head doing anything else related to performance tuning.

Based on these results, we're pretty sure Activiti will meet our needs related to performance.

But curiously, when we switch to the Oracle XA driver (not sure if we will use XA yet) (the driver name is oracle.jdbc.xa.client.OracleXADataSource) the performance decreases about ten fold. Our performance architect suspects "chattiness" between Activiti and the database. Has anyone else come across slow performance when using Activiti with XA?

Brian

P.S. Tests were run against an Oracle server with a single RAC node, with Java code executing inside a WebSphere 7 application server running on Solaris.

[bwestrich]

Another important environmental thing I should mention: we are using history_level = "all".   We need to use this as some of our database queries access context variables on process instances that have already ended.

[ronald_van_kuij]

Our performance architect suspects "chattiness" between Activiti and the database

If he is an architect, can't he actually monitor/analyze this? Or do some profiling? That way it is not 'suspects' anymore… If just changing a driver and not actually needing/doing xa it would strike me as odd that chattiness is the cause. Wouldn't that be the case otherwise as well?

[bwestrich]

Hi Ronald,

Thank you for your reply. I followed up with our architect and have more details to provide on how our application uses XA, as well as some more specifics on our performance test findings and interepretation.

We're interested in XA because our Activiti processes not only do database operations but also send out JMS messages: if there's a database error along the way, we don't want any of the JMS messages to go out. We've coded the application in a way that makes such consistency reasonably certain (i.e. we batch up the JMS messages and don't send them until the very end), but are considering XA as a way to simplify this aspect of our application code. 

We had previously configured our JMS resources to participate in XA, so when we changed from a non-XA JDBC driver to an XA JDBC driver, the database operations were then without any further configuration changes enlisted in the same global transaction as was the JMS operations.

Here's more specifics on what the architect observed on the Oracle server during our performance testing:

When not using XA, about 1/2 the time was being spent in CPU and the other half was spent doing commits. This is likely because Activiti is by design (as a workflow engine, not a batch processing engine) submitting large numbers of small transactions (i.e. one transaction for each state transition).  We're not seeing this as a deficiency in Activiti, just an operating characteristic. As I mentioned before, we are in general very pleased with the performance (and other aspects) of Activiti.

In contrast to the non-XA results, when using XA, very little time was spent doing commits, and most of the time was spent in concurrency wait operations. Why this difference between XA and non-XA? Here's one theory: In the architect's experience, she's seen a JMS message to take about 30msec to send, a fairly long time compared to operations such as a JDBC commit. In a non-XA environment, this work can be done in parallel with the database commits. But in an XA environment, these JMS operations are committed serially with the JDBC commits, which would lead to lower throughput.  This change from parallel to serial processing may well explain the above changes in resource utilization that she observed.

Hopefully the above information sheds some light on your questions. 

I'm still interested in any performance findings that you or anyone else may have on using Activiti with XA, so as to judge the representativeness of our team's findings.

Thanks again,

Brian

[heymjo]

In weblogic you can have a non-xa datasource 'participate' in an xa transaction (described here http://docs.oracle.com/cd/E15051_01/wls/docs103/jta/llr.html). It would be interesting to see if this would remove the overhead you perceive (provided ofcourse websphere has a similar option).

Also, efficient jdbc batching can eliminate a lot of the xa overhead. Looking at the mybatis docs (http://www.mybatis.org/core/java-api.html , look for SqlSessionFactory) it seems that one of the variants of openSession() namely openSession(ExecutorType execType) allows you to enable jdbc batching. A cursory look at the activiti sources reveals that only the basic openSession(connection) is used so it would seem as if  batching optimizations are never performed. Again it would be interesting to see the effect on performance for your case if you change the code to ExecutorType.BATCH there.

As you can tell we are quite interested in this topic as well for our future apps, so please continue to share your findings here

[bwestrich]

Hello heymjo,

Thanks for your interesting reply.

Related to app server extensions to non-XA datasources, the LLR feature of WebLogic looks to be very robust (judging by the info in the link you posted). We use WebSphere, which has a feature called Last Participant Support (LPS). Prompted by your reply, we took a closer look at it and LPS appears to be less robust than LLR and have some limitations (in comparison with XA) that we aren't very comfortable with.  In short, with LPS if the non-XA resource never replies to the transaction manager as to whether it committed or not (for example, due to a database lock or a long running transaction), WebSphere will use a heuristic (guess) when deciding whether to rollback or commit the XA resources.  More info is here: http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.d.... So using LPS may not be a viable option for us. 

Related to batching in mybatis, it looks like this would only require changing a single line of code in the Activiti source code (in DbSqlSession). However, there are a couple things that make me reluctant to try this change:
1. my client is very squeamish about using modified code from open source projects
2. I wonder how many "batchable" statements Activiti includes in each transaction.  If there are a large number of statements going against the same table for example, this change would have a larger potential benefit. But a quick glance at the Activiti schema suggests that within each transaction (in our case, a state transition) there would only be a couple rows inserted/updated, and each of these rows would be for a different table. So I suspect the improvement from batching might be minimal (or could even slow down performance slightly due to the overhead of batching the statements). But I'm really just guessing here, if it were easy for me to try this out in my current environment I would seriously consider doing so. Perhaps you or someone else might be able to?

Thanks again for your insights.  Again, I want to stress that we are in general very happy with Activiti (including its performance) so far.

Brian

[ronald_van_kuij]

submitting large numbers of small transactions (i.e. one transaction for each state transition).

Not necessarilly, depends on the design of your flow. Before each wait state a commit is done, but if you have multiple non-async service tasks and e.g. no usertask in between, the number of transactions is lower.

JMS message to take about 30msec

In JBoss 5 and 6, JMS is not fully XA, but a 'wrapper'  around a normal connection…!!! Might be one of the causes of this increased delay as well.

1. my client is very squeamish about using modified code from open source projects

Why don't you give it a try? If it helps, I don't think the Activiti team is reluctant to incorporate this e.g. as a switch in the config.

And yes, thanks for the detailed analysis. Might also be interesting if someone coul test this against a different DB…