Slow multi-threaded execution due to oversynchronized blocks

[romanoff]

Hi,

I have written a simple standalone test application (a JUnit test) that starts 5000 Runnable tasks on a j.u.c pool of 50 threads. Each task starts and executes the same simple BPMN process. H2 backend is used. There are no jobs or human tasks created during the execution of this process. Since processes are very simple and make no external invocations, they are supposed to be executed very fast. Main aim of this test is to see how fast Activiti can be if used without jobs, tasks and with disabled history, because I mostly need this mode of operation for my soft real-time use-case.

When I measured the performance of my application I noticed that it is rather slow for some reason. So, I decided to run it under a profiler. Profiler has discovered that a lot of time (>50%) is due to thread contention, because many of the worker threads are waiting for entering a critical section at a few places in the Activiti engine. It indicates that something is oversynchronized. Overall, the (ibatis-based) DB-backend seems to be the bottleneck.

Specifically, the ibatis runtime seems to be oversynchronized in the following two methods:
org.apache.ibatis.datasource.pooled.PooledDataSource.popConnection(String, String)
org.apache.ibatis.datasource.pooled.PooledDataSource.pushConnection(PooledConnection)

They synchronize on rather big blocks of code. And interestingly enough, the thread contention gets even worse when I increase the number of threads in my thread pool.

Another blocker is this method:
org.activiti.engine.impl.db.DbIdGenerator.getNextId()
Here we have:
  public synchronized long getNextId() {
    if (lastId<nextId) {
      getNewBlock();
    }
    return nextId++;
  }

I'm not an expert, but may be this synchronized block can be replaced by means of using AtomicLong or AtomicInteger?

Questions:
- Are these findings of a generic nature or is it only me, who sees this problem?
- Is there anything obviously wrong with my test, e.g. RuntimeService.startProcessInstanceByKey() is not supposed to be used from many concurrent threads at the same time?
- Is it possible to completely eliminate a need for a DB backend? I already switched the history off, but I still see quite some DB activity.

Best Regards,
  Leo

[frederikherema1]

Hi,

By default (not using spring datasource and not having altered configuration), the ibatis connection-pool size is fixed to 10. Did you alter the pool size? If not, locked threads makes perfect  since the thead puttrough are trottled by this limit…

Limit can be altered using property 'jdbcMaxActiveConnections' on activiti cfg. Relevant code responsible for create pooled dataSource (ProcessEngineCOnfigurationImpl).

PooledDataSource pooledDataSource =
          new PooledDataSource(ReflectUtil.getClassLoader(), jdbcDriver, jdbcUrl, jdbcUsername, jdbcPassword );
       
        if (jdbcMaxActiveConnections > 0) {
          pooledDataSource.setPoolMaximumActiveConnections(jdbcMaxActiveConnections);
        }
        if (jdbcMaxIdleConnections > 0) {
          pooledDataSource.setPoolMaximumIdleConnections(jdbcMaxIdleConnections);
        }
        if (jdbcMaxCheckoutTime > 0) {
          pooledDataSource.setPoolMaximumCheckoutTime(jdbcMaxCheckoutTime);
        }
        if (jdbcMaxWaitTime > 0) {
          pooledDataSource.setPoolTimeToWait(jdbcMaxWaitTime);
        }

If this isn't the issue, consider using DBCP-datasource pool, which is a proven pooling lib, instead of the built-in (myBatis).

[romanoff]

Hi,

By default (not using spring datasource and not having altered configuration), the ibatis connection-pool size is fixed to 10. Did you alter the pool size? If not, locked threads makes perfect  since the thead puttrough are trottled by this limit…

Limit can be altered using property 'jdbcMaxActiveConnections' on activiti cfg.

Relevant code responsible for create pooled dataSource (ProcessEngineCOnfigurationImpl).

PooledDataSource pooledDataSource =
          new PooledDataSource(ReflectUtil.getClassLoader(), jdbcDriver, jdbcUrl, jdbcUsername, jdbcPassword );
       
        if (jdbcMaxActiveConnections > 0) {
          pooledDataSource.setPoolMaximumActiveConnections(jdbcMaxActiveConnections);
        }
        if (jdbcMaxIdleConnections > 0) {
          pooledDataSource.setPoolMaximumIdleConnections(jdbcMaxIdleConnections);
        }
        if (jdbcMaxCheckoutTime > 0) {
          pooledDataSource.setPoolMaximumCheckoutTime(jdbcMaxCheckoutTime);
        }
        if (jdbcMaxWaitTime > 0) {
          pooledDataSource.setPoolTimeToWait(jdbcMaxWaitTime);
        }


Yes, I already tried to increase jdbcMaxActiveConnections to 100, which should be more than enough with only 50 worker threads in the thread pool. It helped a bit, but not much.

If this isn't the issue, consider using DBCP-datasource pool, which is a proven pooling lib, instead of the built-in (myBatis).

Thanks! This is an interesting hint. How can I replace the myBatis datasource pool via the DBCP pool? Can it be done purely via a configuration or in the client code? Or do I need to alter the Activiti code in the ProcessEngineCOnfigurationImpl?

[romanoff]

Thanks! This is an interesting hint. How can I replace the myBatis datasource pool via the DBCP pool? Can it be done purely via a configuration or in the client code? Or do I need to alter the Activiti code in the ProcessEngineCOnfigurationImpl?

OK. I figured out myself how to do it by reading the user's guide 😉

[jbarrez]

Interesting findings. The synchronized block you see if to allocate a block of ids for the current Activiti engine. In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Keep us tuned about your findings with the DBCP datasource!

[romanoff]

Interesting findings. The synchronized block you see if to allocate a block of ids for the current Activiti engine. In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Keep us tuned about your findings with the DBCP datasource!

If you ask for it, I continue with my reporting 😉

After I switched to the DBCP, I've seen that thread contention due to the thread-pooling has disappeared. Which means that myBatis is really bad for pooling DB connections.

After that problem was solved, org.activiti.engine.impl.db.DbIdGenerator.getNextId() became the main reason of thread contention.

In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Once I did it, org.activiti.engine.impl.db.DbIdGenerator.getNextId() did not appear on the radar any more.

What I observe now is the following: there is no thread contention as such, but there are too many database queries being issued by Activiti. Profiler shows that they consume at least 50-70% of overall time. And almost all of these DB queries are from the ExecutionEntity.remove(), where we have the following snippet:

    // delete all the tasks
    CommandContext commandContext = Context.getCommandContext();
    List<TaskEntity> tasks = (List) new TaskQueryImpl(commandContext)
      .executionId(id)
      .list();
    for (TaskEntity task : tasks) {
      if (replacedBy!=null) {
        task.setExecution(replacedBy);
      } else {
        task.delete(TaskEntity.DELETE_REASON_DELETED);
      }
    }

    List<Job> jobs = new JobQueryImpl(commandContext)
      .executionId(id)
      .list();
    for (Job job: jobs) {
      if (replacedBy!=null) {
        ((JobEntity)job).setExecution((ExecutionEntity) replacedBy);
      } else {
        ((JobEntity)job).delete();
      }
    }

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Please let me know if any improvements are possible. Or if you want me to check something related to the mentioned issues.

[jbarrez]

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Wow great findings! I had already forgotten about that property!

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Yes- I came to the same conclusions a few months ago when I did some profiling.
I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

The stuff you've found is really good stuff to put into a blog (with cool charts ;-), do you have any plans to do such a thing?

[romanoff]

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Wow great findings! I had already forgotten about that property!

It would be nice if you'd update the documentaton and mention it somewhere as well as the fact that the default myBatis config does not scale when you use many concurrent threads.

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Yes- I came to the same conclusions a few months ago when I did some profiling.
I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

Do you have any concrete ideas about when it could be done? It would be nice if it would be fixed rather soon and I guess the fixes are rather simple. The raw performance of the engine for the scenarios like I described in the initial message of this thread is a very important criteria for selecting a concrete engine to be used in a bigger project on our side. Would be nice if I could report something positive and backed by the figures to my management 😉

The stuff you've found is really good stuff to put into a blog (with cool charts ;-), do you have any plans to do such a thing?

I'd like to. But due to the different internal policies of my employee I may not be allowed to do it ;-( In any case, I could provide a source code of the test, so that it can be re-used by users of Activiti or even included into the Activiti's unit/performance tests.

I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

I'm just wondering if this approach would lead to any problems in the cluster setup. I don't know if it is possible at all, but what would happen if other nodes may potentially add tasks/jobs to the process instance running on the current node. If this is possible, then it is difficult to avoid quering the DB. But even in this case, may be some sort of indications (in the process description or in startup configuraton parameters) can be introduced that this process instance will not get new tasks or jobs from other nodes. More over, there is already a flag for disabling job scheduler, IIRC. If this one is disabled (which is a rather radical step ;-), then there can be no jobs or tasks in the DB anyway, or? So, no need to check the DB for them anyway.


Question:
Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

[jbarrez]

Tom is currently working on refactoring the persistence (because it's a mess at some places).
But I doubt that the issue you describe will be fixed soon - as our focus is currently not around persistence. However, if a big customers pops up with this problem, then it will be fixed soon 😉

Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

The services are tightly coupled to BPMN. It is better to use the PVM API as-is. Do note that the PVM API is quite low-level compared to the services.

What do you mean with domain-specific stuff? Or better: why would you want it (now that BPMN is a industrty-adopted standard)?
You could always do stuff like we do with KickStart, where we offer a very simple UI that allows to place some tasks sequentially or parallel, which is then translated to BPMN using JAXB.

[romanoff]

Tom is currently working on refactoring the persistence (because it's a mess at some places). But I doubt that the issue you describe will be fixed soon - as our focus is currently not around persistence. However, if a big customers pops up with this problem, then it will be fixed soon 😉

I see.

Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

The services are tightly coupled to BPMN. It is better to use the PVM API as-is. Do note that the PVM API is quite low-level compared to the services.

Interesting. I hoped that PVM can handle such services like persistence of definitions, jobs (for timers and async continuations) and keeping a history almost independent of the language layer built on top of it. This would greatly reduce the amount of efforts to develop any new languages on top of the PVM. Essentially, you'd just need to define the syntax, a mapping from this syntax to the PVM model and its execution semantics.
But if I understand you correctly, this is not the case. Every language built on top of the PVM needs to implement e.g. its own deployment, history and job/scheduler services. Thus, PVM is mostly used to represent the run-time in-memory model of the process to be executed. Is it correct a correct understanding?

What do you mean with domain-specific stuff? Or better: why would you want it (now that BPMN is a industrty-adopted standard)?

Well, there are two main reasons:
1) BPMN is too bloated and complex in many cases. And in our domain, we mostly concenrate on processes that do not make use of most BPMN features. Plus, we have our own dedicated notation used in this domain and users are used to it.

2) We have the feeling that our processes are more dynamic than usual workflows. For example, we may need to define (and not only select) conditional transitions and order of activities execution at run-time. As far as I understand from reading some forum posts on jBPM and Activiti forums, PVM allows for such a dynamic extensibility (e.g. adding transitions or even acitvities at run-time). But BPMN2.0 does not allow for it out of the box, or?  Of course, Activiti's BPMN2.0 implementation could be probably extended to support it (is it possible, actually? What do you say?), but would it be an industrty-adopted, portable BPMN representation then?

You could always do stuff like we do with KickStart, where we offer a very simple UI that allows to place some tasks sequentially or parallel, which is then translated to BPMN using JAXB.

This is of course something worth consideration. But we are not sure at the moment that we can actually map our process descriptions to the vanilla BPMN2.0 (see the bullet 2 above)