topic Re: Slow multi-threaded execution due to oversynchronized blocks in Alfresco Archive

Slow multi-threaded execution due to oversynchronized blocks

romanoff — Wed, 13 Apr 2011 13:20:47 GMT

Hi,I have written a simple standalone test application (a JUnit test) that starts 5000 Runnable tasks on a j.u.c pool of 50 threads. Each task starts and executes the same simple BPMN process. H2 backend is used. There are no jobs or human tasks created during the execution of this process. Since pr

Re: Slow multi-threaded execution due to oversynchronized blocks

frederikherema1 — Wed, 13 Apr 2011 13:33:04 GMT

Hi,

By default (not using spring datasource and not having altered configuration), the ibatis connection-pool size is fixed to 10. Did you alter the pool size? If not, locked threads makes perfect since the thead puttrough are trottled by this limit…

Limit can be altered using property 'jdbcMaxActiveConnections' on activiti cfg. Relevant code responsible for create pooled dataSource (ProcessEngineCOnfigurationImpl).


 PooledDataSource pooledDataSource = 
          new PooledDataSource(ReflectUtil.getClassLoader(), jdbcDriver, jdbcUrl, jdbcUsername, jdbcPassword );
        
        if (jdbcMaxActiveConnections > 0) {
          pooledDataSource.setPoolMaximumActiveConnections(jdbcMaxActiveConnections);
        }
        if (jdbcMaxIdleConnections > 0) {
          pooledDataSource.setPoolMaximumIdleConnections(jdbcMaxIdleConnections);
        }
        if (jdbcMaxCheckoutTime > 0) {
          pooledDataSource.setPoolMaximumCheckoutTime(jdbcMaxCheckoutTime);
        }
        if (jdbcMaxWaitTime > 0) {
          pooledDataSource.setPoolTimeToWait(jdbcMaxWaitTime);
        }

If this isn't the issue, consider using DBCP-datasource pool, which is a proven pooling lib, instead of the built-in (myBatis).

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Wed, 13 Apr 2011 13:41:54 GMT

Hi,

By default (not using spring datasource and not having altered configuration), the ibatis connection-pool size is fixed to 10. Did you alter the pool size? If not, locked threads makes perfect since the thead puttrough are trottled by this limit…

Limit can be altered using property 'jdbcMaxActiveConnections' on activiti cfg.

Relevant code responsible for create pooled dataSource (ProcessEngineCOnfigurationImpl).
PooledDataSource pooledDataSource = new PooledDataSource(ReflectUtil.getClassLoader(), jdbcDriver, jdbcUrl, jdbcUsername, jdbcPassword ); if (jdbcMaxActiveConnections > 0) { pooledDataSource.setPoolMaximumActiveConnections(jdbcMaxActiveConnections); } if (jdbcMaxIdleConnections > 0) { pooledDataSource.setPoolMaximumIdleConnections(jdbcMaxIdleConnections); } if (jdbcMaxCheckoutTime > 0) { pooledDataSource.setPoolMaximumCheckoutTime(jdbcMaxCheckoutTime); } if (jdbcMaxWaitTime > 0) { pooledDataSource.setPoolTimeToWait(jdbcMaxWaitTime); }

Yes, I already tried to increase jdbcMaxActiveConnections to 100, which should be more than enough with only 50 worker threads in the thread pool. It helped a bit, but not much.

If this isn't the issue, consider using DBCP-datasource pool, which is a proven pooling lib, instead of the built-in (myBatis).

Thanks! This is an interesting hint. How can I replace the myBatis datasource pool via the DBCP pool? Can it be done purely via a configuration or in the client code? Or do I need to alter the Activiti code in the ProcessEngineCOnfigurationImpl?

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Wed, 13 Apr 2011 15:05:26 GMT

Thanks! This is an interesting hint. How can I replace the myBatis datasource pool via the DBCP pool? Can it be done purely via a configuration or in the client code? Or do I need to alter the Activiti code in the ProcessEngineCOnfigurationImpl?

OK. I figured out myself how to do it by reading the user's guide 😉

Re: Slow multi-threaded execution due to oversynchronized blocks

jbarrez — Thu, 14 Apr 2011 08:00:32 GMT

Interesting findings. The synchronized block you see if to allocate a block of ids for the current Activiti engine. In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Keep us tuned about your findings with the DBCP datasource!

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Thu, 14 Apr 2011 08:42:55 GMT

Interesting findings. The synchronized block you see if to allocate a block of ids for the current Activiti engine. In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Keep us tuned about your findings with the DBCP datasource!

If you ask for it, I continue with my reporting 😉

After I switched to the DBCP, I've seen that thread contention due to the thread-pooling has disappeared. Which means that myBatis is really bad for pooling DB connections.

After that problem was solved, org.activiti.engine.impl.db.DbIdGenerator.getNextId() became the main reason of thread contention.

In normal use cases, the fetching of this id-block is infrequent since the block is big enought.
However in your use case of course, the ids are quickly gone. Maybe we would need to make the id-block size configurable.

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Once I did it, org.activiti.engine.impl.db.DbIdGenerator.getNextId() did not appear on the radar any more.

What I observe now is the following: there is no thread contention as such, but there are too many database queries being issued by Activiti. Profiler shows that they consume at least 50-70% of overall time. And almost all of these DB queries are from the ExecutionEntity.remove(), where we have the following snippet:


    // delete all the tasks
    CommandContext commandContext = Context.getCommandContext();
    List<TaskEntity> tasks = (List) new TaskQueryImpl(commandContext)
      .executionId(id)
      .list();
    for (TaskEntity task : tasks) {
      if (replacedBy!=null) {
        task.setExecution(replacedBy);
      } else {
        task.delete(TaskEntity.DELETE_REASON_DELETED);
      }
    }

    List<Job> jobs = new JobQueryImpl(commandContext)
      .executionId(id)
      .list();
    for (Job job: jobs) {
      if (replacedBy!=null) {
        ((JobEntity)job).setExecution((ExecutionEntity) replacedBy);
      } else {
        ((JobEntity)job).delete();
      }
    }

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Please let me know if any improvements are possible. Or if you want me to check something related to the mentioned issues.

Re: Slow multi-threaded execution due to oversynchronized blocks

jbarrez — Fri, 15 Apr 2011 08:04:06 GMT

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Wow great findings! I had already forgotten about that property!

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Yes- I came to the same conclusions a few months ago when I did some profiling.
I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

The stuff you've found is really good stuff to put into a blog (with cool charts ;-), do you have any plans to do such a thing?

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Fri, 15 Apr 2011 09:12:31 GMT

Indeed. And it is already configurable, though not described in the docs yet.
It can be configured like this:
<property name="idBlockSize" value="10000" />

Wow great findings! I had already forgotten about that property!

It would be nice if you'd update the documentaton and mention it somewhere as well as the fact that the default myBatis config does not scale when you use many concurrent threads.

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Yes- I came to the same conclusions a few months ago when I did some profiling.
I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

Do you have any concrete ideas about when it could be done? It would be nice if it would be fixed rather soon and I guess the fixes are rather simple. The raw performance of the engine for the scenarios like I described in the initial message of this thread is a very important criteria for selecting a concrete engine to be used in a bigger project on our side. Would be nice if I could report something positive and backed by the figures to my management 😉

The stuff you've found is really good stuff to put into a blog (with cool charts ;-), do you have any plans to do such a thing?

I'd like to. But due to the different internal policies of my employee I may not be allowed to do it ;-( In any case, I could provide a source code of the test, so that it can be re-used by users of Activiti or even included into the Activiti's unit/performance tests.

I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

I'm just wondering if this approach would lead to any problems in the cluster setup. I don't know if it is possible at all, but what would happen if other nodes may potentially add tasks/jobs to the process instance running on the current node. If this is possible, then it is difficult to avoid quering the DB. But even in this case, may be some sort of indications (in the process description or in startup configuraton parameters) can be introduced that this process instance will not get new tasks or jobs from other nodes. More over, there is already a flag for disabling job scheduler, IIRC. If this one is disabled (which is a rather radical step ;-), then there can be no jobs or tasks in the DB anyway, or? So, no need to check the DB for them anyway.

Question:
Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

Re: Slow multi-threaded execution due to oversynchronized blocks

jbarrez — Mon, 18 Apr 2011 07:12:37 GMT

Tom is currently working on refactoring the persistence (because it's a mess at some places).
But I doubt that the issue you describe will be fixed soon - as our focus is currently not around persistence. However, if a big customers pops up with this problem, then it will be fixed soon 😉

Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

The services are tightly coupled to BPMN. It is better to use the PVM API as-is. Do note that the PVM API is quite low-level compared to the services.

What do you mean with domain-specific stuff? Or better: why would you want it (now that BPMN is a industrty-adopted standard)?
You could always do stuff like we do with KickStart, where we offer a very simple UI that allows to place some tasks sequentially or parallel, which is then translated to BPMN using JAXB.

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Mon, 18 Apr 2011 11:01:31 GMT

Tom is currently working on refactoring the persistence (because it's a mess at some places). But I doubt that the issue you describe will be fixed soon - as our focus is currently not around persistence. However, if a big customers pops up with this problem, then it will be fixed soon 😉

I see.

Is it possible to re-use the services (runtimeService, historyService, etc) with the pure PVM-API based processes? Or is it tightly coupled to the BPMN implementation on top of the PVM? How hard would it be to extend them to support domain-specific custom workflow languages other than BPMN? Would it be a minor or a very big change?

The services are tightly coupled to BPMN. It is better to use the PVM API as-is. Do note that the PVM API is quite low-level compared to the services.

Interesting. I hoped that PVM can handle such services like persistence of definitions, jobs (for timers and async continuations) and keeping a history almost independent of the language layer built on top of it. This would greatly reduce the amount of efforts to develop any new languages on top of the PVM. Essentially, you'd just need to define the syntax, a mapping from this syntax to the PVM model and its execution semantics.
But if I understand you correctly, this is not the case. Every language built on top of the PVM needs to implement e.g. its own deployment, history and job/scheduler services. Thus, PVM is mostly used to represent the run-time in-memory model of the process to be executed. Is it correct a correct understanding?

What do you mean with domain-specific stuff? Or better: why would you want it (now that BPMN is a industrty-adopted standard)?

Well, there are two main reasons:
1) BPMN is too bloated and complex in many cases. And in our domain, we mostly concenrate on processes that do not make use of most BPMN features. Plus, we have our own dedicated notation used in this domain and users are used to it.

2) We have the feeling that our processes are more dynamic than usual workflows. For example, we may need to define (and not only select) conditional transitions and order of activities execution at run-time. As far as I understand from reading some forum posts on jBPM and Activiti forums, PVM allows for such a dynamic extensibility (e.g. adding transitions or even acitvities at run-time). But BPMN2.0 does not allow for it out of the box, or? Of course, Activiti's BPMN2.0 implementation could be probably extended to support it (is it possible, actually? What do you say?), but would it be an industrty-adopted, portable BPMN representation then?

You could always do stuff like we do with KickStart, where we offer a very simple UI that allows to place some tasks sequentially or parallel, which is then translated to BPMN using JAXB.

This is of course something worth consideration. But we are not sure at the moment that we can actually map our process descriptions to the vanilla BPMN2.0 (see the bullet 2 above)

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Mon, 18 Apr 2011 15:08:43 GMT

As you can see, it always tries to get the lists of tasks and jobs and then remove them if required. The interesting part of this observation is that I do not use any tasks and jobs at all, but still pay the price for them! I think it should be optimized somehow…
May be it is possible for the Activiti engine to remember that a certain process instance never created any tasks or jobs? If such a boolen flag would exist, one could avoid issuing such useless DB queries. What do you think?

Another place, where useless DB queries are produced is in VariableScopeImpl, which uses DB always to initialize vars from a DB. It creates a lot of DB traffic. And it does it even on process instance creation, where it is known that no variables could be stored in the DB for this instance already. Again, here it could be optimized in a similar way, i.e. if it is a creation of a process instance and no varaibles were defined yet, there is no need to ask the DB for their values.

Yes- I came to the same conclusions a few months ago when I did some profiling.
I believe they can be optimized somehow (eg by keeping a boolean if jobs are created for example).

I have a few related questions and performance improvement suggestions:

1) How does the historyService work? Is it performing asyncrhonous writes to the DB or does it block the "main" thread of the process execution? If it does not do any async writes yet, may be it should? BTW, some log4j appenders actually support this mode of operation to offload the busy worker threads in Java(EE) apps.

2) Imagine the situation, where user tasks (or more generically wait states e.g. waiting for async responses from certain external services) do not last very long and are completed in just a few milliseconds. This can be the case, where users are not humans, but other processes and can react very quickly. Right now, AFAIK, Activiti stores the state of the current execution into the DB and then waits. Once user task is completed (or waiting in a waiting state is over), Acitiviti would restore the process state from a DB and continue the execution. My profiler shows that these save/restore actions introduce quite some overhead and DB access contention if uses under heavy load and with many concurrent process instances being executed. Therefore, I was thinking about the following optimization:
Defer the DB writes for saving a state of process instances for a certain configurable amount of time (e.g. 1 second). If after this timeout, the user task (or wait state) are still waiting and not finished yet, flush those writes to the DB and proceed as usual. But if it finished in the meantime, then there is no need to store this state anymore and all related pending writes can be just ignored. And a restore operation just reuses the RAM versions of the process instance state. As a result, it makes Activiti usable under high load (and even for certain kinds of soft-real time apps).

What do you think about this idea?

P.S. I'm not sure if it introduces any problems in clustered setups due to this deferred writes.

Re: Slow multi-threaded execution due to oversynchronized blocks

ronald_van_kuij — Mon, 18 Apr 2011 17:48:32 GMT

I have a few related questions and performance improvement suggestions:

1) How does the historyService work? Is it performing asyncrhonous writes to the DB or does it block the "main" thread of the process execution? If it does not do any async writes yet, may be it should? BTW, some log4j appenders actually support this mode of operation to offload the busy worker threads in Java(EE) apps.

It is not doing async writes. I think one of the reasons is that (at least in our situation) we need them… always… They cannot be lost. So they need to be written to some persistent store in a transaction. Sure something could be implemented that writes them to a local files store (quick) and some other process puts them async in the db. But then you also need some option to still retrieve them if the local storage crashes… etc… much more complicated then it looks at first sight. But if you have other cleaver ideas, let me (us know)

2) Imagine the situation, where user tasks (or more generically wait states e.g. waiting for async responses from certain external services) do not last very long and are completed in just a few milliseconds. This can be the case, where users are not humans, but other processes and can react very quickly. Right now, AFAIK, Activiti stores the state of the current execution into the DB and then waits. Once user task is completed (or waiting in a waiting state is over), Acitiviti would restore the process state from a DB and continue the execution. My profiler shows that these save/restore actions introduce quite some overhead and DB access contention if uses under heavy load and with many concurrent process instances being executed. Therefore, I was thinking about the following optimization:
Defer the DB writes for saving a state of process instances for a certain configurable amount of time (e.g. 1 second). If after this timeout, the user task (or wait state) are still waiting and not finished yet, flush those writes to the DB and proceed as usual. But if it finished in the meantime, then there is no need to store this state anymore and all related pending writes can be just ignored. And a restore operation just reuses the RAM versions of the process instance state. As a result, it makes Activiti usable under high load (and even for certain kinds of soft-real time apps).

Isn't this a kind of second level cache? Would (kind of) work for single node systems, but what if there are multiple nodes and other nodes could execute next steps in the process, so yes it might introduce issues unless you use something like teracotta. And wait states including user tasks (at least in our case) last way, way more than one second, so this would not yield much.

If they are short (as in your case, you might choose not to use waitstates and just use service taks so everything is done in one transaction (at least those are the things we take into account when developing a process) . You might need to increase the number of threads/connections though

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Tue, 19 Apr 2011 07:05:15 GMT

I have a few related questions and performance improvement suggestions:

1) How does the historyService work? Is it performing asyncrhonous writes to the DB or does it block the "main" thread of the process execution? If it does not do any async writes yet, may be it should? BTW, some log4j appenders actually support this mode of operation to offload the busy worker threads in Java(EE) apps.

It is not doing async writes. I think one of the reasons is that (at least in our situation) we need them… always… They cannot be lost. So they need to be written to some persistent store in a transaction. Sure something could be implemented that writes them to a local files store (quick) and some other process puts them async in the db. But then you also need some option to still retrieve them if the local storage crashes… etc… much more complicated then it looks at first sight. But if you have other cleaver ideas, let me (us know)

2) Imagine the situation, where user tasks (or more generically wait states e.g. waiting for async responses from certain external services) do not last very long and are completed in just a few milliseconds. This can be the case, where users are not humans, but other processes and can react very quickly. Right now, AFAIK, Activiti stores the state of the current execution into the DB and then waits. Once user task is completed (or waiting in a waiting state is over), Acitiviti would restore the process state from a DB and continue the execution. My profiler shows that these save/restore actions introduce quite some overhead and DB access contention if uses under heavy load and with many concurrent process instances being executed. Therefore, I was thinking about the following optimization:
Defer the DB writes for saving a state of process instances for a certain configurable amount of time (e.g. 1 second). If after this timeout, the user task (or wait state) are still waiting and not finished yet, flush those writes to the DB and proceed as usual. But if it finished in the meantime, then there is no need to store this state anymore and all related pending writes can be just ignored. And a restore operation just reuses the RAM versions of the process instance state. As a result, it makes Activiti usable under high load (and even for certain kinds of soft-real time apps).

Isn't this a kind of second level cache? Would (kind of) work for single node systems, but what if there are multiple nodes and other nodes could execute next steps in the process, so yes it might introduce issues unless you use something like teracotta. And wait states including user tasks (at least in our case) last way, way more than one second, so this would not yield much.

If they are short (as in your case, you might choose not to use waitstates and just use service taks so everything is done in one transaction (at least those are the things we take into account when developing a process) . You might need to increase the number of threads/connections though

Ronald, thanks a lot for your answers!

Few comments:

1)

If they are short (as in your case, you might choose not to use waitstates and just use service taks so everything is done in one transaction (at least those are the things we take into account when developing a process) . You might need to increase the number of threads/connections though

The idea of using service tasks would not scale if we speak about 1000s or 10000s of simultaneous, concurrent process instances, as it implies thread-per-request approach and may also lead to increased threads starvation (all of them just wait for results from external services). The only well-proven way to scale here is to use async approach, i.e. wait states in combination with suspend/resume for process instances.

2) I perfectly understand why Activiti is implemented they way how it is functioning now. It is targeting human-oriented workflows and makes a number of corresponding assumptions about the runtime requirements and behavior of such workflows. The whole business model is currently built around this domain.

But I also want to stress that there is a much bigger (by a few orders of magnitude) world of non-human-oriented workflows (where I come from), e.g. machine-oriented workflows, machine-to-machine, sensors , telcos, trading, etc. This is a huge market with a lot of business opportunities. While BPMN (or PVM) itself is perfectly usable for those domains as well (though "B" becomes not so important eventually), the implementations of these ideas (e.g. Activiti) are not usable for those domains out of the box. The reason for that is that those domains have somewhat different requirements. For example, some of them do not need to maintain history, some others do not need transactions support, third kind does not need the ability to continue process instance executions on a different node in a cluster, because it uses the concept of sticky sessions (and therefore does not need to save the complete process instance state in the DB), etc. And almost all of those have higher, more real-time requirements on performance and latencies simply because machines are much faster than humans 😉

Therefore, I just would like to suggest that in your architecture or system designs you at least consider that some of the hard-coded or built-in decisions are actually based on only a specific use-case from a concrete domain (e.g. CMS or business processes) and are not applicable to the other ones. With that in mind, it would be nice to have such things like persistency, clustering support, asynchrony support, etc more configurable/controlled by policies/pluggable by architectural design, so that alternative implementations could be easily plugged in by Activiti core developers or by project contributors and users. Current implementations of all those features can still be the default implementations delivered out-of-the-box, but they would not prohibit usage of other alternative implementations.

If this becomes possible, then the _same_ engine (and product, and tooling, and solutions around it) would become more usable and efficient in a wide variety of different domains and their combinations, thus making Activiti an even more attractive choice for many new and existing customers and users.

I hope you do not get me wrong. I like Activiti a lot and think it has a great potential. I think overall it is also rather well prepared architecture-wise. I just wanted to highlight some potential opportunities in domains other than pure business processes and architectural implications resulting from the new requirements coming from those domains.

Re: Slow multi-threaded execution due to oversynchronized blocks

jbarrez — Wed, 20 Apr 2011 08:15:10 GMT

Not much tie now to go through all the posts - will do at a later point in time (man, you should write books ;-))

Just wanted to point you to this thread, where Tom added a patch for validation: http://forums.activiti.org/en/viewtopic.php?f=6&t=1521&start=10

Re: Slow multi-threaded execution due to oversynchronized blocks

romanoff — Wed, 20 Apr 2011 08:35:35 GMT

Not much tie now to go through all the posts - will do at a later point in time (man, you should write books ;-))

Who said I'm not doing it already?