topic Re: Contributing functionality in Alfresco Archive

Contributing functionality

ronald_van_kuij — Fri, 07 Jan 2011 15:43:11 GMT

Hola,With years of detailed experience in jBPM, including the BPMN parts of jBPM4, I made the switch to using Activiti. In the company I currently work for, I convinced people to do a proof-of-concept with Activiti. The most important initial tests will be 'crash restistency' regarding transactions

Re: Contributing functionality

tombaeyens — Sat, 08 Jan 2011 20:09:09 GMT

What a start, Ronald!

Foreach: Joram plans to do that one this release.

EJB and JMS activities: would be great if you could contribute those.

Async continuations: I think I need to do that myself as there is already a lot of undocumented design ideas half in the code for that.

What effectively happens is that jobs are put in the db, directly pinned to the server they are created on and put in the threadpool (if there is room).

GREAT IDEAS! Now why didn't I think of that?! 😉 Kudos! You're very welcome to implement this, but do consider what kind of testing you would write for that. ACT-34 would be great if you could take that in one go. Currently we manage the threads ourselves. This should be done with a pluggable thread pool. There some JSR that specifies for appserver environments how you can obtain a threadpool that is managed by the server. (as you're not allowed to manage threads on your own in j2ee environments). Most j2ee servers don't check that by default, but I heard that websphere does.
Pluggability threadpool pluggability should be created in such a way that we don't introduce a dependency on any library when you use our own thread pool.
Another thing related to the JobExecutor is that some of the tests are failing once and a while on windows. We didn't have time yet to figure out if it is the test or the implementation that is the problem.
Also expanding test coverage on the job executor would be good. Especially high throughput or high load simulations would be great to have.
In our opinion, refactoring of the JobExecutor would be the most important and urgent. Would it be ok for you to start with that one?

Expressions in timer duration ACT-429: yep, assign to you if you want. good addition.
Same for dynamic subprocesses issue. Would be a great issue for you to do.

Re: Contributing functionality

ronald_van_kuij — Sun, 09 Jan 2011 16:47:31 GMT

Foreach: Joram plans to do that one this release.

ok, since one part that I already did was not my satisfaction, I'll wait what Joram comes up with. Keep in mind though that loop and serial 'foreach' are easy but the parallel one might require a kind of async functionality if it needs to be limited in one way or another or it might even be more difficukt if all run in one transaction. Not sure if it is technically related to async but it mit be

EJB and JMS activities: would be great if you could contribute those.

I will. I remember the discussion from a while ago (pre Activiti) on how these should be configured. Are those still valid? I'll ask my collegue to implement the shorthand version and do the long bpmn one myself.

Async continuations: I think I need to do that myself as there is already a lot of undocumented design ideas half in the code for that.

The jobhandler functionality was clear, so I made an AsyncJobHandler. Qurious what the rest is, since my implementation is not that complicated at all. Since it scheduled for 5.5 I'd be happy to still help out with you giving some pointers and do the validating. I am already fairly familiar with the internal workings of Activiti 😉

What effectively happens is that jobs are put in the db, directly pinned to the server they are created on and put in the threadpool (if there is room).

GREAT IDEAS! Now why didn't I think of that?! 😉 Kudos! You're very welcome to implement this, but do consider what kind of testing you would write for that.

Well, I just implemented it in Activiti. The ideas are from a collegue of mine, Frederik Wiers

ACT-34 would be great if you could take that in one go.

will give tat a try

Another thing related to the JobExecutor is that some of the tests are failing once and a while on windows. We didn't have time yet to figure out if it is the test or the implementation that is the problem.

I run everything on Linux, but Frederik is still on windows. So hopefully it does not show up anymore with the changes, but if it does we'll check it out

Also expanding test coverage on the job executor would be good. Especially high throughput or high load simulations would be great to have.
In our opinion, refactoring of the JobExecutor would be the most important and urgent. Would it be ok for you to start with that one?

sure, since we want/need to use a JBoss threadpool for other reasons as well.

Re: Contributing functionality

ronald_van_kuij — Sun, 09 Jan 2011 21:15:42 GMT

What a start, Ronald!
Also expanding test coverage on the job executor would be good. Especially high throughput or high load simulations would be great to have.

I used the tweet test to produce messages with unique content. This way it was easy to test if duplicates would be created or messages missed. What went wrong is that hitting the connection limit of the database, the (multi-threaded) creation of messages would fail and the test would sometimes stop. So some changes need to be made there but when I configured things in a way the connection limit was not reached, I could get a fairly constant rate of 420-460 jobs per second produced AND consumed on a 2GB Intel Centrino2 laptop where MySQL was running on that same laptop. (h2 in mem mode was 6000)

Coming week, I'll try to run a test with Activiti instances with an adapted JobExecutor (still with it's own threads) on Linux and Windows against the same remote db and see what happens. We'll also be crashing the DB at that time to see how the jobexecutor (and later on Activiti as a whole) keeps behaving. This area is where there has been bad experiences with JBoss JMS and which lead to a custom queueing solution… Activiti has to be om par here what was custom developed. 🙂 But comparing the custom queue and the jobexecutor gave me the feeling that we could merge some functionality and 'switch'

We'll keep you guys posted.

Re: Contributing functionality

bernd_ruecker — Mon, 10 Jan 2011 08:24:45 GMT

Wow, cool. Especially the JobExecutor stuff seems reasonable and makes a lot of sense. It is very good that you start to focus on these enterprise kind of things. Cool!

Re: Contributing functionality

jbarrez — Mon, 10 Jan 2011 10:24:17 GMT

Awesome! Welcome back, Ronald!

Offloading work from me is always something which I like 😉

Re: Contributing functionality

ronald_van_kuij — Mon, 10 Jan 2011 10:53:47 GMT

Awesome! Welcome back, Ronald!

Offloading work from me is always something which I like 😉

Offloading? No way… you'll get other thing to do, no worries mate…

Re: Contributing functionality

sangv — Wed, 02 Nov 2011 22:11:55 GMT

Ronald -

I know I am reading this a little late, but in the first post here, you had mentioned about dynamically running subprocesses based on a process value and that there was no JIRA issues for that. We needed the same functionality and this issue was created by one of my teammates some time ago

http://jira.codehaus.org/browse/ACT-831

Can you take a look at this and see of this is what you meant. Also, can we have this feature so that we don't have to hard code the sub process key…

Thanks,
Sang

Re: Contributing functionality

ronald_van_kuij — Wed, 02 Nov 2011 23:29:18 GMT

I'll have a look, but important for me to check the correctness of te patch is to have testcases. So if you can provide some, that would make the testing easier. And the next step ten is to have some documentation as well.

Re: Contributing functionality

sangv — Thu, 03 Nov 2011 14:11:25 GMT

Sounds good. I will move my teammates patch to 5.9-SNAPSHOT, add test cases and create another patch.

Thanks,
Sang

Re: Contributing functionality

sangv — Mon, 07 Nov 2011 21:29:39 GMT

Hi Ronald -

Provided an updated patch, test cases and documentation for http://jira.codehaus.org/browse/ACT-831. Please take a look and let me know if you need any other information.

Thanks,
Sang

Re: Contributing functionality

ronald_van_kuij — Tue, 08 Nov 2011 16:23:49 GMT

Thanks, will look into it tomorrow or thursday

Re: Contributing functionality

sangv — Mon, 14 Nov 2011 14:29:53 GMT

Hi - Did you get a chance to look at the updated patch for this issue. Please let me know if you have any questions.

Thanks,
Sang