Throttling solution discussion

[mandas]

Hi,

We are using Activiti-5.10 for a project and according to our client needs, all service tasks are exclusive and asynchronous. There was a need to implement a throttling mechanism for those service tasks in a way that execution metrics are enforced for a specific task id. For example a rule which says that the services tasks of id 'someTask' are allowed to have 5 'simultaneous' executions at the same time, which on async terms means that at most 5 'someTask' can be scheduled for acquiring by the job executor. The remaining tasks have to stay in a queue and wait for the running tasks to fall below 5 instances.

As we couldn't implement such thing through the API and we have made several ad-hoc customizations by overriding classes and db mappings, I 'm posting our approach here if anyone is able to provide any suggestions of the proper way to do through the API, if any, because such an approach is tightly binded to internal activiti code.

We have overriden ProcessEngineConfiguration.getMyBatisXmlConfigurationSteam() to read a different mappings.xml which included one additional table of the job ids which were acquired by the job executor but our mechanism decided to postpone their execution. Furthermore, we provided a different Job.xml which modifies selectNextJobsToExecute and selectExclusiveJobsToExecute queries in order to avoid fetching jobs which are on the new table. Afterwards, we have implemented our custom AcquiredJobsCmd which handles the conversation between the executor and the throttling mechanism. We 've also overriden DefaultJobExecutor.ensureInitialization() to provide our custom AcquiredJobsCmd while at the same time we 've also overriden ExecuteJobsRunnable class, to add a single statement which informs the throttling mechanism that a job execution has been finished. Finally, another thread which is running at a specified interval, it deletes the postponed job ids from the new table, thus making them available for the job executor to acquire.

Thanks

[jbarrez]

That does sound like a valid approach (albeit a bit much internal stuff which is overridden … but this is a specific use case of course).

Having done that, what kind of pluggability would you like to see in Activiti, which wasn't there now? Command swapping? Custom command injection? JobExecutor interfaces? etc….

Thanks for posting, this is definitely interesting!

[mandas]

I think that the most difficult part was to understand at which point I had to plug the adhoc implementation (maybe more documentation for the internal stuff is required). I am not able to see the full picture yet, and probably if I start suggesting additions they would be having only our implementation in mind. At first glance, the ability to provide alternative managers (jobmanager etc) from the config file it would be great. When I come more deeply into this I think I 'd be able to provide you with more generic and elegant suggestions, or maybe I could even help.

Regards,
Dimitris

[mandas]

I think that in JobExecutor the ensureInitialization() method could be like this :

protected void ensureInitialization() {
    if (acquireJobsCmd == null)
        acquireJobsCmd = new AcquireJobsCmd(this);
    if (acquireJobsRunnable == null)
        acquireJobsRunnable = new AcquireJobsRunnable(this); 
}

Therefore it would be able for developers to inject custom cmds through activiti.cfg.xml

[jbarrez]

That would indeed be a solution. I've been thinking about custom command for a while now. Time to put some of these things in code … 😉

Thanks for the suggestion, by the way!