topic parallel instantiation of workflow depending on the DB count in Alfresco Archive

parallel instantiation of workflow depending on the DB count

nirvana — Wed, 15 Jan 2014 11:09:29 GMT

Hi All,Attached the workflow.I'm able to instantiate the workflows parallelly depending in regards with some X count from DB.If the count is more than 22 then am getting stack overflow error.In my case, I have a requirement of handling more than 50000 in one go….

Re: parallel instantiation of workflow depending on the DB count

nirvana — Wed, 15 Jan 2014 11:30:59 GMT

stack trace for the above issue


16:58:00,119 INFO  [stdout] (http–127.0.0.1-8080-1) in PlanningPhase phase Count ==79
16:58:00,134 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningPhase execute() : END
16:58:00,134 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningCycleProcess execute() : START
16:58:00,150 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleProcess updatePlanningCycleWFDetails() : START
16:58:00,150 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningCycle() START
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningCycle() END
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleProcess updatePlanningCycleWFDetails() : END
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() START
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() END
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) PLAANNING PHASE COUNT====79
16:58:00,165 INFO  [stdout] (http–127.0.0.1-8080-1) %%%%%%%%%%%% planningPhaseID is : 22
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) %%%%%%%%%%%% planningPhaseID is : 22
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) execution==ConcurrentExecution[785689]
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningCycleProcess execute() : END
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningPhase execute() : START
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) <<< ======= Phase ID in PlanningPhase is =======>>> 22
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningPhaseProcess updatePlanningPhaseWFDetails() : START
16:58:00,181 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningPhase() START
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningPhase() END
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningPhaseProcess updatePlanningPhaseWFDetails() : END
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() START
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() END
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) in PlanningPhase phase Count ==78
16:58:00,212 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningPhase execute() : END
16:58:00,228 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningCycleProcess execute() : START
16:58:00,228 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleProcess updatePlanningCycleWFDetails() : START
16:58:00,228 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningCycle() START
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : updatePlanningCycle() END
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleProcess updatePlanningCycleWFDetails() : END
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() START
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) In PlanningCycleDAO : getPlanningCycleVO() END
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) PLAANNING PHASE COUNT====78
16:58:00,244 INFO  [stdout] (http–127.0.0.1-8080-1) %%%%%%%%%%%% planningPhaseID is : 23
16:58:00,259 INFO  [stdout] (http–127.0.0.1-8080-1) %%%%%%%%%%%% planningPhaseID is : 23
16:58:00,259 INFO  [stdout] (http–127.0.0.1-8080-1) execution==ConcurrentExecution[785703]
16:58:00,259 INFO  [stdout] (http–127.0.0.1-8080-1) ++++++++++++++++++++++++++++++++++++++++In PlanningCycleProcess execute() : END
16:58:02,665 SEVERE [com.vaadin.Application] (http–127.0.0.1-8080-1) Terminal error:: com.vaadin.event.ListenerMethod$MethodException: Invocation of method buttonClick in org.activiti.e
xplorer.ui.task.TaskDetailPanel$5 failed.
        at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:530) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:164) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.ui.AbstractComponent.fireEvent(AbstractComponent.java:1219) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.ui.Button.fireClick(Button.java:567) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.ui.Button.changeVariables(Button.java:223) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.changeVariables(AbstractCommunicationManager.java:1460) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.handleVariableBurst(AbstractCommunicationManager.java:1404) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.handleVariables(AbstractCommunicationManager.java:1329) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.doHandleUidlRequest(AbstractCommunicationManager.java:761) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.CommunicationManager.handleUidlRequest(CommunicationManager.java:325) [vaadin-6.8.8.jar:6.8.8]
        at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:501) [vaadin-6.8.8.jar:6.8.8]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
        at org.activiti.explorer.filter.ExplorerFilter.doFilter(ExplorerFilter.java:44) [activiti-explorer-5.13.jar:]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
Caused by: java.lang.StackOverflowError
        at java.util.HashMap.put(HashMap.java:372) [rt.jar:1.6.0_30]
        at org.activiti.engine.impl.db.DbSqlSession.cachePut(DbSqlSession.java:340) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.db.DbSqlSession.insert(DbSqlSession.java:116) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.history.HistoryManager.recordActivityStart(HistoryManager.java:178) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.history.handler.ActivityInstanceStartHandler.notify(ActivityInstanceStartHandler.java:30) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:42) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:87) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:87) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:66) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:87) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionDestroyScope.execute(AtomicOperationTransitionDestroyScope.java:116) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:87) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527) [activiti-engine-5.13.jar:5.13]
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerEnd.java:36) [activiti-engin

PLease suggest for getting this out.

thanks in advance.

regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

frederikherema1 — Wed, 15 Jan 2014 12:39:12 GMT

This is a known issue in the command-processing inside the engine. We have an issue open for this for a while now, but fail to make the tail-recursion prevention work in all cases.

A solution whould be to make use of asynchronous activities. I see you're always looping back to the same call-activity to start a process. Perhaps, also take a look at the Multi-instance capabilities. This prevent having a deep sequence-flow stack, since the loop is handled by the multi-instance behaviour rather than depending on process-flow.

Re: parallel instantiation of workflow depending on the DB count

nirvana — Thu, 30 Jan 2014 05:54:05 GMT

Thanks Frederik Heremans.

As suggested I'm using Multi-Instance modelling, I'm able to see 10K instances in 5 mins with 8GB memory and 16M stacksize . But when I configure the count to 100K literally I see 250 instances are created and after that I see OutOfMemory….

Any suggestions plz…

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

frederikherema1 — Thu, 30 Jan 2014 10:14:17 GMT

Are they executed parallel, or sequential (the multi-instance activity, I'm referring to)? Any reason why you would try to automate 100K loops? Why not split it up in multiple processes?

Re: parallel instantiation of workflow depending on the DB count

nirvana — Thu, 30 Jan 2014 12:39:43 GMT

My requirement - as per the diagram above, when I start and complete the first step(planning cycle task) then all the related planning phases should be started and initialized.

Here can you please clarify me what exactly do you mean 'Why not split it up in multiple processes' ?

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

nirvana — Mon, 03 Feb 2014 07:02:29 GMT

Any help or suggestions on the above ?

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

nirvana — Mon, 03 Feb 2014 10:42:24 GMT

Hi,

Also here I have question like in the above multi process instance model when I use Sub process I'm getting same ID for all the 'n' tasks.

But when I use call activiti instead of sub-process I'm getting all the different id's correctly.

Can you please tell me what is the difference between these ? everything else is same ? just changed the sub-process to call activiti everything is working properly.

Watinig for the response.
Thanks in advance.

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

jbarrez — Thu, 06 Feb 2014 11:04:18 GMT

Another solution is to split up processes, and use async on certain steps.

What 'id' do you mean? Database id or activity id?

Re: parallel instantiation of workflow depending on the DB count

nirvana — Fri, 07 Feb 2014 10:06:04 GMT

HI All,

Thanks for the response.
Its a database ID.
tried splitting up the process, but still there no luck ?

Is this the only way ?

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

nirvana — Fri, 07 Feb 2014 10:09:22 GMT

what is difference between using sub-process and call-activiti ?
when using the sub-process I just see the same instance created 'n' number of times.
when using call-activiti i see the 'n' different instances.

Can't we achieve the same with sub-process ?

Thanks in advance.

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

frederikherema1 — Mon, 10 Feb 2014 08:31:17 GMT

Call-activiti wil start a separate process-instance, based on the processDefinition referenced in the calledElement. A sub-process will run inside the process-instance and will execute the activities inside the sub-process. So a call-activity can execute external processDefinition (versioned differently, will always start the latest version of the key in "calledElement"), while a subprocess' activities are always the same version as the main process-definition.

Also, a subprocess has access to the process-instance scope (variables), while a call-activity only "sees" the variables declared in activiti:in, and can only write variables delacred in "activitiut".

Re: parallel instantiation of workflow depending on the DB count

nirvana — Mon, 10 Feb 2014 09:30:22 GMT

Thanks Frederik Heremans.

I tried multi instance, asyn, etc. but I can't instantiate more than 30K.
Even tried splitting up the process with call-activiti calling another call-activiti being both multi instances… for trying X * Y combinations for the call-activities…….
Any more suggestions ?

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

jbarrez — Tue, 11 Feb 2014 15:44:39 GMT

Did you already try to make some steps async as mentioned above

Re: parallel instantiation of workflow depending on the DB count

nirvana — Wed, 12 Feb 2014 09:37:54 GMT

Yes joram.

Tried following
1. Async
2. splitting the process.
3. dividing the Multi instance process for 5000 instances.
4. Running the process through standalone mode, and activiti-explorer just for displaying the tasks.

no luck what so ever…………

Regards,
Nirvana

Re: parallel instantiation of workflow depending on the DB count

jbarrez — Mon, 17 Feb 2014 14:18:55 GMT

Okay, can you put the process xml somewhere so we can verify?
The xml please as simple as possible, while still demonstrating the problem.

Re: parallel instantiation of workflow depending on the DB count

nirvana — Wed, 19 Feb 2014 05:45:08 GMT

Hi Jorram,

Thanks for the reply.

attached 3 process xml's in txt format.

Regards,
Nirvana