RuntimeService.deleteProcessInstance cause StackOverflowErr

[razu]

I use Activity 5.7 and encounter a StackOverflowError after call RuntimeService.deleteProcessInstance on running process with about 4500 instances of UserTask.
I can observe about 10 minutes of querying database, and after that StackOverflowError is thrown.

Any hints how to manage this?

This is fragment of stacktrace:

java.lang.StackOverflowError: null   at java.io.StringReader.ensureOpen(StringReader.java:39) ~[na:1.5.0_22]   at java.io.StringReader.read(StringReader.java:73) ~[na:1.5.0_22]   at org.apache.ibatis.ognl.JavaCharStream.FillBuff(JavaCharStream.java:127) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.JavaCharStream.ReadByte(JavaCharStream.java:149) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.JavaCharStream.readChar(JavaCharStream.java:234) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.JavaCharStream.BeginToken(JavaCharStream.java:168) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParserTokenManager.getNextToken(OgnlParserTokenManager.java:1471) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.jj_scan_token(OgnlParser.java:3419) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.jj_3R_21(OgnlParser.java:3232) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.jj_3_3(OgnlParser.java:3226) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.jj_2_3(OgnlParser.java:2676) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.primaryExpression(OgnlParser.java:1711) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.navigationChain(OgnlParser.java:1409) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.unaryExpression(OgnlParser.java:1361) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.multiplicativeExpression(OgnlParser.java:1100) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.additiveExpression(OgnlParser.java:1011) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.shiftExpression(OgnlParser.java:849) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.relationalExpression(OgnlParser.java:571) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.equalityExpression(OgnlParser.java:456) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.andExpression(OgnlParser.java:397) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.exclusiveOrExpression(OgnlParser.java:338) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.inclusiveOrExpression(OgnlParser.java:279) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.logicalAndExpression(OgnlParser.java:220) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.logicalOrExpression(OgnlParser.java:161) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.conditionalTestExpression(OgnlParser.java:116) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.assignmentExpression(OgnlParser.java:73) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.expression(OgnlParser.java:26) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.OgnlParser.topLevelExpression(OgnlParser.java:16) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.Ognl.parseExpression(Ognl.java:111) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.Ognl.getValue(Ognl.java:435) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.ognl.Ognl.getValue(Ognl.java:414) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.builder.xml.dynamic.ExpressionEvaluator.evaluateIterable(ExpressionEvaluator.java:28) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.builder.xml.dynamic.ForEachSqlNode.apply(ForEachSqlNode.java:36) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.builder.xml.dynamic.MixedSqlNode.apply(MixedSqlNode.java:14) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.builder.xml.dynamic.MixedSqlNode.apply(MixedSqlNode.java:14) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.builder.xml.dynamic.DynamicSqlSource.getBoundSql(DynamicSqlSource.java:22) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.mapping.MappedStatement.getBoundSql(MappedStatement.java:198) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.executor.BaseExecutor.createCacheKey(BaseExecutor.java:115) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:90) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:72) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:75) ~[mybatis-3.0.4.jar:3.0.4]   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:69) ~[mybatis-3.0.4.jar:3.0.4]   at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:193) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.TaskManager.findTasksByQueryCriteria(TaskManager.java:89) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.TaskQueryImpl.executeList(TaskQueryImpl.java:334) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.AbstractQuery.list(AbstractQuery.java:114) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.remove(ExecutionEntity.java:764) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AtomicOperationDeleteCascadeFireActivityEnd.eventNotificationsCompleted(AtomicOperationDeleteCascadeFireActivityEnd.java:61) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:52) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:45) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AtomicOperationDeleteCascade.execute(AtomicOperationDeleteCascade.java:31) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AtomicOperationDeleteCascadeFireActivityEnd.eventNotificationsCompleted(AtomicOperationDeleteCascadeFireActivityEnd.java:66) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:52) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:45) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AtomicOperationDeleteCascade.execute(AtomicOperationDeleteCascade.java:31) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:76) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:481) ~[activiti-engine-5.7.jar:5.7]   at org.activiti.engine.impl.pvm.runtime.AtomicOperationDeleteCascadeFireActivityEnd.eventNotificationsCompleted…‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[ronald_van_kuij]

As Frederik mentioned, a lot of investigation already took place.

How is your history set? You know a lot of entities are created when having 4500 user-tasks, and we can't cascade delete them all. So there will be n times 4500 deletes going on, we can't do magic either you know… You can always look for suggestions on how we can optimize the deletes (however, a lot of thinking and work was perofrmed to get to the way it's going happening right now).

Short term solution: You can always create queries in jdbc yourself, delete things in several batches with explicit commits etc… It will stil take long then, use less memory, but with the risk of having an inconsistent db for processes where it fails half way. So you'd have to do it intelligently.

[frederikherema1]

We are aware of the stackoverflow  and are looking into a solution to fix this.

[ronald_van_kuij]

Did you try increasing the stack size?  e.g. by adding the flag -Xss1024k in the VM Arguments or  in mb by using -Xss1m for example or bigger

[razu]

I've been increasing stack size continuously, and finally after setting -Xss8m I was able to delete process instance, it took about 20 min. My process contains one multi-instance UserTask and 4500 instances of it.
Thank you for your help.

[ronald_van_kuij]

My process contains one multi-instance UserTask and 4500 instances of it.

What kind of process is this if I may ask?

[razu]

The process represent business client campaign. It contains of selection step. Selection should select list of business clients and add them some attributes. Each business client can be selected by many users, user who is responsible for business client select it and set him some attributes.
When all business clients have been selected, process ends generating some summary data.

The selection step is realized as multi-instance UserTask, each instance of this task represents one business client. Person responsible for specific business client takes this instance (via claim in API), add some attributes to it and close it (via complete in API). List of business clients that is processed have size = 4500, so 4500 instances of multi-instance UserTask is generated from that list.