topic Multiple exceptions when stressing a workflow in Alfresco Archive

Multiple exceptions when stressing a workflow

rufini — Tue, 05 Jul 2011 16:11:33 GMT

Hi. I'm having random errors when stressing my app.I made a workflow with a UserTask that times out after 200 sec.With a JMeter, I set up:1) Request via GET the execution id "askForExecutionID"2) Post that execution id with some extra data to process "processData"In JMeter\Thread Group, I set: 100 t

Re: Multiple exceptions when stressing a workflow

rufini — Tue, 05 Jul 2011 20:47:15 GMT

Update:

I've found that the scheduled task (quartz) that trims the history is causing the Deadlock exception.

My task looks like:

  
  this.jdbcTemplate.update(
    "DELETE FROM ACT_HI_ACTINST WHERE END_TIME_ < ?", olderDate);

  this.jdbcTemplate.update(
    "DELETE FROM ACT_HI_PROCINST WHERE END_TIME_ < ?", olderDate);

  this.jdbcTemplate.update(
    "DELETE FROM ACT_HI_TASKINST WHERE END_TIME_ < ?", olderDate);

Any idea how could avoid the deadlock?

Thanks again,
rufini

Re: Multiple exceptions when stressing a workflow

ronald_van_kuij — Tue, 05 Jul 2011 21:28:53 GMT

First of all, isn't the delete of the process instance casading? Might not be, just not sure.

Secondly, maybe an index on the END_TIME might help if it is not already there. Might prevent full tablescans and thus locking. But…. I'm no db expert at all (more STILL a noob actually ops: )

Re: Multiple exceptions when stressing a workflow

rufini — Wed, 06 Jul 2011 15:24:50 GMT

Thanks, ronald!

I added key indexes to the "end_time_" fields, and stressed the app with ridiculous parameters, testing it hundred of times, and it didn't fail once.

It fixed it

Thanks!
rufini

Re: Multiple exceptions when stressing a workflow

ronald_van_kuij — Wed, 06 Jul 2011 16:36:10 GMT

Thanks for reporting back. From now on I will never ever call myself a DB n00b anymore. 🙂

Re: Multiple exceptions when stressing a workflow

crico — Mon, 18 Jun 2012 16:27:48 GMT

I have a listener associated with the event "end" of a process with the following logic:

public void notify(DelegateExecution execution) throws Exception {
            String processInstanceId = execution.getProcessInstanceId();

            Context.getCommandContext().getHistoricProcessInstanceManager()
                    .deleteHistoricProcessInstanceById(processInstanceId);
}

This generates the following exception when I launch multiple instances of the process:

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1064)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:22)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:51)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:29)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:88)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:43)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:122)

What could be the cause?

Thanks.

Re: Multiple exceptions when stressing a workflow

ronald_van_kuij — Mon, 18 Jun 2012 17:00:47 GMT

does this not happen if you start just 1?

Re: Multiple exceptions when stressing a workflow

crico — Mon, 18 Jun 2012 17:08:50 GMT

does this not happen if you start just 1?

No, it only happens when launching multiple processes concurrently. Specifically, the error occurs in the deleteHistoricProcessInstanceById method.

Re: Multiple exceptions when stressing a workflow

crico — Tue, 19 Jun 2012 11:57:00 GMT

does this not happen if you start just 1?

No, it only happens when launching multiple processes concurrently. Specifically, the error occurs in the deleteHistoricProcessInstanceById method.

I have also tried the following code:

     JdbcTemplate jdbcTemplate = (JdbcTemplate) ApplicationContextProvider
                    .getInstance().getBean("jdbcTemplate");

            jdbcTemplate.update(
                    "DELETE FROM ACT_HI_ACTINST WHERE PROC_INST_ID_ < ?",
                    processInstanceId);

            jdbcTemplate.update(
                    "DELETE FROM ACT_HI_PROCINST WHERE PROC_INST_ID_ < ?",
                    processInstanceId);

            jdbcTemplate.update(
                    "DELETE FROM ACT_HI_TASKINST WHERE PROC_INST_ID_ < ?",
                    processInstanceId);

and then I get this error:

org.springframework.dao.CannotAcquireLockException: PreparedStatementCallback; SQL [DELETE FROM ACT_HI_ACTINST WHERE PROC_INST_ID_ < ?]; Lock wait timeout exceeded; try restarting transaction; nested exception is java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction…

Re: Multiple exceptions when stressing a workflow

frederikherema1 — Wed, 20 Jun 2012 07:19:50 GMT

Deleting from the history through the HistoricXXManager (engine IMPL, not API) while the process is ending is generally not a good idea.

1. Can't you delete the history externally (from the method that is calling the API)
2. If you don't want history, why don't you just turn the history-level to none?

Re: Multiple exceptions when stressing a workflow

crico — Fri, 22 Jun 2012 10:15:57 GMT

Deleting from the history through the HistoricXXManager (engine IMPL, not API) while the process is ending is generally not a good idea.

1. Can't you delete the history externally (from the method that is calling the API)
2. If you don't want history, why don't you just turn the history-level to none?

2. I don't want to save the history for a particular case of a process (when a execution of this process does not meet a particular conditions, because this case generates millions of records in database), but I save it for others.

Is there any option to discard processes that do not persist in the history?

Regards.

Re: Multiple exceptions when stressing a workflow

frederikherema1 — Fri, 22 Jun 2012 10:30:50 GMT

There is no per-process history setting.

So the only option there is to accomplish this is to roll your own solution that does periodic historicProcessInstance queries for finished workflows of that certain type (or having certain conditions be met) and delete those through the API…

Perhaps write an activiti-process for this using a timer-start event

Re: Multiple exceptions when stressing a workflow

ronald_van_kuij — Fri, 22 Jun 2012 14:27:12 GMT

Perhaps Activiti could implement support of this element:

 <xsd:element name="auditing" type="tAuditing"/>
 <xsd:complexType name="tAuditing">
  <xsd:complexContent>
   <xsd:extension base="tBaseElement"/>
  </xsd:complexContent>
 </xsd:complexType>

And use this to support per process auditing/history? Would not be to difficult I think.