Activiti deadlock occurs under JMeter performance test

[lepike]

Hi Guys!

One of my project uses Activiti Workflow Engine. The usage of the engine is very comfortable and I'm very satisfied with it, but I have a little problem with concurrency. We have a wicket spring application with embedded activiti workflow engine. It works fine without concurrent users, but during jmeter performance test activiti creates deadlock on its own tables. For example: ACT_RU_JOB, ACT_RU_EXECUTION, ACT_RU_VARIABLE. I've found a thread here (http://forums.activiti.org/en/viewtopic.php?f=6&t=682&start=0) that suggests using queues to start activiti processes. This solution didn't solve the problem, because deadlocks still occuring. I provide some configuration and stacktrace for you.
All answers can be useful for me.

my activiti configuration:

  

 <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
          <property name="databaseType" value="mssql" />
          <property name="dataSource" ref="dataSource" />
          <property name="transactionManager" ref="transactionManagerLugy" />
          <property name="databaseSchemaUpdate" value="true" />
          <property name="jobExecutorActivate" value="true" />
          <property name="deploymentResources" value="classpath*:/diagrams/idm/*.bpmn20.xml" />    
          <property name="history" value="none"/>      
          <property name="jdbcMaxActiveConnections" value="1000"/>
          <property name="jdbcMaxIdleConnections" value="10"/>
          <property name="jdbcMaxWaitTime" value="50000"/>
       </bean>   
       
    
       <bean id="processEngine" class="org.activiti.spring.ProcessEngineFactoryBean">
          <property name="processEngineConfiguration" ref="processEngineConfiguration" />
       </bean>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

stacktrace:

    ### Error querying database.  Cause: org.hibernate.exception.LockAcquisitionException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    ### The error may involve org.activiti.engine.impl.persistence.entity.VariableInstanceEntity.selectVariablesByExecutionId-Inline
    ### The error occurred while setting parameters
    ### Cause: org.hibernate.exception.LockAcquisitionException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
       at org.springframework.jms.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:471)
       at org.springframework.jms.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:355)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
       at java.lang.Thread.run(Thread.java:662)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Thanks for the help!

Levi

[ronald_van_kuij]

Creating indexes on relevant columns (for specifict queries you use) is the first thing to do. This prevents full table scans. Activiti comes with a basic set of indexes and with all the enhancements in the queries this can result in deadlocks

[lepike]

Thx for your help! I will check the activiti queries and the indexes on the tables. Hope it will help with deadlocks. MSSQL database can be a problem? Does it supported well by Activiti?

Best regards,
Levi

[lepike]

I have made some indexes on activiti tables. Now I have only one deadlock. It comes from the selectTaskByQueryCriteriaSql Ibatis sql. I cannot index the tables to make this complex sql to work without deadlock. Have you got any idea?

Thx!

Levi

[heymjo]

Does Activiti set the readonly flag on each connection where it only selects data ? That could help avoid this type of errors as well IIRC.

[ronald_van_kuij]

That could help to yes. I have no idea if Activiti has these kinds of things… there are ibatis config files with everything in it. So if you have suggestions/can try things, feel free

[lepike]

Thanks for the help. The problem still valid. I have no idea to solve…

[mahavirj]

do you mind sharing the indices you created?  also, spring has some transaction settings like propagation_required or propagation_supports that could be tinkered with.  what driver are you using for sqlserver?

[lepike]

JTDS 1.2.4 and the Microsoft JDBC driver were tested also.
The transaction management is the following:
on workflow service class: propagation=supports, readonly=true
on every crud method in this service class: propagation=required, readonly=false

So the startProcess function, every service task function and human task function has propagation=required, readonly=false settings, because they are crud methods in my application.

Is this a bad usage?
Thanks for the help.

Levi

[mahavirj]

looks ok….i use websphere and the article below was useful for txn settings using spring/hibernate. perhaps some of this may be of relevant to your scenario

http://www.ibm.com/developerworks/websphere/techjournal/0609_alcott/0609_alcott.html?ca=drs-