Activiti deadlock occurs under JMeter performance test

[lepike]

Hi Guys!

One of my project uses Activiti Workflow Engine. The usage of the engine is very comfortable and I'm very satisfied with it, but I have a little problem with concurrency. We have a wicket spring application with embedded activiti workflow engine. It works fine without concurrent users, but during jmeter performance test activiti creates deadlock on its own tables. For example: ACT_RU_JOB, ACT_RU_EXECUTION, ACT_RU_VARIABLE. I've found a thread here (http://forums.activiti.org/en/viewtopic.php?f=6&t=682&start=0) that suggests using queues to start activiti processes. This solution didn't solve the problem, because deadlocks still occuring. I provide some configuration and stacktrace for you.
All answers can be useful for me.

my activiti configuration:

  

 <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
          <property name="databaseType" value="mssql" />
          <property name="dataSource" ref="dataSource" />
          <property name="transactionManager" ref="transactionManagerLugy" />
          <property name="databaseSchemaUpdate" value="true" />
          <property name="jobExecutorActivate" value="true" />
          <property name="deploymentResources" value="classpath*:/diagrams/idm/*.bpmn20.xml" />    
          <property name="history" value="none"/>      
          <property name="jdbcMaxActiveConnections" value="1000"/>
          <property name="jdbcMaxIdleConnections" value="10"/>
          <property name="jdbcMaxWaitTime" value="50000"/>
       </bean>   
       
    
       <bean id="processEngine" class="org.activiti.spring.ProcessEngineFactoryBean">
          <property name="processEngineConfiguration" ref="processEngineConfiguration" />
       </bean>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

stacktrace:

    ### Error querying database.  Cause: org.hibernate.exception.LockAcquisitionException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
    ### The error may involve org.activiti.engine.impl.persistence.entity.VariableInstanceEntity.selectVariablesByExecutionId-Inline
    ### The error occurred while setting parameters
    ### Cause: org.hibernate.exception.LockAcquisitionException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
       at org.springframework.jms.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:471)
       at org.springframework.jms.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:355)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:535)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:495)
       at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
       at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)
       at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)
       at java.lang.Thread.run(Thread.java:662)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Thanks for the help!

Levi

[notizklotz]

I've got similar problems on MS SQL 2008. I was able to resolve some deadlock situations by enabling READ_COMMITTED_SNAPSHOT which is an alternative implementation of the READ COMMITED level which is closer to how Oracle and MySQL implement it.
http://msdn.microsoft.com/en-us/library/ee384251%28v=sql.100%29.aspx
http://msdn.microsoft.com/en-us/library/ms177404.aspx

However, I still got deadlocks when multiple users are sending signals in different process instances.

@lepike
Could you please share the index optimizations you did?

[ronald_van_kuij]

Besides the ideces, one important 'improvement' could be using the 'other' id generator:

http://jira.codehaus.org/browse/ACT-789

[flavio_donze]

Just for the record, the answer here "solved" the dead lock problems:
http://stackoverflow.com/questions/6662504/activiti-deadlock-occurs-under-jmeter-performance-test
"I removed all of the timers from my activiti diagram and deadlocks are disappeared. I don't know why, but it works."

[ronald_van_kuij]

Why is removing all the timers a solution? I don't get that. Aounds like havily modifying your process.

[flavio_donze]

Hello Ronald

Your right it is not a solution, just wanted to make clear that it solved the transaction problem.

In my case I just had to remove one.

greets
Flavio

[trond_nordli]

We had a similar problem on MS SQL 2008 (SP3) but this problem seems to have disappeared after we upgraded the project from Activiti version 5.12.1 to Activiti version 5.13.

We have also tested this out successfully in Activi Benchmark project, where Activiti version 5.13 works but the older version does not work (dead locks problem) on MS SQL 2008.

Activiti Benchmark project:
http://www.jorambarrez.be/blog/2012/06/28/the-activiti-performance-showdown/

[frederikherema1]

There were some indexed added to FK-columns to prevent full-table locking of FK-referenced entities. However, not sure this has made the 5.13-realease (for mssql)… Anyway, glad the issue is sorted now