topic Re: Deadlock exception during timer job executions in Alfresco Archive

Deadlock exception during timer job executions

ganeshr — Thu, 03 Jul 2014 16:21:51 GMT

Hi, I'm seeing the below exception in my log file more frequently. Not sure why this exception is occurring. We are using activiti 5.15. Can any help me resolving this issue… Here is the log messagerg.activiti.engine.impl.jobexecutor.AcquireJobsRunnable:97 - exception during job acquisition: ###

Re: Deadlock exception during timer job executions

trademak — Thu, 03 Jul 2014 16:30:23 GMT

It's difficult to tell without knowing more about your environment, the process definitions you are running etc.

Best regards,

Re: Deadlock exception during timer job executions

ganeshr — Thu, 03 Jul 2014 16:36:06 GMT

We are running the application with activiti engine in cluster environment.
And we have a workflow with single human task, on this task we have a timer boundary event which will invoke the service task. This service task will perform operation of sending a notification mail. Timer boundary event has a time duration of 1 hour. so after one hour if task is still active then it should invoke service task. Apart from that this workflow does not do any operations.

Re: Deadlock exception during timer job executions

jbarrez — Mon, 07 Jul 2014 13:07:32 GMT

That is very odd and shouldn't happend. How is this database set up? Whats the transaction isolation level?

How many nodes do you have with Activiti and how many of them are job executors?

Re: Deadlock exception during timer job executions

jaikumar — Wed, 09 Jul 2014 15:15:02 GMT

Hi,

Am also facing the same issue. Once the timer boundary event started, event is not waiting for given duration of 10 minutes i.e(PT10M) and also tried with cron expression as (* 0/10 * * * ?). Please suggest a solution.

Re: Deadlock exception during timer job executions

ganeshr — Wed, 09 Jul 2014 15:34:10 GMT

We are using Oracle database with org.apache.commons.dbcp.BasicDataSource. We have not set any isolation level. Activiti engine is running in 4 nodes and each node has job executor enabled.

Re: Deadlock exception during timer job executions

jbarrez — Wed, 09 Jul 2014 16:14:54 GMT

@jaikumar: you shouldn't use timcecycle here. That is for repeating stuff.
Also, does this give you a deadlock issue?

@ganeshr: Should be fine, oracle has a default correct isolation setting normally.
4 job executors does sound like overkill, 2 should be fine already.
However, that still shouldn't give you these errors….

Is this happening for one process specifically? Could you share it or something similar that looks like it?

Re: Deadlock exception during timer job executions

ganeshr — Tue, 22 Jul 2014 14:06:59 GMT

Unable to figure out for which process it is happening. Here is one of the process

<code>
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlnsmgdc="http://www.omg.org/spec/DD/20100524/DC" xmlnsmgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/test">
<process id="TsmExceptionWorkflowProcess" name="TSM Exception Workflow Process" isExecutable="true">
    <userTask id="TsmExceptionWorkflowHumanTask" name="TSM exception workflow human task" activiti:candidateGroups="${TsmExceptionWorkflowHumanTask_AssignedGroupUser}">
      <extensionElements>
        <activiti:formProperty id="detailPageUrl" name="BPE Detail Page URL" type="string" expression="${detailPageUrl}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="serviceName" name="Service Name" type="string" expression="${serviceName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="deploymentUnitName" name="Deployment Unit Name" type="string" expression="${deploymentUnitName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmSection" name="Tsm Section" type="string" expression="${tsmSection}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmSymbol" name="Tsm Symbol" type="string" expression="${tsmSymbol}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmDate" name="Tsm Date" type="date" expression="${tsmDate}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="CustomActions" name="Custom Actions" type="enum">
          <activiti:value id="CLOSE" name="CLOSE"></activiti:value>
          <activiti:value id="DELETE" name="DELETE"></activiti:value>
        </activiti:formProperty>
        <activiti:formProperty id="eventName" name="Event Name" type="string" expression="${eventName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="ErrorList" name="Error List" type="string" expression="${ErrorList}" writable="false"></activiti:formProperty>
        <activiti:taskListener event="all" expression="#{workflowTaskListner.notify(task)}"></activiti:taskListener>
      </extensionElements>
    </userTask>
    <endEvent id="endevent1" name="End"></endEvent>
    <sequenceFlow id="flow2" name="end" sourceRef="TsmExceptionWorkflowHumanTask" targetRef="endevent1"></sequenceFlow>
    <boundaryEvent id="TsmExceptionExpirationTimeDuration" name="Tsm Exception Expiration Time Duration" attachedToRef="TsmExceptionWorkflowHumanTask" cancelActivity="true">
      <timerEventDefinition>
        <timeDuration>${TsmExceptionWorkflowHumanTask_TaskExpirationDuration}</timeDuration>
      </timerEventDefinition>
    </boundaryEvent>
    <sequenceFlow id="flow3" name="After Task Expiration Time" sourceRef="TsmExceptionExpirationTimeDuration" targetRef="ExpireTaskStateChangeEventTriggerService"></sequenceFlow>
    <serviceTask id="EscalationServiceTask" name="Escalation service task" activiti:delegateExpression="#{escalationServiceTask}">
      <extensionElements>
        <activiti:field name="notificationMailContent">
          <activiti:string>This is to inform you that the following task(s) have been assigned to you: https://${hostUrl}/bpm_worklist/secure/jas/details?domain=${workflowDomain}&tasknumber=${taskId}. %n%n For more information, visit the BPEL Worklist Application:https://${hostUrl}/bpm_worklist/secure/jas/home</activiti:string>
        </activiti:field>
        <activiti:field name="notificationMailSubject">
          <activiti:string>${source} workflow task created (${env} environment)</activiti:string>
        </activiti:field>
        <activiti:field name="reassignMailSubject">
          <activiti:string>UP Workflow Notification - ${env}</activiti:string>
        </activiti:field>
        <activiti:field name="taskDefinitionKey">
          <activiti:string>TsmExceptionWorkflowHumanTask</activiti:string>
        </activiti:field>
      </extensionElements>
    </serviceTask>
    <boundaryEvent id="escalationBoundaryTimer" name="escalationBoundaryTimer" attachedToRef="TsmExceptionWorkflowHumanTask" cancelActivity="false">
      <timerEventDefinition>
        <timeDuration>PT10S</timeDuration>
      </timerEventDefinition>
    </boundaryEvent>
    <startEvent id="startevent1" name="Start"></startEvent>
    <sequenceFlow id="flow14" sourceRef="startevent1" targetRef="TsmExceptionWorkflowHumanTask"></sequenceFlow>
    <sequenceFlow id="flow15" sourceRef="escalationBoundaryTimer" targetRef="EscalationServiceTask"></sequenceFlow>
    <serviceTask id="ExpireTaskStateChangeEventTriggerService" name="Expire task state change event trigger" activiti:delegateExpression="#{workflowTaskStateChangeEventTriggerService}">
      <extensionElements>
        <activiti:field name="actionPerformed">
          <activiti:string>EXPIRED</activiti:string>
        </activiti:field>
      </extensionElements>
    </serviceTask>
    <sequenceFlow id="flow16" sourceRef="ExpireTaskStateChangeEventTriggerService" targetRef="endevent1"></sequenceFlow>
</process>
</code>

We are seeing this exception more frequently, not able to figure out how to resolve. Will this deadlock have any impact on timer jobs on process?

Re: Deadlock exception during timer job executions

jbarrez — Fri, 25 Jul 2014 07:39:39 GMT

> Will this deadlock have any impact on timer jobs on process?

No.

I also don't see anything wrong with this process. Could you try to run it with fewer job executors and see what the impact is?

Re: Deadlock exception during timer job executions

ganeshr — Tue, 05 Apr 2016 18:09:57 GMT

We are not seeing this exception in test environment. We see only this exception in production and we have similar infrastructure in both test and prod environments. We tried with different approaches but we could not resolve this exception.

Re: Deadlock exception during timer job executions

jbarrez — Thu, 07 Apr 2016 10:18:02 GMT

Which version? The latest version has many bugfixes wrt these kind of problems.

Re: Deadlock exception during timer job executions

ganeshr — Thu, 07 Apr 2016 13:11:14 GMT

We are using 5.17 version.

Re: Deadlock exception during timer job executions

jbarrez — Tue, 12 Apr 2016 09:17:12 GMT

Then it is certainly worthwhile upgrading to the latest version. A lot of changes have been made in this area.

Re: Deadlock exception during timer job executions

elpyaz — Tue, 18 Jul 2017 19:49:01 GMT

I am facing the same issue like ganeshr.

Hi ganeshr any luck?

Hi Barrez, could you please help me, how can I disable jobExecutors on few nodes.

If we enable the executor on few nodes, how to handle the fail over

Re: Deadlock exception during timer job executions

ganeshr — Tue, 18 Jul 2017 20:19:43 GMT

We have upgraded to latest version at that time and used async job executor instead. And all of our nodes have async job executor enabled.

We are using below configuration, please verify and may be this might help you.

<bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">

<property name="databaseType" value="oracle" />
<property name="dataSource" ref="dataSource" />
<property name="transactionManager" ref="workflowTransactionManager" />
<property name="history" value="full" />
<property name="databaseSchemaUpdate" value="false" />
<property name="jobExecutorActivate" value="false" />
<property name="asyncExecutorEnabled" value="true" />
<property name="asyncExecutorActivate" value="true" />
<property name="bulkInsertEnabled" value="true" />
<property name="processEngineName" value="default"></property>
<property name="enableDatabaseEventLogging" value="false" />
<property name="delegateExpressionFieldInjectionMode" value= "MIXED" />
<property name="idGenerator">
<bean class="org.activiti.engine.impl.persistence.StrongUuidGenerator" />
</property>
<property name="customMybatisMappers">
<set>
<value>com.workflow.customsql.CustomSqlQueryMapper</value>
</set>
</property>
<property name="typedEventListeners">
<map>
<entry key="VARIABLE_UPDATED">
<list>
<ref bean="variableUpdateEventListner" />
</list>
</entry>
</map>
</property>
</bean>