Hyland Connect

ganeshr · ‎07-03-2014

Hi,

I'm seeing the below exception in my log file more frequently. Not sure why this exception is occurring.
We are using activiti 5.15. Can any help me resolving this issue…

Here is the log message:

org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable:97 - exception during job acquisition:
### Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.updateTimer-Inline
### The error occurred while setting parameters
### SQL: update ACT_RU_JOB      SET REV_ = ?,        LOCK_EXP_TIME_ = ?,        LOCK_OWNER_ = ?,        RETRIES_ = ?,        EXCEPTION_STACK_ID_ = ?,        EXCEPTION_MSG_ = ?,        DUEDATE_ = ?      where ID_= ?       and REV_ = ?
### Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.updateTimer-Inline
### The error occurred while setting parameters
### SQL: update ACT_RU_JOB      SET REV_ = ?,        LOCK_EXP_TIME_ = ?,        LOCK_OWNER_ = ?,        RETRIES_ = ?,        EXCEPTION_STACK_ID_ = ?,        EXCEPTION_MSG_ = ?,        DUEDATE_ = ?      where ID_= ?       and REV_ = ?
### Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

   at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:147)
   at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:558)
   at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:442)
   at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:169)
   at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:116)
   at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:70)
   at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:42)
   at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131)
   at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:40)
   at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
   at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:59)
   at java.lang.Thread.run(Thread.java:744)
Caused by: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource

   at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
   at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
   at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:837)
   at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:445)
   at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
   at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
   at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
   at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1010)
   at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
   at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
   at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3677)
   at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:41)
   at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:66)
   at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:45)
   at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:108)
   at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:145)
   … 11 more

trademak · ‎07-03-2014

It's difficult to tell without knowing more about your environment, the process definitions you are running etc.

Best regards,

ganeshr · ‎07-03-2014

We are running the application with activiti engine in cluster environment.
And we have a workflow with single human task, on this task we have a timer boundary event which will invoke the service task. This service task will perform operation of sending a notification mail. Timer boundary event has a time duration of 1 hour. so after one hour if task is still active then it should invoke service task. Apart from that this workflow does not do any operations.

jbarrez · ‎07-07-2014

That is very odd and shouldn't happend. How is this database set up? Whats the transaction isolation level?

How many nodes do you have with Activiti and how many of them are job executors?

jaikumar · ‎07-09-2014

Hi,

Am also facing the same issue. Once the timer boundary event started, event is not waiting for given duration of 10 minutes i.e(PT10M) and also tried with cron expression as (* 0/10 * * * ?). Please suggest a solution.

ganeshr · ‎07-09-2014

We are using Oracle database with org.apache.commons.dbcp.BasicDataSource. We have not set any isolation level. Activiti engine is running in 4 nodes and each node has job executor enabled.

jbarrez · ‎07-09-2014

@jaikumar: you shouldn't use timcecycle here. That is for repeating stuff.
Also, does this give you a deadlock issue?

@ganeshr: Should be fine, oracle has a default correct isolation setting normally.
4 job executors does sound like overkill, 2 should be fine already.
However, that still shouldn't give you these errors….

Is this happening for one process specifically? Could you share it or something similar that looks like it?

ganeshr · ‎07-22-2014

Unable to figure out for which process it is happening. Here is one of the process

<code>
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns

mgdc="http://www.omg.org/spec/DD/20100524/DC" xmlns

mgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/test">
<process id="TsmExceptionWorkflowProcess" name="TSM Exception Workflow Process" isExecutable="true">
    <userTask id="TsmExceptionWorkflowHumanTask" name="TSM exception workflow human task" activiti:candidateGroups="${TsmExceptionWorkflowHumanTask_AssignedGroupUser}">
      <extensionElements>
        <activiti:formProperty id="detailPageUrl" name="BPE Detail Page URL" type="string" expression="${detailPageUrl}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="serviceName" name="Service Name" type="string" expression="${serviceName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="deploymentUnitName" name="Deployment Unit Name" type="string" expression="${deploymentUnitName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmSection" name="Tsm Section" type="string" expression="${tsmSection}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmSymbol" name="Tsm Symbol" type="string" expression="${tsmSymbol}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="tsmDate" name="Tsm Date" type="date" expression="${tsmDate}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="CustomActions" name="Custom Actions" type="enum">
          <activiti:value id="CLOSE" name="CLOSE"></activiti:value>
          <activiti:value id="DELETE" name="DELETE"></activiti:value>
        </activiti:formProperty>
        <activiti:formProperty id="eventName" name="Event Name" type="string" expression="${eventName}" writable="false"></activiti:formProperty>
        <activiti:formProperty id="ErrorList" name="Error List" type="string" expression="${ErrorList}" writable="false"></activiti:formProperty>
        <activiti:taskListener event="all" expression="#{workflowTaskListner.notify(task)}"></activiti:taskListener>
      </extensionElements>
    </userTask>
    <endEvent id="endevent1" name="End"></endEvent>
    <sequenceFlow id="flow2" name="end" sourceRef="TsmExceptionWorkflowHumanTask" targetRef="endevent1"></sequenceFlow>
    <boundaryEvent id="TsmExceptionExpirationTimeDuration" name="Tsm Exception Expiration Time Duration" attachedToRef="TsmExceptionWorkflowHumanTask" cancelActivity="true">
      <timerEventDefinition>
        <timeDuration>${TsmExceptionWorkflowHumanTask_TaskExpirationDuration}</timeDuration>
      </timerEventDefinition>
    </boundaryEvent>
    <sequenceFlow id="flow3" name="After Task Expiration Time" sourceRef="TsmExceptionExpirationTimeDuration" targetRef="ExpireTaskStateChangeEventTriggerService"></sequenceFlow>
    <serviceTask id="EscalationServiceTask" name="Escalation service task" activiti:delegateExpression="#{escalationServiceTask}">
      <extensionElements>
        <activiti:field name="notificationMailContent">
          <activiti:string>This is to inform you that the following task(s) have been assigned to you: https://${hostUrl}/bpm_worklist/secure/jas/details?domain=${workflowDomain}&tasknumber=${taskId}. %n%n For more information, visit the BPEL Worklist Application:https://${hostUrl}/bpm_worklist/secure/jas/home</activiti:string>
        </activiti:field>
        <activiti:field name="notificationMailSubject">
          <activiti:string>${source} workflow task created (${env} environment)</activiti:string>
        </activiti:field>
        <activiti:field name="reassignMailSubject">
          <activiti:string>UP Workflow Notification - ${env}</activiti:string>
        </activiti:field>
        <activiti:field name="taskDefinitionKey">
          <activiti:string>TsmExceptionWorkflowHumanTask</activiti:string>
        </activiti:field>
      </extensionElements>
    </serviceTask>
    <boundaryEvent id="escalationBoundaryTimer" name="escalationBoundaryTimer" attachedToRef="TsmExceptionWorkflowHumanTask" cancelActivity="false">
      <timerEventDefinition>
        <timeDuration>PT10S</timeDuration>
      </timerEventDefinition>
    </boundaryEvent>
    <startEvent id="startevent1" name="Start"></startEvent>
    <sequenceFlow id="flow14" sourceRef="startevent1" targetRef="TsmExceptionWorkflowHumanTask"></sequenceFlow>
    <sequenceFlow id="flow15" sourceRef="escalationBoundaryTimer" targetRef="EscalationServiceTask"></sequenceFlow>
    <serviceTask id="ExpireTaskStateChangeEventTriggerService" name="Expire task state change event trigger" activiti:delegateExpression="#{workflowTaskStateChangeEventTriggerService}">
      <extensionElements>
        <activiti:field name="actionPerformed">
          <activiti:string>EXPIRED</activiti:string>
        </activiti:field>
      </extensionElements>
    </serviceTask>
    <sequenceFlow id="flow16" sourceRef="ExpireTaskStateChangeEventTriggerService" targetRef="endevent1"></sequenceFlow>
</process>
</code>

We are seeing this exception more frequently, not able to figure out how to resolve. Will this deadlock have any impact on timer jobs on process?

jbarrez · ‎07-25-2014

> Will this deadlock have any impact on timer jobs on process?

No.

I also don't see anything wrong with this process. Could you try to run it with fewer job executors and see what the impact is?

ganeshr · ‎04-05-2016

We are not seeing this exception in test environment. We see only this exception in production and we have similar infrastructure in both test and prod environments. We tried with different approaches but we could not resolve this exception.

Hyland Connect

Deadlock exception during timer job executions