Hi,I am observing an undesired behavior with multithreaded use of Activiti.My program starts a number of threads, each of them starts a new business process. My business process is comprised of 4 Java Services, each of them just printing a message and sleeping for some time. (Service1, Service2a and Service2b in parallel, Service3)The problem is: in some cases the program does not finish, just hangs.I have analyzed the thread dump. Basically, I see a number of threads just waiting to get a db connection, like this:"Thread-16" prio=6 tid=0x000000000725c000 nid=0xec0 in Object.wait() [0x0000000008f9e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000f015dfa0> (a org.apache.commons.pool.impl.GenericObjectPool$Latch) at java.lang.Object.wait(Object.java:485) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1115)My logs suggest that database connections are only closed after the business process finishes, and there are cases when the JDBC connection is requested more than once during thread lifetime. In the effect, a deadlock takes place.For example, Thread-13 calls getConnection twice, and does not close the first one.(there's a little hack to monitor opening and closing db connections - HackedDataSource)process deployed OK!ok, got runtimeService –getConnection called for thread Thread-1 –getConnection called for thread Thread-2 –getConnection called for thread Thread-11 –getConnection called for thread Thread-6 –getConnection called for thread Thread-8 –getConnection finishes for thread Thread-2 –getConnection called for thread Thread-3 –getConnection called for thread Thread-10 –getConnection called for thread Thread-15 –getConnection called for thread Thread-12 –getConnection called for thread Thread-4 –getConnection called for thread Thread-14 –getConnection finishes for thread Thread-1 –getConnection called for thread Thread-7 –getConnection finishes for thread Thread-11Service 1 called in thread Thread-2Service 1 called in thread Thread-1 –getConnection called for thread Thread-5 –getConnection called for thread Thread-9 –getConnection called for thread Thread-13Service 1 called in thread Thread-11Service 1 finishes in thread Thread-1Service 2a called in thread Thread-1Service 1 finishes in thread Thread-2Service 2a called in thread Thread-2Service 1 finishes in thread Thread-11Service 2a called in thread Thread-11Service 2a finishes in thread Thread-1Service 2b called in thread Thread-1Service 2a finishes in thread Thread-11Service 2b called in thread Thread-11Service 2a finishes in thread Thread-2Service 2b called in thread Thread-2Service 2b called in thread Thread-11Service 2b called in thread Thread-1Service 3 called in thread Thread-11Service 3 called in thread Thread-1Service 2b called in thread Thread-2Service 3 called in thread Thread-2Service 3 finishes in thread Thread-11 –Connection.close called for thread Thread-11 –Connection.close finishes for thread Thread-11 –getConnection finishes for thread Thread-6Service 1 called in thread Thread-6Service 3 finishes in thread Thread-1 –Connection.close called for thread Thread-1 –Connection.close finishes for thread Thread-1 –getConnection finishes for thread Thread-8Service 1 called in thread Thread-8Service 3 finishes in thread Thread-2 –Connection.close called for thread Thread-2 –Connection.close finishes for thread Thread-2 –getConnection finishes for thread Thread-3Service 1 called in thread Thread-3Service 1 finishes in thread Thread-6Service 2a called in thread Thread-6Service 2a finishes in thread Thread-6Service 2b called in thread Thread-6Service 1 finishes in thread Thread-8Service 2a called in thread Thread-8Service 1 finishes in thread Thread-3Service 2a called in thread Thread-3Service 2a finishes in thread Thread-8Service 2b called in thread Thread-8Service 2b called in thread Thread-6Service 3 called in thread Thread-6Service 2a finishes in thread Thread-3Service 2b called in thread Thread-3Service 3 finishes in thread Thread-6 –Connection.close called for thread Thread-6 –Connection.close finishes for thread Thread-6 –getConnection finishes for thread Thread-10Service 1 called in thread Thread-10Service 2b called in thread Thread-8Service 3 called in thread Thread-8Service 1 finishes in thread Thread-10Service 2a called in thread Thread-10Service 2b called in thread Thread-3Service 3 called in thread Thread-3Service 3 finishes in thread Thread-8 –Connection.close called for thread Thread-8 –Connection.close finishes for thread Thread-8 –getConnection finishes for thread Thread-15Service 1 called in thread Thread-15Service 2a finishes in thread Thread-10Service 2b called in thread Thread-10Service 3 finishes in thread Thread-3 –Connection.close called for thread Thread-3 –Connection.close finishes for thread Thread-3 –getConnection finishes for thread Thread-12Service 1 called in thread Thread-12Service 1 finishes in thread Thread-15Service 2a called in thread Thread-15Service 1 finishes in thread Thread-12Service 2a called in thread Thread-12Service 2b called in thread Thread-10Service 3 called in thread Thread-10Service 2a finishes in thread Thread-15Service 2b called in thread Thread-15Service 2a finishes in thread Thread-12Service 2b called in thread Thread-12Service 3 finishes in thread Thread-10 –Connection.close called for thread Thread-10 –Connection.close finishes for thread Thread-10 –getConnection finishes for thread Thread-4Service 1 called in thread Thread-4Service 2b called in thread Thread-15Service 3 called in thread Thread-15Service 2b called in thread Thread-12Service 3 called in thread Thread-12Service 3 finishes in thread Thread-15 –Connection.close called for thread Thread-15 –Connection.close finishes for thread Thread-15 –getConnection finishes for thread Thread-14Service 1 called in thread Thread-14Service 1 finishes in thread Thread-4Service 2a called in thread Thread-4Service 3 finishes in thread Thread-12 –Connection.close called for thread Thread-12 –Connection.close finishes for thread Thread-12 –getConnection finishes for thread Thread-7Service 1 called in thread Thread-7Service 1 finishes in thread Thread-7Service 2a called in thread Thread-7Service 2a finishes in thread Thread-4Service 2b called in thread Thread-4Service 1 finishes in thread Thread-14Service 2a called in thread Thread-14Service 2b called in thread Thread-4Service 2a finishes in thread Thread-14Service 2b called in thread Thread-14Service 3 called in thread Thread-4Service 2a finishes in thread Thread-7Service 2b called in thread Thread-7Service 3 finishes in thread Thread-4Service 2b called in thread Thread-7Service 3 called in thread Thread-7 –Connection.close called for thread Thread-4 –Connection.close finishes for thread Thread-4 –getConnection finishes for thread Thread-5Service 1 called in thread Thread-5Service 2b called in thread Thread-14Service 3 called in thread Thread-14Service 3 finishes in thread Thread-7 –Connection.close called for thread Thread-7 –Connection.close finishes for thread Thread-7 –getConnection finishes for thread Thread-9Service 1 called in thread Thread-9Service 1 finishes in thread Thread-5Service 2a called in thread Thread-5Service 3 finishes in thread Thread-14Service 1 finishes in thread Thread-9Service 2a called in thread Thread-9 –Connection.close called for thread Thread-14 –Connection.close finishes for thread Thread-14 –getConnection finishes for thread Thread-13Service 1 called in thread Thread-13Service 2a finishes in thread Thread-5Service 2b called in thread Thread-5Service 1 finishes in thread Thread-13 –getConnection called for thread Thread-13Service 2a finishes in thread Thread-9Service 2b called in thread Thread-5(we're stuck here)Of course the problem is not 100% repeatable, it takes a bad luck with the threads interleave.The main program looks as follows:
package pac1;
import java.util.ArrayList;
import java.util.List;
import javax.sql.DataSource;
import org.activiti.engine.ProcessEngine;
import org.activiti.engine.ProcessEngineConfiguration;
import org.activiti.engine.RepositoryService;
import org.activiti.engine.RuntimeService;
import org.activiti.engine.repository.Deployment;
import org.apache.commons.dbcp.BasicDataSource;
public class Prog3 {
public static void main(String[] args) throws Exception {
ProcessEngine processEngine = null;
BasicDataSource myDataSource = null;
try {
System.out.println("starting!");
int threadNumber = 15;
int maxConnectionNumber = 3;
myDataSource = new org.apache.commons.dbcp.BasicDataSource();
myDataSource.setDriverClassName("org.postgresql.Driver");
myDataSource.setUrl("jdbc:postgresql://127.0.0.1:54322/activiti");
myDataSource.setUsername("activiti");
myDataSource.setPassword("activiti");
myDataSource.setMaxActive(maxConnectionNumber);
myDataSource.setMinIdle(0);
System.out.println("max Active: " + myDataSource.getMaxActive());
DataSource ds = new HackedDataSource(myDataSource);
ProcessEngineConfiguration processEngineConfiguration = ProcessEngineConfiguration
.createStandaloneProcessEngineConfiguration()
.setDataSource(ds);
processEngine = processEngineConfiguration.buildProcessEngine();
System.out.println("processEngine created OK!");
// Deploy the test process
RepositoryService repositoryService = processEngine
.getRepositoryService();
Deployment deployment = repositoryService.createDeployment()
.addClasspathResource("tescior3.bpmn20.xml").deploy();
System.out.println("process deployed OK!");
final RuntimeService runtimeService = processEngine
.getRuntimeService();
System.out.println("ok, got runtimeService");
// Start a number of process instances
List<Thread> myThreads = new ArrayList<Thread>();
for (int i = 0; i < threadNumber; i++) {
Thread thr = new Thread(new Runnable() {
@Override
public void run() {
runtimeService.startProcessInstanceByKey("test3");
}
});
thr.start();
myThreads.add(thr);
}
for (Thread t : myThreads) {
t.join();
}
System.out.println("ok, all threads joined");
} finally {
if (processEngine != null)
try {
System.out.println("about to close process engine…");
processEngine.close();
System.out.println("process engine closed");
} catch (Exception e) {
e.printStackTrace();
}
if (myDataSource != null)
try {
System.out.println("about to close data source…");
myDataSource.close();
System.out.println("data source closed");
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
I am using Activiti 5.4, connected to Postgres, with external DataSource configuration.BTW, if the max connection number is set to 1, I cannot deploy the process 😞Thanks,Marcin Jaskolski
I mean the syncronized method on getNextId() -> which does the second call for an sqlConnection. So if the pool is empty, the thread is stuck in the getNextId() method, which is syncronized. All other treads can't enter and wait as wel, not releasing SQLconnection.
