Activiti using two connections per thread

[gabrielfrancisc]

Hello! I'm having an issue. I'm trying to use Activiti + Camel, using Spring 4, and, I noticed a problem: Sometimes, all connections from pool get locked and never come back, only if I define a borrow timeout it "works", but I got a lot of timeouts when running on multi-thread. I was wondering if it was just a bottleneck or if it could be a deadlock, so, I tried to start my application using just one connection and… it did not start! I try to debug and I then I saw that there are two borrow connections in a row, without returning the first one back to the pool. Well, If I use the default ProcessEngineConfigurationImpl, without Spring transactions and stuff like that, it works, but, the activiti-camel wants the configuration to be a SpringProcessEngineConfiguration instance, so, I'm not able to use the ProcessEngineConfigurationImpl because of it. So, I imagine it could be a problem from SpringProcessEngineConfiguration or the transaction management.

Does anyone know what could be happening?

[trademak]

What is the size of your connection pool?
Do you have long running service tasks that don't free up connections?

Best regards,

[jbarrez]

You need at least two connections, since the engine will use a new transaction to fetch database id's when the pool of reserved id's hjas been used up. Alternatively, you could use the UUID generator, which doesn't need that.

[gabrielfrancisc]

So, that's what could be happening. Okay, so far, so good. Of course I will not use only one connection, I just thought the deadlock while the tasks where running could be the same as the one in startup, but as @jbarrez said, it is not.
Still, If I the thread number is higher or equal to the max connection pool size, I get a deadlock. That doesn't make sense at all, that's okay to get a bottleneck, but, if I do not set the max lifetime of a connection, the application freezes and never come back.
Is there a way to limit the tasks execution to the same number of connections in the pool? Because I will start processes from HTTP and from JMS, so, I cannot limit in the input. For example, If I have 30 connections, I want to limit Activiti to execute only 29 tasks in parallel, because I don't want to get borrow  timeouts all the time, like we are getting right now if the number of tasks running is higher than the connection pool size.

[gabrielfrancisc]

Oh, and I'm not using any connection in the tasks. Actually, I'm not using even the transactional context, just making REST calls with camel.

[jbarrez]

The amount of threads should not matter for your connection pool, that's why you want a connection pool in the first place …

Did you try changing the connection pool framework? Using Tomcat jdbc pool / c3p0 ?

[gabrielfrancisc]

I did. Right now I'm using dbcp2, but I also tried to use dbcp (1.x), c3p0 and BoneCP. Same behavior. I don't think it is the connection pool framework, but, anyway, is there any other framework that I could try?

[gabrielfrancisc]

Just used Tomcat JDBC Pool and HikariCP:

- Tomcat JDBC Pool hangs too, but, after few minutes everything come back to normal, but, I'm not sure it is a solution.
- HikariCP has the same behavior of the others, all the threads hanged util forever.

I thought maybe it was my Spring version, because I'm using Spring 4.1 and Activiti 5.16.4 uses 4.0.6, so, tried to downgrade, but nothing changed.

Is there any issue with Activiti 5.16.4 running on Java 8? If there is, so that could be the problem.

Well, here it is my configuration. Is there anything that I've forgotten?

<java>
@Configuration
@PropertySource("classpath:activiti-db.properties")
@EnableTransactionManagement
public class ActivitiContext {
    @Inject
    private ApplicationContext applicationContext;

    @Inject
    private ActivitiDataSourceInformation activitiDataSourceInformation;

    @Bean
    public ProcessEngineFactoryBean processEngine(ProcessEngineConfigurationImpl processEngineConfiguration) {
        ProcessEngineFactoryBean factoryBean = new ProcessEngineFactoryBean();

        factoryBean.setProcessEngineConfiguration(processEngineConfiguration);
        factoryBean.setApplicationContext(applicationContext);

        return factoryBean;
    }

    @Bean
    public ProcessEngineConfigurationImpl processEngineConfiguration() throws Exception {
        SpringProcessEngineConfiguration configuration = new SpringProcessEngineConfiguration();

        configuration.setDataSource(dataSource());
        configuration.setTransactionManager(transactionManager());
        configuration.setActivityBehaviorFactory(new SpringActivitiCamelBehaviorFactory(applicationContext));
        configuration.setDeploymentResources(new ClassPathResource[]{new ClassPathResource("flows/charge/")});
        configuration.setDatabaseSchemaUpdate("true");

        return configuration;
    }

    @Bean
    public PlatformTransactionManager transactionManager() throws Exception {
        return new DataSourceTransactionManager(dataSource());
    }

    @Bean(destroyMethod = "close")
    public DataSource dataSource() {
        BasicDataSource dataSource = new BasicDataSource();

        dataSource.setUsername(activitiDataSourceInformation.getUsername());
        dataSource.setPassword(activitiDataSourceInformation.getPassword());
        dataSource.setUrl(activitiDataSourceInformation.getUrl());
        dataSource.setDriverClassName(activitiDataSourceInformation.getDriver());
        dataSource.setMaxTotal(activitiDataSourceInformation.getMaxConnections());
        dataSource.setMinIdle(activitiDataSourceInformation.getMinConnectionsIdle());
        dataSource.setMaxWaitMillis(activitiDataSourceInformation.getMaxWait());
        dataSource.setMaxConnLifetimeMillis(activitiDataSourceInformation.getMaxConnectionLifetime());
        dataSource.setMaxIdle(activitiDataSourceInformation.getMaxIdle());

        return dataSource;
    }

    @Bean
    public RuntimeService runtimeService(ProcessEngine processEngine) throws Exception {
        return processEngine.getRuntimeService();
    }

    @Bean
    public RepositoryService repositoryService(ProcessEngine processEngine) throws Exception {
        return processEngine.getRepositoryService();
    }

    @Bean
    public ManagementService managementService(ProcessEngine processEngine) throws Exception {
        return processEngine.getManagementService();
    }

    @Bean
    public HistoryService historyService(ProcessEngine processEngine) throws Exception {
        return processEngine.getHistoryService();
    }

    @Bean
    public TaskService taskService(ProcessEngine processEngine) throws Exception {
        return processEngine.getTaskService();
    }
}
</java>

[jbarrez]

> Is there any issue with Activiti 5.16.4 running on Java 8? If there is, so that could be the problem.

No. I also run on JDK 8.

Could you wrap that config up with a simple unit test that demonstrates the issue? I dont see anything wrong straight away. Maybe a check query if the connection is still valid? But that shouldn't be needed …

[gabrielfrancisc]

Sorry for the delay, the project that will use Activiti here in my company was postponed and only now I started to work with it again.  I couldn't reproduce the problem in a unit test, but I'll try to do it later. I noticed that the threads blocked were waiting for the org.activiti.engine.impl.db.DbIdGenerator.getNextId() method to return, so, I tried to use StrongUuidGenerator instead of  the DbIdGenerator and the problem is gone. I still don't know why, I think maybe it is some issue running with the activiti-camel lib (maybe that's why i couldn't reproduce the problem in a unit test just using activiti),but, for now I will use the StrongUuidGenerator and run some tests.

Thanks again!