Issue with Custom job executor Activiti 5.17

[ganeshr]

We are using Activiti 5.17.0 version and to improve the performance of jobs tried to implement custom job executor in process engine.
After configuring custom job executor with custom pool configuration we are getting UniversalConnectionPoolException while executing the jobs. We are running activiti engine in cluster environment with four instances.
If we remove the custom job executor setting from process engine configurations then everything is working as expected. Not able to figure out why the custom job configurations are giving UCP connection issues.

Can anyone help on this.

Below is the configuration:
<code>
<bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
    <property name="dataSource" ref="customDataSource" />
    <property name="history" value="full" />
    <property name="jobExecutorActivate" value="true" />
    <property name="jobExecutor" ref="customJobExecutor" />
</bean>

  <bean id="customJobExecutor" class="com.uprr.bpm.activiti.custom.CustomJobExecutor">
   <property name="queueSize" value="100"></property>
   <property name="corePoolSize" value="15"></property>
   <property name="maxPoolSize" value="50"></property> 
   <property name="keepAliveTime" value="30"></property>
  </bean>

ucp pool configuration
<ucpool-manager id="ucp.pool.manager">
    <ucpooled-datasource id="ucp.datasource.id"
      user="${ncs.datasource.user}"
      password="${ncs.datasource.pwd}"
      url="${ncs.datasource.bpe.enpr.url}"
      fast-connection-failover-enabled="${ncs.datasource.fast-connection-failover-enabled}"
      ONS-configuration="${ncs.datasource.enpr.ONSConfiguration}"
      initial-pool-size="${ncs.datasource.pool.initial-pool-size}"
      max-pool-size="${ncs.datasource.pool.max-pool-size}"
      min-pool-size="${ncs.datasource.pool.min-pool-size}"
      max-statements="${ncs.datasource.pool.max-statements}"
      validate-connection-on-borrow="true"
      inactive-connection-timeout="${ncs.datasource.pool.inactive-connection-timeout}"
       login-timeout="${ncs.datasource.pool.login-timeout}" />
  </ucpool-manager>

ncs.datasource.fast-connection-failover-enabled = true
ncs.datasource.pool.max-pool-size = 50
ncs.datasource.pool.min-pool-size = 5
ncs.datasource.pool.initial-pool-size = 5
ncs.datasource.pool.max-statements=5
ncs.datasource.pool.inactive-connection-timeout = 60
ncs.datasource.pool.login-timeout = 10

Below is the exception:

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLException: Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: All connections in the Universal Connection Pool are in use
   at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.handleExistingTransaction(AbstractPlatformTransactionManager.java:424)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:348)
   at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
   at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
   at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
   at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
   at org.activiti.engine.impl.jobexecutor.FailedJobListener.execute(FailedJobListener.java:47)
   at org.activiti.spring.SpringTransactionContext$4.afterCompletion(SpringTransactionContext.java:97)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCompletion(TransactionSynchronizationUtils.java:168)
<code>

[vasile_dirla]

HI,
have a look here http://docs.oracle.com/cd/E18283_01/java.112/e12265/optimize.htm
maybe it will help you configuring the connection pool.

just for testing purpose try increasing the "ncs.datasource.pool.max-pool-size" value to 100.  (this is the max connections as I know)
and then use it intenssive for a longer time to see if it will reach the limit again.
if the limit is reach again then there is a problem with reusing connections,
Anyway you should monitor the number of opened connections in the DB server to see if it is increasing in time

[ganeshr]

After setting queueSize and corePoolSize to same then we are not getting the exception. But however we are still monitoring the connections to see if we again receive this exception.

[ganeshr]

We are again seeing the exception. We tried increasing the connections to 100 as well but it is not working. Not sure why it is giving problem when we use custom job executor otherwise we are not getting that exception but the performance of job executor is very slow.
Is there any other way to improve the performance of job executor?

[trademak]

What do you mean with "the performance of the job executor is very slow"? Which job executor are you using exactly?

Best regards,

[ganeshr]

We initially used DefaultJobExecutor, as this is very slow. We have created a custom Job executor and configured the same in process engine configuration with the above configuration. But we are getting the exception I mentioned in my previous comments.
If we get back back to default job executor then we are not getting the connection exceptions.

[jbarrez]

The exception does point to a badly configured connection pool (or a database setting that limits the number of connections).

[jbarrez]

The exception does point to a badly configured connection pool (or a database setting that limits the number of connections).

[ganeshr]

We have set with enough database connection, please see my first comments that has database configurations. Even I increased database connection from 50 to 100 but still facing the same error.