Activiti can't get DB connection

[tomsko]

Hi,
I've put Activiti under some high load and faced problems with DB connections. For first ~minute processing was fine, but later everything freezed just to throw "IJ000453: Unable to get managed connection for java:jboss/activitids" exception 90 secs later.
<!–break–>
Activiti Engine instance has it's own db connection pool from JBoss 7.2 and is configured to consume 20 DB connections at max.

When analyzing thread dump in the 90sec period between the freeze and the stack, I've found that every Activiti operation which keeps DB connection is blocked at DbIdGenerator.getNextId e.g.:

"pool-15-thread-125" prio=10 tid=0x00007ff8381a8800 nid=0x2179 waiting for monitor entry [0x00007ff7e3324000]
   java.lang.Thread.State: BLOCKED (on object monitor)
   at org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:35)
   - waiting to lock <0x000000074c929640> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbSqlSession.insert(DbSqlSession.java:111)
   at org.activiti.engine.impl.persistence.entity.JobEntity.insert(JobEntity.java:83)
   at org.activiti.engine.impl.persistence.entity.JobEntityManager.send(JobEntityManager.java:43)
   at org.activiti.engine.impl.persistence.entity.ExecutionEntity.scheduleAtomicOperationAsync(ExecutionEntity.java:545)
        ….
‍‍‍‍‍‍‍‍‍‍‍

Such threads consume all available pool connections.

On the other hand, threads (there are 2) which actually try to do DbIdGenerator.getNextId are pending on Db connection. E.g.:

"pool-8-thread-101" prio=10 tid=0x0000000041098800 nid=0x2189 waiting on condition [0x00007ff7e4d2e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  <0x00000007461fd8e8> (a java.util.concurrent.Semaphore$FairSync)
   at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
   at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
   at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
   at org.jboss.jca.core.connectionmanager.pool.mcp.Semaphore.tryAcquire(Semaphore.java:59)
   at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:253)
   at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:495)
   at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:374)
   at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:364)
   at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
   at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
   at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
   at org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:84)
   at org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:58)
   at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)
   at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)
   at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)
   at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
   at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)
   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:98)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:62)
   at org.activiti.engine.impl.db.DbSqlSession.selectById(DbSqlSession.java:308)
   at org.activiti.engine.impl.persistence.entity.PropertyEntityManager.findPropertyById(PropertyEntityManager.java:25)
   at org.activiti.engine.impl.cmd.GetNextIdBlockCmd.execute(GetNextIdBlockCmd.java:34)
   at org.activiti.engine.impl.cmd.GetNextIdBlockCmd.execute(GetNextIdBlockCmd.java:24)
   at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
   at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
   at org.activiti.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:64)
   at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:37)
   at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
   at org.activiti.engine.impl.db.DbIdGenerator.getNewBlock(DbIdGenerator.java:43)
   - locked <0x0000000749ac3cc8> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:36)
   - locked <0x0000000749ac3cc8> (a org.activiti.engine.impl.db.DbIdGenerator)
   at org.activiti.engine.impl.db.DbSqlSession.insert(DbSqlSession.java:111)
   at org.activiti.engine.impl.persistence.AbstractManager.insert(AbstractManager.java:50)
        …
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

How to handle such case? Is it a bug or a configuration issue?

[trademak]

Yes the DbIdGenerator needs a connection to the database to retrieve the next block of ids. So if there's no connection available anymore it gets blocked. You could use the StrongUuidGenerator instead or increase the number of connections in your pool.