topic Strange behavior with multithreading in Alfresco Archive

Strange behavior with multithreading

marcinj — Wed, 13 Apr 2011 10:59:15 GMT

Hi,I am observing an undesired behavior with multithreaded use of Activiti.My program starts a number of threads, each of them starts a new business process. My business process is comprised of 4 Java Services, each of them just printing a message and sleeping for some time. (Service1, Service2a and

Re: Strange behavior with multithreading

ronald_van_kuij — Wed, 13 Apr 2011 17:49:51 GMT

If there are no "waitstates" in the process, the connection is indeed only returned after the process finished, so if the number of connections is limited, the behaviour you see is to be expected.

Re: Strange behavior with multithreading

marcinj — Thu, 14 Apr 2011 05:27:30 GMT

Hi,

maybe my previous post was not clear. This is not about the number of connections in the pool. Activiti (or Ibatis ?) seem to request the resources (db connections) in the following fashion:

(let's say there are 2 connections available)


Thread A                                       Thread B
getConnection -> OK!
                                                getConnection -> OK!
getConnection -> wait…
                                                getConnection -> wait…
…now both threads are just stuck forever…

And that's a classic deadlock example.

In my case, it is unclear for me what is the point of the second request ? Avoiding it would prevent the deadlock.

Thanks,
Marcin Jaskolski

Re: Strange behavior with multithreading

ronald_van_kuij — Thu, 14 Apr 2011 07:15:52 GMT

your original post was indeed not clear (enough) and your second post makes it a little more clear. 🙂

Are you sure it is activiti requesting the second connection? In you log I do not see a second connection being retrieved before one s closed in the same thread.

Still then, the pool size has to be tuned for what fuynctionality wise is needed, even if activiti requests a second connection in a thread.

Re: Strange behavior with multithreading

jbarrez — Thu, 14 Apr 2011 08:12:49 GMT

What if you change your pool with a datasource pooling solutiuon such as DBCP? The iBatis default connection pool is fairly simple implemented, last time I went through the code.

Re: Strange behavior with multithreading

frederikherema1 — Thu, 14 Apr 2011 08:20:19 GMT

The getConnection() on mybatis datasource always returns the same connection when transaction is active. The same goos for out spring-implementation (which used TransactionAwareDataSourceProxy), as long as spring-transaction is active the getConnection() returns the same.

What datasource is your hackedDatasource actually extending?

Re: Strange behavior with multithreading

marcinj — Thu, 14 Apr 2011 08:54:31 GMT

Hi Ronald,

thanks for your reply 🙂

Please have a closer look at Thread-13. It asks for the second connection.

It is perfectly agreeable that performance-wise the pool size has to be tuned. But still, if the program blocks, it is a highly problematic situation.
(As a rule of thumb, process should not ask for an additional resource if it's already holding a resource of the same type. :geek: )

I have run a few more tests and I have a more detailed information now.

One of my threads enters the synchronized method: org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:34), and the call proceeds to the point where it asks for additional connection:

"Thread-38" prio=6 tid=0x0000000006f14800 nid=0x10fc in Object.wait() [0x000000000a24c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ee416210> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at java.lang.Object.wait(Object.java:485)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1115)
- locked <0x00000000ee416210> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
…..
at org.activiti.engine.impl.db.DbIdGenerator.getNewBlock(DbIdGenerator.java:41)
- locked <0x00000000ee4112b0> (a org.activiti.engine.impl.db.DbIdGenerator)
at org.activiti.engine.impl.db.DbIdGenerator.getNextId(DbIdGenerator.java:34)
- locked <0x00000000ee4112b0> (a org.activiti.engine.impl.db.DbIdGenerator)
at org.activiti.engine.impl.history.handler.ActivityInstanceStartHandler.notify(ActivityInstanceStartHandler.java:39)
…..

and waits for free resource (sorry, no free resources now, you have to wait)

In the meantime, other threads try to enter the getNextId method (but sorry, it's taken by the previous thread).

I believe this is a real problem. I cannot assume that my database pool is unlimited. :lol:

If it's of any use, I am more than willing to provide source code, thread dumps, etc (too long to post).

Thanks,
Marcin Jaskolski

Re: Strange behavior with multithreading

frederikherema1 — Thu, 14 Apr 2011 09:07:08 GMT

Now I see… The nextId block stuff indeed gets a new connection (to have seperate transaction, to be sure id's aren't taken by others). So basically the current thread can't reuse the same connection for this (so it's actually holding a resource of another type).

This is indeed an issue that was overlooked.

Re: Strange behavior with multithreading

marcinj — Thu, 14 Apr 2011 09:09:12 GMT

What if you change your pool with a datasource pooling solutiuon such as DBCP? The iBatis default connection pool is fairly simple implemented, last time I went through the code.

Hi jbarrez,

thanks for your reply.
I am using the Apache Commons DBCP, mainly to monitor who's asking for connections, closing them, etc. It's configured this way:


   myDataSource = new org.apache.commons.dbcp.BasicDataSource();
   myDataSource.setDriverClassName("org.postgresql.Driver");
   myDataSource.setUrl("jdbc:smileytongue:ostgresql://127.0.0.1:54322/activiti");
   myDataSource.setUsername("activiti");
   myDataSource.setPassword("activiti");
   myDataSource.setMaxActive(maxConnectionNumber);
   myDataSource.setMinIdle(0);
   System.out.println("max Active: " + myDataSource.getMaxActive());

   DataSource ds = new HackedDataSource(myDataSource);
   ProcessEngineConfiguration processEngineConfiguration = ProcessEngineConfiguration
     .createStandaloneProcessEngineConfiguration()
     .setDataSource(ds);

   processEngine = processEngineConfiguration.buildProcessEngine();
   System.out.println("processEngine created OK!");

The classes HackedConnection and HackedDataSource implement java.sql.Connection and javax.sql.DataSource respectively. :


public class HackedConnection implements Connection {

 private Connection conn;

 public HackedConnection(Connection conn) {
  this.conn = conn;
 }

 public void close() throws SQLException {
  String name = Thread.currentThread().getName();
  System.out.println("  –Connection.close called for thread " + name
    + ".");
  HackerDatabase.customerLeaves(name);
  conn.close();
  System.out.println("  –Connection.close finishes for thread " + name
    + ".");
 }
..delegate methods…


public class HackedDataSource implements DataSource {

 private DataSource dataSource;
 private TreeSet<String> customers;

 public HackedDataSource(DataSource dataSource) {
  this.dataSource = dataSource;
  this.customers = new TreeSet<String>();
 }

 public Connection getConnection() throws SQLException {
  String name = Thread.currentThread().getName();
  System.out.println("  –getConnection called for thread " + name + ".");
  synchronized (customers) {
   if (!"main".equals(name)) {
    if (customers.contains(name))
     System.out.println("  —-hey!");
    customers.add(name);
   }
  }

  HackerDatabase.customerAsking(name);
  Connection conn = dataSource.getConnection();
  HackerDatabase.customerServed(name);

  System.out.println("   –getConnection finishes for thread " + name
    + ".");
  return new HackedConnection(conn);
 }
…delegate methods…

HackerDatabase maintains sets of threads waiting for the connection and those holding a connection.

Thanks,
Marcin Jaskolski

Re: Strange behavior with multithreading

frederikherema1 — Thu, 14 Apr 2011 09:19:47 GMT

To summarize:
If no free connections in connection-pool are available, and a new block of ID's is requested, an SQL-connection is requested -> thread has to wait for available SQLConnections. Since this fetching is done in a syncronized block, all other theads that use the ID-generator are blocked. Since they are blocked, they can't return the SQL-connection they use to the pool -> locked untill the pools maxWaitTime expires and the waiting thread gets an exception on the getConnection().

Re: Strange behavior with multithreading

marcinj — Thu, 14 Apr 2011 09:38:19 GMT

To summarize:
If no free connections in connection-pool are available, and a new block of ID's is requested, an SQL-connection is requested -> thread has to wait for available SQLConnections. Since this fetching is done in a syncronized block, all other theads that use the ID-generator are blocked. Since they are blocked, they can't return the SQL-connection they use to the pool -> locked untill the pools maxWaitTime expires and the waiting thread gets an exception on the getConnection().

Hi Frederik,

happy to see we have understanding here.

But the basic problem is not about synchronized methods, (the pool has to employ some sort of internal synchronization anyway), but about the need to get the second resource of the same type (JDBC connection).

Thanks,
Marcin Jaskolski

Re: Strange behavior with multithreading

frederikherema1 — Thu, 14 Apr 2011 10:02:33 GMT

We're talking about the same thing I mean the syncronized method on getNextId() -> which does the second call for an sqlConnection. So if the pool is empty, the thread is stuck in the getNextId() method, which is syncronized. All other treads can't enter and wait as wel, not releasing SQLconnection.

Thanks!

Re: Strange behavior with multithreading

ronald_van_kuij — Thu, 14 Apr 2011 11:23:43 GMT

So you always need the max number of threads + 1 connections in your pool if I understand correctly…

Re: Strange behavior with multithreading

frederikherema1 — Thu, 14 Apr 2011 13:00:18 GMT

Ronald, thats indeed the only workaround there is right now, if you really want to be sure this problem never occurs. We're looking into it

Re: Strange behavior with multithreading

jbarrez — Fri, 15 Apr 2011 07:56:41 GMT

By the way: as described in (and much more interesting findings) http://forums.activiti.org/en/viewtopic.php?f=6&t=1523, the blocksize is configurable:

<property name="idBlockSize" value="10000" />

(didnt know that myself anymore).

Re: Strange behavior with multithreading

marcinj — Mon, 18 Apr 2011 08:06:02 GMT

The workaround seems to work for me ! Thanks a lot!

Marcin Jaskolski

Re: Strange behavior with multithreading

tombaeyens — Tue, 19 Apr 2011 15:40:05 GMT

I've added a patch to the case. Is there a volunteer to validate this patch in a real environment? We would need some proof on a highly concurrent system and we don't have this kind of setup in our QA lab.

Without such confirmation we don't know yet for sure if we are able to include it. So everyone interested in this issue, please check out the patch attached to

http://jira.codehaus.org/browse/ACT-789

Any information that validates the fix and that gives us more confidence it doesn't break anything else, is really appreciated.