topic Re: Java Service Task and HttpClient in Alfresco Archive

Java Service Task and HttpClient

martinfranz — Sun, 02 Nov 2014 13:27:12 GMT

Hi all,I try to write a custom java service task which uses one http client to make one call per service task to an external custom api. PoolingHttpClientConnectionManager > CloseableHTTPClient > Java Service task (async) > 100 * HttpPost / HttpGetThe problem with this is: When I execute mo

Re: Java Service Task and HttpClient

martinfranz — Sun, 02 Nov 2014 22:09:05 GMT

Update: I almost solved the problem.

I forgot to mention that I run actviti-rest and activiti-explorer next together in one tomcat instance. Both use the same mysql "activiti" schema.

I double checked the processEngineConfiguration in both application context files and realized that my activate-rest uses
jobExecutorActivate = false. So I set this to true in both context files.

What happens now:

1. Start 100 process instances of a simple process containing a timer intermediate event and my service task via REST API.
2. Login to activiti-exlporer as admin and can see 100 timer jobs firing in x minutes.
3. ACT_RU_JOB from mysql also shows me these 100 jobs.
4. Than the timer event occurs and i can see in ACT_RU_JOB that the jobs are done.
5. After 3-4 minutes all jobs are done and ACT_RU_JOB is empty.
6. Check the network traffic on the target API side and realized that all 100 calls where made.
7. Everything OK
8. Try to login into activate-explorer again. Freeze! Communication failure.

Nothing happens here. Activiti-explorer is out of reach.
But activit-rest works as aspected. (same time same machine) I can serve task or process definitions via rest api without any problem.

Restart activiti-explorer via tomcat manager does not work.

Shutdown tomcat. Killall java. Startup tomcat. Works! Activit-explorer is now running again.

JAVA_OPTS
-Xms1024m -Xmx2048m -XXermSize=32m -XX:MaxPermSize=512m -Xss2m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Djava.awt.headless=true

Re: Java Service Task and HttpClient

jbarrez — Mon, 03 Nov 2014 10:59:03 GMT

Hmmm you are not the first one reporting this problem. It seems there is a fundamental problem with the job executor + Activiti Explorer in the latest releases. But so far, we haven't found the cause yet.

Thanks for digging into this, much appreciated. So your analysis points in the direction of the threadpool on the Tomcat rather than the database connection pool?

Re: Java Service Task and HttpClient

martinfranz — Mon, 03 Nov 2014 13:37:02 GMT

Could it be a problem that both the activiti-explorer as well as the activiti-rest app running on one tomcat instance?

I mean - with this setup you have two running activiti engines pointing to the same database. Pooling Jobs from one table? Could this work without any problems?

In tomcat you can specify an executor with a max thread size.
<code>
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="150" minSpareThreads="4"/>
</code>

I made another test:

1. Stop activiti-explorer
2. Start 400 process instances via activiti-rest. (in a for loop, whit a delay of 1 second per iteration)
The process definition here is quit simple:
StartEvent > Timer Caching Event > Java Service Task > User Task > End Event

3. ACT_RU_JOB now shows 400 entries (400 timers - they all should fire at the same time)
4. Timer event fires and the number of entries in ACT_RU_JOB slowly begins to decrease.
5. After 12 Minutes all Jobs are done.
6. Startup activiti-explorer via tomcat manager.
7. Everything works as aspected.

The problem with this test for me is that the execution takes to long. It feels like all process instances run in sequence - not parallel.
I tried async true and false on the java service task. Same result.
Normally there should be at least 100 http client executions at the same time. But here there is just one by one. … But this could also be a problem within the connection pool ore with my java service task implementation themselves.

I make some more tests an keep you up to date.

Best regards from Nuremberg

Re: Java Service Task and HttpClient

jbarrez — Wed, 05 Nov 2014 13:12:47 GMT

Thanks, good and well documented explanation.

Just for validation: if you do this on Activiti 5.15, does it happen also? I have a hunch it might be due some 'logic' that was added there.

However, the async true/false that you tried worries me.

Could you hook up jvisualvm to the instance and see if it has indeed 100 threads at least blocked?

Re: Java Service Task and HttpClient

martinfranz — Wed, 05 Nov 2014 21:05:58 GMT

So first of all - sorry for mention that so late - i am currently using Version 5.15.1.

And the second thing: After a bit more digging into this problem I would say that activiti does his job well. My problem here was how I used the HttpClient and also the misunderstanding how Java Service Tasks are executed.

Just imagine the following scenario:

You have a Java Service Task creating and executing a http client directly in the execute method. That could not work. Even the apache docs for http client say that you should never use multiple instances of http clients.
> What happens here: The http client execution blocks the thread which executes the Java Service Task.
> You want recognize this as long as your Service Task comes after a User Task. In this case it is definitely not possible that multiple user execute their task absolutely synchron.
> You definitely will recognize this if your Service Task comes a the beginning of a process or after a timer event. In this case it could happens that the Service Task is executed a 100 times at the same time.

I ended up with the following setup:

1. Create a global connection pool for all your http client executions.
2. Limit the max client connections! (Less is more)
3. Create a global instance of a http client using the connection pool as well as other strategy implementations. (keep alive, connection reuse)
4. In your Java Service Task create the desired http method with headers etc as well as a new thread which executes your method with your single instance http client.

I have tested this scenario with 400 concurrent executions - means 400 process instances starting at the same time.
> Start Event > Timer Event > Java Service Task > User Task.
Results:
1. After the 400 timers fired activiti needed just 2 seconds the generate the 400 User Task. (No blocking anymore)
2. The connection pool with max 20 connections needed about 2 minutes for executing all requests.

I feel save now using the http client within a service task 🙂

There is just one more thing i have not tested jet:
I have started this post because my initial problem was the usage of acitvit rest and activiti explorer in one tomcat instance with on db. I don't know much about the core functionality of the activati engine but could it be possible that two job executors pulling jobs form the same table
won't work?

Best regards.

Re: Java Service Task and HttpClient

jbarrez — Mon, 10 Nov 2014 10:44:35 GMT

Ok, thanks for posting this all back. Very good information for many people!

Two job executors normally have no problems pulling from the same database (it was designed for that).