topic Re: NullPointerException causes jobs to stop executing and ui to become unresponsive in Alfresco Archive

NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Thu, 24 Jul 2014 20:29:30 GMT

I have the 15.1 "activiti-explorer.war" deployed in Tomcat 7 with the properties to enable the demo properties set to false and a MSSQL db configured.I have a process that starts a number (testing with 100) processes using a multiple instance, sequential Java service task.ProcessDefinition processDe

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

frederikherema1 — Fri, 25 Jul 2014 08:11:50 GMT

First of all, Activiti Explorer is a demo-app, used to demonstrate what our API can do. The settings are all default (eg. number of threads used by the job-executor) and the UI is not waterproof.

If you have this requirement (highly concurrent processes, job-heavy), you need to tweak the job-executor to be able to cope with this kind of load, to get any kind of sensible/viable test results. Sometimes, the UI does multiple calls to the API to create a single screen, in different transactions. So NPE's are not surprising with these kind of processes. Again, it's a demo-UI and in actually production apps, there a more checks for this.

That this causes the job executor to freeze is really strange. After rebooting still no activity is even stranger. How long do the sleeps take in the fake-service-calls? If it exceeds 5 minutes (or exceeds the database/connection/transaction timeout), the job will be considered as error in execution, and will be retried. After 3 retries (by default) the will NOT be retried.

My advice:

- Tweak job-executor pool settings to reasonable settings
- Check database-timeout, sleep-times
- Use a decent DB (not H2) to prevent too many table-locks
- Double-check the ACT_RU_JOB table for jobs that have 0 retire lefts, when you think the system is "stuck"

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Tue, 19 Aug 2014 10:53:46 GMT

Thanks for your prompt response and apologies for my delayed follow-up. I wanted to post my resolution in case anyone else comes across this problem.

I am using SQL Server so I doubt table-locks were the problem. Increasing the number of executors helped but didn't eliminate the issue. None of the jobs had 0 retries.

Is the correct pattern to follow here is Service Task -> Receive Task where the service tasks starts another thread which signals the receive tasks? This seems to have alleviated any problems with deadlock and the UI being unresponsive, but raises the additional requirement of needing another framework to manage these threads.

I have followed up with a question on the Service -> Receive pattern here: http://forums.activiti.org/content/correct-way-signal-receivetask

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

jbarrez — Wed, 20 Aug 2014 14:39:49 GMT

No, such a pattern should not be needed.

What could happen is that the threads of the job executor all are lost due to the exception maybe. Another rejection strategy might be useful in that case.

Does it also happen if you completely close down the browser? Or with another browser?

However, becoming unresponsive as a whole app sounds not good! And I have no real idea how it could be caused.

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Wed, 20 Aug 2014 15:04:53 GMT

The observed behaviour is that the UI displays the waiting spinner which goes to red before loading the page after quite some time. If I also watch the logs I can see 1..numJobExecutors tasks complete before the UI updates. The behaviour is the same when closing the browser or logging in from a different browser.

I'll put together a simple example from some older code soon.

I haven't been able to reproduce the full deadlock scenario in quite some time now. We're running the server on a larger instance now so that might have been a contributing factor.

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Thu, 21 Aug 2014 09:34:38 GMT

So this seems to happen whenever the number of pending jobs exceeds the number of executors. One workaround I've considered is partitioning the jobs between two engines: one for these long-running jobs and another for everything else.

This process is just a series of script tasks with 10s sleeps to simulate long-running service tasks.


<?xml version='1.0' encoding='UTF-8'?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:smileysurprised:mgdc="http://www.omg.org/spec/DD/20100524/DC" xmlns:smileysurprised:mgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/processdef">
  <process id="sleepSeries" name="Sleep Series" isExecutable="true">
    <startEvent id="startEvent"/>
    <scriptTask id="sleep1" name="Sleep 1" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep2" name="Sleep 2" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep3" name="Sleep 3" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep4" name="Sleep 4" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <endEvent id="sid-0BD85755-4505-4802-BE88-0F2BE37B0074"/>
    <sequenceFlow id="sid-A245CE56-5A98-41F4-9CBF-E5CDD9C4CA70" sourceRef="startEvent" targetRef="sleep1"/>
    <sequenceFlow id="sid-8D30CB86-6C2A-4E40-BC11-CDFF536A8AC4" sourceRef="sleep1" targetRef="sleep2"/>
    <sequenceFlow id="sid-35BD5D2C-1FE2-4B49-AA2E-DFD3A82F71D7" sourceRef="sleep2" targetRef="sleep3"/>
    <sequenceFlow id="sid-276DE07B-B7F4-46E4-BFFB-2A02DC2D069F" sourceRef="sleep3" targetRef="sleep4"/>
    <sequenceFlow id="sid-2ABFDEB2-CB75-4846-A00A-49586EA0CE2F" sourceRef="sleep4" targetRef="sid-0BD85755-4505-4802-BE88-0F2BE37B0074"/>
  </process>
</definitions>

This process can launch a large number of instances of the process above.


<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:smileysurprised:mgdc="http://www.omg.org/spec/DD/20100524/DC" xmlns:smileysurprised:mgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/test">
    <process id="exampleProcessLauncher" name="Example Process Launcher" isExecutable="true">
        <startEvent id="startEvent" name="Start" activiti:initiator="initiator">
            <extensionElements>
                <activiti:formProperty id="cardinality" name="Cardinality" type="long" required="true"></activiti:formProperty>
            </extensionElements>
        </startEvent>
        <sequenceFlow id="flow1" sourceRef="startEvent" targetRef="parseInputs"></sequenceFlow>
        <scriptTask id="parseInputs" name="Parse Policy IDs" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
            <script>
<![CDATA[
// this step normally parses user inputs
]]>
            </script>
        </scriptTask>
        <sequenceFlow id="flow2" sourceRef="parseInputs" targetRef="launchProcesses"></sequenceFlow>
        <serviceTask id="launchProcesses" name="Service Task" activiti:async="true" activiti:exclusive="false" activiti:class="examples.ExampleProcessLauncher">
            <multiInstanceLoopCharacteristics isSequential="false">
                <loopCardinality>${cardinality}</loopCardinality>
            </multiInstanceLoopCharacteristics>
        </serviceTask>
        <sequenceFlow id="flow3" sourceRef="launchProcesses" targetRef="endevent1"></sequenceFlow>
        <endEvent id="endevent1" name="End"></endEvent>
    </process>
</definitions>

And the service task called by the process above:


package examples;

import org.activiti.engine.FormService;
import org.activiti.engine.IdentityService;
import org.activiti.engine.RepositoryService;
import org.activiti.engine.delegate.DelegateExecution;
import org.activiti.engine.delegate.JavaDelegate;
import org.activiti.engine.repository.ProcessDefinition;
import org.activiti.engine.runtime.ProcessInstance;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;
import java.util.Map;

public class ExampleProcessLauncher implements JavaDelegate {

    private static final Logger log = LoggerFactory.getLogger(ExampleProcessLauncher.class);

 @Override
 public void execute(DelegateExecution execution) throws Exception {
  RepositoryService repositoryService = execution.getEngineServices().getRepositoryService();
  FormService formService = execution.getEngineServices().getFormService();
  IdentityService identityService = execution.getEngineServices().getIdentityService();
  
  String initiator = (String) execution.getVariable("initiator");
  identityService.setAuthenticatedUserId(initiator);
  
  ProcessDefinition processDefinition = repositoryService
    .createProcessDefinitionQuery()
    .processDefinitionKey("sleepSeries")
                .latestVersion()
    .singleResult();
  
  Map<String, String> formProperties = new HashMap<>();
  ProcessInstance processInstance = formService
    .submitStartFormData(processDefinition.getId(), formProperties);
 }

}

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

ivan_postic — Fri, 22 Aug 2014 09:59:53 GMT

Hello,

I am having a similar problem with Activiti Explorer. I deploy and run a simple process which parses messages which are ariving to input queue (RabbitMQ messaging). While the process is waiting for the message to arrive, GUI becomes unresponsive with a red waiting spinner. I am new to activiti and probably doing something wrong, any help would be appreciated.

This is a Service Task Delegate class for receiveing messages:<code>
package hr.biss.activiti.rabbitmq;

import hr.biss.rabbitmq.Queue;
import hr.biss.rabbitmq.XdrQueueInput;

import org.activiti.engine.delegate.DelegateExecution;
import org.activiti.engine.delegate.JavaDelegate;
import org.apache.log4j.Logger;

public class FetchDataDelegate implements JavaDelegate {

XdrQueueInput inputQueue = new XdrQueueInput(new Queue("testActivitiInputQueue", "amqp://localhost"));

@Override
public void execute(DelegateExecution execution) throws Exception {
try {

   String message = new String(inputQueue.receive();

   if(message.isEmpty()) {
    execution.setVariable("messageCorrupted", true);

   } else {
    execution.setVariable("message", message);
    execution.setVariable("messageCorrupted", false);
   }
} catch (Exception e) {
   log.error("");
}
}
}</code>

Thx,
Ivan

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

jbarrez — Sat, 30 Aug 2014 17:00:20 GMT

Im assuming the inputQueue.receive() is blocking - ie it waits until a message is sent. Is that correct? if so, it explains why the ui locks up.

A solution could be to make that service task async. Or did you already do that?

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Tue, 02 Sep 2014 19:59:27 GMT

A simple unit test does not reproduce the issue, but the Activiti Explorer certainly becomes unresponsive while even a modest (10) instances of the process I pasted above are running. All of the tasks are marked async.

It is the DbIdGenerator that seems to cause the deadlock condition. I have attached a thread dump captured while the application was locked up while attempting to display the login page. 'pool-2-thread-8' on line 19 is the thread handling the login.

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

exp2345 — Wed, 03 Sep 2014 23:17:02 GMT

This is likely a synchronisation issue between one or more of the commands used by Explorer and the process engine. I have disabled Explorer's embedded engine and deployed a separate standalone engine. Since making the change I have not observed any of the problems previously reported. If I find some free time I might look into to the cause to avoid having to deploy two apps, but this solution is perfectly fine for our immediate needs.

Re: NullPointerException causes jobs to stop executing and ui to become unresponsive

jbarrez — Mon, 15 Sep 2014 09:55:39 GMT

Hmm thanks for posting back - but it does worry me that it didnt work embedded. If you find the time, would indeed be good to look into it.