NullPointerException causes jobs to stop executing and ui to become unresponsive

[exp2345]

I have the 15.1 "activiti-explorer.war" deployed in Tomcat 7 with the properties to enable the demo properties set to false and a MSSQL db configured.

I have a process that starts a number (testing with 100) processes using a multiple instance, sequential Java service task.

ProcessDefinition processDefinition = repositoryService
            .createProcessDefinitionQuery()
            .processDefinitionKey("invoice")
            .singleResult();
      
      Map<String, String> formProperties = new HashMap<String, String>();
      formProperties.put("key", value);

      ProcessInstance processInstance = formService
            .submitStartFormData(processDefinition.getId(), formProperties);
‍‍‍‍‍‍‍‍‍‍‍‍

The "invoice" process is a series of async script tasks that sleep to simulate external api calls.

If I start this process and do nothing else with Explorer the 100 processes complete normally.

If I try to check on the status of the processes in "My instances" I will eventually get a NullPointerException (sometimes happens immediately upon opening My instances, sometimes I need to open a few instances before it happens. I have seen two different exceptions (see below), and they both seem related to the processInstance being null. The NPE isn't surprising as some of the instances will have completed and been removed from RU while the page loads.

I'm not sure where to start debugging the two problems of:
1) Activiti Explorer stops handling requests after the NPE. It will display the login page but never sends a response back after a login
2) The job executor stops running all jobs after the NPE (logging output from my tasks stop, db tables don't change)

Restarting Tomcat brings Activiti up into the same unresponsive state, and the only way I've found to bring it back is clearing out the RU and HI tables.

Does this have something to do with the configuration of the demo?

com.vaadin.event.ListenerMethod$MethodException: Invocation of method valueChange in org.activiti.explorer.ui.process.ProcessInstancePage$1 failed.
        at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:530)
        at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:164)
        at com.vaadin.ui.AbstractComponent.fireEvent(AbstractComponent.java:1219)
…
Caused by: java.lang.NullPointerException
        at org.activiti.explorer.ui.management.processinstance.ProcessInstanceDetailPanel.<init>(ProcessInstanceDetailPanel.java:107)
        at org.activiti.explorer.ui.process.ProcessInstancePage$1.valueChange(ProcessInstancePage.java:60)
        at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:510)
        … 33 more
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

java.lang.NullPointerException
        at org.activiti.rest.diagram.services.ProcessInstanceHighlightsResource.getHighlighted(ProcessInstanceHighlightsResource.java:53)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
‍‍‍‍‍

[frederikherema1]

First of all, Activiti Explorer is a demo-app, used to demonstrate what our API can do. The settings are all default (eg. number of threads used by the job-executor) and the UI is not waterproof.

If you have this requirement (highly concurrent processes, job-heavy), you need to tweak the job-executor to be able to cope with this kind of load, to get any kind of sensible/viable test results. Sometimes, the UI does multiple calls to the API to create a single screen, in different transactions. So NPE's are not surprising with these kind of processes. Again, it's a demo-UI and in actually production apps, there a more checks for this.

That this causes the job executor to freeze is really strange. After rebooting still no activity is even stranger. How long do the sleeps take in the fake-service-calls? If it exceeds 5 minutes (or exceeds the database/connection/transaction timeout), the job will be considered as error in execution, and will be retried. After 3 retries (by default) the will NOT be retried.

My advice:

- Tweak job-executor pool settings to reasonable settings
- Check database-timeout, sleep-times
- Use a decent DB (not H2) to prevent too many table-locks
- Double-check the ACT_RU_JOB table for jobs that have 0 retire lefts, when you think the system is "stuck"

[exp2345]

Thanks for your prompt response and apologies for my delayed follow-up. I wanted to post my resolution in case anyone else comes across this problem.

I am using SQL Server so I doubt table-locks were the problem. Increasing the number of executors helped but didn't eliminate the issue. None of the jobs had 0 retries.

Is the correct pattern to follow here is Service Task -> Receive Task where the service tasks starts another thread which signals the receive tasks? This seems to have alleviated any problems with deadlock and the UI being unresponsive, but raises the additional requirement of needing another framework to manage these threads.

I have followed up with a question on the Service -> Receive pattern here: http://forums.activiti.org/content/correct-way-signal-receivetask

[jbarrez]

No, such a pattern should not be needed.

What  could happen is that the threads of the job executor all are lost due to the exception maybe. Another rejection strategy might be useful in that case.

Does it also happen if you completely close down the browser? Or with another browser?

However, becoming unresponsive as a whole app sounds not good! And I have no real idea how it could be caused.

[exp2345]

The observed behaviour is that the UI displays the waiting spinner which goes to red before loading the page after quite some time. If I also watch the logs I can see 1..numJobExecutors tasks complete before the UI updates. The behaviour is the same when closing the browser or logging in from a different browser.

I'll put together a simple example from some older code soon.

I haven't been able to reproduce the full deadlock scenario in quite some time now. We're running the server on a larger instance now so that might have been a contributing factor.

[exp2345]

So this seems to happen whenever the number of pending jobs exceeds the number of executors. One workaround I've considered is partitioning the jobs between two engines: one for these long-running jobs and another for everything else.

This process is just a series of script tasks with 10s sleeps to simulate long-running service tasks.

<?xml version='1.0' encoding='UTF-8'?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlnsmgdc="http://www.omg.org/spec/DD/20100524/DC" xmlnsmgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/processdef">
  <process id="sleepSeries" name="Sleep Series" isExecutable="true">
    <startEvent id="startEvent"/>
    <scriptTask id="sleep1" name="Sleep 1" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep2" name="Sleep 2" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep3" name="Sleep 3" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <scriptTask id="sleep4" name="Sleep 4" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
      <script>Thread.sleep(10000)</script>
    </scriptTask>
    <endEvent id="sid-0BD85755-4505-4802-BE88-0F2BE37B0074"/>
    <sequenceFlow id="sid-A245CE56-5A98-41F4-9CBF-E5CDD9C4CA70" sourceRef="startEvent" targetRef="sleep1"/>
    <sequenceFlow id="sid-8D30CB86-6C2A-4E40-BC11-CDFF536A8AC4" sourceRef="sleep1" targetRef="sleep2"/>
    <sequenceFlow id="sid-35BD5D2C-1FE2-4B49-AA2E-DFD3A82F71D7" sourceRef="sleep2" targetRef="sleep3"/>
    <sequenceFlow id="sid-276DE07B-B7F4-46E4-BFFB-2A02DC2D069F" sourceRef="sleep3" targetRef="sleep4"/>
    <sequenceFlow id="sid-2ABFDEB2-CB75-4846-A00A-49586EA0CE2F" sourceRef="sleep4" targetRef="sid-0BD85755-4505-4802-BE88-0F2BE37B0074"/>
  </process>
</definitions>

This process can launch a large number of instances of the process above.

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:activiti="http://activiti.org/bpmn" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlnsmgdc="http://www.omg.org/spec/DD/20100524/DC" xmlnsmgdi="http://www.omg.org/spec/DD/20100524/DI" typeLanguage="http://www.w3.org/2001/XMLSchema" expressionLanguage="http://www.w3.org/1999/XPath" targetNamespace="http://www.activiti.org/test">
    <process id="exampleProcessLauncher" name="Example Process Launcher" isExecutable="true">
        <startEvent id="startEvent" name="Start" activiti:initiator="initiator">
            <extensionElements>
                <activiti:formProperty id="cardinality" name="Cardinality" type="long" required="true"></activiti:formProperty>
            </extensionElements>
        </startEvent>
        <sequenceFlow id="flow1" sourceRef="startEvent" targetRef="parseInputs"></sequenceFlow>
        <scriptTask id="parseInputs" name="Parse Policy IDs" activiti:async="true" activiti:exclusive="false" scriptFormat="groovy" activiti:autoStoreVariables="false">
            <script>
<![CDATA[
// this step normally parses user inputs
]]>
            </script>
        </scriptTask>
        <sequenceFlow id="flow2" sourceRef="parseInputs" targetRef="launchProcesses"></sequenceFlow>
        <serviceTask id="launchProcesses" name="Service Task" activiti:async="true" activiti:exclusive="false" activiti:class="examples.ExampleProcessLauncher">
            <multiInstanceLoopCharacteristics isSequential="false">
                <loopCardinality>${cardinality}</loopCardinality>
            </multiInstanceLoopCharacteristics>
        </serviceTask>
        <sequenceFlow id="flow3" sourceRef="launchProcesses" targetRef="endevent1"></sequenceFlow>
        <endEvent id="endevent1" name="End"></endEvent>
    </process>
</definitions>

And the service task called by the process above:

package examples;

import org.activiti.engine.FormService;
import org.activiti.engine.IdentityService;
import org.activiti.engine.RepositoryService;
import org.activiti.engine.delegate.DelegateExecution;
import org.activiti.engine.delegate.JavaDelegate;
import org.activiti.engine.repository.ProcessDefinition;
import org.activiti.engine.runtime.ProcessInstance;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;
import java.util.Map;

public class ExampleProcessLauncher implements JavaDelegate {

    private static final Logger log = LoggerFactory.getLogger(ExampleProcessLauncher.class);

@Override
public void execute(DelegateExecution execution) throws Exception {
  RepositoryService repositoryService = execution.getEngineServices().getRepositoryService();
  FormService formService = execution.getEngineServices().getFormService();
  IdentityService identityService = execution.getEngineServices().getIdentityService();
 
  String initiator = (String) execution.getVariable("initiator");
  identityService.setAuthenticatedUserId(initiator);
 
  ProcessDefinition processDefinition = repositoryService
    .createProcessDefinitionQuery()
    .processDefinitionKey("sleepSeries")
                .latestVersion()
    .singleResult();
 
  Map<String, String> formProperties = new HashMap<>();
  ProcessInstance processInstance = formService
    .submitStartFormData(processDefinition.getId(), formProperties);
}

}


[ivan_postic]

Hello,

I am having a similar problem with Activiti Explorer. I deploy and run a simple process which parses messages which are ariving to input queue (RabbitMQ messaging). While the process is waiting for the message to arrive, GUI becomes unresponsive with a red waiting spinner. I am new to activiti and probably doing something wrong, any help would be appreciated.

This is a Service Task Delegate class for receiveing messages:<code>
package hr.biss.activiti.rabbitmq;

import hr.biss.rabbitmq.Queue;
import hr.biss.rabbitmq.XdrQueueInput;

import org.activiti.engine.delegate.DelegateExecution;
import org.activiti.engine.delegate.JavaDelegate;
import org.apache.log4j.Logger;

public class FetchDataDelegate implements JavaDelegate {

XdrQueueInput inputQueue = new XdrQueueInput(new Queue("testActivitiInputQueue", "amqp://localhost"));

@Override
public void execute(DelegateExecution execution) throws Exception {
  try {
  
   String message = new String(inputQueue.receive();
  
   if(message.isEmpty()) {
    execution.setVariable("messageCorrupted", true);

   } else {
    execution.setVariable("message", message);
    execution.setVariable("messageCorrupted", false);
   }
  } catch (Exception e) {
   log.error("");
  }
}
}</code>

Thx,
Ivan

[jbarrez]

Im assuming the inputQueue.receive() is blocking - ie it waits until a message is sent. Is that correct? if so, it explains why the ui locks up.

A solution could be to make that service task async. Or did you already do that?

[exp2345]

A simple unit test does not reproduce the issue, but the Activiti Explorer certainly becomes unresponsive while even a modest (10) instances of the process I pasted above are running. All of the tasks are marked async.

It is the DbIdGenerator that seems to cause the deadlock condition. I have attached a thread dump captured while the application was locked up while attempting to display the login page. 'pool-2-thread-8' on line 19 is the thread handling the login.

[exp2345]

This is likely a synchronisation issue between one or more of the commands used by Explorer and the process engine. I have disabled Explorer's embedded engine and deployed a separate standalone engine. Since making the change I have not observed any of the problems previously reported. If I find some free time I might look into to the cause to avoid having to deploy two apps, but this solution is perfectly fine for our immediate needs.