How to handle a race condition signaling an Execution

[blezek]

Hi,

  We start long-running workflows that wait for a signal.  When certain events happen, we signal the workflow by searching for the list of Executions waiting on the signal (and some other criteria).  This can lead to race conditions because the incoming events can come in short order.  Here is example code:

    ExecutionQuery query = runtime.createExecutionQuery().processVariableValueEquals("Key", key).signalEventSubscriptionName(signalName);
    executions.forEach(ex -> {
        logger.info("Signaling workflow: " + ex.getId() + " with " + signalName);
        ...
        runtime.signalEventReceived(signalName, ex.getId(), signalVars);

The problem is that the long-running workflow performs a few checks right after the signal is received, then goes back to waiting on the signal (see image).  Because the above code can be called rapidly, between the query step and the signalEventReceived, the long-running workflow can go from waiting on the signal to being busy with other things.  I do synchronize in the Java code so that only one signal is sent at a time.  

We have tried a combination of "asynchronous" settings on the workflow after the intermediate signal catching event (the "Classify Series" and "Log Message", before it loops back to wait for the next signal).

If the events come slowly enough, the workflow returns to the signal catching event and all works well.

Question: How can I make sure the long-running workflow returns to waiting for the signal before the next signal needs to be sent?

Help?

-dan

[gdharley · Accepted solution]

Dan,

you have to change your design to decouple the signal send and receive logic.

Put the signals onto a queue and then have the receiver loop through each signal/message in it's own time. Before "stopping" at the next "signal" look to see if there is already a message on the queue and process it immediately.

What you describe is a very typical issue when the processing time for signals can exceed the interval between received signals.

Greg

‌

[blezek]

Greg,  Yup, that's what I figured you'd say...  😃  We will document that signal are delivered "best effort" and that workflows should not depend on them being delivered.  The notification logic adds the new information into a workflow variable and continues.  That logic also locks to have a single thread, so we don't clobber the workflow variables.

Thanks much,

-dan