topic Re: Problem with intermediate catch event and retries in Alfresco Archive

Problem with intermediate catch event and retries

fwachs — Tue, 14 May 2013 19:02:05 GMT

I have a very simple problem and I'm not sure why it's happening at all.Maybe anyone can enlighten me somehow This is what's going on:- I'm running activiti 5.11- I have configured Activiti to run with 1 retry only and changed JobExecutor.lockTimeInMillis to 5 hours- I have a process that makes a

Re: Problem with intermediate catch event and retries

fwachs — Tue, 14 May 2013 19:03:29 GMT

This way it's more legible. Sorry for posting more than once.
<code>
{
"data":
[
{
   "ID_": "BLOB",
   "PROC_DEF_ID_": "BLOB",
   "PROC_INST_ID_": "BLOB",
   "EXECUTION_ID_": "BLOB",
   "ACT_ID_": "BLOB",
   "TASK_ID_": null,
   "CALL_PROC_INST_ID_": null,
   "ACT_NAME_": "BLOB",
   "ACT_TYPE_": "BLOB",
   "ASSIGNEE_": null,
   "START_TIME_": "2013-05-11 20:58:17",
   "END_TIME_": "2013-05-12 16:59:17",
   "DURATION_": 72060320
}
]
}
</code>

Re: Problem with intermediate catch event and retries

jbarrez — Wed, 15 May 2013 13:48:57 GMT

Could it be that the job executor simply doesn't poll quickly enough? I believe the default polling interval is 5 minutes?

Revoking retry for good : you'll need to inject your own Job Executor class which extends from the default one and change it to do no retries.

Re: Problem with intermediate catch event and retries

fwachs — Wed, 15 May 2013 14:18:06 GMT

Don't you think 72060320 milisecons aka 20hours is a bit too much for a
<code>
<intermediateCatchEvent id="timerintermediatecatchevent3" name="TimerCatchEvent">
      <timerEventDefinition>
        <timeDuration>PT1M</timeDuration>
      </timerEventDefinition>
    </intermediateCatchEvent>
</code>

How can an intervall that is defaulted to 5 minutes take 20 hours (I've got cases on my DB that duration is more than 40hours)?

Thanks for your reply, I really appreciate it.

Re: Problem with intermediate catch event and retries

jbarrez — Tue, 21 May 2013 11:59:10 GMT

I didnt check the 20 minutes, no. So that is definitely wrong. However if you changed the lockTime to 5 hours, it could explain it right? What happens if you put the time back to the default?

Re: Problem with intermediate catch event and retries

fwachs — Wed, 22 May 2013 19:28:54 GMT

How would changing the lockTime to 5 hours explain it? I'm not following , sorry.

Re: Problem with intermediate catch event and retries

frederikherema1 — Thu, 23 May 2013 07:15:10 GMT

Seems to me that an activity after the timer-event is taking a long time to complete. SInce the JobExecutor.lockTimeInMillis is set to 5 hours, after 5 hours of executing the first run of the timer-fire, the job-executor, will think the job (which is locked by the first thread) has failed and the job is unlocked. SInce the first thread didn't come back yet, the retry-count isn't decremented and the job-executor will execute the job again (in another tread).

This carries on for 4 times, until one of the jobs succeeds, resulting in 20 hours (I guess). What does the process actually do? And what is the state of the ACT_RU_JOB entry, a couple minutes after the timer is due to fire? If it's locked, the job-excutor is working on it.

Re: Problem with intermediate catch event and retries

jbarrez — Thu, 23 May 2013 07:21:15 GMT

Because if there is a failure, it gets locked for 5 hours? That would explain why you see such a huge delay.

Hence my suggestion to test it with default settings, to rule out that is due to your changes to the job executor.

Re: Problem with intermediate catch event and retries

mmaker1234 — Thu, 23 May 2013 14:46:32 GMT

Hello Federico,

In my experience I learned that changing the default values of an external system is the last resort to cope with your own problems. Please explain what was the reason to change the value for JobExecutor.lockTimeInMillis - probably there could be another solution.

I think that the reason the whole process to be retried is … far from "some". You didn't published your process design but let me remind you that "Activiti is going to advance in the process [in a single transaction], until it reaches wait states on each active path of execution" [ Activiti User Guide, Transactions and Concurrency ], i.e. on a transaction roll-back (when an error rise during the process instance execution) Activiti engine retries the activities sequence since the last persisted process state. All this means that if you want to retry only one (the most recent in your case) or a couple of activities then you need to control the persistence state of your process instance. I would suggest you to mark (some of) your activities Asynchronous - then the process state will be persisted on the start of each "asynchronous" activity and the retry (in a case of an error) will start from the last persisted activity.

Additionally, although some workflow engines are tolerant to long transactions, it is not advisable your activities to run longer than a couple of minutes. If you need some tasks to take longer consider to make them asynchronous (this is not related to the activity property) - use one activity to invoke the task (JMS is one of your friends here) and allow the process immediately to proceed to the next activity where to wait for signal that the task was completed. You can even build a complex logic with time-outs, notifications on task result (failure in particular), and (manually) managed task repetition around such an approach.

Hope this helps,
Monique

Re: Problem with intermediate catch event and retries

fwachs — Sat, 25 May 2013 13:58:09 GMT

Thank you all for your replies, I truly appreciate them.
The main problem for me is that there are some jobs that for me , shouldn't be retried even if the transaction fails to be stored due to locking exceptions. For example, sending SMSs or emails, I can't have my users getting multiple emails or sms because of an optimistic lock..If it did send the email or sms then that's all I wanted and I don't really care if the revision is ok or not.. Does this make sense to you?

Concerning the lockTime, I had a process that had a subprocess that was a for-each to send 10k of emails, and the lock time was really small and since the process couldn't finish sending the emails before that lock time, it restarted…and then it restarted again because it never got to send all the emails again, and so on until I killed the app.
How could I avoid this kind of situations?

Re: Problem with intermediate catch event and retries

mmaker1234 — Mon, 27 May 2013 08:28:13 GMT

Hello Federico,

That is what I'm talking about - you should (learn to) control the Activiti transactions in your process design.

Additionally I would not rely on a workflow engine to send 10K messages for me. I would write or extend my application to handle the mailing tasks instead, and the workflow process to notify when and what have to be sent. Then the workflow activity will need only to inform my application to do the job and to expect the application to notify back that the job is done. Additionally I would design the workflow process to control the time for and the result of the task execution and to notify me on time-out or task failure. Such an approach proved its reliability in our solutions, nevertheless this is nothing more than our approach - you can always find your own but always select your tools wisely (just because the ice is a frozen water like the snow and the ski have an edge similar to the skates doesn't mean that you can successfully skiing with skates or play hockey with ski).