topic Re: Cannot log in after the engine ran for a few days. in Alfresco Archive

Cannot log in after the engine ran for a few days.

ssun — Wed, 11 Jun 2014 18:16:41 GMT

After our engine ran for a few days, we accumulate about 8000 running process instances. Most of them have timers running on them.Our program mostly interacts with the engine through REST API, which is working as usual.But the problem is as an admin, I can no longer login through the explorer UI.I c

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 01:06:58 GMT

And the apache process will hang very very bad. It cannot be stopped properly using the apache bin folder command. I have to kill -9 to kill the server.

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 01:38:56 GMT

I see "ghost" process instances. I.E, they are not finished (no end time), but I cannot find any task for them.
This could be the reason. I will try to delete these instances see if it fix anything.

———-
Ignore my comment above, they simply have call activity called sub process and waiting for sub process to finish.

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 01:43:21 GMT

In my ACT_RU_JOB table, I see some jobs got locked, could this be the cause? I have very simple timers, why do they ever need lock?

ID_ REV_ TYPE_ LOCK_EXP_TIME_ LOCK_OWNER_ EXCLUSIVE_ EXECUTION_ID_ PROCESS_INSTANCE_ID_ PROC_DEF_ID_ RETRIES_ EXCEPTION_STACK_ID_ EXCEPTION_MSG_ DUEDATE_ REPEAT_ HANDLER_TYPE_ HANDLER_CFG_
268616 4 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 268615 268606 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:34 NULL timer-transition boundarytimer1
273002 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273001 272992 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273041 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273040 273031 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273170 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273169 273160 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273265 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273264 273244 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273293 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273292 273283 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273321 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273320 273311 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273338 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273337 273328 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273377 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273376 273367 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273405 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273404 273384 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273433 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273432 273412 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273461 3 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273460 273451 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273489 2 timer 2014-06-10 12:04:30 ba9d0eba-54f0-448f-a7cb-921b844c1b67 1 273488 273479 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273517 1 timer NULL NULL 1 273516 273507 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273534 1 timer NULL NULL 1 273533 273524 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273573 1 timer NULL NULL 1 273572 273563 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273590 1 timer NULL NULL 1 273589 273580 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273618 1 timer NULL NULL 1 273617 273608 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
278415 1 timer NULL NULL 1 278414 278405 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 23:59:39 NULL timer-transition boundarytimer1

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 01:49:58 GMT

I manually deleted the timer locks from database. That does not help at all. I still cannot login. It seems I have to close tasks

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 02:21:52 GMT

Something is fundamentally wrong from the beginning. If I start with just activiti-rest , I can stop apache with the apache shell script.
If I start with activiti-explorer , (without try to access anything) immediately I cannot stop apache using the shell script. I have to use kill -9 to kill the process.

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 03:03:50 GMT

I just verified again that if I simply use the REST api to close all the tasks and then I can log in (I have to kill the apache process and restart the server or I still have the same problem).

This is very annoying! Please help!

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 06:04:54 GMT

https://github.com/Activiti/Activiti/blob/master/modules/activiti-explorer/src/main/java/org/activiti/explorer/ui/login/DefaultLoginHandler.java#L42

I added some print statement and I never got to the second print. I believe the problem is somehow inside identityService.checkPassword

     System.out.println("ssun: authenticate 1");
     if (identityService.checkPassword(userName, password)) {
      System.out.println("ssun: authenticate 2");

This is really puzzling. How can a lot of open tasks impact password check?

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 07:14:38 GMT

More debugging lead me through activiti-engine to activiti-spring and this is the line causing the problem.

https://github.com/Activiti/Activiti/blob/master/modules/activiti-spring/src/main/java/org/activiti/spring/SpringTransactionInterceptor.java#L45

When I'm using the bad database data, a lot of things stuck here, simply stuck. And I am not sure how to dig into it because it is spring code from there.

Re: Cannot log in after the engine ran for a few days.

raimana — Thu, 12 Jun 2014 11:59:04 GMT

I know that doesn't solve your problem but have you considered using a directory server instead of a DB for your identity registry? I suppose that won't happen with LDAP…

I'd suggest to add some info regarding your DB, JDBC driver etc that might help putting this into perspective.

Re: Cannot log in after the engine ran for a few days.

ssun — Thu, 12 Jun 2014 20:41:15 GMT

I have tried to not use LDAP. It does not matter.
As long as I use the broken db, right from the start it breaks spring for some reason. See my detailed findings here:
http://forums.activiti.org/content/stuck-springtransactioninterceptor-execute