Cannot log in after the engine ran for a few days.

[ssun]

After our engine ran for a few days, we accumulate about 8000 running process instances. Most of them have timers running on them.
Our program mostly interacts with the engine through REST API, which is working as usual.
But the problem is as an admin, I can no longer login through the explorer UI.
I can open the login page with no problem, but after type in username and password and press login, it will stuck there forever.
There is nothing from the apache log.
If I clone our production database and put it on my local testing environment, I can re-produce the problem.
If I start to close those running processes, after I closed certain amount of them, the problem goes away. I do not have to close everything, just some of therm, I do not think which particular ones that I close or not matters.
This is very strange. I wish someone who is more familiar with the code can shed some light.
We rely on the explorer to deploy new workflow definitions. We can use the REST API but we are not there yet.

Any help is greatly appreciated!

–Gordon

[ssun]

And the apache process will hang very very bad. It cannot be stopped properly using the apache bin folder command. I have to kill -9 to kill the server.

[ssun]

I see "ghost" process instances. I.E, they are not finished (no end time), but I cannot find any task for them.
This could be the reason. I will try to delete these instances see if it fix anything.

———-
Ignore my comment above, they simply have call activity called sub process and waiting for sub process to finish.

[ssun]

In my ACT_RU_JOB table, I see some jobs got locked, could this be the cause? I have very simple timers, why do they ever need lock?

ID_ REV_ TYPE_ LOCK_EXP_TIME_ LOCK_OWNER_ EXCLUSIVE_ EXECUTION_ID_ PROCESS_INSTANCE_ID_ PROC_DEF_ID_ RETRIES_ EXCEPTION_STACK_ID_ EXCEPTION_MSG_ DUEDATE_ REPEAT_ HANDLER_TYPE_ HANDLER_CFG_
268616 4 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 268615 268606 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:34 NULL timer-transition boundarytimer1
273002 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273001 272992 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273041 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273040 273031 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273170 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273169 273160 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273265 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273264 273244 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273293 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273292 273283 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273321 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273320 273311 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273338 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273337 273328 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273377 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273376 273367 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273405 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273404 273384 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273433 5 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273432 273412 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273461 3 timer 2014-06-11 18:33:34 dcecabe8-a17c-4b81-b3bf-2d5c56050ebb 1 273460 273451 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273489 2 timer 2014-06-10 12:04:30 ba9d0eba-54f0-448f-a7cb-921b844c1b67 1 273488 273479 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273517 1 timer NULL NULL 1 273516 273507 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273534 1 timer NULL NULL 1 273533 273524 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273573 1 timer NULL NULL 1 273572 273563 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273590 1 timer NULL NULL 1 273589 273580 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
273618 1 timer NULL NULL 1 273617 273608 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 11:59:30 NULL timer-transition boundarytimer1
278415 1 timer NULL NULL 1 278414 278405 createInboundShipment:2:268322 3 NULL NULL 2014-06-10 23:59:39 NULL timer-transition boundarytimer1

[ssun]

I manually deleted the timer locks from database. That does not help at all. I still cannot login. It seems I have to close tasks

[ssun]

Something is fundamentally wrong from the beginning. If I start with just activiti-rest , I can stop apache with the apache shell script.
If I start with activiti-explorer , (without try to access anything) immediately I cannot stop apache using the shell script. I have to use kill -9 to kill the process.

[ssun]

I just verified again that if I simply use the REST api to close all the tasks and then I can log in (I have to kill the apache process and restart the server or I still have the same problem).

This is very annoying! Please help!

[ssun]

https://github.com/Activiti/Activiti/blob/master/modules/activiti-explorer/src/main/java/org/activit...

I added some print statement and I never got to the second print. I believe the problem is somehow inside identityService.checkPassword

     System.out.println("ssun: authenticate 1");
     if (identityService.checkPassword(userName, password)) {
      System.out.println("ssun: authenticate 2");

This is really puzzling. How can a lot of open tasks impact password check?

[ssun]

More debugging lead me through activiti-engine to activiti-spring and this is the line causing the problem.

https://github.com/Activiti/Activiti/blob/master/modules/activiti-spring/src/main/java/org/activiti/...

When I'm using the bad database data, a lot of things stuck here, simply stuck. And I am not sure how to dig into it because it is spring code from there.

[raimana]

I know that doesn't solve your problem but have you considered using a directory server instead of a DB for your identity registry? I suppose that won't happen with LDAP…

I'd suggest to add some info regarding your DB, JDBC driver etc that might help putting this into perspective.