Newer versions don't reconnect if database fails

[arubiom00_]

After a scheduled database failover we've found that Nuxeo 6.0 has problems reconnecting to the database.

We've reproduced the problem using the downloadable Nuxeo 6.0 virtual machine. The behaviour is the same using 7.1.
5.9.5 reconnects really quick and user does not notice anything.

Steps to reproduce the problem:
1 - Start the VM
2 - Login as root -> /etc/init.d/postgresql restart
3 - Login into nuxeo and try to browse the platform. Errors will appear. Platform will be unusable until Nuxeo is restarted.

Is there anything that should be done to allow reconnects or is it just a bug?

Server.log display tons of errors like this:

2015-01-21 20:09:05,168 WARN  [http-bio-0.0.0.0-8080-exec-64] [org.jboss.seam.jsf.SeamPhaseListener] uncaught exception, passing to exception handler
java.lang.IllegalStateException: Could not commit transaction
    at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:629)
    at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:608)
    at org.jboss.seam.jsf.SeamPhaseListener.handleTransactionsAfterPhase(SeamPhaseListener.java:344)
    at org.jboss.seam.jsf.SeamPhaseListener.afterServletPhase(SeamPhaseListener.java:244)
    at org.jboss.seam.jsf.SeamPhaseListener.afterPhase(SeamPhaseListener.java:195)
    at com.sun.faces.lifecycle.Phase.handleAfterPhase(Phase.java:189)
    at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:107)
    at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
    at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.apidoc.filter.BaseApiDocFilter.doFilter(BaseApiDocFilter.java:57)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
    at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:30)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.wss.servlet.BaseWSSFilter.doFilter(BaseWSSFilter.java:131)
    at org.nuxeo.wss.servlet.FailSafeWSSFilter.doFilter(FailSafeWSSFilter.java:55)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:67)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:136)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:28)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:585)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:68)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:119)
    at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
    at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:398)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:60)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:139)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.apidoc.filter.BaseApiDocFilter.doFilter(BaseApiDocFilter.java:57)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:73)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
Caused by: javax.transaction.RollbackException: Unable to commit: Transaction timeout
    at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
    at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
    at org.nuxeo.runtime.jtajca.NuxeoContainer$UserTransactionImpl.commit(NuxeoContainer.java:573)
    at org.jboss.seam.transaction.UTTransaction.commit(UTTransaction.java:52)
    at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:617)
    ... 80 more
2015-01-21 20:09:05,200 ERROR [http-bio-0.0.0.0-8080-exec-64] [org.jboss.seam.jsf.SeamPhaseListener] swallowing exception
java.lang.IllegalStateException: Could not commit transaction
    at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:629)
    at org.jboss.seam.jsf.SeamPhaseListener.commitOrRollback(SeamPhaseListener.java:608)
    at org.jboss.seam.jsf.SeamPhaseListener.handleTransactionsAfterPhase(SeamPhaseListener.java:344)
    at org.jboss.seam.jsf.SeamPhaseListener.afterServletPhase(SeamPhaseLis

Thank you in advanced

[Florent_Guillau]

If the connection to your database is not reliable, you have to define explicitly a "validation query" that will be used to check for proper connectivity on connection use.

nuxeo.db.validationQuery=SELECT 1

This is currently not present in the default configuration has it has a (small) performance impact and database connections should be reliable anyway.

[arubiom00_]

Adding the parameter solved the issue in the VM.