Connection pool is fully used error

[ioihanguren_]

Hello,

after some time creating and editing documents in Nuxeo 7.2, I get this error:

No ManagedConnections available within configured blocking timeout ( 100 [ms] ) for pool org.apache.geronimo.connector.outbound.SinglePoolMatchAllConnectionInterceptor@19397d12

              javax.el.ELException: org.nuxeo.ecm.core.api.local.LocalException: Failed to load repository default: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
	at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:339)
	at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:348)
	at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58)
	at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
	at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
	at org.nuxeo.ecm.platform.ui.web.rest.services.URLPolicyServiceImpl.navigate(URLPolicyServiceImpl.java:528)
	at org.nuxeo.ecm.platform.ui.web.rest.RestfulPhaseListener.beforePhase(RestfulPhaseListener.java:94)
	at com.sun.faces.lifecycle.Phase.handleBeforePhase(Phase.java:228)
	at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:99)
	at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219)
	at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.wss.servlet.BaseWSSFilter.doFilter(BaseWSSFilter.java:127)
	at org.nuxeo.wss.servlet.FailSafeWSSFilter.doFilter(FailSafeWSSFilter.java:56)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
	at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:46)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:423)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
	at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:120)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
	at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:68)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
	at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:391)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:75)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:73)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)
	at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:110)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:139)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:578)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:36)
	at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:120)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
	at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:68)
	at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:34)
	at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:391)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:168)
	at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233)
	at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:47)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:75)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:73)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.nuxeo.ecm.core.api.local.LocalException: Failed to load repository default: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
	at org.nuxeo.ecm.core.repository.RepositoryService.getSession(RepositoryService.java:228)
	at org.nuxeo.ecm.core.api.local.LocalSession.createSession(LocalSession.java:117)
	at org.nuxeo.ecm.core.api.local.LocalSession.getSession(LocalSession.java:107)
	at org.nuxeo.ecm.core.api.AbstractSession.resolveReference(AbstractSession.java:344)
	at org.nuxeo.ecm.core.api.AbstractSession.getDocument(AbstractSession.java:953)
	at org.nuxeo.ecm.webapp.context.NavigationContextBean.navigateToRef(NavigationContextBean.java:640)
	at org.nuxeo.ecm.webapp.context.NavigationContextBean.navigateTo(NavigationContextBean.java:727)
	at sun.reflect.GeneratedMethodAccessor1275.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
	at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
	at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:78)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.ConversationInterceptor.aroundInvoke(ConversationInterceptor.java:56)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
	at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
	at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
	at org.nuxeo.ecm.webapp.context.NavigationContextBean_$$_javassist_seam_13.navigateTo(NavigationContextBean_$$_javassist_seam_13.java)
	at org.nuxeo.ecm.platform.ui.web.rest.RestHelper.initContextFromRestRequest(RestHelper.java:107)
	at sun.reflect.GeneratedMethodAccessor1274.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
	at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
	at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:78)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.ConversationInterceptor.aroundInvoke(ConversationInterceptor.java:56)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
	at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
	at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
	at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
	at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
	at org.nuxeo.ecm.platform.ui.web.rest.RestHelper_$$_javassist_seam_12.initContextFromRestRequest(RestHelper_$$_javassist_seam_12.java)
	at sun.reflect.GeneratedMethodAccessor2295.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335)
	... 94 more
Caused by: org.nuxeo.ecm.core.api.DocumentException: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
	at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getSession(ConnectionFactoryImpl.java:168)
	at org.nuxeo.ecm.core.repository.RepositoryService.getSession(RepositoryService.java:226)
	... 145 more
Caused by: org.nuxeo.ecm.core.storage.StorageException: Connection pool is fully used, consider increasing nuxeo.vcs.blocking-timeout-millis (currently 100) or nuxeo.vcs.max-pool-size (currently 20)
	at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:114)
	at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getSession(ConnectionFactoryImpl.java:166)
	... 146 more
Caused by: javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 100 [ms] ) for pool org.apache.geronimo.connector.outbound.SinglePoolMatchAllConnectionInterceptor@19397d12
	at org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:86)
	at org.apache.geronimo.connector.outbound.ThreadLocalCachingConnectionInterceptor.getConnection(ThreadLocalCachingConnectionInterceptor.java:70)
	at org.apache.geronimo.connector.outbound.TransactionEnlistingInterceptor.getConnection(TransactionEnlistingInterceptor.java:49)
	at org.apache.geronimo.connector.outbound.TransactionCachingInterceptor.getConnection(TransactionCachingInterceptor.java:101)
	at org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:43)
	at org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39)
	at org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66)
	at org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:77)
	at org.nuxeo.runtime.jtajca.NuxeoContainer$ConnectionManagerWrapper.allocateConnection(NuxeoContainer.java:838)
	at org.nuxeo.ecm.core.storage.sql.ra.ConnectionFactoryImpl.getConnection(ConnectionFactoryImpl.java:100)
	... 147 more

            

I tried increasing nuxeo.vcs.blocking-timeout-millis and nuxeo.vcs.max-pool-size, but after a while, I get the same error.

I've tried searching how can I fix this and why it happens, but with no results. Can someone explain me why it happens and how can I solve the problem?

[Pradeep_Bhat]

I am facing the same issue. Did anybody encountered and got solution?

[Florent_Guillau]

Nuxeo 7.2 is obsolete and unsupported. Please use Nuxeo LTS 2015 (7.10).

[fast_narcis_]

Using the LTS, doesn't fix the problem.

[René_]

I'm having the same problem with Nuxeo 7.10. As said above the respective parameters show no effect

[Florent_Guillau]

It means that there's a lot of work going on in the server and that all its threads are busy. You should use jstack to find out what's going on.

[René_]

I'm quite sure that 2 concurrent users of which one is browsing and the other is uploading a dozen files is not considered "a lot of work going on". It's a test server and therefore I can assure that there is no heavy traffic. Still the problem occurs quite often. After server restart it is gone and when working with the server for a few minutes, then the error occurs, especially when uploading multiple files at once via drag&drop. Multiple files means here 10-15 with an overall size of <20 MB.

[Florent_Guillau]

As you can imagine we have plenty of customers using 2 concurrent users not having this problem. So you must debug on your side to give us an idea of what is going on. To start with this the first tool is jstack.

[René_]

I've uploaded the jstack output and the server log to dropbox for everyone to read

[Florent_Guillau]

A fix was recently pushed (https