LDAP connection leaks for activiti-rest-webapp

[gregdavisfromnj]

I've recently started doing some load testing, and am finding that when deployed in Tomcat, and with version 5.17.0, the REST Web Application is leaking LDAP connections when configured to use LDAP authorization.

I've setup a context XML file to configure the application, with the LDAP config parts like this:

…
<bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration">
  …
 <property name="configurators">
            <list>
                <bean class="org.activiti.ldap.LDAPConfigurator">

                    <!– Server connection params –>
                    <property name="server" value="${ldap.server}"/>
                    <property name="port" value="${ldap.port}"/>
                    <property name="user" value="${ldap.username}"/>
                    <property name="password" value="${ldap.password}"/>

                    <!– Query params –>
                    <property name="baseDn" value="${ldap.baseDn}"/>
                    <property name="queryUserByUserId" value="${ldap.queryUserByUserId}"/>
                    <property name="queryUserByFullNameLike" value="${ldap.queryUserByFullNameLike}"/>
                    <property name="queryGroupsForUser" value="${ldap.queryGroupsForUser}"/>

                    <!– Attribute config –>
                    <property name="userIdAttribute" value="${ldap.userIdAttribute}"/>
                    <property name="userFirstNameAttribute" value="${ldap.userFirstNameAttribute}"/>
                    <property name="userLastNameAttribute" value="${ldap.userLastNameAttribute}"/>

                    <property name="groupIdAttribute" value="${ldap.groupIdAttribute}"/>
                    <property name="groupNameAttribute" value="${ldap.groupNameAttribute}"/>

                    <!– Required for ActiveDirectory –>
                    <property name="customConnectionParameters">
                        <map>
                            <entry key="InitialDirContext" value="Context.REFERRAL"/>
                            <entry key="com.sun.jndi.ldap.connect.pool" value="${ldap.pool.enable}"/>
                            <entry key="com.sun.jndi.ldap.connect.pool.initsize" value="${ldap.pool.initsize}"/>
                            <entry key="com.sun.jndi.ldap.connect.pool.maxsize" value="${ldap.pool.maxsize}"/>
                            <entry key="com.sun.jndi.ldap.connect.pool.prefsize" value="${ldap.pool.prefsize}"/>
                            <entry key="com.sun.jndi.ldap.connect.pool.protocol" value="${ldap.pool.protocol}"/>
                            <entry key="com.sun.jndi.ldap.connect.pool.timeout" value="${ldap.pool.timetoevict}"/>
                        </map>
                    </property>
                </bean>
            </list>
        </property>
  …
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And with a properties file like this to source those property values:

… (some omitted properties)…

# connection pooling
ldap.pool.enable=true
ldap.pool.initsize=2
ldap.pool.maxsize=10
ldap.pool.prefsize=4
ldap.pool.protocol=plain ssl
ldap.pool.timetoevict=180000
‍‍‍‍‍‍‍‍‍‍‍

The trouble is, after a while, I exceed the open file descriptor limit.  I can set the limit higher and higher, but eventually Tomcat just crashes.  I tried to use LDAP pooling to put a cap on the open connections, and to close the unused ones.  But, those pooling settings aren't having any effect.  I can see that the LDAP connections are leaking by finding the PID of tomcat, and using "lsof".  The connections will grow slightly through the ulimit I set (currently 16384).  For example, "sudo lsof -p 12345 | grep ldap | wc -l" will grow from 1 to 16xxx.

I don't see why the LDAP connections are not being closed.  But if they aren't manually closed, I expect the pool timeout value should close the idle connections and control the growth.  But, the connections are not closed, in any event.

Oh yeah, I am using ActiveDirectory as the LDAP server.  It properly authenticates, and I have a custom RestAuthenticator (extended from BasicAuthenticationProvider) that just matches the username against a property to authorize access.  Was something changed on BasicAuthenticationProvider or RestAuthenticator that requires closing the LDAP context manually somehow in any custom implementation?

[gregdavisfromnj]

When I set the connection pooling properties when starting the JVM (rather than or at least in addition to the bean config above), I see the connection count ramp up, then drop every few minutes.  So it looks like I am not setting those properties in the bean config properly… or they don't belong there.

As an aside, I also upgraded to JDK 8 when upgrading to 5.17.0 (that was not a great idea, should have changed one thing at a time).  The difference may be in whether or not pooling is opportunistically turned on in JDK 8 vs 7.  A line from the Oracle docs indicate that the behavior I am seeing is in line with a idle connections waiting to garbage collect, and when no timeout is set.  See the section on Idle Connections at http://docs.oracle.com/javase/7/docs/technotes/guides/jndi/jndi-ldap.html :

When the application is finished with a pooled connection (by invoking Context.close() on all contexts that share the connection), the underlying pooled connection is marked as idle, waiting to be reused. By default, idle connections remain in the pool indefinitely until they are garbage-collected. If the "com.sun.jndi.ldap.connect.pool.timeout" System property has been set, the LDAP provider will automatically close and remove pooled connections that have been idle for more than the specified period.

I have tuned the garbage collection settings to avoid stop-the-world collection events… and it seems that is highlighting the potential for the file descriptor leak when the pooling settings are overly reliant on garbage collection for resource cleanup.

[gregdavisfromnj]

With my pooling properties… there is a different problem.  For an LDAP pool maxsize=10, I can issue 10 HTTP requests to the REST web app.  Thereafter, it hangs, but I don't see any errors in the catalina.out file about exceeding file descriptor limits.

[srini_n]

Hi,
I am running into a hotspot with BasicAuthenticationProvider authenticating requests (with LDAP) to Activiti-Rest. 99% of the time taken by the requests are spent in JNDI. I assume I will need some kind of connection pooling for LDAP. Did you get your configuration working correctly? I am using Activiti 5.19.2.
Thanks

[gregdavisfromnj]

It works, but only with forced periodic garbage collection. At some point I'm going to look into just using Spring LDAP authentication rather than the Activiti authentication. But I suspect I'll have the same problems with the Oracle LDAP client.

If you are burning CPU inside of JNDI, make sure that you have your filters defined as tight as possible. It sounds like you are getting back too many results from the LDAP server to evaluate. Try using JXplorer or LDAPBrowser to replicate the authentication queries, so you see what Activiti is dealing with from the LDAP server.