topic Re: LDAP sync with multiple subsytems connection timeout in Alfresco Forum

LDAP sync with multiple subsytems connection timeout

booltrue — Thu, 29 Jun 2017 12:57:41 GMT

I am using alfresco Community - 5.1.0 (r127059-b7)and synchronizing currently 4 ldap subsytems ad1,ad2,ad3,ad4.ad1 is our domain, ad2,ad3,ad4 is not our domain.From time to time I get a connection timeout from a subsytem.Synchronization and authentication, both work so far.I also tried to increase t

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Fri, 30 Jun 2017 07:43:02 GMT

Does ldap.authentication.java.naming.read.timeout=0 have any effect in the alfresco community version,

due I sometimes still get connection timeout during synchronization.

Re: LDAP sync with multiple subsytems connection timeout

afaust — Fri, 30 Jun 2017 10:12:31 GMT

There is a documentation error. Setting the timeout to 0 does NOT mean it disables the timeout, it just sets it to the literal value of 0. The Alfresco documentation and comments in the sample configuration files are plainly wrong in this instance.

This affects both Enterprise and Community - incidentally I recently had an Enterprise customer run into a similar issue. The only solution was to increase the timeout.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Fri, 30 Jun 2017 15:44:16 GMT

First I tried setting the timeout value to 100000ms, but I got a connection error.

After that I tried to set the timeout value to infinitely with value 0 and -1, I didnt know that it has not the described effect.

But with 100000ms I got a connection error after ~30s too as you can see in the log.

I will try on monday setting the timeout value to 200000ms to see if the connection error appears after the same time.

I thought setting the timeout value to 100000ms for each ldap subsystem would result in a max. waiting time of 100s for each ldap subsystem before a timeout will be received, so I dont understand why the connection error comes before that defined timeout value as you can see in the log.

What is the max allowed value you can set for the timeout value ?

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Mon, 03 Jul 2017 07:58:20 GMT

I increased the timeout value to 200000, but still get connection error after same time.

After the connection error, synchronization stops and doesn't continue with the next ldap subsystem in the chain.

Only when no connection error occurs during synchronization all ldap subsystems will be processed successfully.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Tue, 04 Jul 2017 11:58:54 GMT

Regarding the sync error, I get sometimes the following error, what does it mean ?

Is it also a connection error ?

2017-07-04 13:15:00,234 INFO [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Synchronizing users and groups with user registry 'ad2'
2017-07-04 13:15:00,234 WARN [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Full synchronization with user registry 'ad2'
2017-07-04 13:15:00,234 WARN [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Some users and groups previously created by synchronization with this user registry may be removed.
2017-07-04 13:15:00,250 INFO [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Retrieving groups changed since 03.07.2017 05:39:48 from user registry 'ad2'
2017-07-04 13:15:00,796 ERROR [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Synchronization aborted due to error
org.alfresco.error.AlfrescoRuntimeException: 06040125 Error during LDAP Search. Reason:null
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1303)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.getGroups(LDAPUserRegistry.java:691)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.syncWithPlugin(ChainingUserRegistrySynchronizer.java:973)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronizeInternal(ChainingUserRegistrySynchronizer.java:719)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronize(ChainingUserRegistrySynchronizer.java:448)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob$1.doWork(UserRegistrySynchronizerJob.java:46)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:548)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob.execute(UserRegistrySynchronizerJob.java:42)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1284)
   ... 9 more
Caused by: javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name ''
   at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.search(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapReferralContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreReferrals(Unknown Source)
   ... 12 more
Caused by: com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ''
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(Unknown Source)
   ... 12 more
2017-07-04 13:15:00,874 ERROR [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-9] Synchronization aborted due to error
org.alfresco.error.AlfrescoRuntimeException: 06040125 Error during LDAP Search. Reason:null
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1303)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.getGroups(LDAPUserRegistry.java:691)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.syncWithPlugin(ChainingUserRegistrySynchronizer.java:973)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronizeInternal(ChainingUserRegistrySynchronizer.java:719)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronize(ChainingUserRegistrySynchronizer.java:448)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob$1.doWork(UserRegistrySynchronizerJob.java:46)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:548)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob.execute(UserRegistrySynchronizerJob.java:42)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1284)
   ... 9 more
Caused by: javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name ''
   at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.search(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapReferralContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreReferrals(Unknown Source)
   ... 12 more
Caused by: com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ''
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(Unknown Source)
   ... 12 more
2017-07-04 13:15:00,889 ERROR [org.quartz.core.JobRunShell] [DefaultScheduler_Worker-9] Job DEFAULT.ldapPeopleJobDetail threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 06040125 Error during LDAP Search. Reason:null
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1303)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.getGroups(LDAPUserRegistry.java:691)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.syncWithPlugin(ChainingUserRegistrySynchronizer.java:973)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronizeInternal(ChainingUserRegistrySynchronizer.java:719)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronize(ChainingUserRegistrySynchronizer.java:448)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob$1.doWork(UserRegistrySynchronizerJob.java:46)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:548)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob.execute(UserRegistrySynchronizerJob.java:42)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1284)
   ... 9 more
Caused by: javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name ''
   at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.search(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapReferralContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreReferrals(Unknown Source)
   ... 12 more
Caused by: com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ''
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(Unknown Source)
   ... 12 more
2017-07-04 13:15:00,889 ERROR [org.quartz.core.ErrorLogger] [DefaultScheduler_Worker-9] Job (DEFAULT.ldapPeopleJobDetail threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 06040125 Error during LDAP Search. Reason:null]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 06040125 Error during LDAP Search. Reason:null
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1303)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.getGroups(LDAPUserRegistry.java:691)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.syncWithPlugin(ChainingUserRegistrySynchronizer.java:973)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronizeInternal(ChainingUserRegistrySynchronizer.java:719)
   at org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer.synchronize(ChainingUserRegistrySynchronizer.java:448)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob$1.doWork(UserRegistrySynchronizerJob.java:46)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:548)
   at org.alfresco.repo.security.sync.UserRegistrySynchronizerJob.execute(UserRegistrySynchronizerJob.java:42)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   ... 1 more
Caused by: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name '']
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
   at org.alfresco.repo.security.sync.ldap.LDAPUserRegistry.processQuery(LDAPUserRegistry.java:1284)
   ... 9 more
Caused by: javax.naming.ServiceUnavailableException: DomainDnsZones.a.b.c:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name '']; remaining name ''
   at com.sun.jndi.ldap.Connection.readReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.getSearchReply(Unknown Source)
   at com.sun.jndi.ldap.LdapClient.search(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.doSearch(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.searchAux(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.c_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(Unknown Source)
   at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapReferralContext.search(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreReferrals(Unknown Source)
   ... 12 more
Caused by: com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ''
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.LdapCtx.processReturnCode(Unknown Source)
   at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(Unknown Source)
   ... 12 more

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Thu, 06 Jul 2017 08:17:09 GMT

I still get the same error from time to time.

Can anybody please explain what does this error mean ?

Thanx.

Re: LDAP sync with multiple subsytems connection timeout

afaust — Thu, 06 Jul 2017 08:58:55 GMT

The term "ServiceUnavailableException" appears to sum it up. For some reason or another, your LDAP system appears to be disconnecting / not connected to the network at some times. Increasing the timeout will improve nothing if there is a legitimate connection issue. In similar cases and when customers insist on "but our network is working perfectly" I usually recommend setting up a small verification script / tool that constantly tries to connect to the system in question (every couple of seconds/minutes) and reports via log file if that succeeded. In most cases people will see that - independent of Alfresco - there is a networking issue and the tool fails to connect for a specific amount of time every couple of days.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Thu, 13 Jul 2017 09:16:18 GMT

Sorry for my late reply.

I am just testing the ldap connections of several domains with some services, to see if there occurs a network error on domain side,

or there is an error on Alfresco side.

I had to figure out the trigger time of the services to get the timeout error at the same time as it occurs in Alfresco.

First I used 2 min, but I couldnt fetch the error at same time it occurs on Alfresco side.

I changed now to 30 s, let's see what happens.

I will let you know.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Mon, 17 Jul 2017 07:31:59 GMT

During that weekend occured a connection timout on Alfresco side:

2017-07-15 18:00:00,131 ERROR [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-6] Synchronization aborted due to error

But my service, that is triggering all 30s didn't fetch it:

15.07.2017 17:59:41

15.07.2017 18:00:11
15.07.2017 18:00:41

My service synchronized the same group and all the users properly.

No errors occured !

Re: LDAP sync with multiple subsytems connection timeout

afaust — Mon, 17 Jul 2017 07:52:29 GMT

Is your service only connecting to the LDAP or also processing all the data just like Alfresco would? Also your service is likely not going to exhibit the same behaviour as Alfresco, e.g. Alfresco may have periods where no call is being made to LDAP while the data loaded from the last batch is being processed.

Apart from a verification script, you could also set up a network capturing process to record the exchanges between Alfresco and LDAP (e.g. tcpdump). With that data you could then use Wireshark to analyse the low-level communication behaviour and determine e.g. what side introduces a delay that leads to the timeout.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Mon, 17 Jul 2017 08:24:20 GMT

Alfresco is configured that the scheduler is synchronizing with ldap all 15min.

I thought the synchronization process will only update the groups and users from ldap ?

The connection timeout occurs only in relation with the synchronization trigger.

My service only synchronizes the groups and users as configured in Alfresco, but not more.

I wanted to figure out, if network or ldap connection errors occur in the synced domain.

Re: LDAP sync with multiple subsytems connection timeout

booltrue — Fri, 28 Jul 2017 07:47:37 GMT

I observed the behaviour for 2 weeks. Sometimes quite stable, but today I got within 9h 10 timeouts.

I am synchronizing all 15 min. What data could be processed a lot that causes the timeouts, specially today ?

The timeout only occurs at one domain. But the domain and network is available due my service gets positive response.

If ldap synchronization is not working several times do I still can login ?? Or only new added users are not synced to Alfresco ?

What more effects does aborted sync have to the functionality of Alfresco ?

Didn't anyone have the same problem before?

If still connecton errors occur I try to figure out by capturing the network traffic.