Differential sync and full sync between AD and Alfresco

[erinn]

I have spent a fair amount of time configuring a new alfresco install (community edition 3.2r2) and have run into some issues which I can't seem to get past. I am no LDAP expert so please hang in with me as I try to fumble my way through this.

We have multiple domain controllers in our environment (about 15 I think) and requests to a single IP get handed off to one of those 15 domain controllers. This seems to cause issues for differential syncs. The problem appears to be that a differential sync is keyed off of the modifyTimestamp (which oddly we don't have, but modifyTimestamp appears to just be an alias (not LDAP terminology) for the whenChanged attribute), the problem is as I understand it that the whenChanged/modifyTimestamp is not synced between DCs. See http://blog.joeware.net/2006/10/03/655/.

So I have one of two choices if that information is correct, one I can choose only one DC and sync off of it, running the risk of it failing etc. Or I can do a full sync every time and skip the modifyTimestamp/whenChanged conundrum. Problem is neither one of those options appears to be working, could be a bug, but more than likely it is a mis-configuration on my part. Ideally I suppose I would like a full sync to run every single time, it just makes more sense for our needs at this point. Anyway here is my configuration:

From alfresco-global.properties

synchronization.synchronizeChangesOnly=false
#synchronization.import.cron=0 * * * * ?
synchronization.autoCreatePeopleOnLogin=false
synchronization.syncWhenMissingPeopleLogIn=false
‍‍‍‍‍‍

As a side note either I don't understand the cron entry or it does not work as advertised. With this uncommented, my understanding of cron is that it should run every hour, at the top of the hour, but it ends up running every minute. Anyway, onwards.

My changes.properties for ldap-ad1:

# This flag enables use of this LDAP subsystem for authentication. It may be
# that this subsytem should only be used for synchronization, in which case
# this flag should be set to false.
ldap.authentication.active=true

#
# This properties file brings together the common options for LDAP authentication rather than editing the bean definitions
#
ldap.authentication.allowGuestLogin=false

# How to map the user id entered by the user to that passed through to LDAP
# In Active Directory, this can either be the user principal name (UPN) or DN.
# UPNs are in the form <sAMAccountName>@domain and are held in the userPrincipalName attribute of a user
ldap.authentication.userNameFormat=%s@domain.local

# The LDAP context factory to use
ldap.authentication.java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory

# The URL to connect to the LDAP server 
ldap.authentication.java.naming.provider.url=ldaps://domain.local:636

# The authentication mechanism to use
ldap.authentication.java.naming.security.authentication=simple

# Escape commas entered by the user at bind time
# Useful when using simple authentication and the CN is part of the DN and contains commas
ldap.authentication.escapeCommasInBind=false

# Escape commas entered by the user when setting the authenticated user
# Useful when using simple authentication and the CN is part of the DN and contains commas, and the escaped \, is 
# pulled in as part of an LDAP sync
# If this option is set to true it will break the default home folder provider as space names can not contain \
ldap.authentication.escapeCommasInUid=false

# Comma separated list of user names who should be considered administrators by default
ldap.authentication.defaultAdministratorUserNames=foo

# This flag enables use of this LDAP subsystem for user and group
# synchronization. It may be that this subsytem should only be used for 
# authentication, in which case this flag should be set to false.
ldap.synchronization.active=true

# The default principal to bind with (only used for LDAP sync). This should be a UPN or DN
ldap.synchronization.java.naming.security.principal=foo@domain.local

# The password for the default principal (only used for LDAP sync)
ldap.synchronization.java.naming.security.credentials=password

# If positive, this property indicates that RFC 2696 paged results should be
# used to split query results into batches of the specified size. This
# overcomes any size limits imposed by the LDAP server.
ldap.synchronization.queryBatchSize=1000

# The query to select all objects that represent the groups to import.
ldap.synchronization.groupQuery=(&(objectclass=group)(memberOf=cn=Alfresco Groups,OU=Alfresco,DC=domain,DC=local))

# The query to select objects that represent the groups to import that have changed since a certain time.
ldap.synchronization.groupDifferentialQuery=(&(objectclass=group)(memberOf=cn=Alfresco Groups,OU=Alfresco,DC=domain,DC=local)(!(whenChanged<\={0})))

# The query to select all objects that represent the users to import.
ldap.synchronization.personQuery=(&(objectclass\=user)(memberOf=cn=Alfresco Users,OU=Alfresco,DC=domain,DC=local)(userAccountControl\:1.2.840.113556.1.4.803\:\=512))

# The query to select objects that represent the users to import that have changed since a certain time.
ldap.synchronization.personDifferentialQuery=(&(objectclass\=user)(memberOf=cn=Alfresco Users,OU=Alfresco,DC=domain,DC=local)(userAccountControl\:1.2.840.113556.1.4.803\:\=512)(!(whenChanged<\={0})))

# The group search base restricts the LDAP group query to a sub section of tree on the LDAP server.
ldap.synchronization.groupSearchBase=DC=domain,DC=local

# The user search base restricts the LDAP user query to a sub section of tree on the LDAP server.
ldap.synchronization.userSearchBase=DC=domain,DC=local

# The name of the operational attribute recording the last update time for a group or user.
ldap.synchronization.modifyTimestampAttributeName=whenChanged

# The timestamp format. Unfortunately, this varies between directory servers.
ldap.synchronization.timestampFormat=yyyyMMddHHmmss'.0Z'

# The attribute name on people objects found in LDAP to use as the uid in Alfresco
ldap.synchronization.userIdAttributeName=sAMAccountName

# The attribute on person objects in LDAP to map to the first name property in Alfresco
ldap.synchronization.userFirstNameAttributeName=givenName

# The attribute on person objects in LDAP to map to the last name property in Alfresco
ldap.synchronization.userLastNameAttributeName=sn

# The attribute on person objects in LDAP to map to the email property in Alfresco
ldap.synchronization.userEmailAttributeName=mail

# The attribute on person objects in LDAP to map to the organizational id  property in Alfresco
ldap.synchronization.userOrganizationalIdAttributeName=company

# The default home folder provider to use for people created via LDAP import
ldap.synchronization.defaultHomeFolderProvider=userHomesHomeFolderProvider

# The attribute on LDAP group objects to map to the gid property in Alfrecso
ldap.synchronization.groupIdAttributeName=cn

# The group type in LDAP
ldap.synchronization.groupType=group

# The person type in LDAP
ldap.synchronization.personType=user

# The attribute in LDAP on group objects that defines the DN for its members
ldap.synchronization.groupMemberAttributeName=member

# If true progress estimation is enabled. When enabled, the user query has to be run twice in order to count entries.
ldap.synchronization.enableProgressEstimation=true
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So all this is, I believe, setup to do full syncs, at least from the global properties file, so here is what I am getting in the logs:

14:43:50,136 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] Synchronizing users and groups with user registry 'ldap-ad1'
14:43:50,145 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] Retrieving groups changed since May 12, 2009 11:36:27 AM from user registry 'ldap-ad1'
14:43:50,676 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] ldap-ad1 Group Analysis: Commencing batch of 0 entries
14:43:50,678 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] ldap-ad1 Group Analysis: Completed batch of 0 entries
14:43:50,682 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] Retrieving users changed since Jan 22, 2010 11:49:48 AM from user registry 'ldap-ad1'
14:43:50,829 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] ldap-ad1 User Creation and Association: Commencing batch of 0 entries
14:43:51,046 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] ldap-ad1 User Creation and Association: Completed batch of 0 entries
14:43:51,065 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] Finished synchronizing users and groups with user registry 'ldap-ad1'
14:43:51,066 INFO  [org.alfresco.repo.security.sync.ChainingUserRegistrySynchronizer] 0 user(s) and 0 group(s) processed
‍‍‍‍‍‍‍‍‍‍‍

Now it certainly appears from the logs that this is trying to run a differential sync.

User changes are not getting synced up on a regular basis, if I remove a user from the Alfresco Users group in the AD that change will get propagated over, but if I re-add the user that change will not make it over to Alfresco. Group changes are not propagated either. The only way I can get changes to sync up is via the very first sync from a clean install (Incidentally is there a way to clear the user store in alfresco, for testing purposes, so I don't have to re-install?).

So a number of questions to sum up this ramble:
Does the setup of whenChanged look correct?
Is there a solution to issue of whenChanged/modifyTimestamp not being synced between DCs?
If no is there a way to force a full sync every time?
Is the way I am trying not correct (I assume so)?
Also is there a simple way to just force it once if I want to?
Is that cron entry broken, or malformed on my part?

A lot of questions I know, I appreciate any help that can be offered.

-Erinn

[sergio1024]

Hi,

I have the same problem… The first synchro get all my users (members of my AD group "GRP_ALFRSCO").

If I upgrade the group and reboot alfresco, the log is :
…0 user(s) and 0 group(s) processed…

I tried to install the alfresco.war version 3.3DEV but the problem still…

If you find the solution … I take !! lol

Regards.

[dward]

When you start / restart Alfresco, it always does a differential sync (unless it is the first startup).This is because doing a full sync every time would take too long to start Alfresco.

It's only the sync triggered by the CRON expression that's affected by the synchronization.synchronizeChangesOnly flag.

If you can't rely on timestamps, I would recommend that you configure the differential queries to be identical to the non-differential queries.

See this page for Quartz cron syntax

http://www.quartz-scheduler.org/docs/tutorial/TutorialLesson06.html

[mkalyta]

Dward,

I have the same problem as guys describe - using Windows Server 2008 R2 and Alfresco 3.2. Tried various ways but still does not work as expected.

When you say "If you can't rely on timestamps" does it mean that Alfresco may fail to get the date/time values from AD?
I have noticed that by default for AD LDAP configuration in Alfresco one of the settings is:

# The name of the operational attribute recording the last update time for a group or user.
ldap.synchronization.modifyTimestampAttributeName=modifyTimestamp

but in AD the change attribute is called "whenChanged". So should I use whenChanged instead of modifyTimestamp?

Also - could somebody tell me for which exact purpose it is recommended to grant "Read all inetOrgPerson information" permission to a bind account - this is what is stated here: http://wiki.alfresco.com/wiki/Alfresco_Authentication_Subsystems#Active_Directory_Tips

Thanks

[erinn]

When you start / restart Alfresco, it always does a differential sync (unless it is the first startup).This is because doing a full sync every time would take too long to start Alfresco.

It's only the sync triggered by the CRON expression that's affected by the synchronization.synchronizeChangesOnly flag.

If you can't rely on timestamps, I would recommend that you configure the differential queries to be identical to the non-differential queries.

See this page for Quartz cron syntax

http://www.quartz-scheduler.org/docs/tutorial/TutorialLesson06.html

In hindsight that seems pretty clear thanks for the help and the link to the cron docs, it was seconds of course, who knew they added another field.

I don't know if there is a way to fix this overall but you are pitching this as an enterprise solution and surely in most enterprises you are going to run into more than one domain controller, so although this can be worked around, maybe another route for differential syncs is needed (if it is even possible another way).

As for the other user that is experiencing issues, yours may not be exactly the same as mine, the problem with whenChanged/modifyTimestamp only crops up if you have more than one domain controller and you have no way of ensuring that the domain controller you are talking to this time is the same as the domain controller you talked to during the last sync. But for historical sake here is my new config that works:

ldap.synchronization.groupQuery=(&(objectclass=group)(memberOf=cn=Alfresco Groups,OU=Alfresco,DC=domain,DC=local))

# The query to select objects that represent the groups to import that have changed since a certain time.
ldap.synchronization.groupDifferentialQuery=(&(objectclass=group)(memberOf=cn=Alfresco Groups,OU=Alfresco,DC=domain,DC=local))

# The query to select all objects that represent the users to import.
ldap.synchronization.personQuery=(&(objectclass\=user)(memberOf=cn=Alfresco Users,OU=Alfresco,DC=domain,DC=local)(userAccountControl\:1.2.840.113556.1.4.803\:\=512))

# The query to select objects that represent the users to import that have changed since a certain time.
ldap.synchronization.personDifferentialQuery=(&(objectclass\=user)(memberOf=cn=Alfresco Users,OU=Alfresco,DC=domain,DC=local)(userAccountControl\:1.2.840.113556.1.4.803\:\=512))

The only thing that is different is the removal of the whenChanged/modifyTimestamp part of the query, this makes all queries full queries.

Now if I can figure out why share isn't working with any of this and I will be in business.

Thanks again,
-Erinn

[dward]

It would seem that it is possible to force connection to a single domain controller every time, e.g. LDAP://somedc.domain.com/RootDSE

See http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.adsi.general&...

This will be necessary to enable differential sync, given that whenChanged/modifyTimestamp/uSNChanged are not replicated. Microsoft have acknowledged this limitation. See

http://msdn.microsoft.com/en-us/library/ms677627(VS.85).aspx