Cluster fun - Ehcache remote debugger

[tdell]

I've been trying to set up a PoC for an Alfresco cluster the past few weeks. I'm using Alfresco Enterprise 2.1.1. My latest attempt is the closest I've been, but there's still issues. For example creating a space seems to carry on to the other server(s), creating content takes up to 5 minutes, and modifying content properties doesn't ever seem to.

I've been trying to get the Remote Debugger running to see if that will help me out..

alf0:/opt/alfresco/tomcat/webapps/alfresco/WEB-INF/lib# java -jar ehcache-1.3.0-remote-debugger.jar /opt/alfresco/tomcat/shared/classes/alfresco/extension/ehcache-custom.xml
Increase the net.sf.ehcache.distribution logging level to debug or trace to see distributed cache operations as they occur.
Exception in thread "main" java.lang.NoClassDefFoundError: edu.emory.mathcs.backport.java.util.concurrent.Executors
   at net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver.init(MulticastKeepaliveHeartbeatReceiver.java:81)
   at net.sf.ehcache.distribution.MulticastRMICacheManagerPeerProvider.init(MulticastRMICacheManagerPeerProvider.java:89)
   at net.sf.ehcache.CacheManager.init(CacheManager.java:221)
   at net.sf.ehcache.CacheManager.<init>(CacheManager.java:153)
   at net.sf.ehcache.distribution.RemoteDebugger.main(RemoteDebugger.java:46)‍‍‍‍‍‍‍‍

I have the following in log4j.properties:

log4j.logger.net.sf.ehcache.distribution=DEBUG
log4j.logger.org.alfresco.repo.node.index.IndexTransactionTracker=DEBUG
log4j.logger.org.alfresco.repo.node.index.AVMRemoteSnapshotTracker=DEBUG‍‍‍

I've been going through Ehcache documentation and other forums, and I haven't find anything related to my problem. Am I missing something here?

[francescof]

Seems like you need backport-util-concurrent.jar in your classpath. You can find it here: http://backport-jsr166.sourceforge.net/

[steve]

Hi,
Are both your cluster servers time-synched? - (tt is a good idea to have ntpd running on both)
How are you configuring your cluster?

Steve

[tdell]

I found out about backport few hours after I posted my previous message. Thanks!

For the proof of concept this is in a VM environment. I use VM tools with time sync enabled, but I'll double check on that they are synced.

The cluster is set up as so:

  - 2 Alfresco VMs running in Debian (Currently using multicast discovery and delivery)
  - Single repository shared via NFS
  - Single Mysql database

When I get these issues worked out I'll be adding:

  - Apache load balancer
  - Windows AD
  - Windows DFS server
  - Mysql replication
  - Clustered filesystem

This PoC is to replace Documentum for 1800 users.. (yay!)

While I have you here, there's also one other thing I've been wondering about…

After reading:
  http://ehcache.sourceforge.net/documentation/distributed_design.html

One question about the multicast delivery… In a 2 Alfresco server set up, multicast doesn't seem preferable. Custom TCP delivery seems like the optimal setup, but after further reading I found out Ehcache needs a restart if you add or remove a server from a cluster set up in this fashion.

The Alfresco sample ehcache cluster config uses multicast. Being UDP, packets aren't guaranteed to arrive. It seems likely that eventually there could be cache inconsistency, which would cause really unpredictable results. Ehcache documentation even warns people about this.

There's 3 more delivery mechanisms. One being JMS (Java Message Service). Are there Alfresco clusters out there using any of the alternatives? As well as being a supported configuration?

Thanks for the responses, I'll check on the time synchronization and let you know..