topic Scary log messages with clustering in Alfresco Archive

Scary log messages with clustering

marco_altieri — Tue, 30 Mar 2010 17:54:44 GMT

Hi all,I have a cluster with two alfresco (3.2r2) running in a standard configuration (as described in the wiki pages).Both servers share the database and the file system and the cache is configured as suggested in ehcache-custom.xml.sample.clusterWhen both servers are running the log file (on both

Re: Scary log messages with clustering

pmonks — Fri, 02 Apr 2010 15:07:39 GMT

The issue here is that you've turned on debug logging for EHCache, which is inadvisable since it is used a lot in a cluster and so will generate a lot of log messages. If debug logging is enabled for more than just EHCache then that would also explain the slowness of the system - with debug logging enabled system wide, Alfresco will be generating a tremendous amount of logging output, which slows down the system considerably.

Cheers,
Peter

Re: Scary log messages with clustering

marco_altieri — Fri, 02 Apr 2010 16:26:30 GMT

Hi pmonks, thank you very much for your response.

It would be great if the problem is just a logging issue. I'll do a check asap.

I'm a bit pessimistic about that.
I used log levels suggested in alfresco wiki pages and the amount of log messages does not seem too much.
I saw much more messages when I used DEBUG for hibernate level and these messages didn't lock the server.
It's also difficult to understand why the servers share so many "remove operations" and for such a long period of time (more than 15 minutes).

Marco

Re: Scary log messages with clustering

pmonks — Fri, 02 Apr 2010 17:25:20 GMT

Those remove operations are cache invalidation events, not content removal events. EHCache on one node in the cluster is informing the other node(s) in the cluster that certain cache entries are stale and should therefore be dropped from their caches.

Cheers,
Peter

Re: Scary log messages with clustering

joe_l3 — Fri, 16 Apr 2010 15:26:31 GMT

Hi all,
I had the same problem. I also configured a cluster with two Alfresco nodes (both servers share the database and the file system). When I enabled ehcache with automatic peerDiscovery property on both nodes, the performance of two application servers collapsed and log files were filled with "Remote remove received for key:…." messages !

Here my system configuration:
- Alfresco Comunity Ed. 3.2r2
- Mysql 5.0.77
- Java 1.6.0_16
- Apache Tomcat Server 6.0.18

Possible solution:
In ehcache-custom.xml I tried to disable the automatic peerDiscovery property for cacheManagerPeerProviderFactory. I set "peerDiscovery=manual" and a limited number of rmiUrls. Here an example:

<cacheManagerPeerProviderFactory
            class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
            properties="peerDiscovery=manual,
                        rmiUrls=
         //node2_IP:40001/org.hibernate.cache.StandardQueryCache|
         //node2_IP:40001/org.hibernate.cache.UpdateTimestampsCache|
         //node2_IP:40001/org.alfresco.repo.domain.hibernate.NodeImpl|
         //node2_IP:40001/org.alfresco.repo.domain.hibernate.QNameEntityImpl|
         //node2_IP:40001/org.alfresco.repo.domain.hibernate.NodeImpl.aspects|
         //node2_IP:40001/org.alfresco.repo.domain.hibernate.NodeImpl.properties|
         //node2_IP:40001/org.alfresco.cache.compiledModelsCache"
       />
    <cacheManagerPeerListenerFactory
            class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
            properties="port=40001, socketTimeoutMillis=5000"
    />
…………….‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Obviously you have to do a study to determine what ehcache items to delete from rmiUrls without regression. :idea:

Re: Scary log messages with clustering

marco_altieri — Sat, 17 Apr 2010 20:30:30 GMT

Hi Peter,
I tried disabling log masseges, but the clustering was still totally freezed. Obviously I can't see any message…
The CPUs were 99% allocated to tomcat processes on both servers.

Those remove operations are cache invalidation events, not content removal events. EHCache on one node in the cluster is informing the other node(s) in the cluster that certain cache entries are stale and should therefore be dropped from their caches.

Cheers,
Peter

Re: Scary log messages with clustering

marco_altieri — Fri, 23 Apr 2010 14:27:49 GMT

It doesn't seem a problem related to the database because I imported it on a cluster configured on my PC and the it worked fine.

Re: Scary log messages with clustering

marco_altieri — Tue, 29 Jun 2010 01:46:15 GMT

I realized that the disk used by the cache org.alfresco.cache.ticketsCache grows constantly at a rate of about 10MB/hour.
Is it possible that this cache is the cause of the traffic that freezes the cluster?

Is it still possile to disable clustering for this cache ?

Thanks!
Marco

Re: Scary log messages with clustering

sam13 — Mon, 02 Aug 2010 01:59:16 GMT

You two has a good and full of value conversation,,,.
Thanks…