topic Re: Suggestions for ACS 6.0 in cluster in Alfresco Forum

Suggestions for ACS 6.0 in cluster

marnad — Tue, 04 Jun 2019 15:04:15 GMT

Hi, I installed ACS 6.0 in cluster (2 nodes) under two windows 2016 server failover cluster (in vmware). The server didn't took the load (users sessions) althought i gave them a lot of ram (96g) and CPU (16). Tomcat jvm has 64g. Did someone installed an ACS 6 cluster that is running fine and supp

Re: Suggestions for ACS 6.0 in cluster

angelborroy — Tue, 04 Jun 2019 15:13:17 GMT

Do you mean 200-300 concurrent users or 200-300 users?

Anyway, in both cases, 64 GB and 16 CPUs is over-sized...

Re: Suggestions for ACS 6.0 in cluster

cesarista — Wed, 05 Jun 2019 05:21:03 GMT

Hi:

If you are using virtual machines, you may be sure that your RAM or CPU resources are really (or near) dedicated to your cluster (at least de RAM ones). If you have 200-300 potential users it seems oversized as commented by Angel initially, but having a huge number of documents in your repo, SOLR setup may need a lot JVM and CPU resources. Anyway, some basic profiling may help you to diagnose it (Support Tools or jvisualvm may help to see your JVM use and CPU load).

You should consider to have SOLR Search Services in a dedicated virtual machine and having SSD disks for SOLR indices, if you do not do it yet. Appropiate storage and database resources helps too, and normally the second ones are more relevant. Finally, I use Ubuntu or Linux based OS (when I can) in my setups, but this probably is not as important if you are a Windows administrator.

Regards.

--C.

Re: Suggestions for ACS 6.0 in cluster

marnad — Wed, 05 Jun 2019 13:03:14 GMT

It looks like from my actual installation (4.2), it would reach 100 concurrent users. And it works fine. No problem at all. My actual installation is on a physical server with windows 2012 R2 (128g ram, 32 cpu cores, 64g for tomcat). Nb of documents is about 2 500 000. We don't use share interface, only alfresco explorer. The new installation is for an upgrade to 6.0.1 with the same load. But after 30 concurrents sessions, alfresco became non responsive and slow.

I don't understand what could cause that.

Re: Suggestions for ACS 6.0 in cluster

angelborroy — Wed, 05 Jun 2019 13:16:55 GMT

You need to perform some insights on your 6.0.1 configuration, it should work fine with your current resources.

Re: Suggestions for ACS 6.0 in cluster

marnad — Wed, 05 Jun 2019 13:35:03 GMT

I'm asking that question, because i never heard or read on someone that used my kind of architecture. I mean two windows 2016 failover cluster with a CSV drive (Cluster Shared Volume) for the contentstore. I don't know if i've go to suspect my alfresco installation or my OS solution.

Re: Suggestions for ACS 6.0 in cluster

cesarista — Wed, 05 Jun 2019 15:48:01 GMT

What about db.pool.max parameter in alfresco-global.properties ?

Regards.

--C.

Re: Suggestions for ACS 6.0 in cluster

heiko_robert — Wed, 05 Jun 2019 16:28:55 GMT

If you're talking about cluster the Alfresco support team may be a better audience 😉

100 concurrent users shouldn't a problem depending on the usage scenario, number of nodes and the customizations. We have single instance (no cluster) customers with >500 concurrent, active working users with > 20 mio doc nodes in a 3 vm (alfersco, solr, postgres) installation having 3x 8 Cores and it is very fast.

You should know that a cluster will always cost performance and will never increase it (except some very uncommon read only scenarios).

Anyway: Independant from using a cluster or not: You should focus on common tuning and archtecture best practices which could not be covered by a forum thread.

In short:

* more RAM and (v)CPUs are not always better. More RAM may cause more CPU load and may just hide the real problems

* Run and tune db, repo/share tier, solr on separate systems

* In most cases the transformation, the db and especially their storage are the bottlenecks. Given you have the knowledge and resocurces making the db very fast may be the easier part. Avoid shared SAN LUNs and try to run at least the db on an exclusive cluster aware SSD backed storage. Same for the Solr index.

* it's easier to tune and find bottlenecks in linux instead of in windows

* never ever run docker on windows in production. It will not scale. Docker in production should only an option if you run it directly on bare metal installed linux systems / not VMs or maybe if you know what your're doing only one tier (alfresco, db, solr) on 1 VM. Our customers don't have the required teams for that. For the large systems mentioned before we run even 6.0 / 6.2 in esx VMs without any docker involved.

Re: Suggestions for ACS 6.0 in cluster

marnad — Wed, 05 Jun 2019 16:41:36 GMT

db.pool.max=100 alfresco tell me that they recommended 275.

I will try it but i doubt that it is the only concern.

Thnks

Marc

Re: Suggestions for ACS 6.0 in cluster

marnad — Wed, 05 Jun 2019 16:42:53 GMT

Heiko,

I really apreciate your feedback. It will help us re-think the solution.

Regards

Marc

Re: Suggestions for ACS 6.0 in cluster

marnad — Wed, 05 Jun 2019 17:32:48 GMT

BTW, Heiko, what are the version of alfresco and solr you are running? On which OS?

Thnks

Marc

Re: Suggestions for ACS 6.0 in cluster

heiko_robert — Wed, 05 Jun 2019 21:48:01 GMT

Marc,

most of our customers run on Alfresco 5.1/5.2, Solr 4 but from my experience the versions 5-6 differ only marginally in terms of performance and scalability.

Solr is a memory beast and requires special attention. Solr 6 has more features and better tools to avoid OOM exceptions but you need to handle them. Some recommend to split your index using sharding to overcome hardware limits.

90 % of unplanned downtimes we have seen in the last 10 years were related to an effect which we call thread escalation: If the Alfresco repository waits on or gets exceptions from other components (database, transformation, solr query) it will create new threads until the whole system has eaten all resources. The retrying transaction concept even accelerates the thread escalation. So

the main rule to scale your alfresco is to avoid increasing of threads. Monitor the number of threads and if they are increasing find out the root cause. On heavy used production systems you have only ~30 minutes to avoid a thread escalation and system downtime.
- are there enough free db connections ( expect 1 connection per active client + 20%) and is the db fast enough?
- if you give the jvm a lot of memory: check and monitor garbage collection
- are there increasing response times and number of threads from share?
- is the system waiting on solr response (mostly caused by OOM)? This may be addressed by log and jvm monitoring combined with automated kills/restarts. Check new tools shipped with solr 6 or write your own
- is the system waiting on transformations (mostly requested by share and solr) and therefore creating more (waiting) threads?

Re: Suggestions for ACS 6.0 in cluster

marnad — Thu, 27 Jun 2019 14:04:51 GMT

After many tests, it looks like my problem is a combination of cluster feature + High volume documents. When i test a 2 nodes cluster (2,5 million docs) with 10 concurrent sessions, it take tomcat about 30 min to top his memory usage and CPU. At that moment, alfresco become very slow, almost non-responsive. And most important, we could have more than 40 actives connexion to Oracle. Same test, same server with cluster feature disable, the server runs fine and the database connexion stay low (1-3). On the other end, i've got another setup (identical to the previous cluster) that is for dev, so with less documents (170 000). When i did the same test in cluster, alfresco stay very responsive, no balloon for tomcat, low active connexion to Oracle. So it looks like the volume has an influence when in cluster mode. Did someone experience this kind of situation?

Thanks

Re: Suggestions for ACS 6.0 in cluster

cesarista — Sat, 29 Jun 2019 18:05:27 GMT

Hi Marc, volumetry has always a big influence (but not only for a cluster). The term sizing speaks about this too. It is not the same an Alfresco alf_node_properties table with thousands of records, than one with several hundreds of millions. It is not the same a database size of 10Gb than a database 100 times bigger. It is not same the indices size in SOLR for some thousands of documents, than the size for several millions. The number of documents, the number of metadata properties and ACLs have a big impact on the resources needed by Alfresco and its components such as as SOLR or relational database. For example, the needed JVM for a SOLR instance depends directly on the number of documents of the repository.

https://docs.alfresco.com/5.2/concepts/solrnodes-memory.html

In your cluster, do you have SOLR in a dedicated machine or do you deploy SOLR in both nodes ?

Kind regards.

--C.

Re: Suggestions for ACS 6.0 in cluster

marnad — Tue, 02 Jul 2019 12:25:48 GMT

Hi Cesar,

I did the calculation for SOLR and validate with alfresco, so i hope that it is ok. My alf_node_properties table is about 50 millions records. Solr engine is deploy in both nodes. It really looks like there is a problem when i activate the cluster feature on my server with high volume database. Without cluster activated, the same server react normally. What overhead, cluster is causing?

Regards,

Marc

Re: Suggestions for ACS 6.0 in cluster

cesarista — Thu, 04 Jul 2019 07:11:44 GMT

Hi:

I can't say what is causing this overhead.

But I would point or try with SOLR first. I mean, one of the recommendations/best practices for reducing performance problems (in a cluster or not) is to separate SOLR from the repository. The memory calculation and CPU consumption is only for SOLR. You have a competition on machine resources between SOLR and Alfresco repository each every moment, not only in CPU and JVM memory, also in IO (SOLR indexing/writing and searching/reading) while Alfresco is ingesting data in contentstore. So you can try to:

Configure a third dedicated machine with Alfresco and SOLR similar to your current nodes but in this case, Alfresco will be used only for indexing. This Alfresco node should be out of the cluster and even in read only mode.
Configure your current Alfresco nodes (let say user nodes) in the cluster, for not having SOLR locally and pointing to the new third dedicated machine for searching.

Kind regards.

--C.