Alfresco in multi core enviroment

[adanecki]

Hello,

we have installed Alfresco 5.0.d Community Edition.
Our server is Ubuntu 12.04.2 LTS on Proxmox Virtualization Platform.

System:
CPU : AMD Opteron™ Processor 6180 SE (48 core)
RAM: 64GB

Alfresco is the only VM on this Proxmox Server. There are no other VM.
For Alfresco we set 32 cpu cores, and 60GB of RAM

Java settings:
JAVA_OPTS="-Xms8192M -Xmx50G -Xss1024k
Rest 10Gb is lest for other precesses ,like DB engine, LibreOffice

problem is witch Apache Tomcat, cause it uses only 1 , mostly 2 cpu cores.
and second problem is LibreOffice, which after restart uses 100 % cpu - 1 core.
Of course after LibreOffice restart it's Libre process is back to normal cpu usage.

We have standard Alfresco installation without any java version changes
Tomcat 7.0
java version "1.6.0_35"
OpenJDK Runtime Environment (IcedTea6 1.13.7) (6b35-1.13.7-1ubuntu0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

The issue is , how to force Java and LibreOffice to use other cpu cores.
It's olways 1 core with 100% load, and rest 30 cores are doing nothing.
Overall system load is getting higher, while most cpu core are jobless.

The RAM usage is at 30-40%, so it's not RAM problem , disk usage is the same, in peak 10-15MB, so it's not an issue.


Does any know how to change this behaviour.


Regards
Adam

[afaust]

Hello,

you cannot force a Java process to use more cores than it needs to run its parallel processes / threads. If in the current state there aren't enough parallel executable threads alive within the Java process, you won't see full use of all cores.

What exactly did you do when observing those loads? Did you perform any kind of load testing, e.g. generating lots and lots of HTTP requests against Alfresco?
If you just observed startup without any real user load, than Alfresco just using one or two cores is to be expected - the startup is for the large part not parallizeable.

Please note that your Java version is way too old for Alfresco 5.0.d - Alfresco 5.0 requires at least Java 7 and 5.0.d requires at Java 8 due to an accidental error in the build, or you will face errors at runtime.

Regards
Axel

[adanecki]

Thank You for response.

After my last post we migrated Alfresco to another machine, as we discovered we had network card problem , it worked only in half duplex.

But still we encounter high machine load.

Our java version is 1.8, as it should be
root@alfresco:/opt/alfresco-5.0.d/java/bin# ./java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

Environment settings
JAVA_OPTS="-Xms8192M -Xmx50G -Xss1024k

System is the same as on previous machine. We have installed Alfresco 5.0.d Community Edition.
Our server is Ubuntu 12.04.2 LTS on Proxmox Virtualization Platform.

Server:
CPU : AMD Opteron™ Processor 6180 SE (48 core)
RAM: 64GB

Alfresco is the only VM on this Proxmox Server. There are no other VM.
For Alfresco we set 46 cpu cores, and 60GB of RAM

In attachments are screens showing server load.

Most of the time load is higher than 1.  Vmstat an iotop show that disk are idle most of the time, iowait are at 0 level.


There are 20-40 users using currently Alfresco. We have disabled openoffice, because conversions we very slow, and LibreOffice process in most cases get stuck during this process.

Our users are doing basic operation now, the just upload, download files, use edit offline/online options.

Should we consider that this machine is not enough for Alfresco.

We tried most of performance increase tips found on forums, but still load is much to high


Regards
Adam

[sscbrian]

What's the work type being done by those users (how many/how often are they viewing, uploading, etc)?  I have more users active on a LOT less machine (4 core, 6 GiB RAM, datastore on an iSCSI share, and Windows as the OS) and performance is great.

Large bulk imports will cause slowdowns as it's being indexed, but that tends to eat all the cores on my machine.

[adanecki]

Hello,

for us its very strange, what is going on on this machine.

System summary is :
Repository Information
Edition: Community
Version Number: 5.0.0 (d r99759-b2)
Version Label: d Alfresco version and build number.
Schema: 8 022 Alfresco database schema number.
Repository Identifier: 17e474c5-3846-4ced-bf0b-d280b6432b78 Unique identifier for this repository instance.
System Information
Java Home: /opt/alfresco-5.0.d/java
Java Version: 1.8.0_31
Java VM Vendor: Oracle Corporation
Operating System: Linux
Version: 2.6.32-37-pve
Architecture: amd64
Free Memory (GB): 28,33
Maximum Memory (GB): 49,75
Total Memory (GB): 31,77
CPUs: 46


Currently most of the time, there are 10 users.
They just login in, select their files, they don't even use search function ,just select their sites and files. Afrewards they use Offline Edit option or just download.
And after editing just upload new files.
Average file size is few MB, most Excel files (xls, xlsx). There are also files with size more than 20-30 MB, but most of them have few MB.

Is really hard for us to understand what is going on.
We have disabled LibreOffice 5 by setting "ooo.enabled=false" in alfresco-global.properties, to increase performance, by disabling documents to PDF conversion, but this didn't help allot.

Our storage is getting bigger , day by day, but it's not that huge as it could be.

16G     ./solr4
41G     ./solr4Backup
85G     ./contentstore

in our storage currently we have 90807 files.

I have found something strange in logs. It started few days ago in evening. Nothing was changed that day, but from, this moment server load high all time.

Line count in logs increased very much.
from last days:
173936 localhost_access_log.2015-09-29.txt
196212 localhost_access_log.2015-09-30.txt
550149 localhost_access_log.2015-10-01.txt
547092 localhost_access_log.2015-10-02.txt
534527 localhost_access_log.2015-10-03.txt
580342 localhost_access_log.2015-10-04.txt
600598 localhost_access_log.2015-10-05.txt
562335 localhost_access_log.2015-10-06.txt

Grep "solr" on log 'localhost_access_log.2015-10-06.txt' returns 534502 lines
562335 localhost_access_log.2015-10-06.txt

> grep -c www.alfresco.org localhost_access_log.2015-10-06.txt
397393 entries

127.0.0.1 - CN=Alfresco Repository Client, OU=Unknown, O=Alfresco Software Ltd., L=Maidenhead, ST=UK, C=GB [06/Oct/2015:00:08:15 +0200] "GET /alfresco/service/api/solr/textContent?nodeId=108885&propertyQName=%7bhttp%3a%2f%2fwww.alfresco.org%2fmodel%2fcontent%2f1.0%7dcontent HTTP/1.1" 200 243


> grep -c aclchangesets localhost_access_log.2015-10-06.txt
72222 entries

127.0.0.1 - CN=Alfresco Repository Client, OU=Unknown, O=Alfresco Software Ltd., L=Maidenhead, ST=UK, C=GB [06/Oct/2015:00:04:15 +0200] "GET /alfresco/service/api/solr/aclchangesets?fromTime=1443823276920&toTime=1443830476920&maxResults=2000 HTTP/1.1" 200 129

> grep -c transactions localhost_access_log.2015-10-06.txt
53303 entries

127.0.0.1 - CN=Alfresco Repository Client, OU=Unknown, O=Alfresco Software Ltd., L=Maidenhead, ST=UK, C=GB [06/Oct/2015:00:00:30 +0200] "GET /alfresco/service/api/solr/transactions?minTxnId=0&maxTxnId=2000&maxResults=1 HTTP/1.1" 200 249


> grep -c modelsdiff localhost_access_log.2015-10-06.txt
5760 entries

127.0.0.1 - CN=Alfresco Repository Client, OU=Unknown, O=Alfresco Software Ltd., L=Maidenhead, ST=UK, C=GB [06/Oct/2015:00:03:15 +0200] "POST /alfresco/service/api/solr/modelsdiff HTTP/1.1" 200 37

In log from days before 30.09 there entries witch www.alfresco.org were no more than 5000.
"Solr" entries number was increasing from 24.09 every day, but in some days it was decreasing, yet still very high.

> grep -ci solr localhost_access_log.2015-09-24.txt
68810 entries

> grep -ci solr localhost_access_log.2015-09-25.txt
145864 entries

> grep -ci solr localhost_access_log.2015-09-26.txt
185542 entries

> grep -ci solr localhost_access_log.2015-09-27.txt
217092 entries

> grep -ci solr localhost_access_log.2015-09-28.txt
194486 entries

> grep -ci solr localhost_access_log.2015-09-29.txt
148314 entries

> grep -ci solr localhost_access_log.2015-09-30.txt
163539 entries

I think 'solr' is the issue here. What could be than in this case. Any suggestios?

Regards
Adam

[mrogers]

Do you really have 60GB of physical memory and large numbers of cores.   For example could proxmox be giving you swap space rather than real memory?

SOLR can be quite heavy on CPU and IO and will generally use the resources it is given.   Why are you saying the CPU is too high?

[adanecki]

Hello,

yes, we have dedicated one server only for Alfresco. It's HP ProLiant DL585 G7.
This machine has 4x AMD Opteron™ Processor 6180 SE , each CPU has 12 cores
This servers has also 64GB of RAM.

4GB were reserved for base proxmox system, and rest 60GB are reserved for Ubuntu 12.04 running Alfresco 5.0.d.

On Ubuntu 50GB were reserved for JAVA machine, and rest 10GB for other procesess
Currently Alfreco virtual machine uses 40GB of RAM, 20GB are free.
Swap memory on Proxmox it's not used at all.

We have checked vmstat ant iotop, and I/O operations are quite small. Proxmox have buffered many operations in RAM.

–
Regards
Adam

[adanecki]

hello,

does any one know what could cause low performance of our Alfresco.We have very strong machine , compare to number of users and documents in storage, and yeat performance is poor.
All the time avg cpu load is 2, with peaks up to 8 or more.

Memory usage is at 50%, I/O load is small.

Could anyone help ?

–
Regards
Adam

[ksh]

old topic, but just got the same Problem with 32 cores on latest version 201701.

soffice.bin got stuck converting using 1 core (99%) and the upload runs into the timeout. seems like the socket is blocked.

[afaust]

That soffice process being stuck on 1 core is actually not a bug / issue. The default OOoDirect subsystem that connects to OpenOffice / LibreOffice for conversion can only support one transformation at a time - the Enterprise Edition of Alfresco provides an alternative JODConverter subsystem that can handle multiple transformations at a time. So for Community Edition soffice will always only use at most one core because there is just one single transformation in a single process.

Unfortunately, soffice has a habit of choking on complex MS Office documents leading to constant 100% core usage. This is a know issue and the reason why Enterprise Edition JODConverter has a timeout and active process restart handling. On Community Edition, you have to use some kind of watchdog to monitor the soffice process and restart it yourself when it gets stuck. There are various post on this topic in the Alfresco community, e.g. from Zylk or Contentreich. I have recently created my own, slightly more elaborate watchdog as a small Java library for a customer that performs the same kind of transformations as Alfresco would to test process viability.