Transform service - unlimited database connections

upforsin — Mon, 28 Nov 2022 15:10:17 GMT

Recently Alfresco Transofrm Service run out of memory (OutOfMemoryError, mXm parameter was too small) but it initiated lots (casually there are ~40connections, here it jumped in minutes to >200) of database connections (with "idle-transaction" state) which after a while resulted in not working database which resuled in not working Alfresco Repository and Share.

Is it a bug? What can i do to limit database connections? In the the Alfresco repository there is a db.pool.max parameter, is there something similar in ats?

In the logs I noticed something about "unlimited reties".

I'm running Alfresco Community Edition 7.2 with alfresco-transform-core-aio-boot-2.5.7.jar

Re: Transform service - unlimited database connections

angelborroy — Mon, 28 Nov 2022 15:20:16 GMT

I'm using this configuration to limit the number of threads in Transform Service:

https://github.com/Alfresco/alfresco-docker-installer/blob/master/generators/app/templates/7.3/docker-compose.yml#L126

Re: Transform service - unlimited database connections

upforsin — Tue, 29 Nov 2022 10:46:23 GMT

@angelborroy Thank you, I added "-Dserver.tomcat.threads.max=12 -Dserver.tomcat.threads.min=4" to JAVA_OPTS, the time will tell if it helped.

To set up ATS I relied on alfresco-ansible-deployment, It might be wise to update that config too.

I faced this issue 3 times (even after allocating over 3GB RAM to ats), each time the last processed file was xlsx, and ats run out of memory on the "Scheduled task". Maybe there is a bug somewhere?

ats log file:

2022-11-27 10:30:04.018  INFO 1234 --- [nio-8090-exec-6] org.alfresco.transformer.AIOController   : GET Transform Config version: 2
2022-11-27 10:30:04.018  INFO 1234 --- [nio-8090-exec-4] org.alfresco.transformer.AIOController   : GET Transform Config version: 2
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180              xlsx txt   42.4 MB TikaAuto
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180                targetExtension="txt"
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180                sourceEncoding="UTF-8"
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180                sourceMimetype="application/vnd.openxmlformats-officedocument.spread...sheet"
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180                sourceExtension="xlsx"
2022-11-27 10:32:26.962 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180                targetMimetype="text/plain"
2022-11-27 10:34:38.378  INFO 1234 --- [ntContainer#0-1] o.apache.activemq.util.ThreadPoolUtils   : Waited 2.353 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@7a61b8a9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 57247] to terminate...
2022-11-27 10:35:45.677  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616
2022-11-27 10:37:09.579  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616
2022-11-27 10:39:22.679 ERROR 1234 --- [ ReadCheckTimer] o.a.activemq.thread.SchedulerTimerTask   : Scheduled task error

java.lang.OutOfMemoryError: Java heap space

2022-11-27 10:42:05.107  WARN 1234 --- [ntContainer#0-1] o.s.j.l.DefaultMessageListenerContainer  : Setup of JMS message listener invoker failed for destination 'org.alfresco.transform.engine.aio.acs' - trying to recover. Cause: Could not create JMS transaction; nested exception is java.lang.OutOfMemoryError: Java heap space
2022-11-27 10:42:05.109 DEBUG 1234 --- [nio-8090-exec-2] o.a.transform.router.TransformerDebug    : e180              Java heap space
2022-11-27 10:42:05.117  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616
2022-11-27 10:42:05.120  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616
2022-11-27 10:42:05.161 ERROR 1234 --- [nio-8090-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause

java.lang.OutOfMemoryError: Java heap space

2022-11-27 10:42:06.149  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616
2022-11-27 10:42:07.163  INFO 1234 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport       : Successfully connected to tcp://127.0.0.1:61616

Re: Transform service - unlimited database connections

VS — Thu, 12 Jan 2023 22:58:44 GMT

@angelborroy wrote:
I'm using this configuration to limit the number of threads in Transform Service:
https://github.com/Alfresco/alfresco-docker-installer/blob/master/generators/app/templates/7.3/docker-compose.yml#L126

Are you starting up each component on seperate host? If any of the applications using "-XX:MinRAMPercentage=50 -XX:MaxRAMPercentage=80" are on same host then we will need to reduce the numbers Otherwise heapsizes will become more than the actual memory on the host. Does adding a mem_limit takes care of this issue and caps the memory limit for each application?

topic Re: Transform service - unlimited database connections in Alfresco Forum

Transform service - unlimited database connections

Re: Transform service - unlimited database connections

Re: Transform service - unlimited database connections

Re: Transform service - unlimited database connections