GC overhead limit exceeded/OutOfMemoryError: Java heap space

[Aleksandar]

 Hi, from the past week we have had issues with the APS application. The application crashes multiple times per day and we see "GC overhead limit exceeded/OutOfMemoryError: Java heap space" in their logs. In the beginning, we increased the ram (used by java) from 8GB to 10GB and it helped for a while. Now we use 11GB and we set the GCTimeRatio=80 with parameter in "activiti-app.properties" "datasource.statementCacheNumDeferredCloseThreads=1" and the main ram is 16GB.

`

04:45:48 [C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@29ab6ad8 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
04:52:04 [C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@29ab6ad8 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 2
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@60309a96
on thread: C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@105b28bb
on thread: C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@77eb51c7
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-HelperThread-#2,5,main]
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1417)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-HelperThread-#1,5,main]
com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask.run(BasicResourcePool.java:1417)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-HelperThread-#0,5,main]
java.base@11.0.11/java.lang.Object.wait(Native Method)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

Exception in thread "http-nio-8080-Acceptor-0" Exception in thread "http-nio-8080-exec-8" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
05:04:50 [C3P0PooledConnectionPoolManager[identityToken->2stslqaogzz0k75a43c6|786a9883]-AdminTaskTimer] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner - Task com.mchange.v2.resourcepool.BasicResourcePool$1RefurbishCheckinResourceTask@105b28bb (in deadlocked PoolThread) failed to complete in maximum time 60000ms. Trying interrupt().
```````````````````````````````````````````````

12:10:28 [pool-2-thread-5] ERROR org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at jdk.internal.reflect.GeneratedConstructorAccessor239.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:124)
at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:136)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.instantiate(AbstractEntityTuplizer.java:730)
at org.hibernate.persister.entity.AbstractEntityPersister.instantiate(AbstractEntityPersister.java:4565)
at org.hibernate.internal.SessionImpl.instantiate(SessionImpl.java:1313)
at org.hibernate.internal.SessionImpl.instantiate(SessionImpl.java:1302)
at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1618)
at org.hibernate.loader.Loader.getRow(Loader.java:1509)
at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:719)
at org.hibernate.loader.Loader.processResultSet(Loader.java:949)
at org.hibernate.loader.Loader.doQuery(Loader.java:917)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:348)
at org.hibernate.loader.Loader.doList(Loader.java:2550)
at org.hibernate.loader.Loader.doList(Loader.java:2536)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2366)
at org.hibernate.loader.Loader.list(Loader.java:2361)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1194)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268)
at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:125)
at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:89)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:128)
at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:118)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:494)
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:477)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
12:10:31 [http-nio-8080-exec-9] WARN org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver - Resolved [org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded]
12:10:52 [pool-2-thread-3] WARN com.activiti.service.runtime.ActivitiEventProcessingService - No elastic search client configured, yet configured to process events. This is not possible, no events have processed.

```````````````````````````````

Exception in thread "http-nio-8080-exec-4" Exception in thread "http-nio-8080-Acceptor-0" Exception in thread "ajp-nio-8009-ClientPoller-0" Exception in thread "http-nio-8080-exec-1" Exception in thread "ajp-nio-8009-ClientPoller-1" Exception in thread "http-nio-8080-exec-8" Exception in thread "http-nio-8080-exec-10" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
12:47:01 [activiti-app-rest-Executor-1] INFO com.activiti.service.runtime.integration.alfresco.AlfrescoOnPremiseTicketService - Alfresco ticket cache statistics: CacheStats{hitCount=0, missCount=0, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}

```````````````````````````````

19-Apr-2022 03:47:01.880 SEVERE [http-nio-8080-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint$Poller.events
java.lang.OutOfMemoryError: Java heap space

19-Apr-2022 03:47:01.880 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
java.lang.OutOfMemoryError: Java heap space

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space

```````````````````````````````

01:46:39 [pool-2-thread-5] ERROR org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.OutOfMemoryError: GC overhead limit exceeded
01:46:44 [Thread-11] ERROR org.activiti.engine.impl.interceptor.CommandContext - Error while closing command context
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
### The error may exist in org/activiti/db/mapping/entity/Job.xml
### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.selectNextTimerJobsToExecute-Inline
### The error occurred while setting parameters
### SQL: select RES.* from ACT_RU_JOB RES LEFT OUTER JOIN ACT_RU_EXECUTION PI ON PI.ID_ = RES.PROCESS_INSTANCE_ID_ where (RES.RETRIES_ > 0) and (RES.DUEDATE_ is null or RES.DUEDATE_ <= ?) and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ <= ?) and TYPE_ = 'timer' and ( (RES.EXECUTION_ID_ is null) or (PI.SUSPENSION_STATE_ = 1) ) LIMIT ? OFFSET ?
### Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:122)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113)
at org.activiti.engine.impl.db.DbSqlSession.selectListWithRawParameter(DbSqlSession.java:442)
at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:433)
at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:428)
at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:411)
at org.activiti.engine.impl.persistence.entity.JobEntityManager.findNextTimerJobsToExecute(JobEntityManager.java:157)
at org.activiti.engine.impl.cmd.AcquireTimerJobsCmd.execute(AcquireTimerJobsCmd.java:45)
at org.activiti.engine.impl.cmd.AcquireTimerJobsCmd.execute(AcquireTimerJobsCmd.java:29)
at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.AcquireTimerJobsRunnable.run(AcquireTimerJobsRunnable.java:52)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:335)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:132)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.execute(NewProxyPreparedStatement.java:67)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:62)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:78)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120)
... 18 common frames omitted
Caused by: javax.net.ssl.SSLException: Tag mismatch!
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:122)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1386)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:973)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:306)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952)
at org.postgresql.core.v3.QueryExecutorImpl.flushIfDeadlockRisk(QueryExecutorImpl.java:1299)
at org.postgresql.core.v3.QueryExecutorImpl.sendQuery(QueryExecutorImpl.java:1324)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
... 31 common frames omitted

`