PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

heiko_robert — Sun, 08 Aug 2021 16:09:32 GMT

On a Alfresco 7.0 CE repo successfully migrated from 5.2 with a high number of nodes I now get every morning around 4am a database connection exception from org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl which then breaks alfresco. Alfresco is configured against a postres 13 DB running on a separate server. I haven't seen such an issue before.

Does anybody have a hint, idea?

Alfresco is configured with db.pool.max=350 (postgres: max_connections = 350) and there is only one application user connected the system. As a work around I now deactivated system.workflow.engine.activiti but would like understand the issue

2021-08-08 04:35:39,396 ERROR [org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl] [Thread-46] exception during job acquisition: Could not open JDBC Connection for transaction; nested exception is org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
        at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
        at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl.run(AcquireJobsRunnableImpl.java:54)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:43)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
        at org.postgresql.Driver.makeConnection(Driver.java:465)
        at org.postgresql.Driver.connect(Driver.java:264)
        at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
        ... 9 more
Caused by: javax.net.ssl.SSLException: readHandshakeRecord
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1335)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)
        ... 22 more
        Suppressed: java.net.SocketException: Broken pipe (Write failed)
                at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
                at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
                at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:380)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
                at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
                ... 24 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
        at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
        at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251)
        at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
        at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:679)
        at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:658)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
        at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Finished.java:1011)
        at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Finished.java:874)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
        ... 25 more
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.postgresql.util.PSQLException: SSL error: Remote host terminated the handshake
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
        at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
        at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl.run(AcquireJobsRunnableImpl.java:54)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: SSL error: Remote host terminated the handshake
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:43)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
        at org.postgresql.Driver.makeConnection(Driver.java:465)
        at org.postgresql.Driver.connect(Driver.java:264)
        at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
        ... 9 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
        at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1598)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1426)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)
        ... 22 more
        Suppressed: java.net.SocketException: Broken pipe (Write failed)
                at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
                at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
                at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:380)
                at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
                at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
                ... 24 more
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
        at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
        at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
        ... 26 more

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

heiko_robert — Mon, 09 Aug 2021 07:54:53 GMT

Unfortunately, this post has meanwhile been marked as spam and has been temporarily removed. Therefore, as a work around, I posted the same question at stack overfow

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

cristinamr — Mon, 09 Aug 2021 09:53:15 GMT

Hi @heiko_robert !

Caused by: javax.net.ssl.SSLException: readHandshakeRecord

It's the first time I've seen this error too but doing some research looks you have a problem with your server certificates. Take a look in this and this post I think they are same issue that you are experiencing.

Let us know your tests.

Cheers,

Cris.

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

heiko_robert — Fri, 13 Aug 2021 14:43:05 GMT

The stacktrace was misleading - the jdbc connection problem was caused by memory filling up by the Alfresco trashcan cleaner module: OutOfMemoryError: Java heap space due to no limit on getChildAssocs.

We had ~ 4 million nodes in the trashcan and the module retrieves in every batch run all nodes again and again until the memory has been filled up ...

topic Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl in Alfresco Forum

PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl

Re: PSQLException: SSL error: readHandshakeRecord for AcquireJobsRunnableImpl