Error on replicate data from a alfrescoA to a alfrescoB

[larede]

I followed the following document
https://docs.alfresco.com/content-services/community/admin/replication/

version: "3.8"

services:
  alfrescoA:
    image: alfresco/alfresco-content-repository-community:7.0.0
    restart: on-failure:10
    environment:
      JAVA_TOOL_OPTIONS: "
        -Dencryption.keystore.type=JCEKS
        -Dencryption.cipherAlgorithm=DESede/CBC/PKCS5Padding
        -Dencryption.keyAlgorithm=DESede
        -Dencryption.keystore.location=/usr/local/tomcat/shared/classes/alfresco/extension/keystore/keystore
        -Dmetadata-keystore.password=mp6yc0UD9e
        -Dmetadata-keystore.aliases=metadata
        -Dmetadata-keystore.metadata.password=oKIWzVdEdA
        -Dmetadata-keystore.metadata.algorithm=DESede
        "
      JAVA_OPTS: "
        -Ddb.driver=org.postgresql.Driver
        -Ddb.username=alfrescoA
        -Ddb.password=alfrescoA
        -Ddb.url=jdbc:postgresql://postgresA:5432/alfrescoA
        -Dsolr.host=solr6A
        -Dsolr.port=8983
        -Dsolr.http.connection.timeout=1000
        -Dsolr.secureComms=none
        -Dsolr.base.url=/solr
        -Dindex.subsystem.name=solr6
        -Dooo.enabled=false
        -Djodconverter.enabled=false
        -Dmessaging.broker.url=\"failover:(nio://activemqA:61616)?timeout=3000&jms.useCompression=true\"
        -Ddeployment.method=DOCKER_COMPOSE
        -Dtransform.service.enabled=false
        -Dlocal.transform.service.enabled=false
        -Dcsrf.filter.enabled=false
        -Dreplication.enabled=true
        -Dreplication.transfer.readonly=false
        "
    ports:
      - 8080:8080
    volumes:
      - alfrescoA_data:/usr/local/tomcat/alf_data

  solr6A:
    image: alfresco/alfresco-search-services:2.0.1
    restart: on-failure:10
    healthcheck:
      test: curl -s http://alfrescoA:8080/alfresco/api/-default-/public/cmis/versions/1.1/browser | grep -c -e '401' || exit 1
      interval: 30s
      timeout: 10s
      retries: 30
    environment:
      - SOLR_ALFRESCO_HOST=alfrescoA
      - SOLR_ALFRESCO_PORT=8080
      - SOLR_SOLR_HOST=localhost
      - SOLR_SOLR_PORT=8983
      - SOLR_CREATE_ALFRESCO_DEFAULTS=alfresco,archive
      - ALFRESCO_SECURE_COMMS=none

  activemqA:
    image: alfresco/alfresco-activemq:5.16.1
        
  shareA:
    image: alfresco/alfresco-share:7.0.0
    restart: on-failure:10
    environment:
      REPO_HOST: alfrescoA
      REPO_PORT: "8080"
    ports:
      - 8081:8080

  postgresA:
    image: postgres:13.1
    restart: on-failure:10
    environment:
      - POSTGRES_PASSWORD=alfrescoA
      - POSTGRES_USER=alfrescoA
      - POSTGRES_DB=alfrescoA
    command: postgres -c max_connections=300
    volumes:
      - postgresA_data:/var/lib/postgresql/data

  alfrescoB:
    image: alfresco/alfresco-content-repository-community:7.0.0
    restart: on-failure:10
    environment:
      JAVA_TOOL_OPTIONS: "
        -Dencryption.keystore.type=JCEKS
        -Dencryption.cipherAlgorithm=DESede/CBC/PKCS5Padding
        -Dencryption.keyAlgorithm=DESede
        -Dencryption.keystore.location=/usr/local/tomcat/shared/classes/alfresco/extension/keystore/keystore
        -Dmetadata-keystore.password=mp6yc0UD9e
        -Dmetadata-keystore.aliases=metadata
        -Dmetadata-keystore.metadata.password=oKIWzVdEdA
        -Dmetadata-keystore.metadata.algorithm=DESede
        "
      JAVA_OPTS: "
        -Ddb.driver=org.postgresql.Driver
        -Ddb.username=alfrescoB
        -Ddb.password=alfrescoB
        -Ddb.url=jdbc:postgresql://postgresB:5432/alfrescoB
        -Dooo.enabled=false
        -Djodconverter.enabled=false
        -Dmessaging.broker.url=\"failover:(nio://activemqB:61616)?timeout=3000&jms.useCompression=true\"
        -Ddeployment.method=DOCKER_COMPOSE
        -Dtransform.service.enabled=false
        -Dlocal.transform.service.enabled=false
        -Dreplication.enabled=true
        -Dreplication.transfer.readonly=false
        -Dtransferservice.receiver.enabled=true
        "
    ports:
      - 8082:8080
    volumes:
      - alfrescoB_data:/usr/local/tomcat/alf_data

  activemqB:
    image: alfresco/alfresco-activemq:5.16.1

  shareB:
    image: alfresco/alfresco-share:7.0.0
    restart: on-failure:10
    ports:
      - 8083:8080
    environment:
      - REPO_HOST=alfrescoB
      - REPO_PORT=8080

  postgresB:
    image: postgres:13.1
    restart: on-failure:10
    environment:
      - POSTGRES_PASSWORD=alfrescoB
      - POSTGRES_USER=alfrescoB
      - POSTGRES_DB=alfrescoB
    command: postgres -c max_connections=300
    volumes:
      - postgresB_data:/var/lib/postgresql/data

volumes:
  alfrescoA_data:
    driver: local
  postgresA_data:
    driver: local
  alfrescoB_data:
    driver: local
  postgresB_data:
    driver: local

When replicating directories with few files everything works fine.

When I try to replicate thousands of files, an error occurs in alfrescoB after +/- 10 minutes of running . Error log:

alfrescoB_1 | 2021-08-31 15:56:42,388 ERROR [extensions.webscripts.AbstractRuntime] [http-nio-8080-exec-3] Exception from executeScript: null
alfrescoB_1 | java.lang.NullPointerException
alfrescoB_1 | at org.alfresco.repo.web.scripts.TempOutputStream.getInputStream(TempOutputStream.java:166)
alfrescoB_1 | at org.alfresco.repo.web.scripts.BufferedResponse.writeResponse(BufferedResponse.java:268)
alfrescoB_1 | at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:614)
alfrescoB_1 | at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:664)
alfrescoB_1 | at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:435)
alfrescoB_1 | at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:315)
alfrescoB_1 | at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:399)
alfrescoB_1 | at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
alfrescoB_1 | at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
alfrescoB_1 | at org.alfresco.repo.web.scripts.AlfrescoWebScriptServlet.service(AlfrescoWebScriptServlet.java:43)
alfrescoB_1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
alfrescoB_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
alfrescoB_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
alfrescoB_1 | at jdk.internal.reflect.GeneratedMethodAccessor434.invoke(Unknown Source)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.alfresco.module.aosmodule.service.ContextRootFilter.doFilter(ContextRootFilter.java:93)
alfrescoB_1 | at jdk.internal.reflect.GeneratedMethodAccessor434.invoke(Unknown Source)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.springframework.extensions.webscripts.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:177)
alfrescoB_1 | at jdk.internal.reflect.GeneratedMethodAccessor434.invoke(Unknown Source)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:68)
alfrescoB_1 | at jdk.internal.reflect.GeneratedMethodAccessor434.invoke(Unknown Source)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.alfresco.web.app.servlet.ClearSecurityContextFilter.doFilter(ClearSecurityContextFilter.java:53)
alfrescoB_1 | at jdk.internal.reflect.GeneratedMethodAccessor434.invoke(Unknown Source)
alfrescoB_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
alfrescoB_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
alfrescoB_1 | at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
alfrescoB_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
alfrescoB_1 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
alfrescoB_1 | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
alfrescoB_1 | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
alfrescoB_1 | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
alfrescoB_1 | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
alfrescoB_1 | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
alfrescoB_1 | at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
alfrescoB_1 | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
alfrescoB_1 | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
alfrescoB_1 | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
alfrescoB_1 | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
alfrescoB_1 | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
alfrescoB_1 | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
alfrescoB_1 | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
alfrescoB_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
alfrescoB_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
alfrescoB_1 | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
alfrescoB_1 | at java.base/java.lang.Thread.run(Thread.java:834)

Can you help please?

[abhinavmishra14]

The error seems to be related to temp space, try adding more space for temp folder.

Also enable following debug logs and see what you get:

log4j.logger.org.alfresco.repo.web.scripts.TempOutputStream=DEBUG
log4j.logger.org.alfresco.repo.web.scripts.BufferedResponse=DEBUG
log4j.logger.org.alfresco.repo.web.scripts.RepositoryContainer=DEBUG
log4j.logger.org.alfresco.repo.web.scripts.AlfrescoWebScriptServlet=DEBUG

OR

log4j.logger.org.alfresco.repo.web.scripts=DEBUG

Additionally, you can setup https://github.com/Alfresco/alfresco-community-repo/blob/master/core/src/main/java/org/alfresco/util... 

https://github.com/Alfresco/alfresco-community-repo/blob/master/repository/src/main/resources/alfres...

To cleanup temp space more frequently if that helps. or Use manually invoke temp cleaner job manually from support tools.

[larede]

Hi @abhinavmishra14, I don't have disk problem.
Is it possible to be a configuration / setting? Which property?

PS: I set the log level = debug but I didn't get another error

Thank you for your help!

[venur]

Documentation tells to enable this on nodes

transferservice.receiver.enabled=true

i see you have this set on B node, try setting on both. Not sure if it makes any difference but just a try

[larede]

Hello @venur! Thanks for your reply.
The documentation only requires putting transferservice.receiver.enabled=true in the target alfresco.

However, I followed his recommendation and also tested the setting at alfrescoA. The result was the same .

I believe that @abhinavmishra14 is right! It appears to be a temporary directory space problem ..

I've read and reread the documentation and I can't find any settings to change this behavior.

[larede · Accepted solution]

You can see the research I did and documented
https://github.com/larede/alfresco-transfer-data

The error has been resolved with following properties on alfrescoB

        -Dtransferservice.receiver.lockRefreshTime=60000
        -Dtransferservice.receiver.lockRetryCount=3
        -Dtransferservice.receiver.lockRetryWait=100
        -Dtransferservice.receiver.lockTimeOut=30000000
        -Dwebscripts.encryptTempFiles=false
        -Dwebscripts.memoryThreshold=536870912
        -Dwebscripts.setMaxContentSize=536870912000

[abhinavmishra14]

---

@larede wrote:

You can see the research I did and documented
https://github.com/larede/alfresco-transfer-data

The error has been resolved with following properties on alfrescoB

        -Dtransferservice.receiver.lockRefreshTime=60000
        -Dtransferservice.receiver.lockRetryCount=3
        -Dtransferservice.receiver.lockRetryWait=100
        -Dtransferservice.receiver.lockTimeOut=30000000
        -Dwebscripts.encryptTempFiles=false
        -Dwebscripts.memoryThreshold=536870912
        -Dwebscripts.setMaxContentSize=536870912000

---

Thanks for sharing @larede , glad you figured out the issue.