Getting "Node does not exist" when creating new version via CMIS concurrently.

[dodjavola90]

Hello everyone. I am currently working on upgrading Alfresco from 5.0.d to 25.1. My upgrade path was following: 5.0 -> 5.2 -> 7.4 -> 25.1. Alfresco utilizes MariaDB 10.5 underneath. While the upgrade was mostly straightforward and pain free, I ran into a weird issue. I have a CMIS based script that keeps some files up to date (1k+ files) and script will basically go file by file, find current one in Alfresco via CMIS query, then upload new version via check-out, check-in. This was working great on 5.0.d. However, on 25.1 it's randomly breaking when I run the script with 3+ threads. Script is getting 'Object Info Missing' from the CMIS client and the catalina.out shows following stack trace:

org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/f2bddaef-a344-412a-bdda-efa344712a7c (status:Status[id=517226changeTxnId=8134b6bb-1846-4d54-b4b6-bb18463d542e, dbTxnId=251500, deleted=true])
	at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:187)
	at org.alfresco.repo.node.db.DbNodeServiceImpl.hasAspect_aroundBody38(DbNodeServiceImpl.java:1007)
	at org.alfresco.repo.node.db.DbNodeServiceImpl$AjcClosure39.run(DbNodeServiceImpl.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:164)
	at org.alfresco.traitextender.RouteExtensions.intercept(RouteExtensions.java:100)
	at org.alfresco.repo.node.db.DbNodeServiceImpl.hasAspect(DbNodeServiceImpl.java:1001)
	at jdk.internal.reflect.GeneratedMethodAccessor261.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.alfresco.repo.lock.mem.LockableAspectInterceptor.invoke(LockableAspectInterceptor.java:129)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy42.hasAspect(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor261.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:111)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy42.hasAspect(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor261.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:231)
	at jdk.proxy2/jdk.proxy2.$Proxy57.hasAspect(Unknown Source)
	at org.alfresco.repo.node.MLPropertyInterceptor.getPivotNodeRef(MLPropertyInterceptor.java:329)
	at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:179)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:219)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy42.getProperties(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:216)
	at jdk.proxy2/jdk.proxy2.$Proxy42.getProperties(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.alfresco.repo.node.ContentPropertyRestrictionInterceptor.invoke(ContentPropertyRestrictionInterceptor.java:207)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.audit.DisableAuditableBehaviourInterceptor.invoke(DisableAuditableBehaviourInterceptor.java:120)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at net.sf.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:80)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.security.permissions.impl.ExceptionTranslatorMethodInterceptor.invoke(ExceptionTranslatorMethodInterceptor.java:53)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.audit.AuditMethodInterceptor.invoke(AuditMethodInterceptor.java:166)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.transaction.RetryingTransactionInterceptor$1.execute(RetryingTransactionInterceptor.java:95)
	at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:452)
	at org.alfresco.repo.transaction.RetryingTransactionInterceptor.invoke(RetryingTransactionInterceptor.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy42.getProperties(Unknown Source)
	at org.alfresco.opencmis.CMISNodeInfoImpl.getNodeProps(CMISNodeInfoImpl.java:1117)
	at org.alfresco.opencmis.mapping.DirectProperty.getValueInternal(DirectProperty.java:72)
	at org.alfresco.opencmis.mapping.AbstractProperty.getValue(AbstractProperty.java:124)
	at org.alfresco.opencmis.CMISConnector.getNodeProperties(CMISConnector.java:1952)
	at org.alfresco.opencmis.CMISConnector.createCMISObject(CMISConnector.java:1553)
	at org.alfresco.opencmis.AlfrescoCmisServiceImpl.getObjectInfo(AlfrescoCmisServiceImpl.java:2848)
	at org.alfresco.opencmis.AlfrescoCmisServiceImpl.getObjectInfo(AlfrescoCmisServiceImpl.java:2810)
	at jdk.internal.reflect.GeneratedMethodAccessor555.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.alfresco.opencmis.CMISTransactionAwareHolderInterceptor.invoke(CMISTransactionAwareHolderInterceptor.java:54)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.repo.transaction.RetryingTransactionInterceptor$1.execute(RetryingTransactionInterceptor.java:95)
	at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:452)
	at org.alfresco.repo.transaction.RetryingTransactionInterceptor.invoke(RetryingTransactionInterceptor.java:85)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.opencmis.AlfrescoCmisStreamInterceptor.invoke(AlfrescoCmisStreamInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.opencmis.AlfrescoCmisServiceInterceptor.invoke(AlfrescoCmisServiceInterceptor.java:108)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.alfresco.opencmis.AlfrescoCmisExceptionInterceptor.invoke(AlfrescoCmisExceptionInterceptor.java:90)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy373.getObjectInfo(Unknown Source)
	at lib3party.org.apache.chemistry.opencmis.server.support.wrapper.AbstractCmisServiceWrapper.getObjectInfo(AbstractCmisServiceWrapper.java:478)
	at org.apache.chemistry.opencmis.server.impl.atompub.ObjectService$UpdateProperties.serve(ObjectService.java:750)
	at org.apache.chemistry.opencmis.server.shared.Dispatcher.dispatch(Dispatcher.java:92)
	at org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.dispatch(CmisAtomPubServlet.java:295)
	at org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.service(CmisAtomPubServlet.java:206)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.alfresco.opencmis.CMISServletDispatcher.execute(CMISServletDispatcher.java:238)
	at org.alfresco.opencmis.CMISWebScript.execute(CMISWebScript.java:58)
	at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:506)
	at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:685)
	at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:721)
	at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:413)
	at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:309)
	at org.alfresco.rest.api.PublicApiRepositoryContainer.access$001(PublicApiRepositoryContainer.java:47)
	at org.alfresco.rest.api.PublicApiRepositoryContainer$1.doWork(PublicApiRepositoryContainer.java:84)
	at org.alfresco.repo.tenant.TenantUtil.runAsWork(TenantUtil.java:126)
	at org.alfresco.repo.tenant.TenantUtil.runAsTenant(TenantUtil.java:95)
	at org.alfresco.rest.api.PublicApiRepositoryContainer.executeScript(PublicApiRepositoryContainer.java:80)
	at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:423)
	at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
	at org.alfresco.repo.web.scripts.TenantWebScriptServlet.service(TenantWebScriptServlet.java:82)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.springframework.extensions.webscripts.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:177)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.alfresco.repo.web.filter.beans.NullFilter.doFilter(NullFilter.java:75)
	at jdk.internal.reflect.GeneratedMethodAccessor532.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.alfresco.repo.management.subsystems.ChainingSubsystemProxyFactory$1.invoke(ChainingSubsystemProxyFactory.java:132)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
	at jdk.proxy2/jdk.proxy2.$Proxy164.doFilter(Unknown Source)
	at org.alfresco.repo.web.filter.beans.BeanProxyFilter.doFilter(BeanProxyFilter.java:89)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:68)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.alfresco.web.app.servlet.ClearSecurityContextFilter.doFilter(ClearSecurityContextFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:666)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
	at java.base/java.lang.Thread.run(Thread.java:840)

If I run with a single thread or 2 threads MAX, there are no errors. Anything more than that and I get random errors. Sometimes just couple of documents, sometime dozens. I've tried following so far:

• Upgrading MariaDB to 10.6
• Upgrading CMIS client to the latest version (Apache Chemistry CMIS)
• Trying both MariaDB and MySQL JDBC driver

None of these helped. If I move away from CMIS and call custom webscript that I have in place in charge of document update, the problem goes away. Unfortunately, CMIS is still heavily being used in my system so I have to figure this problem out before I can fully proceed with upgrade. 

I hope someone can help me here. Please let me know if I can provide any more info. 

Thanks in advance!