Unable to import large amounts of files into alfresco.

[m_mind]

We currently have about 32G of files which we currently have on a windows share. As we want to be able to index and search for files, we want to add these files into Alfresco.
However, when we copy the files to the CIFS interface or to the FTP server, after some time errors occur and the connection is closed.
We need to restart Alfresco in order to be able to use the Fileserver interface again.

The error I get when importing through the CIFS interface is:

09:18:41,168 ERROR [org.alfresco.util.transaction.SpringAwareUserTransaction.trace] UserTransaction being garbage collected without a commit() or rollback().
   Started at:
      org.alfresco.util.transaction.SpringAwareUserTransaction.begin(SpringAwareUserTransaction.java:389)
      org.alfresco.filesys.alfresco.AlfrescoDiskDriver.beginTransaction(AlfrescoDiskDriver.java:332)
      org.alfresco.filesys.alfresco.AlfrescoDiskDriver.beginWriteTransaction(AlfrescoDiskDriver.java:180)
      org.alfresco.filesys.repo.ContentDiskDriver.closeFile(ContentDiskDriver.java:1822)
      org.alfresco.jlan.smb.server.NTProtocolHandler.procCloseFile(NTProtocolHandler.java:1386)
      org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:251)
      org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1344)
      org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1436)
      org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:105)
      org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
      java.lang.Thread.run(Thread.java:619)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

but also a longer exception:

11:25:53,320 ERROR [org.hibernate.AssertionFailure] an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use o
f the session)
org.hibernate.AssertionFailure: null id in org.alfresco.repo.domain.hibernate.ChildAssocImpl entry (don't flush the Session after an exception occurs)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.checkId(DefaultFlushEntityEventListener.java:55)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:164)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:120)
        at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
        at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:26)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.flushSession(DirtySessionMethodInterceptor.java:317)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.flushSession(DirtySessionMethodInterceptor.java:282)
        at org.alfresco.repo.node.db.hibernate.HibernateNodeDaoServiceImpl.newChildAssoc(HibernateNodeDaoServiceImpl.java:1588)
        at sun.reflect.GeneratedMethodAccessor659.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.transaction.TransactionalDaoInterceptor.invoke(TransactionalDaoInterceptor.java:68)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.invoke(DirtySessionMethodInterceptor.java:381)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy2.newChildAssoc(Unknown Source)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.createNode(DbNodeServiceImpl.java:319)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.transaction.SingleEntryTransactionResourceInterceptor.invokeInternal(SingleEntryTransactionResourceInterceptor.java:169)
        at org.alfresco.repo.transaction.SingleEntryTransactionResourceInterceptor.invoke(SingleEntryTransactionResourceInterceptor.java:138)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:221)
        at $Proxy8.createNode(Unknown Source)
        at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:279)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:188)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:188)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at org.alfresco.repo.model.filefolder.FileFolderServiceImpl.create(FileFolderServiceImpl.java:729)
        at sun.reflect.GeneratedMethodAccessor688.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.model.filefolder.TempFileMarkerInterceptor.invoke(TempFileMarkerInterceptor.java:83)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.model.ml.MLContentInterceptor.invoke(MLContentInterceptor.java:131)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.model.filefolder.MLTranslationInterceptor.invoke(MLTranslationInterceptor.java:210)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at net.sf.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:80)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.security.permissions.impl.ExceptionTranslatorMethodInterceptor.invoke(ExceptionTranslatorMethodInterceptor.java:49)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.audit.AuditComponentImpl.audit(AuditComponentImpl.java:275)
        at org.alfresco.repo.audit.AuditMethodInterceptor.invoke(AuditMethodInterceptor.java:69)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy53.create(Unknown Source)
        at org.alfresco.filesys.repo.CifsHelper.createNode(CifsHelper.java:367)
        at org.alfresco.filesys.repo.ContentDiskDriver.createDirectory(ContentDiskDriver.java:1656)
        at org.alfresco.jlan.smb.server.CoreProtocolHandler.procCreateDirectory(CoreProtocolHandler.java:479)
        at org.alfresco.jlan.smb.server.CoreProtocolHandler.runProtocol(CoreProtocolHandler.java:3709)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:359)
        at org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1344)
        at org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1436)
        at org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:105)
        at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
        at java.lang.Thread.run(Thread.java:619)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I've also tried to create a relatively small ZIP file (about 160MB) in order to import the files like this, but that also doesn't work.
Is there a way to import large amounts of files without crashing Alfresco?

Thanks in advance.
Peter

[treghy]

Hi I've the same problem!

[m_mind]

Hi Trenghy,
After doing some tests, I ended up with rewireing an import script I used to import files from our php based project environment and modify it to read the directory structure and use a SOAP interface to import each file.
I know that this kind of action should also be possible (and preferably) by using a REST interface, but this was something I already had on the shelf.
Sorry I cannot offer you any answer on how to import it a 'normal' way.

[d_montagni]

Hi all,

I've same problem on both community 3.0 and 3.2 and mysql.

12:05:47,818  DEBUG [org.alfresco.fileserver]    at org.hibernate.id.insert.AbstractSelectingDelegate.performInsert(AbstractSelectingDelegate.java:33)
12:05:47,818  DEBUG [org.alfresco.fileserver]    … 64 more
12:05:47,818  DEBUG [org.alfresco.fileserver] [T2] ** Active transaction after packet processing, cleaning up **
12:05:47,818  ERROR [org.hibernate.AssertionFailure] an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use of the session)
org.hibernate.AssertionFailure: null id in org.alfresco.repo.avm.PlainDirectoryNodeImpl entry (don't flush the Session after an exception occurs)
‍‍‍‍‍‍‍

Does anyone know if this problem will be fix?

thanks in advance
Daniele

[norgan]

Did you check issues.alfresco.com ?

[wesse]

You can zip them and use import from the alfresco UI. It has some issues though with filename encodings (if your windows do not use utf-8)
and then the limits are the session timeout vs. upload speed.

/Micke

[jhuie]

I have the same problem.  I have searched unsuccessfully for a workaround.  It appears that none of the methods for bulk loading files into Alfresco work.  I have been posting this and looking for alternatives with absolutely no response for several days now.  The people I work with are telling me to abandon Alfresco and start looking at Sharepoint. 

[kevinr]

FTP is an effective route. Each file is a separate transaction. Where-as a ZIP import is a single large transaction - so you may need to split the zip files if you hit issues. The WebScript REST APIs also use a separate transaction for each file.

Cheers,

Kev