cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to import large amounts of files into alfresco.

m_mind
Champ in-the-making
Champ in-the-making
We currently have about 32G of files which we currently have on a windows share. As we want to be able to index and search for files, we want to add these files into Alfresco.
However, when we copy the files to the CIFS interface or to the FTP server, after some time errors occur and the connection is closed.
We need to restart Alfresco in order to be able to use the Fileserver interface again.

The error I get when importing through the CIFS interface is:

09:18:41,168 ERROR [org.alfresco.util.transaction.SpringAwareUserTransaction.trace] UserTransaction being garbage collected without a commit() or rollback().
   Started at:
      org.alfresco.util.transaction.SpringAwareUserTransaction.begin(SpringAwareUserTransaction.java:389)
      org.alfresco.filesys.alfresco.AlfrescoDiskDriver.beginTransaction(AlfrescoDiskDriver.java:332)
      org.alfresco.filesys.alfresco.AlfrescoDiskDriver.beginWriteTransaction(AlfrescoDiskDriver.java:180)
      org.alfresco.filesys.repo.ContentDiskDriver.closeFile(ContentDiskDriver.java:1822)
      org.alfresco.jlan.smb.server.NTProtocolHandler.procCloseFile(NTProtocolHandler.java:1386)
      org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:251)
      org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1344)
      org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1436)
      org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:105)
      org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
      java.lang.Thread.run(Thread.java:619)
but also a longer exception:

11:25:53,320 ERROR [org.hibernate.AssertionFailure] an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use o
f the session)
org.hibernate.AssertionFailure: null id in org.alfresco.repo.domain.hibernate.ChildAssocImpl entry (don't flush the Session after an exception occurs)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.checkId(DefaultFlushEntityEventListener.java:55)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:164)
        at org.hibernate.event.def.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:120)
        at org.hibernate.event.def.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:196)
        at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:76)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:26)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.flushSession(DirtySessionMethodInterceptor.java:317)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.flushSession(DirtySessionMethodInterceptor.java:282)
        at org.alfresco.repo.node.db.hibernate.HibernateNodeDaoServiceImpl.newChildAssoc(HibernateNodeDaoServiceImpl.java:1588)
        at sun.reflect.GeneratedMethodAccessor659.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.transaction.TransactionalDaoInterceptor.invoke(TransactionalDaoInterceptor.java:68)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.domain.hibernate.DirtySessionMethodInterceptor.invoke(DirtySessionMethodInterceptor.java:381)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy2.newChildAssoc(Unknown Source)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.createNode(DbNodeServiceImpl.java:319)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.transaction.SingleEntryTransactionResourceInterceptor.invokeInternal(SingleEntryTransactionResourceInterceptor.java:169)
        at org.alfresco.repo.transaction.SingleEntryTransactionResourceInterceptor.invoke(SingleEntryTransactionResourceInterceptor.java:138)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor528.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:221)
        at $Proxy8.createNode(Unknown Source)
        at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:279)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:188)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:188)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy7.createNode(Unknown Source)
        at org.alfresco.repo.model.filefolder.FileFolderServiceImpl.create(FileFolderServiceImpl.java:729)
        at sun.reflect.GeneratedMethodAccessor688.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
        at org.alfresco.repo.model.filefolder.TempFileMarkerInterceptor.invoke(TempFileMarkerInterceptor.java:83)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.model.ml.MLContentInterceptor.invoke(MLContentInterceptor.java:131)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.model.filefolder.MLTranslationInterceptor.invoke(MLTranslationInterceptor.java:210)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at net.sf.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:80)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.security.permissions.impl.ExceptionTranslatorMethodInterceptor.invoke(ExceptionTranslatorMethodInterceptor.java:49)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.alfresco.repo.audit.AuditComponentImpl.audit(AuditComponentImpl.java:275)
        at org.alfresco.repo.audit.AuditMethodInterceptor.invoke(AuditMethodInterceptor.java:69)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy53.create(Unknown Source)
        at org.alfresco.filesys.repo.CifsHelper.createNode(CifsHelper.java:367)
        at org.alfresco.filesys.repo.ContentDiskDriver.createDirectory(ContentDiskDriver.java:1656)
        at org.alfresco.jlan.smb.server.CoreProtocolHandler.procCreateDirectory(CoreProtocolHandler.java:479)
        at org.alfresco.jlan.smb.server.CoreProtocolHandler.runProtocol(CoreProtocolHandler.java:3709)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:359)
        at org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1344)
        at org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1436)
        at org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:105)
        at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
        at java.lang.Thread.run(Thread.java:619)

I've also tried to create a relatively small ZIP file (about 160MB) in order to import the files like this, but that also doesn't work.
Is there a way to import large amounts of files without crashing Alfresco?

Thanks in advance.
Peter
7 REPLIES 7

treghy
Champ in-the-making
Champ in-the-making
Hi I've the same problem!

m_mind
Champ in-the-making
Champ in-the-making
Hi Trenghy,
After doing some tests, I ended up with rewireing an import script I used to import files from our php based project environment and modify it to read the directory structure and use a SOAP interface to import each file.
I know that this kind of action should also be possible (and preferably) by using a REST interface, but this was something I already had on the shelf.
Sorry I cannot offer you any answer on how to import it a 'normal' way.

d_montagni
Champ in-the-making
Champ in-the-making
Hi all,

I've same problem on both community 3.0 and 3.2 and mysql.


12:05:47,818  DEBUG [org.alfresco.fileserver]    at org.hibernate.id.insert.AbstractSelectingDelegate.performInsert(AbstractSelectingDelegate.java:33)
12:05:47,818  DEBUG [org.alfresco.fileserver]    … 64 more
12:05:47,818  DEBUG [org.alfresco.fileserver] [T2] ** Active transaction after packet processing, cleaning up **
12:05:47,818  ERROR [org.hibernate.AssertionFailure] an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use of the session)
org.hibernate.AssertionFailure: null id in org.alfresco.repo.avm.PlainDirectoryNodeImpl entry (don't flush the Session after an exception occurs)

Does anyone know if this problem will be fix?

thanks in advance
Daniele

norgan
Champ in-the-making
Champ in-the-making
Did you check issues.alfresco.com ?

wesse
Champ in-the-making
Champ in-the-making
You can zip them and use import from the alfresco UI. It has some issues though with filename encodings (if your windows do not use utf-8)
and then the limits are the session timeout vs. upload speed.

/Micke

jhuie
Champ in-the-making
Champ in-the-making
I have the same problem.  I have searched unsuccessfully for a workaround.  It appears that none of the methods for bulk loading files into Alfresco work.  I have been posting this and looking for alternatives with absolutely no response for several days now.  The people I work with are telling me to abandon Alfresco and start looking at Sharepoint.  Smiley Sad

kevinr
Star Contributor
Star Contributor
FTP is an effective route. Each file is a separate transaction. Where-as a ZIP import is a single large transaction - so you may need to split the zip files if you hit issues. The WebScript REST APIs also use a separate transaction for each file.

Cheers,

Kev