topic Re: High Volume Files - Alfresco Issues in Alfresco Archive

High Volume Files - Alfresco Issues

ediez — Tue, 15 Apr 2008 13:41:04 GMT

Hi People,I've been testing the Alfresco repository with high volumes of files/folders. I've been trying to upload 3GB folders and I get a handful of errors from the Application Server Console. I'll list herein some of them:- org.alfresco.service.cmr.repository.TemplateException: Error during proces

Re: High Volume Files - Alfresco Issues

mikeh — Tue, 15 Apr 2008 15:49:08 GMT

PostgreSQL was only certified in v2.1.1 which may indicate there was an issue in v2.1 Community.
Please see: http://www.alfresco.com/services/support/stacks/2.1/

Is there any way you can use one of the certified databases?

Full stack traces might be useful, but no guarantees based on the above…

Thanks,
Mike

Re: High Volume Files - Alfresco Issues

ediez — Tue, 15 Apr 2008 16:10:27 GMT

Thanks fro replying Mike!.

I'll find out if we can use any of the certified databases. I've got one question though, why do you state this problem might be caused by a database issue? Are you familiar with the exceptions I posted before? Have you read reports concerning this sort of issues?.

Also wanted to share with you, that we're digging into our Alfresco instance today and we found out that a plug-in called "Thumbnails" , downloaded from the Alfresco forge may be causing these problems to the repository

Please find below the stack traces for some of the errors:

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
        at $Proxy67.processTemplateString(Unknown Source)
        at org.alfresco.repo.action.scheduled.SimpleTemplateActionDefinition.getAction(SimpleTemplateActionDefinition.java:187)
        at org.alfresco.repo.action.scheduled.CronScheduledQueryBasedTemplateActionDefinition.getAction(CronScheduledQueryBasedTemplateActionDefinition.java:268)
        at org.alfresco.repo.action.scheduled.AbstractScheduledAction$JobDefinition$1$2.execute(AbstractScheduledAction.java:570)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:236)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:166)
        at org.alfresco.repo.action.scheduled.AbstractScheduledAction$JobDefinition$1.runTransactionalAction(AbstractScheduledAction.java:564)
        at org.alfresco.repo.action.scheduled.AbstractScheduledAction$JobDefinition$1.doWork(AbstractScheduledAction.java:432)
        at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:269)
        at org.alfresco.repo.action.scheduled.AbstractScheduledAction$JobDefinition.execute(AbstractScheduledAction.java:398)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
23:00:00,025 ERROR [quartz.core.JobRunShell] Job jobGroup.jobD threw an unhandled Exception:
org.alfresco.service.cmr.repository.TemplateException: Error during processing of the template 'No nodes selected'. Please contact your system administrator.

at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2375)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2257)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2557)
        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:92)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:232)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:140)
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:562)
        … 7 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key violates unique constraint "alf_child_assoc_parent_node_id_key"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1548)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1316)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:305)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:101)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2357)

Re: High Volume Files - Alfresco Issues

mikeh — Tue, 15 Apr 2008 18:13:04 GMT

I'm not familiar with the Thumbnails contribution, so can't comment on anything it might be up to.

The reason I pointed to PostgreSQL was the third exception you posted - duplicate keys tend to indicate non-transactionally safe operations; although I must admit I'm guessing that could be the underlying problem.

It sounds like you know what you're doing, so I'd try the test again with an out-of-the-box Alfresco as a test server, watching for the same issues.

Re: High Volume Files - Alfresco Issues

ediez — Tue, 15 Apr 2008 20:46:44 GMT

Hi Mike.

Just did the test with an out-of-the-box Alfresco and the "duplicate key" errors are still being thrown. Addtionally, I got an error related to a transaction that wasn't committed.

INFO: Server startup in 71301 ms
javax.transaction.RollbackException: Transaction didn't commit: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#5701]; nested exception is org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#5701]
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:430)
        at org.alfresco.filesys.server.SrvSession.endTransaction(SrvSession.java:684)
        at org.alfresco.filesys.ftp.FTPSrvSession.run(FTPSrvSession.java:4431)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.springframework.dao.DataIntegrityViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#5701]; nested exception is org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#5701]
        at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:630)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:699)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:570)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:654)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:624)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:307)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:420)
        … 3 more
Caused by: org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#5701]
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2375)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2257)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2557)
        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:92)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:232)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:140)
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:562)
        … 7 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key violates unique constraint "alf_child_assoc_parent_node_id_key"
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1548)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1316)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:305)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:101)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2357)
        … 19 more

I'll do the same test with a MySQL 5.0 database (which is a certified one) and will tell you the results I got soon.!

Thanks for all your cooperation.!

Re: High Volume Files - Alfresco Issues

ediez — Wed, 16 Apr 2008 14:42:02 GMT

Hi Mike,

Just did the test with MySql 5.0 and have got the same errors as I did with PostgreSql. I wanted to ask you if we could considerer these errors as "normal". I'm afraid that these errors might corrupt the repository at any given time.

Any hints?

Please find below the full stack trace

javax.transaction.RollbackException: Transaction didn't commit: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#641]; nested exception is org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#641]
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:430)
        at org.alfresco.filesys.server.SrvSession.endTransaction(SrvSession.java:684)
        at org.alfresco.filesys.ftp.FTPSrvSession.run(FTPSrvSession.java:4431)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.springframework.dao.DataIntegrityViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#641]; nested exception is org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#641]
        at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:630)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:699)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:570)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:654)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:624)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:307)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:420)
        … 3 more
Caused by: org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#641]
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2375)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2257)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2557)
        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:92)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:248)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:232)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:140)
        at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:562)
        … 7 more
Caused by: com.mysql.jdbc.exceptions.MySQLIntegrityConstraintViolationException: Duplicate entry '432-{http://www.alfresco.org/model/content/1.0}contains-1-000000' for key 2
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2870)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
        at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1160)
        at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:685)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1400)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1314)
        at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1299)
        at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:101)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2357)
        … 19 more
javax.transaction.RollbackException: Transaction didn't commit: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#1865]; nested exception is org.hibernate.exception.ConstraintViolationException: could not update: [org.alfresco.repo.domain.hibernate.ChildAssocImpl#1865]

Re: High Volume Files - Alfresco Issues

mikeh — Wed, 16 Apr 2008 20:28:53 GMT

Hmmm.. could you please raise the issue in JIRA copying in both full exceptions?

This looks familiar to an issue we've seen before, but I need another developer with more experience of that area to take a look - and he's on leave at the moment. Raising the issue in JIRA will ensure it gets the proper attention.

Many thanks and sorry I don't have an immediate fix,
Mike

Re: High Volume Files - Alfresco Issues

ediez — Wed, 16 Apr 2008 21:02:19 GMT

Hey Mike,

Actually, I already posted this issue at JIRA and the only response, it's have been handling by Derek Hulley and the only reply I've got is "A later community version should be used." .

You may see the complete thread at http://issues.alfresco.com/browse/AR-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_20766 .

Thanks for your help on this and please let me know if get more information about this issue.

Re: High Volume Files - Alfresco Issues

mikeh — Thu, 17 Apr 2008 08:02:09 GMT

Are you in a position to try with v2.9 (from the nightly build server)?

We'll be releasing 2.9 Final in the coming weeks - although I don't have an exact date for it.

Thanks,
Mike

Re: High Volume Files - Alfresco Issues

ediez — Thu, 17 Apr 2008 13:28:53 GMT

Hey Mike.!

Sure.! We can definitely try that out. I'm just not sure to deploy it to the production environment until the final version is released.

Do you think the 2.9 might have a fix for these issues? Is there any way to get like a "bugs-fix" list of this release, or something like that?

Thanks!

Re: High Volume Files - Alfresco Issues

steve — Thu, 17 Apr 2008 16:35:13 GMT

Hi,

You can look in our bug tracking system for fixed bugs.
http://issues.alfresco.com

Steve

Re: High Volume Files - Alfresco Issues

jzulu2000 — Wed, 30 Jul 2008 14:20:48 GMT

I'm having the same problem uploading files using ftp. It happens when loading a big number of files, and alfresco slows down all the following request.
Before the problem occurrs, the avg time creating a document is 120 ms; after the first occurrence of the error the avg increments to 2300 ms.. it's almost 20 times the normal time.

Re: High Volume Files - Alfresco Issues

jzulu2000 — Thu, 31 Jul 2008 14:25:35 GMT

This problem happened to me when I was trying to save two documents with the same name to alfresco to the same folder at the time, using 2 diferent ftp sessions…
The first ftp session creates the document in memory, but before it persist it, the second session creates it to… then the first session persists it, and after that the second one does it to and the exception is thrown..

the same problem occurrs for example when creating two times the times same folder concurrently.

I think it is because of the hibernate delay to persist the data to the database and make it available to the other sessions.