topic Re: Additional DB table for search/retrieval of hundred million documents in Alfresco Archive

Additional DB table for search/retrieval of hundred million documents

tgrozdek — Thu, 03 Sep 2015 12:04:22 GMT

Hi,I wonder how to implement solution for fast search/retrieval of documents when you might have a hundreds of millions documents in Alfresco 5.0 (One or Community version).Alfresco is primarily used as rudimentary document management system - import document and search/retrieve document, nothing el

Re: Additional DB table for search/retrieval of hundred million documents

mrogers — Thu, 03 Sep 2015 12:17:30 GMT

There shouldn't be any need for that. What is odd about searching that drives you in that direction?

If you do want to continue then I suggest you implement a new search subsystem to work on your own table.

Re: Additional DB table for search/retrieval of hundred million documents

tgrozdek — Thu, 03 Sep 2015 13:17:56 GMT

Hi,

I do not see any big Alfresco implementation, for instance, all Alfresco implementations have a 7 billion documents. Quote from Alfresco site: "Alfresco manages over seven billion documents for more than 1,800 customers in 195 countries, supporting 11 million users in their daily work."
Other examples on Alfresco official site mention tens of millions of documents in some companies but all of this numbers are pretty low for our possible needs, where the numbers may easily reach hundreds of millions of documents even a billion documents if we broaden our scope.

Also, I heard on some Alfresco course that Alfresco has or may have performance issues when number of archived documents reaches million.

My biggest concern about Alfresco is performance and we don't want to end up scaling our servers in dozens or so.

So, we tried to find some other solution for supporting large numbers of documents and one of them is using DB table for import/search/retrieval.

Thanks for Your answer.
Tom.

Re: Additional DB table for search/retrieval of hundred million documents

mrogers — Thu, 03 Sep 2015 13:31:47 GMT

I'm not aware of any archiving problem around a million. Is there a JIRA reference for that?

Also I suspect those numbers are a) conservative and b) out of date. There have been some huge implementations in the last couple of years.

I suggest you try out alfresco 5.1 before re-inventing the wheel.

Re: Additional DB table for search/retrieval of hundred million documents

tgrozdek — Thu, 03 Sep 2015 14:15:04 GMT

I don't know if there is Jira for that, it is just what I heard.

We already have Alfresco 5.1.
So, if we put one hundred million documents on two average Alfresco servers there would be no major performance issues ?

Thanks in advance. Tom.

Re: Additional DB table for search/retrieval of hundred million documents

mrogers — Thu, 03 Sep 2015 16:38:38 GMT

I suggest you try out your use case, before trying to design a solution to a problem that may not exist. At 100M docs SOLR may need to be sharded, but if you were prepared to do your queries in the database then perhaps you don't need it at all.

Re: Additional DB table for search/retrieval of hundred million documents

tgrozdek — Fri, 04 Sep 2015 07:09:21 GMT

Yes, we should try it first though it's not easy thing to do - test environment is weaker than a production one and production environment couldn't be messed around because of other production solutions in use …

Anyway, big thanks for Your time and I think we can go in suggested direction.
Tom.