topic Re: Lucene indexing and performance issue in Alfresco Archive

Lucene indexing and performance issue

jannek — Tue, 05 Feb 2008 14:56:35 GMT

We are using Alfresco to store content that is automatically retrieved from external content providers. We are using a single default content store for storing all of our content. This is because we have to make queries from basically all of the content, which would not be possible if we would have

Re: Lucene indexing and performance issue

pmonks — Sun, 10 Feb 2008 23:49:35 GMT

Like most hierarchical data storage systems (eg. filesystems), Alfresco performance suffers if you attempt to store too much content in a single hierarchy node, and I wouldn't be surprised it that's a large part of the problem you're seeing. Even if you don't require a space hierarchy, I would strongly suggest using one - perhaps a date / time based "hash bucket" structure like that used in the content store. This will help to distribute your content out across multiple spaces, resulting in better performance.

Re: Lucene indexing and performance issue

amh11 — Mon, 11 Feb 2008 12:07:30 GMT

Related to this: is it possible to prevent indexing some of the files that are attached to content? Instructions tell how you can prevent indexing certain metadata attributes, but how do you prevent that certain attached files would not be indexed? Indexing files which would not need to be indexed consumes lot of time and processing power.

Re: Lucene indexing and performance issue

amh11 — Mon, 11 Feb 2008 13:46:23 GMT

to clarify: for certain text file attachements, it would be only needed to store and be able to fetch them, not necessary to index contents of text files.

Re: Lucene indexing and performance issue

andy — Tue, 19 Feb 2008 14:45:31 GMT

Hi

If you add your own property of type d:content it can be defined as unindexed. The default cm:content property is indexed.

Andy

Re: Lucene indexing and performance issue

davidtaylor — Wed, 20 Feb 2008 07:53:25 GMT

Hi,

I can appreciate your concerns since I have implemented an Alfresco-based platform which currently houses 15+ million images (and growing) and makes use of the web client. We implemented our data loader using the JCR interface in preference to web services since testing showed the performance was much better. Currently, after much tuning, we can load 20,000 new images and apply a custom aspect in just under three minutes. Originally the same file took 3+ hours to load due to a variety of indexing and other design issues.

Here are some things we learned in the process:

- Enabling versioning on nodes has a huge negative impact on performance. We saw a 40% improvement in throughput by not applying the versionable aspect to new nodes.

- Distributing content across multiple folder nodes is essential for good performance. We generally try to keep the number of nodes per folder under 10k. Since our image names are serialized, we take a 4 character segment of the file name and use it as a sort of hash value to select a subdirectory. The images are also have a natural distribution by client ID.

- The order of operations when loading files greatly impacts throughput due to database contention issues. Checking for file name collisions inline with the code that performed node insertion caused a major performance hit and occasional database deadlocks (a MS SQL Server specific problem). Separating these operations into a multi-pass process greatly improved throughput.

- We discovered and corrected some database indexing issues that severely impacted MS SQL Server. The addition of some indexes greatly reduced lock contention and eliminated a full table scan that was a major drag on performance.

Hope this information helps.

David

Re: Lucene indexing and performance issue

netdata — Fri, 07 Mar 2008 13:49:57 GMT

Hi David,

Can you help me with this as well please?
We are seeing the same issues as described here.

We are practically doing the same as you do.
Can you please explain me for which tables you created extra indexes in oracle?
We have disabled versioning on all our files so this cannot be an issue.
All our spaces contain at the most 100 files or subspaces.

Adding a document using the API takes about 5 seconds.
I see you are able to do 110 files per second.
And what do you mean by separating the order of operations ?

You could help us a lot.

Re: Lucene indexing and performance issue

davidtaylor — Fri, 07 Mar 2008 18:00:25 GMT

Hi netdata,

The speed at which content can be loaded into Alfresco is negatively impacted by mixing queries and node creation operations. Specifically, I found that the checking for the existence of a node just prior to insertion caused heavy database contention and slowed throughput significantly. The alternative is to check for existence in one pass and then insert or update as required in a subsequent pass. The only downside is that a node could be deleted or added by an external process between passes potentially causing a name collision. This rarely if ever happens in practice and can be dealt with through properly designed recover logic.

I am not sure how significant an impact the above database contention issue will be with Oracle since the project was deployed on SQL Server. I have many years experience developing against Oracle 7.x - 10g for other applications and am inclined to believe that the performance hit would not be as bad due to Oracle's superior concurrency model. At the very least you will not see the dreaded SQL Server "deadlock" exceptions that occur when record insertions trip over read-only query operations on the same tables

Regarding, the indexing improvements, I will need to check with my client before getting into any specific details. The optimization process was a very significant effort and I am not certain that I am at liberty to discuss the details under my NDA. Please contact me by private message if this something you would like to pursue.

Thanks,
David

Re: Lucene indexing and performance issue

dbachem — Thu, 29 May 2008 08:57:12 GMT

Hi David, at the moment we are evaluating Alfresco to use it for a relatively big, muliportal publishing website with lots of reads in parallel (maybe 1000) and writes (up to 200). I heard from an well-experienced Alfresco consultant :!: about performance problems caused by how Alfresco uses Lucene indexing (or something like this). Thus I googled to find anything new about this issue. Ofcourse we need a highly scalable eCMS platform (without workarounds). Could you please inform me about the actual state of these kind of performance problems? You don't support a public bugtracking system for Alfresco dev, do you?

Kind regards, Dirk

Re: Lucene indexing and performance issue

zomurn — Wed, 18 Feb 2009 15:04:24 GMT

I have another question about performance :

The alfresco in production is sometimes rowing. Basically, all is ok.
But not often, when I do a click on a folder for example or a link, I can wait up to 5 seconds at least.
Why this happen ? (lucene background indexing ? multi user ? lucene queries ?)
Otherwise, most of the time, the application is well responding .

Which files do we have to tune for improving performance ?

Thanks

Re: Lucene indexing and performance issue

bwakkie — Fri, 28 Aug 2009 11:05:51 GMT

Some basic thoughts that come up:

The disk file system: did you know that ext4 is very much better in performance then ext3, perhaps it can help. Al least looking into a good file system is worth while looking into.

Externalize Lucene to a different server? Is this possible? Is it possible to replace it for example by Xapian?

Load balanced DB? Master-slave DB (one db to write one to read)