topic Re: Alfresco lucene Index in Alfresco Archive

Alfresco lucene Index

itayh — Mon, 17 Sep 2007 15:28:07 GMT

Hi,I have several questions about lucene in alfresco.I could not find documentations on those issues.1 - What is archive folder under \lucene-indexes means?2 - Can I set <index enabled="false"> for specific items and then they will not be index3 - What <archive>true</archive> in cm

Re: Alfresco lucene Index

itayh — Tue, 18 Sep 2007 11:31:42 GMT

Also what backup-lucene-indexes, indexes-backup means?

I will love to get a link to the documentation of those indexes.

Re: Alfresco lucene Index

itayh — Thu, 20 Sep 2007 09:32:37 GMT

Hi,

We also set the indexBackupJobDetail to run in the middle of the day. We notice something very strange: As long the backup was running we could NOT connect to alfresco. The backup took ~ 90 minutes!

Is it make any sense?

Its look like we will need to disable this job. What will be the impacts of this?

Thanks in advance,
Itay

Re: Alfresco lucene Index

andy — Thu, 04 Oct 2007 11:19:28 GMT

Hi

The back up job should not take anything like that long. It just copies the indexes - it does hold a lock on individual indexes so they can not change. This will be much improved soon.

It is up to you if you need a back up - it can make rebuilding an index faster - you can always rebuild from nothing.

1) There are indexes for all store types and stores - archive just happens to be the store type where archived nodes goes.

2) Changing DDs settings only changes indexing behaviour for each attribute - some stuff will always be indexed.

3) Archiving is what may happen to nodes when you delete them - set to false if you do not want this. They are recoverable from the archive and will be indexed.

4) Backup indexes are created by the back up job - you can delete old ones you do not want - or have non at all

Andy

Re: Alfresco lucene Index

itayh — Sun, 07 Oct 2007 08:15:13 GMT

Hi,

Thank you for your answer.

Can you please tell me what are the differences between recovery with backup index and not (I mean the estimated time difference. How fast is backup with index compare to without).

What is the recommended way to do (with backup or without)?

Also if I set <archive>false</archive> in cm:content and cm:folder will it be ok with system functionality?

Itay

Re: Alfresco lucene Index

itayh — Mon, 08 Oct 2007 08:14:10 GMT

Hi,

I notice that just copy the indexes takes a lot of time (we have a lot of indexes).

Also If I run the SDK FirstFoundationClient program that uses Alfresco Embedded project to create nodes, Will it create indexes? I add many nodes using FirstFoundationClient and could not see any indexes.

Thanks a lot in advance,
Itay

Re: Alfresco lucene Index

andy — Tue, 09 Oct 2007 10:43:20 GMT

Hiu

Can you post your index structure (file names, directories, structure and sizes) and repo version OS etc.

Where are you indexes stored?

The SDK may be storeing data where you do not expect depening on the repository properties settings - in the root folder of the project being run from eclipse.

Andy

Re: Alfresco lucene Index

itayh — Wed, 10 Oct 2007 08:19:59 GMT

Hi,

Those are the indexes that were created using java SDK program (we add many nodes):
# du -sch /opt/alfresco/host-specific/node-7/indexes{,-backup}/*
3.8M    /opt/alfresco/host-specific/node-7/indexes/archive
4.0K    /opt/alfresco/host-specific/node-7/indexes/locks
172K    /opt/alfresco/host-specific/node-7/indexes/system
40K     /opt/alfresco/host-specific/node-7/indexes/user
362M    /opt/alfresco/host-specific/node-7/indexes/workspace
1.7M    /opt/alfresco/host-specific/node-7/indexes-backup/archive
4.0K    /opt/alfresco/host-specific/node-7/indexes-backup/locks
132K    /opt/alfresco/host-specific/node-7/indexes-backup/system
40K     /opt/alfresco/host-specific/node-7/indexes-backup/user
83M     /opt/alfresco/host-specific/node-7/indexes-backup/workspace
450M    total

Those are the indexes that were created using alfresco (we add less node than in SDK):
# du -sch /opt/alfresco/host-specific/mg1/indexes{,-backup}/*
56M     /opt/alfresco/host-specific/mg1/indexes/archive
4.0K    /opt/alfresco/host-specific/mg1/indexes/locks
632K    /opt/alfresco/host-specific/mg1/indexes/system
224K    /opt/alfresco/host-specific/mg1/indexes/user
555M    /opt/alfresco/host-specific/mg1/indexes/workspace
56M     /opt/alfresco/host-specific/mg1/indexes-backup/archive
4.0K    /opt/alfresco/host-specific/mg1/indexes-backup/locks
632K    /opt/alfresco/host-specific/mg1/indexes-backup/system
224K    /opt/alfresco/host-specific/mg1/indexes-backup/user
555M    /opt/alfresco/host-specific/mg1/indexes-backup/workspace
1.2G    total

We use linux as OP
Linux mg1 2.6.18-4-amd64 #1 SMP Fri May 4 00:37:33 UTC 2007 x86_64 GNU/Linux

Also Can you please tell me what are the differences between recovery with backup index and not (I mean the estimated time difference. How fast is backup with index compare to without).

What is the recommended way to do (with backup or without)?

Also if I set <archive>false</archive> in cm:content and cm:folder will it be ok with system functionality?

Thx a lot in advance,
Itay

Re: Alfresco lucene Index

andy — Thu, 18 Oct 2007 16:19:11 GMT

Hi

Can you post the listing of directories in workspace/Spaces?
(The index info would also help)

The index rebuild difference depends on which version you are running. If it does a full rebuild, having the index can be slower and you may be better off deleting it, if it is a partial rebuild then it can be much faster (as it validates stuff is present).

Not archiving is up to you - there is no support to undelete via the UI. When you delete it has gone.

Andy

Re: Alfresco lucene Index

itayh — Wed, 07 Nov 2007 17:05:22 GMT

Hi,

The Index backup job taking more than an hour. This is the most important issue that we try to understand.During this time everything is block. We remove this job for now but we want to understand why it happen.

The index.recovery.mode=VALIDATE in repository properties.

How can we investigate it?

Thx,
Itay

Re: Alfresco lucene Index

andy — Mon, 12 Nov 2007 16:09:25 GMT

Hi

The back up does a file copy while holding an index lock. Check how fast files can be copied around the disks.

What version of Alfresco are you using?

Can you post the directory listings for …
/opt/alfresco/*/*/indexes/workspace
and each sub directory

Andy

Re: Alfresco lucene Index

itayh — Sun, 02 Dec 2007 09:13:21 GMT

Hi,

We are using Alfresco 2.1 community version.

Link to directories list can be found here: http://212.72.51.81/ilya/alfresco.list.txt

Copy time on our server:
(1) node-7, 785M, 1454 files
# time rsync -a node-7 node-7.test
real 2m58.333s
user 0m6.356s
sys 0m10.817s

Thx
Itay

Re: Alfresco lucene Index

andy — Fri, 21 Dec 2007 15:39:54 GMT

Hi

Your index structure has rubbish in it.
You could rebuild this.

This is often caused by failing to close a results set (from the Java API)
Have you written any of your own code or is this a standard Alfresco install?

Andy

Re: Alfresco lucene Index

itayh — Wed, 26 Dec 2007 07:40:42 GMT

Thank you for the answer.

Yes we add some code and the close was not in finally.

Maybe that was the problem.

One more issue. We notice that if we perform action on space (our own action written in java API) and got exception sometimes the space just disappear from the screen.
In this case we need to build the whole index's again (FULL in the index rebuild in repository.properties).

Do you have any idea on that?

Thank you in advance,
Itay

Re: Alfresco lucene Index

ebell — Mon, 16 Jun 2008 21:12:40 GMT

Does someone have any information about what exactly is occurring that takes over an hour? We're looking to use Alfresco with around the clock direct uploads. Is there a way to restrict the backup indexing?

Re: Alfresco lucene Index

robertito — Wed, 01 Oct 2008 09:21:24 GMT

Good Day,

we are using Alfresco 3.0 Labs and we also extended it by using its API. We have the same problem about the index that grows.
Could you explain a little bit more of what you exactly mean about closing the result sets or provide some examples?
Also, you explained that Alfresco takes a lock on the indexes and copies them to the backup directory. But the process takes way too long compared to manual copying the indexes. Is there anything else going on within this process? Maybe shrinking the indexes (we know this takes a long time). Is there no work around for this except turning off the index backup job altogether?

Thank you very much!

Re: Alfresco lucene Index

msporled — Wed, 01 Oct 2008 17:36:24 GMT

Does someone have any information about what exactly is occurring that takes over an hour? We're looking to use Alfresco with around the clock direct uploads. Is there a way to restrict the backup indexing?

Check out scheduled-jobs-context.xml

    <bean id="indexBackupTrigger" class="org.alfresco.util.CronTriggerBean">
        <property name="jobDetail">
            <ref bean="indexBackupJobDetail" />
        </property>
        <property name="scheduler">
            <ref bean="schedulerFactory" />
        </property>
        <!– trigger at 3am each day –>
        <property name="cronExpression">
            <value>0 0 3 * * ?</value>
        </property>
    </bean>
‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: Alfresco lucene Index

andy — Thu, 16 Oct 2008 13:35:19 GMT

Hi

The issue was caused by custom code not closing query results sets, so old index data was left behind leading to a very large backup.
This will happen if result sets are left unclosed.

Andy

Re: Alfresco lucene Index

tejaskanani2920 — Tue, 17 Feb 2009 06:00:11 GMT

Hi ,

I want to do Full Indexing in alfresco. i am using Alfresco 2.1.1
But if i do it by just using adding index.recovery.mode=FULL in repository.properties it will do Full Indexing when i restart the alfresco but it took lot of time as i am having lots of spaces in my alfresco repository and by that time i am not able to use alfresco at all till it finish the Full Indexing.
so for that i create one class which takes transaction range as a parameter (0-16000)and it does reindex for that given range of transactions. And i can run that process without disturbing the alfresco as it reads the transactions from databases directly.
so by that way i just divide the Full Indexing in 2 parts.

for example,
if my alf_transaction table contains 32000 transactions.
then i just divide it to 0-16000 and 160001-32000.
Basically it creates two separate folder like lucene-indexes-0-16000 and lucene-indexes-16001-32000.
so lucene-indexes-0-16000 contains index backup of 0 to 16000 transactions. And lucene-indexes-16001-32000 contains index backup of 16001-32000 transactions.

so my final Full Indexing i want to merge those 2 index folders.
but now m getting the problem that how to merge those two different index folders ?
Is there any way in alfresco that will merge those two indexes folder ??

or if anybody gone through this type of stuff before please guide me …

Thanks,

Re: Alfresco lucene Index

ebell — Thu, 07 May 2009 13:47:12 GMT

Does someone have any information about what exactly is occurring that takes over an hour? We're looking to use Alfresco with around the clock direct uploads. Is there a way to restrict the backup indexing?

See the section on Lucene Index Restoration on http://wiki.alfresco.com/wiki/Backup_and_Restore