topic Re: Chinese documents in Alfresco Archive

Chinese documents

chiho80 — Mon, 07 Nov 2005 04:41:35 GMT

I have added some chinese "word documents" and created a chinese HTML document. I cannot search with the documents when I use chinese terms(Using English term is ok). I have looked into the index using Luke, the chinese words seem not indexed. I suspected that is the problem of the StandardAnalyzer,

Re: Chinese documents

andy — Mon, 07 Nov 2005 15:23:42 GMT

Hi

The tokeniser is specified at the type level and is localisable.
If there is no localisation then the values from dictionaryModel.xml are used.

In the default configuration there is a default localisation bundle that specifies the tokenisers that are used. This is in the file

dataTypeAnalyzers.properties

You can set these for a particular locale by adding something like

dataTypeAnalyzers_zh_CN.properties

in the language pack. I don't think any of the language bundles include this as yet. If no locale specific file is found it falls back to the default.

The next question is: "How is the locale found?"
The default locale is picked up from the server or set when interacting with the client. It is possible that interactively adding a document may produce different tokenisation from indexing as a result of a rule as one could set the locale via the client and the other via the default Java locale in the repository.

I hope this helps. Let me know how you get on.

Regards

Andy

Re: Chinese documents

chiho80 — Wed, 09 Nov 2005 17:40:30 GMT

Hi Andy,

Thanks for your advices. Actually I tried to change dataTypeAnalyzers.properties to use the custom analyzer (which can recognize zh, zh_CN and en). But it has no effect at all.

This time, I have read through the index and seem like the index is not written properly. The locale used to write the index made the chinese characters not readable.

In Hong Kong, we use zh as the main chinese locale. zh_CN is used in China. When we try to write zh content using zh_CN locale, the characters gets strange. When I look into the index, the index looks like zh_CN encoded. I will try to set up a debugger to see where the problem is.

I am sorry that I cannot post the analyzer here for your testing coz the analyzer contains some copyrighted codes. I will try to seek for your advice during the course and post the result / changes in code here when the bug is fixed.

Re: Chinese documents

chiho80 — Tue, 22 Nov 2005 14:21:10 GMT

The problem is solved. The problem is due to the reading of chinese file.
When the LuceneIndexerImpl tries to load the document from file system, it does not specify the charset. This would cause problem when reading CJK documents. Please find the code below I used to solve the problem.

[LuceneIndexerImpl ] At around line 1428, I changed the code to:


InputStreamReader isr = null;
InputStream ris = reader.getContentInputStream();
  try {
    isr = new InputStreamReader(ris,"UTF-8");
  } catch (UnsupportedEncodingException e) {
     isr = new InputStreamReader(ris);
  }
  doc.add(Field.Text("TEXT",isr));
}‍‍‍‍‍‍‍‍‍‍

Re: Chinese documents

derek — Wed, 23 Nov 2005 10:10:35 GMT

Hi,

Thanks for this. You are right and we have fixed this.

During the indexing, we perform a transformation on any text that is not UTF-8 and force it to UTF-8.

Regards

Re: Chinese documents

lhy719 — Thu, 22 Dec 2005 03:17:07 GMT

Hi Derek and Chiho,

I'm a Chinese in Taiwan. I've the Alfresoc 1.1 installed, but I still can't find Chinese file or folder names. Notheless it doesn't work with searching Chinese in any file content.

Does the fix method mentioned above has been apply in Alfresco 1.1? Or other things need to be done for Traditional Chinese?

Thanks!
Hammer Lee

Re: Chinese documents

derek — Tue, 03 Jan 2006 11:01:21 GMT

Hi,

The default analyzer is for English. You will notice that chiho80 was writing a custom analyzer for Chinese. These are fairly standard components to write - you might search the Lucene forums for more information on where to get one.

Or you could beg chiho80 to convince his company to release the code.

Regards

Re: Chinese documents

goodguy — Mon, 23 Jan 2006 03:09:42 GMT

derek

Is this going into Alfresco 1.2?

Re: Chinese documents

derek — Mon, 23 Jan 2006 09:13:12 GMT

That depends on whether you can convince chiho80 to give it to us.
:wink:

Re: Chinese documents

chiho80 — Wed, 25 Jan 2006 06:25:43 GMT

It's been a while since I last been here. I am sorry that it seems not possible to contribute the analyzer after a discussion with my boss, as this is one of the core technologies used in our system (non-alfresco).

If you want to find some open source alternatives, you may try CJKAnalyzer, this is not as accurate as dictionary based Analyzer though. You may contact me for help.

Re: Chinese documents

akinori — Sat, 25 Feb 2006 06:31:32 GMT

Hi,

Have you succeeded to enable CJKAnalyzer in Alfresco?

I'm Japanese. There are two major options to analyze Japanese contents in Lucene. JapaneseAnalyzer and CJKAnalyzer. So far I failed both of them.
I tried JaneseAnalysis first. Because it looked better. JapaneseAnalyzer requires another software called Sen to make dictionaly in advance, and it may require System.property stuff which I'm not sure if I already set properly in Tomcat context or not. I saw lots of errors and warnings in the bootstrap message.

So now I'm trying to use CJKAnalyzer anyway. It seemed to work. No error message. But I cannot still find double byte charactors in index.

What I did…
1. Copy lucene-ja.jar which includes CJKAnalyzer to alfresco's lib directory.
2. Make dataTypeAnalyzers_ja_JP.properies pointing to CJKAnalyzer.
3. Restart Alfresco (1.2 RC2)
4. Import a word file including English and Japanese text.

After that,
1. the word file was found by a query with an English word.
2. the word file wasn't found by any query with a Japanese word (longer than 3 charactors I tried).

Re: Chinese documents

akinori — Wed, 01 Mar 2006 07:08:44 GMT

Hi again,

Still I've never succeeded to search with Japanese Contents.
Anyway I found more facts.

1. Both of CJKAnalyzer and JapaneseAnalyzer ran without any error. (Only when I remove the jar file of the analyzer, Alfresco gave me exceptions messages. I believe this means my properties file points to the analyzer properly)
2. I can find Japanese file name and topic titile, but I cannot find any files whose contents include Japanese charactors with Japanese keywords.
3. English keywords still work. If the target file includes both of English words and Japanese words, I can find it with any of those English words.

Sorry, I know it's off-topic in strict sense. Title is Chinese and I'm writing about Japanese issue. But I supposed my situation looked similar to chiho80's case.
I was wondering if someone could instruct me the way to check how lucene works in Alfresco's context now.

Akinori

Re: Chinese documents

andy — Wed, 22 Mar 2006 09:53:09 GMT

Hi

Apologies, this issue has mainly been discussed under French alaysis.

There was a bug tokenising the TEXT attribute in the index - it was always using the standard analyser and not picking up the one defined for text attributres. This is fixed in the future 1.2.1 build (tested with the french analyser). These changes will be merged into the HEAD code stream in due course. The code can be obtained from svn.

I will look into merge and 1.2.1 timings.

Regards

Andy

Re: Chinese documents

derek — Wed, 22 Mar 2006 16:23:56 GMT

The HEAD now contains all the latest bug fixes from the V1.2.0 branch (future 1.2.1 release).