REST API search with custom properties returns no results

[sanzoghenzo]

I'm trying to use REST API to search in the repository for documents with custom properties.

I succesfully created a custom model with this aspects (among others):

<aspect name="sws:projectIdentifier">
    <title>SWS Project Code</title>
    <description>Unique project identifier</description>
    <properties>
        <property name="sws:projectCode">
            <type>d:text</type>
            <mandatory>true</mandatory>
            <constraints>
                <constraint ref="sws:projectCodeFormat" />
            </constraints>
        </property>
    </properties>
</aspect>
<aspect name="sws:bimLinkable">
    <title>BIM Linkable</title>
    <description>Defines the link to a BIM element</description>
    <properties>
        <property name="sws:modelUrn">
            <type>d:text</type>
        </property>
        <property name="sws:elementUniqueId">
            <type>d:text</type>
        </property>
        <property name="sws:functionalSystem">
            <type>d:text</type>
        </property>
        <property name="sws:elementCategory">
            <type>d:text</type>
        </property>
    </properties>
</aspect>

I added these aspects and set the properties to one of my documents in the repository (I can't embed an image, so you have to take my word for it), and if I try to search in Share for, i.e. 

TYPE:"cm:content" AND sws:functionalSystem:"Access Shaft" AND sws:projectCode:"MIWP024"

I correctly get the document, but if I try to use the REST API, I get an empty list.

This is the request:

POST http://localhost:8082/alfresco/api/-default-/public/search/versions/1/search

here's the body:

{ 
	"query": {
		"query": "TYPE:\"cm:content\" AND sws:funcitonalSystem:\"Access Shaft\" AND sws:projectCode:\"MIWP024\""
	},
	"include": ["properties"]
}

The response body is:

{
    "list": {
        "pagination": {
            "count": 0,
            "hasMoreItems": false,
            "totalItems": 0,
            "skipCount": 0,
            "maxItems": 100
        },
        "context": {
            "consistency": {
                "lastTxId": 635
            }
        },
        "entries": []
    }
}

Properties are there, I can see them in Share and If I search for name:

{ 
	"query": {
		"query": "TYPE:\"cm:content\" AND name:\"MC-03\""
	},
	"include": ["properties"]
}

I get the element:

{
    "list": {
        "pagination": {
            "count": 1,
            "hasMoreItems": false,
            "totalItems": 1,
            "skipCount": 0,
            "maxItems": 100
        },
        "context": {
            "consistency": {
                "lastTxId": 635
            }
        },
        "entries": [
            {
                "entry": {
                    "isFile": true,
                    "createdByUser": {
                        "id": "admin",
                        "displayName": "Administrator"
                    },
                    "modifiedAt": "2019-09-24T10:23:31.677+0000",
                    "nodeType": "cm:content",
                    "content": {
                        "mimeType": "application/pdf",
                        "mimeTypeName": "Adobe PDF Document",
                        "sizeInBytes": 431392,
                        "encoding": "UTF-8"
                    },
                    "parentId": "ec429f71-1d5c-46cb-8b1b-a56820cadd02",
                    "createdAt": "2019-09-24T10:03:05.980+0000",
                    "isFolder": false,
                    "search": {
                        "score": 1.0
                    },
                    "modifiedByUser": {
                        "id": "admin",
                        "displayName": "Administrator"
                    },
                    "name": "MC-03 - TUNNEL CONSTRUCTION SHAFT - PLAN SECTION - INITIAL ROCK SUPPORT.pdf",
                    "location": "nodes",
                    "id": "9d1eb08b-cf01-4fb2-ae32-710adbfb5159",
                    "properties": {
                        "cm:title": "D:\\Users\\02_TempPDF - Sheet - MC-03 - TUNNEL CONSTRUCTION SHAFT - PLAN SECTION - INITIAL ROCK SUPPORT",
                        "sws:projectCode": "MIWP024",
                        "cm:versionType": "MAJOR",
                        "cm:versionLabel": "1.0",
                        "sws:modelUrn": "dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6enJwdWEyZW9sdG5yeHBzazdwbjNhNXB2bmVkcDIyd2ZfNWQ4OGVlZmUwNWM1NzUyOTc4Zjg4NzRiL0JDUlVBX0MtQy1CSU0tMDMtMDAtMDAtMDMubndj",
                        "cm:author": "xxxxxx",
                        "sws:functionalSystem": "Access Shaft",
                        "sws:elementCategory": "Rock Support",
                        "cm:lastThumbnailModification": [
                            "doclib:1569319514231",
                            "imgpreview:1569329952072"
                        ]
                    }
                }
            }
        ]
    }
}

I cannot see any error in the logs (docker-compose logs -f doesn't output anything when I submit the request).

Is there something wrong in my syntax?

Thanks in advance for the attention!

[abhinavmishra14]

Take a look at this post: https://hub.alfresco.com/t5/alfresco-content-services-forum/advanced-search-on-custom-model-properti...

[sanzoghenzo]

Hi,

thanks for the quick reply.

I saw that post, I'll try to disable tokenization and see what changes. Anyway...

What I don't understand is why the same exact search (aside from escape characters) works fine in Share (directly from the simple search box since I have not yet added the aspects in the advanced search ui) and in Node Browser, but not via REST API.

Also, does this mean that I cannot perform search on tokenized indices via REST API? that sounds like a bug to me.

Regards

[abhinavmishra14]

You can very well perform search on tokenized values. There are different settings as per use cases. 

This is what documentation says: 

Default index setting:

<index enabled="true">   
   <atomic>true</atomic>   
   <stored>false</stored>
   <tokenised>true</tokenised>
</index>

 The property value is tokenized when it is indexed (<tokenized>true</tokenized>), so if the value is "Access Shaft" it will be tokenized into two strings that will be indexed separately i.e. "Access", and "Shaft" . You can also use false, which will just tokenize the value as one item i.e. just "Access Shaft". It also possible to set it to both, which means that "Access Shaft", "Access", and "Shaft" will be in the index.

Related to share side results, i am not exactly sure about this behavior. Try enabling the log to see what value is being parsed and sent to the solr query. I am guessing that, it is returning the result based on first token or last token. 

[sanzoghenzo]

Hi, thanks again!

Indeed, disabling the tokenization for "phrase-like" fields worked.

After many tests, I found out that I'm facing another problem: I've set up a rule that executes a script for every file added to a folder and its subfolder; that script adds the aspect swsrojectIdentifier and sets the property swsrojectCode (reading the value from the root folder, which is a custom type "project").

when I search for swsrojectCode, only the folder (with the manually set project code) is returned; the propagated property is ignored by the search.

Here's the script I execute:

// propagates the project code to all the files in the container
var projectCode = space.properties["sws:projectCode"];
// making sure the aspect projectIdentifier is on
if (!document.hasAspect("sws:projectIdentifier")) {
	document.addAspect("sws:projectIdentifier");
}
// sets the project code
document.properties["sws:projectCode"] = projectCode;
document.save();

I looked around the web but I didn't found any issue regarding indexing of property set via scripts.

Any hint?

TIA

[jpotts]

It doesn't matter how the property is set.

Have you used the node browser to confirm that the document has the property set to the value you intended?

[sanzoghenzo]

Hi Jeff,

first of all, thanks for your guides, they helped me a lot with my customization.

Yes, as I've stated on my first reply the node browser returns the correct results.

Here's the log of the solr query when using the node browser:

alfresco-content-model-acs_1       | 2019-09-27 07:28:02,117  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-10] Sent :/solr/alfresco/afts?wt=json&fl=DBID%2Cscore&rows=1000&df=TEXT&start=0&locale=it_IT&alternativeDic=DEFAULT_DICTIONARY&fq=%7B%21afts%7DAUTHORITY_FILTER_FROM_JSON&fq=%7B%21afts%7DTENANT_FILTER_FROM_JSON
alfresco-content-model-acs_1       | 2019-09-27 07:28:02,118  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-10]    with: {"tenants":[""],"locales":["it_IT"],"defaultNamespace":"http://www.alfresco.org/model/content/1.0","textAttributes":[],"defaultFTSOperator":"OR","defaultFTSFieldOperator":"OR","anyDenyDenies":true,"query":"TYPE:\"cm:content\" AND sws:functionalSystem:\"Access Shaft\" AND sws:elementCategory:\"Rock Support\" AND sws:projectCode:MIWP024","templates":[{"template":"ISNODE:T","name":"KEY_REPLACELONESTAR"}],"allAttributes":[],"queryConsistency":"DEFAULT","authorities":["GROUP_EVERYONE","ROLE_ADMINISTRATOR","ROLE_AUTHENTICATED","admin"]}
alfresco-content-model-acs_1       | 2019-09-27 07:28:02,118  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-10] Got: 1 in 5 ms

And this is when I use the REST API via Postman:

alfresco-content-model-acs_1       | 2019-09-27 07:28:10,114  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-7] Sent :/solr/alfresco/afts?wt=json&fl=DBID%2Cscore&rows=100&df=TEXT&start=0&locale=en_US&alternativeDic=DEFAULT_DICTIONARY&fq=%7B%21afts%7DAUTHORITY_FILTER_FROM_JSON&fq=%7B%21afts%7DTENANT_FILTER_FROM_JSON
alfresco-content-model-acs_1       | 2019-09-27 07:28:10,118  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-7]    with: {"tenants":[""],"locales":["en_US"],"defaultNamespace":"http://www.alfresco.org/model/content/1.0","textAttributes":[],"defaultFTSOperator":"OR","defaultFTSFieldOperator":"OR","anyDenyDenies":true,"query":"TYPE:\"cm:content\" AND sws:functionalSystem:\"Access Shaft\" AND sws:elementCategory:\"Rock Support\" AND sws:projectCode:MIWP024","templates":[{"template":"ISNODE:T","name":"KEY_REPLACELONESTAR"}],"allAttributes":[],"queryConsistency":"DEFAULT","authorities":["GROUP_EVERYONE","ROLE_ADMINISTRATOR","ROLE_AUTHENTICATED","admin"]}
alfresco-content-model-acs_1       | 2019-09-27 07:28:10,118  DEBUG [impl.solr.SolrQueryHTTPClient] [http-nio-8080-exec-7] Got: 0 in 367 ms

the only thing that changes is the locale, could this be the issue?

UDATE: well, I answered myself: if I add the following to the REST API request:

"localization": { "locales": ["it_IT"] },

the results shows correctly.

I don't understand why the locale matters, the query looks the same to me...

[sanzoghenzo · Accepted solution]

Hi Jeff,

first of all thanks for your helpful guides!

The node browser works fine, but you led me in the right direction: comparing the output of the solr query logger I discovered that the node browser uses the "it_IT" locale, but REST API uses "us_US" by default.

Adding

"localization": { "locales": ["it_IT"] },

to my REST API request body gives me the result I expected.

My final answer is: why on the earth the results changes if I (don't) specify the locale?

[jpotts]

Probably your solr server is indexing using one locale but you are searching using another. When strings are analyzed, broken into terms, and indexed, it matters which locale is used. Similarly, queries also go through the same process.

Maybe someone else will have a more precise answer and somewhere to look to change your solr locale settings.

Glad you figured out what was causing the problem.