topic Re: Search content without extension in Alfresco Forum

Search content without extension

mattjourdan — Fri, 10 Mar 2017 15:01:14 GMT

Hello,I use alfresco community 5.0.d and i would like to know if it is possible to search in Alfresco all the files thatdon't have an extension. I do not know how to do the search.Thanks, Matthieu

Re: Search content without extension

mehe — Sat, 11 Mar 2017 18:32:11 GMT

Depends on which Search you want to use. If using the "Aikau" Search in Share or the Alfresco FTS , the Searchstring !=cm:name:*.??? should do it. It should find all nodes not having a name that ends with a three character extension.

Re: Search content without extension

afaust — Mon, 13 Mar 2017 08:42:37 GMT

The question isn't necessarily a matter of which UI you use (Aikau faceted search or Node Browser for instances), but if the search services support this type of query. The problem with a wildcard based approach in FTS is that it will by design only scale to a certain amount of documents in the system. This is a result of how the query is translated to the underlying Lucene system in SOLR. Also, the pattern *.??? assumes that all extensions are three-letter extensions only which might have been the standard in the old DOS 8.3 world but all modern MS Office extensions are four-lettered ones.

Without having done a similar query myself on a large document base (i.e. more than just a couple tens of thousands of documents), I would assume the best way to work with this is by doing a CMIS query using the LIKE operator on cmis:name. The reasoning behind this is that a CMIS query using LIKE can actually be applied against the database instead of the SOLR index, and thus is not limited by the index query rewrite restrictions. The only thing you need to ensure is that the additional indexes for transactional metadata queries have been applied on the database system.

Re: Search content without extension

mehe — Mon, 13 Mar 2017 11:32:17 GMT

Hi Axel, I mentioned "Aikau" because it's the easiest way to test the FTS String. The query performs well on large document sets (tested with 1000.000 doc repo ) , but paging throu large resultset gets slower for following pages (and gets worse page by page)

It's true it finds only three character extensions, but is easy to adapt 🙂

I used ??? because I thought Solr would internally invert the query string (???.*) which would not be so expensive - do you know if this is correct?

Re: Search content without extension

afaust — Mon, 13 Mar 2017 11:50:31 GMT

I can't say how SOLR / Lucene handles this low level. I just remember issues with running into maxBooleanClause limits with Alfresco SOLR before due to the way that Alfresco was rewriting wildcard queries before sending them off to the SOLR / Lucene layer. Though this may have changed in Alfresco 5.0 or later versions...

Re: Search content without extension

mehe — Mon, 13 Mar 2017 12:17:41 GMT

Max boolean clauses should be no problem here - Hits you when using big "or" conjunctions. I hoped to eliminate that by using the '=' Operator. (You see that I used 'hope' - what would we do without it 🙂

Re: Search content without extension

cesarista — Mon, 13 Mar 2017 23:13:48 GMT

Hi:

I tried in Alfresco Search with a small set in a site --> TYPE:"cm:content" AND !=cm:name:*.*

And then I played a little bit with mimetype facet, considering "Binary File (Octet Stream)", HTML and text mimetypes. I obtained some meaningful list, although not exactly accurate.

- What about a database query for doing LIKEs ?

- What about a recursive javascript function checking the filenames with some js regex ?

Regards.

--C.

Re: Search content without extension

afaust — Tue, 14 Mar 2017 07:55:19 GMT

Recursive analysis via JavaScript is out of the question. This will be extremely slow, load too much data into Memory (overwhelming Caches) and potentially lock rows/tables in the DB (some DB systems have lock escalation functionalitites when too many rows are read in a transaction).

As I said, you could try CMIS SQL queries using LIKE against the DB - in that case you would basically only test for the presence of a dot, e.g. do a where cmis:name NOT LIKE '%.%' query...

Re: Search content without extension

mehe — Tue, 14 Mar 2017 09:00:46 GMT

I think you could use a recursive script, but only for admin use/purpose. Runtime of the script would be long - the browser will probably run in a timeout error, but there should be no lock problem because the script would only do read access which will result in "Shared locks" on the db which are not causing lock escalations BUT if there is any insert or update request on the dataset, it will block the script until the Update/insert is completed.

But your cmis variant is far better. Is there any restriction on the number of fetched results when executing a cmis db query like in the early search service?

Re: Search content without extension

mattjourdan — Tue, 14 Mar 2017 09:12:49 GMT

Hi,

Thanks a lof for your answers.

I will try CMIS SQL queries

Matthieu

Re: Search content without extension

cesarista — Tue, 14 Mar 2017 09:39:59 GMT

Hi:

You can try CMIS SQL queries in node browser (cmis-strict or cmis-alfresco) or CMIS workbench first, before writing a custom CMIS code for example in java or python.

Regards.

--C.

Re: Search content without extension

mehe — Thu, 23 Mar 2017 17:28:18 GMT

Hi,

a last update for this one

TYPE:"cm:content" !=cm:name:*.?*

Did it for me. Found all files without an, at least one character, extension. AND is implicit in the newer Alfresco versions.

‌ filter cm:content was a good idea, missed it in the first shot (boaah... so many nodes without extension )

I used the slingshot search via an angularjs SPA, so the count of documents without extension was present in milliseconds. A feature of Solr, which gives you the count of matches directly in the result header.

Re: Search content without extension

mehe — Tue, 04 Apr 2017 20:42:18 GMT

As I wrote in another thread (Get all Childs of a Node )

I totally forgot about GitHub - ciber/alfresco-js-batch-executer: Alfresco easy bulk processing with JavaScript
this gives you a new root scope object which enables you to write a serverside JavaScript that iterates over the elements without the common problems like runtime, memory consumption, transaction size...

just for completeness...