topic timing between document.delete() and running query in Alfresco Archive

timing between document.delete() and running query

mcs130 — Sun, 16 Sep 2012 21:41:12 GMT

HelloWe are doing some POC work with AF4 (4.0.e) Community Edition and Apache Chemistry 0.8.0-SNAPSHOT source (built JARs from this)Much is working fine.We recently noticed when either running a couple of our unit tests or performing the 2 operations in series in our custom UI's business logic, we s

Re: timing between document.delete() and running query

mrogers — Mon, 17 Sep 2012 06:50:35 GMT

It depends upon the search engine being used. You are probably using solr which will have an asynchronous delay.

Need details of your exception to go forward.

Re: timing between document.delete() and running query

mcs130 — Mon, 17 Sep 2012 10:35:55 GMT

I will get the later today when I get in but you ARE CORRECT - our Community Edition is using SOLR.

Thanks

Re: timing between document.delete() and running query

mrogers — Mon, 17 Sep 2012 11:00:19 GMT

So when you use SOLR for a search it may fail to find recently added content or find recently deleted content. Its just a fact of life with the so called "eventual consistency",

I'm assuming that the error will be in your custom code.

There are a couple of alternate approaches
a) don't use solr as your working method to list contents of recently modified folders. There are many other APIs available. This is how Alfresco Share works.
b) Check that a node exists prior to using it, this will probably sort out your delete case.

Re: timing between document.delete() and running query

mcs130 — Mon, 17 Sep 2012 11:26:06 GMT

Thanks for the quick response…but when you say there are many other APIs available, we are constrained to make these calls through CMIS only. We cannot use any product specific approach with what is being developed. Are you referring to something AF-specific?

Mark

Re: timing between document.delete() and running query

mcs130 — Mon, 17 Sep 2012 14:17:49 GMT

here is last part of the stack trace from our logs involving the attempt to make the Query call:

I suppose this confirms what you expected?


[9/17/12 9:55:09:284 EDT] 0000001d SystemErr     R org.apache.chemistry.opencmis.commons.exceptions.CmisRuntimeException: Node does not exist: missing://missing/missing(null)
[9/17/12 9:55:09:284 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.convertStatusCode(AbstractAtomPubService.java:452)
[9/17/12 9:55:09:284 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.post(AbstractAtomPubService.java:570)
[9/17/12 9:55:09:285 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.bindings.spi.atompub.DiscoveryServiceImpl.query(DiscoveryServiceImpl.java:142)
[9/17/12 9:55:09:285 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.runtime.SessionImpl$3.fetchPage(SessionImpl.java:562)
[9/17/12 9:55:09:285 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.runtime.util.AbstractIterator.getCurrentPage(AbstractIterator.java:132)
[9/17/12 9:55:09:285 EDT] 0000001d SystemErr     R    at org.apache.chemistry.opencmis.client.runtime.util.CollectionIterator.hasNext(CollectionIterator.java:48)


‍‍‍‍‍‍‍‍‍‍‍

Re: timing between document.delete() and running query

mrogers — Mon, 17 Sep 2012 15:58:39 GMT

Sorry - I don't known about Open CMIS :mrgreen: But I would have thought that there are list methods available that could be used in place of search.

Re: timing between document.delete() and running query

mcs130 — Mon, 17 Sep 2012 16:26:06 GMT

Not sure what you are saying about "Search" vs "list"

To recall:

We perform 3 basic calls (for brevity, focused solely on the calls made via CMIS Session object which we do everything through and which works for everything so far against Alfresco 4)

We build the queryString like anything else: (we do this dynamically w/ variable substitution, but to illustrate I am showing the captured String value)

Step 1:


String queryString = SELECT * FROM acme:documents  WHERE IN_FOLDER('workspace://SpacesStore/c98cd558-e159-44fa-927a-7e9b17241dff') AND CONTAINS ('engineering')

cmisSession.query(queryString, false);

‍‍‍‍‍‍

We get back 12 records - everything is exactly as expected.

Step 2:

One of the records in the list is displayed in the custom UI being built, is then marked (check box) for a delete. The next call is then made:


document.delete(true);
‍‍‍

Here the document instance correlates to the selected item marked by the user for Delete.

This also works, the document is in fact deleted from the repository exactly as expected.

We then attempt to run this immediately AFTER the delete() call is made - same exact code as Step 1.

Step 3:


String queryString = SELECT * FROM acme:documents  WHERE IN_FOLDER('workspace://SpacesStore/c98cd558-e159-44fa-927a-7e9b17241dff') AND CONTAINS ('engineering')

cmisSession.query(queryString, false);
‍‍‍‍‍

We EXPECT to get back 11 records - INSTEAD we get the exception thrown UNLESS we purposefully hold with a sleep() between Step 2 and 3.

We are not using an Alfresco-specific APIs for this as we cannot for this work. We are adhering to the CMIS SQL only (a subset of SQL-92 as in the specification for CMIS)

Is this perhaps clearer?

Re: timing between document.delete() and running query

jpotts — Mon, 17 Sep 2012 16:34:53 GMT

Mark is suggesting that you essentially have two options:
1. Keep Solr, but use a non-search call like Folder.getChildren(), Session.getObject(), or Session.getObjectByPath(). Granted this may not meet your needs because you are also doing a full-text search to get your result set.
2. Switch from Solr to Lucene. This is a simple configuration change, but if you are writing a tool that must work with Alfresco regardless of how it is configured, this may not be something you can depend on.

Jeff

Re: timing between document.delete() and running query

fpa1974 — Mon, 17 Sep 2012 22:24:39 GMT

I actually would like to ask a question on the eventual consistency theme here. Basically the sequence of events here is:
1. Perform a query
2. Delete a document
3. Perform a query
In mind, when I execute the query for the second time, I might get in the result set the document that was deleted or not, depending on the timing. But I would not expect to get an exception. This is what I understand as eventual consistency. Is there something that can be done so AF4 behaves that way with solr enabled?

Thanks,
Florian

P.S. I work with Mark on this project.

Re: timing between document.delete() and running query

jpotts — Mon, 17 Sep 2012 23:02:05 GMT

Agreed, you should not get an exception. But it looks as though the exception may be coming from OpenCMIS, not Alfresco. I am wondering if you are holding on to that iterator when you shouldn't be. For example, if you delete the object using the web client instead of code and then invoke your page, does the exception happen?

Jeff

Re: timing between document.delete() and running query

andy — Tue, 18 Sep 2012 10:33:40 GMT

Hi

You run a query - you get a list of nodes - then you try and get them …
Why do they have to still be there? Someone else could have deleted it ….changed its permissions… etc etc
Eventual consistency is just worse - you are more liekly to have nodes that do not exists in the results.

missing:missing//missing is used to indicate a missing node - you have found a node that longer exists for some reason
When you make a CMIS call for this ID it will not be there.
As this can happen at any time you have to expect it - using any API - hold onto data it can change ….there is no "repeatable read" between calls.

Andy

Re: timing between document.delete() and running query

mcs130 — Tue, 18 Sep 2012 21:09:38 GMT

Perhaps I am missing something, so please bear with me on this:

1- The scenario we described involved making a call to pass an SQL query String

cmisSession.query(queryString, false);‍

.
2 -We get back an ItemIterable<QueryResult> - it's correct.
3 -We iterate through and display the list of documents (relevant info only) in the UI - works great.
4 -The user then selects a document from the list…behind the scenes that application has the objectId of this ONE document and fires of the delete() method on it with its business logic. The delete works fine because when you refresh the local Alfresco repository web page on the Folder that has the documents, there is ONE less… URL form looks like: http://XX.XX.XX.XX:8081/share/page/repository#filter=path%7C%2FAcmeLibrary%2Facme-data%2F9999777700005555%2Feedocs&page=1
5 -Then, the same exact SQL query String is passed in again to perform an entirely NEW query, expecting to get back a new ItemIterable<QueryResult> - why should this be a problem? The query() method is on the Session object. We are not recreating this Session, we are using the current one we have - is that a problem? I would think it shouldn't be.

It is obtained like the following:

Session org.apache.chemistry.opencmis.client.api.SessionFactory.createSession(Map<String, String> parameters)‍

What "node" is being referred to here in the statement below?

missing:missing//missing is used to indicate a missing node - you have found a node that longer exists for some reason
When you make a CMIS call for this ID it will not be there.

I would expect simply to get a new ItemIterable<QueryResult>, reflective of the same results except now not including the Document object that correlates to the one objectId that I just deleted… of which nowhere in the second call is there any reference to this objectId.

Is the problem actually the "state" of the cmis:Folder's objectId we have - of which I am re-using since the SQL is using the IN_FOLDER predicate? We don't get it again, since the Folder objectId does not change and for the purpose of the IN_FOLDER predicate, you pass the String of the Id and not the object instance itself.

A) We find if we wait up to 20 seconds, and make the SAME call - it works, which we CANNOT actually implement OR
b) Switching to Lucene from Solr - the problem then goes away and is no longer seen.

Am I missing the point altogether?

Thanks

Re: timing between document.delete() and running query

karstene — Mon, 24 Sep 2012 12:46:43 GMT

I am facing the same issue. A query call fails with the Soap Fault message "Runtime error. Message: org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: missing://missing/missing(null)"

In my case, the previous call was a CancelCheckOut, wich deletes the working copy. The next query returns the fault and no result set.

I read your statement

You run a query - you get a list of nodes - then you try and get them …
Why do they have to still be there? Someone else could have deleted it ….changed its permissions… etc etc

Andy

and would agree if the fault happens in subsequent calls after the query, when trying to access non-existing nodes. But the query itself has to return a valid node set, with or without the deleted node, depending on timing. Instead, it returns a fault and no result set at all.

thanks
Karsten

Re: timing between document.delete() and running query

t_sato — Tue, 25 Sep 2012 11:33:03 GMT

Hi,

Are you guys aware of this issue?

Solr CMIS Query After Delete a Node Throws CmisRuntimeException: Node does not exist

This was closed as fixed at 4.0.2. But this may persist. Other test cases must be helpful to fix completely.

Re: timing between document.delete() and running query

mcs130 — Tue, 25 Sep 2012 19:23:45 GMT

We are using 4.0.e Community Edition. Is it possible that this is why we are seeing this? We wanted to use Solr as it is configured OOTB, but did have to switch the configuration to use Lucene to get around the problem for now. You mention that the defect fix applies to 4.0.2.

Mark

Re: timing between document.delete() and running query

jpotts — Thu, 11 Oct 2012 20:41:58 GMT

Means it should be fixed in 4.2a Community Edition. Would be great if someone could confirm.

Jeff

Re: timing between document.delete() and running query

mcs130 — Wed, 31 Oct 2012 20:47:07 GMT

Just a new update: We are currently using 4.2.a COMMUNITY Edition with the OOTB Solr configuration (Postgress) and are still seeing this issue. I will, for now, go in an switch the index configuration to Lucene to get around this. We've been using the Community Edition for POC work.

Currently, our company is now looking at Alfresco 4.2 Enterprise as a solution for this ECM initiative and perhaps when we have a chance to install the 4.2 Enterprise version, maybe it is fixed in that version?

Thanks

Mark

Re: timing between document.delete() and running query

jspuchau — Fri, 16 Nov 2012 18:46:25 GMT

Hi everybody

We are facing a very similar issue in our organization, but we are not using Solr, as we are using Alfresco Enterprise 3.4.10.

Our problem appears during execution of stress test on our cmis layer. When we mix deletion with methods using the ItemIterable (and PageFetcher) classes from OpenCMIS, Alfresco returns some odd errors to identical requests.

It's worth to say that it happens when using atom/pub implementation of opencmis, but not using soap implementation.

I don't have access to logs as Im at the airport, but on monday I will post it.

Regards, and thanks for the help. I hope we be able to get a solution.

Re: timing between document.delete() and running query

mcs130 — Fri, 16 Nov 2012 19:26:08 GMT

I will say that we have recently observed:

Community Edition: 4.2.a - Exception is no longer thrown back upon the immediate SELECT query after the delete is made, but there is a lag due to Solr indexing (which eventually catches up) - the "eventual consistency" behavior seems to be expected with Solr

Enterperise Edition: 4.1 - Exception is no longer thrown back upon the immediate SELECT query after the delete is made, but there is a lag due to Solr indexing (which eventually catches up) - the "eventual consistency" behavior seems to be expected with Solr