Missing .bin file after restore, favourites are missing as well

[streetturtle]

After we did Alfresco restore some random users couldn't login to Alfresco. In the logs for each login attempt there is following stacktrace:

REPO_ 2015-11-16 16:46:12,569  ERROR [extensions.webscripts.AbstractRuntime] [http-apr-9443-exec-9] Exception from executeScript - redirecting to status template error: 10160432 Wrapped Exception (with status template): 10166691 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js': 10166690 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin, mimetype=text/plain, size=0, encoding=UTF-8, locale=en_US]
org.springframework.extensions.webscripts.WebScriptException: 10160432 Wrapped Exception (with status template): 10166691 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js': 10166690 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin, mimetype=text/plain, size=0, encoding=UTF-8, locale=en_US]
   at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:
   …
   at java.lang.Thread.run(Thread.java:744)
Caused by: org.alfresco.scripts.ScriptException: 10166691 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js': 10166690 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin, mimetype=text/plain, size=0, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.jscript.RhinoScriptProcessor.execute(RhinoScriptProcessor.java:204)
        … 
   at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
   … 33 more
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 10166690 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin, mimetype=text/plain, size=0, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:401)
   …   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6._c_Common_getFavourites_11(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js:666)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6.call(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6._c_getDoclist_20(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js:1111)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6.call(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js)
   at org.mozilla.javascript.optimizer.OptRuntime.callName0(OptRuntime.java:74)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6._c_script_0(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js:1389)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6.call(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js)
   at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
   at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6.call(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js)
   at org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6.exec(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js)
   at org.alfresco.repo.jscript.RhinoScriptProcessor.executeScriptImpl(RhinoScriptProcessor.java:502)
   at org.alfresco.repo.jscript.RhinoScriptProcessor.execute(RhinoScriptProcessor.java:200)
   … 38 more
Caused by: org.springframework.dao.ConcurrencyFailureException: store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin no longer exists
   at org.alfresco.repo.content.EmptyContentReader.getDirectReadableChannel(EmptyContentReader.java:56)
   at org.alfresco.repo.content.AbstractContentReader.getReadableChannel(AbstractContentReader.java:271)
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:375)
   … 62 more
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This part looks interesting:

Caused by: org.springframework.dao.ConcurrencyFailureException: store://2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin no longer exists
‍‍‍

If I create this file in the

alf_data/contentstore/2015/7/28/10/17/5f8a8649-9f9e-472b-b6c1-4cc21d28155f.bin‍

with empty curly brackets (

{}‍

) user can login and it solves the problem. But we have to many users so it would be a total mess to create such file for each user who has this problem.

But in the stacktrace I found another interesting part:

org.mozilla.javascript.gen.classpath__alfresco_templates_webscripts_org_alfresco_slingshot_documentlibrary_doclist_get_js_6._c_Common_getFavourites_11(classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js:666)
‍‍‍

So I would guess that js method

getFavourites‍

in doclist.js is causing this error. Which means that when user logins to Alfresco main page should be shown, but it has dashlet with favorites, which couldn't be loaded due to missing file. This would explain the error.
I also tried to call a preference webscript which returns user preferences (and favourites files as well): https://someco.com/alfresco/service/api/people/someusername/preferences and for some user it failed with exactly the same error:

{
  "status": {
    "code": 500,
    "name": "Internal Error",
    "description": "An error inside the HTTP server which prevented it from fulfilling the request."
  },

  "message": "org.alfresco.service.cmr.repository.ContentIOException: 10166868 Failed to open stream onto channel: \n   accessor: ContentAccessor[ contentUrl=store:\/\/2015\/7\/28\/11\/18\/71f8a94e-2fbb-4ec1-a12a-56b8330ebd3c.bin, mimetype=text\/plain, size=0, encoding=UTF-8, locale=en_US]",
  "exception": "",

  "callstack": [

  ],

  "server": "Community v5.0.0 (c r91299-b145) schema 8,009",
  "time": "Nov 16, 2015 6:28:40 PM"
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

So I think that my guess with favourites was right: for some reason after restore all files in the contentstore which contained users' favourites disappeared.

<b>Now reasonable question is how it could be possible?</b>
Because apart from this problem Alfresco works fine, we have more than 60000 documents there and none of them are missed.

<b>And another question: how it could be solved?</b>
As a workaround I think could be useful to get list of all users who used favourites functionality, so we can tell which users will have similar problem with login. 

[douglascrp]

It seems you have restored from a non consistent backup.

Are you sure the database and the contentstore were saved at the same time and with the Alfresco service stopped?

How do you perform your backups?