Disaster recovery CRISIS

[globaladvisors_]

Hello

Thanks in advance for any help. This is an absolute crisis for us.

We have been running Alfresco 4.2b on Debian and Postgres - it has been stable and reliable.

Our server crashed last week and we have been attempting to restore from hot-backups via Bacula.

We restored the full Alfresco directory and repository and the Postgres SQL dump. These were backed up within hours of one another over a weekend when there was no user activity on the server.

When we start up the recovered server, we get the following error in the logs:

<blockquote>09:55:34,130 ERROR [org.alfresco.repo.descriptor.RepositoryDescriptorDAOImpl] updateDescriptor:
org.springframework.dao.DuplicateKeyException:
### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "protocol"
  Detail: Key (protocol, identifier)=(system, system) already exists.
### The error may involve alfresco.node.insert.insert_Store-Inline
### The error occurred while setting parameters
### Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "protocol"
  Detail: Key (protocol, identifier)=(system, system) already exists.
; SQL []; ERROR: duplicate key value violates unique constraint "protocol"
  Detail: Key (protocol, identifier)=(system, system) already exists.; nested exception is org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "protocol"
  Detail: Key (protocol, identifier)=(system, system) already exists.
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:241)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:71)
        at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:346)
        at $Proxy7.insert(Unknown Source)
        at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:231)
        at org.alfresco.repo.domain.node.ibatis.NodeDAOImpl.insertStore(NodeDAOImpl.java:296)
        at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.newStore(AbstractNodeDAOImpl.java:805)
        at org.alfresco.repo.node.db.DbNodeServiceImpl.createStore(DbNodeServiceImpl.java:245)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at $Proxy9.createStore(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
        at $Proxy37.createStore(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.alfresco.repo.tagging.TagScopePropertyMethodInterceptor.invoke(TagScopePropertyMethodInterceptor.java:152)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.node.MLPropertyInterceptor.invoke(MLPropertyInterceptor.java:306)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:269)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
        at $Proxy9.createStore(Unknown Source)
        at org.alfresco.repo.descriptor.RepositoryDescriptorDAOImpl.getDescriptorNodeRef(RepositoryDescriptorDAOImpl.java:433)
        at org.alfresco.repo.descriptor.RepositoryDescriptorDAOImpl.updateDescriptor(RepositoryDescriptorDAOImpl.java:187)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl$NOOPLicenseService.verifyLicense(DescriptorServiceImpl.java:354)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl$4.execute(DescriptorServiceImpl.java:266)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl$4.execute(DescriptorServiceImpl.java:260)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:433)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl.bootstrap(DescriptorServiceImpl.java:277)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl.access$100(DescriptorServiceImpl.java:52)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl$3.doWork(DescriptorServiceImpl.java:206)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl$3.doWork(DescriptorServiceImpl.java:202)
        at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:529)
        at org.alfresco.repo.descriptor.DescriptorServiceImpl.onBootstrap(DescriptorServiceImpl.java:210)
        at org.springframework.extensions.surf.util.AbstractLifecycleBean.onApplicationEvent(AbstractLifecycleBean.java:56)
        at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEventInternal(SafeApplicationEventMulticaster.java:209)
        at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEvent(SafeApplicationEventMulticaster.java:180)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:911)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:428)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
        at org.alfresco.web.app.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:63)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)</blockquote>

I've attached full log.

I have tried the upgrade process by installing 4.2.c starting successfully, stopping, replacing with the backed up alf_data, including replacing the solr directory with the solrBackup directory.

Any ideas?


Marc Wilson

[michaelböckling]

This looks like an inconsistent backup. Are you sure your DB, ContentStore and Index have been backed up and restored correctly? Are you using Lucene or Solr?
You could try to start with a fresh index…

[globaladvisors_]

Hi Michael

Thanks for the response. We're using Solr. I followed the instructions for a hot backup restore to the letter.

I have set index recovery to full. But Alfresco never gets there. I have additionally restored from the solrBackup directory to the solr directory to avoid index corruption.

You will see from the log that it tries to insert into a table:

<blockquote>Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "protocol"
Detail: Key (protocol, identifier)=(system, system) already exists
### The error may involve alfresco.node.insert.insert_Store-Inline
### The error occurred while setting parameters</blockquote>

[globaladvisors_]

Hi

I discovered the underlying problem. In setting up the new server, Postgres had upgraded to version 9.1.8-1 on the latest Debian Testing.

Alfresco's JDBC is not comptiable with this version. I downgraded to the version shipped as part of the Alfresco 4.2.c bundle.

I believe other SQL errors being reported might be due to the same issue.

Well. Three days of my life gone, more grey hair and lost sleep. Thought our backups had failed…