Serious startup issue with 4.2.d community

[smcardle]

Hi All

We have just upgraded to <strong>4.2.d</strong> and the startup time for the alfresco.war is <strong>REALLY REALLY BAD……</strong>

On our previous version, <strong>Community - v4.1.0 (4180)</strong> (this version was never officially released but we had an early release and got it working properly) the startup time was just a few minutes. This was upgraded from 3.4.x and we had no performance or startup issues with either…..

With this <strong>4.2.d</strong> release the Search subsystem startup is taking about <strong>half an hour!!!!</strong> What gives ???

<blockcode>
…
2013-10-24 08:50:33,733  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1]
2013-10-24 08:50:33,905  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1] complete
2013-10-24 08:51:14,835  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Search' subsystem, ID: [Search, managed, lucene]
2013-10-24 09:22:26,778  WARN  [cache.node.nodesTransactionalCache] [localhost-startStop-1] Transactional update cache 'org.alfresco.cache.node.nodesTransactionalCache' is full (125000).
2013-10-24 09:22:31,812  WARN  [cache.node.aspectsTransactionalCache] [localhost-startStop-1] Transactional update cache 'org.alfresco.cache.node.aspectsTransactionalCache' is full (65000).
2013-10-24 09:22:31,865  WARN  [cache.node.propertiesTransactionalCache] [localhost-startStop-1] Transactional update cache 'org.alfresco.cache.node.propertiesTransactionalCache' is full (65000).
2013-10-24 09:27:22,344  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Search' subsystem, ID: [Search, managed, lucene] complete
2013-10-24 09:27:23,562  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'thirdparty' subsystem, ID: [thirdparty, default]
…
</blockcode>

As you can see from the above, the time taken to start the Search subsystem is 08:51:14,835 (Starting) to 09:27:22,344 (Complete) <strong>36 minutes !!! HOLY C%$P Batman……</strong> All of our installs have the same JVM settings and have 4gig heap allocated and they all run on Centos 6

We have Lucene configured as the index solution rather than SOLR on all our systems, with the exact same amount of data in each (millions or small records copied from a production instance) and went through a schema upgrade cycle to the new 4.2.d schema that completed successfully.

Because we run SQLServer 2008 as our DB, for each version upgrade I first copy and migrate the upgrade scripts from MySQL to make them SQLServer complient

(By the way, there is an issue in the code regarding upgrades in the ExportDB.java - SQLServer returns "YES " - with a Space when checking the nullability of a column. To fix this change the parseBoolean method to the following - only the trim is really require here)

    private boolean parseBoolean(String nullableString)
    {
        // TODO: what about (from the javadoc):
        // empty string — if the nullability for the parameter is unknown
        if (nullableString.trim().toLowerCase().equals("no"))
        {
            return false;
        }
        if (nullableString.trim().toLowerCase().equals("yes"))
        {
            return true;
        }
        
        throw new IllegalArgumentException("Unsupported term \"" + nullableString +
                    "\", perhaps this database doesn't use YES/NO for booleans?");
        
    }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Some help here please people….. We really can't have a system taking over half an hour to start..


Regards

Steve

[eswbitto]

We are using 4.2.d on Centos 6.4 (postgresql) and have 4 GB of ram on a VM via proxmox. This server is dedicated to Alfresco only. I will say it does gobble up the entire 4 GB at times. Do you have other services running on yours?

Server start up time is like 15 seconds. Restarting takes a good 2-3 minutes, but that is probably do to the fact open/close all the sockets.

If I were you I would try installing a second instance of alfresco with different service name, but use postgresql from the binary installer to try and rule out some area's.

Have you tried using solr instead of lucene?
Maybe re-indexing?
If you have a test lab setup then maybe try those.

[smcardle]

Hi.

A full re-index has been completed and we do not use Postgresql. This would require a lot of work in migrating the existing large amount of data from SQLServer….

I already have another 4.2.d instance with the exact same setup (on a test VM) that only take a couple of minutes to restart.

The difference between the two is that on the slow instance we have a HUGE amount of data copied from a production instance, I am talking several million pieces of very small content. The existing production server (Community - v4.1.0 (4180)) only take a few minutes to start with the same amount of data and is also configured for Lucene.

We do not intend to switch to SOLR indexing just at the moment as some significant retesting of features will be required.

We never use the Alfresco or Share UI as we access everything through Webscripts from out application.

Oh yeah… Thanks for the CSRF and SAMEORIGIN Policy addition being switched on by default Alfresco, nothing worked at first.

Anyway, configuring for a different database etc will not explain what the big difference in 4.2.d release Lucene Search Service start up is…


Regards

Steve

[shazada]

Hi Steve, Did you get it fixed? I have the same issue for a while at a customer.
And even setting the index.recovery mode to NONE doesn't help.
Total startup varies from 30 minutes to 45 min on a 4.2.e version.

I'll probably switch to SOLR in 6 months, but not at the moment.

[mrogers]

If its not a lucene re-index then where is the problem?  The alfresco log may help you narrow down what is taking time.

[eradan]

Hello,

same issue here on Alfresco 4.2.f and PostgreSQL and we use lucene (not solr) as search subsystem.

We uploaded 100.000 documents. We filed them in different folders in order to have less the 1.000 documents in a folder.
Even though, when Alfresco starts, it takes long time to be fully up and running.

A FULL REINDEX startup takes 4 hours to be accomplished!
Using index.recovery.mode = NONE it takes 1 hour!!

During startup I noticed that the server is busy: 98% of the CPU working on SELECT and many other IDLE database processes.
Any ideas of what's going on?

Thanks,
Ivan

[michel_s]

Hi,

We were having the same problem while migrating a huge repository instance from 2.2.8 to 4.2 (~ 1.5m nodes). Tracing the queries, I discovered this one :

SELECT node.id AS id,
         node.version AS version,
         store.id AS store_id,
         store.protocol AS protocol,
         store.identifier AS identifier,
         node.uuid AS uuid,
         node.type_qname_id AS type_qname_id,
         node.locale_id AS locale_id,
         node.acl_id AS acl_id,
         txn.id AS txn_id,
         txn.change_txn_id AS txn_change_id,
         node.audit_creator AS audit_creator,
         node.audit_created AS audit_created,
         node.audit_modifier AS audit_modifier,
         node.audit_modified AS audit_modified,
         node.audit_accessed AS audit_accessed
  FROM alf_node node
         JOIN alf_store store ON (store.id = node.store_id)
         JOIN alf_transaction txn ON (txn.id = node.transaction_id)
 WHERE      node.store_id = 63517816
         AND node.uuid IN ( ….. )
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The Oracle Explain Plan revealed that the 2 consecutives join have a cardinatily of about 21 billions rows due to a full table scan. In our test environment, this query will run on 42 hours…

Replacing the JOINs by LEFT JOINs gave better results : 3 seconds…

This can be changed in alfresco/ibatis/org.hibernate.dialect.Dialect/node-common-SqlMap.xml on the following queries :
<ul>
<li>select_NodeById</li>
<li>select_NodeByNodeRef</li>
<li>select_NodesByUuids</li>
<li>select_NodesByIds</li>
<li>select_NodesWithAspectIds</li>
<li>select_TxnNodes</li>
</ul>


Hope this will help…

[eradan]

Hello Michel,

thank you for reply!

We also tried to upgrade to Alfresco Community 5.0.a and the query issues have been resolved in that version.

Thanks,
Ivan

[ivan_plestina]

Hi,

We also hit this issue but not on all installations. Going through the MySQL slow log we noticed that this query is taking a lot of time to execute (though when we use explain, plan seems trivial):

select
            node.id             as id,
            node.version        as version,
            store.id            as store_id,
            store.protocol      as protocol,
            store.identifier    as identifier,
            node.uuid           as uuid,
            node.type_qname_id  as type_qname_id,
            node.locale_id      as locale_id,
            node.acl_id         as acl_id,
            txn.id              as txn_id,
            txn.change_txn_id   as txn_change_id,
            node.audit_creator  as audit_creator,
            node.audit_created  as audit_created,
            node.audit_modifier as audit_modifier,
            node.audit_modified as audit_modified,
            node.audit_accessed  as audit_accessed
     
        from
            alf_node node
            join alf_store store on (store.id = node.store_id)
            join alf_transaction txn on (txn.id = node.transaction_id)
        where
            node.store_id = 5 and
            node.uuid in 
             (  
                '0240cf0e-c8ce-4a80-bc0f-0ff64cef3874'
             , 
                '02526c27-b909-41c2-b3f2-2a468a0984d4'
             , 
                '035c7552-e0ee-4e88-a8e3-5f7ce30f0fd3'
             , 
                '03c6d216-d07f-4863-b50c-835b09cb5873'
             , 
                '03d0fe19-dfac-4899-a3d8-b7303ed5ecbe'
             , 
                '056f4e89-4292-47c6-a6e3-ae3ef7be55a7'
             , 
                '058416a9-b07b-4685-8f3d-c1bf62ec6633'
             , 
                '060d17d8-b8bc-41a3-acc1-e70bac7f7df1'
             , 
                '0633724b-3a71-45e3-9196-50a63e13ad98'
             , 
                '08f78446-8a0f-4bab-838e-3a02255cc935'
             , 
                '0a32e9e7-2c74-4dd7-ac37-aedc6aeecb6a'
             , 
                '0a3c3e37-a632-4e91-aaf0-dd2617cf8d1b'
             , 
                '0a825548-d8ed-4cd4-8b92-76ef9192198f'
             , 
                '0b786dcc-c79d-4336-b02e-16c0addb154e'
             , 
                '0c2d6390-3d67-4715-b995-354230030fde'
             , 
                '0de77e10-9dcc-4091-a0b1-4665fe479fea'
             , 
                '0e721e00-b015-4ca7-824d-68de1a763fe4'
             , 
                '11c510dd-7c9b-46b5-93a0-37a08553f862'
             , 
                '12d39bd1-aacf-48cb-b9c8-3232564b4f59'
             , 
                '13a6871d-c6ea-4ebd-bd84-2c7ff5c2081b'
             , 
                '14caff49-8b98-4a01-b313-3ad1d073db96'
             , 
                '14f472ff-a4ce-4061-a093-d4aa9725e066'
             , 
                '16813da6-a38d-4030-8d32-d10198206553'
             , 
                '18231a1b-8271-49aa-91d3-cc22928cf50b'
             , 
                '18996d90-7a9c-484b-82e6-3929f20c045f'
             , 
                '18b5a79d-37c0-4e37-8878-e7ec89cc1b5d'
             , 
                '18c7ba75-d0e4-4a5a-aa4e-4e1e8b099440'
             , 
                '1acd9d18-9f66-4cd1-a2bc-6a3a99834a77'
             , 
                '1b023b3a-8c26-4095-9794-8c12b42df41c'
             , 
                '1bb79ffc-ee3e-4817-9ff5-7de8d591086c'
             , 
                '1c21186b-419d-40b2-89bd-e4b22d5411aa'
             , 
                '1c9c9ed0-b06b-41b9-9fa7-4326d3ee5972'
             , 
                '1cbc6fdb-3f61-4a1e-9a86-7890372bda8c'
             , 
                '1cf19849-3510-41a5-8545-95304abc4402'
             , 
                '20987081-91a6-4e8b-81e4-23c181a1aa16'
             , 
                '21bc5863-a30b-4d87-8068-46dda7e0da4d'
             , 
                '2284125a-5410-445e-9688-b27b6ba72331'
             , 
                '23e4f44e-371f-4250-a165-d2a5a028c7b5'
             , 
                '2633cb60-8474-454a-9f00-d3c4db4ffdad'
             , 
                '26441ebd-9da2-43a1-89eb-1d3119cfef90'
             , 
                '26827e88-e2de-4958-a8b4-ab0b0dfba178'
             , 
                '26b87a28-d1a4-4870-b616-18bb86a96dcd'
             , 
                '2a094ad4-2c54-461f-bb13-d1ef7c7cf9b4'
             , 
                '2c1a3d12-2c30-417f-a9f5-ffcb3b7605ce'
             , 
                '2cee1317-e4bf-42ff-b48f-d0fb6a099908'
             , 
                '2e32ffb6-b034-41e0-887c-a18aa76a602d'
             , 
                '308ac89c-e42a-4de8-8a76-8278f7093d00'
             , 
                '32e86224-13bb-43f6-8c96-f235a735b7d1'
             , 
                '32f9b871-587d-4cbe-b4dd-38763b523aae'
             , 
                '35191f93-8c35-4ccc-ab01-82c92a4f64b7'
             , 
                '3772ac9e-4e94-4e38-ac24-fde86149105d'
             , 
                '37aeda2a-2f61-421e-87c9-68685fee7048'
             , 
                '37c172a3-a110-4769-b1d9-96e441521402'
             , 
                '38193b1e-44ad-48af-9a29-80141de1beac'
             , 
                '39d02591-f819-4e25-8212-bddf23b9e9db'
             , 
                '3b9d4efd-7aeb-43b1-8434-4f9ca476c622'
             , 
                '3c2e24f0-8c40-4a95-824a-22e6f47ff063'
             , 
                '3c766c59-1d08-43f7-842d-b34dbafde9e3'
             , 
                '3dad1851-133f-476e-8be7-f4a9da9a86fe'
             , 
                '3ec15e54-8d14-4e09-97b7-3dd45d58bfe5'
             , 
                '3feccb6c-8c03-4765-bb7d-1dcbe5fe4606'
             , 
                '43c52422-9ed9-4998-81ac-cabf2cd2a2a3'
             , 
                '43fa4757-6e53-474a-9247-47fe83f54c73'
             , 
                '452db52a-eb00-4b28-9430-ef9af15ce4a2'
             , 
                '465d2de9-80f4-4495-9507-691904a7801c'
             , 
                '48b86b2b-5a4f-4725-9e22-13135c9d9cb0'
             , 
                '48e7b9de-c115-4cff-94b8-8c484e4fd9d4'
             , 
                '4a137dde-0c2e-446f-9fd8-c0241ede4333'
             , 
                '4a869040-7ca2-4567-9f3d-605be494abf3'
             , 
                '4b1ebfe8-f4b6-4554-a5d5-28d032faa862'
             , 
                '4b2c1298-2da9-47d8-978f-657f40338f93'
             , 
                '4c1e3b11-19e2-467d-a4ae-8181ff3653a3'
             , 
                '4f1ff807-e180-4d6e-9141-630ef1f935e3'
             , 
                '50ee5115-954c-489d-9c50-9a39e11f9029'
             , 
                '52f7e6f0-de88-473b-a235-ec3a4d38953a'
             , 
                '54bcc7c9-3e34-4cb6-96a1-815b183a7dbd'
             , 
                '5801d82f-ecc0-404b-a1af-57f51d4b31f6'
             , 
                '589be353-454b-424e-8ac8-79c02e4c789d'
             , 
                '597bc87b-0420-4bcb-ba39-f94ad85473e2'
             , 
                '59d5cc3b-9d47-4fa9-a4da-b5e73bf3c6f2'
             , 
                '5ba32863-93e4-4e46-be34-a45bd284d314'
             , 
                '5bff1f6d-383b-4199-81e9-7b154168966c'
             , 
                '5ce45a2f-925c-4726-9a17-7696b11aadf1'
             , 
                '5d611392-e3f2-4627-89fa-c7f836084347'
             , 
                '5f95c3e7-842f-4aac-8be1-c2494bfe6827'
             , 
                '602f5ab8-88d7-4962-8ce3-31bee0a947b9'
             , 
                '60e70966-ce0e-490f-afef-4d2675c30e27'
             , 
                '624adec2-b044-4aa9-8870-347ad6659455'
             , 
                '629df595-72db-4ce3-8b2a-75b6c3af98c8'
             , 
                '6303df6b-3cae-43ad-bf78-2ca213669ff3'
             , 
                '658aa074-1f55-44ec-9254-84ccc244c480'
             , 
                '662967db-6c3f-4d89-98a3-819e752bd14d'
             , 
                '66d12175-d27f-4e16-b033-a40c08c4f795'
             , 
                '677345ee-bc59-4c0a-804f-ae07c9816e03'
             , 
                '67955e02-6307-4d92-bea5-0cc9fc445707'
             , 
                '67cc4d2a-36a6-4356-b279-7e5132c5f476'
             , 
                '6b963b87-2ccb-4562-a876-03bdc0504445'
             , 
                '6d7ac004-10fd-40fa-a31a-582310389460'
             , 
                '6dd31335-f925-442b-902b-d12aa311532c'
             , 
                '6e944dca-c22a-4c65-b680-d62ccd0ccd66'
             , 
                '7030a48f-d352-4864-b799-4f7b0d9f7211'
             , 
                '721aaa5a-8860-4b80-bb01-bdab0a25490a'
             , 
                '72856164-400f-475b-bee4-b6a458fdd7cd'
             , 
                '729eac9c-56a0-4f2f-a23c-2150aec1321b'
             , 
                '73710699-d2ad-442b-8ece-7c8ce2997a01'
             , 
                '74303e1d-2bd6-4331-8236-3fb7684de012'
             , 
                '7529a1e4-7c59-4c52-80cd-1ebc61abd7e2'
             , 
                '75fd4802-b720-4a74-87c8-4d8919c64281'
             , 
                '7600ce8f-e1cd-4484-9d98-6c6efabbc83f'
             , 
                '771f7198-a0d0-4fb0-af6f-4f87d2afb64d'
             , 
                '77278a51-d68f-43f6-afff-885f28aca2d3'
             , 
                '7971da4f-0ab4-4ae5-b0b1-20206ec0be3d'
             , 
                '7ad305b6-c1ec-46f6-b7b3-ac23b6bc717c'
             , 
                '7b520d21-6d50-4312-ab7d-acc3f3b9068d'
             , 
                '7bd2f7de-462b-4d1b-84a5-8aab115be28c'
             , 
                '7c70f57f-e443-4959-b3e1-557bf7e05ba0'
             , 
                '7ef3a3a0-a8a0-47ed-8cc1-f386c31e321f'
             , 
                '813fa8c8-3a72-4453-8f49-792dd83e0f2c'
             , 
                '81b744f7-ab02-4bd4-9f98-ce1c0cd8426c'
             , 
                '833c2808-9437-4228-9d19-cb3dbde4c1bc'
             , 
                '83b339ef-5c12-4d5d-be63-5ecea314223c'
             , 
                '8459b954-7d44-4ff4-9e8b-dcb962dd82ed'
             , 
                '85299443-d9f8-417b-b989-47e64930c8cf'
             , 
                '85399bc6-22ed-4c9d-91da-096dc0e247e4'
             , 
                '859ade33-4391-4658-b20c-a753b0b18eb3'
             , 
                '8605a8f2-70d2-4a3b-8b98-9e59e707ad72'
             , 
                '8661e4f0-bf20-4c69-8b04-c9fab5cf55f4'
             , 
                '8694e0ed-0272-49d3-9fbb-2b397586be3d'
             , 
                '8695cb46-b019-432d-86ed-a77362a7ce0b'
             , 
                '89bf3f04-05c1-42e4-892d-7cba0d7d16f6'
             , 
                '8b9e8cdb-65ff-4151-89e6-ad3e9de36201'
             , 
                '90702e4f-1725-4509-86ea-b95759324c67'
             , 
                '92a4c843-41de-4912-88a7-3e629b661c25'
             , 
                '943f9c9d-93ec-49e6-9cd1-482896e440e1'
             , 
                '94cbb60a-a2f6-4407-b16c-8109d5902edd'
             , 
                '94d9beb9-86a2-4736-9dfa-13ab7d301a38'
             , 
                '9523edc0-dd04-457d-882c-73004a41bc7a'
             , 
                '95c3a5bd-8953-413c-9544-4c536e842161'
             , 
                '995a35fa-0729-4c81-98bd-b73d03fb1caa'
             , 
                '99f749c4-5e6d-4a39-a4df-7e665499d10b'
             , 
                '9a2ed319-4644-47ef-9e4e-c7b03d0a14cf'
             , 
                '9aa6cacd-546d-48d8-9333-c554d805c51e'
             , 
                '9bff8b3f-53e8-439a-916d-9f5b88811f6a'
             , 
                '9c98e9cf-273d-4c26-a07d-83be8df35ada'
             , 
                '9cc5e9f7-495c-4bd3-ba1b-22a0133106d2'
             , 
                '9d161b36-e9ca-42a0-b5a4-00f61c9ff013'
             , 
                '9e2456ab-ccb6-4b6f-90e3-11dba9f931e7'
             , 
                '9eb6d773-e507-4771-a153-b240abe64276'
             , 
                '9ec38940-c1ad-42f7-a2af-e4911c7d4e8e'
             , 
                '9f2ae2d9-a19a-4be4-b510-084e3dc128ac'
             , 
                '9f33b515-dff4-4118-bf66-9ce13f8513cf'
             , 
                '9f5deac1-0091-4654-80eb-57d852786e04'
             , 
                '9fa31fb9-5ca1-47de-82b4-40e670d67379'
             , 
                'a05e79e6-4455-4e3a-904a-8abfba5b89ee'
             , 
                'a1667263-36e4-4830-8cc5-ea9c6a5d3778'
             , 
                'a1eed4ed-9cfe-4cf1-9e95-9827e1bdedcf'
             , 
                'a20d9809-8f68-4ed1-be14-b1b008ed825b'
             , 
                'a21d1f90-30c1-4ee7-b847-da07db559ad2'
             , 
                'a306e93e-4dfb-41a5-8065-b50a7f1a39fc'
             , 
                'a5307932-000b-4c68-ac85-8f0f4b2d599f'
             , 
                'a5488493-0e13-4816-bc48-eb41fbe19fcf'
             , 
                'a5ccc624-dd3b-4876-9127-399e0c6d53c1'
             , 
                'a8bc703f-c6e1-486c-8e60-cac7841cb37a'
             , 
                'a98e6f4f-6b09-4312-b3b2-807a0d1f65fa'
             , 
                'ae88b2cc-cb4a-4883-83fc-54b71aeb0c52'
             , 
                'b1c99a27-98b9-4e54-ab4a-7c2d9ae88448'
             , 
                'b2e90bb1-2bab-4e29-9153-da6bfffd3b7e'
             , 
                'b35ee13f-2c7a-4ab5-9b83-3dc9302feaa9'
             , 
                'b3a07240-9394-4f65-be7b-c7bb0b493611'
             , 
                'b3cb6717-347c-4db7-8fd1-c4fa2ffd6bf5'
             , 
                'b4afa3f5-bb14-4fdd-a8c7-b4fda000be21'
             , 
                'b4e920fd-58e5-4cfd-a5c8-0beebee359df'
             , 
                'b8628419-5e74-4876-ac10-e39114115d5d'
             , 
                'b8d32202-ca57-4937-95a4-a23f8ad6701e'
             , 
                'ba73dbbb-0f67-42dc-8ca7-44ccf1c23c52'
             , 
                'bb21c4c9-f069-4eb4-9b52-b20f0245f5ec'
             , 
                'bb5b9310-6305-42c9-bc0c-10894a7554cd'
             , 
                'bbbcdb2f-2b54-49d0-8909-2ce82d98598c'
             , 
                'bc77c970-3ed1-41fe-a80e-ac97d7718f59'
             , 
                'bce9b67f-08f3-4fcc-860e-042575bb715b'
             , 
                'bcf1c7d1-b913-402e-a5d9-534a2ad8d47d'
             , 
                'bfff1ac7-dcdf-497e-9fe7-da67f460cdb8'
             , 
                'c047ff2f-ea01-4581-b25d-c9c68c490a4d'
             , 
                'c0a40b21-d4b8-469e-868b-7320bc96c640'
             , 
                'c14aba7c-10d1-4e78-9bd8-335113764571'
             , 
                'c1de0b63-4b2c-42ef-bbf2-6792ac84a806'
             , 
                'c2c0ce6a-30b9-4907-b73d-87956e490b7d'
             , 
                'c4062dfd-a00c-4b14-aef7-2b17d5f2d6fe'
             , 
                'c43745f7-ea74-4528-a8ea-eafcd635282b'
             , 
                'c7583cc8-8c43-4465-81b3-57e100072134'
             , 
                'c80062f5-029d-4018-b174-fefa87e6a9c8'
             , 
                'c8f8d4c8-575a-4e13-802a-02b3cd6d55b6'
             , 
                'cc727b65-2efd-4312-9c54-4d0d10cf67a3'
             , 
                'ccd07a63-8ee1-4f26-96b3-47da81f33382'
             , 
                'ce7332c3-54f3-4b5e-b2dd-38096fdbbae5'
             , 
                'cf472f2d-edbc-4f1a-a1c6-0b167f3f31e4'
             , 
                'cf62b08e-0816-4348-a5a7-febd5e08d42d'
             , 
                'd01e9cd2-6dcf-4e5e-9511-a797b08d2ad5'
             , 
                'd09c097c-306d-4da6-8880-02157c71a880'
             , 
                'd123c327-dcd2-480e-b901-d3df46efe4fe'
             , 
                'd23005b0-0c8e-440d-8954-59cb83aa2c3c'
             , 
                'd23dc92d-b4cf-47a5-b950-3e6d967ab1d2'
             , 
                'd51595bc-df3a-4688-b9aa-375db58a3dba'
             , 
                'd57e8e43-b6cc-485f-a4f2-63dcc4cb5ef6'
             , 
                'd63cbd21-dc86-426f-80d3-07930f0d27aa'
             , 
                'd6e24f6d-1441-4689-a2be-716d08f60ce7'
             , 
                'd87da8eb-2dfb-48d7-9a1b-65058bbc82ed'
             , 
                'd94f2e38-92ce-435f-8716-75a3f93e529b'
             , 
                'd9e6bc3e-91d2-4fa1-b7f5-130bb70303dc'
             , 
                'db017523-b517-4cf5-b423-e37fdebe86d5'
             , 
                'dc4853a6-de88-4f77-adc4-964db93c2e11'
             , 
                'dd384a84-934e-48c0-a870-9957edbab126'
             , 
                'de45ad34-85d9-4d8c-b830-3fdae389fbde'
             , 
                'e06ce267-d114-43db-b023-5eb0e2581b2c'
             , 
                'e08a20a2-71ac-4f0b-b13d-103bf5d44fff'
             , 
                'e0c1807c-c37a-4357-b3a0-042c224ea970'
             , 
                'e101245a-441a-4093-8b6f-1ef3b064b3f3'
             , 
                'e16c28c8-ff55-4954-a562-3c04819985f1'
             , 
                'e1e254d6-df33-47f1-92b8-eee41df285e9'
             , 
                'e253d9d7-796b-465e-a7c6-1f0708c5e04a'
             , 
                'e2dbd6d1-8f15-4964-9471-8c9d99138e54'
             , 
                'e3b2f73b-c5db-4337-aa4b-cb01294daa21'
             , 
                'e4a799d9-11a5-4d8e-ae33-6f67b9fd31c0'
             , 
                'e56d0082-bbc9-40d6-8ed4-442f98c1c3c1'
             , 
                'e598b7f5-30c5-4f4f-aeda-5cff346d4668'
             , 
                'e67231a9-b223-40ad-a257-d40e9f8c9e95'
             , 
                'e6a23b79-6bb6-4a0a-915d-d06f6aa4152e'
             , 
                'e85e92b9-c744-446f-bf23-c009e3f4cfbd'
             , 
                'ea4051d7-5b1c-4107-928d-e914ca1cac73'
             , 
                'ea4d436e-7cb8-4e2e-bcaa-6db65cd3dad9'
             , 
                'eaedbb3c-4187-4e90-aa60-46254e8bd8af'
             , 
                'eb0d67ed-1655-4870-8dfc-fcdffa83f9fe'
             , 
                'eb37f3e8-5263-4cd0-af10-bcc29df732e7'
             , 
                'ef5df55c-76aa-4a28-97c5-8bd6523a1bb3'
             , 
                'ef905a33-66ff-47fd-a859-d129a9e0f02c'
             , 
                'f00e5b2f-7527-4c75-b676-03b830adb27f'
             , 
                'f08391e7-7aec-46b1-8c65-06db59ad9b86'
             , 
                'f15a6ac7-15ce-4b1d-8a76-92466824bc81'
             , 
                'f1eb21fc-1dca-463f-82b6-cb5ab93babeb'
             , 
                'f1ebec5c-7586-442d-bf01-0c1d460cbda0'
             , 
                'f3e0c57d-417f-4d3a-a933-21bbf5906839'
             , 
                'f482fafb-cdae-48ca-9528-957fbd2a0f03'
             , 
                'f511cf72-979e-4898-adc1-c3fb61aeba5c'
             , 
                'f68a2d92-4a01-42dc-ad41-d813af640e06'
             , 
                'f7540cb4-8af1-401b-9971-dc391083d4e4'
             , 
                'f838375d-950f-4dc9-b4e5-0283288de056'
             , 
                'f83bc757-0af7-4283-97cd-3e4c8fbfb410'
             , 
                'f8d4e136-57f2-4a1c-adc5-512697edcccb'
             , 
                'f90fce2d-339e-4dac-8ea3-ee9b12f1bace'
             , 
                'f92c2e1a-817b-4b7e-be35-df1ed12cee4b'
             , 
                'faa609d9-d94c-42b6-a713-a6c054823ae9'
             , 
                'fb0fcc51-0f1f-4866-ad10-5c3833d2b3ec'
             , 
                'fc03d3d8-2e97-4095-b8e6-ae1f6d5b2387'
             , 
                'fdf40ff9-0bb1-4e68-b30e-fe37f6a11e6e'
             , 
                'fe5d8967-c688-4323-9c54-ad8b3551595a'
             , 
                'ffcd0b92-56b4-4976-a9b4-eb484ffb4b72'
             );
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

store_id 5 is actually the archive store. Checking to see how many deleted docs are there, and there are alot of them:

mysql> select count(*) from alf_node where store_id=5
+———-+
| count(*) |
+———-+
|   373344 |
+———-+
1 row in set (0.34 sec)
‍‍‍‍‍‍‍‍‍

On installations where we don't experience slow startup, there are no that many nodes in archive store. Can anybody please confirm the same? Solution would be simple then - just purge the deleted nodes…