Scaling live-indexing-mediation

[agermain]

Hi everyone,

I’m looking for some insights on how others handle this situation. We are using Alfresco 23.x with Alfresco Search Enterprise and have been experiencing issues during periods of high document ingestion. When the ingestion rate increases, we observe a significant delay in the replication between our OpenSearch indexes and the Alfresco database.

After further analysis, we found that once our Alfresco instances finish publishing events to the alfresco.repo.event2 topic, a large number of messages remain marked as “inFlight”. In other words, many messages are waiting to be ACKed by the live-indexing-mediation service.

Our OpenSearch cluster is barely doing any work during these peaks, and the live-indexing-metadata service is essentially waiting for live-indexing-mediation to process events. For context, we only index metadata, no content or path indexing.

Since live-indexing-mediation is a singleton service, how can we improve its throughput or overall performance? Has anyone faced a similar bottleneck, and how did you approach or resolve the situation?

Thanks in advance for any advice or shared experience!

[angelborroy]

This may seem naive, but have you tried increasing the number of concurrent consumers on the mediation inbound channel (in.alfresco.event.channel)?

Even though live-indexing-mediation runs as a singleton, it can process multiple events in parallel. By default, the mediation inbound channel is quite conservative, and under high ingestion rates it easily becomes the bottleneck you describe.

In practice, adding concurrentConsumers to the mediation channel often unlocks a lot of throughput, for example: &concurrentConsumers=8

[agermain]

Hi Angelborroy,

This is not naïve at all. I also used to assume that having multiple concurrent consumers on a topic would simply duplicate each event once per consumer.

A few days ago, I tested it with 20 concurrent consumers and initially thought that each event was being multiplied by 20 — meaning that for every single event in the topic, I saw 20 processed messages in the org.alfresco.search.medata.event. I may have jumped to conclusions, though, so I’ll run the test again on Monday morning and update this post with accurate results.

For reference, here is the configuration I used:

-Dalfresco.event.topic=activemq:topic:alfresco.repo.event2?concurrentConsumers=20

[angelborroy · Accepted solution]

I was reviewing the implementation and I have to say that this behaviour is actually intentional.

The live-indexing-mediation service consumes repository events using a durable topic subscription on alfresco.repo.event2. Because of that, the broker explicitly forbids multiple concurrent consumers on the same durable identity, which is why concurrentConsumers > 1 fails with the “durable consumer is in use” exception.

The original choice of a durable topic subscription was made to guarantee no loss of repository events across restarts or outages, deterministic order processing of events and exactly one logical consumer performing indexing side effects

From a repository consistency point of view, this is the safest possible default: losing or reordering events is worse than being slow.

That said, this design also means that classic JMS 1.1 durable subscriptions do not scale horizontally. Increasing concurrentConsumers on the same durable subscription is fundamentally incompatible with how JMS durable topics work.

To increase throughput, you may move concurrency after message consumption (parallel processing inside the mediator) or switch to a broker-level fan-out pattern (for example queues / virtual destinations).

[agermain]

Hi Angelborroy,

Thank you again for your time.

Regarding parallel processing inside the mediator, I don't believe I have access to the underlying code since it's a paid service.

However, using virtual destinations does seem like a promising approach.

This will allow the mediator to consume from a queue and therefore add multiple consumers.

I’ll explore that option further. Thanks a lot for your help!

[agermain]

Hi everyone,

I’ve tried several approaches, but unfortunately none of them worked.

I was originally using version 4.1 of search-service-enterprise, so it seems that the property in.alfresco.event.channel did not exist at that time. I upgraded to 5.2 to run new tests.

I attempted to override in.alfresco.event.channel with the following value:

-Din.alfresco.event.channel=consumer-sjms:topic:alfresco.repo.event2?durableSubscriptionName=LiveIndexingSubscription&clientId=LiveIndexing&concurrentConsumers=10

However, I consistently get an error whenever concurrentConsumers is greater than 1:

Caused by: jakarta.jms.JMSException: Durable consumer is in use for client: LiveIndexing and subscriptionName: LiveIndexingSubscription

It looks like it’s not possible to have multiple consumers on the same clientId (which makes sense for durable subscriptions).
Is there another way to achieve concurrency in this setup?

Thanks for your help!