Load+Concurrency Testing Utility

[hubick]

Hi,

I have created a small CMIS based utility for performing load and concurrency testing.

You can find the code at: https://github.com/hubick/au-cms-util-test-concurrency

This test creates a number of simultaneous CMIS clients, where each creates a document, repeatedly increments a number in the title+content while (optionally) rotating through the nodes in the server cluster for each increment (also verifying the previous update), deletes that document, and does it again with a new document.  The number of cluster nodes, clients, documents, and increments are all configurable.

If you navigate to the project page, you can click on 'tags' to grab a zip file of the latest release, which should build using Maven (see included README.txt for details).

I would be very curious to see the kinds of numbers other people are getting on various hardware+database configurations.

[hubick]

We are just building a new 2-node 64-bit Intel+RHEL6 server each w 64GB RAM using Alfresco Enterprise 3.3.5 and PostgreSQL.

We haven't done much performance tuning yet, but the following are the current numbers using 2, 5, 10 (default), and 20 concurrent clients:

Concurrency test completed, with 2 concurrent clients spread across 2 servers, each client performing 5 increments to each of 20 documents in a total of 93 seconds (average 4.6 seconds/document).

Concurrency test completed, with 5 concurrent clients spread across 2 servers, each client performing 5 increments to each of 20 documents in a total of 211 seconds (average 10.6 seconds/document).

Concurrency test completed, with 10 concurrent clients spread across 2 servers, each client performing 5 increments to each of 20 documents in a total of 383 seconds (average 19.1 seconds/document).

Concurrency test completed, with 20 concurrent clients spread across 2 servers, each client performing 5 increments to each of 20 documents in a total of 801 seconds (average 40.0 seconds/document).

Those numbers show that as the number of concurrent clients increases, we see an almost exactly corresponding linear increase in time/document

[afaust]

Hello,

just doing a quick look over the source code, this appears not to be too surprising to me. As far as I glanced over the code for the test client, it attempts to create / delete a file in one and the same folder through concurrent clients. This of course is bound to run into synchronisation issues on the database, as all concurrent clients attempt to modify the child associations of the folder at the same time. The more clients attempt this, the longer the "waiting line" in the database locking mechanism, which directly translates in to the linear increase in time / document.

To really test concurrent usage / load, tests should aim to realistically distribute content manipulation over a wide range of the content hierarchy and not be confined to selected subtrees. The Alfresoc remote BM e.g. distributes across content stores and multi-layered folder structures.

Regards
Axel

[hubick]

As far as I glanced over the code for the test client, it attempts to create / delete a file in one and the same folder through concurrent clients. This of course is bound to run into synchronisation issues on the database, as all concurrent clients attempt to modify the child associations of the folder at the same time. The more clients attempt this, the longer the "waiting line" in the database locking mechanism, which directly translates in to the linear increase in time / document.

To really test concurrent usage / load, tests should aim to realistically distribute content manipulation over a wide range of the content hierarchy and not be confined to selected subtrees. The Alfresoc remote BM e.g. distributes across content stores and multi-layered folder structures.

This is great feedback, thanks!

I'm fairly sure the majority of the time for the test comes from the increments to the metadata/content, not creation/deletion of the documents.  Regardless, I have tagged version 1.1, where each client now creates/increments/deletes files within it's own folder created underneath the main test folder.

I don't think the test still being rooted in the same subtree should have any real impact, as everything ultimately shares one root folder anyhow, all just being nodes and relationship rows in the database, regardless of path within the repository.  I'm hoping the additional node of separation should provide enough isolation between each client to ensure child-associations aren't a potential issue now.

That said, with these changes in place, I experienced no appreciable difference in runtime using 10 clients

[hubick]

I posted a new 1.3.0 release of my utility after testing it against the new OpenCMIS implementation in Alfresco 4.0.0.  As a workaround for https://issues.alfresco.com/jira/browse/ALF-11263 it now has configuration properties for manipulating the versioning policy used to create/increment documents.  This release also trims out some extraneous update/refresh requests, allows you to configure the number of subfolders used to distribute the test load across the repository, and the logging is improved as well:

Concurrency test completed, with 10 concurrent clients spread across 5 folders on 2 servers [http://mycluster01/alfresco/service/api/cmis, http://mycluster02/alfresco/service/api/cmis] running 'Alfresco Repository (Enterprise)' version '3.3.5 (383)', each client performing 5 increments (not explicitly versioned) to each of 20 documents (versioning='none') in a total of 319 seconds (average 15.9 seconds/document).