IMAP and memory usage

[phileas]

Hi,

I've installed Alfresco CE 4.0d on a machine running tomcat 6 with Sun/Oracle Java 1.6.

There are about 40 users how regulary (Mac Mail / Thunderbird almost always open all day) use Alfresco's IMAP Server.

The problem is that the system is always running out of memory after a few days. (4GB java heap with 1GB NewGen)

Analysis of a java heap dump after the system was running for about one and a half days shows the greenmail ImapServer / ImapHandler class as possible memory leak. There are more than 500 instances of ImapHander using more than 800MB of RAM.
OldGen never really stabilizes but keeps growing until something like "java.lang.OutOfMemoryError: GC overhead limit exceeded" happens.

Could this really be a leak or is this much memory consumption to be expected for this number of IMAP connections?
How much memory should I assign for the JVM or is there some other tuning I have to do?

I would be happy to hear any suggestions.

Best Regards,

p.

[afaust]

Hello,

4 GiB heap for a 40 users installation are more than enough - from my experience, I would only increase NewGen to about 2 GiB (most of the memory used is only used for short lived data structures, and increasing NewGen can improve performance as not as much "garbage" is promoted to OldGen whose clean-up is more expensive).

Are you able to analyse your heap dump in more detail? 500 instances with 800MiB RAM in a 4 GiB environment do not sound like they may be the biggest fish / main culprit. It would be good to know what else is kept in memory (may be transitively via the ImapServer) and how much memory space is used. Tools like the Eclipse Memory Analyzer tool are great for investigating this.

Are you able to reproduce the increased memory footprint in a controlled environment? Were you able to identify a particular usage pattern that shortens the time until your memory runs out?

Regards
Axel

[phileas]

Hi,

thanks for your help!

The 500 instances / 800MiB RAM of Imap were from a heap dump taken some time before the system crashed. Eclipse Memory Analysis identifed those classes as a possible memory leak.
I am right know analysing another dump but this takes some time on my machine.

What I see on the live system is that especially when users start opening their mail clients in the morning and also a bit after lunch the heap grows fast (OldGen that is) more then 500MiB in 10 minutes. Otherwise it grows constantly at about 100MiB per hour. Forcing a GC via jconsole takes about 20 seconds and only frees up maybe 10MiB.

After running without restart since friday the alfresco has now 3.77 GiB of OldGen (I've raised Max Heap to 5GiB) and still rising.

While I'm waiting for the heap analysis to finish:

Are there any tunables in the IMAP subsystem which influence e.g. how long IMAP connection information is cached or anything like that?

Best Regards,

p.

[afaust]

Hello,

unfortunately, I don't see any configuration options that would relate to memory tuning of the IMAPServer.

Just out of curiosity and in order to help identify the root cause: Would it be possible to provide a heap dump (e.g. via the Alfresco Cloud) for distributed analysis? Before answering to hastily, please bear in mind that it may be possible to extract some potentially sensitive information from the dump (usually not passwords, but content of emails e.g.).

Since MAT has identified these instances / classes as a potential memory leak, you need to identify what holds them in memory (why your GCs do not remove them). Usually, the identified elements are just the symptom / largest affected elements of a leak, but not the leak itself. This involves analysing the "Paths to GC Roots" for those 500 elements.

Regards
Axel

[phileas]

Hi,

I can not give you the heap dump because of the e-mails that are potentially in there. But maybe you can help me make sense of what I see in the MAT.

I have opened another heap dump. It shows me that at it's creation time there where 2.8 GiB of 4GiB filled. 1.8 GiB of this belong to the class com.icegreen.greenmail.imap.ImapServer which is "Problem Suspect 1" the rest is ehcache and smaller stuff.

In the "accumulated objects" I see com.icegreen.greenmail.imap.ImapServer -> java.util.Vector -> java.lang.Object -> and then 987 objects of class
com.icegreen.greenmail.imap.ImapHandler. That's also the path to gc roots.

Additionaly I did a "netstat -tupn | grep 143" on the server and tracked it over time. Right now there are 149 open connections to the imap port on the server and those where also open thru the night. But nobody hat an e-Mail client connected in the night. And there was no incoming traffic on port 143.

So for me it seems that the imap connections are for some reason never really closed (maybe on the client side) and alfresco keeps them alive indefinitely. Hence the growing memory.

Shouldn't alfresco dump those connections after some time?

Best Regards,

p.

[afaust]

Hello,

I have just checked the code for ImapServer / ImapHandler (this is not Alfresco code, but a 3rd party library used in Alfresco). It turns out ImapHandlers in the handlers list are only fully removed when the ImapServer is shut down. The handlers vector grows indefinitely until that point. This should usually not be a major issue, since once a particular connection is closed, the ImapHandler resets its internal state which should free most of the associated memory. Can you check the ImapServer / ImapHandler for individual instances that stick out due to their size? I would expect a majority of very small instances and some significantly larger. The latter are the ones that warrant further investigation.

In terms of connections being kept alive: As soon as a client cuts the connection (clean disconnect e.g. by mail client or forced disconnect when the OS shuts down), this should be automatically detected by Alfresco (as the handlers constantly check for new command input) and also free the socket on the server. Have you performed a cross-check on the clients the connections are reported to be comming from? I know, this is rather tiresome for 149 connections, but maybe you discover a common pattern before you reach 15.

Regards
Axel