Re: Large number of EventManager threads causing OutOfMemoryException

From: Andrus Adamchik (andru..bjectstyle.org)
Date: Thu Apr 29 2010 - 21:47:49 EDT

  • Next message: Andrus Adamchik: "Re: Spring + Cayenne"

    Hi Vic,

    There are two possible explanations - EventManager leaking threads
    (seems rather unlikely) or you are starting multiple Cayenne stacks
    (or create multiple EventManagers as a side effect of some other
    action).

    A few ways to detect this from a thread dump:

    1. Check the names of the cayenne-edt- threads. Specifically check the
    last part of the name - the number after the last dash. E.g. in
    "cayenne-edt-16165743-0", this number is "0". If this number is non-
    repeating and increments indefinitely, that may indicate a leak,
    however if this number is something rather low (e.g. between 0 and 4)
    and repeats many times, this means that multiple EventManagers are
    started somehow.

    2. Check the address of the lock of the even thread pool:

    > org.apache.cayenne.event.EventManager
    > $DispatchThread.run(EventManager.java:476)
    > - locked <0x96d28c40> (a
    > java.util.Collections$SynchronizedList)

    (it is "0x96d28c40" in the example above). If it is the same for all
    threads, then it is a leak. If it is different for most threads, then
    you have multiple EM's.

    Andrus

    On Apr 30, 2010, at 1:07 AM, Victor Leung wrote:

    > Hi all,
    >
    > We are using Cayenne 3.0RC3. Our application has a web tier and a
    > business /
    > data access tier (both deployed on Glassfish v2.1.1). Communication
    > between
    > the two tiers is through the use of Cayenne Web Service.
    >
    > We have been encountering OutOfMemoryException after a couple of
    > days of
    > routine usage. Heap dumps show a large number of daemon threads
    > associated
    > with EventManager. As an example, there are some 13,000 entries in
    > the heap
    > dump similar to this:
    >
    > "cayenne-edt-16165743-0" daemon prio=10 tid=0x576d3800
    > nid=0x5b2a in Object.wait() [0x847ad000]
    > java.lang.Thread.State: TIMED_WAITING (on object monitor)
    > at java.lang.Object.wait(Native Method)
    > at
    > org.apache.cayenne.event.EventManager
    > $DispatchThread.run(EventManager.java:476)
    > - locked <0x96d28c40> (a
    > java.util.Collections$SynchronizedList)
    >
    > There are no evidence of any deadlocked threads.
    >
    > We have since turned on monitoring on the JVM. It appears that the
    > number of
    > daemon threads stays more-or-less constant whilst we are just
    > reading from
    > the DB, but will creep up slowly over time whenever we do any sort
    > of DB
    > updates.
    >
    > Any hints as to how we can get around this problem would be much
    > appreciated!
    >
    > Thanks in advance,
    > Vic



    This archive was generated by hypermail 2.0.0 : Thu Apr 29 2010 - 21:48:43 EDT