Hi Andrus,
Thank you -- you've hit the nail right on the head!
In our web tier, we've been starting up a new ClientChannel when creating
each session-specific CayenneContext. I've since changed the code so that
all CayenneContexts are now created using a shared ClientChannel instance
(Singleton pattern).
I've run through a couple of user scenarios, and the server's thread count
seem to have stabilised. We'll try and run through some soak testing in the
next 2 weeks, but I'm pretty confident that we've resolved the problem.
Thank you once again.
Cheers,
Vic
On Sat, May 1, 2010 at 3:12 PM, Andrus Adamchik <andru..bjectstyle.org>wrote:
> Or actually, client can be a problem as well. Each ClientChannel that you
> create would start an EventManager. So you may want to do
> ClientChannel.getEventManager().shutdown() before a ClientChannel goes out
> of scope.
>
> Andrus
>
>
>
> On May 1, 2010, at 1:09 PM, Andrus Adamchik wrote:
>
> I don't think running things on the same server vs. separate server is an
>> issue. Hw is your business tier implemented though? Is that an EJB or a web
>> app? Something in its lifecycle makes Cayenne start multiple times and this
>> is what you need to figure out.
>>
>> Andrus
>>
>> On Apr 30, 2010, at 11:14 AM, Victor Leung wrote:
>>
>> Hi Andrus,
>>>
>>> Thank you for your prompt and insightful reply!
>>>
>>> Looking through the thread dumps as suggested, it definitely seems like
>>> we
>>> are starting multiple Cayenne stacks / EventManagers -- on point (1), the
>>> number after the last dash is rather low, and on point (2), the address
>>> of
>>> the lock are pretty much all different.
>>>
>>> In our current test environment (where the problem is occurring), we have
>>> deployed both the web tier and the business tier as separate EARs on the
>>> same server. As mentioned earlier, communication between the two tiers is
>>> through the use of Cayenne Web Service. Is this a supported
>>> configuration,
>>> or do we need to deploy the two EARs on separate servers?
>>>
>>> Thanks again,
>>> Vic
>>>
>>>
>>> On Fri, Apr 30, 2010 at 11:47 AM, Andrus Adamchik <
>>> andru..bjectstyle.org>wrote:
>>>
>>> Hi Vic,
>>>>
>>>> There are two possible explanations - EventManager leaking threads
>>>> (seems
>>>> rather unlikely) or you are starting multiple Cayenne stacks (or create
>>>> multiple EventManagers as a side effect of some other action).
>>>>
>>>> A few ways to detect this from a thread dump:
>>>>
>>>> 1. Check the names of the cayenne-edt- threads. Specifically check the
>>>> last
>>>> part of the name - the number after the last dash. E.g. in
>>>> "cayenne-edt-16165743-0", this number is "0". If this number is
>>>> non-repeating and increments indefinitely, that may indicate a leak,
>>>> however
>>>> if this number is something rather low (e.g. between 0 and 4) and
>>>> repeats
>>>> many times, this means that multiple EventManagers are started somehow.
>>>>
>>>> 2. Check the address of the lock of the even thread pool:
>>>>
>>>>
>>>>
>>>> org.apache.cayenne.event.EventManager$DispatchThread.run(EventManager.java:476)
>>>>
>>>>> - locked <0x96d28c40> (a
>>>>> java.util.Collections$SynchronizedList)
>>>>>
>>>>>
>>>> (it is "0x96d28c40" in the example above). If it is the same for all
>>>> threads, then it is a leak. If it is different for most threads, then
>>>> you
>>>> have multiple EM's.
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Apr 30, 2010, at 1:07 AM, Victor Leung wrote:
>>>>
>>>> Hi all,
>>>>
>>>>>
>>>>> We are using Cayenne 3.0RC3. Our application has a web tier and a
>>>>> business
>>>>> /
>>>>> data access tier (both deployed on Glassfish v2.1.1). Communication
>>>>> between
>>>>> the two tiers is through the use of Cayenne Web Service.
>>>>>
>>>>> We have been encountering OutOfMemoryException after a couple of days
>>>>> of
>>>>> routine usage. Heap dumps show a large number of daemon threads
>>>>> associated
>>>>> with EventManager. As an example, there are some 13,000 entries in the
>>>>> heap
>>>>> dump similar to this:
>>>>>
>>>>> "cayenne-edt-16165743-0" daemon prio=10 tid=0x576d3800
>>>>> nid=0x5b2a in Object.wait() [0x847ad000]
>>>>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.cayenne.event.EventManager$DispatchThread.run(EventManager.java:476)
>>>>> - locked <0x96d28c40> (a
>>>>> java.util.Collections$SynchronizedList)
>>>>>
>>>>> There are no evidence of any deadlocked threads.
>>>>>
>>>>> We have since turned on monitoring on the JVM. It appears that the
>>>>> number
>>>>> of
>>>>> daemon threads stays more-or-less constant whilst we are just reading
>>>>> from
>>>>> the DB, but will creep up slowly over time whenever we do any sort of
>>>>> DB
>>>>> updates.
>>>>>
>>>>> Any hints as to how we can get around this problem would be much
>>>>> appreciated!
>>>>>
>>>>> Thanks in advance,
>>>>> Vic
>>>>>
>>>>>
>>>>
>>>>
>>
>>
>
This archive was generated by hypermail 2.0.0 : Sun May 02 2010 - 08:24:01 EDT