Do you see DataContext instances in the memory profile? I wonder how
many of those do you have, as those are the only Cayenne objects that
have hard reference to the listeners.
Andrus
On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
> Hi,
>
> the JVM has 1 GB of memory and i already tried to run the GC
> manually but it doesn't make a difference. I can start the GC from
> the profiler tool but the number of instances does not change.
> I guess that there are some other references to this objects so
> that the weak ref. does not expire.
>
> And there are no EventManager exception in the log files.
>
>
> ayhan
>
>> -----Ursprüngliche Nachricht-----
>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>> Gesendet: Mittwoch, 21. Februar 2007 14:17
>> An: use..ayenne.apache.org
>> Betreff: Re: possible bug / memory leak in DispatchQueue and
>> EventManager?
>>
>> Seems like your assessment of the EventManager leaking is correct.
>>
>> Now the cause is not that clear. A shot in the dark - this is due to
>> a combination of lots of spare memory in JVM (so weak references are
>> not collected fast enough) and slow custom 'equals' and 'hashCode'
>> methods in invocation.
>>
>> How much heap size do you have in your server? Also can you try to
>> add a request filter that would do 'System.gc()' on every few
>> thousands requests, and see if it makes any difference.
>>
>> Also - do you see any EventManager exceptions in the logs?
>>
>> Andrus
>>
>>
>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
>>> Hi,
>>>
>>>
>>>
>>> i think i found a possible bug / memory leak within the
>>> DispatchQueue and EventManager.
>>>
>>>
>>>
>>>
>>>
>>> First a little bit about my setup:
>>>
>>>
>>>
>>> I have 3 servers. Each server runs an axis service. The service
>>> uses cayenne 1.2.1 to connect to a database. It reads customer and
>>> account information from the DB etc..
>>>
>>> The servers are using cayenne's shared caching with javagroups as
>>> the messaging service so that changes made from one server are
>>> dispatched to the other servers.
>>>
>>>
>>>
>>> The avarage connections per second is somewhere around 4-5.
>>>
>>>
>>>
>>> However I have a very strange problem with this setup so I startet
>>> to search for the reason. The problem is that with nearly constant
>>> and unchanging usage the load of each server increases over time.
>>> To further test this I created a test server with a similar setup.
>>> There I created a test program that creates totally constant usage.
>>> But even with the unchanging usage the load of the server is
>>> increasing until the cpu load is so high that the requests can not
>>> be processed anymore.
>>>
>>>
>>>
>>> I installed a java profiler to trying to pinpoint the location of
>>> this error and this is what I found out.
>>>
>>>
>>>
>>> I let the server run for 24 hours and then stopped the program
>>> which creates the test usage.
>>>
>>> But even while the server was idle there where still a lot of
>>> instances in the java heap after the GC run.
>>>
>>>
>>>
>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
>>>
>>>
>>>
>>> Please note the HashMap, WeakReference and Invocation counts. I
>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
>>> not sure.
>>>
>>>
>>>
>>> Now the following image shows the cpu profile with incoming
>>> connections.
>>>
>>>
>>>
>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
>>>
>>>
>>>
>>> As you can see 58% of the cpu time is used within HashSet.add().
>>>
>>>
>>>
>>> So when I consider the two facts i think that there might be a
>>> possible problem with the EventManager. The first table tells us
>>> that there are over 1 million instances of HashSet's and cayenne
>>> Invocations. So it seems like the set's within the DispatchQueue
>>> are not recycled properly so that the object count rises over time
>>> which result in extremly high processtime when trying to add new
>>> HashSet's.
>>>
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> Ayhan
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Ayhan Kondoz
>>>
>>>
>>>
>>> Software-Entwicklung
>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> --
>>> ------------
>>>
>>> Telefon: +49 (0) 40 513 06 616
>>>
>>> Telefax: +49 (0) 40 513 06 998 616
>>>
>>> E-Mail: ayhan.kondo..reenet-ag.de <mailto:ayhan.kondoz@freenet-
>>> ag.de>
>>>
>>> Website: http://www.freenet.de <http://www.freenet.de/> ; http://
>>> www.freenet-ag.de <http://www.freenet-ag.de/>
>>>
>>> --------------------------------------------------------------------
>>> --
>>> ------------
>>>
>>> freenet.de AG
>>>
>>> Deelbögenkamp 4c
>>>
>>> 22297 Hamburg
>>>
>>> --------------------------------------------------------------------
>>> --
>>> ------------
>>>
>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
>>>
>>> Vorstand: Eckhard Spoerr (Vors.),
>>> Axel Krieger, Stephan Esch, Eric Berger
>>>
>>> Amtsgericht Hamburg HRB 74048
>>>
>>>
>>>
>>>
>>>
>>> Diese Information ist ausschließlich für die adressierte Person
>>> oder Organisation bestimmt und könnte vertrauliches und/oder
>>> privilegiertes Material enthalten. Personen oder Organisationen,
>>> für die diese Information nicht bestimmt ist, ist es nicht
>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
>>> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese Nachricht
>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem Absender in
>>> Verbindung zu setzen und das Material von Ihrem Computer zu löschen.
>>>
>>>
>>>
>>> The information transmitted is intended only for the person or
>>> entity to which it is addressed and may contain confidential and/or
>>> privileged material. Any review, retransmission, dissemination or
>>> other use of, or taking of any action in reliance upon, this
>>> information by persons or entities other than the intended
>>> recipient is prohibited. If you received this in error, please
>>> contact the sender and delete the material from any computer.
>>>
>>>
>>>
>
>
This archive was generated by hypermail 2.0.0 : Wed Feb 21 2007 - 08:49:56 EST
