I didn't. I'd appreciate if you could do that, as you have all the
details.
http://issues.apache.org/cayenne/
Thanks
Andrus
On Mar 15, 2007, at 3:45 PM, Ayhan Kondoz wrote:
> Hi Andrus,
>
> did you log this as a bug? And do you need anything else from me?
>
> At the moment I added a cronjob to restart the WebService every
> night which prevents this bug from causing any harm but I really
> would like to fix it properly and remove the cronjob.
>
> Thanx
> Ayhan Kondoz
>
>> -----Ursprüngliche Nachricht-----
>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>> Gesendet: Mittwoch, 21. Februar 2007 15:20
>> An: use..ayenne.apache.org
>> Betreff: Re: AW: AW: possible bug / memory leak in DispatchQueue and
>> EventManager?
>>
>> 30 is good. If you had a retained DataContext somewhere you'd get the
>> same number as the number of stuck invocations.
>>
>> 1 million of Invocations is bad.
>>
>> I guess this needs to be logged as a possible bug to investigate,
>> with as many details as possible on the JVM version, etc.
>>
>> Andrus
>>
>>
>> On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote:
>>> Yes seems live there are DataContext instances.
>>>
>>> http://img73.imageshack.us/img73/3068/memta6.jpg
>>>
>>> I started the GC a few times before I took this screenshot.
>>> As you can see there are still 30 DataContext instances. No idea
>>> why thought...
>>>
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>>>> Gesendet: Mittwoch, 21. Februar 2007 14:49
>>>> An: use..ayenne.apache.org
>>>> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and
>>>> EventManager?
>>>>
>>>> Do you see DataContext instances in the memory profile? I wonder
>>>> how
>>>> many of those do you have, as those are the only Cayenne objects
>>>> that
>>>> have hard reference to the listeners.
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> the JVM has 1 GB of memory and i already tried to run the GC
>>>>> manually but it doesn't make a difference. I can start the GC from
>>>>> the profiler tool but the number of instances does not change.
>>>>> I guess that there are some other references to this objects so
>>>>> that the weak ref. does not expire.
>>>>>
>>>>> And there are no EventManager exception in the log files.
>>>>>
>>>>>
>>>>> ayhan
>>>>>
>>>>>> -----Ursprüngliche Nachricht-----
>>>>>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>>>>>> Gesendet: Mittwoch, 21. Februar 2007 14:17
>>>>>> An: use..ayenne.apache.org
>>>>>> Betreff: Re: possible bug / memory leak in DispatchQueue and
>>>>>> EventManager?
>>>>>>
>>>>>> Seems like your assessment of the EventManager leaking is
>>>>>> correct.
>>>>>>
>>>>>> Now the cause is not that clear. A shot in the dark - this is
>>>>>> due to
>>>>>> a combination of lots of spare memory in JVM (so weak references
>>>>>> are
>>>>>> not collected fast enough) and slow custom 'equals' and
>>>>>> 'hashCode'
>>>>>> methods in invocation.
>>>>>>
>>>>>> How much heap size do you have in your server? Also can you
>>>>>> try to
>>>>>> add a request filter that would do 'System.gc()' on every few
>>>>>> thousands requests, and see if it makes any difference.
>>>>>>
>>>>>> Also - do you see any EventManager exceptions in the logs?
>>>>>>
>>>>>> Andrus
>>>>>>
>>>>>>
>>>>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> i think i found a possible bug / memory leak within the
>>>>>>> DispatchQueue and EventManager.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> First a little bit about my setup:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have 3 servers. Each server runs an axis service. The service
>>>>>>> uses cayenne 1.2.1 to connect to a database. It reads
>>>>>>> customer and
>>>>>>> account information from the DB etc..
>>>>>>>
>>>>>>> The servers are using cayenne's shared caching with
>>>>>>> javagroups as
>>>>>>> the messaging service so that changes made from one server are
>>>>>>> dispatched to the other servers.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The avarage connections per second is somewhere around 4-5.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> However I have a very strange problem with this setup so I
>>>>>>> startet
>>>>>>> to search for the reason. The problem is that with nearly
>>>>>>> constant
>>>>>>> and unchanging usage the load of each server increases over
>>>>>>> time.
>>>>>>> To further test this I created a test server with a similar
>>>>>>> setup.
>>>>>>> There I created a test program that creates totally constant
>>>>>>> usage.
>>>>>>> But even with the unchanging usage the load of the server is
>>>>>>> increasing until the cpu load is so high that the requests
>>>>>>> can not
>>>>>>> be processed anymore.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I installed a java profiler to trying to pinpoint the
>>>>>>> location of
>>>>>>> this error and this is what I found out.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I let the server run for 24 hours and then stopped the program
>>>>>>> which creates the test usage.
>>>>>>>
>>>>>>> But even while the server was idle there where still a lot of
>>>>>>> instances in the java heap after the GC run.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please note the HashMap, WeakReference and Invocation counts. I
>>>>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
>>>>>>> not sure.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Now the following image shows the cpu profile with incoming
>>>>>>> connections.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As you can see 58% of the cpu time is used within HashSet.add().
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> So when I consider the two facts i think that there might be a
>>>>>>> possible problem with the EventManager. The first table tells us
>>>>>>> that there are over 1 million instances of HashSet's and cayenne
>>>>>>> Invocations. So it seems like the set's within the DispatchQueue
>>>>>>> are not recycled properly so that the object count rises over
>>>>>>> time
>>>>>>> which result in extremly high processtime when trying to add new
>>>>>>> HashSet's.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Ayhan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ayhan Kondoz
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Software-Entwicklung
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> Telefon: +49 (0) 40 513 06 616
>>>>>>>
>>>>>>> Telefax: +49 (0) 40 513 06 998 616
>>>>>>>
>>>>>>> E-Mail: ayhan.kondo..reenet-ag.de
>>>>>>> <mailto:ayhan.kondo..reenet-
>>>>>>> ag.de>
>>>>>>>
>>>>>>> Website: http://www.freenet.de <http://www.freenet.de/> ;
>>>>>>> http://
>>>>>>> www.freenet-ag.de <http://www.freenet-ag.de/>
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> freenet.de AG
>>>>>>>
>>>>>>> Deelbögenkamp 4c
>>>>>>>
>>>>>>> 22297 Hamburg
>>>>>>>
>>>>>>> ----------------------------------------------------------------
>>>>>>> --
>>>>>>> --
>>>>>>> --
>>>>>>> ------------
>>>>>>>
>>>>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
>>>>>>>
>>>>>>> Vorstand: Eckhard Spoerr (Vors.),
>>>>>>> Axel Krieger, Stephan Esch, Eric Berger
>>>>>>>
>>>>>>> Amtsgericht Hamburg HRB 74048
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Diese Information ist ausschließlich für die adressierte Person
>>>>>>> oder Organisation bestimmt und könnte vertrauliches und/oder
>>>>>>> privilegiertes Material enthalten. Personen oder Organisationen,
>>>>>>> für die diese Information nicht bestimmt ist, ist es nicht
>>>>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
>>>>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu
>>>>>>> sehen,
>>>>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese
>>>>>>> Nachricht
>>>>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem
>>>>>>> Absender in
>>>>>>> Verbindung zu setzen und das Material von Ihrem Computer zu
>>>>>>> löschen.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The information transmitted is intended only for the person or
>>>>>>> entity to which it is addressed and may contain confidential
>>>>>>> and/or
>>>>>>> privileged material. Any review, retransmission,
>>>>>>> dissemination or
>>>>>>> other use of, or taking of any action in reliance upon, this
>>>>>>> information by persons or entities other than the intended
>>>>>>> recipient is prohibited. If you received this in error, please
>>>>>>> contact the sender and delete the material from any computer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
This archive was generated by hypermail 2.0.0 : Thu Mar 15 2007 - 12:21:33 EDT
