Hi Andrus,
did you log this as a bug? And do you need anything else from me?
At the moment I added a cronjob to restart the WebService every night which prevents this bug from causing any harm but I really would like to fix it properly and remove the cronjob.
Thanx
Ayhan Kondoz
> -----Ursprüngliche Nachricht-----
> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
> Gesendet: Mittwoch, 21. Februar 2007 15:20
> An: use..ayenne.apache.org
> Betreff: Re: AW: AW: possible bug / memory leak in DispatchQueue and
> EventManager?
>
> 30 is good. If you had a retained DataContext somewhere you'd get the
> same number as the number of stuck invocations.
>
> 1 million of Invocations is bad.
>
> I guess this needs to be logged as a possible bug to investigate,
> with as many details as possible on the JVM version, etc.
>
> Andrus
>
>
> On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote:
> > Yes seems live there are DataContext instances.
> >
> > http://img73.imageshack.us/img73/3068/memta6.jpg
> >
> > I started the GC a few times before I took this screenshot.
> > As you can see there are still 30 DataContext instances. No idea
> > why thought...
> >
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
> >> Gesendet: Mittwoch, 21. Februar 2007 14:49
> >> An: use..ayenne.apache.org
> >> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and
> >> EventManager?
> >>
> >> Do you see DataContext instances in the memory profile? I wonder how
> >> many of those do you have, as those are the only Cayenne objects that
> >> have hard reference to the listeners.
> >>
> >> Andrus
> >>
> >>
> >> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
> >>
> >>> Hi,
> >>>
> >>> the JVM has 1 GB of memory and i already tried to run the GC
> >>> manually but it doesn't make a difference. I can start the GC from
> >>> the profiler tool but the number of instances does not change.
> >>> I guess that there are some other references to this objects so
> >>> that the weak ref. does not expire.
> >>>
> >>> And there are no EventManager exception in the log files.
> >>>
> >>>
> >>> ayhan
> >>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
> >>>> Gesendet: Mittwoch, 21. Februar 2007 14:17
> >>>> An: use..ayenne.apache.org
> >>>> Betreff: Re: possible bug / memory leak in DispatchQueue and
> >>>> EventManager?
> >>>>
> >>>> Seems like your assessment of the EventManager leaking is correct.
> >>>>
> >>>> Now the cause is not that clear. A shot in the dark - this is
> >>>> due to
> >>>> a combination of lots of spare memory in JVM (so weak references
> >>>> are
> >>>> not collected fast enough) and slow custom 'equals' and 'hashCode'
> >>>> methods in invocation.
> >>>>
> >>>> How much heap size do you have in your server? Also can you try to
> >>>> add a request filter that would do 'System.gc()' on every few
> >>>> thousands requests, and see if it makes any difference.
> >>>>
> >>>> Also - do you see any EventManager exceptions in the logs?
> >>>>
> >>>> Andrus
> >>>>
> >>>>
> >>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
> >>>>> Hi,
> >>>>>
> >>>>>
> >>>>>
> >>>>> i think i found a possible bug / memory leak within the
> >>>>> DispatchQueue and EventManager.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> First a little bit about my setup:
> >>>>>
> >>>>>
> >>>>>
> >>>>> I have 3 servers. Each server runs an axis service. The service
> >>>>> uses cayenne 1.2.1 to connect to a database. It reads customer and
> >>>>> account information from the DB etc..
> >>>>>
> >>>>> The servers are using cayenne's shared caching with javagroups as
> >>>>> the messaging service so that changes made from one server are
> >>>>> dispatched to the other servers.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The avarage connections per second is somewhere around 4-5.
> >>>>>
> >>>>>
> >>>>>
> >>>>> However I have a very strange problem with this setup so I startet
> >>>>> to search for the reason. The problem is that with nearly constant
> >>>>> and unchanging usage the load of each server increases over time.
> >>>>> To further test this I created a test server with a similar setup.
> >>>>> There I created a test program that creates totally constant
> >>>>> usage.
> >>>>> But even with the unchanging usage the load of the server is
> >>>>> increasing until the cpu load is so high that the requests can not
> >>>>> be processed anymore.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I installed a java profiler to trying to pinpoint the location of
> >>>>> this error and this is what I found out.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I let the server run for 24 hours and then stopped the program
> >>>>> which creates the test usage.
> >>>>>
> >>>>> But even while the server was idle there where still a lot of
> >>>>> instances in the java heap after the GC run.
> >>>>>
> >>>>>
> >>>>>
> >>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
> >>>>>
> >>>>>
> >>>>>
> >>>>> Please note the HashMap, WeakReference and Invocation counts. I
> >>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
> >>>>> not sure.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Now the following image shows the cpu profile with incoming
> >>>>> connections.
> >>>>>
> >>>>>
> >>>>>
> >>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
> >>>>>
> >>>>>
> >>>>>
> >>>>> As you can see 58% of the cpu time is used within HashSet.add().
> >>>>>
> >>>>>
> >>>>>
> >>>>> So when I consider the two facts i think that there might be a
> >>>>> possible problem with the EventManager. The first table tells us
> >>>>> that there are over 1 million instances of HashSet's and cayenne
> >>>>> Invocations. So it seems like the set's within the DispatchQueue
> >>>>> are not recycled properly so that the object count rises over time
> >>>>> which result in extremly high processtime when trying to add new
> >>>>> HashSet's.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Ayhan
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Ayhan Kondoz
> >>>>>
> >>>>>
> >>>>>
> >>>>> Software-Entwicklung
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> Telefon: +49 (0) 40 513 06 616
> >>>>>
> >>>>> Telefax: +49 (0) 40 513 06 998 616
> >>>>>
> >>>>> E-Mail: ayhan.kondo..reenet-ag.de
> >>>>> <mailto:ayhan.kondo..reenet-
> >>>>> ag.de>
> >>>>>
> >>>>> Website: http://www.freenet.de <http://www.freenet.de/> ; http://
> >>>>> www.freenet-ag.de <http://www.freenet-ag.de/>
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> freenet.de AG
> >>>>>
> >>>>> Deelbögenkamp 4c
> >>>>>
> >>>>> 22297 Hamburg
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> --
> >>>>> --
> >>>>> ------------
> >>>>>
> >>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
> >>>>>
> >>>>> Vorstand: Eckhard Spoerr (Vors.),
> >>>>> Axel Krieger, Stephan Esch, Eric Berger
> >>>>>
> >>>>> Amtsgericht Hamburg HRB 74048
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Diese Information ist ausschließlich für die adressierte Person
> >>>>> oder Organisation bestimmt und könnte vertrauliches und/oder
> >>>>> privilegiertes Material enthalten. Personen oder Organisationen,
> >>>>> für die diese Information nicht bestimmt ist, ist es nicht
> >>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
> >>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
> >>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese
> >>>>> Nachricht
> >>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem
> >>>>> Absender in
> >>>>> Verbindung zu setzen und das Material von Ihrem Computer zu
> >>>>> löschen.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The information transmitted is intended only for the person or
> >>>>> entity to which it is addressed and may contain confidential
> >>>>> and/or
> >>>>> privileged material. Any review, retransmission, dissemination or
> >>>>> other use of, or taking of any action in reliance upon, this
> >>>>> information by persons or entities other than the intended
> >>>>> recipient is prohibited. If you received this in error, please
> >>>>> contact the sender and delete the material from any computer.
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >
This archive was generated by hypermail 2.0.0 : Thu Mar 15 2007 - 09:46:32 EDT
