AW: possible bug / memory leak in DispatchQueue and EventManager?

From: Ayhan Kondoz (Ayhan.Kondo..reenet-ag.de)
Date: Wed Feb 21 2007 - 08:37:40 EST

  • Next message: Andrus Adamchik: "Re: AW: possible bug / memory leak in DispatchQueue and EventManager?"

    Hi,

    the JVM has 1 GB of memory and i already tried to run the GC manually but it doesn't make a difference. I can start the GC from the profiler tool but the number of instances does not change.
    I guess that there are some other references to this objects so that the weak ref. does not expire.

    And there are no EventManager exception in the log files.

    ayhan

    > -----Ursprüngliche Nachricht-----
    > Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
    > Gesendet: Mittwoch, 21. Februar 2007 14:17
    > An: use..ayenne.apache.org
    > Betreff: Re: possible bug / memory leak in DispatchQueue and EventManager?
    >
    > Seems like your assessment of the EventManager leaking is correct.
    >
    > Now the cause is not that clear. A shot in the dark - this is due to
    > a combination of lots of spare memory in JVM (so weak references are
    > not collected fast enough) and slow custom 'equals' and 'hashCode'
    > methods in invocation.
    >
    > How much heap size do you have in your server? Also can you try to
    > add a request filter that would do 'System.gc()' on every few
    > thousands requests, and see if it makes any difference.
    >
    > Also - do you see any EventManager exceptions in the logs?
    >
    > Andrus
    >
    >
    > On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
    > > Hi,
    > >
    > >
    > >
    > > i think i found a possible bug / memory leak within the
    > > DispatchQueue and EventManager.
    > >
    > >
    > >
    > >
    > >
    > > First a little bit about my setup:
    > >
    > >
    > >
    > > I have 3 servers. Each server runs an axis service. The service
    > > uses cayenne 1.2.1 to connect to a database. It reads customer and
    > > account information from the DB etc..
    > >
    > > The servers are using cayenne's shared caching with javagroups as
    > > the messaging service so that changes made from one server are
    > > dispatched to the other servers.
    > >
    > >
    > >
    > > The avarage connections per second is somewhere around 4-5.
    > >
    > >
    > >
    > > However I have a very strange problem with this setup so I startet
    > > to search for the reason. The problem is that with nearly constant
    > > and unchanging usage the load of each server increases over time.
    > > To further test this I created a test server with a similar setup.
    > > There I created a test program that creates totally constant usage.
    > > But even with the unchanging usage the load of the server is
    > > increasing until the cpu load is so high that the requests can not
    > > be processed anymore.
    > >
    > >
    > >
    > > I installed a java profiler to trying to pinpoint the location of
    > > this error and this is what I found out.
    > >
    > >
    > >
    > > I let the server run for 24 hours and then stopped the program
    > > which creates the test usage.
    > >
    > > But even while the server was idle there where still a lot of
    > > instances in the java heap after the GC run.
    > >
    > >
    > >
    > > http://img206.imageshack.us/img206/9769/memorysy5.jpg
    > >
    > >
    > >
    > > Please note the HashMap, WeakReference and Invocation counts. I
    > > pressume that the $ObjectProvider_*** is cayenne aswell but I am
    > > not sure.
    > >
    > >
    > >
    > > Now the following image shows the cpu profile with incoming
    > > connections.
    > >
    > >
    > >
    > > http://img339.imageshack.us/img339/2441/cpuci0.jpg
    > >
    > >
    > >
    > > As you can see 58% of the cpu time is used within HashSet.add().
    > >
    > >
    > >
    > > So when I consider the two facts i think that there might be a
    > > possible problem with the EventManager. The first table tells us
    > > that there are over 1 million instances of HashSet's and cayenne
    > > Invocations. So it seems like the set's within the DispatchQueue
    > > are not recycled properly so that the object count rises over time
    > > which result in extremly high processtime when trying to add new
    > > HashSet's.
    > >
    > >
    > >
    > >
    > >
    > > Thanks
    > >
    > > Ayhan
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > > Ayhan Kondoz
    > >
    > >
    > >
    > > Software-Entwicklung
    > >
    > >
    > >
    > > ----------------------------------------------------------------------
    > > ------------
    > >
    > > Telefon: +49 (0) 40 513 06 616
    > >
    > > Telefax: +49 (0) 40 513 06 998 616
    > >
    > > E-Mail: ayhan.kondo..reenet-ag.de <mailto:ayhan.kondoz@freenet-
    > > ag.de>
    > >
    > > Website: http://www.freenet.de <http://www.freenet.de/> ; http://
    > > www.freenet-ag.de <http://www.freenet-ag.de/>
    > >
    > > ----------------------------------------------------------------------
    > > ------------
    > >
    > > freenet.de AG
    > >
    > > Deelbögenkamp 4c
    > >
    > > 22297 Hamburg
    > >
    > > ----------------------------------------------------------------------
    > > ------------
    > >
    > > Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
    > >
    > > Vorstand: Eckhard Spoerr (Vors.),
    > > Axel Krieger, Stephan Esch, Eric Berger
    > >
    > > Amtsgericht Hamburg HRB 74048
    > >
    > >
    > >
    > >
    > >
    > > Diese Information ist ausschließlich für die adressierte Person
    > > oder Organisation bestimmt und könnte vertrauliches und/oder
    > > privilegiertes Material enthalten. Personen oder Organisationen,
    > > für die diese Information nicht bestimmt ist, ist es nicht
    > > gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
    > > anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
    > > Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese Nachricht
    > > irrtümlich erhalten haben, bitten wir Sie, sich mit dem Absender in
    > > Verbindung zu setzen und das Material von Ihrem Computer zu löschen.
    > >
    > >
    > >
    > > The information transmitted is intended only for the person or
    > > entity to which it is addressed and may contain confidential and/or
    > > privileged material. Any review, retransmission, dissemination or
    > > other use of, or taking of any action in reliance upon, this
    > > information by persons or entities other than the intended
    > > recipient is prohibited. If you received this in error, please
    > > contact the sender and delete the material from any computer.
    > >
    > >
    > >



    This archive was generated by hypermail 2.0.0 : Wed Feb 21 2007 - 08:38:17 EST