Seems like your assessment of the EventManager leaking is correct.
Now the cause is not that clear. A shot in the dark - this is due to
a combination of lots of spare memory in JVM (so weak references are
not collected fast enough) and slow custom 'equals' and 'hashCode'
methods in invocation.
How much heap size do you have in your server? Also can you try to
add a request filter that would do 'System.gc()' on every few
thousands requests, and see if it makes any difference.
Also - do you see any EventManager exceptions in the logs?
Andrus
On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
> Hi,
>
>
>
> i think i found a possible bug / memory leak within the
> DispatchQueue and EventManager.
>
>
>
>
>
> First a little bit about my setup:
>
>
>
> I have 3 servers. Each server runs an axis service. The service
> uses cayenne 1.2.1 to connect to a database. It reads customer and
> account information from the DB etc..
>
> The servers are using cayenne's shared caching with javagroups as
> the messaging service so that changes made from one server are
> dispatched to the other servers.
>
>
>
> The avarage connections per second is somewhere around 4-5.
>
>
>
> However I have a very strange problem with this setup so I startet
> to search for the reason. The problem is that with nearly constant
> and unchanging usage the load of each server increases over time.
> To further test this I created a test server with a similar setup.
> There I created a test program that creates totally constant usage.
> But even with the unchanging usage the load of the server is
> increasing until the cpu load is so high that the requests can not
> be processed anymore.
>
>
>
> I installed a java profiler to trying to pinpoint the location of
> this error and this is what I found out.
>
>
>
> I let the server run for 24 hours and then stopped the program
> which creates the test usage.
>
> But even while the server was idle there where still a lot of
> instances in the java heap after the GC run.
>
>
>
> http://img206.imageshack.us/img206/9769/memorysy5.jpg
>
>
>
> Please note the HashMap, WeakReference and Invocation counts. I
> pressume that the $ObjectProvider_*** is cayenne aswell but I am
> not sure.
>
>
>
> Now the following image shows the cpu profile with incoming
> connections.
>
>
>
> http://img339.imageshack.us/img339/2441/cpuci0.jpg
>
>
>
> As you can see 58% of the cpu time is used within HashSet.add().
>
>
>
> So when I consider the two facts i think that there might be a
> possible problem with the EventManager. The first table tells us
> that there are over 1 million instances of HashSet's and cayenne
> Invocations. So it seems like the set's within the DispatchQueue
> are not recycled properly so that the object count rises over time
> which result in extremly high processtime when trying to add new
> HashSet's.
>
>
>
>
>
> Thanks
>
> Ayhan
>
>
>
>
>
>
>
> Ayhan Kondoz
>
>
>
> Software-Entwicklung
>
>
>
> ----------------------------------------------------------------------
> ------------
>
> Telefon: +49 (0) 40 513 06 616
>
> Telefax: +49 (0) 40 513 06 998 616
>
> E-Mail: ayhan.kondo..reenet-ag.de <mailto:ayhan.kondoz@freenet-
> ag.de>
>
> Website: http://www.freenet.de <http://www.freenet.de/> ; http://
> www.freenet-ag.de <http://www.freenet-ag.de/>
>
> ----------------------------------------------------------------------
> ------------
>
> freenet.de AG
>
> Deelbögenkamp 4c
>
> 22297 Hamburg
>
> ----------------------------------------------------------------------
> ------------
>
> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
>
> Vorstand: Eckhard Spoerr (Vors.),
> Axel Krieger, Stephan Esch, Eric Berger
>
> Amtsgericht Hamburg HRB 74048
>
>
>
>
>
> Diese Information ist ausschließlich für die adressierte Person
> oder Organisation bestimmt und könnte vertrauliches und/oder
> privilegiertes Material enthalten. Personen oder Organisationen,
> für die diese Information nicht bestimmt ist, ist es nicht
> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese Nachricht
> irrtümlich erhalten haben, bitten wir Sie, sich mit dem Absender in
> Verbindung zu setzen und das Material von Ihrem Computer zu löschen.
>
>
>
> The information transmitted is intended only for the person or
> entity to which it is addressed and may contain confidential and/or
> privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this
> information by persons or entities other than the intended
> recipient is prohibited. If you received this in error, please
> contact the sender and delete the material from any computer.
>
>
>
This archive was generated by hypermail 2.0.0 : Wed Feb 21 2007 - 08:17:07 EST
