Re: Extract large data from database

From: Michael Gentry (mgentr..asslight.net)
Date: Wed Mar 18 2009 - 10:24:17 EDT

  • Next message: Andrus Adamchik: "Re: Extract large data from database"

    Are you doing the queries in the same DataContext? If so, try
    discarding the old DC and create a new one. (DC creation is cheap.)

    You could also try calling DC's invalidateObjects(Collection objects),
    but discarding the DC is probably easier and more efficient.

    mrg

    On Wed, Mar 18, 2009 at 7:14 AM, mr.abanjo <mr.abanj..mail.com> wrote:
    > Hi,i'm developing an application that must load a large ammount of data
    > (100.000 records) and stream them out  from a servlet.
    >
    > I created a "for" cycle that for every step load 200 record from the
    > database (using datacontext.performquery method), and then flush the data in
    > the serlvet output stream.
    > In this way i suppose that the max ammount of memory allocated by cayenne is
    > :
    >
    > object size * 200
    >
    > but seems that this is not true. When i try to get the data from the
    > servlet, after few seconds, i receive an "out of memory" error. (my 500Mb
    > allocated for the Heap are reached in a few time... :-(....)
    >
    > Which is the best practice in this case?
    > I can increase the heap, but i want to find a workaround that allow me to
    > have a small heap allocated during this operation.
    >
    > I found this article:
    > http://cayenne.apache.org/doc20/iterating-through-data-rows.html
    > Is this the right way?
    >
    > The servlet i'm implementig will be called by a different webapp
    > (backoffice.. in a server 2 server way ) with a low traffic but i'm warried
    > about the warning reported in the article:
    >
    > "In web applications, programmers must ensure that no open ResultIterators
    > are kept between HTTP requests. Failure to do so may result in too many
    > database connections being locked, thus quickly exhausting connection pool.
    > In general, an application with Web GUI is NOT a good candidate for
    > implementation using ResultIterators."
    >
    > Thanks
    > Davide
    >



    This archive was generated by hypermail 2.0.0 : Wed Mar 18 2009 - 10:24:55 EDT