Re: paged query slow when fetching big lists

From: Andrus Adamchik (andru..bjectstyle.org)
Date: Sat Jun 23 2007 - 03:38:34 EDT

  • Next message: Andrus Adamchik: "Re: paged query slow when fetching big lists"

    Ari, Marcin --

    going through the code I noticed one inefficiency - the elements
    array access is synchronized in 'fillIn' method. Since 'fillIn' is
    called from constructor, such synchronization is unneeded and only
    slows things down. I just checked a fixed version to trunk. Could you
    try it out?

    Andrus

    On Jun 23, 2007, at 12:33 AM, Aristedes Maniatis wrote:
    > On 22/06/2007, at 11:10 PM, Michael Gentry wrote:
    >
    >> Marcin, this thread might be of interest to you ...
    >>
    >> http://mail-archives.apache.org/mod_mbox/cayenne-dev/200705.mbox/
    >> browser
    >>
    >> Look at the "Paging and SQL queries" thread on May 25.
    >
    >
    > Yes, this is the same project we are working on. I started some
    > performance profiling and Marcin has been able now to take it much
    > further. What is it about:
    >
    >> elements.add(it.nextObjectId(entity));
    >
    > which is so slow? The code gets a little complex at that point and
    > we are having difficulty tracing it through to the exact
    > performance problem in the underlying code. Is it the speed of
    > adding the object id to the Collection or the speed of creating an
    > object id itself? 0.5ms doesn't sound slow, but it doesn't scale well.
    >
    > Andrus, I got the impression from the previous thread that you
    > suspected something slightly different. That the performance
    > problem might be in the fat query itself, but from our tests this
    > isn't the case. If I've got this right, the way it works is:
    >
    > 1. perform regular query to get all columns but return result in
    > iterator
    > 2. iterate through first page and build full objects
    > 3. iterate through other pages and build just objectids (this is
    > the slow part for us)
    > 4. when another page is fetched perform another query and fetch
    > those objects from the DB
    >
    > So, getting just primary keys from the DB may not be any faster if
    > the performance problem is simply in the construction of objectIds.
    > If the performance problem is in the numerous resizings of the
    > Collection (each time it runs out of space, then it is increased by
    > 50% or 100% in size), then the solution could be as simple as
    > figuring out the size of the iterator and sizing the collection
    > appropriately from the beginning.
    >
    > Any ideas on how to discover the exact cause of the performance hit?
    >
    >
    > Ari Maniatis
    >
    >
    >
    >
    > -------------------------->
    > ish
    > http://www.ish.com.au
    > Level 1, 30 Wilson Street Newtown 2042 Australia
    > phone +61 2 9550 5001 fax +61 2 9550 4001
    > GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
    >
    >



    This archive was generated by hypermail 2.0.0 : Sat Jun 23 2007 - 03:39:01 EDT