Hi
Recently we have found that fetching a list of 100,000 records using
ROP with paging and no cache takes a long time, about 50 seconds in
our case. We have profiled the cpu usage and the result shows that
99% of time is spent in IncrementalFaultList, within the fillIn()
method.
The fillIn method works (in my opinion) in a bit strange fashion: it
does execute query at once, stores the query result in
java.sql.ResultSet, and than iterates through the result either
creating the whole DataRow or just ObjectId. If there is a need the
DataRows are faulted at the end of the method.
From our testing it came up that this bit of code :
while (it.hasNextRow()) {
elements.add(it.nextObjectId(entity));
}
is where all the time is spent. Each iteration in this loop takes
about 0.5ms, which multiplied by 100,000 takes almost 50 seconds.
nextObjectId method consists of two parts: fetching the next result
from ResultSet and creating a ObjectId, but I was unable to check
which one takes the most time, anyway I think that this approach is
somewhat wrong, since always 99% of the records will be fetched as
ObjectId and never faulted, so my ideas to enhance this are:
* fetch only Pk columns and create all ObjectIds at once, get rid of
the iterating process if possible
* use already existing method resolveInterval() to fault the required
range of records
If the creation of ObjectId and getting the results from ResultSet
cannot be speed up (because it simply has to happen, and it does not
depend on the way it is done), the only choice will be to implement
some more complex solution using sql LIMIT statement.
I would like to mention that we are using some DataContext decorators
and life-cycle callbacks, but I don't believe those are important
factors in this case.
Whatever is the solution, i think it is pretty crucial that it will
be implemented soon, since the usability of the ROP without fast
paging is rather low.
With regards
Marcin
-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001 fax +61 2 9550 4001
This archive was generated by hypermail 2.0.0 : Thu Jun 21 2007 - 20:35:42 EDT