Hi Andrus
I had not much time to check, but with the fix the 100k records load
in 30 instead of 50 seconds. It is some improvement, but not enough.
I'll do some more profiling tomorrow and let you know.
By the way, we are using netbeans for profiling, the benefit : it is
free. I will evaluate the yourkit as we are moving away from netbeans
as a development platform.
Marcin
On 23/06/2007, at 5:38 PM, Andrus Adamchik wrote:
> Ari, Marcin --
>
> going through the code I noticed one inefficiency - the elements
> array access is synchronized in 'fillIn' method. Since 'fillIn' is
> called from constructor, such synchronization is unneeded and only
> slows things down. I just checked a fixed version to trunk. Could
> you try it out?
>
> Andrus
>
>
>
> On Jun 23, 2007, at 12:33 AM, Aristedes Maniatis wrote:
>> On 22/06/2007, at 11:10 PM, Michael Gentry wrote:
>>
>>> Marcin, this thread might be of interest to you ...
>>>
>>> http://mail-archives.apache.org/mod_mbox/cayenne-dev/200705.mbox/
>>> browser
>>>
>>> Look at the "Paging and SQL queries" thread on May 25.
>>
>>
>> Yes, this is the same project we are working on. I started some
>> performance profiling and Marcin has been able now to take it much
>> further. What is it about:
>>
>>> elements.add(it.nextObjectId(entity));
>>
>> which is so slow? The code gets a little complex at that point and
>> we are having difficulty tracing it through to the exact
>> performance problem in the underlying code. Is it the speed of
>> adding the object id to the Collection or the speed of creating an
>> object id itself? 0.5ms doesn't sound slow, but it doesn't scale
>> well.
>>
>> Andrus, I got the impression from the previous thread that you
>> suspected something slightly different. That the performance
>> problem might be in the fat query itself, but from our tests this
>> isn't the case. If I've got this right, the way it works is:
>>
>> 1. perform regular query to get all columns but return result in
>> iterator
>> 2. iterate through first page and build full objects
>> 3. iterate through other pages and build just objectids (this is
>> the slow part for us)
>> 4. when another page is fetched perform another query and fetch
>> those objects from the DB
>>
>> So, getting just primary keys from the DB may not be any faster if
>> the performance problem is simply in the construction of
>> objectIds. If the performance problem is in the numerous resizings
>> of the Collection (each time it runs out of space, then it is
>> increased by 50% or 100% in size), then the solution could be as
>> simple as figuring out the size of the iterator and sizing the
>> collection appropriately from the beginning.
>>
>> Any ideas on how to discover the exact cause of the performance hit?
>>
>>
>> Ari Maniatis
>>
>>
>>
>>
>> -------------------------->
>> ish
>> http://www.ish.com.au
>> Level 1, 30 Wilson Street Newtown 2042 Australia
>> phone +61 2 9550 5001 fax +61 2 9550 4001
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
>>
>>
>
Marcin
This archive was generated by hypermail 2.0.0 : Mon Jun 25 2007 - 06:32:30 EDT