On 13/11/09 10:04 PM, Hans Pikkemaat wrote:
> I ran some tests using 3.0b with SQLTemplate in combination with
> prefetching and found
> a possible new problem.
>
> It seems that when running the query in eg 1 minute, it takes about 2
> minutes before cayenne
> has constructed the prefetched objects.
>
> My query produces 2.5 million records. The query will take about 30
> minutes. Construction
> of the objects will then take an extra hour.
Just to be clear about what you are doing:
* Cayenne 3.0 beta 1
* SQL template query
* prefeching across to-many join
* paging on
You expect the first query (which gets hollow objects) to NOT include the prefetch JOIN, but when fetching a page of results, it should use the prefetch. Cayenne is constructing the first query which includes the JOIN and that makes it take 30 minutes in your database to return 2.5 million records.
Is that correct?
My opinions:
1. Does prefetch really help here anyway? You are only getting (say) 100 records at a time, so the extra queries to follow the relations may not be that significant.
2. Do you really want to fetch 2.5 million rows? If so, I assume this is not a user interface :-) Perhaps Cayenne (or any ORM for that matter) is not the best way to batch process that many rows.
Ari
----------------------------> Aristedes Maniatis GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
This archive was generated by hypermail 2.0.0 : Fri Nov 13 2009 - 06:45:37 EST