The mistake I made was to create a new nested context for each CSV...not
a new parent context.
On 24 May 2010 20:32, Steve Wells <websystem..mail.com> wrote:
> Typical Cayenne users exceedingly fast helpful answers! I've tried a few
> more things based on your replies but nothing positive so far.
>
> Andrey, I am only creating *most* new objects in datacontext. But I am
> selecting the related (master a.k.a 'rel') data-objects and "connecting"
> them to the detail objects in the order of 1-2000 (could be 1-1 or 1-3000).
> one thing I did notice was that I was selecting the 'rel' object using
> shared cache. I set cache to NONE for that select and all the
> create/inserts ran approx twice as fast (I do have some basic timings in
> place). Perhaps this is an interesting separate thread of discussion.
>
> Juergen, yes I already had a new ObjectContext for each CSV and I was
> commiting each 400 (and 200 and 1000 etc). I also created a new
> ObjectContext at the same point, though I expect moving the 'related' object
> to the new Context may negate the point of creating the new context...so
> same result.
>
> Andrus, I've deleted the reverse relationship and the same issue
> persists. What I did do was remove the forward relationship
> 'l.setToRelatedObject(rel)' and it all worked fine...no OutOfMemory. But
> I do need both relationship directions. At least I have some direction in
> which to try.
>
> Thanks all for your help.
>
> On 24 May 2010 17:21, Juergen Saar <juerge..saar.org> wrote:
>
>> First try: use a new ObjectContext for each csv-File ...
>>
>> If this doesn't solve the memory-problem you should try to make a commit
>> after let's say 200 inserts and then create a new ObjectContext ...
>>
>> 2010/5/24 Steve Wells <websystem..mail.com>
>>
>> > Hi All,
>> >
>> > I am importing from a lot of csv files (2200+) and each csv has between
>> 0
>> > and 3000 rows, each row will create a new DataObject in a DataContext
>> etc :
>> >
>> > The code is simple enough and distilled here is:
>> >
>> > for each csv file
>> > RelatedObj rel = getRelatedObj()
>> > for each line in csv file
>> > Line l = new Line()
>> > l.setxxx for each column of csv // 5 cols are set
>> > l.setToRelatedObject(rel)
>> > dataContext.registerNewObject(l)
>> > end for each line in csv file
>> > dataContext.commitChanges()
>> > end
>> >
>> > This goes well for a while but no matter how high I set the heap size I
>> > eventually get OutOfMemoryException....I can see the speed of each
>> commit
>> > slowdown over time as the heap size is diminshed.
>> >
>> > I have tried a few different ways around this but all end up the same.
>> > Initially tuning the SQL server instance, but everything points to
>> objects
>> > in memory not being de-allocated.
>> >
>> > Has anyone had experience with Cayenne (3.0RC3) and this number of
>> inserts
>> > they could advise with?
>> >
>> > Cheers,
>> >
>> > Steve
>> >
>>
>
>
This archive was generated by hypermail 2.0.0 : Wed May 26 2010 - 03:59:14 UTC