Oh, that'll do it ... :-)
On Tue, May 25, 2010 at 11:58 PM, Steve Wells <websystem..mail.com> wrote:
> The mistake I made was to create a new nested context for each CSV...not
> a new parent context.
>
>
> On 24 May 2010 20:32, Steve Wells <websystem..mail.com> wrote:
>
>> Typical Cayenne users exceedingly fast helpful answers! I've tried a few
>> more things based on your replies but nothing positive so far.
>>
>> Andrey, I am only creating *most* new objects in datacontext. But I am
>> selecting the related (master a.k.a 'rel') data-objects and "connecting"
>> them to the detail objects in the order of 1-2000 (could be 1-1 or 1-3000).
>> one thing I did notice was that I was selecting the 'rel' object using
>> shared cache. I set cache to NONE for that select and all the
>> create/inserts ran approx twice as fast (I do have some basic timings in
>> place). Perhaps this is an interesting separate thread of discussion.
>>
>> Juergen, yes I already had a new ObjectContext for each CSV and I was
>> commiting each 400 (and 200 and 1000 etc). I also created a new
>> ObjectContext at the same point, though I expect moving the 'related' object
>> to the new Context may negate the point of creating the new context...so
>> same result.
>>
>> Andrus, I've deleted the reverse relationship and the same issue
>> persists. What I did do was remove the forward relationship
>> 'l.setToRelatedObject(rel)' and it all worked fine...no OutOfMemory. But
>> I do need both relationship directions. At least I have some direction in
>> which to try.
>>
>> Thanks all for your help.
>>
>> On 24 May 2010 17:21, Juergen Saar <juerge..saar.org> wrote:
>>
>>> First try: use a new ObjectContext for each csv-File ...
>>>
>>> If this doesn't solve the memory-problem you should try to make a commit
>>> after let's say 200 inserts and then create a new ObjectContext ...
>>>
>>> 2010/5/24 Steve Wells <websystem..mail.com>
>>>
>>> > Hi All,
>>> >
>>> > I am importing from a lot of csv files (2200+) and each csv has between
>>> 0
>>> > and 3000 rows, each row will create a new DataObject in a DataContext
>>> etc :
>>> >
>>> > The code is simple enough and distilled here is:
>>> >
>>> > for each csv file
>>> > RelatedObj rel = getRelatedObj()
>>> > for each line in csv file
>>> > Line l = new Line()
>>> > l.setxxx for each column of csv // 5 cols are set
>>> > l.setToRelatedObject(rel)
>>> > dataContext.registerNewObject(l)
>>> > end for each line in csv file
>>> > dataContext.commitChanges()
>>> > end
>>> >
>>> > This goes well for a while but no matter how high I set the heap size I
>>> > eventually get OutOfMemoryException....I can see the speed of each
>>> commit
>>> > slowdown over time as the heap size is diminshed.
>>> >
>>> > I have tried a few different ways around this but all end up the same.
>>> > Initially tuning the SQL server instance, but everything points to
>>> objects
>>> > in memory not being de-allocated.
>>> >
>>> > Has anyone had experience with Cayenne (3.0RC3) and this number of
>>> inserts
>>> > they could advise with?
>>> >
>>> > Cheers,
>>> >
>>> > Steve
>>> >
>>>
>>
>>
>
This archive was generated by hypermail 2.0.0 : Wed May 26 2010 - 12:42:07 UTC