Actually you can do batch processing with Cayenne, as DataContext uses
weak references to committed objects. So doing commit after each X
inserted objects should release the memory. However "rel" object
reference may prevent that from happening if it is the same object for
all inserted objects. For one thing make sure there's no reverse
relationship from rel to l's.
Andrus
On May 24, 2010, at 10:09 AM, Andrey Razumovsky wrote:
> ORM technologies require additional memory than just plain JDBC.
> Million
> records you read will lead to creating million data objects and this
> is not
> a case for ORM, because huge size of memory is needed. Moreover,
> you'll see
> major slowdowns during the commit. So the actual advice is to use
> plain SQL
> (or say Cayenne's SQLTemplate) for such great inserts
>
> 2010/5/24 Steve Wells <websystem..mail.com>
>
>> Hi All,
>>
>> I am importing from a lot of csv files (2200+) and each csv has
>> between 0
>> and 3000 rows, each row will create a new DataObject in a
>> DataContext etc :
>>
>> The code is simple enough and distilled here is:
>>
>> for each csv file
>> RelatedObj rel = getRelatedObj()
>> for each line in csv file
>> Line l = new Line()
>> l.setxxx for each column of csv // 5 cols are set
>> l.setToRelatedObject(rel)
>> dataContext.registerNewObject(l)
>> end for each line in csv file
>> dataContext.commitChanges()
>> end
>>
>> This goes well for a while but no matter how high I set the heap
>> size I
>> eventually get OutOfMemoryException....I can see the speed of each
>> commit
>> slowdown over time as the heap size is diminshed.
>>
>> I have tried a few different ways around this but all end up the
>> same.
>> Initially tuning the SQL server instance, but everything points to
>> objects
>> in memory not being de-allocated.
>>
>> Has anyone had experience with Cayenne (3.0RC3) and this number of
>> inserts
>> they could advise with?
>>
>> Cheers,
>>
>> Steve
>>
>
>
>
> --
> Andrey
This archive was generated by hypermail 2.0.0 : Mon May 24 2010 - 07:20:27 UTC