Hi everyone,
I'm facing the following situation:
I'm importing arbitrary spreadsheets with quite large numbers of rows. A
row represents a recipient of a mailing. The spreadsheet can contain
arbitrary columns, so I chose the following entity model:
* RecipientSet
-> m Fields (i.e. spreadsheet column titles)
-> n Recipients
-> m Values (one for each Field)
Given a spreadsheet with 20 columns and 25.000 rows, this results in
about 25.000 x 21 DataObjects in the context. The application consumes
quite a lot of money, a profiler run showed that a large chunk of the
memory is needed for Cayenne's difference objects. And committing the
object context changes can take a couple of minutes.
Is there a best practise to approach this situation?
I considered to use a different entity model, e.g. use a generic table
with columns "col_1" to "col_100" so that I wouldn't need extra objects
for the values. But this feels like a bad workaround.
Of course I could commit the transaction each 1000 rows or so, but I'd
rather commit the whole spreadsheet to the DB in a single transaction.
Thanks a lot for any hints!
-- Andreas
-- Andreas Hartmann, CTO BeCompany GmbH http://www.becompany.ch Tel.: +41 (0) 43 818 57 01
This archive was generated by hypermail 2.0.0 : Thu Aug 13 2009 - 05:37:23 EDT