On 23/09/2007, at 2:23 PM, Philip Copeland wrote:
> Did you have an issue committing every "several thousand records".
> What
> would you do if it failed at that stage - did you figure out a way to
> resume again at a known point? Handling large imports is never easy.
Initially we had some memory issues, but improvements by Andrus in
the builds of Cayenne 3 around April 2007 fixed that nicely. Then we
experimented with different batch sizes and their effect on speed. We
found that 1000 worked well for us, so I think that's what we left it
at in the end.
We prevent failure during commit by running validateForSave() on each
record. If it fails we remove it from the context and write out a log
entry. In our case we regularly see hundreds of errors in the import
data, but thankfully they mostly aren't our problem to fix. That way
we continue right to the end, even if some records fail. This allows
the customer to fix the errors and either import them along, or
return to a backup database and rerun the whole thing.
Of course, your goals may be quite different.
Ari Maniatis
-------------------------->
Aristedes Maniatis
phone +61 2 9660 9700
PGP fingerprint 08 57 20 4B 80 69 59 E2 A9 BF 2D 48 C2 20 0C C8
This archive was generated by hypermail 2.0.0 : Sun Sep 23 2007 - 04:20:36 EDT