Yeah fully implementing streaming may require some non trivial effort.
But from your initial description it appeared that you have enough
memory to process even the largest blobs in isolation. So maybe
there's a memory leak somewhere that can be detected with a profiler
and fixed by changing some cache settings / DataContext scope without
rewriting Cayenne?
Andrus
On May 28, 2010, at 2:32 PM, MGargan..scholar.com wrote:
> Hi Tore,
>
> I finally got around to looking at your code in the jira
> attachments. It looks like it will need a little bit of updating to
> get
> it 3.0 ready, but at the end of the day it still pulls the whole thing
> into memory at some point. I might ping you over the next week, if
> you
> don't mind, about this as I quickly (i.e. panic-mode) try to cobble
> something together. I'm just learning cayenne so a lot of the
> internals
> are still not very clear to me.
>
> Thanks.
> -Mike
>
>
>
>
> From:
> Tore Halset <halse..vv.ntnu.no>
> To:
> use..ayenne.apache.org
> Date:
> 05/25/2010 07:58 AM
> Subject:
> Re: Blobs in the DataContext
>
>
>
> Hello.
>
> I tried to implement support for streaming blobs back in 2006.
> Sorry, but
> it never got completed. I still think it is a nice feature. If you
> want to
> work on this issue, you might want to take a look at
> streaming_blob_try2.zip from https://issues.apache.org/jira/browse/CAY-316
>
> Regards,
> - Tore.
>
> On 21. mai 2010, at 23.27, MGargan..scholar.com wrote:
>
>> Hi,
>>
>> I'm using cayenne to store large files in BLOBs as a process
> runs.
>> The first step of the process is storing large files (~ 600MB) and
>> they
>> are ending up in the DB just fine, then we run some tasks and get
>> some
>> output files, and then store the large output files (~ 500MB) to
>> the DB.
>
>> The output files are not making it into the DB. In fact it appears
>> that
>
>> the whole program is just sitting and waiting, for what, i have no
>> idea
>> and after you try and spawn another thread in the program it throws
>> an
> out
>> of memory exception. I was trying to figure out why the larger input
>> files got persisted fine, but the large output files cause a
>> problem and
>
>> the only thing I could think of was that when the BLOBs are created
>> they
>
>> are cached in the DataContext and are never cleared eventually just
>> causing the memory to be exhausted. Is this possible? Anything else
>> anyone can think of?
>>
>> note: I'm also compressing the stream in memory as I'm adding it to
>> the
>> byte[], but still... it works for the input files. also, each of
>> these
>> phases of the process is followed by a commit, so all the input files
> are
>> committed together and all the output files should be committed
>> together
>
>> as well, but this never happens.
>>
>> Thank you for any help you may be able to provide.
>> -Mike
>
>
>
This archive was generated by hypermail 2.0.0 : Mon May 31 2010 - 10:13:14 UTC