> Some database products offer "streaming" to and from BLOBs which is
> one get-around for these problems. This means you can theoretically
> get away with not having to hold the whole hunk of data in memory at
> once.
Yeah, I was wondering the same thing. Blob is an interface that you
can implement as a pointer to a file or a remote URL. How many drivers
are smart enough to actually use 'getBinaryStream' instead of
'getBytes' inside the prepared statement?
Andrus
On May 22, 2010, at 9:42 AM, Andrew Lindesay wrote:
> Hi there Mike;
>
> I'd have to agree with Ari there; small BLOBs (usually in a sub-
> table) work fine with an object-relational mapping system like
> Cayenne, but trying to use an object-relational technology for big
> BLOBs is generally troublesome owing to the cost of shifting those
> big hunks of data around and also gobbling-up all that memory.
>
> Some database products offer "streaming" to and from BLOBs which is
> one get-around for these problems. This means you can theoretically
> get away with not having to hold the whole hunk of data in memory at
> once.
>
> Some time ago I was having to work cross-database with BLOBs of
> arbitrary size and had some such troubles. For this reason I wrote
> a system which lays down a whole series of smaller BLOBs which are
> then linked by a header-table holding some very basic meta-data such
> as a "short unique code" to link into the object-relational world.
> It's non-transactional across the whole data, but generally special
> handling is required to deal with large data sets anyway. In java,
> I then have an input/output stream writing to and reading from this
> data structure. There are some other advantages to this system such
> as being able to do "out of order" writes to the stream.
>
> That is actually part of my "lestuff" project which is open-source
> so you are welcome to use that if you would like; drop me a note and
> I'll give you some pointers. Otherwise, maybe this gives you some
> ideas.
>
> Regards;
>
>> I'm using cayenne to store large files in BLOBs as a process runs.
>> The first step of the process is storing large files (~ 600MB) and
>> they
>> are ending up in the DB just fine, then we run some tasks and get
>> some
>> output files, and then store the large output files (~ 500MB) to
>> the DB.
> ...
>> note: I'm also compressing the stream in memory as I'm adding it to
>> the
>> byte[], but still... it works for the input files. also, each of
>> these
>
> ___
> Andrew Lindesay
> www.silvereye.co.nz
>
>
This archive was generated by hypermail 2.0.0 : Mon May 24 2010 - 06:33:22 UTC