Hi there Mike;
I'd have to agree with Ari there; small BLOBs (usually in a sub-table) work fine with an object-relational mapping system like Cayenne, but trying to use an object-relational technology for big BLOBs is generally troublesome owing to the cost of shifting those big hunks of data around and also gobbling-up all that memory.
Some database products offer "streaming" to and from BLOBs which is one get-around for these problems. This means you can theoretically get away with not having to hold the whole hunk of data in memory at once.
Some time ago I was having to work cross-database with BLOBs of arbitrary size and had some such troubles. For this reason I wrote a system which lays down a whole series of smaller BLOBs which are then linked by a header-table holding some very basic meta-data such as a "short unique code" to link into the object-relational world. It's non-transactional across the whole data, but generally special handling is required to deal with large data sets anyway. In java, I then have an input/output stream writing to and reading from this data structure. There are some other advantages to this system such as being able to do "out of order" writes to the stream.
That is actually part of my "lestuff" project which is open-source so you are welcome to use that if you would like; drop me a note and I'll give you some pointers. Otherwise, maybe this gives you some ideas.
Regards;
> I'm using cayenne to store large files in BLOBs as a process runs.
> The first step of the process is storing large files (~ 600MB) and they
> are ending up in the DB just fine, then we run some tasks and get some
> output files, and then store the large output files (~ 500MB) to the DB.
...
> note: I'm also compressing the stream in memory as I'm adding it to the
> byte[], but still... it works for the input files. also, each of these
___
Andrew Lindesay
www.silvereye.co.nz
This archive was generated by hypermail 2.0.0 : Sat May 22 2010 - 06:43:31 UTC