Re: high-volume offline processing using cayenne?

From: Arndt Brenschede (a..iamos.de)
Date: Sun Mar 02 2003 - 14:07:30 EST

  • Next message: Craig Miskell: "Re: high-volume offline processing using cayenne?"

    Andrus Adamchik wrote:

    > Interesting that I used a very similar custom code on one of my
    > projects. The fact that two people independently came to the same idea
    > clearly indicates that our base prefetching API needs some more work.

    O.k., changed the mailing-list...

    I've the suspicion that we use the word "prefetching"
    for two different problems, and before I started
    discussing with you, internally in our team I was
    talking about "batch-loading" for the problem that
    was bothering me.

    - the first problem is, simply speaking, to
       merge two (or 3, or 4, ..) queries on
       *different* object-types into a single query
       by means of database "joins".
       This is the relavant problem for interactive
       applications.

    - the second problem is to merge a large number (e.g. 100)
       of queries on a single object-type into a single query.
       This is the relevant problem for batch-processing.
       (that's why I called it "batch-loading"...)

    I understand that there is an overlap, because also
    when you set a prefetching-request on the query, the
    engine can of course decide which of the 2 mechanisms
    to use.
    But it would indeeed be a limitation to require that
    the list of objects the prefetching is working on must
    itself result from a query. The 2 examples are:

    - iterated queries: here, the mapping layer does
       not know about the "batching-unit" (o.k., you
       could extend the API to tell him...)

    - but think of a 2-step "prefetch":

       A (1)----(N) B (1)---N) C

       If you query on "A", and only after evaluating
       "B" you know whether you need "C".
       Then it's good to have a method where
       you can pass a filtered list of "B"'s to
       trigger the loading of "C".

    > BTW, Craig, who is currently working on fixing prefetching
    > bugs posted a comment to your message on Cayenne-devel,
    > suggesting to add these methods to Cayenne (which we
    > will most likely do soon).

    Nice, but please consider that this was not release quality,
    it does definitly not work on compound-keys, and I didn't
    even thought about flattened relationships, and there
    a some implicit assumptions whithout crosschecks
    (e.g. it is assumed that the list of objects is of the
    same type, but there's no check and no error message...).

    (But I assume you mean adding the API, not the actual code...)

    >> For keeping track of the objets in the context
    >> I now tried the simpler solution of putting a
    >> second map into ObjectStore. (->see patch)

    > Need to look at this some more to see where this fits in Cayenne.

    same as above: just a quick hack.
    - It does not handle object-id-changes
    - probably it doesn't need to be a map (a list would do)
    - no sanity checks at all
       (are the objects to be unregistered in a COMMITED state?
        do the remaining objects have relations to them?, etc...)

    regards,

    Arndt



    This archive was generated by hypermail 2.0.0 : Sun Mar 02 2003 - 13:53:40 EST