If you are fetching 500-5000 products at a time, I'd seriously
consider using pagination since it is unlikely they will look at
500-5000 products at a time. On your SelectQuery object, do a
setPageSize(10) -- or some other reasonable number (how ever many
products you show on one page). This will reduce the memory
footprint. See:
http://cayenne.apache.org/doc/paginated-queries.html
With a paginated query, Cayenne fetches all of the PKs in (500-5000),
but then only fetches one page of data objects (say 10, using the PKs
it fetched previously) as you are looking at them, which will be more
efficient for large sets where you don't use all of the data at one
time.
As for one context/application, this is not what the thread context
(which you are using) does. The Cayenne filter creates or restores a
session-based context that is kept around for that user for the life
of the session. For some applications this makes perfect sense. You
might want to keep their shopping cart objects around from
request-to-request, for example. For a catalog type application,
though, I personally would probably use a brand-new context in each
request for fetching catalog data, letting it go at the end of the
request. This will allow those contexts and data objects to be
garbage collected much sooner. When needing to copy something from
the temporary catalog context to the user's session context (because
Cayenne needs related objects in the same context), you need to use
localObject():
http://cayenne.apache.org/doc/moving-objects-between-contexts.html
I know that was long-winded, but I hope it gave you some ideas.
Also, you said, "I do not want to do this if this is not your normal
procedure." There isn't really a normal procedure. Each application
has requirements that drive how you approach it. In one application I
kept 10,000+ records in an application-level object that was shared by
all sessions. These records were read-mostly and expensive to read in
(required several minutes due to the number of joins) and I cached
them and controlled access to them. This approach made sense -- for
that application.
mrg
PS. Add the paginated queries first and see how much that buys you.
That should be easy to do. Changing the way you are using the context
will be much more time consuming.
On Thu, Sep 17, 2009 at 10:22 AM, Joe Baldwin <jfbaldwi..arthlink.net> wrote:
> Michael
>
>> 2) The BaseContext (which is really a DataContext) will stay around as
>> long as the session does. I suppose you could always bind a new
>> context to the thread (to allow the old one to GC) or you could
>> invalidate objects when you are done using them. However, given that
>> you said you are reading in relatively few records, this may not be a
>> big win.
>
> Sorry I mislead you on this one. Even for one or two users (the current
> usage until we go live), the query result sets are returning about 500
> products, but we anticipate scaling to 5000 in the future. We are only
> updating about 5 every day (I say this because there was a mention of this
> in the docs concerning BaseContext memory management.)
>
> Also, each Product has a list of Photo references (file system paths), Audio
> references, and Video references. But I think these do not trip the fault
> until they are actually viewed (if I understand the docs).
>
> RE bind a new context to a thread
>
> I do not want to do this if this is not your normal procedure. The problem
> for the customer part is that I don't know exactly where they get a session
> and so this could make things more complex.
>
>
>> 3) Once your session is gone, it would be hard to message the context
>> since it is also gone.
>
>
> Well now that is a good point. :) Again, I was under the impression that
> the default behavior was to have only *one* BaseContext (DataContext) per
> application. But you are saying that each session gets a new BaseContext.
>
> If that is true then wouldn't all the Data Objects fetched via that context
> be released at the end of the session? If this is true then my analysis is
> *ALL* wrong. If the BaseContext goes away at the end of each session, and
> if the DataObjects go away with it, then the problem lies only within the
> session.
>
> Do I have this logic correct or did I misunderstand you?
>
> Thanks,
> Joe
>
>
>
>
> On Sep 17, 2009, at 9:58 AM, Michael Gentry wrote:
>
>> 1) DataContext dc = (DataContext) BaseContext.getThreadObjectContext();
>>
>> 2) The BaseContext (which is really a DataContext) will stay around as
>> long as the session does. I suppose you could always bind a new
>> context to the thread (to allow the old one to GC) or you could
>> invalidate objects when you are done using them. However, given that
>> you said you are reading in relatively few records, this may not be a
>> big win.
>>
>> 3) Once your session is gone, it would be hard to message the context
>> since it is also gone.
>>
>>
>> On Thu, Sep 17, 2009 at 9:42 AM, Joe Baldwin <jfbaldwi..arthlink.net>
>> wrote:
>>>
>>> Michael,
>>>
>>> I just checked my web.xml and it appears am using the same filter
>>>
>>> <!-- Cayenne -->
>>> <context-param>
>>> <param-name>cayenne.configuration.path</param-name>
>>> <param-value>/WEB-INF/config/cayenne-files</param-value>
>>> </context-param>
>>> <filter>
>>> <filter-name>CayenneFilter</filter-name>
>>>
>>>
>>> <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
>>> </filter>
>>> <filter-mapping>
>>> <filter-name>CayenneFilter</filter-name>
>>> <url-pattern>/*</url-pattern>
>>> </filter-mapping>
>>>
>>> In the code I am accessing the BaseContext via the following call:
>>> ObjectContext oc = BaseContext.getThreadObjectContext();
>>>
>>> So, unless I have misread, it appears we are using almost exactly the
>>> same
>>> code.
>>>
>>> Questions:
>>> 1. I am not sure what you mean by "you could do a cast there".
>>> 2. Should I release this BaseContext and create a new one (as has been
>>> suggested) or should I simple rely on the BaseContext to manage the
>>> memory?
>>> 3. Is there some way to message the BaseContext to determine if it has
>>> properly released memory after a session is complete?
>>>
>>> Thanks,
>>> Joe
>>>
>>>
>>>
>>>
>>>
>>> On Sep 17, 2009, at 9:18 AM, Michael Gentry wrote:
>>>
>>>> Hi Joe,
>>>>
>>>> The fact that you are seeing a spike with 50-100 concurrent users
>>>> doesn't surprise me. After your sessions timeout, the memory usage
>>>> goes back down. This seems expected to me.
>>>>
>>>> As for my session-based filter, I'm pretty much using what Cayenne
>>>> provides. In my web.xml file for the servlet engine (Tomcat in your
>>>> case), I have:
>>>>
>>>> <filter>
>>>> <filter-name>Cayenne Filter</filter-name>
>>>>
>>>>
>>>> <filter-class>org.apache.cayenne.conf.WebApplicationContextFilter</filter-class>
>>>> </filter>
>>>> <filter-mapping>
>>>> <filter-name>Cayenne Filter</filter-name>
>>>> <url-pattern>/*</url-pattern>
>>>> </filter-mapping>
>>>>
>>>> To get your context after that:
>>>>
>>>> private ObjectContext objectContext =
>>>> BaseContext.getThreadObjectContext();
>>>>
>>>> I think the filter actually creates a DataContext, so you could do a
>>>> cast
>>>> there.
>>>>
>>>
>>>
>
>
This archive was generated by hypermail 2.0.0 : Thu Sep 17 2009 - 10:52:26 EDT