Re: out of memory using ResultIterator

From: Emanuele Maiarelli (emanuel..engozzi.com)
Date: Thu Dec 24 2009 - 03:48:12 EST

  • Next message: Andrus Adamchik: "Re: out of memory using ResultIterator"

        I'm trying to figureout a solution using 2.0.4.

        I don't feel confortable in switching to 3.0 at this stage of the
    project.

        Thank you,

        Emanuele
    Andrus Adamchik ha scritto:
    > This is likely an issue with MySQL driver caching returned data. The
    > solution was implemented in Cayenne 3.0 (which is now in beta):
    >
    > query.setStatementFetchSize(1000);
    >
    > This propagates the fetch size setting to the JDBC statement, reducing
    > the driver memory footprint.
    >
    > Andrus
    >
    > On Dec 24, 2009, at 9:43 AM, Emanuele Maiarelli wrote:
    >
    >> im using cayenne 2.0.4, and i need to fetch an huge ammout of data, i
    >> did that way:
    >>
    >>
    >> public static StringBuffer traceProts;
    >>
    >>
    >> static {
    >> traceProts = new StringBuffer();
    >> traceProts.append("SELECT ");
    >> traceProts.append("PROTOCOLLI.* ");
    >> traceProts.append("FROM PROTOCOLLI, ");
    >> traceProts.append("( ");
    >> traceProts.append(" SELECT ");
    >> traceProts.append(" FORM_PROT_FK ");
    >> traceProts.append(" FROM ");
    >> traceProts.append(" ( ");
    >> traceProts.append(" SELECT ");
    >> traceProts.append(" FORM_PROT_FK,SUM(IF(ME_PK is not
    >> null,1,0)) AS ESPORTAZIONI ");
    >> traceProts.append(" FROM FORMULARI ");
    >> traceProts.append(" LEFT JOIN MOV_EXPORTED ON
    >> (FORM_PK=ME_FORMULARI_FK) ");
    >> traceProts.append(" GROUP BY FORM_PROT_FK ");
    >> traceProts.append(" ) ");
    >> traceProts.append(" EXPS ");
    >> traceProts.append(" WHERE ESPORTAZIONI=0 ");
    >> traceProts.append(") ");
    >> traceProts.append("UNEXP ");
    >> traceProts.append("where PROT_PK=UNEXP.FORM_PROT_FK");
    >> }
    >>
    >> this query returns an huge ammont of row,
    >>
    >> and im fetching row using IteratedQuery, that's the code im using
    >>
    >> StringBuffer mq=new StringBuffer();
    >> Vector<EXPProtocolli> toRet=new Vector<EXPProtocolli>();
    >> mq.append(traceProts);
    >> mq.append(" AND PROT_CHIUSO=true AND PROT_RIPARTITO=true");
    >> SQLTemplate rawSelect = new SQLTemplate(Protocolli.class,
    >> mq.toString());
    >> ResultIterator v=ctx.performIteratedQuery(rawSelect);
    >> while (v.hasNextRow())
    >> {
    >> EXPProtocolli prt=new EXPProtocolli();
    >> DataRow dataRow = (DataRow) v.nextDataRow();
    >>
    >> Protocolli p=(Protocolli)
    >> ctx.objectFromDataRow(Protocolli.class, dataRow,false);
    >>
    >> this.bindEXPProtMsg(p,prt);
    >> prt.setStato(ProtStato.PROT_CARICO_ESPORTABILE);
    >> toRet.add(prt);
    >> }
    >> return toRet;
    >>
    >> when running this method i always get an heap space problem
    >>
    >> java.lang.OutOfMemoryError: Java heap space
    >> at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
    >> at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
    >> at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
    >> at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
    >> at
    >> com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
    >> at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
    >> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
    >> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
    >> at
    >> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
    >>
    >> at
    >> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995)
    >> at
    >> org.apache.cayenne.access.jdbc.SQLTemplateAction.execute(SQLTemplateAction.java:135)
    >>
    >> at
    >> org.apache.cayenne.access.jdbc.SQLTemplateAction.performAction(SQLTemplateAction.java:107)
    >>
    >> at
    >> org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:59)
    >>
    >> at
    >> org.apache.cayenne.access.DataNode.performQueries(DataNode.java:273)
    >> at
    >> org.apache.cayenne.access.DataDomainLegacyQueryAction.execute(DataDomainLegacyQueryAction.java:81)
    >>
    >> at
    >> org.apache.cayenne.access.DataDomain$1.transform(DataDomain.java:725)
    >> at
    >> org.apache.cayenne.access.DataDomain.runInTransaction(DataDomain.java:826)
    >>
    >> at
    >> org.apache.cayenne.access.DataDomain.performQueries(DataDomain.java:722)
    >> at
    >> org.apache.cayenne.access.DataContext.internalPerformIteratedQuery(DataContext.java:1329)
    >>
    >> at
    >> org.apache.cayenne.access.DataContext.performIteratedQuery(DataContext.java:1295)
    >>
    >> at services.EXP.Factory.getProtocolliCaricoNonEsp(Factory.java:85)
    >> at services.EXP.EXPServicePBE.getData(EXPServicePBE.java:54)
    >> at services.EXPExporter.processRequest(EXPExporter.java:52)
    >> ....
    >>
    >> the error is generated by: ResultIterator
    >> v=ctx.performIteratedQuery(rawSelect);
    >>
    >> any hints?
    >>
    >>
    >>
    >>
    >>
    >

    This archive was generated by hypermail 2.0.0 : Thu Dec 24 2009 - 03:58:27 EST