out of memory using ResultIterator

From: Emanuele Maiarelli (emanuel..engozzi.com)
Date: Thu Dec 24 2009 - 02:43:58 EST

  • Next message: Andrus Adamchik: "Re: out of memory using ResultIterator"

    im using cayenne 2.0.4, and i need to fetch an huge ammout of data, i
    did that way:

     public static StringBuffer traceProts;

        static {
            traceProts = new StringBuffer();
            traceProts.append("SELECT ");
            traceProts.append("PROTOCOLLI.* ");
            traceProts.append("FROM PROTOCOLLI, ");
            traceProts.append("( ");
            traceProts.append(" SELECT ");
            traceProts.append(" FORM_PROT_FK ");
            traceProts.append(" FROM ");
            traceProts.append(" ( ");
            traceProts.append(" SELECT ");
            traceProts.append(" FORM_PROT_FK,SUM(IF(ME_PK is not
    null,1,0)) AS ESPORTAZIONI ");
            traceProts.append(" FROM FORMULARI ");
            traceProts.append(" LEFT JOIN MOV_EXPORTED ON
    (FORM_PK=ME_FORMULARI_FK) ");
            traceProts.append(" GROUP BY FORM_PROT_FK ");
            traceProts.append(" ) ");
            traceProts.append(" EXPS ");
            traceProts.append(" WHERE ESPORTAZIONI=0 ");
            traceProts.append(") ");
            traceProts.append("UNEXP ");
            traceProts.append("where PROT_PK=UNEXP.FORM_PROT_FK");
        }

    this query returns an huge ammont of row,

    and im fetching row using IteratedQuery, that's the code im using

            StringBuffer mq=new StringBuffer();
             Vector<EXPProtocolli> toRet=new Vector<EXPProtocolli>();
            mq.append(traceProts);
            mq.append(" AND PROT_CHIUSO=true AND PROT_RIPARTITO=true");
            SQLTemplate rawSelect = new SQLTemplate(Protocolli.class,
    mq.toString());
           
            ResultIterator v=ctx.performIteratedQuery(rawSelect);
            while (v.hasNextRow())
            {
                EXPProtocolli prt=new EXPProtocolli();
                DataRow dataRow = (DataRow) v.nextDataRow();

                Protocolli p=(Protocolli)
    ctx.objectFromDataRow(Protocolli.class, dataRow,false);

                this.bindEXPProtMsg(p,prt);
                prt.setStato(ProtStato.PROT_CARICO_ESPORTABILE);
                toRet.add(prt);
            }
            return toRet;

    when running this method i always get an heap space problem

    java.lang.OutOfMemoryError: Java heap space
            at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621)
            at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398)
            at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816)
            at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467)
            at
    com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
            at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
            at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2542)
            at
    com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1734)
            at
    com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:995)
            at
    org.apache.cayenne.access.jdbc.SQLTemplateAction.execute(SQLTemplateAction.java:135)
            at
    org.apache.cayenne.access.jdbc.SQLTemplateAction.performAction(SQLTemplateAction.java:107)
            at
    org.apache.cayenne.access.DataNodeQueryAction.runQuery(DataNodeQueryAction.java:59)
            at
    org.apache.cayenne.access.DataNode.performQueries(DataNode.java:273)
            at
    org.apache.cayenne.access.DataDomainLegacyQueryAction.execute(DataDomainLegacyQueryAction.java:81)
            at
    org.apache.cayenne.access.DataDomain$1.transform(DataDomain.java:725)
            at
    org.apache.cayenne.access.DataDomain.runInTransaction(DataDomain.java:826)
            at
    org.apache.cayenne.access.DataDomain.performQueries(DataDomain.java:722)
            at
    org.apache.cayenne.access.DataContext.internalPerformIteratedQuery(DataContext.java:1329)
            at
    org.apache.cayenne.access.DataContext.performIteratedQuery(DataContext.java:1295)
            at services.EXP.Factory.getProtocolliCaricoNonEsp(Factory.java:85)
            at services.EXP.EXPServicePBE.getData(EXPServicePBE.java:54)
            at services.EXPExporter.processRequest(EXPExporter.java:52)
    ....

    the error is generated by: ResultIterator
    v=ctx.performIteratedQuery(rawSelect);

    any hints?

    This archive was generated by hypermail 2.0.0 : Thu Dec 24 2009 - 02:54:15 EST