Big big fecth..??

Author Message

Jorge estévez

Saturday 24 October 2009 8:45:39 pm

I have to allow my client to select a node that has many products, once selected I must display a list of all the products within and some of its attributes.

The problem arises when this kind of “fetch” has to deal with so many nodes that the memory gets compromised (I have whole control of apache at the server , but I would like to keep memory used for running scripts not to big) .

I need to know how to deal with a really big fetch even if it takes some seconds or even minutes but in some way I can free memory withing the fetching of groups of nodes or else.

Any Ideas?

Diseño Web Cuba
Web Design Cuba
www.elfosdesign.com

Gabriel Finkelstein

Monday 26 October 2009 10:43:17 am

If you're displaying all items on the same page, you should page them.

Piotrek Karaś

Monday 26 October 2009 11:28:51 pm

Can you illustrate your example with some numbers? How many nodes do you have to display? Also, won't caching layer take care of the problem (generate once and server from cache later on)?

--
Company: mediaSELF Sp. z o.o., http://www.mediaself.pl
eZ references: http://ez.no/partners/worldwide_partners/mediaself
eZ certified developer: http://ez.no/certification/verify/272585
eZ blog: http://ez.ryba.eu

André R.

Tuesday 27 October 2009 6:53:19 am

I'll add another: Why?

But ignoring the fact that this sounds like a big design mistake, here is how it can be done:

A. Fetch some of the items at once (50 or 100 nodes), if your on 4.1 and lower then make sure you add "load_data_map, false()" parameter (was added in 3.10). And create a template operator that clears object + data_map + attribute cache (search for "cronjob memory" or something to find out how you do that).

B. Same as above, except fetch all nodes at once instead of some using "as_object, false()" parameter, and create a template operator that creates the objects using "eZContentObject::fillNodeListAttributes( $listOfNodes );".

Page will probably be slower, but you'll avoid eating memory.

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Jorge estévez

Wednesday 28 October 2009 4:46:53 am

Hello,

My client, at different offices around the world is populating the site with products, lots of them, at the head office there is a need to know the list of codes of products that are already inserted in a folder-node (and its folder-childs) to be able to compare with the products that they have in their database, reasonable need indeed).

Taking into consideration usability I built a tree of nodes (similar to the ez admin page) that has checkmarks (all nodes with its corresponding checkmark), So the head office admin clicks the checkmark and a button that makes a call to an url using jAVASCRIPT like this one

http://www.thesite.com/index.php/esl/content/view/getdupcode/181

the 181 is added to the url based on the selection of the node in the tree, and the getupcode is a view mode that get's all the product nodes (within the node's tree) and displays them.

Getting all the products takes too much time and memory, thousands of nodes (and still more to come) lies beneath different branches of the selected node.

I tried to page the output as to display 200 products codes from page to page, by doing so

http://www.thesite.com/index.php/esl/content/view/getdupcode/181/(offset)/50

by some reason the pages goes to it's previous state (before selecting and clicking the button) that is

http://www.thesite.com/index.php/esl/content/view/full/181

And displays nothing (not even the information that should be there before the selection and click to the button)

Thanks.

Diseño Web Cuba
Web Design Cuba
www.elfosdesign.com

Romain Petit

Monday 02 November 2009 2:07:30 pm

Hi,

If you are already using javascript to load your content you can fetch nodes one after another with several consecutive ajax calls, and insert them in your page ( or 10 by 10... ).

Is the memory limit reached ? If so, is that during the fetch or during the template processing ?

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.

eZ debug

Timing: Jan 30 2025 16:40:15
Script start
Timing: Jan 30 2025 16:40:15
Module start 'layout'
Timing: Jan 30 2025 16:40:15
Module start 'content'
Timing: Jan 30 2025 16:40:16
Module end 'content'
Timing: Jan 30 2025 16:40:16
Script end

Main resources:

Total runtime1.7373 sec
Peak memory usage8,192.0000 KB
Database Queries72

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0071 588.0078151.1953
Module start 'layout' 0.00710.0049 739.2031220.6406
Module start 'content' 0.01201.7241 959.84384,601.2031
Module end 'content' 1.73610.0009 5,561.046915.8828
Script end 1.7370  5,576.9297 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00390.2242160.0002
Check MTime0.00130.0764160.0001
Mysql Total
Database connection0.00210.123510.0021
Mysqli_queries1.547689.0834720.0215
Looping result0.00100.0572700.0000
Template Total1.618593.220.8092
Template load0.00220.127120.0011
Template processing1.616293.032120.8081
Template load and register function0.00180.102910.0018
states
state_id_array0.00170.098710.0017
state_identifier_array0.00100.054720.0005
Override
Cache load0.00190.1101370.0001
Sytem overhead
Fetch class attribute can translate value0.00200.116650.0004
Fetch class attribute name0.00240.138390.0003
XML
Image XML parsing0.02121.218050.0042
class_abstraction
Instantiating content class attribute0.00000.000990.0000
General
dbfile0.07994.6002300.0027
String conversion0.00000.000540.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
6content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
9content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
3content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
3content/datatype/view/ezxmltags/literal.tpl<No override>extension/community/design/standard/templates/content/datatype/view/ezxmltags/literal.tplEdit templateOverride template
1print_pagelayout.tpl<No override>extension/community/design/community/templates/print_pagelayout.tplEdit templateOverride template
 Number of times templates used: 23
 Number of unique templates used: 6

Time used to render debug report: 0.0005 secs