Forums / Developer / How to build an update CLI script without memory problem ?

How to build an update CLI script without memory problem ?

Author Message

Bertrand Dunogier

Wednesday 18 November 2009 9:15:20 am

Hi Kristof !

Well, such an iterator would indeed be a good idea. But based on how developement is organized, and on real life, I think that applying this to ezPersistentObject is a bit optimistic, especially since it is based to be heavily modified soon.

What we will probably start with is a global in-memory cache API, maybe linked to memcache and other cache mechanisms, so that we don't have all these hardcoded global calls everywhere. The iterator idea, with automatic cleanup, etc, is a very good point though.

I'm very eager to hear any idea you might have in that area though !

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Gaetano Giunta

Thursday 19 November 2009 1:13:38 am

A low-hanging fruit that might be preliminary work to cache refactoring: tag in comments all methods that cache results with a custom phpdoc tag

Principal Consultant International Business
Member of the Community Project Board

Mark Simon

Thursday 19 November 2009 1:52:35 am

Wher writing CLI scritps I try to keep the fetch on the "outside of the for/foreach" as small as possible.

Best way would be a SQL fetch with all the nodes I need.
(The advantage is: I can use LIMIT to run the script for a part of the result. Change the offset and append the next result to the output file.)

If You need to use the fetch functions, I would try the 'load_data_map' paramter like this:

$parentNodeID = 5;

$users =& eZContentObjectTreeNode::subTreeByNodeId( 
      array( 'ClassFilterType' => 'include',
  'ClassFilterArray' => array( 'user' ),
  'SortBy' => array( 'published', false),
                'LoadDataMap' => false
 ),          
      $parentNodeID);

$i = 0;
foreach( $users as $user ) {
      $usrObj = eZContentObject::fetch( $user->attribute('contentobject_id') );      
      $dataMap =& $usrObj->attribute( 'data_map' );

      ... do something

      unset( $dataMap );
      unset( $usrObj );
}

'inside' the foreach fetch the datamap and unset all variables that are being used.

www.all2e.com

Kristof Coomans

Sunday 06 December 2009 1:26:10 am

@Bertrand which kind of changes are you talking about, regarding eZPersistentObject? Is this for the 4.3 development cycle? Or for "Project V"? I did not see anything about this on the roadmap.

independent eZ Publish developer and service provider | http://blog.coomanskristof.be | http://ezpedia.org

Bertrand Dunogier

Sunday 06 December 2009 2:20:15 am

ProjectV ? Never heard of that. I've heard of Big Balls though. End of private joke.

This change is something we've been talking about, and I also know it has already been partially implemented by a member of the team. There is nothing concrete yet, but it is clear that managing in-memory cache the way we currently do in eZ is a poor approach. This needs to be consolidated with a system wide API, but it's definitely not my role to decide when it is gonna be done. Sorry :(

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

eZ debug

Timing: Jan 30 2025 19:08:38
Script start
Timing: Jan 30 2025 19:08:38
Module start 'content'
Timing: Jan 30 2025 19:08:38
Module end 'content'
Timing: Jan 30 2025 19:08:38
Script end

Main resources:

Total runtime0.2815 sec
Peak memory usage8,192.0000 KB
Database Queries141

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0071 589.7500370.2578
Module start 'content' 0.00710.0165 960.00781,010.1484
Module end 'content' 0.02350.2579 1,970.15633,899.5859
Script end 0.2815  5,869.7422 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00461.6164200.0002
Check MTime0.00140.5035200.0001
Mysql Total
Database connection0.00070.231310.0007
Mysqli_queries0.189767.36271410.0013
Looping result0.00110.38681390.0000
Template Total0.257691.510.2576
Template load0.00070.255310.0007
Template processing0.256991.236110.2569
Override
Cache load0.00050.173710.0005
Sytem overhead
Fetch class attribute can translate value0.00150.516810.0015
XML
Image XML parsing0.00030.119710.0003
General
dbfile0.01134.0132200.0006
String conversion0.00000.001730.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 1
 Number of unique templates used: 1

Time used to render debug report: 0.0001 secs