How to build an update CLI script without memory problem ?

Wednesday 18 November 2009 9:15:20 am

Hi Kristof !

Well, such an iterator would indeed be a good idea. But based on how developement is organized, and on real life, I think that applying this to ezPersistentObject is a bit optimistic, especially since it is based to be heavily modified soon.

What we will probably start with is a global in-memory cache API, maybe linked to memcache and other cache mechanisms, so that we don't have all these hardcoded global calls everywhere. The iterator idea, with automatic cleanup, etc, is a very good point though.

I'm very eager to hear any idea you might have in that area though !

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

Thursday 19 November 2009 1:52:35 am

Wher writing CLI scritps I try to keep the fetch on the "outside of the for/foreach" as small as possible.

Best way would be a SQL fetch with all the nodes I need.
(The advantage is: I can use LIMIT to run the script for a part of the result. Change the offset and append the next result to the output file.)

If You need to use the fetch functions, I would try the 'load_data_map' paramter like this:

$parentNodeID = 5;

$users =& eZContentObjectTreeNode::subTreeByNodeId( 
      array( 'ClassFilterType' => 'include',
  'ClassFilterArray' => array( 'user' ),
  'SortBy' => array( 'published', false),
                'LoadDataMap' => false
 ),          
      $parentNodeID);

$i = 0;
foreach( $users as $user ) {
      $usrObj = eZContentObject::fetch( $user->attribute('contentobject_id') );      
      $dataMap =& $usrObj->attribute( 'data_map' );

      ... do something

      unset( $dataMap );
      unset( $usrObj );
}

'inside' the foreach fetch the datamap and unset all variables that are being used.

www.all2e.com

Sunday 06 December 2009 2:20:15 am

ProjectV ? Never heard of that. I've heard of Big Balls though. End of private joke.

This change is something we've been talking about, and I also know it has already been partially implemented by a member of the team. There is nothing concrete yet, but it is clear that managing in-memory cache the way we currently do in eZ is a poor approach. This needs to be consolidated with a system wide API, but it's definitely not my role to decide when it is gonna be done. Sorry :(

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier