Import memory problem

Author Message

Olivier Versanne

Monday 22 June 2009 1:24:56 am

Hi,

I made a script to import some data from a CSV file to eZPublish database.

There is the PHP script, very basic without filling attribute content :

$class = eZContentClass::fetchByIdentifier( 'model' );

while ( ($data = fgets($f, $bufferSize)) !== false){

$res = explode(';',$data);
$object = $class->instantiate();
$object->setAttribute('remote_id', $refid);

$assignment = eZNodeAssignment::create( array( 'contentobject_id' => $object->attribute( 'id' ),   'contentobject_version' => 1, 'parent_node' => $parentId, sort_order' => 0, 'is_main' => 1 ) );
$assignment->store();
$object->store();
eZOperationHandler::execute( 'content', 'publish', array( 'object_id' => $object->attribute( 'id' ), 'version' => $object->attribute( 'current_version' ) ) );
}

And, nearby 300 objects inserted, I've got a problem of memory usage overflow. I'm using Wamp and in my php.ini file memory capacity is limited to 128 Mb. I need to import near 3000 products, and I don't think I can change on the production server the capacity to 1,25 Gb!

eZPublish version: 4.0

Do you have any idea of how to solve this problem? Is there existing some script that made my script?

Thanks for your help!

Jean-Luc Nguyen

Monday 22 June 2009 2:20:59 am

Hello,

After each new created objet, try to clear cache:

eZContentObject::clearCache( $newObjectCreatedID );

Where $newObjectCreatedID is the contentobject_id of you new created object.

http://www.acidre.com

Olivier Versanne

Monday 22 June 2009 5:46:43 am

Thanks, that works better! But, it doesn't work for Attribute storage and Assignement storage. I try to find same functions in these classes without success. Do you have any idea?

Marc Sallaberremborde

Monday 22 June 2009 12:11:03 pm

Hello,

Actually, the right method is :

eZContentCacheManager::clearContentObject( $objectID );

More details here : http://pubsvn.ez.no/doxygen/4.0/html/classeZContentCacheManager.html#381a14270e658e789757c8568b6d3b70

Olivier Versanne

Tuesday 23 June 2009 3:30:06 am

Hello,

I'm using these three functions and they're working quite well for some objects.

eZContentCacheManager::clearContentCache($object->attribute('id'));
eZContentObject::clearCache();
eZContentCacheManager::clearAllContentCache();

But, for the attribute storage of an object:

$dataMap = $object->dataMap();
[...] 
$dataMap[$key]->setAttribute( $dataType, $content );
$dataMap[$key]->store(); // here

And for the publication of an object:

eZOperationHandler::execute( 'content', 'publish', array( 'object_id' => $object->attribute( 'id' ), 'version' => $object->attribute( 'current_version' ) ) );

the three previous functions don't work (memory is still used).

What should I do to free this memory?

Thanks a lot!

Jean-Luc Nguyen

Tuesday 23 June 2009 3:36:52 am

unset( $dataMap );

?

http://www.acidre.com

Heath

Tuesday 23 June 2009 3:50:31 am

Hey you guys .. you really have to search and read the forums to learn these answers have already been shared a hundred times before, the best answer that stops all performance problems is making a master -> worker script using two scripts, one doing the work and one doing the iterating.

<i>http://ez.no/developer/forum/developer/how_to_build_an_update_cli_script_without_memory_problem

more unset global cache ideasin this one,
http://ez.no/developer/forum/developer/memory_leak_on_content_publish</i>

Cheers,
Heath

Brookins Consulting | http://brookinsconsulting.com/
Certified | http://auth.ez.no/certification/verify/380350
Solutions | http://projects.ez.no/users/community/brookins_consulting
eZpedia community documentation project | http://ezpedia.org

Olivier Versanne

Tuesday 23 June 2009 6:30:18 am

@Jean-Luc Nguyen: I'm already doing this unset.

@Marc Sallaberremborde: Your two links are interesting and I update my source code to add propositions like : unset ($GLOBALS['....']). without success.

This problem is existing for two years? There is no other trick that making two scripts?

Francisca Hernández

Tuesday 23 June 2009 2:34:05 pm

hi

we use xmlimport extension (in 4.1) to import objects to ezpublish (first we convert from excel or csv to xml with pentaho) and them we import, the system support 150 objects in each action more or less.

if you want to use your script i recommend to create ezcontentobjecttreenodelite.php as we did for other functionalities.

good luck, fran

Iguana IT, SL - http://www.iguanait.com

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.