How to speed up import of 1M objects?

«

Previous topic

|

|

Next topic

»

Author	Message
zurgutt -	Wednesday 22 December 2010 3:38:19 pm I have to migrate lots of content from one ez installation to other (4.0 -> 4.3). It is not a straight upgrade, there are custom scripts to convert objects to new classes etc. Problem is, there is nearly a million objects, so while export runs at reasonable speed, the import/publish operations are slow and by my estimates would take days to finish. I can dedicate a server for this operation and tune it specificly. It is a reasonably fast box with Xeon E5520@2.27GHz and 12G of ram. Can you suggest any specific tuneups or tricks to temporarily speed up insert/publish operations for the duration of import? Certified eZ developer looking for projects. zurgutt at gg.ee
Jérôme Vieilledent	Wednesday 22 December 2010 10:03:20 pm Hi Zurgutt SQLIImport tunes up some performance settings for imports such as : View cache deactivation (only for the script) Delayed indexing Once the import process is over, a cleanup cronjob runs to clear the cache and trigger indexing. If you're not using this extension, maybe you should consider it. You could do your transformation stuffs in your important handler :)
Ivo Lukac	Thursday 23 December 2010 4:41:51 am I second everything what Jerome wrote. With additional few notes: 1. most important thing is to spread nodes over lot of parent nodes. We had lot of bad experience with importing thousands of objects under same node as single publish is a bit slower with every new sibling. I didn't have time to investigate why is that, maybe it can be avoided somehow... 2. to reduce single publish try to hack temporary "publish" operation definition in kernel/content/operation_defintion.php and remove every method that is not crucial, like: post_publish, remove-temporary-drafts, create-notification, register-search-object, generate-object-view-cache, clear-object-view-cache, pre_publish. Maybe even some others. You need to know exactly what you are doing, of course. Try different hacks with couple of thousands and measure the single average publish time.... http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
gilles guirand	Thursday 23 December 2010 1:22:55 pm I agree, @Ivo : When you tell "hack" : you mean execute a specific static PHP method and/or unset some INI values before importing datas, i guess :) ? -- Gilles Guirand eZ Community Board Member http://twitter.com/gandbox http://www.gandbox.fr
Ivo Lukac	Tuesday 28 December 2010 3:18:31 am " I agree, @Ivo : When you tell "hack" : you mean execute a specific static PHP method and/or unset some INI values before importing datas, i guess :) ? " No, with hack I mean go to kernel/content/operation_defintion.php and comment out some parts of publish method :) temporary just for importing http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Ivo Lukac	Tuesday 28 December 2010 5:43:44 am Aditionaly, it could be lucrative performance wise to hack out some features (e.g. browserecent, etc), but generally I think those should be possible to disable through ini settings. http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.

eZ debug

Timing:	Jan 30 2025 16:39:19
Script start
Timing:	Jan 30 2025 16:39:19
Module start 'layout'
Timing:	Jan 30 2025 16:39:19
Module start 'content'
Timing:	Jan 30 2025 16:39:19
Module end 'content'
Timing:	Jan 30 2025 16:39:19
Script end

Main resources:

Total runtime	0.0333 sec
Peak memory usage	4,096.0000 KB
Database Queries	3

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0076	589.2734	151.2109
Module start 'layout'	0.0077	0.0050	740.4844	220.7109
Module start 'content'	0.0127	0.0185	961.1953	1,006.4922
Module end 'content'	0.0311	0.0021	1,967.6875	37.9766
Script end	0.0332		2,005.6641

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0030	9.1435	14	0.0002
Check MTime	0.0012	3.4690	14	0.0001
Mysql Total
Database connection	0.0013	3.9321	1	0.0013
Mysqli_queries	0.0033	9.8545	3	0.0011
Looping result	0.0000	0.0509	1	0.0000
Template Total	0.0015	4.5	1	0.0015
Template load	0.0011	3.4267	1	0.0011
Template processing	0.0003	1.0285	1	0.0003
Override
Cache load	0.0007	2.2248	1	0.0007
General
dbfile	0.0035	10.5677	8	0.0004
String conversion	0.0000	0.0272	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded	Edit	Override
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 1 Number of unique templates used: 1

Time used to render debug report: 0.0001 secs