Forums / General / How to speed up import of 1M objects?

How to speed up import of 1M objects?

«

Previous topic

|

|

Next topic

»

Author	Message
zurgutt -	Wednesday 22 December 2010 3:38:19 pm I have to migrate lots of content from one ez installation to other (4.0 -> 4.3). It is not a straight upgrade, there are custom scripts to convert objects to new classes etc. Problem is, there is nearly a million objects, so while export runs at reasonable speed, the import/publish operations are slow and by my estimates would take days to finish. I can dedicate a server for this operation and tune it specificly. It is a reasonably fast box with Xeon E5520@2.27GHz and 12G of ram. Can you suggest any specific tuneups or tricks to temporarily speed up insert/publish operations for the duration of import? Certified eZ developer looking for projects. zurgutt at gg.ee
Jérôme Vieilledent	Wednesday 22 December 2010 10:03:20 pm Hi Zurgutt SQLIImport tunes up some performance settings for imports such as : View cache deactivation (only for the script) Delayed indexing Once the import process is over, a cleanup cronjob runs to clear the cache and trigger indexing. If you're not using this extension, maybe you should consider it. You could do your transformation stuffs in your important handler :)
Ivo Lukac	Thursday 23 December 2010 4:41:51 am I second everything what Jerome wrote. With additional few notes: 1. most important thing is to spread nodes over lot of parent nodes. We had lot of bad experience with importing thousands of objects under same node as single publish is a bit slower with every new sibling. I didn't have time to investigate why is that, maybe it can be avoided somehow... 2. to reduce single publish try to hack temporary "publish" operation definition in kernel/content/operation_defintion.php and remove every method that is not crucial, like: post_publish, remove-temporary-drafts, create-notification, register-search-object, generate-object-view-cache, clear-object-view-cache, pre_publish. Maybe even some others. You need to know exactly what you are doing, of course. Try different hacks with couple of thousands and measure the single average publish time.... http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
gilles guirand	Thursday 23 December 2010 1:22:55 pm I agree, @Ivo : When you tell "hack" : you mean execute a specific static PHP method and/or unset some INI values before importing datas, i guess :) ? -- Gilles Guirand eZ Community Board Member http://twitter.com/gandbox http://www.gandbox.fr
Ivo Lukac	Tuesday 28 December 2010 3:18:31 am " I agree, @Ivo : When you tell "hack" : you mean execute a specific static PHP method and/or unset some INI values before importing datas, i guess :) ? " No, with hack I mean go to kernel/content/operation_defintion.php and comment out some parts of publish method :) temporary just for importing http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Ivo Lukac	Tuesday 28 December 2010 5:43:44 am Aditionaly, it could be lucrative performance wise to hack out some features (e.g. browserecent, etc), but generally I think those should be possible to disable through ini settings. http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac

eZ debug

Timing:	Jan 18 2025 02:20:14
Script start
Timing:	Jan 18 2025 02:20:14
Module start 'content'
Timing:	Jan 18 2025 02:20:14
Module end 'content'
Timing:	Jan 18 2025 02:20:14
Script end

Main resources:

Total runtime	0.0154 sec
Peak memory usage	2,048.0000 KB
Database Queries	4

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0050	588.8594	180.8438
Module start 'content'	0.0050	0.0046	769.7031	102.5703
Module end 'content'	0.0096	0.0057	872.2734	78.6875
Script end	0.0153		950.9609

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0021	13.8803	12	0.0002
Check MTime	0.0011	6.8589	12	0.0001
Mysql Total
Database connection	0.0007	4.6634	1	0.0007
Mysqli_queries	0.0021	13.5861	4	0.0005
Looping result	0.0000	0.0790	2	0.0000
Template Total	0.0053	34.3	1	0.0053
Template load	0.0008	5.4685	1	0.0008
Template processing	0.0044	28.7484	1	0.0044
Override
Cache load	0.0006	3.8769	1	0.0006
General
dbfile	0.0012	7.5277	10	0.0001
String conversion	0.0000	0.0310	3	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded	Edit	Override
1	pagelayout.tpl	<No override>	extension/sevenx/design/simple/templates/pagelayout.tpl
Number of times templates used: 1 Number of unique templates used: 1

Time used to render debug report: 0.0001 secs