eZFind 2.0 very slow

Next topic

Author	Message
H-Works Agency	Tuesday 22 September 2009 1:50:24 am Hello, We installed eZFind on a medium size website (300.000 contents, 2,5 millions in content attributes table). We tried to enable eZFind on this site and its a complete failure : - Indexation take like 7 or 8 hours - the site become so slow it can't be used Its not a Solr problem because direct query on the server is very quick. Are those problems related to the fact my DB is in MYISAM format ? What are the potential aspects of ezpublish connector to Solr which can cause such performance problems ? Thanx for any help ! EZP is Great
Sander van den Akker	Tuesday 22 September 2009 2:34:39 am Do you have many attributes of type ezkeyword (which automatically relates objects to each other)? This is a known performance hog because of infinite loops. eZ Publish certified developer http://auth.ez.no/certification/verify/392313
H-Works Agency	Tuesday 22 September 2009 2:46:56 am Uhm yes we have a big quantity of related objects & ezkeywords....is there a way to correct this bug/problem ? EZP is Great
Paul Borgermans	Tuesday 22 September 2009 5:11:11 am Hi Martin What ezp version are you using? MyISAM is in every aspect a bad choice and not supported What the relation is with ez find and the performance problem: - if it is for indexing - or browsing/searching - or both? otoh, the slow indexing may be due to: - not using the dedicated Solr indexing script ( extension/ezfind/bin/php/updatesearchindexsolr.php ) - a bug in ez find 2.0 where teh spell checker is rebuilt on every "commit" to Solr ... But I need to know more details to help you out on this Paul eZ Publish, eZ Find, Solr expert consulting and training http://twitter.com/paulborgermans
H-Works Agency	Tuesday 22 September 2009 6:23:21 am Hi Paul & thanx for your answer, Ok for the table engine, we're gonna switch to InnoDB ! (Though i had a very bad experience with this engine with a corrupted ezcontentobject_attribute table which could never be repaired = restore = data lost). eZFind indexing is reaching 100% although i find it very slow (5 to 6 hours) with provided indexing script (extension/ezfind/bin/php/updatesearchindexsolr.php). Once indexed search is 10 times slower than with the default search engine, and the whole site is slow. The problem seems to be related to the modification/publication of objects, it looks like everytime someone is modifying something ezfind is working on something. When making a "top" while trying to display a page we identified a "java" SolR process taking 120% of CPU. Given that we don't understand what its doing. There is a few points about eZFind we don't understand : - How new objects are indexed ? - Do we have to remove our cron job "updatesearchindex" and install eZFind one instead ? - After indexation this script is always restarting from scratch so how can it index new contents ? I hope this is clear enough for you to leed me on a few solutions. Thanx anyway for your time. Martin EZP is Great
Ivo Lukac	Tuesday 22 September 2009 6:55:49 am First of all enable DelayedIndexing. http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Sander van den Akker	Tuesday 22 September 2009 7:01:41 am I have the same problem. When eZ Find is enabled, publishing content is much slower. By disabling the 'searchable' flag on some class attributes I managed to reduce the response time. However, this is not always an option. You might also want to read this thread: http://ez.no/developer/forum/extensions/ez_find/ezfind2_indexing_speed_incredibly_low_er/re_ezfind2_indexing_speed_incredibly_low_er__2 eZ Publish certified developer http://auth.ez.no/certification/verify/392313
Paul Borgermans	Tuesday 22 September 2009 9:42:36 am A few things to improve the speed and large sites: - in [IndexOptions] set OptimizeOnCommit=disabled. Then configure an additional cron that calls the script extension/cronjobs/ezfoptimizeindex.php once a day or so (during low traffic hours) This is most probably going to solve most performance problems For additional performance gains: - patch the solr.war file with the one from trunk in extension/ezfind/java/webapps/solr.war - check that the spellcheck configuration is set to rebuild upon optimize in extension/ezfind/java/solr/conf/solrconfig.xml (this is the default, but may not work until you patched solr.war) This otherwise also causes high CPU load, as the spell chek index would otherwise be built on every object update (and with several 100k of objects, that takes a lot of time) Abut delayed indexing: this should not be much of a concern with the above changes, but in any case, you should use the ezfind provided cronjob ezfindexcontent.php (to be run rather regularly, like every 10 minutes) in combination with the daily optilize cronjob hth Paul eZ Publish, eZ Find, Solr expert consulting and training http://twitter.com/paulborgermans
Paul Borgermans	Tuesday 22 September 2009 9:47:35 am One additional note for delayed indexing: There is a bug in eZPublish 4.1.3 that causes delayed indexing only to work when edits are done once between runs of the ezfindexcontent.php cronjob. Please use the patched file from svn: http://pubsvn.ez.no/nextgen/release/4.1.4/kernel/content/ezcontentoperationcollection.php to avoid this bug as well hth Paul eZ Publish, eZ Find, Solr expert consulting and training http://twitter.com/paulborgermans
Carlos Revillo	Wednesday 23 September 2009 12:35:14 pm I second Paul about optimizeoncommit setting. We had this problem until we disable it. from then, we have delayed indexing disabled but good perfomance too.
H-Works Agency	Tuesday 11 January 2011 9:25:22 am Great !!! Disabling this variable resolved my problem. Thanx Paul. EZP is Great
H-Works Agency	Thursday 27 January 2011 3:59:09 am Hello everyone, I have still a problem on this : In fact my speed problem were totally resolved with the modifications in ezfind.ini mentionned above but the solr index isn't updated after running : php runcronjobs.php -q -s siteaccess ezfindexcontent 2>&1 My question is : What exact cronjob script is supposed to run to update the solr index every 10 minutes ? Documentation says "run a cronjob if you delay indexing" but never mention explicitly which cronjob. When i read the ezfindexcontent code i seems to wait for lines inside ezpending_actions sql table to know which objects to process. Given this my second question is : Why this table is always empty after an admin content publish when "OptimizeOnCommit=disabled" ? My third question is : Do i need to manually insert a line in ezpending_actions to fire solr index update ? I find no documentation on this, any help would be great. Cheers EZP is Great
Gaetano Giunta	Saturday 29 January 2011 2:37:38 am q: "What exact cronjob script is supposed to run to update the solr index every 10 minutes ?" a: the ezfindexcontent one The pending actions table should be filled up by eZP automatically at the time objects are published (thsi only happens if delayedindexing is enabled, otherwise they get indexed immediately). So you do not need to touch it. optimizeoncommit is a setting that only affects sending optimize commands to solr, it has no impact on who/when the indexation takes place. So I recommend that to debug you - make sure have no cronjobs running at all - publish an object - look at ezpendingactions table. No lines in there => you have actually not enabled delayedindexing correctly - run the ezfindexobject cronjob, and check that the line in ezpendingactions is gone Principal Consultant International Business Member of the Community Project Board
H-Works Agency	Saturday 29 January 2011 3:02:23 am Thank you Gaetano for this clear debugging process. It works after setting DelayedIndexing=enabled in site.ini. Thanx ! EZP is Great

eZ debug

Timing:	Jan 18 2025 03:09:06
Script start
Timing:	Jan 18 2025 03:09:06
Module start 'layout'
Timing:	Jan 18 2025 03:09:06
Module start 'content'
Timing:	Jan 18 2025 03:09:07
Module end 'content'
Timing:	Jan 18 2025 03:09:07
Script end

Main resources:

Total runtime	1.1549 sec
Peak memory usage	4,096.0000 KB
Database Queries	98

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0047	587.9141	152.6250
Module start 'layout'	0.0047	0.0026	740.5391	39.4453
Module start 'content'	0.0073	1.1462	779.9844	854.0156
Module end 'content'	1.1535	0.0014	1,634.0000	32.1719
Script end	1.1549		1,666.1719

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0030	0.2614	16	0.0002
Check MTime	0.0013	0.1125	16	0.0001
Mysql Total
Database connection	0.0007	0.0569	1	0.0007
Mysqli_queries	1.0281	89.0187	98	0.0105
Looping result	0.0011	0.0971	96	0.0000
Template Total	1.1228	97.2	2	0.5614
Template load	0.0018	0.1594	2	0.0009
Template processing	1.1210	97.0616	2	0.5605
Template load and register function	0.0001	0.0101	1	0.0001
states
state_id_array	0.0009	0.0781	1	0.0009
state_identifier_array	0.0015	0.1288	2	0.0007
Override
Cache load	0.0019	0.1632	111	0.0000
Sytem overhead
Fetch class attribute can translate value	0.0012	0.1008	6	0.0002
Fetch class attribute name	0.0016	0.1403	19	0.0001
XML
Image XML parsing	0.0031	0.2672	6	0.0005
class_abstraction
Instantiating content class attribute	0.0003	0.0252	27	0.0000
General
dbfile	0.0015	0.1299	44	0.0000
String conversion	0.0000	0.0006	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	full/forum_topic.tpl	extension/sevenx/design/simple/override/templates/full/forum_topic.tpl
13	content/datatype/view/ezimage.tpl	<No override>	extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tpl
14	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
22	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
7	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
1	content/datatype/view/ezxmltags/literal.tpl	<No override>	extension/community/design/standard/templates/content/datatype/view/ezxmltags/literal.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 59 Number of unique templates used: 7

Time used to render debug report: 0.0001 secs