Need for speed - How to use eZ Find search fetch instead of standard content list/tree fetch

Tuesday 13 July 2010 1:27:46 am - 10 replies

Introduction

This tutorial should give eZ Publish developers some new ideas on how to use eZ Find. The ever growing need for speed can be achieved by using eZ Find search function instead of standard content list/tree fetch functions. Solr engine, used by eZ Find, is far more superior regarding read speed than mysql (at least in the way eZ Publish uses them). This fact becomes apparent when dealing with 10 or 100 thousands of objects in your database with complicated eZ Publish SQL queries starting to slow down rapidly. If you need text search capabilities difference in read speed becomes even more obvious.

10 replies

Author	Message
Bruce Morrison	Tuesday 20 July 2010 9:50:08 pm Hi Ivo Great article! You can do class exclusions without changes to ezfind.ini . See: http://share.ez.no/forums/extensions/ez-find/exclude-class-on-the-fly Passing raw solr attribute names to the eZ find filter allows much more control over queries. Cheers Bruce My Blog: http://www.stuffandcontent.com/ Follow me on twitter: http://twitter.com/brucemorrison Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish
Ivo Lukac	Thursday 22 July 2010 3:24:57 am " Hi Ivo Great article! You can do class exclusions without changes to ezfind.ini . See: http://share.ez.no/forums/extensions/ez-find/exclude-class-on-the-fly Passing raw solr attribute names to the eZ find filter allows much more control over queries. Cheers Bruce " Thanks, I did notice that a bit too late :) http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
H-Works Agency	Wednesday 19 January 2011 7:01:48 am I'll bookmark this article ! This is a great help. EZP is Great
Philipp Kamps	Sunday 27 March 2011 11:41:22 am I did 2 tests: 1) content fetch function wrapped in eZDebug::accumulatorStart / Stop 2) ezfind search function wrapped in eZDebug::accumulatorStart / Stop The content fetch got executed in 0.0197 sec versus ezfind search 0.1307. So a simple content fetch is almost 10 times faster than eZ Find. As the article explains eZ Find does quite a long round-trip to get you the search result: - parsing input array to a solr string - open curl/socket connection to send of request - parse result string from solr - re-fetches objects from DB - builds ezp objects If you replace content fetches with ezfind fetches because of performance reasons, you want to make sure that the DB queries from a content fetch take a very long time to get executed. Like the article says it's the case in eZ Publish setups with a large number of objects. Even then, you should start with content fetch containing an attribute filter. Those content fetches are usually the slowest because the SQL statement joins the ezcontentobject_attribute table. http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Bertrand Dunogier	Sunday 27 March 2011 2:49:36 pm Thanks Philipp ! It would be very interesting to run these tests on different volumes of content, in different situations, and see when eZ Find becomes better. It is also possible that eZ Find will better handle high loads, but possibilities aren't facts :-) Bertrand Dunogier eZ Systems Engineering, Lyon http://twitter.com/bdunogier http://gplus.to/BertrandDunogier
Ivo Lukac	Monday 28 March 2011 9:44:33 am " I did 2 tests: 1) content fetch function wrapped in eZDebug::accumulatorStart / Stop 2) ezfind search function wrapped in eZDebug::accumulatorStart / Stop The content fetch got executed in 0.0197 sec versus ezfind search 0.1307. So a simple content fetch is almost 10 times faster than eZ Find. As the article explains eZ Find does quite a long round-trip to get you the search result: - parsing input array to a solr string - open curl/socket connection to send of request - parse result string from solr - re-fetches objects from DB - builds ezp objects If you replace content fetches with ezfind fetches because of performance reasons, you want to make sure that the DB queries from a content fetch take a very long time to get executed. Like the article says it's the case in eZ Publish setups with a large number of objects. Even then, you should start with content fetch containing an attribute filter. Those content fetches are usually the slowest because the SQL statement joins the ezcontentobject_attribute table. " Thanks Phillip. Maybe I didn't emphasized it enough but these advices where intended for situations when you have large database. E.g. you have several millions rows in attribute table or far more. On a average site the performance gains will be low (or even worse as you measured). I planned to add some measurements but, as usually happens, didn't find to do that. If you have some larger site with a bit more complicated queries, try to repeat your tests and share it with us :) http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Philipp Kamps	Monday 28 March 2011 11:08:10 am I just run my tests against a DB with 250-300 content objects: eZ Find --------------- eZ Find is pretty constant at ~0.2 sec. My guess is that the eZ Find wrapper takes most of the execution time (solr is probably very efficient to handle the request). Content Fetch ---------------------- For content fetch functions, I use no attribute filter nor attribute sorting and limit the fetch to 10 objects (same as the ezfind result). The speed is not very consistent: Most queries get executed between 0.05sec and 0.1sec. Sometimes I see some spikes up to 0.3sec 0.5sec. In my test, I'm bypassing db query cache but it looks like there is some other cache making most of my sql queries a little faster than eZ Find. As soon as you have it in the query cache it's super fast (eZ Find doesn't have that type of cache). I'm sure it's a different stories if you add an attribute filter or attribute sorting (joins ezcontentobject_attribute). I previous tests I saw execution times of a couple seconds for those types of fetches. http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Ivo Lukac	Monday 28 March 2011 12:35:18 pm " I just run my tests against a DB with 250-300 content objects: eZ Find --------------- eZ Find is pretty constant at ~0.2 sec. My guess is that the eZ Find wrapper takes most of the execution time (solr is probably very efficient to handle the request). Content Fetch ---------------------- For content fetch functions, I use no attribute filter nor attribute sorting and limit the fetch to 10 objects (same as the ezfind result). The speed is not very consistent: Most queries get executed between 0.05sec and 0.1sec. Sometimes I see some spikes up to 0.3sec 0.5sec. In my test, I'm bypassing db query cache but it looks like there is some other cache making most of my sql queries a little faster than eZ Find. As soon as you have it in the query cache it's super fast (eZ Find doesn't have that type of cache). I'm sure it's a different stories if you add an attribute filter or attribute sorting (joins ezcontentobject_attribute). I previous tests I saw execution times of a couple seconds for those types of fetches. " In Mysql you have a in memory query cache which you can size based on server RAM. So you can use lets say 1G. Query cache will cache results for queries so if all results of most used queries fit in 1G you will never have speed issues :) The problem is that with large sites you have lots of different queries with lots of joins so you will hit that barrier soon. Then you should think about solr & ezfind, as you can have millions of nodes, the query will always be fast. And solr is much easier to scale then Mysql. This is especially true if you need to do some text searching. BTW do you have a ez db with at least 10000 nodes? http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Philipp Kamps	Monday 28 March 2011 12:41:02 pm Sorry - I missed 3 zeros: my DB has 250.000 300.000 content objects http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Bertrand Dunogier	Tuesday 29 March 2011 6:34:17 am In any case, it is very likely that eZ Find will be faster for very complex searches, with mutliple conditions. It should also be able to handle filtering a bit better in some extreme conditions. Bertrand Dunogier eZ Systems Engineering, Lyon http://twitter.com/bdunogier http://gplus.to/BertrandDunogier

You must be logged in to post messages in this topic!

eZ debug

Timing:	Jan 18 2025 02:05:54
Script start
Timing:	Jan 18 2025 02:05:54
Module start 'layout'
Timing:	Jan 18 2025 02:05:54
Module start 'content'
Timing:	Jan 18 2025 02:05:54
Module end 'content'
Timing:	Jan 18 2025 02:05:54
Script end

Main resources:

Total runtime	0.2561 sec
Peak memory usage	4,096.0000 KB
Database Queries	99

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0064	588.4219	152.6875
Module start 'layout'	0.0064	0.0029	741.1094	39.6172
Module start 'content'	0.0092	0.2457	780.7266	939.4063
Module end 'content'	0.2549	0.0012	1,720.1328	38.3984
Script end	0.2561		1,758.5313

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0031	1.2267	16	0.0002
Check MTime	0.0013	0.5245	16	0.0001
Mysql Total
Database connection	0.0009	0.3681	1	0.0009
Mysqli_queries	0.1419	55.3820	99	0.0014
Looping result	0.0010	0.3958	97	0.0000
Template Total	0.2213	86.4	2	0.1107
Template load	0.0020	0.7616	2	0.0010
Template processing	0.2194	85.6427	2	0.1097
Template load and register function	0.0001	0.0519	1	0.0001
states
state_id_array	0.0015	0.5797	2	0.0007
state_identifier_array	0.0012	0.4604	3	0.0004
Override
Cache load	0.0019	0.7565	98	0.0000
Sytem overhead
Fetch class attribute name	0.0016	0.6294	16	0.0001
Fetch class attribute can translate value	0.0009	0.3614	5	0.0002
class_abstraction
Instantiating content class attribute	0.0000	0.0181	21	0.0000
XML
Image XML parsing	0.0037	1.4551	5	0.0007
General
dbfile	0.0034	1.3450	41	0.0001
String conversion	0.0000	0.0027	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	forum_topic/full.tpl	extension/community_design/design/suncana/override/templates/forum_topic/full.tpl
11	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
22	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
10	content/datatype/view/ezimage.tpl	<No override>	extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tpl
2	content/datatype/view/ezxmltags/strong.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/strong.tpl
2	content/datatype/view/ezxmltags/link.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/link.tpl
8	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
3	content/datatype/view/ezxmltags/quote.tpl	datatype/ezxmltext/quote.tpl	extension/ezwebin/design/ezwebin/override/templates/datatype/ezxmltext/quote.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 60 Number of unique templates used: 9

Time used to render debug report: 0.0001 secs