Need for speed - How to use eZ Find search fetch instead of standard content list/tree fetch

Tuesday 13 July 2010 1:27:46 am - 10 replies

Introduction

This tutorial should give eZ Publish developers some new ideas on how to use eZ Find. The ever growing need for speed can be achieved by using eZ Find search function instead of standard content list/tree fetch functions. Solr engine, used by eZ Find, is far more superior regarding read speed than mysql (at least in the way eZ Publish uses them). This fact becomes apparent when dealing with 10 or 100 thousands of objects in your database with complicated eZ Publish SQL queries starting to slow down rapidly. If you need text search capabilities difference in read speed becomes even more obvious.

10 replies

Author	Message
Bruce Morrison	Tuesday 20 July 2010 9:50:08 pm Hi Ivo Great article! You can do class exclusions without changes to ezfind.ini . See: http://share.ez.no/forums/extensions/ez-find/exclude-class-on-the-fly Passing raw solr attribute names to the eZ find filter allows much more control over queries. Cheers Bruce My Blog: http://www.stuffandcontent.com/ Follow me on twitter: http://twitter.com/brucemorrison Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish
Ivo Lukac	Thursday 22 July 2010 3:24:57 am " Hi Ivo Great article! You can do class exclusions without changes to ezfind.ini . See: http://share.ez.no/forums/extensions/ez-find/exclude-class-on-the-fly Passing raw solr attribute names to the eZ find filter allows much more control over queries. Cheers Bruce " Thanks, I did notice that a bit too late :) http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
H-Works Agency	Wednesday 19 January 2011 7:01:48 am I'll bookmark this article ! This is a great help. EZP is Great
Philipp Kamps	Sunday 27 March 2011 11:41:22 am I did 2 tests: 1) content fetch function wrapped in eZDebug::accumulatorStart / Stop 2) ezfind search function wrapped in eZDebug::accumulatorStart / Stop The content fetch got executed in 0.0197 sec versus ezfind search 0.1307. So a simple content fetch is almost 10 times faster than eZ Find. As the article explains eZ Find does quite a long round-trip to get you the search result: - parsing input array to a solr string - open curl/socket connection to send of request - parse result string from solr - re-fetches objects from DB - builds ezp objects If you replace content fetches with ezfind fetches because of performance reasons, you want to make sure that the DB queries from a content fetch take a very long time to get executed. Like the article says it's the case in eZ Publish setups with a large number of objects. Even then, you should start with content fetch containing an attribute filter. Those content fetches are usually the slowest because the SQL statement joins the ezcontentobject_attribute table. http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Bertrand Dunogier	Sunday 27 March 2011 2:49:36 pm Thanks Philipp ! It would be very interesting to run these tests on different volumes of content, in different situations, and see when eZ Find becomes better. It is also possible that eZ Find will better handle high loads, but possibilities aren't facts :-) Bertrand Dunogier eZ Systems Engineering, Lyon http://twitter.com/bdunogier http://gplus.to/BertrandDunogier
Ivo Lukac	Monday 28 March 2011 9:44:33 am " I did 2 tests: 1) content fetch function wrapped in eZDebug::accumulatorStart / Stop 2) ezfind search function wrapped in eZDebug::accumulatorStart / Stop The content fetch got executed in 0.0197 sec versus ezfind search 0.1307. So a simple content fetch is almost 10 times faster than eZ Find. As the article explains eZ Find does quite a long round-trip to get you the search result: - parsing input array to a solr string - open curl/socket connection to send of request - parse result string from solr - re-fetches objects from DB - builds ezp objects If you replace content fetches with ezfind fetches because of performance reasons, you want to make sure that the DB queries from a content fetch take a very long time to get executed. Like the article says it's the case in eZ Publish setups with a large number of objects. Even then, you should start with content fetch containing an attribute filter. Those content fetches are usually the slowest because the SQL statement joins the ezcontentobject_attribute table. " Thanks Phillip. Maybe I didn't emphasized it enough but these advices where intended for situations when you have large database. E.g. you have several millions rows in attribute table or far more. On a average site the performance gains will be low (or even worse as you measured). I planned to add some measurements but, as usually happens, didn't find to do that. If you have some larger site with a bit more complicated queries, try to repeat your tests and share it with us :) http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Philipp Kamps	Monday 28 March 2011 11:08:10 am I just run my tests against a DB with 250-300 content objects: eZ Find --------------- eZ Find is pretty constant at ~0.2 sec. My guess is that the eZ Find wrapper takes most of the execution time (solr is probably very efficient to handle the request). Content Fetch ---------------------- For content fetch functions, I use no attribute filter nor attribute sorting and limit the fetch to 10 objects (same as the ezfind result). The speed is not very consistent: Most queries get executed between 0.05sec and 0.1sec. Sometimes I see some spikes up to 0.3sec 0.5sec. In my test, I'm bypassing db query cache but it looks like there is some other cache making most of my sql queries a little faster than eZ Find. As soon as you have it in the query cache it's super fast (eZ Find doesn't have that type of cache). I'm sure it's a different stories if you add an attribute filter or attribute sorting (joins ezcontentobject_attribute). I previous tests I saw execution times of a couple seconds for those types of fetches. http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Ivo Lukac	Monday 28 March 2011 12:35:18 pm " I just run my tests against a DB with 250-300 content objects: eZ Find --------------- eZ Find is pretty constant at ~0.2 sec. My guess is that the eZ Find wrapper takes most of the execution time (solr is probably very efficient to handle the request). Content Fetch ---------------------- For content fetch functions, I use no attribute filter nor attribute sorting and limit the fetch to 10 objects (same as the ezfind result). The speed is not very consistent: Most queries get executed between 0.05sec and 0.1sec. Sometimes I see some spikes up to 0.3sec 0.5sec. In my test, I'm bypassing db query cache but it looks like there is some other cache making most of my sql queries a little faster than eZ Find. As soon as you have it in the query cache it's super fast (eZ Find doesn't have that type of cache). I'm sure it's a different stories if you add an attribute filter or attribute sorting (joins ezcontentobject_attribute). I previous tests I saw execution times of a couple seconds for those types of fetches. " In Mysql you have a in memory query cache which you can size based on server RAM. So you can use lets say 1G. Query cache will cache results for queries so if all results of most used queries fit in 1G you will never have speed issues :) The problem is that with large sites you have lots of different queries with lots of joins so you will hit that barrier soon. Then you should think about solr & ezfind, as you can have millions of nodes, the query will always be fast. And solr is much easier to scale then Mysql. This is especially true if you need to do some text searching. BTW do you have a ez db with at least 10000 nodes? http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Philipp Kamps	Monday 28 March 2011 12:41:02 pm Sorry - I missed 3 zeros: my DB has 250.000 300.000 content objects http://www.mugo.ca Mugo Web, eZ Partner in Vancouver, Canada
Bertrand Dunogier	Tuesday 29 March 2011 6:34:17 am In any case, it is very likely that eZ Find will be faster for very complex searches, with mutliple conditions. It should also be able to handle filtering a bit better in some extreme conditions. Bertrand Dunogier eZ Systems Engineering, Lyon http://twitter.com/bdunogier http://gplus.to/BertrandDunogier

You must be logged in to post messages in this topic!