|
Author
|
Message
|
|
Jérôme Vieilledent
|
Wednesday 18 March 2009 3:50:39 pm
Hi
I just read the documentation about the EZP cluster feature and, as caches and storage are stored in the database, I'm worried about the performance...
Isn't it slower to put these resources in the DB ? I'm regularly asked about potentially high traffic websites configuration and load balancer stuffs, and I'm currently used to replying to use a filer (NFS), not knowing very much about the cluster feature performance.
Can anyone give me explanation ?
Thanks ! :-)
|
|
Jérôme Renard
|
Thursday 19 March 2009 1:12:35 am
Hi Jérôme
I just read the documentation about the EZP cluster feature and, as caches and storage are stored in the database, I'm worried about the performance...
Using eZ Publish in cluster mode (DB handler) is always slower, but this is a trade-off between using one mamoth server or several nodes in a cluster.
Isn't it slower to put these resources in the DB ? I'm regularly asked about potentially high traffic websites configuration and load balancer stuffs, and I'm currently used to replying to use a filer (NFS), not knowing very much about the cluster feature performance.
Storing caches and in the database is indeed slower, this is why we always recommend to use a reverse proxy to service (at least) images, since they are all stored in the database. If you want to improve latency for clustered images, you can use mod_ezclustered_image [http://projects.ez.no/mod_ezclustered_image] in conjuction with a good reverse proxy like Varnish (please no Squid). On the other hand there are quite a few "replication" mecanisms in the underliying clustering system. eZ Publish will query the cluster database only if the file is stale or if it does not exists on the node's local FS, this dramatically reduces the amount of SQL queries to the cluster (I am not talking about content at all here, this is completely different). Before eZ Publish 4.1 we had an issue with NFS (and others NFS like File Systems) because NFS does not support *flock* calls (which does not mean NFS does not support locks), unfortunately eZ Publish locks a lot of stuff when it comes to caches and cache purges / update, the result was that sometimes it took more than 20 seconds to update a ViewCache file when a content was updated. This was not acceptable. And this issue is fixed with the new Stale Cache feature (more informations to come in a article in a near future). From the most basic point of views, this feature is more NFS friendly. I will not dive into details here. 'Hope that helps.
|
|
Jérôme Vieilledent
|
Thursday 19 March 2009 1:38:09 am
Hi Jérôme and Thank you ! When you're talking about a reverse proxy like Varnish, can you explain me what it's supposed to do here ? I'm not familiar with it. Thanks
|
|
Jérôme Renard
|
Thursday 19 March 2009 2:03:40 am
Hi Jérôme and Thank you !
no worries :)
When you're talking about a reverse proxy like Varnish, can you explain me what it's supposed to do here ? I'm not familiar with it.
Sure, In a "normal" mode (I take only one node to avoid all Load Balancer stuff which will make the schema harder to read and is not that relevant here) you have basically this :
Internet -> Apache -> eZ Publish
Which means for each HTTP request you hit Apache which returns you the result you expect (from binary files, videos...) to plain/* file like HTML, CSS, JS, whatever you want. The problem with that type of architecture is that Apache is over sollicitated. This is where a reverse proxy comes into play, with a reverse proxy (like Varnish) you now have this :
Internet -> Varnish -> Apache -> eZ Publish
In that case, Varnish will intercept any possible request and return the expected result from its cache without even requesting Apache and eZ Publish which reduces load especially when images comes from a Database ;) Please note that what is described here is the most basic view possible, just for you to get the concept of what a Reverse Proxy is actually. :)
|
|
Bertrand Dunogier
|
Thursday 19 March 2009 2:48:07 am
Hi Jerome !
I'm not sure Jerome mentionned this, but since eZ publish 3.10, we have an additional performances feature when it comes to clustering:
- files are stored to the database (it can be a distinct MySQL server, it always helps)
- when requested, the files are copied locally to the filesystem, at the exact location they would have been placed in without a DB cluster
- when requested again, the database is checked to ensure that the file is not expired (since it can have been expired by another cluster node) then used from the filesystem. It makes a huge difference. We have plans for further improvements in that direction, but we can't reveal anything until we have decided we'll do it. Keep in touch ;)
Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier
|
|
Jérôme Renard
|
Thursday 19 March 2009 2:55:27 am
Hi Bertrand
I'm not sure Jerome mentionned this, but since eZ publish 3.10, we have an additional performances feature when it comes to clustering:
- files are stored to the database (it can be a distinct MySQL server, it always helps)
- when requested, the files are copied locally to the filesystem, at the exact location they would have been placed in without a DB cluster
- when requested again, the database is checked to ensure that the file is not expired (since it can have been expired by another cluster node) then used from the filesystem.
This what I called a "replication" system but that was obscure I guess. :)
|
|
Jérôme Vieilledent
|
Thursday 19 March 2009 3:08:26 am
Thanks both for your quick replies ^^
I guess that it is advised to put Varnish in front even without cluster ?
We have a big client for which we have integrated eZ Publish 4.0.1 and they chose to use NFS with a dedicated filer and 2 front Apache webservers. The webservers DocumentRoot actually points to a unique eZ Publish instance on this filer.
It works well (we have a full Akamai CDN) but I wonder if it is a good choice vs cluster mode... The main problem with this architecture is that PHP currently doesn't support flocks with NFS (but the problem should be resolved with the new stale cache in 4.1, shouldn't it ?) What do you think ?
|
|
Jérôme Vieilledent
|
Thursday 19 March 2009 3:10:33 am
Another point : Do you have recommendation for Varnish conf ?
|
|
Jérôme Renard
|
Thursday 19 March 2009 3:25:15 am
I guess that it is advised to put Varnish in front even without cluster ?
Yep.
We have a big client for which we have integrated eZ Publish 4.0.1 and they chose to use NFS with a dedicated filer and 2 front Apache webservers. The webservers DocumentRoot actually points to a unique eZ Publish instance on this filer.
It works well (we have a full Akamai CDN) but I wonder if it is a good choice vs cluster mode...
Akamai's CDN != eZ Publish cluster mode.
The main problem with this architecture is that PHP currently doesn't support flocks with NFS (but the problem should be resolved with the new stale cache in 4.1, shouldn't it ?)
Actually the problem is the following :
- locking with PHP is usually done through flock
- eZ Publish uses flock
- NFS does not support flock
- it is possible to lock on NFS using pecl_dio
- pecl_dio is no longer maintained http://pecl.php.net/package/dio, but still available on CVS : http://cvs.php.net/viewvc.cgi/pecl/dio/
|
|
Jérôme Renard
|
Thursday 19 March 2009 3:26:17 am
Another point : Do you have recommendation for Varnish conf ?
You might be interested in this thread :
http://projects.ez.no/ezvlogin/forum/general/working_with_custom_headers/re_working_with_custom_headers5 :)
|
|
Jérôme Vieilledent
|
Thursday 19 March 2009 3:30:35 am
Thanks ! Akamai's CDN != eZ Publish cluster mode.
Actually, I was talking about full NFS configuration (with the whole ezp instance on a filer) vs cluster mode
|
|
Bertrand Dunogier
|
Friday 20 March 2009 1:47:05 pm
My personal opinion is that a software based approach of clustering is our best option. The reason is actually simple: depending on a distributed file system makes us too depending of 3rd party software evolutions. Dealing with NFS locks, file attributes caching and so on is a pain. Really :) But the FACT is that eZ publish performs decently on correctly scaled & tuned platforms. But there still are a few issues (caches generation in a high concurrency situation due to improper support of locking). My feeling about the currently DB based cluster is that it is the approach that will let us implement the most scalable platform, because we do have control over it. We can still take advantage of widely available software (we do depend on a few MySQL features), and it is also a good thing. I want to tell you: please try ! We've made tests with a DB based cluster, but not in an environnement like the one you describe. We would need to setup powerful servers, tune them, implement a project that replicates a real life situation, and so on. We will work on it, but you know how it is ;)
These are ideas we have discussed internally. Specifications can be expected in the future. Any future...
- memcache layer that would prevent database checks of local cache metadata
- mixed storage: use the database to manage metadata, store the files on distributed filesystem
Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier
|
|
Gaetano Giunta
|
Friday 20 March 2009 3:44:28 pm
To add some further insight:
- flock on nfs is not necessarily unsupported: it just varies greatly from one nfs server/client implementation to the other. So ymmv. Here: http://nfs.sourceforge.net/, it says that the linux nfs client only supports flock since kernel 2.6.12.
Real life tests have shown filers to be very good at handling nfs serving for eZP, linux servers being quite bad and solaris ones being worse. - in the long term, not having to rely on very costly filers to store cache is surely a good idea. But getting it done right takes a lot of time and testing. Cluster code is much better in 4.1 than in 4.0 than in 3.9 etc... - ...otoh if you already have paid for a filer, you might as well take advantage of it! - if your filer is connected via fibre (san), you can try to use ocfs2 or gfs as shared-storage filesystem, instead of nfs. If your storage is attached via gigabit-ethernet, you might do the same using iscsi, but results so far indicate that that config is no better than plain nfs - some dbs are much (much) better than others at the kind of load that the management of a huge pile of blobs imposes. And I am thinking about Oracle here... - a reverse proxy can be much more cost-effective than a cluster setup anyway. the cluster setup remains of course your best bet for availability.
- a simple reverse proxy solution can even be an nginx/lighttpd process, put in front of apache, on the same server. The benefits are multiple:
- a light proxy process means less ram used when user requests a static file and the proxy process serves it without connecting to the apache process
- when apache+php is done preparing the html output, it can send it immediately to the proxy process and start serving another 'heavy request'. The proxy process buffers the output until the client has finished receiving it, which can be a long time if the client is on a slow network connection - reverse proxies can cache full html pages too, besides static content. But then you will spend a lot of time to properly configure timeouts and expiry rules. Look for the cache-expiry-handler in ezflow to get some extra ideas - there are at least 2 eZP extensions for varnish that look very promising
Principal Consultant International Business
Member of the Community Project Board
|
|
Bertrand Dunogier
|
Friday 20 March 2009 4:04:12 pm
Gaetano, we should start a document somewhere that lists the possible setups (regarding DFS at least) and the known issues / strength, don't you think ?
Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier
|
|
Jérôme Vieilledent
|
Friday 20 March 2009 4:54:46 pm
We will work on it, but you know how it is ;)
Yes I do know ! ;-)
OK, I understand, the best scalable option is the DB cluster. You're right, Bertrand, when you say that tuning NFS is a real pain...
On the other hand, it's really tempting to use a filer when you have it (this is case for us), but it has to be a REAL filer (not Linux or Solaris server) @Gaetano : So you advise to use Oracle instead of MySQL for the cluster tables ? What is the gain ? Gaetano, we should start a document somewhere that lists the possible setups (regarding DFS at least) and the known issues / strength, don't you think ?
It would be really great ! Subsidiary question : How Memcache could be used in (and also out) a clustered environment ? Thanks :-)
|
|
Jérôme Renard
|
Saturday 21 March 2009 12:07:19 pm
Hello Jérôme, I totally agree with what Geatano and Bertrand said above, working with multiple (open source, proprietary) implementations of NFS is a pain. Concerning Memcached we are working on a new cache API which would give us the ability to cache anything we want and store it wherever we want, so Memcached could be used. But all this work is in its very early stage and we have to work a lot more before delivering something correct. Cheers :)
|
|
Kai Krämer
|
Tuesday 24 March 2009 6:13:08 am
We have just published an article about some drawbacks of the eZ cluster. It also describes an alternative concept called "Distributed Cache" that solves most of the problems. http://www.ymc.ch/weblog/some_due_modifications_to_the_ez_cluster
|