eZ - what is ripping our resources?

Wednesday 24 March 2004 5:10:17 am

I LOVE when people leave their work, the evening/night is getting closer..the visitors of our website are almost gone and are going home for the evening, back to their families..and the awful slow eZ site is getting back to "normal"(a usable shape where processing only takes from 2-8 seconds, and it's possible to use the web again).

IT's LOVELY. We ran other sites on the same server earlier, but had to move everything to another server, because of the big resource-need eZ publish is having.

I hereby ask for some tuningtips of eZ publish. Haven't found this doc. anywhere on www.ez.no

The eZ Systems team reccomend using a PHP-accellerator.
Since this is absolutely neccessary to run an eZ publish site with "ok" results, I need tuning tips. We use Turck mmcache, but there must be any good ez docs on this (made by ez, so the eZ team know how to best tune the customers sites). Is this available eZ?

Which settings is the best for an eZ publish site (for a small site, medium site and a large site), how is it possible to administrate this accellerator(empty cache(implement in cron-jobs for clearcache), tune, conflicts-checking etc.).

We're running eZ publish 3.3 and it's making us crazy.. Some of the editors are complaining, and tell me the publishing is taking too much of their time..they're waiting and waiting...and the web stands still when they publish. Not just in the admin-part, but the whole web. Is it better to change CMS-system than develope and develope a system which has given us a slow-site from the very beginning?

We have checked out the database, and one "thing" that Emil Webber also said(in another posting) was, that there are missing A LOT of indexes in the db which make eZ publish REALLY SLOW.

Shall we(the eZ customers make the indexes ourself), or shall this actually be a part of eZ publish Content Management System?

We bought a complete eZ publish solution(also got ez to tune our site after every upgrade they've done), but we still got problems.

The whole web hangs when people publish content. We got a large db, many editors, very heavy trafic, our hardware is very good, and we got 1 GB connection. All the other sites are running wonderful, and this is the first time we've got problems with any php-mysql application ever.

PLEASE eZ, can you soon post any doc. with tuning tips on the PHP accellerators you recommend for running eZ "stable", and can you give us an overview of the databasemodel, and where it's important to make indexes in the database for speeding up the site.

Tnx.

Wednesday 24 March 2004 1:01:54 pm

I know..but, we've tried different PHP accellerators..and our latest one is the one from Turck..but the whole ez publish system is just totally "weird".

Why so much queries to get data, and to update data in the database? I think a solution is that eZ publish has to be rewritten regarding the queries and so on.

There are so many CMS-systems on the market. They offer very much the same functionallity as ez publish, so if the ez team want their customers to be satisfied, they really have to make some changes.

Gnite

Thursday 25 March 2004 3:26:09 am

Server
Dell 2650, 2x1.6 GHz
Pentium 4 Xeon cpu's, 2GB RAM, and 200GB diskspace on RAID5 controller.

OS
Redhat 9

Database: 800mb

select count(*) from ezcontentobject = 19049
select count(*) from ezcontentobject_attribute = 332778

Total hits february: 10450185
Total visits february: 255241

80 editors, and the whole system(not only the admin-part) hangs when they are publishing content.

20-25 roles, 20 sections

eZ have set up the the system for us, and done the upgrade to the 3.3 v., and they've used cache-blocks in the templates yes. They've checked our hardware and software, our configuration etc. many times, but I haven't got any answers of what we can do. Have had speed-problems since we started using ez publish, and after a change in the role-system(they told me there was a bug here half a year ago or so, and something was changed), and now the publishing process stops the whole web. It was not like this earlier, but I don't have a clue why this is happening. To heavy queries which locks the queries in mysql, or what? hmm..

I'm checking the indexes in the database now, and will add some to check if it helps in the select-processes. Unfortunately there are almost no documentation on tuning ez, adding indexes, configuring PHP accellerators on www.ez.no

Our experiences are that everything hangs when publishing content, once the publishing-process is finished, then the web works again.

The time it takes to publish content is from 3 seconds, and up to around 30 sec.

I'm interested in getting in touch with companies running 3.x sites with a minimum of a 500 mb database, many editors, many sections, many roles and some traffic. It would be nice to check out your 3.x site and check the speed etc.

Thursday 25 March 2004 6:37:22 am

Well..small amount of content is going much faster than if the text which is beeing published is very long..but I've never seen anything like it in other systems.

We have always followed the recommendation from the ez team.

We're dropping to go for apache 2 which we actually would like to do, but this is not recommended with ez publish yet, so we wait.

We're running PHP 4.3.4 with Turck MMCache v2.4.6.

Thursday 25 March 2004 7:14:24 am

Yes, I remember Balazs.

But what about the PHP versions? Is eZ publish stable with both v. 4.2.x and 4.3.x?

Does it exist any ez-doc on tuning, how to config PHP-accellerators, creating indexes on the most important places in the db etc.?

Thursday 25 March 2004 3:43:38 pm

> Our experiences are that everything hangs when publishing content, once the
> publishing-process is finished, then the web works again.
>
> The time it takes to publish content is from 3 seconds, and up to around 30 sec.

To me this sounds like the updates are waiting to get a lock on the table. i.e. they have to wait until all the selects have finished.

From http://www.mysql.com/doc/en/Table_locking.html
"To achieve a very high lock speed, MySQL uses table locking (instead of page, row, or column locking) for all storage engines except InnoDB and BDB. For large tables, table locking is much better than row locking for most applications, but there are some pitfalls."and"Table locking enables many threads to read from a table at the same time, but if a thread wants to write to a table, it must first get exclusive access. During the update, all other threads that want to access this particular table must wait until the update is done."

and

"Table locking is also disadvantageous under the following scenario:
* A client issues a SELECT that takes a long time to run.
* Another client then issues an UPDATE on a used table. This client will wait until the SELECT is finished.
* Another client issues another SELECT statement on the same table. As UPDATE has higher priority than SELECT, this SELECT will wait for the UPDATE to finish. It will also wait for the first SELECT to finish! "

Zinistry, thanks for all the information, the one thing you didn't specifically answer is weather the database is on the same server as eZ - I get the feeling that it is.

I'm pretty confident that given the situation you have mentioned that the database will be the source of the problem.

Things you could look at -

+ check and optimize all tables Look at http://www.mysql.com/doc/en/myisamchk_syntax.html
+ Change the storage engine from the default to one that supports row locking such as InnoDB http://www.mysql.com/doc/en/InnoDB_overview.html

"InnoDB provides MySQL with a transaction-safe (ACID compliant) storage engine with commit, rollback, and crash recovery capabilities. InnoDB does locking on row level and also provides an Oracle-style consistent non-locking read in SELECT statements. These features increase multi-user concurrency and performance."

I do not know how this may effect ezPublish but it may be worth a try

+ use PostgreSQL - I take it that it is still supported?

+ Look into load balancing the database - admin off the master - front end off the slave(s)
http://www.mysql.com/doc/en/Replication.htmlhttp://ez.no/community/forum/install_configuration/load_balancing_and_useslaveservertrue
http://ez.no/community/forum/install_configuration/enterprise_setup

Plenty to look at but I would focus on the database.

Cheers
Bruce

P.S. I'm also interested in other large sites.
Our largest ez site had 45,543.00 session in Feb 2 editors and a DB of 42Mb ( a shadow of what you are dealing with) Would be interested to know about the ez setup!

My Blog: http://www.stuffandcontent.com/
Follow me on twitter: http://twitter.com/brucemorrison
Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish

Friday 26 March 2004 6:32:27 am

Hi,

a little story:
This week I made an ez "hack": I've changed the behaviour of the "url-alias". German "umlauts" (and other special characters are now translated in a better way, e.g. ö => oe and instead of using the underline character, I use the "-". (Google like this.)

But the main reason was: I made a small site for a client who publishes content belonging to "homeopathy". The german word of "homeopathy" is "Homöopathie". And the translated url was - in a normal ez installation e.g.
/seminare/hom_opathie

"Homo_pathie" -> The first part of the word is "Homo" which can be translated to "gay" in english. And my client wasn't "amused".

Okay, the update worked fine, all new urls are working. But I've overseen something: The speed of the search engine spiders. 2 hours after the upload of the hack my server was under very heavy pressure: Average server load of 20 up to 40. Oh no!

The reason: 5 spiders (google, yahoo and three others) were crawling all my sites without remorse. The cache wasn't filled (because of the change of the urls I had to delete all my cache ...) So I got 45.000 page views only from spiders at this day (!).

Small change - big effect :-)

All in all I can say:
The main reasons why ez3 is slow:
-) The template engine is [*censored*]. The ez crew is knowing that, but unfortunately the "template compiler" isn't stable yet.
-) The ez3 caching system is [*censored*] -> now I am using an own "add-on" which stores the cache-input in the db. (Otherwise I've around 100.000 cache-files which are deleted if someone publishes content ...) -> see my other postings.
But the cache-db-addon is - of course - only helping if the cache is filled with data. (In my case, the cache was empty). And cache is only helping if many users are viewing the same page again and again (the search spiders were crawling only "new pages").

Tuning the server (apache / mysql) should be the last step. Of course it's necerssary and important, but it shouldn't be the first step.

-> Gnite: If you want to exchange experiences, my email is emil.webber@gmx.at

Kind regards,
Emil.

Best wishes,
Georg.

--
http://www.schicksal.com Horoskop website which uses eZ Publish since 2004

Monday 29 March 2004 9:12:42 am

Zinistry,

In our experience it is unlikely to be the database that causes the bottleneck. We normally find the database uses about 3% of the CPU time.
It seems to me that part of the problem is that when your editors publish the system needs to regenerate the cache every time. This means that the accelerators will be useless as they will be constantly recacheing cahce pages.

You could try turning off the cache within eZ and see if this helps with things. That way you will not be constantly deleting and recreating cache files. This would only be a stop gap solution, hopefully 3.4 will solve some of your issues as they are reducing the memory payload of eZ.

oh, just a thought, when was the last time this box was patched? I know that in December Redhat patched their 2.4 kernel with some 2.6 kernel features. This might speed things up. I know it did for us.

Hope this helps

--tony

Tony Wood : twitter.com/tonywood
Vision with Technology
Experts in eZ Publish consulting & development

Power to the Editor!

Free eZ Training : http://www.VisionWT.com/training
eZ Future Podcast : http://www.VisionWT.com/eZ-Future

Tuesday 20 April 2004 4:34:13 am

Hi Zinistry,

eZ publish deletes all cache blocks when any content is published, as you probably know. This would cause the slowdown when you have many editors working. One way to improve performance in this situation is to use the "ignore_content_expiry" parameter in your cache blocks.

http://ez.no/ez_publish/documentation/incoming/cache_block_optimization
(see "Expiry")

The more complex your templates are (pagelayout, menus etc.) the more this will help. When the content of the cache blocks actually needs to change, you must clear the cache blocks "manually" (using "Setup" -> "Cache" in admin). But you probably don't need to do this often, only when the menu content and similar parts should change.

Tuesday 20 April 2004 7:01:59 am

Hi,

-> clearcache.sh / blank pages

I think, that problem occurs because ez don't use the file-locking at the cache-section, as I mentioned somewhere ago (bug reports / forums).

I've the same problem if I delete the whole cache and many visits are at the same time on my sites. So, if the cache is empty and two instances tries to write / append the same cache-file, some weird results can happen.

So, the only way around is:
Delete the cache more then once and pray that not much visitors are coming at the moment.

Kind regards,
Emil.

Best wishes,
Georg.

--
http://www.schicksal.com Horoskop website which uses eZ Publish since 2004