Dedicated server crashed with few sites

Author Message

John Smith

Monday 04 January 2010 8:39:48 am

Hi guys,

Hope someone can help me. We got few sites (around 10) running on a dedicated server with the following specification and it crashed on the day when they were needed. At that time there were around 800 users and around 22,195 hits.

Intel(R) Core(TM)2 Duo CPU E4600 @ 2.40GHz

Speed: 2394.103

Cache Size: 2048 KB

Total Memory 2064780kB

Total Swap 497944 kB

Is there anyway to check whether it is something wrong with ezPublish or the server. I have already checked the debug for all sites there are no warnings or errors? I am using ezPublish 4.0.1, Please help.

Pål Messenlien

Monday 04 January 2010 1:31:37 pm

I have seen the same happend with just 1 eZ site. We released a portal for registrations. I noticed something was wrong the minute i released the site. As soon the users started logging in and registering the server went nuts. It all crashed within an hour. We fixed this with some code changes on cache blocks etc. But the main fix was adding a php accelerator and tuning both mysql and php. After a couple hours we had the site back up and running again. I would first look into the php accelerator for your server. We use APC. Also check if php got enough memory, and check for tuning on mysql. If possible move mysql to another server, it will take some of the load off. Logged in users doing lots of stuff in eZ seems to be very power hungry. There should be lots of posts here in the forums about tuning php and servers.

These are what i found that might be usefull:

Other things you should consider:

  • Upgrading eZ to a later version. 4.2 seems to run allot easier than 4.0.1 was. (just upgraded a large site)
  • Adding Varnish to the sites

Hope some of this help.

--------------------------------------------
Høgskolen i Lillehammer
Lillehammer University College
http://www.hil.no
--------------------------------------------
Messenlien IT
http://messenlien.com
-------------------------------------------

John Smith

Tuesday 05 January 2010 9:44:44 am

Thanks for you kind feedback.

Tuning MySQL means allocation of memory towards database. I did try few settings (max connections, key buffer, table cache) as discussed in the article this afternoon, but now apache web server started hanging.

Still stuck quite frustrating...I know the setting would be different for different server specifications but is there any general set of check points which needs to be checked before releasing the site to the public so that it would not crash the server. Any help would be appreciated.

Jérôme Vieilledent

Wednesday 06 January 2010 2:13:02 pm

Hi John

What about Apache max_clients ? Try to tune it (here's a tiny tutorial ;)).

Can you be more precise on your server configuration please ? PHP/Apache/MySQL/eZ Publish versions, PHP configuration (memory_size, APC config if you have it...). Is Apache in prefork mode ? Is PHP compiled as a module ? Do you run your server on Linux (which distro) ?

Also, you should try to install monitoring tools such as Cacti or Nagios.

You can monitor your eZ Publish websites with jvMonitoring (simple, you can develop new handlers) or eZSNMPd.

First things to check for performance :

  • Cache configuration (is view cache activated, did you disable INI File mtime check in config.php, are your cache blocks in pagelayout efficient, ...)
  • Opcode cache such as APC (highly recommended, with 128MB at least)
  • A reverse proxy cache eventually, such as Varnish

Hope this helps :)

John Smith

Friday 08 January 2010 12:51:47 am

Cheers mate,

I will revert back with all the question you asked for. By the way checking few first things as suggested by you:

View cached is activated with the following settings. Please have a look.

[ContentSettings]
ViewCaching=enabled
PreViewCache=enabled

[TemplateSettings]
TemplateCache=enabled
TemplateCompile=enabled
TemplateOptimization=enabled

I did not get the point regarding INI file mtime check in config.php, Please explain.

Cache blocks seems fine in pagelayout.tpl. Is there any tutorial where I can get full information about Cache Blocks.

Cheers

Jérôme Vieilledent

Friday 08 January 2010 1:35:30 am

Hi John

OK everything seems fine with your cache config. Also check if DevelopmentMode is disabled ([TemplateSettings]).

About config.php stuff, it's from eZ Publish 4.1. You can consider upgrading to 4.2 (huge performance improvements) ;)

John Smith

Friday 08 January 2010 9:21:16 am

cheers,

It would be favour if you can suggest me how to cache the following.

I have written some code to display the latest events and random images from gallery on home page. I have to use the following:

{set-block scope=root variable=cache_ttl}0{/set-block}

The reason I am using set-block is to exclude it from cache. If I do not write it then the system would not update events when added from the administration interface. But it also increases the number of mysql queries and load on the server.

What is the best way to do it?

Gaetano Giunta

Friday 08 January 2010 9:44:20 am

A couple of hints:

- mysql queries might be killing your app. You can find out how many queries are executed per-page by enabling the debug output (it can be done without disabling caches) and looking at the bottom of the page for the table line that reads "Mysql_queries" (or Mysqli_queries). Anything above 100 is plain suicide. Below 10, you'r gold.

- disabling view cache on hp is a bad idea, really.

You can set it to eg 10 minutes if you want latest news appearing, just not in real time:

<span>{</span><a href="http://doc.ez.no/set-block" mce_href="http://doc.ez.no/set-block" target="ez_no_documentation"><span>set-block</span></a> <span>scope</span><span>=</span>root <span>variable</span><span>=</span>cache_ttl<span>}36</span>0<span>{</span><span>/</span><a href="http://doc.ez.no/set-block" mce_href="http://doc.ez.no/set-block" target="ez_no_documentation"><span>set-block</span></a><span>}</span>

You can use a cache block where you put in the "keys" param a random value calculated between eg 1 and 10 if what you want is some random pictures - you will have the cache block executed about 10 times before it is fully warmed, and successive hits will not execute it anymore

If your news do not update very often, even better than a periodic expiry is one based on actual linking of content editing action with cached expiration. look up the docs about the "smart view cache" for that

- a lot of developers always fetch the current node in the pagelayout, because it is not there and they need just a bit of info from it. Do not do it, use the persistent_variable instead (look it up in the forums/docs)

- make sure your cache blocks are properly tuned: if they do not depend on content expiration, use "ignore_content_expiry", if they depend on expiration on a particular node, set it in "subtree_expiry", and do not forget plain "expiry" timeout (http://ez.no/doc/ez_publish/technical_manual/4_x/reference/template_functions/miscellaneous/cache_block)

Principal Consultant International Business
Member of the Community Project Board

John Smith

Friday 15 January 2010 1:15:28 am

Hi Gaetano,

Thanks for your suggestions, I tried all the points you suggested and it has reduce the load dramatically.

One thing which needs to be confirmed with you. I did link content editing action with cached expiration to display latest events on the home page. It worked every well with smart view cache enabled, which means if somebody add/edit event - the home page cache get cleared and updates the list.

If the content editing action is not triggered means nobody add or edit any event, then it does not update the list of events automatically on homepage which has already held yesterday or before as I am display all latest event today onwards. Is there any trick to do that as well.

Cheers,

Gaetano Giunta

Friday 15 January 2010 1:42:42 am

Sorry, I do not understand fully your request.

If you want the homepage expired every day event if no editing action has been happening, you can set up a cronjob that expires its cache. The php call to expire the cache of an object is:

 eZContentCacheManager::clearContentCache( $contentObject->attribute('id') );

Principal Consultant International Business
Member of the Community Project Board

John Smith

Friday 15 January 2010 2:03:25 am

Yes, I want the homepage cache to be expired everyday in the event if no editing action has been happening. Is it possible to explain in more detail how to implement and setup as cronjob.

Much appreciated.

Cheers,

John Smith

Tuesday 19 January 2010 7:06:58 am

Anyone ??

Nicolas Pastorino

Monday 01 February 2010 8:39:44 am

"

Anyone ??

"

Hi John,

Late answer on this, hopefully helpful anyways. Check these concise resources in order to understand how to setup a custom cronjob:

  • http://ez.no/doc/ez_publish/technical_manual/4_x/features/cronjobs
  • http://ez.no/doc/ez_publish/technical_manual/4_x/features/cronjobs/the_cronjob_scripts
  • http://ez.no/doc/ez_publish/technical_manual/4_x/features/cronjobs/configuring_cronjobs
  • http://ez.no/doc/ez_publish/technical_manual/4_x/features/cronjobs/running_cronjobs

The code contained in the cronjob you apparently want to setup would be the line Gaetano gave :

eZContentCacheManager::clearContentCache( $homePageObjectId );

where $homePageObjectId can be fetched from an INI file.

How is your website going now ?
Do you have uptodate stats (pv, hits, unique visitors, etc) ?
Cheers,

--
Nicolas Pastorino
Director Community - eZ
Member of the Community Project Board

eZ Publish Community on twitter: http://twitter.com/ezcommunity

t : http://twitter.com/jeanvoye
G+ : http://plus.tl/jeanvoye

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.

eZ debug

Timing: Jan 18 2025 10:35:07
Script start
Timing: Jan 18 2025 10:35:07
Module start 'layout'
Timing: Jan 18 2025 10:35:07
Module start 'content'
Timing: Jan 18 2025 10:35:08
Module end 'content'
Timing: Jan 18 2025 10:35:08
Script end

Main resources:

Total runtime0.9700 sec
Peak memory usage4,096.0000 KB
Database Queries95

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0057 588.0313152.6406
Module start 'layout' 0.00570.0025 740.671939.4766
Module start 'content' 0.00820.9597 780.1484818.0703
Module end 'content' 0.96790.0021 1,598.218832.1406
Script end 0.9700  1,630.3594 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00350.3647160.0002
Check MTime0.00130.1325160.0001
Mysql Total
Database connection0.00080.077510.0008
Mysqli_queries0.849687.5798950.0089
Looping result0.00120.1210930.0000
Template Total0.936996.620.4685
Template load0.00200.201020.0010
Template processing0.934996.380920.4675
Template load and register function0.00020.024310.0002
states
state_id_array0.00100.099510.0010
state_identifier_array0.00080.087320.0004
Override
Cache load0.00200.20201250.0000
Sytem overhead
Fetch class attribute can translate value0.00090.088750.0002
Fetch class attribute name0.00170.1728160.0001
XML
Image XML parsing0.00200.205150.0004
class_abstraction
Instantiating content class attribute0.00000.0041180.0000
General
dbfile0.00440.4516290.0002
String conversion0.00000.001040.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
13content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
28content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
9content/datatype/view/ezxmltags/link.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/link.tplEdit templateOverride template
9content/datatype/view/ezxmltags/li.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/li.tplEdit templateOverride template
4content/datatype/view/ezxmltags/ul.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/ul.tplEdit templateOverride template
5content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
2content/datatype/view/ezxmltags/emphasize.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/emphasize.tplEdit templateOverride template
3content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
4content/datatype/view/ezxmltags/literal.tpl<No override>extension/community/design/standard/templates/content/datatype/view/ezxmltags/literal.tplEdit templateOverride template
1content/datatype/view/ezxmltags/quote.tpldatatype/ezxmltext/quote.tplextension/ezwebin/design/ezwebin/override/templates/datatype/ezxmltext/quote.tplEdit templateOverride template
1print_pagelayout.tpl<No override>extension/community/design/community/templates/print_pagelayout.tplEdit templateOverride template
 Number of times templates used: 80
 Number of unique templates used: 12

Time used to render debug report: 0.0001 secs