Forums / General / cronjob resource hog

cronjob resource hog

« 
Previous topic
|
General
|
Next topic
 »
Author Message

Andre Felipe Machado

Monday 31 October 2005 2:41:37 am

Hello,
The runcronjob is depleting resources at my machine.
It exahusts ram and maximum open files limits.
The site syndicates some rss external feeds and then links are included. So, the site "grows".
At the next cronjob, the additional links and pages are checked and memory is filled, maximum open files are reached, site crashes with Ezpublish fatal error or error 403, forbidden access(!), database acess fails (unacessible mysql table files).
Is there a way to optimize the cronjobs?
Today, its code imposes a practical limit to the site size, number of pages and number of links.
What can I do?
Regards.
Andre Felipe

---
A Debian user never dies. Issues a last command:
shutdown -h now

http://www.techforce.com.br

Andre Felipe Machado

Wednesday 02 November 2005 7:30:20 am

Hello
Found the culprit and an ugly workaround.
The cronjob does not scale to bigger sites.
It starts linkchecking and this process is slow, because each link must connect or time out.
At my site, each instance of linkcheck use almost 6 hours to complete.
As each instance is already a resource hog on its own, 6 instances running simultaneausly exhausts system resources.
I had to reconfigure cronjob from hourly to at each 6 hours...
The rss export import are now very bad. Workflow....
A more reasonable solution could be using independent link check cron job from the other tasks.
The link check could run once a day.
Other solution could be check many links in parallel mode.
Regards.
Andre Felipe

---
A Debian user never dies. Issues a last command:
shutdown -h now

http://www.techforce.com.br

Mark Marsiglio

Wednesday 02 November 2005 10:10:12 am

I removed linkcheck from the list of jobs run on a regular basis in cronjobs.ini to solve the same problem.

I am in the process of creating a separate cron job to run only link check once a week. Have not got it done yet though...

http://www.thinkcreative.com
Turning Ideas Into Strategic Solutions

Kåre Køhler Høvik

Thursday 03 November 2005 12:15:33 am

Checking all links will always be a heavy operation for the system. It's not recommended to run it as part of the normal cronjobs. Use the cronjob-part to run the linkcheck as separate process : http://ez.no/products/ez_publish_cms/documentation/installation/the_cronjob_script

Kåre Høvik

eZ debug

Timing: Jan 18 2025 11:31:17
Script start
Timing: Jan 18 2025 11:31:17
Module start 'content'
Timing: Jan 18 2025 11:31:18
Module end 'content'
Timing: Jan 18 2025 11:31:18
Script end

Main resources:

Total runtime1.0792 sec
Peak memory usage4,096.0000 KB
Database Queries199

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0097 587.6094180.8516
Module start 'content' 0.00970.8923 768.4609601.2969
Module end 'content' 0.90200.1770 1,369.7578337.3906
Script end 1.0791  1,707.1484 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00450.4164210.0002
Check MTime0.00160.1448210.0001
Mysql Total
Database connection0.00220.205410.0022
Mysqli_queries0.982791.05731990.0049
Looping result0.00460.42791970.0000
Template Total1.034195.820.5170
Template load0.00310.288920.0016
Template processing1.030995.532120.5155
Template load and register function0.00020.015010.0002
states
state_id_array0.00190.178410.0019
state_identifier_array0.00100.096220.0005
Override
Cache load0.00200.1842360.0001
Sytem overhead
Fetch class attribute can translate value0.00150.138640.0004
Fetch class attribute name0.00180.170870.0003
XML
Image XML parsing0.00210.193640.0005
class_abstraction
Instantiating content class attribute0.00000.001580.0000
General
dbfile0.00510.4699370.0001
String conversion0.00000.001030.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
4content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
4content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
2content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
4content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 16
 Number of unique templates used: 6

Time used to render debug report: 0.0002 secs