Forums / General / cronjob resource hog
Andre Felipe Machado
Monday 31 October 2005 2:41:37 am
Hello, The runcronjob is depleting resources at my machine. It exahusts ram and maximum open files limits. The site syndicates some rss external feeds and then links are included. So, the site "grows". At the next cronjob, the additional links and pages are checked and memory is filled, maximum open files are reached, site crashes with Ezpublish fatal error or error 403, forbidden access(!), database acess fails (unacessible mysql table files). Is there a way to optimize the cronjobs? Today, its code imposes a practical limit to the site size, number of pages and number of links. What can I do? Regards.Andre Felipe
--- A Debian user never dies. Issues a last command: shutdown -h now http://www.techforce.com.br
Wednesday 02 November 2005 7:30:20 am
Hello Found the culprit and an ugly workaround. The cronjob does not scale to bigger sites. It starts linkchecking and this process is slow, because each link must connect or time out. At my site, each instance of linkcheck use almost 6 hours to complete. As each instance is already a resource hog on its own, 6 instances running simultaneausly exhausts system resources. I had to reconfigure cronjob from hourly to at each 6 hours... The rss export import are now very bad. Workflow.... A more reasonable solution could be using independent link check cron job from the other tasks. The link check could run once a day. Other solution could be check many links in parallel mode. Regards.Andre Felipe
Mark Marsiglio
Wednesday 02 November 2005 10:10:12 am
I removed linkcheck from the list of jobs run on a regular basis in cronjobs.ini to solve the same problem.
I am in the process of creating a separate cron job to run only link check once a week. Have not got it done yet though...
http://www.thinkcreative.com Turning Ideas Into Strategic Solutions
Kåre Køhler Høvik
Thursday 03 November 2005 12:15:33 am
Checking all links will always be a heavy operation for the system. It's not recommended to run it as part of the normal cronjobs. Use the cronjob-part to run the linkcheck as separate process : http://ez.no/products/ez_publish_cms/documentation/installation/the_cronjob_script
Kåre Høvik