Forums / General / Huge database

Huge database

Author Message

Christian Johansen

Saturday 11 July 2009 4:17:54 am

A database dump for my site, which is a blog with a few hundred entries, is coming upwards og 900MB. The size seems excessive, and I was wondering if there are any cleanup utilities to run on it?

I've cleaned out the trash, but that don't seem to help. Also, the database schema is fairly complex, so guessing which table is holding heaps of unused data is not simple. Any hints on how to slim my database?

Heath

Saturday 11 July 2009 4:34:38 am

Sadly the size is quite common.

Save your self some headache and simply adapt.

I've seen much larger sizes in GB of exports of active clean eZ Publish installation databases.

I suggest reading up on the php shell script, flatten.php and cleanup.php

These should help you reduce some database size.

<i>[0] http://ez.no/doc/ez_publish/upgrading/the_system_upgrade_scripts</i>

Cheers,
Heaht

Brookins Consulting | http://brookinsconsulting.com/
Certified | http://auth.ez.no/certification/verify/380350
Solutions | http://projects.ez.no/users/community/brookins_consulting
eZpedia community documentation project | http://ezpedia.org

Christian Johansen

Saturday 11 July 2009 4:54:45 am

Thanks, I'll read up on them. 900MBs of database for a simple blog is insane. The size is not really a problem in terms of storage space, but it is causing me headaches in terms of backup. Takes too long to backup a copy of the database.

If I can't considerably reduce the db size I fear I'll migrate the site to another solution :(

Thanks for your help!

kracker (the)

Saturday 11 July 2009 4:58:31 am

Migrate to wordpress Christian which can be very small.

You will never get eZ to fit the size
of db I know your looking for.

Also your prolly using the content tree
to store comments for your blog
which is also a huge performance
problem just waiting to be understood.

Cheers,
<i>//kracker</i>

Member since: 2001.07.13 || http://ezpedia.se7enx.com/

Karnichi Mohamed

Saturday 11 July 2009 5:49:49 am

Hi Christian,

As eZ Publish stores sessions informations in the database, you have to enable session.gc_probability so the expired sessions are purged:

so in your php.ini check if your have these parameters:

session.gc_probability = 1
session.gc_divisor = 100

If you look at your database size you will find that the ezsession table is huge !

For now you can truncate the ezsession table :

mysql >truncate table ezsession

or use the cleanup script

php update/common/scripts/cleanup.php session
php update/common/scripts/cleanup.php expired_session

Mo

http://www.amiralweb.com Certified eZPublish Expert

Max Keil

Saturday 11 July 2009 2:00:31 pm

Hi Christian,

can you tell us which of the tables causes such a large size of the dump? If it is ezsession then you should run clean up scripts like my previous writers told you - if it is ezcontentobject_attribute you may check the number of versions you store and the number of attributes in this classes.

Mit freundlichen Grüßen
Best regards

Max Keil

____________________________________________________________
eZ Publish Gold Partner - http://www.all2e.com
http://ez.no/partners/worldwide_partners/all2e_gmbh

Christian Johansen

Saturday 18 July 2009 1:07:55 pm

I cleaned up the sessions, and wow. I mean, WOW! Incredible:

-rw-r--r-- 1 christian users 904M 2009-07-18 21:58 20090718-1.sql
-rw-r--r-- 1 christian users  25M 2009-07-18 22:02 20090718-2.sql

That's pre and post session cleanup. Problem solved :) Thanks alot all!

Kracker: I'd be interested to know how you recon storing comments in the content tree is a problem? In what other way can I really store them? If you're thinking about the tree menu, I obviously don't show them there. I have quite a bit of content, and haven't had any real performance trouble yet...

kracker (the)

Saturday 18 July 2009 1:15:42 pm

Wow you sure proved me wrong. That's quite small.

Checkout, http://projects.ez.no/cscomment

Cheers,
//kracker

Member since: 2001.07.13 || http://ezpedia.se7enx.com/

Christian Johansen

Saturday 18 July 2009 2:10:29 pm

Yeah, I know. At first I was certain that my data had been corrupted, but it's still sane. Cool!

Thanks for the link, I'll check it out!

kracker (the)

Sunday 19 July 2009 11:40:19 am

I asked Heath to add a node about this topic on eZpedia,
<i>http://ezpedia.org/en/ez/solution_how_to_shrink_or_reduce_the_site_of_ez_publish_database_backup_dump_files</i>

Cheers,<i>
//kracker</i>

Member since: 2001.07.13 || http://ezpedia.se7enx.com/

eZ debug

Timing: Jan 18 2025 04:07:50
Script start
Timing: Jan 18 2025 04:07:50
Module start 'content'
Timing: Jan 18 2025 04:07:51
Module end 'content'
Timing: Jan 18 2025 04:07:51
Script end

Main resources:

Total runtime0.8510 sec
Peak memory usage4,096.0000 KB
Database Queries221

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0061 587.5313180.8594
Module start 'content' 0.00610.6862 768.3906760.4766
Module end 'content' 0.69230.1586 1,528.8672344.4688
Script end 0.8509  1,873.3359 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00410.4875210.0002
Check MTime0.00150.1748210.0001
Mysql Total
Database connection0.00080.089310.0008
Mysqli_queries0.744387.46762210.0034
Looping result0.00270.32142190.0000
Template Total0.822696.720.4113
Template load0.00210.245120.0010
Template processing0.820596.425320.4103
Template load and register function0.00020.027510.0002
states
state_id_array0.00170.196110.0017
state_identifier_array0.00100.116920.0005
Override
Cache load0.00190.2256660.0000
Sytem overhead
Fetch class attribute can translate value0.00150.176360.0002
Fetch class attribute name0.00170.1976140.0001
XML
Image XML parsing0.00280.326160.0005
class_abstraction
Instantiating content class attribute0.00000.0041160.0000
General
dbfile0.00360.4178480.0001
String conversion0.00000.000830.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
10content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
20content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
6content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
7content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
4content/datatype/view/ezxmltags/literal.tpl<No override>extension/community/design/standard/templates/content/datatype/view/ezxmltags/literal.tplEdit templateOverride template
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 49
 Number of unique templates used: 7

Time used to render debug report: 0.0002 secs