Forums / Setup & design / large datasets

large datasets

Author Message

Dennis Jessen

Friday 09 March 2007 6:31:32 am

Hello

i have the oppertunity to use eZ publish to develop an webshop, the experience I have with eZ is from rather 'small content' sites.

at some point in time the webshop will contain around 2.5 million objects/products with possibly as much as 30 attributes. This is just the products.
On average i would think that 5 of these attributes would be empty / non-existant.

i havent had any experinces yet with datasets of this size, and wouldn't mind some input from those that have.

some questions i have, among many!

Should I use the contentobject model to store products, giving many rows of contentObject_attributes ( 25*2.5mil ) * 2.5mil contentObjects.

Would I be better of extending the persistentObject, and if I do, what could the possible drawbacks be ( caching etc ? ).

To start with we have a dedicated server with mysql and one dedicated webserver, and possibly we would use the ezDBcachehandler to prepare for more webservers.
would this be overkill and are there any issues like image fetching etc.

Obviously all this is something I just have to test to gain my own experinces, but nonetheless any input would be much appreciated.

grettings

Dennis

eZ debug

Timing: Jan 31 2025 04:27:15
Script start
Timing: Jan 31 2025 04:27:15
Module start 'content'
Timing: Jan 31 2025 04:27:15
Module end 'content'
Timing: Jan 31 2025 04:27:15
Script end

Main resources:

Total runtime0.4639 sec
Peak memory usage2,048.0000 KB
Database Queries141

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0092 587.8203180.8516
Module start 'content' 0.00920.0235 768.671989.7109
Module end 'content' 0.03260.4312 858.3828519.2266
Script end 0.4638  1,377.6094 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00481.0384200.0002
Check MTime0.00160.3399200.0001
Mysql Total
Database connection0.00110.233210.0011
Mysqli_queries0.360477.69351410.0026
Looping result0.00160.34441390.0000
Template Total0.430492.810.4304
Template load0.00120.248110.0012
Template processing0.429392.535410.4293
Override
Cache load0.00070.156110.0007
Sytem overhead
Fetch class attribute can translate value0.00100.222110.0010
XML
Image XML parsing0.00030.055410.0003
General
dbfile0.04399.4673200.0022
String conversion0.00000.001330.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 1
 Number of unique templates used: 1

Time used to render debug report: 0.0002 secs