Forums / Developer / 1000000 (one milion) versioned multi-located objects

1000000 (one milion) versioned multi-located objects

Author Message

Piotrek Karaś

Thursday 23 October 2008 7:05:56 am

Hi,

Has anyone got and would be willing to share (on public or privately) experiences of using eZ Publish for managing around one milion (1000000) of content objects with several object versions and on average 20-50 object locations?

I'm interested in any aspects of such implementation: hardware, practical problems (if any), and so on.

Or maybe there are some known case studies available for such projects?

Much appreciated,
Piotrek

--
Company: mediaSELF Sp. z o.o., http://www.mediaself.pl
eZ references: http://ez.no/partners/worldwide_partners/mediaself
eZ certified developer: http://ez.no/certification/verify/272585
eZ blog: http://ez.ryba.eu

Ivo Lukac

Thursday 23 October 2008 8:45:49 am

Hello Piotrek,

Did some testing with 300000 objects with several versions each placed randomly in 20 folders. Hardware was nothing special, just development machine. Main purpose was to see how would such implementation work.

There was one big issue which was slowing done publishing process (publish time was increasing as total number of objects). The problem was in view cache clearing keyword related objects because I had keywords in all objects (eZContentCacheManager::appendKeywordNodeIDs() is very slow).

There is no setting option to disable this so kernel hack is the saver :)

http://www.linkedin.com/in/ivolukac
http://www.netgen.hr/eng/blog
http://twitter.com/ilukac

André R.

Thursday 23 October 2008 9:40:58 am

>There is no setting option to disable this so kernel hack is the saver :)

No but there is a patch that are looking for feedback:
http://issues.ez.no/IssueView.php?Id=13429&activeItem=4

Adds setting to limit keyword relations to clear view cache for, and setting to let you have total control over this with viewcache.ini rules(instead of just adding rules to the hardcoded defaults).

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Ivo Lukac

Thursday 23 October 2008 10:16:12 am

Great :)

When will it be in trunk and not like patch?

http://www.linkedin.com/in/ivolukac
http://www.netgen.hr/eng/blog
http://twitter.com/ilukac

André R.

Thursday 23 October 2008 11:01:04 am

Where there is some feedback on it =)
For instance should the keyword limit be set by default to for example 500? or 100? or..

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Ivo Lukac

Thursday 23 October 2008 11:42:00 pm

I think the most important is that you can disable this because you don't need it always. For example if you have keywords and show them in full view linking to content/keyword/something you don't need this. You need this only when you have listed keyword related objects.

Right?

http://www.linkedin.com/in/ivolukac
http://www.netgen.hr/eng/blog
http://twitter.com/ilukac

André R.

Friday 24 October 2008 12:47:58 am

yes. Witch reminds me that there should be possible to control this by class :)

eZ Online Editor 5: http://projects.ez.no/ezoe || eZJSCore (Ajax): http://projects.ez.no/ezjscore || eZ Publish EE http://ez.no/eZPublish/eZ-Publish-Enterprise-Subscription
@: http://twitter.com/andrerom

Piotrek Karaś

Tuesday 18 November 2008 8:45:06 pm

Thanks for your feedback, Ivo, and sorry for the delay.

I'm justing finishing a script for running similar tests in a flexible manner, I'll share that as soon as it's ready. Unfortunately, I will also need to buy a PC for running those tests to reach numbers we're talking about before christmas ;)

I will use a real-life thesaurus to feed objects with content, and will definitely include XMLBlock and Keywords datatypes testing, as well.

I will get back on this issue.

--
Company: mediaSELF Sp. z o.o., http://www.mediaself.pl
eZ references: http://ez.no/partners/worldwide_partners/mediaself
eZ certified developer: http://ez.no/certification/verify/272585
eZ blog: http://ez.ryba.eu

eZ debug

Timing: Jan 18 2025 11:15:12
Script start
Timing: Jan 18 2025 11:15:12
Module start 'content'
Timing: Jan 18 2025 11:15:13
Module end 'content'
Timing: Jan 18 2025 11:15:14
Script end

Main resources:

Total runtime1.9500 sec
Peak memory usage4,096.0000 KB
Database Queries211

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0087 587.8594180.8125
Module start 'content' 0.00871.7620 768.6719656.6719
Module end 'content' 1.77080.1791 1,425.3438345.0547
Script end 1.9499  1,770.3984 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00440.2257210.0002
Check MTime0.00160.0829210.0001
Mysql Total
Database connection0.00190.095010.0019
Mysqli_queries1.835794.13932110.0087
Looping result0.00290.14812090.0000
Template Total1.905997.720.9529
Template load0.00230.117020.0011
Template processing1.903697.620220.9518
Template load and register function0.00010.005110.0001
states
state_id_array0.00080.038510.0008
state_identifier_array0.00940.482120.0047
Override
Cache load0.00210.1054430.0000
Sytem overhead
Fetch class attribute can translate value0.00170.084840.0004
Fetch class attribute name0.00110.0548110.0001
XML
Image XML parsing0.00200.100740.0005
class_abstraction
Instantiating content class attribute0.00000.0021160.0000
General
dbfile0.00340.1764420.0001
String conversion0.00000.000330.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
8content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
8content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
10content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
3content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 31
 Number of unique templates used: 6

Time used to render debug report: 0.0002 secs