large datasets

Next topic

Author	Message
Dennis Jessen	Friday 09 March 2007 6:31:32 am Hello i have the oppertunity to use eZ publish to develop an webshop, the experience I have with eZ is from rather 'small content' sites. at some point in time the webshop will contain around 2.5 million objects/products with possibly as much as 30 attributes. This is just the products. On average i would think that 5 of these attributes would be empty / non-existant. i havent had any experinces yet with datasets of this size, and wouldn't mind some input from those that have. some questions i have, among many! Should I use the contentobject model to store products, giving many rows of contentObject_attributes ( 252.5mil ) 2.5mil contentObjects. Would I be better of extending the persistentObject, and if I do, what could the possible drawbacks be ( caching etc ? ). To start with we have a dedicated server with mysql and one dedicated webserver, and possibly we would use the ezDBcachehandler to prepare for more webservers. would this be overkill and are there any issues like image fetching etc. Obviously all this is something I just have to test to gain my own experinces, but nonetheless any input would be much appreciated. grettings Dennis

Author

Message

Friday 09 March 2007 6:31:32 am

Hello

i have the oppertunity to use eZ publish to develop an webshop, the experience I have with eZ is from rather 'small content' sites.

at some point in time the webshop will contain around 2.5 million objects/products with possibly as much as 30 attributes. This is just the products.
On average i would think that 5 of these attributes would be empty / non-existant.

i havent had any experinces yet with datasets of this size, and wouldn't mind some input from those that have.

some questions i have, among many!

Should I use the contentobject model to store products, giving many rows of contentObject_attributes ( 25*2.5mil ) * 2.5mil contentObjects.

Would I be better of extending the persistentObject, and if I do, what could the possible drawbacks be ( caching etc ? ).

To start with we have a dedicated server with mysql and one dedicated webserver, and possibly we would use the ezDBcachehandler to prepare for more webservers.
would this be overkill and are there any issues like image fetching etc.

Obviously all this is something I just have to test to gain my own experinces, but nonetheless any input would be much appreciated.

grettings

Dennis

eZ debug

Timing:	Jan 19 2025 03:30:54
Script start
Timing:	Jan 19 2025 03:30:54
Module start 'layout'
Timing:	Jan 19 2025 03:30:54
Module start 'content'
Timing:	Jan 19 2025 03:30:56
Module end 'content'
Timing:	Jan 19 2025 03:30:56
Script end

Main resources:

Total runtime	1.8982 sec
Peak memory usage	4,096.0000 KB
Database Queries	46

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0058	589.0313	152.6094
Module start 'layout'	0.0058	0.0035	741.6406	39.4297
Module start 'content'	0.0094	1.8871	781.0703	402.6328
Module end 'content'	1.8965	0.0016	1,183.7031	8.1953
Script end	1.8981		1,191.8984

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0037	0.1932	16	0.0002
Check MTime	0.0013	0.0702	16	0.0001
Mysql Total
Database connection	0.0007	0.0348	1	0.0007
Mysqli_queries	1.8535	97.6476	46	0.0403
Looping result	0.0005	0.0260	44	0.0000
Template Total	1.8590	97.9	2	0.9295
Template load	0.0023	0.1228	2	0.0012
Template processing	1.8567	97.8135	2	0.9283
Template load and register function	0.0003	0.0157	1	0.0003
states
state_id_array	0.0018	0.0944	1	0.0018
state_identifier_array	0.0011	0.0579	2	0.0005
Override
Cache load	0.0019	0.0981	16	0.0001
Sytem overhead
Fetch class attribute can translate value	0.0008	0.0447	1	0.0008
Fetch class attribute name	0.0010	0.0543	1	0.0010
XML
Image XML parsing	0.0005	0.0241	1	0.0005
class_abstraction
Instantiating content class attribute	0.0000	0.0003	1	0.0000
General
dbfile	0.0012	0.0607	10	0.0001
String conversion	0.0000	0.0005	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	full/forum_topic.tpl	extension/sevenx/design/simple/override/templates/full/forum_topic.tpl
1	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
3	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
2	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 8 Number of unique templates used: 5

Time used to render debug report: 0.0003 secs