Problem with Import of Text to UTF-Database

Next topic

Author	Message
Rose Futchko	Monday 31 October 2005 7:19:27 pm Recently, we imported over 3,000 text articles to the eZpublish database using the provided import routine on release 3.70. It seemed that this conversion function normalized the XML to eZ publish compliant code and improperly stripped some special characters, converting them to question marks, and did not convert others. For example, 1) The special character are encoded properly as UTF-8 in the import XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94'). 2) In the samples, these characters are incorrectly coded in the MySQL database simply as '3F' (the question mark). We had thought they might be in Latin1, Unicode, or encoded as HTML character entities, but they are not. 3) The eZ Online Editor allows you to use the special character entry button to enter special characters. You can also enter special character using the Windows "Character Map." These characters are then stored in the database as proper UTF-8, just like in the XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94'). Has anyone run into this problem and if they have, do you have a work around solution that you can share for updating these special characters in the mysql database? Thank you.
Xavier Dutoit	Monday 31 October 2005 11:59:59 pm Hi, What is the routine you've used ? Don't get it. Could you detail that steps ? As for the chars, I always have problem with win-1251 chars when I switch to utf-8. I wrote a wee dirty script to convert the win chars (the quotes, dash and the al), to normal chars. I dump the database, apply it and reimport the modified dump. However, if the chars are already screwed in the mysql database, that's probably not going to work. X+ http://www.sydesy.com

Author

Message

Monday 31 October 2005 7:19:27 pm

Recently, we imported over 3,000 text articles to the eZpublish database using the provided import routine on release 3.70. It seemed that this conversion function normalized the XML to eZ publish compliant code and improperly stripped some special characters, converting them to question marks, and did not convert others.

For example,

1) The special character are encoded properly as UTF-8 in the import XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94').

2) In the samples, these characters are incorrectly coded in the MySQL database simply as '3F' (the question mark). We had thought they might be in Latin1, Unicode, or encoded as HTML character entities, but they are not.

3) The eZ Online Editor allows you to use the special character entry button to enter special characters. You can also enter special character using the Windows "Character Map." These characters are then stored in the database as proper UTF-8, just like in the XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94').

Has anyone run into this problem and if they have, do you have a work around solution that you can share for updating these special characters in the mysql database?

Thank you.

Xavier Dutoit

Monday 31 October 2005 11:59:59 pm

Hi,

What is the routine you've used ? Don't get it.
Could you detail that steps ?

As for the chars, I always have problem with win-1251 chars when I switch to utf-8.

I wrote a wee dirty script to convert the win chars (the quotes, dash and the al), to normal chars. I dump the database, apply it and reimport the modified dump. However, if the chars are already screwed in the mysql database, that's probably not going to work.

X+

http://www.sydesy.com

eZ debug

Timing:	Jan 18 2025 15:59:56
Script start
Timing:	Jan 18 2025 15:59:56
Module start 'layout'
Timing:	Jan 18 2025 15:59:56
Module start 'content'
Timing:	Jan 18 2025 15:59:57
Module end 'content'
Timing:	Jan 18 2025 15:59:57
Script end

Main resources:

Total runtime	0.8087 sec
Peak memory usage	4,096.0000 KB
Database Queries	54

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0063	589.1641	152.6406
Module start 'layout'	0.0063	0.0045	741.8047	39.4766
Module start 'content'	0.0108	0.7966	781.2813	521.7656
Module end 'content'	0.8074	0.0012	1,303.0469	8.2813
Script end	0.8086		1,311.3281

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0034	0.4198	16	0.0002
Check MTime	0.0012	0.1519	16	0.0001
Mysql Total
Database connection	0.0010	0.1201	1	0.0010
Mysqli_queries	0.7596	93.9264	54	0.0141
Looping result	0.0004	0.0545	52	0.0000
Template Total	0.7736	95.7	2	0.3868
Template load	0.0022	0.2750	2	0.0011
Template processing	0.7713	95.3826	2	0.3857
Template load and register function	0.0002	0.0188	1	0.0002
states
state_id_array	0.0008	0.0952	1	0.0008
state_identifier_array	0.0007	0.0816	2	0.0003
Override
Cache load	0.0019	0.2317	18	0.0001
Sytem overhead
Fetch class attribute can translate value	0.0006	0.0716	2	0.0003
Fetch class attribute name	0.0014	0.1754	3	0.0005
XML
Image XML parsing	0.0009	0.1060	2	0.0004
class_abstraction
Instantiating content class attribute	0.0000	0.0010	3	0.0000
General
dbfile	0.0048	0.5925	16	0.0003
String conversion	0.0000	0.0013	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	full/forum_topic.tpl	extension/sevenx/design/simple/override/templates/full/forum_topic.tpl
2	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
3	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
1	content/datatype/view/ezimage.tpl	<No override>	extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tpl
1	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 9 Number of unique templates used: 6

Time used to render debug report: 0.0001 secs