Forums / Developer / Problem with Import of Text to UTF-Database

Problem with Import of Text to UTF-Database

Author Message

Rose Futchko

Monday 31 October 2005 7:19:27 pm

Recently, we imported over 3,000 text articles to the eZpublish database using the provided import routine on release 3.70. It seemed that this conversion function normalized the XML to eZ publish compliant code and improperly stripped some special characters, converting them to question marks, and did not convert others.

For example,

1) The special character are encoded properly as UTF-8 in the import XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94').

2) In the samples, these characters are incorrectly coded in the MySQL database simply as '3F' (the question mark). We had thought they might be in Latin1, Unicode, or encoded as HTML character entities, but they are not.

3) The eZ Online Editor allows you to use the special character entry button to enter special characters. You can also enter special character using the Windows "Character Map." These characters are then stored in the database as proper UTF-8, just like in the XML (ex. the euro sign is 'E2-82-AC'; the em-dash is 'E2-80-94').

Has anyone run into this problem and if they have, do you have a work around solution that you can share for updating these special characters in the mysql database?

Thank you.

Xavier Dutoit

Monday 31 October 2005 11:59:59 pm

Hi,

What is the routine you've used ? Don't get it.
Could you detail that steps ?

As for the chars, I always have problem with win-1251 chars when I switch to utf-8.

I wrote a wee dirty script to convert the win chars (the quotes, dash and the al), to normal chars. I dump the database, apply it and reimport the modified dump. However, if the chars are already screwed in the mysql database, that's probably not going to work.

X+

http://www.sydesy.com

eZ debug

Timing: Jan 18 2025 18:27:24
Script start
Timing: Jan 18 2025 18:27:24
Module start 'content'
Timing: Jan 18 2025 18:27:25
Module end 'content'
Timing: Jan 18 2025 18:27:25
Script end

Main resources:

Total runtime0.8878 sec
Peak memory usage4,096.0000 KB
Database Queries191

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0099 588.9766180.8125
Module start 'content' 0.00990.7754 769.7891526.8594
Module end 'content' 0.78530.1025 1,296.6484332.8984
Script end 0.8878  1,629.5469 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00420.4762210.0002
Check MTime0.00170.1868210.0001
Mysql Total
Database connection0.00170.193310.0017
Mysqli_queries0.821492.51661910.0043
Looping result0.00160.18121890.0000
Template Total0.851295.920.4256
Template load0.00220.250120.0011
Template processing0.849095.622720.4245
Template load and register function0.00010.016210.0001
states
state_id_array0.00160.179510.0016
state_identifier_array0.00090.100220.0004
Override
Cache load0.00180.2029180.0001
Sytem overhead
Fetch class attribute can translate value0.00080.090130.0003
Fetch class attribute name0.00090.100130.0003
XML
Image XML parsing0.00080.092830.0003
class_abstraction
Instantiating content class attribute0.00000.000930.0000
General
dbfile0.00560.6311270.0002
String conversion0.00000.001130.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
2content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
3content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
1content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
1content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 9
 Number of unique templates used: 6

Time used to render debug report: 0.0002 secs