Forums / Extensions / eZ Find / Indexer script stops indexing at 36% without error message

Indexer script stops indexing at 36% without error message

Author Message

Benjamin Plaquevent

Tuesday 11 March 2008 3:13:39 am

Hi,

I currently use eZFind 1.0.0beta2 on eZPublish 4.0.

I have a strange eZFind indexer script behaviour : It starts indexing and stops at 36% without any error message... (when run without special debug / loging params)

Indexer script terminates with following output:

Creating a new thread: 2645
.................................................. 34.19%
...................................................................... 35.17%
...................................................................... 36.15%
..........Process finished: 2643
Process finished: 2645
Optimizing. Please wait ...
Indexing took 242.269028902 secs ( average: 29.5745602831 objects/sec )
Finished updating the search index.

Solr server output:

INFO: Closing Searcher@fbf51d main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}

eZPublish error log:

[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: BEGIN WORK
[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: COMMIT
[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: DELETE FROM ezsession WHERE session_key='d7bddb3cb470f958bc95d88d835477fc'
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: BEGIN WORK
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: COMMIT
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: DELETE FROM ezsession WHERE session_key='d7bddb3cb470f958bc95d88d835477fc'

So it appears that there is a mysql connexion error but I can't figure why.
I try to index ~20.000 ezpublish objects.
Trying to index few basic objects works normaly.

Does anyone can help please?

Benjamin Plaquevent
Project Manager
Smile

Andreas Adelsberger

Tuesday 11 March 2008 3:30:46 am

Did you start your start.jar in extension/ezfind/java with enought memory heap like

extension/ezfind/java:

java -Xms1500M -Xmx1500M -jar start.jar 

?

---------------------------------------
Styleflasher New Media OG
Websites. Games/Multimedia.

Benjamin Plaquevent

Tuesday 11 March 2008 3:47:41 am

I start Solr with:

java -Xms512M -Xmx512M -jar start.jar

But even if I change memory heap, indexer script keep on stoping at the same point (36%).

I don't hava a clue of what's going wrong...

I changed php.ini memory_limit too but no success.

Benjamin Plaquevent
Project Manager
Smile

Heath

Tuesday 11 March 2008 5:17:51 am

Ahh! I had heard this would become a problem with eZ Find ...

We have seen this design flaw before in the php indexing cronjob/scripts in the past releases.

We created an alternative solution which has been used by several customers with great success.

http://projects.ez.no/updatesearchindex

Now eZ Find appears at a glance to offer the same problem. I think the solution for you might be along similar lines.

In the end the problem logic surrounds how it indexes sites with any more than a small amount of content object content.

Our solution provides for an incremental indexing solution which might be able to be adapted to meet your needs with eZ find.

IE: use the logic/design pattern not actual code of updatesearchindex extension.

Cheers,
Heath

Brookins Consulting | http://brookinsconsulting.com/
Certified | http://auth.ez.no/certification/verify/380350
Solutions | http://projects.ez.no/users/community/brookins_consulting
eZpedia community documentation project | http://ezpedia.org

Benjamin Plaquevent

Friday 14 March 2008 8:34:30 am

I finally found a temporary work around:
I disabled fork by setting $useFork to false in updatesearchindexsolr.php.
I can't figure out what exactly happens but I think it is in relation with concurrent processes running.
I noticed that indexer stops when it finishes indexing the first top node...

Shouldn't I report this is the issue tracker?

Benjamin Plaquevent
Project Manager
Smile

Kåre Køhler Høvik

Friday 14 March 2008 8:55:45 am

Hi

Are you using persistent MySQL connections ?
Which PHP version are you using ?

It sounds like a MySQL connection problem. This was fixed, but not sure it was done in the beta2.

Kåre Høvik

Benjamin Plaquevent

Friday 14 March 2008 9:41:33 am

Hi,

I use PHP 5.2.0.
I tried with both UsePersistentConnection=enabled and UsePersistentConnection=disabled, no success.
Is there a more recent version of eZ Find for eZ Publish 4.0 available over there?

Thank you for your response.

Benjamin Plaquevent
Project Manager
Smile

Kåre Køhler Høvik

Friday 21 March 2008 7:41:36 am

Hi

A new version of eZ Find for eZP4 should be out by the beginning of April.

Kåre Høvik

Francisca Hernández

Sunday 25 May 2008 3:10:11 pm

Hi,

I use ezfind in ezp4 and i have to put $useFork = false; because i get HTTP error 500.

thanks,
fran

Iguana IT, SL - http://www.iguanait.com

Ronan Guilloux

Tuesday 23 September 2008 9:21:02 am

Disabling $useFork to false in updatesearchindexsolr.php solved a indexation problem of us, too.

Thanks to Francisca & Benjamin.

--
Ronan Guilloux
www.mazedia.fr
www.coolforest.net

eZ debug

Timing: Jan 30 2025 00:50:19
Script start
Timing: Jan 30 2025 00:50:19
Module start 'content'
Timing: Jan 30 2025 00:50:19
Module end 'content'
Timing: Jan 30 2025 00:50:20
Script end

Main resources:

Total runtime0.2152 sec
Peak memory usage2,048.0000 KB
Database Queries141

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0059 588.2344180.7422
Module start 'content' 0.00590.0055 768.9766111.1797
Module end 'content' 0.01140.2038 880.1563543.2188
Script end 0.2152  1,423.3750 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00321.4968200.0002
Check MTime0.00140.6312200.0001
Mysql Total
Database connection0.00070.325210.0007
Mysqli_queries0.172480.08481410.0012
Looping result0.00120.56411390.0000
Template Total0.203494.510.2034
Template load0.00090.421510.0009
Template processing0.202594.091910.2025
Override
Cache load0.00070.302510.0007
Sytem overhead
Fetch class attribute can translate value0.00130.588210.0013
XML
Image XML parsing0.00020.106810.0002
General
dbfile0.00502.3397200.0003
String conversion0.00000.001830.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 1
 Number of unique templates used: 1

Time used to render debug report: 0.0001 secs