Indexer script stops indexing at 36% without error message

Author Message

Benjamin Plaquevent

Tuesday 11 March 2008 3:13:39 am

Hi,

I currently use eZFind 1.0.0beta2 on eZPublish 4.0.

I have a strange eZFind indexer script behaviour : It starts indexing and stops at 36% without any error message... (when run without special debug / loging params)

Indexer script terminates with following output:

Creating a new thread: 2645
.................................................. 34.19%
...................................................................... 35.17%
...................................................................... 36.15%
..........Process finished: 2643
Process finished: 2645
Optimizing. Please wait ...
Indexing took 242.269028902 secs ( average: 29.5745602831 objects/sec )
Finished updating the search index.

Solr server output:

INFO: Closing Searcher@fbf51d main
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}

eZPublish error log:

[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: BEGIN WORK
[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: COMMIT
[ Mar 11 2008 11:07:02 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: DELETE FROM ezsession WHERE session_key='d7bddb3cb470f958bc95d88d835477fc'
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: BEGIN WORK
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: COMMIT
[ Mar 11 2008 11:07:08 ] [] eZMySQLDB:
Query error: MySQL server has gone away. Query: DELETE FROM ezsession WHERE session_key='d7bddb3cb470f958bc95d88d835477fc'

So it appears that there is a mysql connexion error but I can't figure why.
I try to index ~20.000 ezpublish objects.
Trying to index few basic objects works normaly.

Does anyone can help please?

Benjamin Plaquevent
Project Manager
Smile

Andreas Adelsberger

Tuesday 11 March 2008 3:30:46 am

Did you start your start.jar in extension/ezfind/java with enought memory heap like

extension/ezfind/java:

java -Xms1500M -Xmx1500M -jar start.jar 

?

---------------------------------------
Styleflasher New Media OG
Websites. Games/Multimedia.

Benjamin Plaquevent

Tuesday 11 March 2008 3:47:41 am

I start Solr with:

java -Xms512M -Xmx512M -jar start.jar

But even if I change memory heap, indexer script keep on stoping at the same point (36%).

I don't hava a clue of what's going wrong...

I changed php.ini memory_limit too but no success.

Benjamin Plaquevent
Project Manager
Smile

Heath

Tuesday 11 March 2008 5:17:51 am

Ahh! I had heard this would become a problem with eZ Find ...

We have seen this design flaw before in the php indexing cronjob/scripts in the past releases.

We created an alternative solution which has been used by several customers with great success.

http://projects.ez.no/updatesearchindex

Now eZ Find appears at a glance to offer the same problem. I think the solution for you might be along similar lines.

In the end the problem logic surrounds how it indexes sites with any more than a small amount of content object content.

Our solution provides for an incremental indexing solution which might be able to be adapted to meet your needs with eZ find.

IE: use the logic/design pattern not actual code of updatesearchindex extension.

Cheers,
Heath

Brookins Consulting | http://brookinsconsulting.com/
Certified | http://auth.ez.no/certification/verify/380350
Solutions | http://projects.ez.no/users/community/brookins_consulting
eZpedia community documentation project | http://ezpedia.org

Benjamin Plaquevent

Friday 14 March 2008 8:34:30 am

I finally found a temporary work around:
I disabled fork by setting $useFork to false in updatesearchindexsolr.php.
I can't figure out what exactly happens but I think it is in relation with concurrent processes running.
I noticed that indexer stops when it finishes indexing the first top node...

Shouldn't I report this is the issue tracker?

Benjamin Plaquevent
Project Manager
Smile

Kåre Køhler Høvik

Friday 14 March 2008 8:55:45 am

Hi

Are you using persistent MySQL connections ?
Which PHP version are you using ?

It sounds like a MySQL connection problem. This was fixed, but not sure it was done in the beta2.

Kåre Høvik

Benjamin Plaquevent

Friday 14 March 2008 9:41:33 am

Hi,

I use PHP 5.2.0.
I tried with both UsePersistentConnection=enabled and UsePersistentConnection=disabled, no success.
Is there a more recent version of eZ Find for eZ Publish 4.0 available over there?

Thank you for your response.

Benjamin Plaquevent
Project Manager
Smile

Kåre Køhler Høvik

Friday 21 March 2008 7:41:36 am

Hi

A new version of eZ Find for eZP4 should be out by the beginning of April.

Kåre Høvik

Francisca Hernández

Sunday 25 May 2008 3:10:11 pm

Hi,

I use ezfind in ezp4 and i have to put $useFork = false; because i get HTTP error 500.

thanks,
fran

Iguana IT, SL - http://www.iguanait.com

Ronan Guilloux

Tuesday 23 September 2008 9:21:02 am

Disabling $useFork to false in updatesearchindexsolr.php solved a indexation problem of us, too.

Thanks to Francisca & Benjamin.

--
Ronan Guilloux
www.mazedia.fr
www.coolforest.net

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.

eZ debug

Timing: Jan 18 2025 10:36:14
Script start
Timing: Jan 18 2025 10:36:14
Module start 'layout'
Timing: Jan 18 2025 10:36:14
Module start 'content'
Timing: Jan 18 2025 10:36:16
Module end 'content'
Timing: Jan 18 2025 10:36:16
Script end

Main resources:

Total runtime1.2890 sec
Peak memory usage4,096.0000 KB
Database Queries88

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0053 589.3359152.6563
Module start 'layout' 0.00530.0034 741.992239.5234
Module start 'content' 0.00871.2787 781.5156805.7031
Module end 'content' 1.28740.0015 1,587.218824.0859
Script end 1.2889  1,611.3047 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00320.2510160.0002
Check MTime0.00140.1076160.0001
Mysql Total
Database connection0.00080.062410.0008
Mysqli_queries1.181591.6594880.0134
Looping result0.00130.0999860.0000
Template Total1.257797.620.6288
Template load0.00210.160020.0010
Template processing1.255697.407320.6278
Template load and register function0.00010.009510.0001
states
state_id_array0.00080.061610.0008
state_identifier_array0.00120.093720.0006
Override
Cache load0.00190.1471760.0000
Sytem overhead
Fetch class attribute can translate value0.00090.066360.0001
Fetch class attribute name0.00180.1379160.0001
XML
Image XML parsing0.01421.099060.0024
class_abstraction
Instantiating content class attribute0.00000.0031200.0000
General
dbfile0.01240.9601460.0003
String conversion0.00010.004040.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1node/view/full.tplfull/forum_topic.tplextension/sevenx/design/simple/override/templates/full/forum_topic.tplEdit templateOverride template
10content/datatype/view/ezimage.tpl<No override>extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tplEdit templateOverride template
10content/datatype/view/ezxmltext.tpl<No override>extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tplEdit templateOverride template
19content/datatype/view/ezxmltags/paragraph.tpl<No override>extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tplEdit templateOverride template
5content/datatype/view/ezxmltags/literal.tpl<No override>extension/community/design/standard/templates/content/datatype/view/ezxmltags/literal.tplEdit templateOverride template
6content/datatype/view/ezxmltags/line.tpl<No override>design/standard/templates/content/datatype/view/ezxmltags/line.tplEdit templateOverride template
1print_pagelayout.tpl<No override>extension/community/design/community/templates/print_pagelayout.tplEdit templateOverride template
 Number of times templates used: 52
 Number of unique templates used: 7

Time used to render debug report: 0.0002 secs