Problem with really BIG solr indices

Thursday 19 March 2009 4:52:13 am

Hi all guys,

let's try to explain the problem encountered with eZFind 2.0.0 (also with previous versions).

Background: we've indexed in the solr engine about 30k eZContentObjects, also with 226k external XML files, this generates about 5 GB of index data.

Currently we have started the solr engine with this command:

/usr/bin/java -server -Xmx600m -Xms600m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:NewRatio=5 -jar start.jar

Our main problem, is that every time that a commit is executed in the solr engine, all index files (remember, 5GB data) are regenerated from scratch, the size of the data folder grows up until about 10 GB, then the old files are gone and the new one remain in the data folder.
As we have delayed indexing enabled, this is not a critical problem on publishing content, but it is when deleting something, as every time we remove some object the system freezes until indices are regenerated.

Anyone out there with similar escenario that can guide us?

Thanks for reading!
Xavier

--
Xavier Serna
eZ Publish Certified Developer
Departament de Software
Microblau S.L. - http://www.microblau.net
+34 937 466 205

Friday 10 April 2009 12:32:55 am

@Xavier :
Any feedback on your issue ? Did the proposed solution of disabling the OptimizeOnCommit directive + setting up a daily 'optimize' workflow work ?

@Ali :
This index size is very surprising. Did the indexed content base grow a lot between the eZ Find 1.x usage and eZ Find 2.0 ? Are external elements indexed ( through the DataImportHandler Solr extension for instance ? ) too ? Websites crawled ?

Best regards,

--
Nicolas Pastorino
Director Community - eZ
Member of the Community Project Board

eZ Publish Community on twitter: http://twitter.com/ezcommunity

t : http://twitter.com/jeanvoye
G+ : http://plus.tl/jeanvoye

Thursday 16 April 2009 12:50:04 am

Hi Nico,

many thanks for your proposed solution, it seems to work fine now disabling optimizeoncommit.
Only one detail, in the updatesearchindexsolr.php on each commit, every 1000 objects, it's forced an optimize, not respecting the setting in the ini file. I believe that this should be updated, because reindexation of the whole xml files takes more than 4 hours.

thanks!

--
Xavier Serna
eZ Publish Certified Developer
Departament de Software
Microblau S.L. - http://www.microblau.net
+34 937 466 205