Forums / Developer / eZ Find / Solr config for partial search on latin chars

eZ Find / Solr config for partial search on latin chars

Author Message

Bjørnar Grøtterud

Friday 13 May 2011 8:16:29 am

Hi,

We have an install with eZ find 2.3 configured with ReversedWildcardFilterFactory. It works good on partial wildcard searches, but we still have issues when searching for words that starts with an latin character, when the query is to, three or four characters long.

An example:

We have indexed an object named Ølbolle

Query - result:

øl - no result

ølb - no result

ølbo - no result

ølbol - no result

ølboll - ølbolle

ølbolle - ølbolle

When searchin for word starting with non-latin characters, this is not an issue.

Our indez-analyser is set up as follows:

<analyzer type="index">

<tokenizer class="solr.WhitespaceTokenizerFactory"/>


<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />


<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>


<filter class="solr.LowerCaseFilterFactory"/>
<filter class="ISOLatin1AccentFilterFactory"/>

<filter class="solr.ReversedWildcardFilterFactory" withOriginal="true" maxPosAsterisk="3" maxPosQuestion="2" maxFractionAsterisk="0.33"/>
</analyzer>

Any experience / suggestions greatly appreciated.

Paul Borgermans

Monday 16 May 2011 11:58:44 am

Hello Bjørnar

I guess you mean that using normal wildcards (asterix) after the characters which are added in your template logic

The culprit is actually the ISOLatin filter factory: when doing wildcard searches, the analysis steps are not done.There is still no stable resolution in Solr (where the "problem" actually is). The same for lowercasing by the way.

If you really want to rely so much on wildcards (which I dont recommend either actually), best is to remove the ISOlatin "normalisation" as well (both index and query analysis steps)

Furthermore, I think you did not remove the stemming step for the query part of your text field type in schema.xml (otherwise you should not get a match for ølboll). You should absolutely remove the stemming part there as well.

It would be good that you email me your schema.xml for closer inspection

Cheers

Paul

eZ Publish, eZ Find, Solr expert consulting and training
http://twitter.com/paulborgermans

Bjørnar Grøtterud

Thursday 19 May 2011 12:42:11 am

Hi Paul,

thanks for your reply.

The reason why we have added wildcards, is that we found this as the only solution for partial search for both first and last part of a word. It actually works well, exept for latin chars.

Ill try your suggestion on removing the ISOLatin1AccentFilterFactory.

We have already removed the SnowballPorterFilterFactory from the query part in our schema.xml.

Ill send you our schema.xml, thanks for taking a look!

Best regards

Bjørnar

eZ debug

Timing: Jan 29 2025 13:38:54
Script start
Timing: Jan 29 2025 13:38:54
Module start 'content'
Timing: Jan 29 2025 13:38:54
Module end 'content'
Timing: Jan 29 2025 13:38:54
Script end

Main resources:

Total runtime0.1354 sec
Peak memory usage2,048.0000 KB
Database Queries141

Timing points:

CheckpointStart (sec)Duration (sec)Memory at start (KB)Memory used (KB)
Script start 0.00000.0055 588.0781180.8203
Module start 'content' 0.00550.0051 768.898498.0703
Module end 'content' 0.01070.1246 866.9688527.3594
Script end 0.1353  1,394.3281 

Time accumulators:

 Accumulator Duration (sec) Duration (%) Count Average (sec)
Ini load
Load cache0.00332.4507200.0002
Check MTime0.00130.9301200.0001
Mysql Total
Database connection0.00080.613210.0008
Mysqli_queries0.091567.59671410.0006
Looping result0.00110.79711390.0000
Template Total0.124391.810.1243
Template load0.00070.506810.0007
Template processing0.123691.310710.1236
Override
Cache load0.00050.353210.0005
Sytem overhead
Fetch class attribute can translate value0.00060.468610.0006
XML
Image XML parsing0.00020.148510.0002
General
dbfile0.00584.3174200.0003
String conversion0.00000.003530.0000
Note: percentages do not add up to 100% because some accumulators overlap

CSS/JS files loaded with "ezjscPacker" during request:

CacheTypePacklevelSourceFiles
CSS0extension/community/design/community/stylesheets/ext/jquery.autocomplete.css
extension/community_design/design/suncana/stylesheets/scrollbars.css
extension/community_design/design/suncana/stylesheets/tabs.css
extension/community_design/design/suncana/stylesheets/roadmap.css
extension/community_design/design/suncana/stylesheets/content.css
extension/community_design/design/suncana/stylesheets/star-rating.css
extension/community_design/design/suncana/stylesheets/syntax_and_custom_tags.css
extension/community_design/design/suncana/stylesheets/buttons.css
extension/community_design/design/suncana/stylesheets/tweetbox.css
extension/community_design/design/suncana/stylesheets/jquery.fancybox-1.3.4.css
extension/bcsmoothgallery/design/standard/stylesheets/magnific-popup.css
extension/sevenx/design/simple/stylesheets/star_rating.css
extension/sevenx/design/simple/stylesheets/libs/fontawesome/css/all.min.css
extension/sevenx/design/simple/stylesheets/main.v02.css
extension/sevenx/design/simple/stylesheets/main.v02.res.css
JS0extension/ezjscore/design/standard/lib/yui/3.17.2/build/yui/yui-min.js
extension/ezjscore/design/standard/javascript/jquery-3.7.0.min.js
extension/community_design/design/suncana/javascript/jquery.ui.core.min.js
extension/community_design/design/suncana/javascript/jquery.ui.widget.min.js
extension/community_design/design/suncana/javascript/jquery.easing.1.3.js
extension/community_design/design/suncana/javascript/jquery.ui.tabs.js
extension/community_design/design/suncana/javascript/jquery.hoverIntent.min.js
extension/community_design/design/suncana/javascript/jquery.popmenu.js
extension/community_design/design/suncana/javascript/jScrollPane.js
extension/community_design/design/suncana/javascript/jquery.mousewheel.js
extension/community_design/design/suncana/javascript/jquery.cycle.all.js
extension/sevenx/design/simple/javascript/jquery.scrollTo.js
extension/community_design/design/suncana/javascript/jquery.cookie.js
extension/community_design/design/suncana/javascript/ezstarrating_jquery.js
extension/community_design/design/suncana/javascript/jquery.initboxes.js
extension/community_design/design/suncana/javascript/app.js
extension/community_design/design/suncana/javascript/twitterwidget.js
extension/community_design/design/suncana/javascript/community.js
extension/community_design/design/suncana/javascript/roadmap.js
extension/community_design/design/suncana/javascript/ez.js
extension/community_design/design/suncana/javascript/ezshareevents.js
extension/sevenx/design/simple/javascript/main.js

Templates used to render the page:

UsageRequested templateTemplateTemplate loadedEditOverride
1pagelayout.tpl<No override>extension/sevenx/design/simple/templates/pagelayout.tplEdit templateOverride template
 Number of times templates used: 1
 Number of unique templates used: 1

Time used to render debug report: 0.0001 secs