Accented characters are not working in solr search

Next topic

Author	Message
Praveen Kumar	Tuesday 16 August 2011 5:00:23 pm Hi, This is Praveen. I am using apache-solr in our project to support search on cities. I having a problem with the accented characters while searching. For example: My city name is 'vrély'. if i search for 'vr', it is giving the result. But if i search for 'vrél', it is not giving any results. But if i search without accented characters** like 'vre*', it again give results. My city field type is "text" and my schema.xml for this as follows: <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> </analyzer> </fieldType> Any suggestions or solution to resolve my problem is appreciable. Thanks in Advance... Regards, Praveen Kumar
Ivo Lukac	Wednesday 17 August 2011 12:59:15 am There could be 2 things: - either your index and query analyzer are not the same (e.g. there is a small difference: catenateWords="0" catenateNumbers="0") so tokens are not the same in both situations or - the "é" character is somehow badly encoded when sent to solr as a query I had a similar problem before when I used jetty, it didn't support utf-8 queries very well. I switched to tomcat. Could be that jetty resolved those issues in newer version, I didn't check. Anyway, you need to be aware that "vrély" is always tokenized as "vrely", that is why you are finding it with vr* and vre* http://www.linkedin.com/in/ivolukac http://www.netgen.hr/eng/blog http://twitter.com/ilukac
Philippe VINCENT-ROYOL	Wednesday 17 August 2011 1:24:20 am Just a question : which version of solr do you use? Certified Developer (4.1): http://auth.ez.no/certification/verify/272607 Certified Developer (4.4): http://auth.ez.no/certification/verify/377321 G+ : http://plus.tl/dspe Twitter : http://twitter.com/dspe

Author

Message

Tuesday 16 August 2011 5:00:23 pm

Hi,
This is Praveen. I am using apache-solr in our project to support search on cities. I having a problem with the accented characters while searching.
For example:
My city name is 'vrély'.
if i search for 'vr*', it is giving the result.
But if i search for 'vrél*', it is not giving any results.
But if i search without accented characters like 'vre*', it again give results.
My city field type is "text" and my schema.xml for this as follows:
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>


<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/>
</analyzer>
</fieldType>
Any suggestions or solution to resolve my problem is appreciable.
Thanks in Advance...
Regards,
Praveen Kumar

Ivo Lukac

Wednesday 17 August 2011 12:59:15 am

There could be 2 things:

- either your index and query analyzer are not the same (e.g. there is a small difference: catenateWords="0" catenateNumbers="0") so tokens are not the same in both situations or

- the "é" character is somehow badly encoded when sent to solr as a query

I had a similar problem before when I used jetty, it didn't support utf-8 queries very well. I switched to tomcat. Could be that jetty resolved those issues in newer version, I didn't check.

Anyway, you need to be aware that "vrély" is always tokenized as "vrely", that is why you are finding it with vr* and vre*

http://www.linkedin.com/in/ivolukac
http://www.netgen.hr/eng/blog
http://twitter.com/ilukac

Philippe VINCENT-ROYOL

Wednesday 17 August 2011 1:24:20 am

Just a question : which version of solr do you use?

Certified Developer (4.1): http://auth.ez.no/certification/verify/272607
Certified Developer (4.4): http://auth.ez.no/certification/verify/377321

G+ : http://plus.tl/dspe
Twitter : http://twitter.com/dspe

eZ debug

Timing:	Jan 18 2025 02:59:24
Script start
Timing:	Jan 18 2025 02:59:24
Module start 'layout'
Timing:	Jan 18 2025 02:59:24
Module start 'content'
Timing:	Jan 18 2025 02:59:25
Module end 'content'
Timing:	Jan 18 2025 02:59:25
Script end

Main resources:

Total runtime	0.9001 sec
Peak memory usage	4,096.0000 KB
Database Queries	59

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0055	588.0469	152.6406
Module start 'layout'	0.0055	0.0040	740.6875	39.5000
Module start 'content'	0.0095	0.8892	780.1875	595.1641
Module end 'content'	0.8987	0.0013	1,375.3516	16.1250
Script end	0.9000		1,391.4766

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0032	0.3504	16	0.0002
Check MTime	0.0013	0.1480	16	0.0001
Mysql Total
Database connection	0.0007	0.0752	1	0.0007
Mysqli_queries	0.8485	94.2752	59	0.0144
Looping result	0.0005	0.0513	57	0.0000
Template Total	0.8698	96.6	2	0.4349
Template load	0.0018	0.1971	2	0.0009
Template processing	0.8680	96.4435	2	0.4340
Template load and register function	0.0001	0.0158	1	0.0001
states
state_id_array	0.0008	0.0842	1	0.0008
state_identifier_array	0.0016	0.1824	2	0.0008
Override
Cache load	0.0015	0.1666	48	0.0000
Sytem overhead
Fetch class attribute can translate value	0.0004	0.0462	3	0.0001
Fetch class attribute name	0.0008	0.0893	5	0.0002
XML
Image XML parsing	0.0011	0.1250	3	0.0004
class_abstraction
Instantiating content class attribute	0.0000	0.0012	5	0.0000
General
dbfile	0.0020	0.2245	24	0.0001
String conversion	0.0000	0.0011	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	full/forum_topic.tpl	extension/sevenx/design/simple/override/templates/full/forum_topic.tpl
3	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
3	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
2	content/datatype/view/ezxmltags/strong.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/strong.tpl
3	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
2	content/datatype/view/ezimage.tpl	<No override>	extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 15 Number of unique templates used: 7

Time used to render debug report: 0.0001 secs