Indexing content of files using Solr (ezfind)

Next topic

Author	Message
Laurence Bonhomme	Wednesday 01 October 2008 1:36:23 am Hi there, I'm trying to index the content of files (txt, pdf) using eZFind + Solr and have troubles with it. eZPublish 4.0.1 eZFind 1.0.0 beta2 Linux Debian 4 First of all, I've installed the eZPublish and eZFind package as recommended. When I create a new media/file and upload a file (txt or pdf), indexing works perfectly and I can make searches (I can find my words into the database table ezkeyword as well). Because I found the raw Search a bit "light", I decided to test with the Solr. ... And everything gets wrong now. Pretty sure that the thing is well installed because I can search for articles contents or file summary into the admin Solr search. But nothing about the <b>content of the uploaded file</b> itself. What am I doing wrong? Is there a trick? Having a look at the Solr guide (http://wiki.apache.org/solr/), I found this : "Solr has an extensible DocumentHandler architecture that allows you to feed it XML and CSV documents. There is now a patch file available as part of SOLR-284 that adds support for parsing rich binary formats. " Do we have to patch the provided Solr? Would anyone be so kind to help? Thanks a lot Laurence
Christian Rößler	Wednesday 22 October 2008 11:01:54 am Laurence, perhaps a bit late but better late than never :-) I have had pretty the same problems. Perhaps this link will help you: http://ez.no/developer/articles/indexing_multiple_binary_file_types I was able to setup a generic binaryfilehandler was was called for every physical file by ezfind. This binaryfilehandler was calling several external programs (pdftotxt, doc2txt), the parsed contents of each file was printed to stdout and catched by the binaryfilehandler, later on returned to ezflow, which saved it into ezsolr-index via a http-request. The tricky point is to get ezfind use the custom file-handler to parse the binaryfile's content that ezflow/ezsolr can work with. The above supplied link contains a full featured howto + downloads to get it working. If you need any further help, feel free to reply to this post. chris. Hannover, Germany eZ-Certified http://auth.ez.no/certification/verify/395613
Paul Borgermans	Wednesday 22 October 2008 2:21:15 pm Something to point out here: the configuration for indexing files like pdf, word, ... depends on the configuration of eZ publish to convert these to plain text. It has nothing to do with the search plugin used (default, Solr/eZ Find, ...). We'll improve the conversion mechanism options in eZ Publish for the next iteration of eZ Publish (4.1), I'm investigating a few more options to handle also more file formats. You'll learn more about that very soon (< 3 weeks) Paul eZ Publish, eZ Find, Solr expert consulting and training http://twitter.com/paulborgermans
Geoff Bentley	Wednesday 25 February 2009 3:21:12 pm Check out Paul's ezTika extension ( http://projects.ez.no/eztika ) which draws on the Apache Tika toolkit ( http://lucene.apache.org/tika/ ) - this works seamlessly (so far) with eZ Find.

eZ debug

Timing:	Jan 31 2025 04:18:07
Script start
Timing:	Jan 31 2025 04:18:07
Module start 'layout'
Timing:	Jan 31 2025 04:18:07
Module start 'content'
Timing:	Jan 31 2025 04:18:08
Module end 'content'
Timing:	Jan 31 2025 04:18:08
Script end

Main resources:

Total runtime	1.0515 sec
Peak memory usage	4,096.0000 KB
Database Queries	64

Timing points:

Checkpoint	Start (sec)	Duration (sec)	Memory at start (KB)	Memory used (KB)
Script start	0.0000	0.0048	588.3828	151.2422
Module start 'layout'	0.0048	0.0023	739.6250	36.7109
Module start 'content'	0.0071	1.0436	776.3359	1,027.4531
Module end 'content'	1.0507	0.0008	1,803.7891	11.7969
Script end	1.0515		1,815.5859

Time accumulators:

Accumulator	Duration (sec)	Duration (%)	Count	Average (sec)
Ini load
Load cache	0.0031	0.2911	16	0.0002
Check MTime	0.0013	0.1198	16	0.0001
Mysql Total
Database connection	0.0006	0.0616	1	0.0006
Mysqli_queries	0.9917	94.3073	64	0.0155
Looping result	0.0006	0.0548	62	0.0000
Template Total	1.0184	96.8	2	0.5092
Template load	0.0020	0.1893	2	0.0010
Template processing	1.0164	96.6566	2	0.5082
Template load and register function	0.0012	0.1120	1	0.0012
states
state_id_array	0.0015	0.1382	1	0.0015
state_identifier_array	0.0021	0.2041	2	0.0011
Override
Cache load	0.0017	0.1611	40	0.0000
Sytem overhead
Fetch class attribute can translate value	0.0005	0.0488	4	0.0001
Fetch class attribute name	0.0020	0.1876	6	0.0003
XML
Image XML parsing	0.0034	0.3242	4	0.0009
class_abstraction
Instantiating content class attribute	0.0000	0.0010	6	0.0000
General
dbfile	0.0041	0.3868	23	0.0002
String conversion	0.0000	0.0006	4	0.0000
Note: percentages do not add up to 100% because some accumulators overlap

Templates used to render the page:

Usage	Requested template	Template	Template loaded
1	node/view/full.tpl	full/forum_topic.tpl	extension/sevenx/design/simple/override/templates/full/forum_topic.tpl
4	content/datatype/view/ezxmltext.tpl	<No override>	extension/community_design/design/suncana/templates/content/datatype/view/ezxmltext.tpl
11	content/datatype/view/ezxmltags/paragraph.tpl	<No override>	extension/ezwebin/design/ezwebin/templates/content/datatype/view/ezxmltags/paragraph.tpl
7	content/datatype/view/ezxmltags/line.tpl	<No override>	design/standard/templates/content/datatype/view/ezxmltags/line.tpl
2	content/datatype/view/ezimage.tpl	<No override>	extension/sevenx/design/simple/templates/content/datatype/view/ezimage.tpl
1	print_pagelayout.tpl	<No override>	extension/community/design/community/templates/print_pagelayout.tpl
Number of times templates used: 26 Number of unique templates used: 6

Time used to render debug report: 0.0001 secs