<HTML
><HEAD
><TITLE
>indexer configuration</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.73
"><LINK
REL="HOME"
TITLE="mnoGoSearch 3.2 reference manual"
HREF="index.html"><LINK
REL="UP"
TITLE="Indexing"
HREF="msearch-indexing.html"><LINK
REL="PREVIOUS"
TITLE="Content-Encoding support
"
HREF="msearch-content-enc.html"><LINK
REL="NEXT"
TITLE="Extended indexing features"
HREF="msearch-extended-indexing.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="mnogo.css"><META
NAME="Description"
CONTENT="mnoGoSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, mnoGoSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="sect1"
BGCOLOR="#EEEEEE"
TEXT="#000000"
LINK="#000080"
VLINK="#800080"
ALINK="#FF0000"
><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
>mnoGoSearch 3.2 reference manual: Full-featured search engine software</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="msearch-content-enc.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
>Chapter 3. Indexing</TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="msearch-extended-indexing.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><DIV
CLASS="sect1"
><H1
CLASS="sect1"
><A
NAME="indexer-configuration"
>indexer configuration</A
></H1
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="follow"
>Specifying WEB space to be indexed</A
></H2
><P
>When indexer tries to insert a new URL into database or
is trying to index an existing one, it first of all checks whether
this URL has corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
> or <B
CLASS="command"
>Subnet</B
> command given in
<TT
CLASS="filename"
>indexer.conf</TT
>. URLs without corresponding <B
CLASS="command"
>Server</B
>, <B
CLASS="command"
>Realm</B
>
or <B
CLASS="command"
>Subnet</B
> command are not indexed. By default those URLs which are
already in database and have no Server/Realm/Subnet commands will be deleted
from database. It may happen for example after removing some
Server/Realm/Subnet commands from <TT
CLASS="filename"
>indexer.conf</TT
>.</P
><P
>These commands have following format:
<PRE
CLASS="programlisting"
>[Server | Realm | Subnet] [method] [subsection] [CaseType] [MatchType] [CmpType] pattern [alias]</PRE
>
</P
><P
>Mandatory parameter <TT
CLASS="option"
>pattern</TT
> specify an URL, or it part, or pattern to compare.
</P
><P
> Optional parameter <TT
CLASS="option"
>method</TT
> specify an document action for this command.
May take values:
<TT
CLASS="literal"
>Allow</TT
>, <TT
CLASS="literal"
>Disallow</TT
>, <TT
CLASS="literal"
>HrefOnly</TT
>,
<TT
CLASS="literal"
>CheckOnly</TT
>, <TT
CLASS="literal"
>Skip</TT
>, <TT
CLASS="literal"
>CheckMP3</TT
>,
<TT
CLASS="literal"
>CheckMP3Only</TT
>. By default, the value <TT
CLASS="literal"
>Allow</TT
> is used.
<P
></P
><OL
TYPE="1"
><LI
><P
> <B
CLASS="command"
>Allow</B
>
<A
NAME="AEN979"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>Allow</TT
> specify that all corresponding documets will be indexed
and scaned for new links. Depends on <TT
CLASS="literal"
>Content-Type</TT
> appropriate external parser is executed if need.
</P
></LI
><LI
><P
> <B
CLASS="command"
>Disallow</B
>
<A
NAME="AEN989"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>Disallow</TT
> specify that all corresponding documents will be ignored and
deleted from database, if its was placed into before.
</P
></LI
><LI
><P
> <B
CLASS="command"
>HrefOnly</B
>
<A
NAME="AEN998"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>HrefOnly</TT
> specify that all corresponding documents will be only
scaned for new links (not indexed). This is useful, for example, for mail archives indexing, when index pages is only scaned to
detect new messages for indexing.
<PRE
CLASS="programlisting"
> Server HrefOnly Page http://www.mail-archive.com/general%40mnogosearch.org/
Server Allow Path http://www.mail-archive.com/general%40mnogosearch.org/
</PRE
>
</P
></LI
><LI
><P
> <B
CLASS="command"
>CheckOnly</B
>
<A
NAME="AEN1008"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>CheckOnly</TT
> specify that all corresponding documents wiil be
requested by HTTP HEAD request, not HTTP GET, i.e. inly brief info about documents (size, last modified, content type) will be
fetched. This allow, for example, check links on your site:
<PRE
CLASS="programlisting"
> Server HrefOnly http://www.mnogosearch.org/
Realm CheckOnly *
</PRE
>
</P
><P
> These commands instruct <B
CLASS="command"
>indexer</B
> to scan all documents on <TT
CLASS="literal"
>www.mnogosearch.org</TT
> site and
collect all links. Brief info about every document found will be requested by HEAD method.
After indexing done, <B
CLASS="command"
>indexer -S</B
> command will show status for all documents from this site.
</P
></LI
><LI
><P
> <B
CLASS="command"
>Skip</B
>
<A
NAME="AEN1022"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>Skip</TT
> specify that all corresponding documents will be skiped
while indexing. This is useful when need temporaly disable reindexing several sites, but able search on.
These documents will maked as expired.
</P
></LI
><LI
><P
> <B
CLASS="command"
>CheckMP3</B
>
<A
NAME="AEN1031"
></A
>
</P
><P
> Value <TT
CLASS="literal"
>CheckMP3</TT
> specify that corresponding documents will be checked
for MP3 tags along if its <TT
CLASS="literal"
>Conten-Type</TT
> is equal to <TT
CLASS="literal"
>audio/mpeg</TT
>.
This is useful, for example, if remote server supply <TT
CLASS="literal"
>application/octet-stream</TT
> as
<TT
CLASS="literal"
>Content-Type</TT
> for MP3 files. If this tag is present, these files will indexed as MP3 file, otherwise its
will be processed according to <TT
CLASS="literal"
>Content-Type</TT
>.
</P
></LI
><LI
><P
> <B
CLASS="command"
>CheckMP3Only</B
>
<A
NAME="AEN1045"
></A
>
</P
><P
> This value is equal to <TT
CLASS="literal"
>CheckMP3</TT
>, but if MP3 tag is not present,
processing on <TT
CLASS="literal"
>Content-Type</TT
> will not be taken.
</P
></LI
></OL
>
</P
><P
> Use optional <TT
CLASS="option"
>subsection</TT
>
parameter to specify server's checking behavior. Values of subsection
are the same with "Follow" command arguments. Subsection value must be
one of the following: page, path, site, world and has "path" value by
default. If subsection is not specified, current "Follow" value will
be used. So, the only <TT
CLASS="literal"
>Server site http://localhost/</TT
>
command and combination of <TT
CLASS="literal"
>Follow site</TT
> and
<TT
CLASS="literal"
>Server http://localhost/</TT
> have the same
effect.
<P
></P
><OL
TYPE="1"
><LI
><P
> <TT
CLASS="literal"
>path</TT
> subsection</P
><P
>When indexer seeks for a
"Server" command corresponding to an URL it checks that the discovered
URL starts with URL given in Server command argument but without
trailing file name. For example, if <TT
CLASS="literal"
>Server path
http://localhost/path/to/index.html</TT
> is given, all URLs which
have <TT
CLASS="literal"
>http://localhost/path/to/</TT
> at the beginning
correspond to this Server command.</P
><P
>The following commands have the same effect except that they insert different URLs into database:</P
><P
> <PRE
CLASS="programlisting"
> Server path http://localhost/path/to/index.html
Server path http://localhost/path/to/index
Server path http://localhost/path/to/index.cgi?q=bla
Server path http://localhost/path/to/index?q=bla
</PRE
>
</P
></LI
><LI
><P
> <TT
CLASS="literal"
>site</TT
> subsection</P
><P
>indexer checks that the
discovered URL have the same hostname with URL given in Server
command. For example, <TT
CLASS="literal"
>Server site
http://localhost/path/to/a.html</TT
> will allow to index whole
<TT
CLASS="literal"
>http://localhost/</TT
> server. </P
></LI
><LI
><P
> <TT
CLASS="literal"
>world</TT
> subsection</P
><P
>If world subsection is specified
in Server command, it has the same effect that URL is considered to
match this Server command. See explanation below.</P
></LI
><LI
><P
> <TT
CLASS="literal"
>page</TT
> subsection</P
><P
>This subsection describes the only one URL given in Server argument.</P
></LI
><LI
><P
>subsection in <TT
CLASS="literal"
>news://</TT
> schema</P
><P
>Subsection is always considered
as "site" for news:// URL schema. This is because news:// schema has
no nested paths like ftp:// or http:// Use <TT
CLASS="literal"
>Server
news://news.server.com/</TT
> to index whole news server or for
example <TT
CLASS="literal"
>Server news://news.server.com/udm</TT
> to index
all messages from "udm" hierarchy.</P
></LI
></OL
>
</P
><P
> Optional parameter <TT
CLASS="literal"
>CaseType</TT
> is specify the case sensivity for string comparison, it can take one of follow value:
<TT
CLASS="literal"
>case</TT
> - caseinsensitive comparison, or <TT
CLASS="literal"
>nocase</TT
> - casesensitive comparison.
</P
><P
> Optional parameter <TT
CLASS="literal"
>CmpType</TT
> is specify the type of comparison and can take two value:
<TT
CLASS="literal"
>Regex</TT
> and <TT
CLASS="literal"
>String</TT
>.
<TT
CLASS="literal"
>String</TT
> wildcards is default
match type. You can use ? and * signs in URLMask parameters, they
means "one character" and "any number of characters" respectively. For
example, if you want to index all HTTP sites in .ru domain, use this
command:<PRE
CLASS="programlisting"
>Realm http://*.ru/*</PRE
>
</P
><P
>Regex comparison type takes a regular expression
as it's argument. Activate regex comparison type using <TT
CLASS="option"
>Regex</TT
>
keyword. For example, you can describe everything in .ru domain using
regex comparison type: <PRE
CLASS="programlisting"
>Realm Regex ^http://.*\.ru/</PRE
>
</P
><P
>Optional parameter <TT
CLASS="literal"
>MatchType</TT
> means match type. There
are <TT
CLASS="literal"
>Match</TT
> and <TT
CLASS="literal"
>NoMatch</TT
> possible values with <TT
CLASS="literal"
>Match</TT
> as
default. <TT
CLASS="literal"
>Realm NoMatch</TT
> has reverse effect. It means
that URL that does not match given <TT
CLASS="option"
>pattern</TT
> will correspond to this
<B
CLASS="command"
>Realm</B
> command. For example, use this command to index everything
without .com domain:<PRE
CLASS="programlisting"
>Realm NoMatch http://*.com/*</PRE
>
</P
><P
>Optional <TT
CLASS="option"
>alias</TT
> argument allows providing very
complicated URL rewrite more powerful than other aliasing
mechanism. Take a look <A
HREF="msearch-indexer-configuration.html#aliases"
>the Section called <I
>Aliases</I
></A
> for <TT
CLASS="option"
>alias</TT
> argument usage
explanation. <TT
CLASS="option"
>Alias</TT
> works only with <TT
CLASS="option"
>Regex</TT
> comparison type and has no
effect with <TT
CLASS="option"
>String</TT
> type.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-server"
>Server command
<A
NAME="AEN1118"
></A
></A
></H3
><P
>This is the main command of the
<TT
CLASS="filename"
>indexer.conf</TT
> file. It is used to add servers or
their parts to be indexed.
This command also says indexer to insert given URL into database at startup.</P
><P
>E.g. command <TT
CLASS="literal"
>Server
http://localhost/</TT
> allows to index whole
<TT
CLASS="literal"
>http://localhost/</TT
> server. It also makes indexer
insert given URL into database at startup. You can also specify some
path to index server subsection: <TT
CLASS="literal"
>Server
http://localhost/subsection/</TT
>. It also says indexer to insert
given URL at startup.</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>You can suppress indexer behavior to add
URL given in Server command by using -q indexer command line
argument. It is useful when you have hundreds or thousands Server
commands and their URLs are already in database. This allows having
more quick indexer startup.</P
></BLOCKQUOTE
></DIV
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-realm"
>Realm command
<A
NAME="AEN1131"
></A
></A
></H3
><P
>Realm command is a more powerful means of describing web area to be indexed.
It works almost like <B
CLASS="command"
>Server</B
> command but takes
a regular expression or string wildcards as it's <TT
CLASS="option"
>pattern</TT
> parameter.
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-subnet"
>Subnet command
<A
NAME="AEN1139"
></A
></A
></H3
><P
>Subnet command is another way to describe web area to be indexed.
It works almost like <B
CLASS="command"
>Server</B
> command but takes
a string wildcards as it's <TT
CLASS="option"
>pattern</TT
> argument which is compared against IP
address instead of URL. Argument may have * and ? signs, they means
"one character" and "any number of characters" respectively. For
example, if you want to index all HTTP sites in your local subnet,
use this command:<PRE
CLASS="programlisting"
>Subnet 192.168.*.*</PRE
>
</P
><P
>You may use "NoMatch" optional argument. For
example, if you want to index everything without
<TT
CLASS="literal"
>195.x.x.x</TT
> subnet, use:<PRE
CLASS="programlisting"
>Subnet NoMatch 195.*.*.*</PRE
>
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-difparam"
>Using different parameter for server and it's subsections</A
></H3
><P
>Indexer seeks for "Server" and "Realm" commands
in order of their appearance. Thus if you want to give different
parameters to e.g. whole server and its subsection you should add
subsection line before whole server's. Imagine that you have server
subdirectory which contains news articles. Surely those articles are
to be reindexed more often than the rest of the server. The following
combination may be useful in such cases:</P
><P
> <PRE
CLASS="programlisting"
> # Add subsection
Period 200000
Server http://servername/news/
# Add server
Period 600000
Server http://servername/
</PRE
>
</P
><P
>These commands give different reindexing period
for <TT
CLASS="filename"
>/news/</TT
> subdirectory comparing with the period
of server as a whole. indexer will choose the first "Server" record
for the <TT
CLASS="filename"
>http://servername/news/page1.html</TT
> as far
as it matches and was given first.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-default"
>Default indexer behavior</A
></H3
><P
>The default behavior of indexer is to follow
through links having correspondent Server/Realm command in the
<TT
CLASS="filename"
>indexer.conf</TT
> file. It also jumps between servers
if both of them are present in <TT
CLASS="filename"
>indexer.conf</TT
>
either directly in Server command or indirectly in Realm command. For
example, there are two Server commands:</P
><P
> <PRE
CLASS="programlisting"
> Server http://www/
Server http://web/
</PRE
>
</P
><P
>When indexing
<TT
CLASS="filename"
>http://www/page1.html</TT
> indexer WILL follow the
link <TT
CLASS="filename"
>http://web/page2.html</TT
> if the last one has
been found. Note that these pages are on different servers, but BOTH
of them have correspondent Server record.</P
><P
>If one of the Server command is deleted, indexer
will remove all expired URLs from this server during next
reindexing.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="follow-f"
>Using "indexer -f <filename></A
></H3
><P
>The third scheme is very useful for
<TT
CLASS="literal"
>indexer -i -f url.txt</TT
> running. You may maintain
required servers in the <TT
CLASS="filename"
>url.txt</TT
>. When new URL is
added into <TT
CLASS="filename"
>url.txt</TT
> indexer will index the server
of this URL during next startup. </P
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="aliases"
>Aliases</A
></H2
><P
>mnoGoSearch has an alias support making it possible to
index sites taking information from another location. For example, if
you index local web server, it is possible to take pages directly from
disk without involving your web server in indexing process. Another
example is building of search engine for primary site and using its
mirror while indexing. There are several ways of using aliases.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-conf"
>Alias indexer.conf command
<A
NAME="AEN1179"
></A
></A
></H3
><P
>Format of "Alias" <TT
CLASS="filename"
>indexer.conf</TT
> command:
<PRE
CLASS="programlisting"
> Alias <masterURL> <mirrorURL>
</PRE
>
</P
><P
>E.g. you wish to index
<TT
CLASS="literal"
>http://search.mnogo.ru/</TT
> using nearest German mirror
<TT
CLASS="literal"
>http://www.gstammw.de/mirrors/mnoGoSearch/</TT
>. Add
these lines in your indexer.conf:
<PRE
CLASS="programlisting"
> Server http://search.mnogo.ru/
Alias http://search.mnogo.ru/ http://www.gstammw.de/mirrors/mnoGoSearch/
</PRE
>
</P
><P
> <TT
CLASS="filename"
>search.cgi</TT
> will display
URLs from master site <TT
CLASS="literal"
>http://search.mnogo.ru/</TT
> but
indexer will take corresponding page from mirror site
<TT
CLASS="literal"
>http://www.gstammw.de/mirrors/mnoGoSearch/</TT
>.</P
><P
>Another example. If you want to index everything
in udm.net domain and one of servers, for example
<TT
CLASS="literal"
>http://home.udm.net/</TT
> is stored on local machine in
<TT
CLASS="literal"
>/home/httpd/htdocs/</TT
> directory. These commands will
be useful:
<PRE
CLASS="programlisting"
> Realm http://*.udm.net/
Alias http://home.udm.net/ file:/home/httpd/htdocs/
</PRE
>
</P
><P
>Indexer will take <TT
CLASS="literal"
>home.udm.net</TT
> from local disk and index other sites using HTTP.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="aliases-diff"
>Different aliases for server parts</A
></H3
><P
>Aliases are searched in the order of their
appearance in <TT
CLASS="filename"
>indexer.conf</TT
>. So, you can create
different aliases for server and its parts:
<PRE
CLASS="programlisting"
> # First, create alias for example for /stat/ directory which
# is not under common location:
Alias http://home.udm.net/stat/ file:/usr/local/stat/htdocs/
# Then create alias for the rest of the server:
Alias http://home.udm.net/ file:/usr/local/apache/htdocs/
</PRE
>
</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>if you change the order of these
commands, alias for <TT
CLASS="filename"
>/stat/</TT
> directory will never
be found.</P
></BLOCKQUOTE
></DIV
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-server"
>Using alias in Server command</A
></H3
><P
>You may specify location used by indexer as an optional argument for Server command:
<PRE
CLASS="programlisting"
> Server http://home.udm.net/ file:/home/httpd/htdocs/
</PRE
>
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-realm"
>Using alias in Realm command</A
></H3
><P
>Aliases in Realm command is a very powerful
feature based on regular expressions. The idea of aliases in Realm
command implementation is similar to how PHP
<TT
CLASS="function"
>preg_replace()</TT
> function works. Aliases in Realm
command work only if "regex" match type is used and does not work with
"string" match type.</P
><P
>Use this syntax for Realm aliases:
<PRE
CLASS="programlisting"
> Realm regex <URL_pattern> <alias_pattern>
</PRE
>
</P
><P
>Indexer searches URL for matches to URL_pattern
and builds an URL alias using alias_pattern. alias_pattern may contain
references of the form $n. Where n is a number in the range of
0-9. Every such reference will be replaced by text captured by the
n'th parenthesized pattern. $0 refers to text matched by the whole
pattern. Opening parentheses are counted from left to right (starting
from 1) to obtain the number of the capturing subpattern.</P
><P
>Example: your company hosts several hundreds
users with their domains in the form of
<TT
CLASS="literal"
>www.username.yourname.com</TT
>. Every user's site is
stored on disk in "htdocs" under user's home directory:
<TT
CLASS="literal"
>/home/username/htdocs/</TT
>.</P
><P
>You may write this command into indexer.conf
(note that dot '.' character has a special meaning in regular
expressions and must be escaped with '\' sign when dot is used in
usual meaning):
<PRE
CLASS="programlisting"
> Realm regex (http://www\.)(.*)(\.yourname\.com/)(.*) file:/home/$2/htdocs/$4
</PRE
>
</P
><P
>Imagine indexer process
<TT
CLASS="filename"
>http://www.john.yourname.com/news/index.html</TT
>
page. It will build patterns from $0 to $4:</P
><P
CLASS="literallayout"
><br>
$0 = 'http://www.john.yourname.com/news/index.htm' (whole patter match)<br>
$1 = 'http://www.' subpattern matches '(http://www\.)'<br>
$2 = 'john' subpattern matches '(.*)'<br>
$3 = '.yourname.com/' subpattern matches '(\.yourname\.com/)'<br>
$4 = '/news/index.html' subpattern matches '(.*)'<br>
</P
><P
>Then indexer will compose alias using $2 and $4 patterns:
<PRE
CLASS="programlisting"
> file:/home/john/htdocs/news/index.html
</PRE
>
</P
><P
>and will use the result as document location to fetch it.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-prog"
>AliasProg command
<A
NAME="AEN1231"
></A
></A
></H3
><P
>You may also specify "AliasProg" command for
aliasing purposes. AliasProg is useful for major web hosting companies
which want to index their web space taking documents directly from a
disk without having to involve web server in indexing
process. Documents layout may be very complex to describe it using
alias in Realm command. AliasProg is an external program that can be
called, that takes a URL and returns one string with the appropriate
alias to stdout. Use $1 to pass URL to command line.</P
><P
>For example this AliasProg command uses
'replace' command from MySQL distribution and replaces URL substring
<TT
CLASS="literal"
>http://www.apache.org/</TT
> to
<TT
CLASS="literal"
>file:/usr/local/apache/htdocs/</TT
>:
<PRE
CLASS="programlisting"
> AliasProg "echo $1 | /usr/local/mysql/bin/mysql/replace http://www.apache.org/ file:/usr/local/apache/htdocs/"
</PRE
>
</P
><P
>You may also write your own very complex program to process URLs.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-reverse"
>ReverseAlias command
<A
NAME="AEN1242"
></A
></A
></H3
><P
>The ReverseAlias
<TT
CLASS="filename"
>indexer.conf</TT
> command allows URL mapping before
URL is inserted into database. Unlike Alias command, that triggers
mapping right before a document is downloaded, ReverseAlias command
triggers mapping after the link is found.
<PRE
CLASS="programlisting"
> ReverseAlias http://name2/ http://name2.yourname.com/
Server http://name2.yourname.com/
</PRE
>
</P
><P
>All links with short server name will be mapped
to links with full server name before they are inserted into
database.</P
><P
>One of the possible use is cutting various unnecessary strings like <TT
CLASS="literal"
>PHPSESSION=XXXX</TT
>
</P
><P
>E.g. cutting from URL like
<TT
CLASS="literal"
>http://www/a.php?PHPSESSION=XXX</TT
>, when PHPSESSION is
the only parameter. The question sign is deleted as well:
<PRE
CLASS="programlisting"
> ReverseAlias regex (http://[^?]*)[?]PHPSESSION=[^&]*$ $1$2
</PRE
>
</P
><P
>Cutting from URL like
<TT
CLASS="literal"
>w/a.php?PHPSESSION=xxx&..</TT
>, i.e. when PHPSESSION
is the first parameter, but there are other parameters following
it. The '&' sign after PHPSESSION is deleted as well. Question
mark is not deleted:
<PRE
CLASS="programlisting"
> ReverseAlias regex (http://[^?]*[?])PHPSESSION=[^&]*&(.*) $1$2
</PRE
>
</P
><P
>Cutting from URL like
<TT
CLASS="literal"
>http://www/a.php?a=b&PHPSESSION=xxx</TT
> or
<TT
CLASS="literal"
>http://www/a.php?a=b&PHPSESSION=xxx&c=d</TT
>,
where PHPSESSION is not the first parameter. The '&' sign before
PHPSESSION is deleted:
<PRE
CLASS="programlisting"
> ReverseAlias regex (http://[^&]*)&PHPSESSION=[^&]*(.*) $1$2
</PRE
>
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="alias-search"
>Alias in search.htm search template
<A
NAME="AEN1263"
></A
></A
></H3
><P
>It is also possible to define aliases in search
template (<TT
CLASS="filename"
>search.htm</TT
>). The Alias command in
<TT
CLASS="filename"
>search.htm</TT
> is identical to the one in
<TT
CLASS="filename"
>indexer.conf</TT
>, however it is active during
searching and not indexation.</P
><P
>The syntax of the
<TT
CLASS="filename"
>search.htm</TT
> Alias command is the same as in
<TT
CLASS="filename"
>indexer.conf</TT
>:
<PRE
CLASS="programlisting"
> Alias <find-prefix> <replace-prefix>
</PRE
>
</P
><P
>For example, there is the following command in <TT
CLASS="filename"
>search.htm</TT
>:
<PRE
CLASS="programlisting"
> Alias http://localhost/ http://www.mnogo.ru/
</PRE
>
</P
><P
>Search returned a page with the following URL:
<PRE
CLASS="programlisting"
> http://localhost/news/article10.html
</PRE
>
</P
><P
>As a result, the $(DU) variable will be replace NOT with this URL:
<PRE
CLASS="programlisting"
> http://localhost/news/article10.html
</PRE
>
</P
><P
>but with the following URL (that results in processing with Alias):
<PRE
CLASS="programlisting"
> http://www.mnogo.ru/news/article10.html
</PRE
>
</P
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="srvtable"
>ServerTable
<A
NAME="AEN1285"
></A
></A
></H2
><P
>Since version 3.2.7 <SPAN
CLASS="application"
>mnoGoSearch</SPAN
> has "ServerTable"
<TT
CLASS="filename"
>indexer.conf</TT
> command. This works in SQL version
and has no effect on built-in database.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="srvtable-load"
>Loading servers table</A
></H3
><P
>When "ServerTable srv" is specified, indexer will load
servers information from given "srv" SQL table. Check the structure in
<TT
CLASS="filename"
>create/mysql/create.txt</TT
> file. If there is no
structure example for your database, take it as an example. Please
send us structure for your database at
<TT
CLASS="email"
><<A
HREF="mailto:devel@mnogosearch.org"
>devel@mnogosearch.org</A
>></TT
>!</P
><P
>You may use several "ServerTable" command to load servers information from different tables. </P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="srvtable-structure"
>Server table structure</A
></H3
><P
>Servers table consists of all necessary fields which
describe servers parameters. Field names have correspondent
<TT
CLASS="filename"
>indexer.conf</TT
> commands. For example, "period"
field corresponds "Period" <TT
CLASS="filename"
>indexer.conf</TT
>
command. Default field values are the same with default
<TT
CLASS="filename"
>indexer.conf</TT
> parameters.</P
><P
>"gindex" field corresponds "Index" command. Name is slightly changed to avoid SQL reserved word usage.</P
><P
>"gindex","deletebad","use_robots","delete_no_server","use_clones" are actually boolean parameters. 0 means "no", 1 means "yes".</P
><P
>"follow" field values are 0 for "page", 1 for "path", 2 for "site" and 4 for "world".</P
><DIV
CLASS="note"
><BLOCKQUOTE
CLASS="note"
><P
><B
>Note: </B
>Only those servers are read from the table where "active"
field has 1 value and "parent" field has 0 value. This is useful to allow users to submit new URLs
into servers table and give administrator a possibility to approve
added URLs.</P
></BLOCKQUOTE
></DIV
></DIV
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="flushsrvtable"
>FlushServerTable
<A
NAME="AEN1310"
></A
></A
></H2
><P
> Flush server.active to inactive for all server table records.
Use this command to diactivate all command in servertable before load new from <TT
CLASS="filename"
>indexer.conf</TT
>
or from other servertable.
</P
></DIV
><DIV
CLASS="sect2"
><H2
CLASS="sect2"
><A
NAME="pars"
>External parsers
<A
NAME="AEN1317"
></A
></A
></H2
><P
>mnoGoSearch indexer can use external parsers to index various file types (mime types).</P
><P
>Parser is an executable program which converts one of
the mime types to <TT
CLASS="literal"
>text/plain</TT
> or
<TT
CLASS="literal"
>text/html</TT
>. For example, if you have postscript
files, you can use ps2ascii parser (filter), which reads postscript
file from stdin and produces ascii to stdout.</P
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-sup"
>Supported parser types</A
></H3
><P
>Indexer supports four types of parsers that can:</P
><P
></P
><UL
><LI
><P
>read data from stdin and send result to stdout</P
></LI
><LI
><P
>read data from file and send result to stdout</P
></LI
><LI
><P
>read data from file and send result to file</P
></LI
><LI
><P
>read data from stdin and send result to file</P
></LI
></UL
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-setup"
>Setting up parsers</A
></H3
><P
></P
><OL
TYPE="1"
><LI
><P
>Configure mime types</P
><P
>Configure your web server to
send appropriate "Content-Type" header. For apache, have a look at
<TT
CLASS="filename"
>mime.types</TT
> file, most mime types are already
defined there.</P
><P
> <A
NAME="AEN1343"
></A
>
If you want to index local files
use "AddType" command in <TT
CLASS="filename"
>indexer.conf</TT
> to
associate file name extensions with their mime types. For
example:
<PRE
CLASS="programlisting"
> AddType text/html *.html
</PRE
>
</P
></LI
><LI
><P
> <A
NAME="AEN1350"
></A
>
Add parsers</P
><P
>Add lines with parsers definitions. Lines have the following format with three arguments:
<PRE
CLASS="programlisting"
> Mime <from_mime> <to_mime> <command line>
</PRE
>
</P
><P
>For example, the following line defines parser for man pages:
<PRE
CLASS="programlisting"
> # Use deroff for parsing man pages ( *.man )
Mime application/x-troff-man text/plain deroff
</PRE
>
</P
><P
>This parser will take data from stdin and output result to stdout.</P
><P
>Many parsers can not operate on
stdin and require a file to read from. In this case indexer creates a
temporary file in /tmp and will remove it when parser exits. Use $1
macro in parser command line to substitute file name. For example,
Mime command for "catdoc" MS Word to ASCII converters may look like
this:
<PRE
CLASS="programlisting"
> Mime application/msword text/plain "/usr/bin/catdoc -a $1"
</PRE
>
</P
><P
>If your parser writes result
into output file, use $2 macro. indexer will replace $2 by temporary
file name, start parser, read result from this temporary file then
remove it. For example:
<PRE
CLASS="programlisting"
> Mime application/msword text/plain "/usr/bin/catdoc -a $1 >$2"
</PRE
>
</P
><P
>The parser above will read data
from first temporary file and write result to second one. Both
temporary files will be removed when parser exists. Note that result
of usage of this parser will be absolutely the same with the previous
one, but they use different execution mode: file->stdout and
file->file correspondingly.</P
></LI
></OL
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="ParserTimeOut"
>Avoid indexer hang on parser execution</A
></H3
><P
> <A
NAME="AEN1366"
></A
>
To avoid a indexer hang on parser execution you may specify in your <TT
CLASS="filename"
>indexer.conf</TT
>
amount of time in seconds for parser execution by <TT
CLASS="literal"
>ParserTimeOut</TT
> commnad. For example:
<PRE
CLASS="programlisting"
> ParserTimeOut 600
</PRE
>
</P
><P
> Default value is 300 seconds, i.e. 5 minutes.
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-pipes"
>Pipes in parser's command line</A
></H3
><P
>You can use pipes in parser's command line. For
example, these lines will be useful to index gzipped man pages from
local disk:
<PRE
CLASS="programlisting"
> AddType application/x-gzipped-man *.1.gz *.2.gz *.3.gz *.4.gz
Mime application/x-gzipped-man text/plain "zcat | deroff"
</PRE
>
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-char"
>Charsets and parsers
<A
NAME="AEN1379"
></A
></A
></H3
><P
>Some parsers can produce output in other charset
than given in LocalCharset command. Specify charset to make indexer
convert parser's output to proper one. For example, if your catdoc is
configured to produce output in windows-1251 charset but LocalCharset
is koi8-r, use this command for parsing MS Word documents:
<PRE
CLASS="programlisting"
> Mime application/msword "text/plain; charset=windows-1251" "catdoc -a $1"
</PRE
>
</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-udmurl"
>UDM_URL variable</A
></H3
><P
>When executing a parser indexer creates UDM_URL
environment variable with an URL being processed as a value. You can
use this variable in parser scripts.</P
></DIV
><DIV
CLASS="sect3"
><H3
CLASS="sect3"
><A
NAME="pars-links"
>Some third-party parsers
<A
NAME="AEN1389"
></A
></A
></H3
><P
></P
><UL
><LI
><P
>RPM parser by Mario Lang <TT
CLASS="email"
><<A
HREF="mailto:lang@zid.tu-graz.ac.at"
>lang@zid.tu-graz.ac.at</A
>></TT
>
<PRE
CLASS="programlisting"
> /usr/local/bin/rpminfo:
#!/bin/bash
/usr/bin/rpm -q --queryformat="<html><head><title>RPM: %{NAME} %{VERSION}-%{RELEASE}(%{GROUP})</title><meta name=\"description\" content=\"%{SUMMARY}\"></head><body>%{DESCRIPTION}\n</body></html>" -p $1
indexer.conf:
Mime application/x-rpm text/html "/usr/local/bin/rpminfo $1"
It renders to such nice RPM information:
3. RPM: mysql 3.20.32a-3 (Applications/Databases) [4]
Mysql is a SQL (Structured Query Language) database server.
Mysql was written by Michael (monty) Widenius. See the CREDITS
file in the distribution for more credits for mysql and related
things....
(application/x-rpm) 2088855 bytes
</PRE
>
</P
></LI
><LI
><P
>catdoc MS Word to text converter</P
><P
> <A
HREF="http://freshmeat.net/redir/catdoc/1055/url_homepage/"
TARGET="_top"
>Home page</A
>, also listed on <A
HREF="http://freshmeat.net/"
TARGET="_top"
>Freshmeat</A
>
<PRE
CLASS="programlisting"
> indexer.conf:
Mime application/msword text/plain "catdoc $1"
</PRE
>
</P
></LI
><LI
><P
>xls2csv MS Excel to text converter</P
><P
>It is supplied with catdoc.
<PRE
CLASS="programlisting"
> indexer.conf:
Mime application/vnd.ms-excel text/plain "xls2csv $1"
</PRE
>
</P
></LI
><LI
><P
>pdftotext Adobe PDF converter</P
><P
>Supplied with xpdf project.</P
><P
> <A
HREF="http://freshmeat.net/redir/xpdf/12080/url_homepage/"
TARGET="_top"
>Homepage</A
>, also listed on <A
HREF="http://freshmeat.net/"
TARGET="_top"
>Freshmeat</A
>
<PRE
CLASS="programlisting"
> indexer.conf:
Mime application/pdf text/plain "pdftotext $1 -"
</PRE
>
</P
></LI
><LI
><P
>unrtf RTF to html converter</P
><P
> <A
HREF="ftp://ftp.gnu.org/pub/gnu/unrtf/"
TARGET="_top"
>Homepage</A
>
<PRE
CLASS="programlisting"
> indexer.conf:
Mime text/rtf* text/html "/usr/local/mnogosearch/sbin/unrtf --html $1
Mime application/rtf text/html "/usr/local/mnogosearch/sbin/unrtf --html $1
</PRE
>
</P
></LI
></UL
><P
>Please feel free to contribute your scripts and
parsers configuration to
<TT
CLASS="email"
><<A
HREF="mailto:general@mnogosearch.org"
>general@mnogosearch.org</A
>></TT
>.</P
></DIV
></DIV
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="msearch-content-enc.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="msearch-extended-indexing.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>Content-Encoding support
<A
NAME="AEN927"
></A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="msearch-indexing.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>Extended indexing features</TD
></TR
></TABLE
></DIV
></BODY
></HTML
>
