<HTML ><HEAD ><TITLE >Cache mode storage</TITLE ><META NAME="GENERATOR" CONTENT="Modular DocBook HTML Stylesheet Version 1.73 "><LINK REL="HOME" TITLE="mnoGoSearch 3.2 reference manual" HREF="index.html"><LINK REL="UP" TITLE="Storing mnoGoSearch data " HREF="msearch-howstore.html"><LINK REL="PREVIOUS" TITLE="Storing mnoGoSearch data " HREF="msearch-howstore.html"><LINK REL="NEXT" TITLE="mnoGoSearch performance issues " HREF="msearch-perf.html"><LINK REL="STYLESHEET" TYPE="text/css" HREF="mnogo.css"><META NAME="Description" CONTENT="mnoGoSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META NAME="Keywords" CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, mnoGoSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD ><BODY CLASS="sect1" BGCOLOR="#EEEEEE" TEXT="#000000" LINK="#000080" VLINK="#800080" ALINK="#FF0000" ><DIV CLASS="NAVHEADER" ><TABLE SUMMARY="Header navigation table" WIDTH="100%" BORDER="0" CELLPADDING="0" CELLSPACING="0" ><TR ><TH COLSPAN="3" ALIGN="center" >mnoGoSearch 3.2 reference manual: Full-featured search engine software</TH ></TR ><TR ><TD WIDTH="10%" ALIGN="left" VALIGN="bottom" ><A HREF="msearch-howstore.html" ACCESSKEY="P" >Prev</A ></TD ><TD WIDTH="80%" ALIGN="center" VALIGN="bottom" >Chapter 5. Storing mnoGoSearch data</TD ><TD WIDTH="10%" ALIGN="right" VALIGN="bottom" ><A HREF="msearch-perf.html" ACCESSKEY="N" >Next</A ></TD ></TR ></TABLE ><HR ALIGN="LEFT" WIDTH="100%"></DIV ><DIV CLASS="sect1" ><H1 CLASS="sect1" ><A NAME="cachemode" >Cache mode storage</A ></H1 ><A NAME="AEN2011" ></A ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachemode-intro" >Introduction</A ></H2 ><P >Beginning from version 3.1.5 mnoGoSearch supports new words "cache" storage mode able to index and search quickly through several millions of documents.</P ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachemode-str" >Cache mode word indexes structure</A ></H2 ><P >The main idea of cache storage mode is that word index is stored on disk rather than SQL database. URL information (table "url") however is kept in SQL database. Word index is divided into 8192 files using 32 bit word_id built with CRC32 of the word. Index is located in files under <TT CLASS="filename" >/var/tree</TT > directory of mnoGoSearch installation. </P ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachemode-tools" >Cache mode tools</A ></H2 ><P >There are tree additional programs <TT CLASS="filename" >cached</TT >, <TT CLASS="filename" >splitter</TT > and <TT CLASS="filename" >mkind</TT > used in "cache mode" indexing.</P ><P > <TT CLASS="filename" >cached</TT > is a TCP daemon which collects word information from indexers and stores it on your hard disk. It can operate in two modes, as old <TT CLASS="filename" >cachelogd</TT > daemon to logs data only, and in new mode, when <TT CLASS="filename" >cachelogd</TT > and <TT CLASS="filename" >splitter</TT > functionality are combined.</P ><P > <TT CLASS="filename" >splitter</TT > is a program to create fast word indexes using data collected by <TT CLASS="filename" >cached</TT >. Those indexes are used later in search process.</P ><P ><TT CLASS="filename" >mkind</TT > is a tool to create search limits by tags, category, etc. </P ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachemode-start" >Starting cache mode</A ></H2 ><P >To start "cache mode" follow these steps:</P ><P ></P ><OL TYPE="1" ><LI ><P >Start <TT CLASS="filename" >cached</TT > server:</P ><P > <TT CLASS="userinput" ><B >cd /usr/local/mnogosearch/sbin </B ></TT ></P ><P ><TT CLASS="userinput" ><B >./cached & 2>cached.out</B ></TT > </P ><P >It will write some debug information into <TT CLASS="filename" >cached.out</TT > file. <TT CLASS="filename" >cached</TT > also creates a <TT CLASS="filename" >cached.pid</TT > file in /var directory of base mnoGoSearch installation.</P ><P ><TT CLASS="filename" >cached</TT > listens to TCP connections and can accept several indexers from different machines. Theoretical number of indexers connections is equal to 128. In old mode <TT CLASS="filename" >cached</TT > stores information sent by indexers in <TT CLASS="filename" >/var/splitter/</TT > directory of mnoGoSearch installation. In new mode it stores in <TT CLASS="filename" >/var/tree/</TT > directory.</P ><P >By default, <TT CLASS="filename" >cached</TT > starts in new mode. To run it in old mode, i.e. logs only mode, run it with -l switch:</P ><P ><TT CLASS="userinput" ><B >cached -l</B ></TT ></P ><P >Or by specify <A NAME="AEN2062" ></A > <B CLASS="command" >LogsOnly yes</B > command in your <TT CLASS="filename" >indexer.conf</TT >.</P ><P >You can specify port for <TT CLASS="filename" >cached</TT > to use without recompiling. In order to do that, please run </P ><P > <TT CLASS="userinput" ><B > ./cached -p8000 </B ></TT > </P ><P >where <TT CLASS="literal" >8000</TT > is the port number you choose.</P ><P >You can as well specify a directory to store data (it is <TT CLASS="literal" >/var</TT > directory by default) with this command:</P ><P > <TT CLASS="userinput" ><B > ./cached -w /path/to/var/dir </B ></TT > </P ></LI ><LI ><P >Configure your indexer.conf as usual and add these two lines: <PRE CLASS="programlisting" > DBMode cache LogdAddr localhost:7000 </PRE > </P ><P > <A NAME="AEN2081" ></A > <B CLASS="command" >LogdAddr</B > command is used to specify <TT CLASS="filename" >cached</TT > location. Each indexer will connect to <TT CLASS="filename" >cached</TT > on given address at startup.</P ></LI ><LI ><P >Run indexers. Several indexers can be executed simultaneously. Note that you may install indexers on different machines and then execute them with the same <TT CLASS="filename" >cached</TT > server. This distributed system allows making indexing faster. </P ></LI ><LI ><P >Creating word index. This stage is no needs, if <TT CLASS="filename" >cached</TT > runs in new, i.e. combined, mode. When some information is gathered by indexers and collected in <TT CLASS="filename" >/var/splitter/</TT > directory by <TT CLASS="filename" >cached</TT > it is possible to create fast word indexes. <TT CLASS="filename" >splitter</TT > program is responsible for this. It is installed in <TT CLASS="filename" >/sbin</TT > directory. Note that indexes can be created anytime without interrupting current indexing process.</P ><P >Indexes are to be created in the following two steps:</P ><P ></P ><OL TYPE="i" ><LI ><P >Sending -HUP signal to <TT CLASS="filename" >cached</TT >. <TT CLASS="filename" >cached</TT > will flush all buffers to logs files on hard disk. You can use <TT CLASS="filename" >cached.pid</TT > file to do this:</P ><P > <TT CLASS="userinput" ><B > kill -HUP `cat /usr/local/mnogosearch/var/cached.pid` </B ></TT > </P ></LI ><LI ><P >Building word index. Run <TT CLASS="filename" >splitter</TT > without any arguments:</P ><P > <TT CLASS="userinput" ><B > /usr/local/mnogosearch/sbin/splitter </B ></TT > </P ><P >It will take sequentially all 4096 prepared files in <TT CLASS="filename" >/var/splitter/</TT > directory and use them to build fast word index. Processed logs in <TT CLASS="filename" >/var/splitter/</TT > directory are truncated after this operation.</P ></LI ></OL ></LI ></OL ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachelog-sevspl" >Optional usage of several splitters</A ></H2 ><P >splitter has two command line arguments: <TT CLASS="literal" >-f [first file] -t [second file]</TT > which allows limiting used files range. If no parameters are specified splitter distributes all 4096 prepared files. You can limit files range using -f and -t keys specifying parameters in HEX notation. For example, <TT CLASS="literal" >splitter -f 000 -t A00</TT > will create word indexes using files in the range from 000 to A00. These keys allow using several splitters at the same time. It usually gives more quick indexes building. For example, this shell script starts four splitters in background:</P ><PRE CLASS="programlisting" > #!/bin/sh splitter -f 000 -t 3f0 & splitter -f 400 -t 7f0 & splitter -f 800 -t bf0 & splitter -f c00 -t ff0 & </PRE ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachelog-runspl" >Using run-splitter script</A ></H2 ><P >There is a <TT CLASS="filename" >run-splitter</TT > script in <TT CLASS="filename" >/sbin</TT > directory of mnoGoSearch installation. It helps to execute subsequently all three indexes building steps.</P ><P >"run-splitter" has these two command line parameters:</P ><P > <TT CLASS="userinput" ><B > run-splitter --hup --split </B ></TT > </P ><P >or a short version:</P ><P > <TT CLASS="userinput" ><B > run-splitter -k -s </B ></TT > </P ><P >Each parameter activates corresponding indexes building step. <TT CLASS="filename" >run-splitter</TT > executes all three steps of index building in proper order:</P ><P ></P ><OL TYPE="1" ><LI ><P >Sending -HUP signal to cached. <TT CLASS="literal" >--hup</TT > (or <TT CLASS="literal" >-k</TT >) run-splitter arguments are responsible for this.</P ></LI ><LI ><P >Running splitter. Keys <TT CLASS="literal" >--split</TT > (or <TT CLASS="literal" >-s</TT >).</P ></LI ></OL ><P >In most cases just run "run-splitter" script with all <TT CLASS="literal" >-k -s</TT > arguments. Separate usage of those three flags which correspond to three steps of indexes building is rarely required. </P ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="cachelog-search" >Doing search</A ></H2 ><P >To start using search.cgi in the "cache mode" edit as usually your <TT CLASS="filename" >search.htm</TT > template and add this line: <TT CLASS="literal" >DBMode cache</TT > </P ></DIV ><DIV CLASS="sect2" ><H2 CLASS="sect2" ><A NAME="limits" >Using search limits</A ></H2 ><P >To use search limits in cache mode, you should add appropriate <TT CLASS="literal" >Limit</TT > comand(s) to your <TT CLASS="filename" >indexer.conf</TT > and to <TT CLASS="filename" >search.htm</TT > or <TT CLASS="filename" >searchd.conf</TT > (if <TT CLASS="literal" >searchd</TT > is used). </P ><P ><A NAME="AEN2158" ></A > To use, for example, search limit by tag, by category and by site, add follow lines to <TT CLASS="filename" >search.htm</TT > or to <TT CLASS="filename" >searchd.conf</TT >, if <TT CLASS="literal" >searchd</TT > is used. </P ><PRE CLASS="programlisting" > Limit t:tag Limit c:catategory Limit site:siteid </PRE ><P > where <TT CLASS="literal" >t</TT > - name of CGI parameter (&t=) for this constraint, <TT CLASS="literal" >tag</TT > - type of constraint. </P ><P >Instead of tag/category/siteid in example above you can use any of values from table below: <DIV CLASS="table" ><A NAME="AEN2169" ></A ><P ><B >Table 5-1. Cache limit types</B ></P ><TABLE BORDER="1" CLASS="CALSTABLE" ><TBODY ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >category</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Category limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >tag</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Tag limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >time</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Time limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >hostname</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Hostname (url) limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >language</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Language limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >content</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >Content-Type limit.</TD ></TR ><TR ><TD ALIGN="LEFT" VALIGN="MIDDLE" >siteid</TD ><TD ALIGN="LEFT" VALIGN="MIDDLE" >url.site_id limit.</TD ></TR ></TBODY ></TABLE ></DIV > </P ></DIV ></DIV ><DIV CLASS="NAVFOOTER" ><HR ALIGN="LEFT" WIDTH="100%"><TABLE SUMMARY="Footer navigation table" WIDTH="100%" BORDER="0" CELLPADDING="0" CELLSPACING="0" ><TR ><TD WIDTH="33%" ALIGN="left" VALIGN="top" ><A HREF="msearch-howstore.html" ACCESSKEY="P" >Prev</A ></TD ><TD WIDTH="34%" ALIGN="center" VALIGN="top" ><A HREF="index.html" ACCESSKEY="H" >Home</A ></TD ><TD WIDTH="33%" ALIGN="right" VALIGN="top" ><A HREF="msearch-perf.html" ACCESSKEY="N" >Next</A ></TD ></TR ><TR ><TD WIDTH="33%" ALIGN="left" VALIGN="top" >Storing mnoGoSearch data</TD ><TD WIDTH="34%" ALIGN="center" VALIGN="top" ><A HREF="msearch-howstore.html" ACCESSKEY="U" >Up</A ></TD ><TD WIDTH="33%" ALIGN="right" VALIGN="top" >mnoGoSearch performance issues <A NAME="AEN2196" ></A ></TD ></TR ></TABLE ></DIV ></BODY ></HTML >
