<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML
><HEAD
><TITLE
>Robots</TITLE
><META
NAME="GENERATOR"
CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
REL="HOME"
TITLE="mnoGoSearch 3.3.9 reference manual"
HREF="index.html"><LINK
REL="UP"
TITLE="mnoGoSearch command reference"
HREF="msearch-cmdref.html"><LINK
REL="PREVIOUS"
TITLE="ReverseAlias"
HREF="msearch-cmdref-reversealias.html"><LINK
REL="NEXT"
TITLE="SaveSectionSize"
HREF="msearch-cmdref-savesectionsize.html"><LINK
REL="STYLESHEET"
TYPE="text/css"
HREF="mnogo.css"><META
NAME="Description"
CONTENT="mnoGoSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META
NAME="Keywords"
CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, mnoGoSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD
><BODY
CLASS="refentry"
BGCOLOR="#EEEEEE"
TEXT="#000000"
LINK="#000080"
VLINK="#800080"
ALINK="#FF0000"
><!--#include virtual="body-before.html"--><DIV
CLASS="NAVHEADER"
><TABLE
SUMMARY="Header navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TH
COLSPAN="3"
ALIGN="center"
><SPAN
CLASS="application"
>mnoGoSearch</SPAN
> 3.3.9 reference manual: Full-featured search engine software</TH
></TR
><TR
><TD
WIDTH="10%"
ALIGN="left"
VALIGN="bottom"
><A
HREF="msearch-cmdref-reversealias.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="80%"
ALIGN="center"
VALIGN="bottom"
></TD
><TD
WIDTH="10%"
ALIGN="right"
VALIGN="bottom"
><A
HREF="msearch-cmdref-savesectionsize.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
></TABLE
><HR
ALIGN="LEFT"
WIDTH="100%"></DIV
><H1
><A
NAME="cmdref-robots"
></A
>Robots</H1
><DIV
CLASS="refnamediv"
><A
NAME="AEN11948"
></A
><H2
>Name</H2
><B
CLASS="command"
>Robots</B
> -- defines whether to use <TT
CLASS="filename"
>robots.txt</TT
><P
><B
></B
><TT
CLASS="filename"
>indexer.conf</TT
></P
></DIV
><DIV
CLASS="refsynopsisdiv"
><A
NAME="AEN11955"
></A
><H2
>Synopsis</H2
><P
><B
CLASS="command"
>Robots</B
> {yes | no}</P
></DIV
><DIV
CLASS="refsect1"
><A
NAME="AEN11961"
></A
><H2
>Description</H2
><P
><B
CLASS="command"
>Robots</B
> defines whether to use
<TT
CLASS="filename"
>robots.txt</TT
> and
<TT
CLASS="literal"
><META NAME="robots"></TT
> exclusions.
</P
><P
> Setting <B
CLASS="command"
>Robots</B
> to <TT
CLASS="literal"
>no</TT
> can be useful when
running <SPAN
CLASS="application"
>mnoGoSearch</SPAN
> for
<A
HREF="msearch-indexing.html#general-linkval"
>site validation</A
> purposes,
as well as when indexing your own Web site.
</P
><P
> The default value is <TT
CLASS="literal"
>yes</TT
>, for polite crawling purposes.
</P
></DIV
><DIV
CLASS="refsect1"
><A
NAME="AEN11974"
></A
><H2
>Scope</H2
><P
> <B
CLASS="command"
>Robots</B
> can be used multiple times and
affects on all following <B
CLASS="command"
><A
HREF="msearch-cmdref-server.html"
>Server</A
></B
>
and <B
CLASS="command"
><A
HREF="msearch-cmdref-realm.html"
>Realm</A
></B
> commands
until the end of the configuration file, or until the next
<B
CLASS="command"
>Robots</B
> command.
</P
></DIV
><DIV
CLASS="refsect1"
><A
NAME="AEN11983"
></A
><H2
>Examples</H2
><DIV
CLASS="informalexample"
><P
></P
><A
NAME="AEN11985"
></A
><PRE
CLASS="programlisting"
> Robots no
</PRE
><P
></P
></DIV
></DIV
><DIV
CLASS="refsect1"
><A
NAME="AEN11987"
></A
><H2
>See also</H2
><P
> <A
HREF="msearch-cmdref-crawldelay.html"
>CrawlDelay</A
>,
<A
HREF="msearch-cmdref-index.html"
>Index</A
>,
<A
HREF="msearch-cmdref-indexif.html"
>IndexIf</A
>,
<A
HREF="msearch-cmdref-noindexif.html"
>NoIndexIf</A
>.
</P
></DIV
><DIV
CLASS="NAVFOOTER"
><HR
ALIGN="LEFT"
WIDTH="100%"><TABLE
SUMMARY="Footer navigation table"
WIDTH="100%"
BORDER="0"
CELLPADDING="0"
CELLSPACING="0"
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
><A
HREF="msearch-cmdref-reversealias.html"
ACCESSKEY="P"
>Prev</A
></TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="index.html"
ACCESSKEY="H"
>Home</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
><A
HREF="msearch-cmdref-savesectionsize.html"
ACCESSKEY="N"
>Next</A
></TD
></TR
><TR
><TD
WIDTH="33%"
ALIGN="left"
VALIGN="top"
>ReverseAlias</TD
><TD
WIDTH="34%"
ALIGN="center"
VALIGN="top"
><A
HREF="msearch-cmdref.html"
ACCESSKEY="U"
>Up</A
></TD
><TD
WIDTH="33%"
ALIGN="right"
VALIGN="top"
>SaveSectionSize</TD
></TR
></TABLE
></DIV
><!--#include virtual="body-after.html"--></BODY
></HTML
>
