<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML ><HEAD ><TITLE >Robots</TITLE ><META NAME="GENERATOR" CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK REL="HOME" TITLE="mnoGoSearch 3.3.9 reference manual" HREF="index.html"><LINK REL="UP" TITLE="mnoGoSearch command reference" HREF="msearch-cmdref.html"><LINK REL="PREVIOUS" TITLE="ReverseAlias" HREF="msearch-cmdref-reversealias.html"><LINK REL="NEXT" TITLE="SaveSectionSize" HREF="msearch-cmdref-savesectionsize.html"><LINK REL="STYLESHEET" TYPE="text/css" HREF="mnogo.css"><META NAME="Description" CONTENT="mnoGoSearch - Full Featured Web site Open Source Search Engine Software over the Internet and Intranet Web Sites Based on SQL Database. It is a Free search software covered by GNU license."><META NAME="Keywords" CONTENT="shareware, freeware, download, internet, unix, utilities, search engine, text retrieval, knowledge retrieval, text search, information retrieval, database search, mining, intranet, webserver, index, spider, filesearch, meta, free, open source, full-text, udmsearch, website, find, opensource, search, searching, software, udmsearch, engine, indexing, system, web, ftp, http, cgi, php, SQL, MySQL, database, php3, FreeBSD, Linux, Unix, mnoGoSearch, MacOS X, Mac OS X, Windows, 2000, NT, 95, 98, GNU, GPL, url, grabbing"></HEAD ><BODY CLASS="refentry" BGCOLOR="#EEEEEE" TEXT="#000000" LINK="#000080" VLINK="#800080" ALINK="#FF0000" ><!--#include virtual="body-before.html"--><DIV CLASS="NAVHEADER" ><TABLE SUMMARY="Header navigation table" WIDTH="100%" BORDER="0" CELLPADDING="0" CELLSPACING="0" ><TR ><TH COLSPAN="3" ALIGN="center" ><SPAN CLASS="application" >mnoGoSearch</SPAN > 3.3.9 reference manual: Full-featured search engine software</TH ></TR ><TR ><TD WIDTH="10%" ALIGN="left" VALIGN="bottom" ><A HREF="msearch-cmdref-reversealias.html" ACCESSKEY="P" >Prev</A ></TD ><TD WIDTH="80%" ALIGN="center" VALIGN="bottom" ></TD ><TD WIDTH="10%" ALIGN="right" VALIGN="bottom" ><A HREF="msearch-cmdref-savesectionsize.html" ACCESSKEY="N" >Next</A ></TD ></TR ></TABLE ><HR ALIGN="LEFT" WIDTH="100%"></DIV ><H1 ><A NAME="cmdref-robots" ></A >Robots</H1 ><DIV CLASS="refnamediv" ><A NAME="AEN11948" ></A ><H2 >Name</H2 ><B CLASS="command" >Robots</B > -- defines whether to use <TT CLASS="filename" >robots.txt</TT ><P ><B ></B ><TT CLASS="filename" >indexer.conf</TT ></P ></DIV ><DIV CLASS="refsynopsisdiv" ><A NAME="AEN11955" ></A ><H2 >Synopsis</H2 ><P ><B CLASS="command" >Robots</B > {yes | no}</P ></DIV ><DIV CLASS="refsect1" ><A NAME="AEN11961" ></A ><H2 >Description</H2 ><P ><B CLASS="command" >Robots</B > defines whether to use <TT CLASS="filename" >robots.txt</TT > and <TT CLASS="literal" ><META NAME="robots"></TT > exclusions. </P ><P > Setting <B CLASS="command" >Robots</B > to <TT CLASS="literal" >no</TT > can be useful when running <SPAN CLASS="application" >mnoGoSearch</SPAN > for <A HREF="msearch-indexing.html#general-linkval" >site validation</A > purposes, as well as when indexing your own Web site. </P ><P > The default value is <TT CLASS="literal" >yes</TT >, for polite crawling purposes. </P ></DIV ><DIV CLASS="refsect1" ><A NAME="AEN11974" ></A ><H2 >Scope</H2 ><P > <B CLASS="command" >Robots</B > can be used multiple times and affects on all following <B CLASS="command" ><A HREF="msearch-cmdref-server.html" >Server</A ></B > and <B CLASS="command" ><A HREF="msearch-cmdref-realm.html" >Realm</A ></B > commands until the end of the configuration file, or until the next <B CLASS="command" >Robots</B > command. </P ></DIV ><DIV CLASS="refsect1" ><A NAME="AEN11983" ></A ><H2 >Examples</H2 ><DIV CLASS="informalexample" ><P ></P ><A NAME="AEN11985" ></A ><PRE CLASS="programlisting" > Robots no </PRE ><P ></P ></DIV ></DIV ><DIV CLASS="refsect1" ><A NAME="AEN11987" ></A ><H2 >See also</H2 ><P > <A HREF="msearch-cmdref-crawldelay.html" >CrawlDelay</A >, <A HREF="msearch-cmdref-index.html" >Index</A >, <A HREF="msearch-cmdref-indexif.html" >IndexIf</A >, <A HREF="msearch-cmdref-noindexif.html" >NoIndexIf</A >. </P ></DIV ><DIV CLASS="NAVFOOTER" ><HR ALIGN="LEFT" WIDTH="100%"><TABLE SUMMARY="Footer navigation table" WIDTH="100%" BORDER="0" CELLPADDING="0" CELLSPACING="0" ><TR ><TD WIDTH="33%" ALIGN="left" VALIGN="top" ><A HREF="msearch-cmdref-reversealias.html" ACCESSKEY="P" >Prev</A ></TD ><TD WIDTH="34%" ALIGN="center" VALIGN="top" ><A HREF="index.html" ACCESSKEY="H" >Home</A ></TD ><TD WIDTH="33%" ALIGN="right" VALIGN="top" ><A HREF="msearch-cmdref-savesectionsize.html" ACCESSKEY="N" >Next</A ></TD ></TR ><TR ><TD WIDTH="33%" ALIGN="left" VALIGN="top" >ReverseAlias</TD ><TD WIDTH="34%" ALIGN="center" VALIGN="top" ><A HREF="msearch-cmdref.html" ACCESSKEY="U" >Up</A ></TD ><TD WIDTH="33%" ALIGN="right" VALIGN="top" >SaveSectionSize</TD ></TR ></TABLE ></DIV ><!--#include virtual="body-after.html"--></BODY ></HTML >