<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Robots exclusion application for Django — django-robots 3.1.0 documentation</title>
<script type="text/javascript" src="_static/js/modernizr.min.js"></script>
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="History" href="history.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="#" class="icon icon-home"> django-robots
</a>
<div class="version">
3.1
</div>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<ul>
<li class="toctree-l1"><a class="reference internal" href="history.html">History</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="#">django-robots</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="#">Docs</a> »</li>
<li>Robots exclusion application for Django</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/index.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<div class="section" id="robots-exclusion-application-for-django">
<h1>Robots exclusion application for Django<a class="headerlink" href="#robots-exclusion-application-for-django" title="Permalink to this headline">¶</a></h1>
</div>
<div class="section" id="django-robots">
<h1>Django Robots<a class="headerlink" href="#django-robots" title="Permalink to this headline">¶</a></h1>
<a class="reference external image-reference" href="https://jazzband.co/"><img alt="Jazzband" src="https://jazzband.co/static/img/badge.svg" /></a>
<a class="reference external image-reference" href="https://travis-ci.org/jazzband/django-robots"><img alt="https://travis-ci.org/jazzband/django-robots.svg?branch=master" src="https://travis-ci.org/jazzband/django-robots.svg?branch=master" /></a>
<a class="reference external image-reference" href="https://codecov.io/gh/jazzband/django-robots"><img alt="https://codecov.io/gh/jazzband/django-robots/branch/master/graph/badge.svg" src="https://codecov.io/gh/jazzband/django-robots/branch/master/graph/badge.svg" /></a>
<a class="reference external image-reference" href="https://pypi.python.org/pypi/django-robots"><img alt="https://img.shields.io/pypi/v/django-robots.svg" src="https://img.shields.io/pypi/v/django-robots.svg" /></a>
<p>This is a basic Django application to manage robots.txt files following the
<a class="reference external" href="http://en.wikipedia.org/wiki/Robots_exclusion_standard">robots exclusion protocol</a>, complementing the <a class="reference external" href="http://www.djangoproject.com/">Django</a> <a class="reference external" href="http://docs.djangoproject.com/en/dev/ref/contrib/sitemaps/">Sitemap contrib app</a>.</p>
<p>For installation instructions, see the documentation <a class="reference external" href="https://django-robots.readthedocs.io/en/latest/#installation">install section</a>;
for instructions on how to use this application, and on
what it provides, see the file “overview.txt” in the “docs/”
directory or on ReadTheDocs: <a class="reference external" href="https://django-robots.readthedocs.io/">https://django-robots.readthedocs.io/</a></p>
<div class="section" id="supported-django-versions">
<h2>Supported Django versions<a class="headerlink" href="#supported-django-versions" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>Django 2.0</li>
<li>Django 1.11</li>
</ul>
<p>For older Django versions (1.6-1.10) use <code class="docutils literal notranslate"><span class="pre">django-robots==3.0</span></code>.</p>
</div>
<div class="section" id="supported-python-version">
<h2>Supported Python version<a class="headerlink" href="#supported-python-version" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>Python 2.7 (Django 1.11 only)</li>
<li>Python 3.4, 3.5, 3.6</li>
</ul>
<p>Contents:</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="history.html">History</a></li>
</ul>
</div>
<div class="section" id="installation">
<h3>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h3>
<p>Use your favorite Python installer to install it from PyPI:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">pip</span> <span class="n">install</span> <span class="n">django</span><span class="o">-</span><span class="n">robots</span>
</pre></div>
</div>
<p>Or get the source from the application site at:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">github</span><span class="o">.</span><span class="n">com</span><span class="o">/</span><span class="n">jazzband</span><span class="o">/</span><span class="n">django</span><span class="o">-</span><span class="n">robots</span><span class="o">/</span>
</pre></div>
</div>
<p>To install the sitemap app, then follow these steps:</p>
<ol class="arabic simple">
<li>Add <code class="docutils literal notranslate"><span class="pre">'robots'</span></code> to your <a class="reference external" href="http://docs.djangoproject.com/en/dev/ref/settings/#installed-apps">INSTALLED_APPS</a> setting.</li>
<li>Make sure <code class="docutils literal notranslate"><span class="pre">'django.template.loaders.app_directories.Loader'</span></code>
is in your TEMPLATES setting. It’s in there by default, so
you’ll only need to change this if you’ve changed that setting.</li>
<li>Make sure you’ve installed the <a class="reference external" href="http://docs.djangoproject.com/en/dev/ref/contrib/sites/">sites framework</a>.</li>
<li>Run the <code class="docutils literal notranslate"><span class="pre">migrate</span></code> management command</li>
</ol>
</div>
</div>
<div class="section" id="sitemaps">
<h2>Sitemaps<a class="headerlink" href="#sitemaps" title="Permalink to this headline">¶</a></h2>
<p>By default a <code class="docutils literal notranslate"><span class="pre">Sitemap</span></code> statement is automatically added to the resulting
robots.txt by reverse matching the URL of the installed <a class="reference external" href="http://docs.djangoproject.com/en/dev/ref/contrib/sitemaps/">Sitemap contrib app</a>.
This is especially useful if you allow every robot to access your whole site,
since it then gets URLs explicitly instead of searching every link.</p>
<p>To change the default behaviour to omit the inclusion of a sitemap link,
change the <code class="docutils literal notranslate"><span class="pre">ROBOTS_USE_SITEMAP</span></code> setting in your Django settings file to:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_USE_SITEMAP</span> <span class="o">=</span> <span class="kc">False</span>
</pre></div>
</div>
<p>In case you want to use specific sitemap URLs instead of the one that is
automatically discovered, change the <code class="docutils literal notranslate"><span class="pre">ROBOTS_SITEMAP_URLS</span></code> setting to:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_SITEMAP_URLS</span> <span class="o">=</span> <span class="p">[</span>
<span class="s1">'http://www.example.com/sitemap.xml'</span><span class="p">,</span>
<span class="p">]</span>
</pre></div>
</div>
<p>If the sitemap is wrapped in a decorator, dotted path reverse to discover
the sitemap URL does not work.
To overcome this, provide a name to the sitemap instance in <code class="docutils literal notranslate"><span class="pre">urls.py</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">urlpatterns</span> <span class="o">=</span> <span class="p">[</span>
<span class="o">...</span>
<span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^sitemap.xml$'</span><span class="p">,</span> <span class="n">cache_page</span><span class="p">(</span><span class="mi">60</span><span class="p">)(</span><span class="n">sitemap_view</span><span class="p">),</span> <span class="p">{</span><span class="s1">'sitemaps'</span><span class="p">:</span> <span class="p">[</span><span class="o">...</span><span class="p">]},</span> <span class="n">name</span><span class="o">=</span><span class="s1">'cached-sitemap'</span><span class="p">),</span>
<span class="o">...</span>
<span class="p">]</span>
</pre></div>
</div>
<p>and inform django-robots about the view name by adding the followin setting:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_SITEMAP_VIEW_NAME</span> <span class="o">=</span> <span class="s1">'cached-sitemap'</span>
</pre></div>
</div>
<p>Use <code class="docutils literal notranslate"><span class="pre">ROBOTS_SITEMAP_VIEW_NAME</span></code> also if you use custom sitemap views (e.g.: wagtail custom sitemaps).</p>
<div class="section" id="initialization">
<h3>Initialization<a class="headerlink" href="#initialization" title="Permalink to this headline">¶</a></h3>
<p>To activate robots.txt generation on your Django site, add this line to your
<a class="reference external" href="http://docs.djangoproject.com/en/dev/topics/http/urls/">URLconf</a>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">url</span><span class="p">(</span><span class="sa">r</span><span class="s1">'^robots\.txt'</span><span class="p">,</span> <span class="n">include</span><span class="p">(</span><span class="s1">'robots.urls'</span><span class="p">)),</span>
</pre></div>
</div>
<p>This tells Django to build a robots.txt when a robot accesses <code class="docutils literal notranslate"><span class="pre">/robots.txt</span></code>.
Then, please sync your database to create the necessary tables and create
<code class="docutils literal notranslate"><span class="pre">Rule</span></code> objects in the admin interface or via the shell.</p>
</div>
<div class="section" id="rules">
<h3>Rules<a class="headerlink" href="#rules" title="Permalink to this headline">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">Rule</span></code> - defines an abstract rule which is used to respond to crawling web
robots, using the <a class="reference external" href="http://en.wikipedia.org/wiki/Robots_exclusion_standard">robots exclusion protocol</a>, a.k.a. robots.txt.</p>
<p>You can link multiple URL pattern to allows or disallows the robot identified
by its user agent to access the given URLs.</p>
<p>The crawl delay field is supported by some search engines and defines the
delay between successive crawler accesses in seconds. If the crawler rate is a
problem for your server, you can set the delay up to 5 or 10 or a comfortable
value for your server, but it’s suggested to start with small values (0.5-1),
and increase as needed to an acceptable value for your server. Larger delay
values add more delay between successive crawl accesses and decrease the
maximum crawl rate to your web server.</p>
<p>The <a class="reference external" href="http://docs.djangoproject.com/en/dev/ref/contrib/sites/">sites framework</a> is used to enable multiple robots.txt per Django instance.
If no rule exists it automatically allows every web robot access to every URL.</p>
<p>Please have a look at the <a class="reference external" href="http://www.robotstxt.org/db.html">database of web robots</a> for a full list of
existing web robots user agent strings.</p>
</div>
<div class="section" id="host-directive">
<h3>Host directive<a class="headerlink" href="#host-directive" title="Permalink to this headline">¶</a></h3>
<p>By default a <code class="docutils literal notranslate"><span class="pre">Host</span></code> statement is automatically added to the resulting
robots.txt to avoid mirrors and select the main website properly.</p>
<p>To change the default behaviour to omit the inclusion of host directive,
change the <code class="docutils literal notranslate"><span class="pre">ROBOTS_USE_HOST</span></code> setting in your Django settings file to:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_USE_HOST</span> <span class="o">=</span> <span class="kc">False</span>
</pre></div>
</div>
<p>if you want to prefix the domain with the current request protocol
(<strong>http</strong> or <strong>https</strong> as in <code class="docutils literal notranslate"><span class="pre">Host:</span> <span class="pre">https://www.mysite.com</span></code>) add this setting:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_USE_SCHEME_IN_HOST</span> <span class="o">=</span> <span class="kc">True</span>
</pre></div>
</div>
</div>
<div class="section" id="urls">
<h3>URLs<a class="headerlink" href="#urls" title="Permalink to this headline">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">Url</span></code> - defines a case-sensitive and exact URL pattern which is used to
allow or disallow the access for web robots. Case-sensitive.</p>
<p>A missing trailing slash does also match files which start with the name of
the given pattern, e.g., <code class="docutils literal notranslate"><span class="pre">'/admin'</span></code> matches <code class="docutils literal notranslate"><span class="pre">/admin.html</span></code> too.</p>
<p>Some major search engines allow an asterisk (<code class="docutils literal notranslate"><span class="pre">*</span></code>) as a wildcard to match any
sequence of characters and a dollar sign (<code class="docutils literal notranslate"><span class="pre">$</span></code>) to match the end of the URL,
e.g., <code class="docutils literal notranslate"><span class="pre">'/*.jpg$'</span></code> can be used to match all jpeg files.</p>
</div>
<div class="section" id="caching">
<h3>Caching<a class="headerlink" href="#caching" title="Permalink to this headline">¶</a></h3>
<p>You can optionally cache the generation of the <code class="docutils literal notranslate"><span class="pre">robots.txt</span></code>. Add or change
the <code class="docutils literal notranslate"><span class="pre">ROBOTS_CACHE_TIMEOUT</span></code> setting with a value in seconds in your Django
settings file:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ROBOTS_CACHE_TIMEOUT</span> <span class="o">=</span> <span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">24</span>
</pre></div>
</div>
<p>This tells Django to cache the <code class="docutils literal notranslate"><span class="pre">robots.txt</span></code> for 24 hours (86400 seconds).
The default value is <code class="docutils literal notranslate"><span class="pre">None</span></code> (no caching).</p>
</div>
<div class="section" id="bugs-and-feature-requests">
<h3>Bugs and feature requests<a class="headerlink" href="#bugs-and-feature-requests" title="Permalink to this headline">¶</a></h3>
<p>As always your mileage may vary, so please don’t hesitate to send feature
requests and bug reports:</p>
<blockquote>
<div><a class="reference external" href="https://github.com/jazzband/django-robots/issues">https://github.com/jazzband/django-robots/issues</a></div></blockquote>
</div>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="history.html" class="btn btn-neutral float-right" title="History" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright 2008-2016, Jannis Leidel
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
