<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>stem.descriptor.remote — Stem 1.1.0 documentation</title>
<link rel="stylesheet" href="../../../_static/haiku.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/print.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../../../',
VERSION: '1.1.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../../../_static/jquery.js"></script>
<script type="text/javascript" src="../../../_static/underscore.js"></script>
<script type="text/javascript" src="../../../_static/doctools.js"></script>
<script type="text/javascript" src="../../../_static/theme_extras.js"></script>
<link rel="shortcut icon" href="../../../_static/favicon.png"/>
<link rel="top" title="Stem 1.1.0 documentation" href="../../../index.html" />
<link rel="up" title="stem" href="../../stem.html" />
</head>
<body>
<div class="header"><img class="rightlogo" src="../../../_static/logo.png" alt="Logo"/><h1 class="heading"><a href="../../../index.html">
<span>Stem Docs</span></a></h1>
<h2 class="heading"><span>stem.descriptor.remote</span></h2>
</div>
<div class="topnav">
<p>
<ul id="navbar">
<li><a href="../../../index.html">Home</a></li>
<li><a href="../../../tutorials.html">Tutorials</a>
<ul>
<li><a href="../../../tutorials/the_little_relay_that_could.html">Hello World</a></li>
<li><a href="../../../tutorials/to_russia_with_love.html">Client Usage</a></li>
<li><a href="../../../tutorials/tortoise_and_the_hare.html">Event Listening</a></li>
<li><a href="../../../tutorials/mirror_mirror_on_the_wall.html">Tor Descriptors</a></li>
<li><a href="../../../tutorials/east_of_the_sun.html">Utilities</a></li>
<li><a href="../../../tutorials/double_double_toil_and_trouble.html">Examples</a></li>
</ul>
</li>
<li><a href="../../../api.html">API</a>
<ul>
<li><a href="../../../api/control.html">stem.control</a></li>
<li><a href="../../../api/connection.html">stem.connection</a></li>
<li><a href="../../../api/socket.html">stem.socket</a></li>
<li><a href="../../../api/process.html">stem.process</a></li>
<li><a href="../../../api/response.html">stem.response</a></li>
<li><a href="../../../api/exit_policy.html">stem.exit_policy</a></li>
<li><a href="../../../api/version.html">stem.version</a></li>
<li><a href="../../../api.html#descriptors">Descriptors</a></li>
<li><a href="../../../api.html#utilities">Utilities</a></li>
</ul>
</li>
<li><a href="https://trac.torproject.org/projects/tor/wiki/doc/stem">Development</a>
<ul>
<li><a href="../../../faq.html">FAQ</a></li>
<li><a href="../../../change_log.html">Change Log</a></li>
<li><a href="https://trac.torproject.org/projects/tor/wiki/doc/stem/bugs">Bug Tracker</a></li>
<li><a href="../../../download.html">Download</a></li>
</ul>
</li>
</ul>
</p>
</div>
<div class="content">
<h1>Source code for stem.descriptor.remote</h1><div class="highlight"><pre>
<span class="c"># Copyright 2013, Damian Johnson and The Tor Project</span>
<span class="c"># See LICENSE for licensing information</span>
<span class="sd">"""</span>
<span class="sd">Module for remotely retrieving descriptors from directory authorities and</span>
<span class="sd">mirrors. This is most easily done through the</span>
<span class="sd">:class:`~stem.descriptor.remote.DescriptorDownloader` class, which issues</span>
<span class="sd">:class:`~stem.descriptor.remote.Query` instances to get you the descriptor</span>
<span class="sd">content. For example...</span>
<span class="sd">::</span>
<span class="sd"> from stem.descriptor.remote import DescriptorDownloader</span>
<span class="sd"> downloader = DescriptorDownloader(</span>
<span class="sd"> use_mirrors = True,</span>
<span class="sd"> timeout = 10,</span>
<span class="sd"> )</span>
<span class="sd"> query = downloader.get_server_descriptors()</span>
<span class="sd"> print "Exit Relays:"</span>
<span class="sd"> try:</span>
<span class="sd"> for desc in query.run():</span>
<span class="sd"> if desc.exit_policy.is_exiting_allowed():</span>
<span class="sd"> print " %s (%s)" % (desc.nickname, desc.fingerprint)</span>
<span class="sd"> print</span>
<span class="sd"> print "Query took %0.2f seconds" % query.runtime</span>
<span class="sd"> except Exception as exc:</span>
<span class="sd"> print "Unable to retrieve the server descriptors: %s" % exc</span>
<span class="sd">If you don't care about errors then you can also simply iterate over the query</span>
<span class="sd">itself...</span>
<span class="sd">::</span>
<span class="sd"> for desc in downloader.get_server_descriptors():</span>
<span class="sd"> if desc.exit_policy.is_exiting_allowed():</span>
<span class="sd"> print " %s (%s)" % (desc.nickname, desc.fingerprint)</span>
<span class="sd">::</span>
<span class="sd"> get_authorities - Provides tor directory information.</span>
<span class="sd"> DirectoryAuthority - Information about a tor directory authority.</span>
<span class="sd"> Query - Asynchronous request to download tor descriptors</span>
<span class="sd"> |- start - issues the query if it isn't already running</span>
<span class="sd"> +- run - blocks until the request is finished and provides the results</span>
<span class="sd"> DescriptorDownloader - Configurable class for issuing queries</span>
<span class="sd"> |- use_directory_mirrors - use directory mirrors to download future descriptors</span>
<span class="sd"> |- get_server_descriptors - provides present server descriptors</span>
<span class="sd"> |- get_extrainfo_descriptors - provides present extrainfo descriptors</span>
<span class="sd"> |- get_microdescriptors - provides present microdescriptors</span>
<span class="sd"> |- get_consensus - provides the present consensus or router status entries</span>
<span class="sd"> |- get_key_certificates - provides present authority key certificates</span>
<span class="sd"> +- query - request an arbitrary descriptor resource</span>
<span class="sd">.. data:: MAX_FINGERPRINTS</span>
<span class="sd"> Maximum number of descriptors that can requested at a time by their</span>
<span class="sd"> fingerprints.</span>
<span class="sd">.. data:: MAX_MICRODESCRIPTOR_HASHES</span>
<span class="sd"> Maximum number of microdescriptors that can requested at a time by their</span>
<span class="sd"> hashes.</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">io</span>
<span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">threading</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="kn">import</span> <span class="nn">zlib</span>
<span class="kn">import</span> <span class="nn">stem.descriptor</span>
<span class="kn">from</span> <span class="nn">stem</span> <span class="kn">import</span> <span class="n">Flag</span>
<span class="kn">from</span> <span class="nn">stem.util</span> <span class="kn">import</span> <span class="n">log</span>
<span class="c"># Tor has a limited number of descriptors we can fetch explicitly by their</span>
<span class="c"># fingerprint or hashes due to a limit on the url length by squid proxies.</span>
<span class="n">MAX_FINGERPRINTS</span> <span class="o">=</span> <span class="mi">96</span>
<span class="n">MAX_MICRODESCRIPTOR_HASHES</span> <span class="o">=</span> <span class="mi">92</span>
<span class="c"># We commonly only want authorities that vote in the consensus, and hence have</span>
<span class="c"># a v3ident.</span>
<span class="n">HAS_V3IDENT</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">auth</span><span class="p">:</span> <span class="n">auth</span><span class="o">.</span><span class="n">v3ident</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">_guess_descriptor_type</span><span class="p">(</span><span class="n">resource</span><span class="p">):</span>
<span class="c"># Attempts to determine the descriptor type based on the resource url. This</span>
<span class="c"># raises a ValueError if the resource isn't recognized.</span>
<span class="k">if</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/tor/server/'</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'server-descriptor 1.0'</span>
<span class="k">elif</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/tor/extra/'</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'extra-info 1.0'</span>
<span class="k">elif</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/tor/micro/'</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'microdescriptor 1.0'</span>
<span class="k">elif</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/tor/status-vote/'</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'network-status-consensus-3 1.0'</span>
<span class="k">elif</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/tor/keys/'</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'dir-key-certificate-3 1.0'</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unable to determine the descriptor type for '</span><span class="si">%s</span><span class="s">'"</span> <span class="o">%</span> <span class="n">resource</span><span class="p">)</span>
<div class="viewcode-block" id="Query"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.Query">[docs]</a><span class="k">class</span> <span class="nc">Query</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Asynchronous request for descriptor content from a directory authority or</span>
<span class="sd"> mirror. These can either be made through the</span>
<span class="sd"> :class:`~stem.descriptor.remote.DescriptorDownloader` or directly for more</span>
<span class="sd"> advanced usage.</span>
<span class="sd"> To block on the response and get results either call</span>
<span class="sd"> :func:`~stem.descriptor.remote.Query.run` or iterate over the Query. The</span>
<span class="sd"> :func:`~stem.descriptor.remote.Query.run` method pass along any errors that</span>
<span class="sd"> arise...</span>
<span class="sd"> ::</span>
<span class="sd"> from stem.descriptor.remote import Query</span>
<span class="sd"> query = Query(</span>
<span class="sd"> '/tor/server/all.z',</span>
<span class="sd"> block = True,</span>
<span class="sd"> timeout = 30,</span>
<span class="sd"> )</span>
<span class="sd"> print "Current relays:"</span>
<span class="sd"> if not query.error:</span>
<span class="sd"> for desc in query:</span>
<span class="sd"> print desc.fingerprint</span>
<span class="sd"> else:</span>
<span class="sd"> print "Unable to retrieve the server descriptors: %s" % query.error</span>
<span class="sd"> ... while iterating fails silently...</span>
<span class="sd"> ::</span>
<span class="sd"> print "Current relays:"</span>
<span class="sd"> for desc in Query('/tor/server/all.z', 'server-descriptor 1.0'):</span>
<span class="sd"> print desc.fingerprint</span>
<span class="sd"> In either case exceptions are available via our 'error' attribute.</span>
<span class="sd"> Tor provides quite a few different descriptor resources via its directory</span>
<span class="sd"> protocol (see section 4.2 and later of the `dir-spec</span>
<span class="sd"> <https://gitweb.torproject.org/torspec.git/blob/HEAD:/dir-spec.txt>`_).</span>
<span class="sd"> Commonly useful ones include...</span>
<span class="sd"> ===================================== ===========</span>
<span class="sd"> Resource Description</span>
<span class="sd"> ===================================== ===========</span>
<span class="sd"> /tor/server/all.z all present server descriptors</span>
<span class="sd"> /tor/server/fp/<fp1>+<fp2>+<fp3>.z server descriptors with the given fingerprints</span>
<span class="sd"> /tor/extra/all.z all present extrainfo descriptors</span>
<span class="sd"> /tor/extra/fp/<fp1>+<fp2>+<fp3>.z extrainfo descriptors with the given fingerprints</span>
<span class="sd"> /tor/micro/d/<hash1>-<hash2>.z microdescriptors with the given hashes</span>
<span class="sd"> /tor/status-vote/current/consensus.z present consensus</span>
<span class="sd"> /tor/keys/all.z key certificates for the authorities</span>
<span class="sd"> /tor/keys/fp/<v3ident1>+<v3ident2>.z key certificates for specific authorities</span>
<span class="sd"> ===================================== ===========</span>
<span class="sd"> The '.z' suffix can be excluded to get a plaintext rather than compressed</span>
<span class="sd"> response. Compression is handled transparently, so this shouldn't matter to</span>
<span class="sd"> the caller.</span>
<span class="sd"> :var str resource: resource being fetched, such as '/tor/server/all.z'</span>
<span class="sd"> :var str descriptor_type: type of descriptors being fetched (for options see</span>
<span class="sd"> :func:`~stem.descriptor.__init__.parse_file`), this is guessed from the</span>
<span class="sd"> resource if **None**</span>
<span class="sd"> :var list endpoints: (address, dirport) tuples of the authority or mirror</span>
<span class="sd"> we're querying, this uses authorities if undefined</span>
<span class="sd"> :var int retries: number of times to attempt the request if downloading it</span>
<span class="sd"> fails</span>
<span class="sd"> :var bool fall_back_to_authority: when retrying request issues the last</span>
<span class="sd"> request to a directory authority if **True**</span>
<span class="sd"> :var str content: downloaded descriptor content</span>
<span class="sd"> :var Exception error: exception if a problem occured</span>
<span class="sd"> :var bool is_done: flag that indicates if our request has finished</span>
<span class="sd"> :var str download_url: last url used to download the descriptor, this is</span>
<span class="sd"> unset until we've actually made a download attempt</span>
<span class="sd"> :var float start_time: unix timestamp when we first started running</span>
<span class="sd"> :var float timeout: duration before we'll time out our request</span>
<span class="sd"> :var float runtime: time our query took, this is **None** if it's not yet</span>
<span class="sd"> finished</span>
<span class="sd"> :var bool validate: checks the validity of the descriptor's content if</span>
<span class="sd"> **True**, skips these checks otherwise</span>
<span class="sd"> :var stem.descriptor.__init__.DocumentHandler document_handler: method in</span>
<span class="sd"> which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`</span>
<span class="sd"> :var dict kwargs: additional arguments for the descriptor constructor</span>
<span class="sd"> :param bool start: start making the request when constructed (default is **True**)</span>
<span class="sd"> :param bool block: only return after the request has been completed, this is</span>
<span class="sd"> the same as running **query.run(True)** (default is **False**)</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">resource</span><span class="p">,</span> <span class="n">descriptor_type</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">endpoints</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">retries</span> <span class="o">=</span> <span class="mi">2</span><span class="p">,</span> <span class="n">fall_back_to_authority</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">timeout</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">start</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">block</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">DocumentHandler</span><span class="o">.</span><span class="n">ENTRIES</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">resource</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/'</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Resources should start with a '/': </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">resource</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">resource</span> <span class="o">=</span> <span class="n">resource</span>
<span class="k">if</span> <span class="n">descriptor_type</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">descriptor_type</span> <span class="o">=</span> <span class="n">descriptor_type</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">descriptor_type</span> <span class="o">=</span> <span class="n">_guess_descriptor_type</span><span class="p">(</span><span class="n">resource</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">endpoints</span> <span class="o">=</span> <span class="n">endpoints</span> <span class="k">if</span> <span class="n">endpoints</span> <span class="k">else</span> <span class="p">[]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">retries</span> <span class="o">=</span> <span class="n">retries</span>
<span class="bp">self</span><span class="o">.</span><span class="n">fall_back_to_authority</span> <span class="o">=</span> <span class="n">fall_back_to_authority</span>
<span class="bp">self</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_done</span> <span class="o">=</span> <span class="bp">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">download_url</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">start_time</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
<span class="bp">self</span><span class="o">.</span><span class="n">runtime</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span>
<span class="bp">self</span><span class="o">.</span><span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span>
<span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span> <span class="o">=</span> <span class="n">kwargs</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread_lock</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">RLock</span><span class="p">()</span>
<span class="k">if</span> <span class="n">start</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">if</span> <span class="n">block</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<div class="viewcode-block" id="Query.start"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.Query.start">[docs]</a> <span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Starts downloading the scriptors if we haven't started already.</span>
<span class="sd"> """</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread_lock</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span>
<span class="n">name</span> <span class="o">=</span> <span class="s">"Descriptor Query"</span><span class="p">,</span>
<span class="n">target</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_download_descriptors</span><span class="p">,</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">retries</span><span class="p">,)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span><span class="o">.</span><span class="n">setDaemon</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</div>
<div class="viewcode-block" id="Query.run"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.Query.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">suppress</span> <span class="o">=</span> <span class="bp">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Blocks until our request is complete then provides the descriptors. If we</span>
<span class="sd"> haven't yet started our request then this does so.</span>
<span class="sd"> :param bool suppress: avoids raising exceptions if **True**</span>
<span class="sd"> :returns: list for the requested :class:`~stem.descriptor.__init__.Descriptor` instances</span>
<span class="sd"> :raises:</span>
<span class="sd"> Using the iterator can fail with the following if **suppress** is</span>
<span class="sd"> **False**...</span>
<span class="sd"> * **ValueError** if the descriptor contents is malformed</span>
<span class="sd"> * **socket.timeout** if our request timed out</span>
<span class="sd"> * **urllib2.URLError** for most request failures</span>
<span class="sd"> Note that the urllib2 module may fail with other exception types, in</span>
<span class="sd"> which case we'll pass it along.</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_run</span><span class="p">(</span><span class="n">suppress</span><span class="p">))</span>
</div>
<span class="k">def</span> <span class="nf">_run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">suppress</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread_lock</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_downloader_thread</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">error</span><span class="p">:</span>
<span class="k">if</span> <span class="n">suppress</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">raise</span> <span class="bp">self</span><span class="o">.</span><span class="n">error</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">content</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">if</span> <span class="n">suppress</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">'BUG: _download_descriptors() finished without either results or an error'</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">parse_file</span><span class="p">(</span>
<span class="n">io</span><span class="o">.</span><span class="n">BytesIO</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">content</span><span class="p">),</span>
<span class="bp">self</span><span class="o">.</span><span class="n">descriptor_type</span><span class="p">,</span>
<span class="n">validate</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">validate</span><span class="p">,</span>
<span class="n">document_handler</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">document_handler</span><span class="p">,</span>
<span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span>
<span class="p">)</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">results</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="n">exc</span> <span class="c"># encountered a parsing error</span>
<span class="k">if</span> <span class="n">suppress</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">raise</span> <span class="bp">self</span><span class="o">.</span><span class="n">error</span>
<span class="k">def</span> <span class="nf">__iter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_run</span><span class="p">(</span><span class="bp">True</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">def</span> <span class="nf">_pick_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">use_authority</span> <span class="o">=</span> <span class="bp">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides a url that can be queried. If we have multiple endpoints then one</span>
<span class="sd"> will be picked randomly.</span>
<span class="sd"> :param bool use_authority: ignores our endpoints and uses a directory</span>
<span class="sd"> authority instead</span>
<span class="sd"> :returns: **str** for the url being queried by this request</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">use_authority</span> <span class="ow">or</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoints</span><span class="p">:</span>
<span class="n">authority</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="n">HAS_V3IDENT</span><span class="p">,</span> <span class="n">get_authorities</span><span class="p">()</span><span class="o">.</span><span class="n">values</span><span class="p">()))</span>
<span class="n">address</span><span class="p">,</span> <span class="n">dirport</span> <span class="o">=</span> <span class="n">authority</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">authority</span><span class="o">.</span><span class="n">dir_port</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">address</span><span class="p">,</span> <span class="n">dirport</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoints</span><span class="p">)</span>
<span class="k">return</span> <span class="s">"http://</span><span class="si">%s</span><span class="s">:</span><span class="si">%i</span><span class="s">/</span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">dirport</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">resource</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s">'/'</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">_download_descriptors</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">retries</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">use_authority</span> <span class="o">=</span> <span class="n">retries</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">fall_back_to_authority</span>
<span class="bp">self</span><span class="o">.</span><span class="n">download_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pick_url</span><span class="p">(</span><span class="n">use_authority</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">download_url</span><span class="p">,</span> <span class="n">timeout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">download_url</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s">'.z'</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">zlib</span><span class="o">.</span><span class="n">decompress</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">content</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">runtime</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">start_time</span>
<span class="n">log</span><span class="o">.</span><span class="n">trace</span><span class="p">(</span><span class="s">"Descriptors retrieved from '</span><span class="si">%s</span><span class="s">' in </span><span class="si">%0.2f</span><span class="s">s"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">download_url</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">runtime</span><span class="p">))</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">exc</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="n">retries</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Unable to download descriptors from '</span><span class="si">%s</span><span class="s">' (</span><span class="si">%i</span><span class="s"> retries remaining): </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">download_url</span><span class="p">,</span> <span class="n">retries</span><span class="p">,</span> <span class="n">exc</span><span class="p">))</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_download_descriptors</span><span class="p">(</span><span class="n">retries</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Unable to download descriptors from '</span><span class="si">%s</span><span class="s">': </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">download_url</span><span class="p">,</span> <span class="n">exc</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="n">exc</span>
<span class="k">finally</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_done</span> <span class="o">=</span> <span class="bp">True</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader">[docs]</a><span class="k">class</span> <span class="nc">DescriptorDownloader</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Configurable class that issues :class:`~stem.descriptor.remote.Query`</span>
<span class="sd"> instances on your behalf.</span>
<span class="sd"> :param bool use_mirrors: downloads the present consensus and uses the directory</span>
<span class="sd"> mirrors to fetch future requests, this fails silently if the consensus</span>
<span class="sd"> cannot be downloaded</span>
<span class="sd"> :param default_args: default arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">use_mirrors</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="o">**</span><span class="n">default_args</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_default_args</span> <span class="o">=</span> <span class="n">default_args</span>
<span class="n">authorities</span> <span class="o">=</span> <span class="nb">filter</span><span class="p">(</span><span class="n">HAS_V3IDENT</span><span class="p">,</span> <span class="n">get_authorities</span><span class="p">()</span><span class="o">.</span><span class="n">values</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_endpoints</span> <span class="o">=</span> <span class="p">[(</span><span class="n">auth</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">auth</span><span class="o">.</span><span class="n">dir_port</span><span class="p">)</span> <span class="k">for</span> <span class="n">auth</span> <span class="ow">in</span> <span class="n">authorities</span><span class="p">]</span>
<span class="k">if</span> <span class="n">use_mirrors</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">use_directory_mirrors</span><span class="p">()</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Retrieved directory mirrors (took </span><span class="si">%0.2f</span><span class="s">s)"</span> <span class="o">%</span> <span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_time</span><span class="p">))</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">"Unable to retrieve directory mirrors: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">exc</span><span class="p">)</span>
<div class="viewcode-block" id="DescriptorDownloader.use_directory_mirrors"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.use_directory_mirrors">[docs]</a> <span class="k">def</span> <span class="nf">use_directory_mirrors</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Downloads the present consensus and configures ourselves to use directory</span>
<span class="sd"> mirrors, in addition to authorities.</span>
<span class="sd"> :returns: :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`</span>
<span class="sd"> from which we got the directory mirrors</span>
<span class="sd"> :raises: **Exception** if unable to determine the directory mirrors</span>
<span class="sd"> """</span>
<span class="n">authorities</span> <span class="o">=</span> <span class="nb">filter</span><span class="p">(</span><span class="n">HAS_V3IDENT</span><span class="p">,</span> <span class="n">get_authorities</span><span class="p">()</span><span class="o">.</span><span class="n">values</span><span class="p">())</span>
<span class="n">new_endpoints</span> <span class="o">=</span> <span class="nb">set</span><span class="p">([(</span><span class="n">auth</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">auth</span><span class="o">.</span><span class="n">dir_port</span><span class="p">)</span> <span class="k">for</span> <span class="n">auth</span> <span class="ow">in</span> <span class="n">authorities</span><span class="p">])</span>
<span class="n">consensus</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_consensus</span><span class="p">(</span><span class="n">document_handler</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">DocumentHandler</span><span class="o">.</span><span class="n">DOCUMENT</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">consensus</span><span class="o">.</span><span class="n">routers</span><span class="o">.</span><span class="n">values</span><span class="p">():</span>
<span class="k">if</span> <span class="n">Flag</span><span class="o">.</span><span class="n">V2DIR</span> <span class="ow">in</span> <span class="n">desc</span><span class="o">.</span><span class="n">flags</span><span class="p">:</span>
<span class="n">new_endpoints</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">desc</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">desc</span><span class="o">.</span><span class="n">dir_port</span><span class="p">))</span>
<span class="c"># we need our endpoints to be a list rather than set for random.choice()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_endpoints</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">new_endpoints</span><span class="p">)</span>
<span class="k">return</span> <span class="n">consensus</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_server_descriptors"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_server_descriptors">[docs]</a> <span class="k">def</span> <span class="nf">get_server_descriptors</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fingerprints</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the server descriptors with the given fingerprints. If no</span>
<span class="sd"> fingerprints are provided then this returns all descriptors in the present</span>
<span class="sd"> consensus.</span>
<span class="sd"> :param str,list fingerprints: fingerprint or list of fingerprints to be</span>
<span class="sd"> retrieved, gets all descriptors if **None**</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the server descriptors</span>
<span class="sd"> :raises: **ValueError** if we request more than 96 descriptors by their</span>
<span class="sd"> fingerprints (this is due to a limit on the url length by squid proxies).</span>
<span class="sd"> """</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/server/all.z'</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">fingerprints</span> <span class="o">=</span> <span class="p">[</span><span class="n">fingerprints</span><span class="p">]</span>
<span class="k">if</span> <span class="n">fingerprints</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">)</span> <span class="o">></span> <span class="n">MAX_FINGERPRINTS</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unable to request more than </span><span class="si">%i</span><span class="s"> descriptors at a time by their fingerprints"</span> <span class="o">%</span> <span class="n">MAX_FINGERPRINTS</span><span class="p">)</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/server/fp/</span><span class="si">%s</span><span class="s">.z'</span> <span class="o">%</span> <span class="s">'+'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">resource</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_extrainfo_descriptors"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_extrainfo_descriptors">[docs]</a> <span class="k">def</span> <span class="nf">get_extrainfo_descriptors</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fingerprints</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the extrainfo descriptors with the given fingerprints. If no</span>
<span class="sd"> fingerprints are provided then this returns all descriptors in the present</span>
<span class="sd"> consensus.</span>
<span class="sd"> :param str,list fingerprints: fingerprint or list of fingerprints to be</span>
<span class="sd"> retrieved, gets all descriptors if **None**</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the extrainfo descriptors</span>
<span class="sd"> :raises: **ValueError** if we request more than 96 descriptors by their</span>
<span class="sd"> fingerprints (this is due to a limit on the url length by squid proxies).</span>
<span class="sd"> """</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/extra/all.z'</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">fingerprints</span> <span class="o">=</span> <span class="p">[</span><span class="n">fingerprints</span><span class="p">]</span>
<span class="k">if</span> <span class="n">fingerprints</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">)</span> <span class="o">></span> <span class="n">MAX_FINGERPRINTS</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unable to request more than </span><span class="si">%i</span><span class="s"> descriptors at a time by their fingerprints"</span> <span class="o">%</span> <span class="n">MAX_FINGERPRINTS</span><span class="p">)</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/extra/fp/</span><span class="si">%s</span><span class="s">.z'</span> <span class="o">%</span> <span class="s">'+'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">fingerprints</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">resource</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_microdescriptors"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_microdescriptors">[docs]</a> <span class="k">def</span> <span class="nf">get_microdescriptors</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hashes</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the microdescriptors with the given hashes. To get these see the</span>
<span class="sd"> 'microdescriptor_hashes' attribute of</span>
<span class="sd"> :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3`. Note</span>
<span class="sd"> that these are only provided via a microdescriptor consensus (such as</span>
<span class="sd"> 'cached-microdesc-consensus' in your data directory).</span>
<span class="sd"> :param str,list hashes: microdescriptor hash or list of hashes to be</span>
<span class="sd"> retrieved</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the microdescriptors</span>
<span class="sd"> :raises: **ValueError** if we request more than 92 microdescriptors by their</span>
<span class="sd"> hashes (this is due to a limit on the url length by squid proxies).</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hashes</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">hashes</span> <span class="o">=</span> <span class="p">[</span><span class="n">hashes</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">hashes</span><span class="p">)</span> <span class="o">></span> <span class="n">MAX_MICRODESCRIPTOR_HASHES</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unable to request more than </span><span class="si">%i</span><span class="s"> microdescriptors at a time by their hashes"</span> <span class="o">%</span> <span class="n">MAX_MICRODESCRIPTOR_HASHES</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="s">'/tor/micro/d/</span><span class="si">%s</span><span class="s">.z'</span> <span class="o">%</span> <span class="s">'-'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">hashes</span><span class="p">),</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_consensus"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_consensus">[docs]</a> <span class="k">def</span> <span class="nf">get_consensus</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">authority_v3ident</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the present router status entries.</span>
<span class="sd"> :param str authority_v3ident: fingerprint of the authority key for which</span>
<span class="sd"> to get the consensus, see `'v3ident' in tor's config.c</span>
<span class="sd"> <https://gitweb.torproject.org/tor.git/blob/f631b73:/src/or/config.c#l816>`_</span>
<span class="sd"> for the values.</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the router status</span>
<span class="sd"> entries</span>
<span class="sd"> """</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/status-vote/current/consensus'</span>
<span class="k">if</span> <span class="n">authority_v3ident</span><span class="p">:</span>
<span class="n">resource</span> <span class="o">+=</span> <span class="s">'/</span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">authority_v3ident</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">resource</span> <span class="o">+</span> <span class="s">'.z'</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_vote"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_vote">[docs]</a> <span class="k">def</span> <span class="nf">get_vote</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">authority</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the present vote for a given directory authority.</span>
<span class="sd"> :param stem.descriptor.remote.DirectoryAuthority authority: authority for which to retrieve a vote for</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the router status</span>
<span class="sd"> entries</span>
<span class="sd"> """</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/status-vote/current/authority'</span>
<span class="k">if</span> <span class="ow">not</span> <span class="s">'endpoint'</span> <span class="ow">in</span> <span class="n">query_args</span><span class="p">:</span>
<span class="n">query_args</span><span class="p">[</span><span class="s">'endpoints'</span><span class="p">]</span> <span class="o">=</span> <span class="p">[(</span><span class="n">authority</span><span class="o">.</span><span class="n">address</span><span class="p">,</span> <span class="n">authority</span><span class="o">.</span><span class="n">dir_port</span><span class="p">)]</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">resource</span> <span class="o">+</span> <span class="s">'.z'</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.get_key_certificates"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.get_key_certificates">[docs]</a> <span class="k">def</span> <span class="nf">get_key_certificates</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">authority_v3idents</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the key certificates for authorities with the given fingerprints.</span>
<span class="sd"> If no fingerprints are provided then this returns all present key</span>
<span class="sd"> certificates.</span>
<span class="sd"> :param str authority_v3idents: fingerprint or list of fingerprints of the</span>
<span class="sd"> authority keys, see `'v3ident' in tor's config.c</span>
<span class="sd"> <https://gitweb.torproject.org/tor.git/blob/f631b73:/src/or/config.c#l816>`_</span>
<span class="sd"> for the values.</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the key certificates</span>
<span class="sd"> :raises: **ValueError** if we request more than 96 key certificates by</span>
<span class="sd"> their identity fingerprints (this is due to a limit on the url length by</span>
<span class="sd"> squid proxies).</span>
<span class="sd"> """</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/keys/all.z'</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">authority_v3idents</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">authority_v3idents</span> <span class="o">=</span> <span class="p">[</span><span class="n">authority_v3idents</span><span class="p">]</span>
<span class="k">if</span> <span class="n">authority_v3idents</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">authority_v3idents</span><span class="p">)</span> <span class="o">></span> <span class="n">MAX_FINGERPRINTS</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unable to request more than </span><span class="si">%i</span><span class="s"> key certificates at a time by their identity fingerprints"</span> <span class="o">%</span> <span class="n">MAX_FINGERPRINTS</span><span class="p">)</span>
<span class="n">resource</span> <span class="o">=</span> <span class="s">'/tor/keys/fp/</span><span class="si">%s</span><span class="s">.z'</span> <span class="o">%</span> <span class="s">'+'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">authority_v3idents</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">resource</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">)</span>
</div>
<div class="viewcode-block" id="DescriptorDownloader.query"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DescriptorDownloader.query">[docs]</a> <span class="k">def</span> <span class="nf">query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">resource</span><span class="p">,</span> <span class="o">**</span><span class="n">query_args</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Issues a request for the given resource.</span>
<span class="sd"> :param str resource: resource being fetched, such as '/tor/server/all.z'</span>
<span class="sd"> :param query_args: additional arguments for the</span>
<span class="sd"> :class:`~stem.descriptor.remote.Query` constructor</span>
<span class="sd"> :returns: :class:`~stem.descriptor.remote.Query` for the descriptors</span>
<span class="sd"> :raises: **ValueError** if resource is clearly invalid or the descriptor</span>
<span class="sd"> type can't be determined when 'descriptor_type' is **None**</span>
<span class="sd"> """</span>
<span class="n">args</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_default_args</span><span class="p">)</span>
<span class="n">args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">query_args</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="s">'endpoints'</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="n">args</span><span class="p">[</span><span class="s">'endpoints'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_endpoints</span>
<span class="k">if</span> <span class="ow">not</span> <span class="s">'fall_back_to_authority'</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="n">args</span><span class="p">[</span><span class="s">'fall_back_to_authority'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">return</span> <span class="n">Query</span><span class="p">(</span>
<span class="n">resource</span><span class="p">,</span>
<span class="o">**</span><span class="n">args</span>
<span class="p">)</span>
</div></div>
<div class="viewcode-block" id="DirectoryAuthority"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.DirectoryAuthority">[docs]</a><span class="k">class</span> <span class="nc">DirectoryAuthority</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Tor directory authority, a special type of relay `hardcoded into tor</span>
<span class="sd"> <https://gitweb.torproject.org/tor.git/blob/f631b73:/src/or/config.c#l816>`_</span>
<span class="sd"> that enumerates the other relays within the network.</span>
<span class="sd"> At a very high level tor works as follows...</span>
<span class="sd"> 1. A volunteer starts up a new tor relay, during which it sends a `server</span>
<span class="sd"> descriptor <server_descriptor.html>`_ to each of the directory</span>
<span class="sd"> authorities.</span>
<span class="sd"> 2. Each hour the directory authorities make a `vote <networkstatus.html>`_</span>
<span class="sd"> that says who they think the active relays are in the network and some</span>
<span class="sd"> attributes about them.</span>
<span class="sd"> 3. The directory authorities send each other their votes, and compile that</span>
<span class="sd"> into the `consensus <networkstatus.html>`_. This document is very similar</span>
<span class="sd"> to the votes, the only difference being that the majority of the</span>
<span class="sd"> authorities agree upon and sign this document. The idividual relay entries</span>
<span class="sd"> in the vote or consensus is called `router status entries</span>
<span class="sd"> <router_status_entry.html>`_.</span>
<span class="sd"> 4. Tor clients (people using the service) download the consensus from one of</span>
<span class="sd"> the authorities or a mirror to determine the active relays within the</span>
<span class="sd"> network. They in turn use this to construct their circuits and use the</span>
<span class="sd"> network.</span>
<span class="sd"> :var str nickname: nickname of the authority</span>
<span class="sd"> :var str address: IP address of the authority, presently they're all IPv4 but</span>
<span class="sd"> this may not always be the case</span>
<span class="sd"> :var int or_port: port on which the relay services relay traffic</span>
<span class="sd"> :var int dir_port: port on which directory information is available</span>
<span class="sd"> :var str fingerprint: relay fingerprint</span>
<span class="sd"> :var str v3ident: identity key fingerprint used to sign votes and consensus</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nickname</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">address</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">or_port</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">dir_port</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">fingerprint</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">v3ident</span> <span class="o">=</span> <span class="bp">None</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nickname</span> <span class="o">=</span> <span class="n">nickname</span>
<span class="bp">self</span><span class="o">.</span><span class="n">address</span> <span class="o">=</span> <span class="n">address</span>
<span class="bp">self</span><span class="o">.</span><span class="n">or_port</span> <span class="o">=</span> <span class="n">or_port</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dir_port</span> <span class="o">=</span> <span class="n">dir_port</span>
<span class="bp">self</span><span class="o">.</span><span class="n">fingerprint</span> <span class="o">=</span> <span class="n">fingerprint</span>
<span class="bp">self</span><span class="o">.</span><span class="n">v3ident</span> <span class="o">=</span> <span class="n">v3ident</span>
</div>
<span class="n">DIRECTORY_AUTHORITIES</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">'moria1'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'moria1'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'128.31.0.39'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">9101</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">9131</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'9695DFC35FFEB861329B9F1AB04C46397020CE31'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'D586D18309DED4CD6D57C18FDB97EFA96D330566'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'tor26'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'tor26'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'86.59.21.38'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'847B1F850344D7876491A54892F904934E4EB85D'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'dizum'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'dizum'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'194.109.206.212'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'7EA6EAD6FD83083C538F44038BBFA077587DD755'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'E8A9C45EDE6D711294FADF8E7951F4DE6CA56B58'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'Tonga'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'Tonga'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'82.94.251.203'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'4A0CCD2DDC7995083D73F5D667100C8A5831F16D'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="c"># does not vote in the consensus</span>
<span class="p">),</span>
<span class="s">'turtles'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'turtles'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'76.73.17.194'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">9090</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">9030</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'F397038ADC51336135E7B80BD99CA3844360292B'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'27B6B5996C426270A5C95488AA5BCEB6BCC86956'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'gabelmoo'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'gabelmoo'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'212.112.245.170'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'F2044413DAC2E02E3D6BCF4735A19BCA1DE97281'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'ED03BB616EB2F60BEC80151114BB25CEF515B226'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'dannenberg'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'dannenberg'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'193.23.244.244'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'7BE683E65D48141321C5ED92F075C55364AC7123'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'585769C78764D58426B8B52B6651A5A71137189A'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'urras'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'urras'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'208.83.223.34'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'0AD3FA884D18F89EEA2D89C019379E0E7FD94417'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'80550987E1D626E3EBA5E5E75A458DE0626D088C'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'maatuska'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'maatuska'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'171.25.193.9'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'BD6A829255CB08E66FBE7D3748363586E46B3810'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'49015F787433103580E3B66A1707A00E60F2D15B'</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">'Faravahar'</span><span class="p">:</span> <span class="n">DirectoryAuthority</span><span class="p">(</span>
<span class="n">nickname</span> <span class="o">=</span> <span class="s">'Faravahar'</span><span class="p">,</span>
<span class="n">address</span> <span class="o">=</span> <span class="s">'154.35.32.5'</span><span class="p">,</span>
<span class="n">or_port</span> <span class="o">=</span> <span class="mi">443</span><span class="p">,</span>
<span class="n">dir_port</span> <span class="o">=</span> <span class="mi">80</span><span class="p">,</span>
<span class="n">fingerprint</span> <span class="o">=</span> <span class="s">'CF6D0AAFB385BE71B8E111FC5CFF4B47923733BC'</span><span class="p">,</span>
<span class="n">v3ident</span> <span class="o">=</span> <span class="s">'EFCBE720AB3A82B99F9E953CD5BF50F7EEFC7B97'</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">}</span>
<div class="viewcode-block" id="get_authorities"><a class="viewcode-back" href="../../../api/descriptor/remote.html#stem.descriptor.remote.get_authorities">[docs]</a><span class="k">def</span> <span class="nf">get_authorities</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Provides the Tor directory authority information as of **Tor commit 00bcc25</span>
<span class="sd"> (8/27/13)**. The directory information hardcoded into Tor and occasionally</span>
<span class="sd"> changes, so the information this provides might not necessarily match your</span>
<span class="sd"> version of tor.</span>
<span class="sd"> :returns: dict of str nicknames to :class:`~stem.descriptor.remote.DirectoryAuthority` instances</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">DIRECTORY_AUTHORITIES</span><span class="p">)</span></div>
</pre></div>
</div>
<div class="bottomnav">
</div>
<div class="footer">
</div>
</body>
</html>
