<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>stem.descriptor.__init__ — Stem 1.1.0 documentation</title>
<link rel="stylesheet" href="../../../_static/haiku.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../../../_static/print.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../../../',
VERSION: '1.1.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../../../_static/jquery.js"></script>
<script type="text/javascript" src="../../../_static/underscore.js"></script>
<script type="text/javascript" src="../../../_static/doctools.js"></script>
<script type="text/javascript" src="../../../_static/theme_extras.js"></script>
<link rel="shortcut icon" href="../../../_static/favicon.png"/>
<link rel="top" title="Stem 1.1.0 documentation" href="../../../index.html" />
<link rel="up" title="stem" href="../../stem.html" />
</head>
<body>
<div class="header"><img class="rightlogo" src="../../../_static/logo.png" alt="Logo"/><h1 class="heading"><a href="../../../index.html">
<span>Stem Docs</span></a></h1>
<h2 class="heading"><span>stem.descriptor.__init__</span></h2>
</div>
<div class="topnav">
<p>
<ul id="navbar">
<li><a href="../../../index.html">Home</a></li>
<li><a href="../../../tutorials.html">Tutorials</a>
<ul>
<li><a href="../../../tutorials/the_little_relay_that_could.html">Hello World</a></li>
<li><a href="../../../tutorials/to_russia_with_love.html">Client Usage</a></li>
<li><a href="../../../tutorials/tortoise_and_the_hare.html">Event Listening</a></li>
<li><a href="../../../tutorials/mirror_mirror_on_the_wall.html">Tor Descriptors</a></li>
<li><a href="../../../tutorials/east_of_the_sun.html">Utilities</a></li>
<li><a href="../../../tutorials/double_double_toil_and_trouble.html">Examples</a></li>
</ul>
</li>
<li><a href="../../../api.html">API</a>
<ul>
<li><a href="../../../api/control.html">stem.control</a></li>
<li><a href="../../../api/connection.html">stem.connection</a></li>
<li><a href="../../../api/socket.html">stem.socket</a></li>
<li><a href="../../../api/process.html">stem.process</a></li>
<li><a href="../../../api/response.html">stem.response</a></li>
<li><a href="../../../api/exit_policy.html">stem.exit_policy</a></li>
<li><a href="../../../api/version.html">stem.version</a></li>
<li><a href="../../../api.html#descriptors">Descriptors</a></li>
<li><a href="../../../api.html#utilities">Utilities</a></li>
</ul>
</li>
<li><a href="https://trac.torproject.org/projects/tor/wiki/doc/stem">Development</a>
<ul>
<li><a href="../../../faq.html">FAQ</a></li>
<li><a href="../../../change_log.html">Change Log</a></li>
<li><a href="https://trac.torproject.org/projects/tor/wiki/doc/stem/bugs">Bug Tracker</a></li>
<li><a href="../../../download.html">Download</a></li>
</ul>
</li>
</ul>
</p>
</div>
<div class="content">
<h1>Source code for stem.descriptor.__init__</h1><div class="highlight"><pre>
<span class="c"># Copyright 2012-2013, Damian Johnson and The Tor Project</span>
<span class="c"># See LICENSE for licensing information</span>
<span class="sd">"""</span>
<span class="sd">Package for parsing and processing descriptor data.</span>
<span class="sd">**Module Overview:**</span>
<span class="sd">::</span>
<span class="sd"> parse_file - Parses the descriptors in a file.</span>
<span class="sd"> Descriptor - Common parent for all descriptor file types.</span>
<span class="sd"> |- get_path - location of the descriptor on disk if it came from a file</span>
<span class="sd"> |- get_archive_path - location of the descriptor within the archive it came from</span>
<span class="sd"> |- get_bytes - similar to str(), but provides our original bytes content</span>
<span class="sd"> |- get_unrecognized_lines - unparsed descriptor content</span>
<span class="sd"> +- __str__ - string that the descriptor was made from</span>
<span class="sd">.. data:: DocumentHandler (enum)</span>
<span class="sd"> Ways in which we can parse a</span>
<span class="sd"> :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`.</span>
<span class="sd"> Both **ENTRIES** and **BARE_DOCUMENT** have a 'thin' document, which doesn't</span>
<span class="sd"> have a populated **routers** attribute. This allows for lower memory usage</span>
<span class="sd"> and upfront runtime. However, if read time and memory aren't a concern then</span>
<span class="sd"> **DOCUMENT** can provide you with a fully populated document.</span>
<span class="sd"> =================== ===========</span>
<span class="sd"> DocumentHandler Description</span>
<span class="sd"> =================== ===========</span>
<span class="sd"> **ENTRIES** Iterates over the contained :class:`~stem.descriptor.router_status_entry.RouterStatusEntry`. Each has a reference to the bare document it came from (through its **document** attribute).</span>
<span class="sd"> **DOCUMENT** :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` with the :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` it contains (through its **routers** attribute).</span>
<span class="sd"> **BARE_DOCUMENT** :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` **without** a reference to its contents (the :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` are unread).</span>
<span class="sd"> =================== ===========</span>
<span class="sd">"""</span>
<span class="n">__all__</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"export"</span><span class="p">,</span>
<span class="s">"reader"</span><span class="p">,</span>
<span class="s">"remote"</span><span class="p">,</span>
<span class="s">"extrainfo_descriptor"</span><span class="p">,</span>
<span class="s">"server_descriptor"</span><span class="p">,</span>
<span class="s">"microdescriptor"</span><span class="p">,</span>
<span class="s">"networkstatus"</span><span class="p">,</span>
<span class="s">"router_status_entry"</span><span class="p">,</span>
<span class="s">"tordnsel"</span><span class="p">,</span>
<span class="s">"parse_file"</span><span class="p">,</span>
<span class="s">"Descriptor"</span><span class="p">,</span>
<span class="p">]</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">stem.prereq</span>
<span class="kn">import</span> <span class="nn">stem.util.enum</span>
<span class="kn">import</span> <span class="nn">stem.util.str_tools</span>
<span class="k">try</span><span class="p">:</span>
<span class="c"># added in python 2.7</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">stem.util.ordereddict</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="n">KEYWORD_CHAR</span> <span class="o">=</span> <span class="s">"a-zA-Z0-9-"</span>
<span class="n">WHITESPACE</span> <span class="o">=</span> <span class="s">" </span><span class="se">\t</span><span class="s">"</span>
<span class="n">KEYWORD_LINE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s">"^([</span><span class="si">%s</span><span class="s">]+)(?:[</span><span class="si">%s</span><span class="s">]+(.*))?$"</span> <span class="o">%</span> <span class="p">(</span><span class="n">KEYWORD_CHAR</span><span class="p">,</span> <span class="n">WHITESPACE</span><span class="p">))</span>
<span class="n">PGP_BLOCK_START</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s">"^-----BEGIN ([</span><span class="si">%s%s</span><span class="s">]+)-----$"</span> <span class="o">%</span> <span class="p">(</span><span class="n">KEYWORD_CHAR</span><span class="p">,</span> <span class="n">WHITESPACE</span><span class="p">))</span>
<span class="n">PGP_BLOCK_END</span> <span class="o">=</span> <span class="s">"-----END </span><span class="si">%s</span><span class="s">-----"</span>
<span class="n">DocumentHandler</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">enum</span><span class="o">.</span><span class="n">UppercaseEnum</span><span class="p">(</span>
<span class="s">"ENTRIES"</span><span class="p">,</span>
<span class="s">"DOCUMENT"</span><span class="p">,</span>
<span class="s">"BARE_DOCUMENT"</span><span class="p">,</span>
<span class="p">)</span>
<div class="viewcode-block" id="parse_file"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.parse_file">[docs]</a><span class="k">def</span> <span class="nf">parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">descriptor_type</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">DocumentHandler</span><span class="o">.</span><span class="n">ENTRIES</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Simple function to read the descriptor contents from a file, providing an</span>
<span class="sd"> iterator for its :class:`~stem.descriptor.__init__.Descriptor` contents.</span>
<span class="sd"> If you don't provide a **descriptor_type** argument then this automatically</span>
<span class="sd"> tries to determine the descriptor type based on the following...</span>
<span class="sd"> * The @type annotation on the first line. These are generally only found in</span>
<span class="sd"> the `descriptor archives <https://metrics.torproject.org>`_.</span>
<span class="sd"> * The filename if it matches something from tor's data directory. For</span>
<span class="sd"> instance, tor's 'cached-descriptors' contains server descriptors.</span>
<span class="sd"> This is a handy function for simple usage, but if you're reading multiple</span>
<span class="sd"> descriptor files you might want to consider the</span>
<span class="sd"> :class:`~stem.descriptor.reader.DescriptorReader`.</span>
<span class="sd"> Descriptor types include the following, including further minor versions (ie.</span>
<span class="sd"> if we support 1.1 then we also support everything from 1.0 and most things</span>
<span class="sd"> from 1.2, but not 2.0)...</span>
<span class="sd"> ========================================= =====</span>
<span class="sd"> Descriptor Type Class</span>
<span class="sd"> ========================================= =====</span>
<span class="sd"> server-descriptor 1.0 :class:`~stem.descriptor.server_descriptor.RelayDescriptor`</span>
<span class="sd"> extra-info 1.0 :class:`~stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor`</span>
<span class="sd"> microdescriptor 1.0 :class:`~stem.descriptor.microdescriptor.Microdescriptor`</span>
<span class="sd"> directory 1.0 **unsupported**</span>
<span class="sd"> network-status-2 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV2` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV2`)</span>
<span class="sd"> dir-key-certificate-3 1.0 :class:`~stem.descriptor.networkstatus.KeyCertificate`</span>
<span class="sd"> network-status-consensus-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)</span>
<span class="sd"> network-status-vote-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)</span>
<span class="sd"> network-status-microdesc-consensus-3 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryMicroV3` (with a :class:`~stem.descriptor.networkstatus.NetworkStatusDocumentV3`)</span>
<span class="sd"> bridge-network-status 1.0 :class:`~stem.descriptor.router_status_entry.RouterStatusEntryV3` (with a :class:`~stem.descriptor.networkstatus.BridgeNetworkStatusDocument`)</span>
<span class="sd"> bridge-server-descriptor 1.0 :class:`~stem.descriptor.server_descriptor.BridgeDescriptor`</span>
<span class="sd"> bridge-extra-info 1.1 :class:`~stem.descriptor.extrainfo_descriptor.BridgeExtraInfoDescriptor`</span>
<span class="sd"> torperf 1.0 **unsupported**</span>
<span class="sd"> bridge-pool-assignment 1.0 **unsupported**</span>
<span class="sd"> tordnsel 1.0 :class:`~stem.descriptor.tordnsel.TorDNSEL`</span>
<span class="sd"> ========================================= =====</span>
<span class="sd"> If you're using **python 3** then beware that the open() function defaults to</span>
<span class="sd"> using text mode. **Binary mode** is strongly suggested because it's both</span>
<span class="sd"> faster (by my testing by about 33x) and doesn't do universal newline</span>
<span class="sd"> translation which can make us misparse the document.</span>
<span class="sd"> ::</span>
<span class="sd"> my_descriptor_file = open(descriptor_path, 'rb')</span>
<span class="sd"> :param str,file descriptor_file: path or opened file with the descriptor contents</span>
<span class="sd"> :param str descriptor_type: `descriptor type <https://metrics.torproject.org/formats.html#descriptortypes>`_, this is guessed if not provided</span>
<span class="sd"> :param bool validate: checks the validity of the descriptor's content if</span>
<span class="sd"> **True**, skips these checks otherwise</span>
<span class="sd"> :param stem.descriptor.__init__.DocumentHandler document_handler: method in</span>
<span class="sd"> which to parse the :class:`~stem.descriptor.networkstatus.NetworkStatusDocument`</span>
<span class="sd"> :param dict kwargs: additional arguments for the descriptor constructor</span>
<span class="sd"> :returns: iterator for :class:`~stem.descriptor.__init__.Descriptor` instances in the file</span>
<span class="sd"> :raises:</span>
<span class="sd"> * **ValueError** if the contents is malformed and validate is True</span>
<span class="sd"> * **TypeError** if we can't match the contents of the file to a descriptor type</span>
<span class="sd"> * **IOError** if unable to read from the descriptor_file</span>
<span class="sd"> """</span>
<span class="c"># if we got a path then open that file for parsing</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="p">(</span><span class="nb">bytes</span><span class="p">,</span> <span class="nb">unicode</span><span class="p">)):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">)</span> <span class="k">as</span> <span class="n">desc_file</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">parse_file</span><span class="p">(</span><span class="n">desc_file</span><span class="p">,</span> <span class="n">descriptor_type</span><span class="p">,</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">return</span>
<span class="c"># The tor descriptor specifications do not provide a reliable method for</span>
<span class="c"># identifying a descriptor file's type and version so we need to guess</span>
<span class="c"># based on its filename. Metrics descriptors, however, can be identified</span>
<span class="c"># by an annotation on their first line...</span>
<span class="c"># https://trac.torproject.org/5651</span>
<span class="n">initial_position</span> <span class="o">=</span> <span class="n">descriptor_file</span><span class="o">.</span><span class="n">tell</span><span class="p">()</span>
<span class="n">first_line</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">str_tools</span><span class="o">.</span><span class="n">_to_unicode</span><span class="p">(</span><span class="n">descriptor_file</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="n">metrics_header_match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="s">"^@type (\S+) (\d+).(\d+)$"</span><span class="p">,</span> <span class="n">first_line</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">metrics_header_match</span><span class="p">:</span>
<span class="n">descriptor_file</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">initial_position</span><span class="p">)</span>
<span class="n">descriptor_path</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="s">'name'</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s">'<undefined>'</span> <span class="k">if</span> <span class="n">descriptor_path</span> <span class="ow">is</span> <span class="bp">None</span> <span class="k">else</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">descriptor_file</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">if</span> <span class="n">descriptor_type</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">descriptor_type_match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="s">"^(\S+) (\d+).(\d+)$"</span><span class="p">,</span> <span class="n">descriptor_type</span><span class="p">)</span>
<span class="k">if</span> <span class="n">descriptor_type_match</span><span class="p">:</span>
<span class="n">desc_type</span><span class="p">,</span> <span class="n">major_version</span><span class="p">,</span> <span class="n">minor_version</span> <span class="o">=</span> <span class="n">descriptor_type_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">_parse_metrics_file</span><span class="p">(</span><span class="n">desc_type</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">major_version</span><span class="p">),</span> <span class="nb">int</span><span class="p">(</span><span class="n">minor_version</span><span class="p">),</span> <span class="n">f</span><span class="p">,</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"The descriptor_type must be of the form '<type> <major_version>.<minor_version>'"</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">metrics_header_match</span><span class="p">:</span>
<span class="c"># Metrics descriptor handling</span>
<span class="n">desc_type</span><span class="p">,</span> <span class="n">major_version</span><span class="p">,</span> <span class="n">minor_version</span> <span class="o">=</span> <span class="n">metrics_header_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">_parse_metrics_file</span><span class="p">(</span><span class="n">desc_type</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">major_version</span><span class="p">),</span> <span class="nb">int</span><span class="p">(</span><span class="n">minor_version</span><span class="p">),</span> <span class="n">f</span><span class="p">,</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c"># Cached descriptor handling. These contain multiple descriptors per file.</span>
<span class="k">if</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">"cached-descriptors"</span><span class="p">:</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">server_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">"cached-extrainfo"</span><span class="p">:</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">extrainfo_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">"cached-microdescs"</span><span class="p">:</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">microdescriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">"cached-consensus"</span><span class="p">:</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">filename</span> <span class="o">==</span> <span class="s">"cached-microdesc-consensus"</span><span class="p">:</span>
<span class="n">file_parser</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">f</span><span class="p">:</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">is_microdescriptor</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="n">file_parser</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">file_parser</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">):</span>
<span class="k">if</span> <span class="n">descriptor_path</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">desc</span><span class="o">.</span><span class="n">_set_path</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">descriptor_path</span><span class="p">))</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">return</span>
<span class="c"># Not recognized as a descriptor file.</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s">"Unable to determine the descriptor's type. filename: '</span><span class="si">%s</span><span class="s">', first line: '</span><span class="si">%s</span><span class="s">'"</span> <span class="o">%</span> <span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">first_line</span><span class="p">))</span>
</div>
<span class="k">def</span> <span class="nf">_parse_metrics_file</span><span class="p">(</span><span class="n">descriptor_type</span><span class="p">,</span> <span class="n">major_version</span><span class="p">,</span> <span class="n">minor_version</span><span class="p">,</span> <span class="n">descriptor_file</span><span class="p">,</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="c"># Parses descriptor files from metrics, yielding individual descriptors. This</span>
<span class="c"># throws a TypeError if the descriptor_type or version isn't recognized.</span>
<span class="k">if</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"server-descriptor"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">server_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">is_bridge</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"bridge-server-descriptor"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">server_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">is_bridge</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"extra-info"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">extrainfo_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">is_bridge</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"microdescriptor"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">microdescriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"bridge-extra-info"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="c"># version 1.1 introduced a 'transport' field...</span>
<span class="c"># https://trac.torproject.org/6257</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">extrainfo_descriptor</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">is_bridge</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"network-status-2"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">document_type</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">NetworkStatusDocumentV2</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">document_type</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"dir-key-certificate-3"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file_key_certs</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="ow">in</span> <span class="p">(</span><span class="s">"network-status-consensus-3"</span><span class="p">,</span> <span class="s">"network-status-vote-3"</span><span class="p">)</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">document_type</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">NetworkStatusDocumentV3</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">document_type</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"network-status-microdesc-consensus-3"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">document_type</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">NetworkStatusDocumentV3</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">document_type</span><span class="p">,</span> <span class="n">is_microdescriptor</span> <span class="o">=</span> <span class="bp">True</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"bridge-network-status"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">document_type</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">BridgeNetworkStatusDocument</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">networkstatus</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">document_type</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="n">document_handler</span> <span class="o">=</span> <span class="n">document_handler</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">elif</span> <span class="n">descriptor_type</span> <span class="o">==</span> <span class="s">"tordnsel"</span> <span class="ow">and</span> <span class="n">major_version</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">document_type</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">tordnsel</span><span class="o">.</span><span class="n">TorDNSEL</span>
<span class="k">for</span> <span class="n">desc</span> <span class="ow">in</span> <span class="n">stem</span><span class="o">.</span><span class="n">descriptor</span><span class="o">.</span><span class="n">tordnsel</span><span class="o">.</span><span class="n">_parse_file</span><span class="p">(</span><span class="n">descriptor_file</span><span class="p">,</span> <span class="n">validate</span> <span class="o">=</span> <span class="n">validate</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">desc</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s">"Unrecognized metrics descriptor format. type: '</span><span class="si">%s</span><span class="s">', version: '</span><span class="si">%i</span><span class="s">.</span><span class="si">%i</span><span class="s">'"</span> <span class="o">%</span> <span class="p">(</span><span class="n">descriptor_type</span><span class="p">,</span> <span class="n">major_version</span><span class="p">,</span> <span class="n">minor_version</span><span class="p">))</span>
<div class="viewcode-block" id="Descriptor"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.Descriptor">[docs]</a><span class="k">class</span> <span class="nc">Descriptor</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Common parent for all types of descriptors.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">contents</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_path</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_archive_path</span> <span class="o">=</span> <span class="bp">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_raw_contents</span> <span class="o">=</span> <span class="n">contents</span>
<div class="viewcode-block" id="Descriptor.get_path"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.Descriptor.get_path">[docs]</a> <span class="k">def</span> <span class="nf">get_path</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the absolute path that we loaded this descriptor from.</span>
<span class="sd"> :returns: **str** with the absolute path of the descriptor source</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_path</span>
</div>
<div class="viewcode-block" id="Descriptor.get_archive_path"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.Descriptor.get_archive_path">[docs]</a> <span class="k">def</span> <span class="nf">get_archive_path</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> If this descriptor came from an archive then provides its path within the</span>
<span class="sd"> archive. This is only set if the descriptor came from a</span>
<span class="sd"> :class:`~stem.descriptor.reader.DescriptorReader`, and is **None** if this</span>
<span class="sd"> descriptor didn't come from an archive.</span>
<span class="sd"> :returns: **str** with the descriptor's path within the archive</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_archive_path</span>
</div>
<div class="viewcode-block" id="Descriptor.get_bytes"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.Descriptor.get_bytes">[docs]</a> <span class="k">def</span> <span class="nf">get_bytes</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the ASCII **bytes** of the descriptor. This only differs from</span>
<span class="sd"> **str()** if you're running python 3.x, in which case **str()** provides a</span>
<span class="sd"> **unicode** string.</span>
<span class="sd"> :returns: **bytes** for the descriptor's contents</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_raw_contents</span>
</div>
<div class="viewcode-block" id="Descriptor.get_unrecognized_lines"><a class="viewcode-back" href="../../../api/descriptor/descriptor.html#stem.descriptor.__init__.Descriptor.get_unrecognized_lines">[docs]</a> <span class="k">def</span> <span class="nf">get_unrecognized_lines</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides a list of lines that were either ignored or had data that we did</span>
<span class="sd"> not know how to process. This is most common due to new descriptor fields</span>
<span class="sd"> that this library does not yet know how to process. Patches welcome!</span>
<span class="sd"> :returns: **list** of lines of unrecognized content</span>
<span class="sd"> """</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
</div>
<span class="k">def</span> <span class="nf">_set_path</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_path</span> <span class="o">=</span> <span class="n">path</span>
<span class="k">def</span> <span class="nf">_set_archive_path</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_archive_path</span> <span class="o">=</span> <span class="n">path</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="n">stem</span><span class="o">.</span><span class="n">prereq</span><span class="o">.</span><span class="n">is_python_3</span><span class="p">():</span>
<span class="k">return</span> <span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">str_tools</span><span class="o">.</span><span class="n">_to_unicode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_raw_contents</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_raw_contents</span>
</div>
<span class="k">def</span> <span class="nf">_get_bytes_field</span><span class="p">(</span><span class="n">keyword</span><span class="p">,</span> <span class="n">content</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Provides the value corresponding to the given keyword. This is handy to fetch</span>
<span class="sd"> values specifically allowed to be arbitrary bytes prior to converting to</span>
<span class="sd"> unicode.</span>
<span class="sd"> :param str keyword: line to look up</span>
<span class="sd"> :param bytes content: content to look through</span>
<span class="sd"> :returns: **bytes** value on the given line, **None** if the line doesn't</span>
<span class="sd"> exist</span>
<span class="sd"> :raises: **ValueError** if the content isn't bytes</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Content must be bytes, got a </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="nb">type</span><span class="p">(</span><span class="n">content</span><span class="p">))</span>
<span class="n">line_match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">str_tools</span><span class="o">.</span><span class="n">_to_bytes</span><span class="p">(</span><span class="s">"^(opt )?</span><span class="si">%s</span><span class="s">(?:[</span><span class="si">%s</span><span class="s">]+(.*))?$"</span> <span class="o">%</span> <span class="p">(</span><span class="n">keyword</span><span class="p">,</span> <span class="n">WHITESPACE</span><span class="p">)),</span> <span class="n">content</span><span class="p">,</span> <span class="n">re</span><span class="o">.</span><span class="n">MULTILINE</span><span class="p">)</span>
<span class="k">if</span> <span class="n">line_match</span><span class="p">:</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">line_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">return</span> <span class="n">b</span><span class="s">""</span> <span class="k">if</span> <span class="n">value</span> <span class="ow">is</span> <span class="bp">None</span> <span class="k">else</span> <span class="n">value</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">_read_until_keywords</span><span class="p">(</span><span class="n">keywords</span><span class="p">,</span> <span class="n">descriptor_file</span><span class="p">,</span> <span class="n">inclusive</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">ignore_first</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">skip</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span> <span class="n">end_position</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span> <span class="n">include_ending_keyword</span> <span class="o">=</span> <span class="bp">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Reads from the descriptor file until we get to one of the given keywords or reach the</span>
<span class="sd"> end of the file.</span>
<span class="sd"> :param str,list keywords: keyword(s) we want to read until</span>
<span class="sd"> :param file descriptor_file: file with the descriptor content</span>
<span class="sd"> :param bool inclusive: includes the line with the keyword if True</span>
<span class="sd"> :param bool ignore_first: doesn't check if the first line read has one of the</span>
<span class="sd"> given keywords</span>
<span class="sd"> :param bool skip: skips buffering content, returning None</span>
<span class="sd"> :param int end_position: end if we reach this point in the file</span>
<span class="sd"> :param bool include_ending_keyword: provides the keyword we broke on if **True**</span>
<span class="sd"> :returns: **list** with the lines until we find one of the keywords, this is</span>
<span class="sd"> a two value tuple with the ending keyword if include_ending_keyword is</span>
<span class="sd"> **True**</span>
<span class="sd"> """</span>
<span class="n">content</span> <span class="o">=</span> <span class="bp">None</span> <span class="k">if</span> <span class="n">skip</span> <span class="k">else</span> <span class="p">[]</span>
<span class="n">ending_keyword</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">keywords</span><span class="p">,</span> <span class="p">(</span><span class="nb">bytes</span><span class="p">,</span> <span class="nb">unicode</span><span class="p">)):</span>
<span class="n">keywords</span> <span class="o">=</span> <span class="p">(</span><span class="n">keywords</span><span class="p">,)</span>
<span class="k">if</span> <span class="n">ignore_first</span><span class="p">:</span>
<span class="n">first_line</span> <span class="o">=</span> <span class="n">descriptor_file</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">if</span> <span class="n">content</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="n">first_line</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">content</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">first_line</span><span class="p">)</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">last_position</span> <span class="o">=</span> <span class="n">descriptor_file</span><span class="o">.</span><span class="n">tell</span><span class="p">()</span>
<span class="k">if</span> <span class="n">end_position</span> <span class="ow">and</span> <span class="n">last_position</span> <span class="o">>=</span> <span class="n">end_position</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">descriptor_file</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
<span class="k">break</span> <span class="c"># EOF</span>
<span class="n">line_match</span> <span class="o">=</span> <span class="n">KEYWORD_LINE</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">str_tools</span><span class="o">.</span><span class="n">_to_unicode</span><span class="p">(</span><span class="n">line</span><span class="p">))</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line_match</span><span class="p">:</span>
<span class="c"># no spaces or tabs in the line</span>
<span class="n">line_keyword</span> <span class="o">=</span> <span class="n">stem</span><span class="o">.</span><span class="n">util</span><span class="o">.</span><span class="n">str_tools</span><span class="o">.</span><span class="n">_to_unicode</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">line_keyword</span> <span class="o">=</span> <span class="n">line_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">line_keyword</span> <span class="ow">in</span> <span class="n">keywords</span><span class="p">:</span>
<span class="n">ending_keyword</span> <span class="o">=</span> <span class="n">line_keyword</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">inclusive</span><span class="p">:</span>
<span class="n">descriptor_file</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">last_position</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">content</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">content</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">break</span>
<span class="k">elif</span> <span class="n">content</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">content</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">include_ending_keyword</span><span class="p">:</span>
<span class="k">return</span> <span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">ending_keyword</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">content</span>
<span class="k">def</span> <span class="nf">_get_pseudo_pgp_block</span><span class="p">(</span><span class="n">remaining_contents</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Checks if given contents begins with a pseudo-Open-PGP-style block and, if</span>
<span class="sd"> so, pops it off and provides it back to the caller.</span>
<span class="sd"> :param list remaining_contents: lines to be checked for a public key block</span>
<span class="sd"> :returns: **str** with the armor wrapped contents or None if it doesn't exist</span>
<span class="sd"> :raises: **ValueError** if the contents starts with a key block but it's</span>
<span class="sd"> malformed (for instance, if it lacks an ending line)</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">remaining_contents</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span> <span class="c"># nothing left</span>
<span class="n">block_match</span> <span class="o">=</span> <span class="n">PGP_BLOCK_START</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">remaining_contents</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">if</span> <span class="n">block_match</span><span class="p">:</span>
<span class="n">block_type</span> <span class="o">=</span> <span class="n">block_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">block_lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">end_line</span> <span class="o">=</span> <span class="n">PGP_BLOCK_END</span> <span class="o">%</span> <span class="n">block_type</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">remaining_contents</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Unterminated pgp style block (looking for '</span><span class="si">%s</span><span class="s">'):</span><span class="se">\n</span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">end_line</span><span class="p">,</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">block_lines</span><span class="p">)))</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">remaining_contents</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">block_lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="n">line</span> <span class="o">==</span> <span class="n">end_line</span><span class="p">:</span>
<span class="k">return</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">block_lines</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="k">def</span> <span class="nf">_get_descriptor_components</span><span class="p">(</span><span class="n">raw_contents</span><span class="p">,</span> <span class="n">validate</span><span class="p">,</span> <span class="n">extra_keywords</span> <span class="o">=</span> <span class="p">()):</span>
<span class="sd">"""</span>
<span class="sd"> Initial breakup of the server descriptor contents to make parsing easier.</span>
<span class="sd"> A descriptor contains a series of 'keyword lines' which are simply a keyword</span>
<span class="sd"> followed by an optional value. Lines can also be followed by a signature</span>
<span class="sd"> block.</span>
<span class="sd"> To get a sub-listing with just certain keywords use extra_keywords. This can</span>
<span class="sd"> be useful if we care about their relative ordering with respect to each</span>
<span class="sd"> other. For instance, we care about the ordering of 'accept' and 'reject'</span>
<span class="sd"> entries because this influences the resulting exit policy, but for everything</span>
<span class="sd"> else in server descriptors the order does not matter.</span>
<span class="sd"> :param str raw_contents: descriptor content provided by the relay</span>
<span class="sd"> :param bool validate: checks the validity of the descriptor's content if</span>
<span class="sd"> True, skips these checks otherwise</span>
<span class="sd"> :param list extra_keywords: entity keywords to put into a separate listing</span>
<span class="sd"> with ordering intact</span>
<span class="sd"> :returns:</span>
<span class="sd"> **collections.OrderedDict** with the 'keyword => (value, pgp key) entries'</span>
<span class="sd"> mappings. If a extra_keywords was provided then this instead provides a two</span>
<span class="sd"> value tuple, the second being a list of those entries.</span>
<span class="sd"> """</span>
<span class="n">entries</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
<span class="n">extra_entries</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># entries with a keyword in extra_keywords</span>
<span class="n">remaining_lines</span> <span class="o">=</span> <span class="n">raw_contents</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="k">while</span> <span class="n">remaining_lines</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">remaining_lines</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="c"># V2 network status documents explicitly can contain blank lines...</span>
<span class="c">#</span>
<span class="c"># "Implementations MAY insert blank lines for clarity between sections;</span>
<span class="c"># these blank lines are ignored."</span>
<span class="c">#</span>
<span class="c"># ... and server descriptors end with an extra newline. But other documents</span>
<span class="c"># don't say how blank lines should be handled so globally ignoring them.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
<span class="k">continue</span>
<span class="c"># Some lines have an 'opt ' for backward compatibility. They should be</span>
<span class="c"># ignored. This prefix is being removed in...</span>
<span class="c"># https://trac.torproject.org/projects/tor/ticket/5124</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"opt "</span><span class="p">):</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="p">[</span><span class="mi">4</span><span class="p">:]</span>
<span class="n">line_match</span> <span class="o">=</span> <span class="n">KEYWORD_LINE</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line_match</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">validate</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s">"Line contains invalid characters: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">line</span><span class="p">)</span>
<span class="n">keyword</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">line_match</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="k">if</span> <span class="n">value</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">value</span> <span class="o">=</span> <span class="s">''</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">block_contents</span> <span class="o">=</span> <span class="n">_get_pseudo_pgp_block</span><span class="p">(</span><span class="n">remaining_lines</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">exc</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">validate</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">raise</span> <span class="n">exc</span>
<span class="k">if</span> <span class="n">keyword</span> <span class="ow">in</span> <span class="n">extra_keywords</span><span class="p">:</span>
<span class="n">extra_entries</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">"</span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">keyword</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">entries</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">keyword</span><span class="p">,</span> <span class="p">[])</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">value</span><span class="p">,</span> <span class="n">block_contents</span><span class="p">))</span>
<span class="k">if</span> <span class="n">extra_keywords</span><span class="p">:</span>
<span class="k">return</span> <span class="n">entries</span><span class="p">,</span> <span class="n">extra_entries</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">entries</span>
<span class="c"># importing at the end to avoid circular dependencies on our Descriptor class</span>
<span class="kn">import</span> <span class="nn">stem.descriptor.server_descriptor</span>
<span class="kn">import</span> <span class="nn">stem.descriptor.extrainfo_descriptor</span>
<span class="kn">import</span> <span class="nn">stem.descriptor.networkstatus</span>
<span class="kn">import</span> <span class="nn">stem.descriptor.microdescriptor</span>
<span class="kn">import</span> <span class="nn">stem.descriptor.tordnsel</span>
</pre></div>
</div>
<div class="bottomnav">
</div>
<div class="footer">
</div>
</body>
</html>
