<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Working with non-python tests</title>
<link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '2.3.4.1',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="top" title="None" href="../index.html" />
<link rel="up" title="Usages and Examples" href="index.html" />
<link rel="next" title="Talks and Tutorials" href="../talks.html" />
<link rel="prev" title="Changing standard (Python) test discovery" href="pythoncollection.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../talks.html" title="Talks and Tutorials"
accesskey="N">next</a></li>
<li class="right" >
<a href="pythoncollection.html" title="Changing standard (Python) test discovery"
accesskey="P">previous</a> |</li>
<li><a href="../contents.html">pytest-2.3.4.1</a> »</li>
<li><a href="index.html" accesskey="U">Usages and Examples</a> »</li>
<g:plusone></g:plusone>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<div id="searchbox" style="display: none">
<form class="search" action="../search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Search" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
<h3>quicklinks</h3>
<div style="text-align: left; font-size: 100%; vertical-align: middle;">
<table>
<tr>
<td>
<a href="../index.html">home</a>
</td><td>
<a href="../contents.html">TOC/contents</a>
</td></tr><tr><td>
<a href="../getting-started.html">install</a>
</td><td>
<a href="../changelog.html">changelog</a>
</td></tr><tr><td>
<a href="index.html">examples</a>
</td><td>
<a href="../customize.html">customize</a>
</td></tr><tr><td>
<a href="https://bitbucket.org/hpk42/pytest/issues?status=new&status=open">issues[bb]</a>
</td><td>
<a href="../contact.html">contact</a>
</td></tr></table>
</div>
<h3><a href="../contents.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Working with non-python tests</a><ul>
<li><a class="reference internal" href="#a-basic-example-for-specifying-tests-in-yaml-files">A basic example for specifying tests in Yaml files</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="pythoncollection.html"
title="previous chapter">Changing standard (Python) test discovery</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../talks.html"
title="next chapter">Talks and Tutorials</a></p>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="working-with-non-python-tests">
<span id="non-python-tests"></span><h1>Working with non-python tests<a class="headerlink" href="#working-with-non-python-tests" title="Permalink to this headline">¶</a></h1>
<div class="section" id="a-basic-example-for-specifying-tests-in-yaml-files">
<span id="yaml-plugin"></span><h2>A basic example for specifying tests in Yaml files<a class="headerlink" href="#a-basic-example-for-specifying-tests-in-yaml-files" title="Permalink to this headline">¶</a></h2>
<p>Here is an example <tt class="docutils literal"><span class="pre">conftest.py</span></tt> (extracted from Ali Afshnars special purpose <a class="reference external" href="http://bitbucket.org/aafshar/pytest-yamlwsgi/src/tip/pytest_yamlwsgi.py">pytest-yamlwsgi</a> plugin). This <tt class="docutils literal"><span class="pre">conftest.py</span></tt> will collect <tt class="docutils literal"><span class="pre">test*.yml</span></tt> files and will execute the yaml-formatted content as custom tests:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># content of conftest.py</span>
<span class="kn">import</span> <span class="nn">pytest</span>
<span class="k">def</span> <span class="nf">pytest_collect_file</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="k">if</span> <span class="n">path</span><span class="o">.</span><span class="n">ext</span> <span class="o">==</span> <span class="s">".yml"</span> <span class="ow">and</span> <span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"test"</span><span class="p">):</span>
<span class="k">return</span> <span class="n">YamlFile</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">parent</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">YamlFile</span><span class="p">(</span><span class="n">pytest</span><span class="o">.</span><span class="n">File</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">collect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">yaml</span> <span class="c"># we need a yaml parser, e.g. PyYAML</span>
<span class="n">raw</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fspath</span><span class="o">.</span><span class="n">open</span><span class="p">())</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">spec</span> <span class="ow">in</span> <span class="n">raw</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">yield</span> <span class="n">YamlItem</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="p">,</span> <span class="n">spec</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">YamlItem</span><span class="p">(</span><span class="n">pytest</span><span class="o">.</span><span class="n">Item</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">spec</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">YamlItem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">parent</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">spec</span> <span class="o">=</span> <span class="n">spec</span>
<span class="k">def</span> <span class="nf">runtest</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">spec</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="c"># some custom test execution (dumb example follows)</span>
<span class="k">if</span> <span class="n">name</span> <span class="o">!=</span> <span class="n">value</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">YamlException</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">repr_failure</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">excinfo</span><span class="p">):</span>
<span class="sd">""" called when self.runtest() raises an exception. """</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">YamlException</span><span class="p">):</span>
<span class="k">return</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="o">.</span><span class="n">join</span><span class="p">([</span>
<span class="s">"usecase execution failed"</span><span class="p">,</span>
<span class="s">" spec failed: </span><span class="si">%r</span><span class="s">: </span><span class="si">%r</span><span class="s">"</span> <span class="o">%</span> <span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="mi">3</span><span class="p">],</span>
<span class="s">" no further details known at this point."</span>
<span class="p">])</span>
<span class="k">def</span> <span class="nf">reportinfo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fspath</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s">"usecase: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span>
<span class="k">class</span> <span class="nc">YamlException</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
<span class="sd">""" custom exception for error reporting. """</span>
</pre></div>
</div>
<p>You can create a simple example file:</p>
<div class="highlight-python"><pre># test_simple.yml
ok:
sub1: sub1
hello:
world: world
some: other
</pre>
</div>
<p>and if you installed <a class="reference external" href="http://pypi.python.org/pypi/PyYAML/">PyYAML</a> or a compatible YAML-parser you can
now execute the test specification:</p>
<div class="highlight-python"><pre>nonpython $ py.test test_simple.yml
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.5
collected 2 items
test_simple.yml .F
================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
spec failed: 'some': 'other'
no further details known at this point.
==================== 1 failed, 1 passed in 0.05 seconds ====================</pre>
</div>
<p>You get one dot for the passing <tt class="docutils literal"><span class="pre">sub1:</span> <span class="pre">sub1</span></tt> check and one failure.
Obviously in the above <tt class="docutils literal"><span class="pre">conftest.py</span></tt> you’ll want to implement a more
interesting interpretation of the yaml-values. You can easily write
your own domain specific testing language this way.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last"><tt class="docutils literal"><span class="pre">repr_failure(excinfo)</span></tt> is called for representing test failures.
If you create custom collection nodes you can return an error
representation string of your choice. It
will be reported as a (red) string.</p>
</div>
<p><tt class="docutils literal"><span class="pre">reportinfo()</span></tt> is used for representing the test location and is also
consulted when reporting in <tt class="docutils literal"><span class="pre">verbose</span></tt> mode:</p>
<div class="highlight-python"><pre>nonpython $ py.test -v
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.5 -- /home/hpk/p/pytest/.tox/regen/bin/python
collecting ... collected 2 items
test_simple.yml:1: usecase: ok PASSED
test_simple.yml:1: usecase: hello FAILED
================================= FAILURES =================================
______________________________ usecase: hello ______________________________
usecase execution failed
spec failed: 'some': 'other'
no further details known at this point.
==================== 1 failed, 1 passed in 0.05 seconds ====================</pre>
</div>
<p>While developing your custom test collection and execution it’s also
interesting to just look at the collection tree:</p>
<div class="highlight-python"><pre>nonpython $ py.test --collectonly
=========================== test session starts ============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.5
collected 2 items
<YamlFile 'test_simple.yml'>
<YamlItem 'ok'>
<YamlItem 'hello'>
============================= in 0.05 seconds =============================</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../talks.html" title="Talks and Tutorials"
>next</a></li>
<li class="right" >
<a href="pythoncollection.html" title="Changing standard (Python) test discovery"
>previous</a> |</li>
<li><a href="../contents.html">pytest-2.3.4.1</a> »</li>
<li><a href="index.html" >Usages and Examples</a> »</li>
<g:plusone></g:plusone>
</ul>
</div>
<div class="footer">
© Copyright 2012, holger krekel.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-7597274-13']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
<script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
</body>
</html>
