<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Working with non-python tests</title> <link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" /> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> <script type="text/javascript"> var DOCUMENTATION_OPTIONS = { URL_ROOT: '../', VERSION: '2.3.4.1', COLLAPSE_INDEX: false, FILE_SUFFIX: '.html', HAS_SOURCE: true }; </script> <script type="text/javascript" src="../_static/jquery.js"></script> <script type="text/javascript" src="../_static/underscore.js"></script> <script type="text/javascript" src="../_static/doctools.js"></script> <link rel="top" title="None" href="../index.html" /> <link rel="up" title="Usages and Examples" href="index.html" /> <link rel="next" title="Talks and Tutorials" href="../talks.html" /> <link rel="prev" title="Changing standard (Python) test discovery" href="pythoncollection.html" /> </head> <body> <div class="related"> <h3>Navigation</h3> <ul> <li class="right" style="margin-right: 10px"> <a href="../talks.html" title="Talks and Tutorials" accesskey="N">next</a></li> <li class="right" > <a href="pythoncollection.html" title="Changing standard (Python) test discovery" accesskey="P">previous</a> |</li> <li><a href="../contents.html">pytest-2.3.4.1</a> »</li> <li><a href="index.html" accesskey="U">Usages and Examples</a> »</li> <g:plusone></g:plusone> </ul> </div> <div class="sphinxsidebar"> <div class="sphinxsidebarwrapper"> <div id="searchbox" style="display: none"> <form class="search" action="../search.html" method="get"> <input type="text" name="q" size="18" /> <input type="submit" value="Search" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> <script type="text/javascript">$('#searchbox').show(0);</script> <h3>quicklinks</h3> <div style="text-align: left; font-size: 100%; vertical-align: middle;"> <table> <tr> <td> <a href="../index.html">home</a> </td><td> <a href="../contents.html">TOC/contents</a> </td></tr><tr><td> <a href="../getting-started.html">install</a> </td><td> <a href="../changelog.html">changelog</a> </td></tr><tr><td> <a href="index.html">examples</a> </td><td> <a href="../customize.html">customize</a> </td></tr><tr><td> <a href="https://bitbucket.org/hpk42/pytest/issues?status=new&status=open">issues[bb]</a> </td><td> <a href="../contact.html">contact</a> </td></tr></table> </div> <h3><a href="../contents.html">Table Of Contents</a></h3> <ul> <li><a class="reference internal" href="#">Working with non-python tests</a><ul> <li><a class="reference internal" href="#a-basic-example-for-specifying-tests-in-yaml-files">A basic example for specifying tests in Yaml files</a></li> </ul> </li> </ul> <h4>Previous topic</h4> <p class="topless"><a href="pythoncollection.html" title="previous chapter">Changing standard (Python) test discovery</a></p> <h4>Next topic</h4> <p class="topless"><a href="../talks.html" title="next chapter">Talks and Tutorials</a></p> </div> </div> <div class="document"> <div class="documentwrapper"> <div class="bodywrapper"> <div class="body"> <div class="section" id="working-with-non-python-tests"> <span id="non-python-tests"></span><h1>Working with non-python tests<a class="headerlink" href="#working-with-non-python-tests" title="Permalink to this headline">¶</a></h1> <div class="section" id="a-basic-example-for-specifying-tests-in-yaml-files"> <span id="yaml-plugin"></span><h2>A basic example for specifying tests in Yaml files<a class="headerlink" href="#a-basic-example-for-specifying-tests-in-yaml-files" title="Permalink to this headline">¶</a></h2> <p>Here is an example <tt class="docutils literal"><span class="pre">conftest.py</span></tt> (extracted from Ali Afshnars special purpose <a class="reference external" href="http://bitbucket.org/aafshar/pytest-yamlwsgi/src/tip/pytest_yamlwsgi.py">pytest-yamlwsgi</a> plugin). This <tt class="docutils literal"><span class="pre">conftest.py</span></tt> will collect <tt class="docutils literal"><span class="pre">test*.yml</span></tt> files and will execute the yaml-formatted content as custom tests:</p> <div class="highlight-python"><div class="highlight"><pre><span class="c"># content of conftest.py</span> <span class="kn">import</span> <span class="nn">pytest</span> <span class="k">def</span> <span class="nf">pytest_collect_file</span><span class="p">(</span><span class="n">parent</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span> <span class="k">if</span> <span class="n">path</span><span class="o">.</span><span class="n">ext</span> <span class="o">==</span> <span class="s">".yml"</span> <span class="ow">and</span> <span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"test"</span><span class="p">):</span> <span class="k">return</span> <span class="n">YamlFile</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">parent</span><span class="p">)</span> <span class="k">class</span> <span class="nc">YamlFile</span><span class="p">(</span><span class="n">pytest</span><span class="o">.</span><span class="n">File</span><span class="p">):</span> <span class="k">def</span> <span class="nf">collect</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="kn">import</span> <span class="nn">yaml</span> <span class="c"># we need a yaml parser, e.g. PyYAML</span> <span class="n">raw</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">fspath</span><span class="o">.</span><span class="n">open</span><span class="p">())</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">spec</span> <span class="ow">in</span> <span class="n">raw</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="k">yield</span> <span class="n">YamlItem</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="bp">self</span><span class="p">,</span> <span class="n">spec</span><span class="p">)</span> <span class="k">class</span> <span class="nc">YamlItem</span><span class="p">(</span><span class="n">pytest</span><span class="o">.</span><span class="n">Item</span><span class="p">):</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">parent</span><span class="p">,</span> <span class="n">spec</span><span class="p">):</span> <span class="nb">super</span><span class="p">(</span><span class="n">YamlItem</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">parent</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">spec</span> <span class="o">=</span> <span class="n">spec</span> <span class="k">def</span> <span class="nf">runtest</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">spec</span><span class="o">.</span><span class="n">items</span><span class="p">():</span> <span class="c"># some custom test execution (dumb example follows)</span> <span class="k">if</span> <span class="n">name</span> <span class="o">!=</span> <span class="n">value</span><span class="p">:</span> <span class="k">raise</span> <span class="n">YamlException</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span> <span class="k">def</span> <span class="nf">repr_failure</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">excinfo</span><span class="p">):</span> <span class="sd">""" called when self.runtest() raises an exception. """</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">YamlException</span><span class="p">):</span> <span class="k">return</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="o">.</span><span class="n">join</span><span class="p">([</span> <span class="s">"usecase execution failed"</span><span class="p">,</span> <span class="s">" spec failed: </span><span class="si">%r</span><span class="s">: </span><span class="si">%r</span><span class="s">"</span> <span class="o">%</span> <span class="n">excinfo</span><span class="o">.</span><span class="n">value</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="mi">3</span><span class="p">],</span> <span class="s">" no further details known at this point."</span> <span class="p">])</span> <span class="k">def</span> <span class="nf">reportinfo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">fspath</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="s">"usecase: </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="k">class</span> <span class="nc">YamlException</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span> <span class="sd">""" custom exception for error reporting. """</span> </pre></div> </div> <p>You can create a simple example file:</p> <div class="highlight-python"><pre># test_simple.yml ok: sub1: sub1 hello: world: world some: other </pre> </div> <p>and if you installed <a class="reference external" href="http://pypi.python.org/pypi/PyYAML/">PyYAML</a> or a compatible YAML-parser you can now execute the test specification:</p> <div class="highlight-python"><pre>nonpython $ py.test test_simple.yml =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.5 collected 2 items test_simple.yml .F ================================= FAILURES ================================= ______________________________ usecase: hello ______________________________ usecase execution failed spec failed: 'some': 'other' no further details known at this point. ==================== 1 failed, 1 passed in 0.05 seconds ====================</pre> </div> <p>You get one dot for the passing <tt class="docutils literal"><span class="pre">sub1:</span> <span class="pre">sub1</span></tt> check and one failure. Obviously in the above <tt class="docutils literal"><span class="pre">conftest.py</span></tt> you’ll want to implement a more interesting interpretation of the yaml-values. You can easily write your own domain specific testing language this way.</p> <div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last"><tt class="docutils literal"><span class="pre">repr_failure(excinfo)</span></tt> is called for representing test failures. If you create custom collection nodes you can return an error representation string of your choice. It will be reported as a (red) string.</p> </div> <p><tt class="docutils literal"><span class="pre">reportinfo()</span></tt> is used for representing the test location and is also consulted when reporting in <tt class="docutils literal"><span class="pre">verbose</span></tt> mode:</p> <div class="highlight-python"><pre>nonpython $ py.test -v =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.5 -- /home/hpk/p/pytest/.tox/regen/bin/python collecting ... collected 2 items test_simple.yml:1: usecase: ok PASSED test_simple.yml:1: usecase: hello FAILED ================================= FAILURES ================================= ______________________________ usecase: hello ______________________________ usecase execution failed spec failed: 'some': 'other' no further details known at this point. ==================== 1 failed, 1 passed in 0.05 seconds ====================</pre> </div> <p>While developing your custom test collection and execution it’s also interesting to just look at the collection tree:</p> <div class="highlight-python"><pre>nonpython $ py.test --collectonly =========================== test session starts ============================ platform linux2 -- Python 2.7.3 -- pytest-2.3.5 collected 2 items <YamlFile 'test_simple.yml'> <YamlItem 'ok'> <YamlItem 'hello'> ============================= in 0.05 seconds =============================</pre> </div> </div> </div> </div> </div> </div> <div class="clearer"></div> </div> <div class="related"> <h3>Navigation</h3> <ul> <li class="right" style="margin-right: 10px"> <a href="../talks.html" title="Talks and Tutorials" >next</a></li> <li class="right" > <a href="pythoncollection.html" title="Changing standard (Python) test discovery" >previous</a> |</li> <li><a href="../contents.html">pytest-2.3.4.1</a> »</li> <li><a href="index.html" >Usages and Examples</a> »</li> <g:plusone></g:plusone> </ul> </div> <div class="footer"> © Copyright 2012, holger krekel. Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3. </div> <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-7597274-13']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script> <script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script> </body> </html>