<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Testing Topic Guide — Horizon 2012.2.3 documentation</title>
<link rel="stylesheet" href="../_static/nature.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/tweaks.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '2012.2.3',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<script type="text/javascript" src="../_static/jquery.tweet.js"></script>
<link rel="top" title="Horizon 2012.2.3 documentation" href="../index.html" />
<link rel="next" title="The run_tests.sh Script" href="../ref/run_tests.html" />
<link rel="prev" title="DataTables Topic Guide" href="tables.html" />
</head>
<body>
<div id="header">
<h1 id="logo"><a href="http://www.openstack.org/">OpenStack</a></h1>
<ul id="navigation">
<li><a href="http://www.openstack.org/" title="Go to the Home page" class="link">Home</a></li>
<li><a href="http://www.openstack.org/projects/" title="Go to the OpenStack Projects page">Projects</a></li>
<li><a href="http://www.openstack.org/user-stories/" title="Go to the User Stories page" class="link">User Stories</a></li>
<li><a href="http://www.openstack.org/community/" title="Go to the Community page" class="link">Community</a></li>
<li><a href="http://www.openstack.org/blog/" title="Go to the OpenStack Blog">Blog</a></li>
<li><a href="http://wiki.openstack.org/" title="Go to the OpenStack Wiki">Wiki</a></li>
<li><a href="http://docs.openstack.org/" title="Go to OpenStack Documentation" class="current">Documentation</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="testing-topic-guide">
<h1>Testing Topic Guide<a class="headerlink" href="#testing-topic-guide" title="Permalink to this headline">¶</a></h1>
<p>Having good tests in place is absolutely critical for ensuring a stable,
maintainable codebase. Hopefully that doesn’t need any more explanation.</p>
<p>However, what defines a “good” test is not always obvious, and there are
a lot of common pitfalls that can easily shoot your test suite in the
foot.</p>
<p>If you already know everything about testing but are fed up with trying to
debug why a specific test failed, you can skip the intro and jump
stright to <a class="reference internal" href="#debugging-unit-tests"><em>Debugging Unit Tests</em></a>.</p>
<div class="section" id="an-overview-of-testing">
<h2>An overview of testing<a class="headerlink" href="#an-overview-of-testing" title="Permalink to this headline">¶</a></h2>
<p>There are three main types of tests, each with their associated pros and cons:</p>
<div class="section" id="unit-tests">
<h3>Unit tests<a class="headerlink" href="#unit-tests" title="Permalink to this headline">¶</a></h3>
<p>These are isolated, stand-alone tests with no external dependencies. They are
written from the a perspective of “knowing the code”, and test the assumptions
of the codebase and the developer.</p>
<p>Pros:</p>
<ul class="simple">
<li>Generally lightweight and fast.</li>
<li>Can be run anywhere, anytime since they have no external dependencies.</li>
</ul>
<p>Cons:</p>
<ul class="simple">
<li>Easy to be lax in writing them, or lazy in constructing them.</li>
<li>Can’t test interactions with live external services.</li>
</ul>
</div>
<div class="section" id="functional-tests">
<h3>Functional tests<a class="headerlink" href="#functional-tests" title="Permalink to this headline">¶</a></h3>
<p>These are generally also isolated tests, though sometimes they may interact
with other services running locally. The key difference between functional
tests and unit tests, however, is that functional tests are written from the
perspective of the user (who knows nothing about the code) and only knows
what they put in and what they get back. Essentially this is a higher-level
testing of “does the result match the spec?”.</p>
<p>Pros:</p>
<ul class="simple">
<li>Ensures that your code <em>always</em> meets the stated functional requirements.</li>
<li>Verifies things from an “end user” perspective, which helps to ensure
a high-quality experience.</li>
<li>Designing your code with a functional testing perspective in mind helps
keep a higher-level viewpoint in mind.</li>
</ul>
<p>Cons:</p>
<ul class="simple">
<li>Requires an additional layer of thinking to define functional requirements
in terms of inputs and outputs.</li>
<li>Often requires writing a separate set of tests and/or using a different
testing framework from your unit tests.</li>
<li>Don’t offer any insight into the quality or status of the underlying code,
only verifies that it works or it doesn’t.</li>
</ul>
</div>
<div class="section" id="integration-tests">
<h3>Integration Tests<a class="headerlink" href="#integration-tests" title="Permalink to this headline">¶</a></h3>
<p>This layer of testing involves testing all of the components that your
codebase interacts with or relies on in conjunction. This is equivalent to
“live” testing, but in a repeatable manner.</p>
<p>Pros:</p>
<ul class="simple">
<li>Catches <em>many</em> bugs that unit and functional tests will not.</li>
<li>Doesn’t rely on assumptions about the inputs and outputs.</li>
<li>Will warn you when changes in external components break your code.</li>
</ul>
<p>Cons:</p>
<ul class="simple">
<li>Difficult and time-consuming to create a repeatable test environment.</li>
<li>Did I mention that setting it up is a pain?</li>
</ul>
</div>
<div class="section" id="so-what-should-i-write">
<h3>So what should I write?<a class="headerlink" href="#so-what-should-i-write" title="Permalink to this headline">¶</a></h3>
<p>A few simple guidelines:</p>
<ol class="arabic simple">
<li>Every bug fix should have a regression test. Period.</li>
<li>When writing a new feature, think about writing unit tests to verify
the behavior step-by-step as you write the feature. Every time you’d
go to run your code by hand and verify it manually, think “could I
write a test to do this instead?”. That way when the feature is done
and you’re ready to commit it you’ve already got a whole set of tests
that are more thorough than anything you’d write after the fact.</li>
<li>Write tests that hit every view in your application. Even if they
don’t assert a single thing about the code, it tells you that your
users aren’t getting fatal errors just by interacting with your code.</li>
</ol>
</div>
</div>
<div class="section" id="what-makes-a-good-unit-test">
<h2>What makes a good unit test?<a class="headerlink" href="#what-makes-a-good-unit-test" title="Permalink to this headline">¶</a></h2>
<p>Limiting our focus just to unit tests, there are a number of things you can
do to make your unit tests as useful, maintainable, and unburdensome as
possible.</p>
<div class="section" id="test-data">
<h3>Test data<a class="headerlink" href="#test-data" title="Permalink to this headline">¶</a></h3>
<p>Use a single, consistent set of test data. Grow it over time, but do everything
you can not to fragment it. It quickly becomes unmaintainable and perniciously
out-of-sync with reality.</p>
<p>Make your test data as accurate to reality as possible. Supply <em>all</em> the
attributes of an object, provide objects in all the various states you may want
to test.</p>
<p>If you do the first suggestion above <em>first</em> it makes the second one far less
painful. Write once, use everywhere.</p>
<p>To make your life even easier, if your codebase doesn’t have a built-in
ORM-like function to manage your test data you can consider buidling (or
borrowing) one yourself. Being able to do simple retrieval queries on your
test data is incredibly valuable.</p>
</div>
<div class="section" id="mocking">
<h3>Mocking<a class="headerlink" href="#mocking" title="Permalink to this headline">¶</a></h3>
<p>Mocking is the practice of providing stand-ins for objects or pieces of code
you don’t need to test. While convenient, they should be used with <em>extreme</em>
caution.</p>
<p>Why? Because overuse of mocks can rapidly land you in a situation where you’re
not testing any real code. All you’ve done is verified that your mocking
framework returns what you tell it to. This problem can be very tricky to
recognize, since you may be mocking things in <tt class="docutils literal"><span class="pre">setUp</span></tt> methods, other modules,
etc.</p>
<p>A good rule of thumb is to mock as close to the source as possible. If you have
a function call that calls an external API in a view , mock out the external
API, not the whole function. If you mock the whole function you’ve suddenly
lost test coverage for an entire chunk of code <em>inside</em> your codebase. Cut the
ties cleanly right where your system ends and the external world begins.</p>
<p>Similarly, don’t mock return values when you could construct a real return
value of the correct type with the correct attributes. You’re just adding
another point of potential failure by exercising your mocking framework instead
of real code. Following the suggestions for testing above will make this a lot
less burdensome.</p>
</div>
<div class="section" id="assertions-and-verification">
<h3>Assertions and verification<a class="headerlink" href="#assertions-and-verification" title="Permalink to this headline">¶</a></h3>
<p>Think long and hard about what you really want to verify in your unit test. In
particular, think about what custom logic your code executes.</p>
<p>A common pitfall is to take a known test object, pass it through your code,
and then verify the properties of that object on the output. This is all well
and good, except if you’re verifying properties that were untouched by your
code. What you want to check are the pieces that were <em>changed</em>, <em>added</em>, or
<em>removed</em>. Don’t check the object’s id attribute unless you have reason to
suspect it’s not the object you started with. But if you added a new attribute
to it, be damn sure you verify that came out right.</p>
<p>It’s also very common to avoid testing things you really care about because
it’s more difficult. Verifying that the proper messages were displayed to the
user after an action, testing for form errors, making sure exception handling
is tested... these types of things aren’t always easy, but they’re extremely
necessary.</p>
<p>To that end, Horizon includes several custom assertions to make these tasks
easier. <tt class="xref py py-meth docutils literal"><span class="pre">assertNoFormErrors()</span></tt>,
<tt class="xref py py-meth docutils literal"><span class="pre">assertMessageCount()</span></tt>, and
<tt class="xref py py-meth docutils literal"><span class="pre">asertNoMessages()</span></tt> all exist for exactly these
purposes. Moreover, they provide useful output when things go wrong so you’re
not left scratching your head wondering why your view test didn’t redirect
as expected when you posted a form.</p>
</div>
</div>
<div class="section" id="debugging-unit-tests">
<span id="id1"></span><h2>Debugging Unit Tests<a class="headerlink" href="#debugging-unit-tests" title="Permalink to this headline">¶</a></h2>
<div class="section" id="tips-and-tricks">
<h3>Tips and tricks<a class="headerlink" href="#tips-and-tricks" title="Permalink to this headline">¶</a></h3>
<ol class="arabic simple">
<li>Use <tt class="xref py py-meth docutils literal"><span class="pre">assertNoFormErrors()</span></tt> immediately after
your <tt class="docutils literal"><span class="pre">client.post</span></tt> call for tests that handle form views. This will
immediately fail if your form POST failed due to a validation error and
tell you what the error was.</li>
<li>Use <tt class="xref py py-meth docutils literal"><span class="pre">assertMessageCount()</span></tt> and
<tt class="xref py py-meth docutils literal"><span class="pre">asertNoMessages()</span></tt> when a piece of code is
failing inexplicably. Since the core error handlers attach user-facing
error messages (and since the core logging is silenced during test runs)
these methods give you the dual benefit of verifying the output you expect
while clearly showing you the problematic error message if they fail.</li>
<li>Use Python’s <tt class="docutils literal"><span class="pre">pdb</span></tt> module liberally. Many people don’t realize it works
just as well in a test case as it does in a live view. Simply inserting
<tt class="docutils literal"><span class="pre">import</span> <span class="pre">pdb;</span> <span class="pre">pdb.set_trace()</span></tt> anywhere in your codebase will drop the
interpreter into an interactive shell so you can explore your test
environment and see which of your assumptions about the code isn’t,
in fact, flawlessly correct.</li>
</ol>
</div>
<div class="section" id="common-pitfalls">
<h3>Common pitfalls<a class="headerlink" href="#common-pitfalls" title="Permalink to this headline">¶</a></h3>
<p>There are a number of typical (and non-obvious) ways to break the unit tests.
Some common things to look for:</p>
<ol class="arabic simple">
<li>Make sure you stub out the method exactly as it’s called in the code
being tested. For example, if your real code calls
<tt class="docutils literal"><span class="pre">api.keystone.tenant_get</span></tt>, stubbing out <tt class="docutils literal"><span class="pre">api.tenant_get</span></tt> (available
for legacy reasons) will fail.</li>
<li>When defining the expected input to a stubbed call, make sure the
arguments are <em>identical</em>, this includes <tt class="docutils literal"><span class="pre">str</span></tt> vs. <tt class="docutils literal"><span class="pre">int</span></tt> differences.</li>
<li>Make sure your test data are completely in line with the expected inputs.
Again, <tt class="docutils literal"><span class="pre">str</span></tt> vs. <tt class="docutils literal"><span class="pre">int</span></tt> or missing properties on test objects will
kill your tests.</li>
<li>Make sure there’s nothing amiss in your templates (particularly the
<tt class="docutils literal"><span class="pre">{%</span> <span class="pre">url</span> <span class="pre">%}</span></tt> tag and its arguments). This often comes up when refactoring
views or renaming context variables. It can easily result in errors that
you might not stumble across while clicking around the development server.</li>
<li>Make sure you’re not redirecting to views that no longer exist, e.g.
the <tt class="docutils literal"><span class="pre">index</span></tt> view for a panel that got combined (such as instances &
volumes).</li>
<li>Make sure your mock calls are in order before calling <tt class="docutils literal"><span class="pre">mox.ReplayAll</span></tt>.
The order matters.</li>
<li>Make sure you repeat any stubbed out method calls that happen more than
once. They don’t automatically repeat, you have to explicitly define them.
While this is a nuisance, it makes you acutely aware of how many API
calls are involved in a particular function.</li>
</ol>
</div>
<div class="section" id="understanding-the-output-from-mox">
<h3>Understanding the output from <tt class="docutils literal"><span class="pre">mox</span></tt><a class="headerlink" href="#understanding-the-output-from-mox" title="Permalink to this headline">¶</a></h3>
<p>Horizon uses <tt class="docutils literal"><span class="pre">mox</span></tt> as its mocking framework of choice, and while it
offers many nice features, its output when a test fails can be quite
mysterious.</p>
<div class="section" id="unexpected-method-call">
<h4>Unexpected Method Call<a class="headerlink" href="#unexpected-method-call" title="Permalink to this headline">¶</a></h4>
<p>This occurs when you stubbed out a piece of code, and it was subsequently
called in a way that you didn’t specify it would be. There are two reasons
this tends to come up:</p>
<ol class="arabic simple">
<li>You defined the expected call, but a subtle difference crept in. This
may be a string versus integer difference, a string versus unicode
difference, a slightly off date/time, or passing a name instead of an id.</li>
<li>The method is actually being called <em>multiple times</em>. Since mox uses
a call stack internally, it simply pops off the expected method calls to
verify them. That means once a call is used once, it’s gone. An easy way
to see if this is the case is simply to copy and paste your method call a
second time to see if the error changes. If it does, that means your method
is being called more times than you think it is.</li>
</ol>
</div>
<div class="section" id="expected-method-never-called">
<h4>Expected Method Never Called<a class="headerlink" href="#expected-method-never-called" title="Permalink to this headline">¶</a></h4>
<p>This one is the opposite of the unexpected method call. This one means you
tol mox to expect a call and it didn’t happen. This is almost always the
result of an error in the conditions of the test. Using the
<tt class="xref py py-meth docutils literal"><span class="pre">assertNoFormErrors()</span></tt> and
<tt class="xref py py-meth docutils literal"><span class="pre">assertMessageCount()</span></tt> will make it readily apparent
what the problem is in the majority of cases. If not, then use <tt class="docutils literal"><span class="pre">pdb</span></tt> and
start interrupting the code flow to see where things are getting off track.</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="../index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Testing Topic Guide</a><ul>
<li><a class="reference internal" href="#an-overview-of-testing">An overview of testing</a><ul>
<li><a class="reference internal" href="#unit-tests">Unit tests</a></li>
<li><a class="reference internal" href="#functional-tests">Functional tests</a></li>
<li><a class="reference internal" href="#integration-tests">Integration Tests</a></li>
<li><a class="reference internal" href="#so-what-should-i-write">So what should I write?</a></li>
</ul>
</li>
<li><a class="reference internal" href="#what-makes-a-good-unit-test">What makes a good unit test?</a><ul>
<li><a class="reference internal" href="#test-data">Test data</a></li>
<li><a class="reference internal" href="#mocking">Mocking</a></li>
<li><a class="reference internal" href="#assertions-and-verification">Assertions and verification</a></li>
</ul>
</li>
<li><a class="reference internal" href="#debugging-unit-tests">Debugging Unit Tests</a><ul>
<li><a class="reference internal" href="#tips-and-tricks">Tips and tricks</a></li>
<li><a class="reference internal" href="#common-pitfalls">Common pitfalls</a></li>
<li><a class="reference internal" href="#understanding-the-output-from-mox">Understanding the output from <tt class="docutils literal"><span class="pre">mox</span></tt></a><ul>
<li><a class="reference internal" href="#unexpected-method-call">Unexpected Method Call</a></li>
<li><a class="reference internal" href="#expected-method-never-called">Expected Method Never Called</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="tables.html"
title="previous chapter">DataTables Topic Guide</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../ref/run_tests.html"
title="next chapter">The <tt class="docutils literal docutils literal docutils literal"><span class="pre">run_tests.sh</span></tt> Script</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/topics/testing.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="../ref/run_tests.html" title="The run_tests.sh Script"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="tables.html" title="DataTables Topic Guide"
accesskey="P">previous</a> |</li>
<li><a href="../index.html">Horizon 2012.2.3 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2012, OpenStack, LLC.
Last updated on Feb 08, 2013.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
</body>
</html>
