<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Writing Test Suites</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/test_server-3.5.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Test Server</strong><br><strong>User's Guide</strong><br><small>Version 3.5.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Test Server Basics" expanded="false">Test Server Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id56984">Introduction</a></li>
<li title="Getting started"><a href="basics_chapter.html#id56907">Getting started</a></li>
<li title="Definition of terms"><a href="basics_chapter.html#id61500">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Test Structure and Test Specifications" expanded="false">Test Structure and Test Specifications<ul>
<li><a href="test_spec_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_spec_chapter.html#id62057">Test structure</a></li>
<li title="Test specifications"><a href="test_spec_chapter.html#id57989">Test specifications</a></li>
<li title="Test Specification Files"><a href="test_spec_chapter.html#id63174">Test Specification Files</a></li>
<li title="Configuration cases"><a href="test_spec_chapter.html#id60965">Configuration cases</a></li>
<li title="The parallel property and nested configuration cases"><a href="test_spec_chapter.html#id56819">The parallel property and nested configuration cases</a></li>
<li title="Repeated execution of test cases"><a href="test_spec_chapter.html#id56843">Repeated execution of test cases</a></li>
<li title="Shuffled test case order"><a href="test_spec_chapter.html#id59974">Shuffled test case order</a></li>
<li title="Skipping test cases"><a href="test_spec_chapter.html#id60025">Skipping test cases</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Writing Test Suites" expanded="true">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id62420">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id62477">Test suites</a></li>
<li title="Init per test case"><a href="write_test_chapter.html#id62508">Init per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id62581">Test cases</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id62718">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id62776">Execution environment</a></li>
</ul>
</li>
<li id="no" title="Running Test Suites" expanded="false">Running Test Suites<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the test server controller"><a href="run_test_chapter.html#id62898">Using the test server controller</a></li>
</ul>
</li>
<li id="no" title="Write you own test server framework" expanded="false">Write you own test server framework<ul>
<li><a href="write_framework_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="write_framework_chapter.html#id62979">Introduction</a></li>
<li title="Interfacing the test server controller from Erlang"><a href="write_framework_chapter.html#id62997">Interfacing the test server controller from Erlang</a></li>
<li title="Interfacing the test server controller from the operating system."><a href="write_framework_chapter.html#id63107">Interfacing the test server controller from the operating system.</a></li>
<li title="Framework callback functions"><a href="write_framework_chapter.html#id63412">Framework callback functions</a></li>
<li title="Other concerns"><a href="write_framework_chapter.html#id63442">Other concerns</a></li>
</ul>
</li>
<li id="no" title="Examples" expanded="false">Examples<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite"><a href="example_chapter.html#id63550">Test suite</a></li>
<li title="Test specification file"><a href="example_chapter.html#id63589">Test specification file</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>3 Writing Test Suites</h1>
<h3><a name="id62420">3.1
Support for test suite authors</a></h3>
<p>The <span class="code">test_server</span> module provides some useful functions
to support the test suite author. This includes:
</p>
<ul>
<li>Starting and stopping slave or peer nodes</li>
<li>Capturing and checking stdout output</li>
<li>Retrieving and flushing process message queue</li>
<li>Watchdog timers</li>
<li>Checking that a function crashes</li>
<li>Checking that a function succeeds at least m out of n times</li>
<li>Checking .app files</li>
</ul>
<p>Please turn to the reference manual for the <span class="code">test_server</span>
module for details about these functions.
</p>
<h3><a name="id62477">3.2
Test suites</a></h3>
<p>A test suite is an ordinary Erlang module that contains test
cases. It's recommended that the module has a name on the form
*_SUITE.erl. Otherwise, the directory function will not find the
modules (by default).
</p>
<p>For some of the test server support, the test server include
file <span class="code">test_server.hrl</span> must be included. Never include it
with the full path, for portability reasons. Use the compiler
include directive instead.
</p>
<p>The special function <span class="code">all(suite)</span> in each module is called
to get the test specification for that module. The function
typically returns a list of test cases in that module, but any
test specification could be returned. Please see the chapter
about test specifications for details about this.
</p>
<h3><a name="id62508">3.3
Init per test case</a></h3>
<p>In each test suite module, the functions
<span class="code">init_per_testcase/2</span> and <span class="code">end_per_testcase/2</span> must be
implemented.
</p>
<p><span class="code">init_per_testcase</span> is called before each test case in the
test suite, giving a (limited) possibility for initialization.
</p>
<p><span class="code">end_per_testcase/2</span> is called after each test case is
completed, giving a possibility to clean up.
</p>
<p>The first argument to these functions is the name of the test
case. This can be used to do individual initialization and cleanup for
each test cases.
</p>
<p>The second argument is a list of tuples called
<span class="code">Config</span>. The first element in a <span class="code">Config</span> tuple
should be an atom - a key value to be used for searching.
<span class="code">init_per_testcase/2</span> may modify the <span class="code">Config</span>
parameter or just return it as is. Whatever is retuned by
<span class="code">init_per_testcase/2</span> is given as <span class="code">Config</span> parameter to
the test case itself.
</p>
<p>The return value of <span class="code">end_per_testcase/2</span> is ignored by the
test server.
</p>
<h3><a name="id62581">3.4
Test cases</a></h3>
<p>The smallest unit that the test server is concerned with is a
test case. Each test case can in turn test many things, for
example make several calls to the same interface function with
different parameters.
</p>
<p>It is possible to put many or few tests into each test
case. How many things each test case tests is up to the author,
but here are some things to keep in mind.
</p>
<p>Very small test cases often leads to more code, since
initialization has to be duplicated. Larger code, especially with
a lot of duplication, increases maintenance and reduces
readability.
</p>
<p>Larger test cases make it harder to tell what went wrong if it
fails, and force us to skip larger portions of test code if a
specific part fails. These effects are accentuated when running on
multiple platforms because test cases often have to be skipped.
</p>
<p>A test case generally consists of three parts, the
documentation part, the specification part and the execution
part. These are implemented as three clauses of the same function.
</p>
<p>The documentation clause matches the argument '<span class="code">doc</span>' and
returns a list for strings describing what the test case tests.
</p>
<p>The specification clause matches the argument '<span class="code">suite</span>'
and returns the test specification for this particular test
case. If the test specification is an empty list, this indicates
that the test case is a leaf test case, i.e. one to be executed.
</p>
<p><strong>Note that the specification clause of a test case is executed on the test server controller host. This means that if target is remote, the specification clause is probably executed on a different platform than the one tested.</strong></p>
<p>The execution clause implements the actual test case. It takes
one argument, <span class="code">Config</span>, which contain configuration
information like <span class="code">data_dir</span> and <span class="code">priv_dir</span>. See <span class="bold_code"><a href="#data_priv_dir">Data and Private Directories</a></span> for
more information about these.
</p>
<p>The <span class="code">Config</span> variable can also contain the
<span class="code">nodenames</span> key, if requested by the <span class="code">require_nodenames</span>
command in the test suite specification file. All <span class="code">Config</span>
items should be extracted using the <span class="code">?config</span> macro. This is
to ensure future compatibility if the <span class="code">Config</span> format
changes. See the reference manual for <span class="code">test_server</span> for
details about this macro.
</p>
<p>If the execution clause crashes or exits, it is considered a
failure. If it returns <span class="code">{skip,Reason}</span>, the test case is
considered skipped. If it returns <span class="code">{comment,String}</span>,
the string will be added in the 'Comment' field on the HTML
result page. If the execution clause returns anything else, it is
considered a success, unless it is <span class="code">{'EXIT',Reason}</span> or
<span class="code">{'EXIT',Pid,Reason}</span> which can't be distinguished from a
crash, and thus will be considered a failure.
</p>
<h3><a name="id62718">3.5
Data and Private Directories</a></h3>
<a name="data_priv_dir"></a>
<p>The data directory (<span class="code">data_dir</span>) is the directory where the test
module has its own files needed for the testing. A compiler test
case may have source files to feed into the compiler, a release
upgrade test case may have some old and new release of
something. A graphics test case may have some icons and a test
case doing a lot of math with bignums might store the correct
answers there. The name of the <span class="code">data_dir</span> is the the name of
the test suite and then "_data". For example,
<span class="code">"some_path/foo_SUITE.beam"</span> has the data directory
<span class="code">"some_path/foo_SUITE_data/"</span>.
</p>
<p>The <span class="code">priv_dir</span> is the test suite's private directory. This
directory should be used when a test case needs to write to
files. The name of the private directory is generated by the test
server, which also creates the directory.
</p>
<p><strong>Warning:</strong> Do not depend on current directory to be
writable, or to point to anything in particular. All scratch files
are to be written in the <span class="code">priv_dir</span>, and all data files found
in <span class="code">data_dir</span>. If the current directory has to be something
specific, it must be set with <span class="code">file:set_cwd/1</span>.
</p>
<h3><a name="id62776">3.6
Execution environment</a></h3>
<p>Each time a test case is about to be executed, a new process is
created with <span class="code">spawn_link</span>. This is so that the test case will
have no dependencies to earlier tests, with respect to process flags,
process links, messages in the queue, other processes having registered
the process, etc. As little as possible is done to change the initial
context of the process (what is created by plain spawn). Here is a
list of differences:
</p>
<ul>
<li>It has a link to the test server. If this link is removed,
the test server will not know when the test case is finished,
just wait infinitely.
</li>
<li>It often holds a few items in the process dictionary, all
with names starting with '<span class="code">test_server_</span>'. This is to keep
track of if/where a test case fails.
</li>
<li>There is a top-level catch. All of the test case code is
catched, so that the location of a crash can be reported back to
the test server. If the test case process is killed by another
process (thus the catch code is never executed) the test server
is not able to tell where the test case was executing.
</li>
<li>It has a special group leader implemented by the test
server. This way the test server is able to capture the io that
the test case provokes. This is also used by some of the test
server support functions.
</li>
</ul>
<p>There is no time limit for a test case, unless the test case
itself imposes such a limit, by calling
<span class="code">test_server:timetrap/1</span> for example. The call can be made
in each test case, or in the <span class="code">init_per_testcase/2</span>
function. Make sure to call the corresponding
<span class="code">test_server:timetrap_cancel/1</span> function as well, e.g in the
<span class="code">end_per_testcase/2</span> function, or else the test cases will
always fail.
</p>
</div>
<div class="footer">
<hr>
<p>Copyright © 2002-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
17
>
i386
>
media
>
updates
>
by-pkgid
>
675c8c8167236dfcf8d66da674f931e8
>
files
>
1486
