<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Test Structure and Test Specifications</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/test_server-3.5.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Test Server</strong><br><strong>User's Guide</strong><br><small>Version 3.5.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Test Server Basics" expanded="false">Test Server Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id56984">Introduction</a></li>
<li title="Getting started"><a href="basics_chapter.html#id56907">Getting started</a></li>
<li title="Definition of terms"><a href="basics_chapter.html#id61500">Definition of terms</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Test Structure and Test Specifications" expanded="true">Test Structure and Test Specifications<ul>
<li><a href="test_spec_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_spec_chapter.html#id62057">Test structure</a></li>
<li title="Test specifications"><a href="test_spec_chapter.html#id57989">Test specifications</a></li>
<li title="Test Specification Files"><a href="test_spec_chapter.html#id63174">Test Specification Files</a></li>
<li title="Configuration cases"><a href="test_spec_chapter.html#id60965">Configuration cases</a></li>
<li title="The parallel property and nested configuration cases"><a href="test_spec_chapter.html#id56819">The parallel property and nested configuration cases</a></li>
<li title="Repeated execution of test cases"><a href="test_spec_chapter.html#id56843">Repeated execution of test cases</a></li>
<li title="Shuffled test case order"><a href="test_spec_chapter.html#id59974">Shuffled test case order</a></li>
<li title="Skipping test cases"><a href="test_spec_chapter.html#id60025">Skipping test cases</a></li>
</ul>
</li>
<li id="no" title="Writing Test Suites" expanded="false">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id62420">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id62477">Test suites</a></li>
<li title="Init per test case"><a href="write_test_chapter.html#id62508">Init per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id62581">Test cases</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id62718">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id62776">Execution environment</a></li>
</ul>
</li>
<li id="no" title="Running Test Suites" expanded="false">Running Test Suites<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the test server controller"><a href="run_test_chapter.html#id62898">Using the test server controller</a></li>
</ul>
</li>
<li id="no" title="Write you own test server framework" expanded="false">Write you own test server framework<ul>
<li><a href="write_framework_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="write_framework_chapter.html#id62979">Introduction</a></li>
<li title="Interfacing the test server controller from Erlang"><a href="write_framework_chapter.html#id62997">Interfacing the test server controller from Erlang</a></li>
<li title="Interfacing the test server controller from the operating system."><a href="write_framework_chapter.html#id63107">Interfacing the test server controller from the operating system.</a></li>
<li title="Framework callback functions"><a href="write_framework_chapter.html#id63412">Framework callback functions</a></li>
<li title="Other concerns"><a href="write_framework_chapter.html#id63442">Other concerns</a></li>
</ul>
</li>
<li id="no" title="Examples" expanded="false">Examples<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite"><a href="example_chapter.html#id63550">Test suite</a></li>
<li title="Test specification file"><a href="example_chapter.html#id63589">Test specification file</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>2 Test Structure and Test Specifications</h1>
<h3><a name="id62057">2.1
Test structure</a></h3>
<p>A test consists of a set of test cases. Each test case is
implemented as an erlang function. An erlang module implementing
one or more test cases is called a test suite.
</p>
<h3><a name="id57989">2.2
Test specifications</a></h3>
<p>A test specification is a specification of which test suites
and test cases to run and which to skip. A test specification can
also group several test cases into conf cases with init and
cleanup functions (see section about configuration cases
below). In a test there can be test specifications on three
different levels:
</p>
<p>The top level is a test specification file which roughly
specifies what to test for a whole application. The test
specification in such a file is encapsulated in a topcase
command.
</p>
<p>Then there is a test specification for each test suite,
specifying which test cases to run within the suite. The test
specification for a test suite is returned from the
<span class="code">all(suite)</span> function in the test suite module.
</p>
<p>And finally there can be a test specification per test case,
specifying sub test cases to run. The test specification for a
test case is returned from the specification clause of the test
case.
</p>
<p>When a test starts, the total test specification is built in a
tree fashion, starting from the top level test specification.
</p>
<p>The following are the valid elements of a test
specification. The specification can be one of these elements or a
list with any combination of the elements:
</p>
<dl>
<dt><strong><span class="code">{Mod, Case}</span></strong></dt>
<dd>This specifies the test case Mod:Case/1
</dd>
<dt><strong><span class="code">{dir, Dir}</span></strong></dt>
<dd>This specifies all modules <span class="code">*_SUITE</span> in the directory
<span class="code">Dir</span>
</dd>
<dt><strong><span class="code">{dir, Dir, Pattern}</span></strong></dt>
<dd>This specifies all modules <span class="code">Pattern*</span> in the
directory <span class="code">Dir</span>
</dd>
<dt><strong><span class="code">{conf, Init, TestSpec, Fin}</span></strong></dt>
<dd>This is a configuration case. In a test specification
file, <span class="code">Init</span> and <span class="code">Fin</span> must be
<span class="code">{Mod,Func}</span>. Inside a module they can also be just
<span class="code">Func</span>. See the section named Configuration Cases below for
more information about this.
</dd>
<dt><strong><span class="code">{conf, Properties, Init, TestSpec, Fin}</span></strong></dt>
<dd>This is a configuration case as explained above, but
which also takes a list of execution properties for its group
of test cases and nested sub-groups.
</dd>
<dt><strong><span class="code">{make, Init, TestSpec, Fin}</span></strong></dt>
<dd>This is a special version of a conf case which is only
used by the test server framework <span class="code">ts</span>. <span class="code">Init</span> and
<span class="code">Fin</span> are make and unmake functions for a data
directory. <span class="code">TestSpec</span> is the test specification for the
test suite owning the data directory in question. If the make
function fails, all tests in the test suite are skipped. The
difference between this "make case" and a normal conf case is
that for the make case, <span class="code">Init</span> and <span class="code">Fin</span> are given with
arguments (<span class="code">{Mod,Func,Args}</span>), and that they are executed
on the controller node (i.e. not on target).
</dd>
<dt><strong><span class="code">Case</span></strong></dt>
<dd>This can only be used inside a module, i.e. not a test
specification file. It specifies the test case
<span class="code">CurrentModule:Case</span>.
</dd>
</dl>
<h3><a name="id63174">2.3
Test Specification Files</a></h3>
<p>A test specification file is a text file containing the top
level test specification (a topcase command), and possibly one or
more additional commands. A "command" in a test specification file
means a key-value tuple ended by a dot-newline sequence.
</p>
<p>The following commands are valid:
</p>
<dl>
<dt><strong><span class="code">{topcase, TestSpec}</span></strong></dt>
<dd>This command is mandatory in all test specification
files. <span class="code">TestSpec</span> is the top level test specification of a
test.
</dd>
<dt><strong><span class="code">{skip, {Mod, Comment}}</span></strong></dt>
<dd>This specifies that all cases in the module <span class="code">Mod</span>
shall be skipped. <span class="code">Comment</span> is a string.
</dd>
<dt><strong><span class="code">{skip, {Mod, Case, Comment}}</span></strong></dt>
<dd>This specifies that the case <span class="code">Mod:Case</span> shall be
skipped.
</dd>
<dt><strong><span class="code">{skip, {Mod, CaseList, Comment}}</span></strong></dt>
<dd>This specifies that all cases <span class="code">Mod:Case</span>, where
<span class="code">Case</span> is in <span class="code">CaseList</span>, shall be skipped.
</dd>
<dt><strong><span class="code">{nodes, Nodes}</span></strong></dt>
<dd>
<span class="code">Nodes</span> is a list of nodenames available to the test
suite. It will be added to the <span class="code">Config</span> argument to all
test cases. <span class="code">Nodes</span> is a list of atoms.
</dd>
<dt><strong><span class="code">{require_nodenames, Num}</span></strong></dt>
<dd>Specifies how many nodenames the test suite will
need. Theese will be automatically generated and inserted into the
<span class="code">Config</span> argument to all test cases. <span class="code">Num</span> is an
integer.
</dd>
<dt><strong><span class="code">{hosts, Hosts}</span></strong></dt>
<dd>This is a list of available hosts on which to start slave
nodes. It is used when the <span class="code">{remote, true}</span> option is given
to the <span class="code">test_server:start_node/3</span> function. Also, if
<span class="code">{require_nodenames, Num}</span> is contained in a test
specification file, the generated nodenames will be spread over
all hosts given in this <span class="code">Hosts</span> list. The hostnames are
atoms or strings.
</dd>
<dt><strong><span class="code">{diskless, true}</span></strong></dt>
<dd>Adds <span class="code">{diskless, true}</span> to the <span class="code">Config</span> argument
to all test cases. This is kept for backwards compatibility and
should not be used. Use a configuration case instead.
</dd>
<dt><strong><span class="code">{ipv6_hosts, Hosts}</span></strong></dt>
<dd>Adds <span class="code">{ipv6_hosts, Hosts}</span> to the <span class="code">Config</span>
argument to all test cases.</dd>
</dl>
<p>All test specification files shall have the extension
".spec". If special test specification files are needed for
Windows or VxWorks platforms, additional files with the
extension ".spec.win" and ".spec.vxworks" shall be
used. This is useful e.g. if some test cases shall be skipped on
these platforms.
</p>
<p>Some examples for test specification files can be found in the
Examples section of this user's guide.
</p>
<h3><a name="id60965">2.4
Configuration cases</a></h3>
<p>If a group of test cases need the same initialization, a so called
<strong>configuration</strong> or <strong>conf</strong> case can be used. A conf
case consists of an initialization function, the group of test cases
needing this initialization and a cleanup or finalization function.
</p>
<p>If the init function in a conf case fails or returns
<span class="code">{skip,Comment}</span>, the rest of the test cases in the conf case
(including the cleanup function) are skipped. If the init function
succeeds, the cleanup function will always be called, even if some
of the test cases in between failed.
</p>
<p>Both the init function and the cleanup function in a conf case
get the <span class="code">Config</span> parameter as only argument. This parameter
can be modified or returned as is. Whatever is returned by the
init function is given as <span class="code">Config</span> parameter to the rest of
the test cases in the conf case, including the cleanup function.
</p>
<p>If the <span class="code">Config</span> parameter is changed by the init function,
it must be restored by the cleanup function. Whatever is returned
by the cleanup function will be given to the next test case called.
</p>
<p>The optional <span class="code">Properties</span> list can be used to specify
execution properties for the test cases and possibly nested
sub-groups of the configuration case. The available properties are:</p>
<div class="example"><pre>
Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
Shuffle = shuffle | {shuffle,Seed}
Seed = {integer(),integer(),integer()}
RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
repeat_until_any_ok | repeat_until_any_fail
N = integer() | forever</pre></div>
<p>If the <span class="code">parallel</span> property is specified, Test Server will execute
all test cases in the group in parallel. If <span class="code">sequence</span> is specified,
the cases will be executed in a sequence, meaning if one case fails, all
following cases will be skipped. If <span class="code">shuffle</span> is specified, the cases
in the group will be executed in random order. The <span class="code">repeat</span> property
orders Test Server to repeat execution of the cases in the group a given
number of times, or until any, or all, cases fail or succeed.</p>
<p>Properties may be combined so that e.g. if <span class="code">shuffle</span>,
<span class="code">repeat_until_any_fail</span> and <span class="code">sequence</span> are all specified, the test
cases in the group will be executed repeatedly and in random order until
a test case fails, when execution is immediately stopped and the rest of
the cases skipped.</p>
<p>The properties for a conf case is always printed on the top of the HTML log
for the group's init function. Also, the total execution time for a conf case
can be found at the bottom of the log for the group's end function.</p>
<p>Configuration cases may be nested so that sets of grouped cases can be
configured with the same init- and end functions.</p>
<h3><a name="id56819">2.5
The parallel property and nested configuration cases</a></h3>
<p>If a conf case has a parallel property, its test cases will be spawned
simultaneously and get executed in parallel. A test case is not allowed
to execute in parallel with the end function however, which means
that the time it takes to execute a set of parallel cases is equal to the
execution time of the slowest test case in the group. A negative side
effect of running test cases in parallel is that the HTML summary pages
are not updated with links to the individual test case logs until the
end function for the conf case has finished.</p>
<p>A conf case nested under a parallel conf case will start executing in
parallel with previous (parallel) test cases (no matter what properties the
nested conf case has). Since, however, test cases are never executed in
parallel with the init- or the end function of the same conf case, it's
only after a nested group of cases has finished that any remaining parallel
cases in the previous conf case get spawned.</p>
<h3><a name="id56843">2.6
Repeated execution of test cases</a></h3>
<a name="repeated_cases"></a>
<p>A conf case may be repeated a certain number of times
(specified by an integer) or indefinitely (specified by <span class="code">forever</span>).
The repetition may also be stopped prematurely if any or all cases
fail or succeed, i.e. if the property <span class="code">repeat_until_any_fail</span>,
<span class="code">repeat_until_any_ok</span>, <span class="code">repeat_until_all_fail</span>, or
<span class="code">repeat_until_all_ok</span> is used. If the basic <span class="code">repeat</span>
property is used, status of test cases is irrelevant for the repeat
operation.</p>
<p>It is possible to return the status of a conf case (ok or
failed), to affect the execution of the conf case on the level above.
This is accomplished by, in the end function, looking up the value
of <span class="code">tc_group_properties</span> in the <span class="code">Config</span> list and checking the
result of the finished test cases. If status <span class="code">failed</span> should be
returned from the conf case as a result, the end function should return
the value <span class="code">{return_group_result,failed}</span>. The status of a nested conf
case is taken into account by Test Server when deciding if execution
should be repeated or not (unless the basic <span class="code">repeat</span> property is used).</p>
<p>The <span class="code">tc_group_properties</span> value is a list of status tuples,
each with the key <span class="code">ok</span>, <span class="code">skipped</span> and <span class="code">failed</span>. The
value of a status tuple is a list containing names of test cases
that have been executed with the corresponding status as result.</p>
<p>Here's an example of how to return the status from a conf case:</p>
<div class="example"><pre>
conf_end_function(Config) ->
Status = ?config(tc_group_result, Config),
case proplists:get_value(failed, Status) of
[] -> % no failed cases
{return_group_result,ok};
_Failed -> % one or more failed
{return_group_result,failed}
end.</pre></div>
<p>It is also possible in the end function to check the status of
a nested conf case (maybe to determine what status the current conf case should
return). This is as simple as illustrated in the example above, only the
name of the end function of the nested conf case is stored in a tuple
<span class="code">{group_result,EndFunc}</span>, which can be searched for in the status lists.
Example:</p>
<div class="example"><pre>
conf_end_function_X(Config) ->
Status = ?config(tc_group_result, Config),
Failed = proplists:get_value(failed, Status),
case lists:member({group_result,conf_end_function_Y}, Failed) of
true ->
{return_group_result,failed};
false ->
{return_group_result,ok}
end;
...</pre></div>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>When a conf case is repeated, the init- and end functions
are also always called with each repetition.</p></p></div>
</div>
<h3><a name="id59974">2.7
Shuffled test case order</a></h3>
<p>The order that test cases in a conf case are executed, is under normal
circumstances the same as the order defined in the test specification.
With the <span class="code">shuffle</span> property set, however, Test Server will instead
execute the test cases in random order.</p>
<p>The user may provide a seed value (a tuple of three integers) with
the shuffle property: <span class="code">{shuffle,Seed}</span>. This way, the same shuffling
order can be created every time the conf case is executed. If no seed value
is given, Test Server creates a "random" seed for the shuffling operation
(using the return value of <span class="code">erlang:now()</span>). The seed value is always
printed to the log file of the init function so that it can be used to
recreate the same execution order in subsequent test runs.</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>If execution of a conf case with shuffled test cases is repeated,
the seed will not be reset in between turns.</p></p></div>
</div>
<p>If a nested conf case is specified in a conf case with a <span class="code">shuffle</span>
property, the execution order of the nested cases in relation to the test cases
(and other conf cases) is also random. The order of the test cases in the nested
conf case is however not random (unless, of course, this one also has a
<span class="code">shuffle</span> property).</p>
<h3><a name="id60025">2.8
Skipping test cases</a></h3>
<p>It is possible to skip certain test cases, for example if you
know beforehand that a specific test case fails. This might be
functionality which isn't yet implemented, a bug that is known but
not yet fixed or some functionality which doesn't work or isn't
applicable on a specific platform.
</p>
<p>There are several different ways to state that a test case
should be skipped:</p>
<ul>
<li>Using the <span class="code">{skip,What}</span> command in a test
specification file
</li>
<li>Returning <span class="code">{skip,Reason}</span> from the
<span class="code">init_per_testcase/2</span> function
</li>
<li>Returning <span class="code">{skip,Reason}</span> from the specification
clause of the test case
</li>
<li>Returning <span class="code">{skip,Reason}</span> from the execution clause
of the test case
</li>
</ul>
<p>The latter of course means that the execution clause is
actually called, so the author must make sure that the test case
is not run. For more information about the different clauses in a
test case, see the chapter about writing test cases.
</p>
<p>When a test case is skipped, it will be noted as <span class="code">SKIPPED</span>
in the HTML log.
</p>
</div>
<div class="footer">
<hr>
<p>Copyright © 2002-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
17
>
i386
>
media
>
updates
>
by-pkgid
>
675c8c8167236dfcf8d66da674f931e8
>
files
>
1483
