<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Dependencies between Test Cases and Suites</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/common_test-1.6.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Common Test</strong><br><strong>User's Guide</strong><br><small>Version 1.6.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Common Test Basics" expanded="false">Common Test Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id66143">Introduction</a></li>
<li title="Test Suite Organisation"><a href="basics_chapter.html#id61412">Test Suite Organisation</a></li>
<li title="Support Libraries"><a href="basics_chapter.html#id67203">Support Libraries</a></li>
<li title="Suites and Test Cases"><a href="basics_chapter.html#id61632">Suites and Test Cases</a></li>
<li title="External Interfaces"><a href="basics_chapter.html#id63930">External Interfaces</a></li>
</ul>
</li>
<li id="no" title="Getting Started" expanded="false">Getting Started<ul>
<li><a href="getting_started_chapter.html">
Top of chapter
</a></li>
<li title="Are you new around here?"><a href="getting_started_chapter.html#id61356">Are you new around here?</a></li>
<li title="Test case execution"><a href="getting_started_chapter.html#id63050">Test case execution</a></li>
<li title="A simple test suite"><a href="getting_started_chapter.html#id63909">A simple test suite</a></li>
<li title="A test suite with configuration functions"><a href="getting_started_chapter.html#id68094">A test suite with configuration functions</a></li>
<li title="What happens next?"><a href="getting_started_chapter.html#id62545">What happens next?</a></li>
</ul>
</li>
<li id="no" title="Installation" expanded="false">Installation<ul>
<li><a href="install_chapter.html">
Top of chapter
</a></li>
<li title="General information"><a href="install_chapter.html#id66699">General information</a></li>
</ul>
</li>
<li id="no" title="Writing Test Suites" expanded="false">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id71900">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id70761">Test suites</a></li>
<li title="Init and end per suite"><a href="write_test_chapter.html#id70806">Init and end per suite</a></li>
<li title="Init and end per test case"><a href="write_test_chapter.html#id70917">Init and end per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id71162">Test cases</a></li>
<li title="Test case info function"><a href="write_test_chapter.html#id71322">Test case info function</a></li>
<li title="Test suite info function"><a href="write_test_chapter.html#id71521">Test suite info function</a></li>
<li title="Test case groups"><a href="write_test_chapter.html#id71611">Test case groups</a></li>
<li title="The parallel property and nested groups"><a href="write_test_chapter.html#id72823">The parallel property and nested groups</a></li>
<li title="Parallel test cases and IO"><a href="write_test_chapter.html#id72860">Parallel test cases and IO</a></li>
<li title="Repeated groups"><a href="write_test_chapter.html#id72908">Repeated groups</a></li>
<li title="Shuffled test case order"><a href="write_test_chapter.html#id73046">Shuffled test case order</a></li>
<li title="Group info function"><a href="write_test_chapter.html#id73105">Group info function</a></li>
<li title="Info functions for init- and end-configuration"><a href="write_test_chapter.html#id73139">Info functions for init- and end-configuration</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id73202">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id73327">Execution environment</a></li>
<li title="Timetrap timeouts"><a href="write_test_chapter.html#id73364">Timetrap timeouts</a></li>
<li title="Logging - categories and verbosity levels"><a href="write_test_chapter.html#id73549">Logging - categories and verbosity levels</a></li>
<li title="Illegal dependencies"><a href="write_test_chapter.html#id73756">Illegal dependencies</a></li>
</ul>
</li>
<li id="no" title="Test Structure" expanded="false">Test Structure<ul>
<li><a href="test_structure_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_structure_chapter.html#id73930">Test structure</a></li>
<li title="Skipping test cases"><a href="test_structure_chapter.html#id73944">Skipping test cases</a></li>
<li title="Definition of terms"><a href="test_structure_chapter.html#id74020">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Examples and Templates" expanded="false">Examples and Templates<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite example"><a href="example_chapter.html#id74246">Test suite example</a></li>
<li title="Test suite templates"><a href="example_chapter.html#id74298">Test suite templates</a></li>
</ul>
</li>
<li id="no" title="Running Tests" expanded="false">Running Tests<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the Common Test Framework"><a href="run_test_chapter.html#id74488">Using the Common Test Framework</a></li>
<li title="Automatic compilation of test suites and help modules"><a href="run_test_chapter.html#id74528">Automatic compilation of test suites and help modules</a></li>
<li title="Running tests from the OS command line"><a href="run_test_chapter.html#id74645">Running tests from the OS command line</a></li>
<li title="Running tests from the Erlang shell or from an Erlang program"><a href="run_test_chapter.html#id75142">Running tests from the Erlang shell or from an Erlang program</a></li>
<li title="Test case group execution"><a href="run_test_chapter.html#id75304">Test case group execution</a></li>
<li title="Running the interactive shell mode"><a href="run_test_chapter.html#id75821">Running the interactive shell mode</a></li>
<li title="Step by step execution of test cases with the Erlang Debugger"><a href="run_test_chapter.html#id76003">Step by step execution of test cases with the Erlang Debugger</a></li>
<li title="Test Specifications"><a href="run_test_chapter.html#id76102">Test Specifications</a></li>
<li title="Running tests from the Web based GUI"><a href="run_test_chapter.html#id76643">Running tests from the Web based GUI</a></li>
<li title="Log files"><a href="run_test_chapter.html#id76735">Log files</a></li>
<li title="HTML Style Sheets"><a href="run_test_chapter.html#id77053">HTML Style Sheets</a></li>
<li title="Repeating tests"><a href="run_test_chapter.html#id77201">Repeating tests</a></li>
<li title="Silent Connections"><a href="run_test_chapter.html#id77412">Silent Connections</a></li>
</ul>
</li>
<li id="no" title="External Configuration Data" expanded="false">External Configuration Data<ul>
<li><a href="config_file_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="config_file_chapter.html#id77631">General</a></li>
<li title="Syntax"><a href="config_file_chapter.html#id77671">Syntax</a></li>
<li title="Requiring and reading configuration data"><a href="config_file_chapter.html#id77695">Requiring and reading configuration data</a></li>
<li title="Using configuration variables defined in multiple files"><a href="config_file_chapter.html#id77839">Using configuration variables defined in multiple files</a></li>
<li title="Encrypted configuration files"><a href="config_file_chapter.html#id77870">Encrypted configuration files</a></li>
<li title="Opening connections by using configuration data"><a href="config_file_chapter.html#id77934">Opening connections by using configuration data</a></li>
<li title="User specific configuration data formats"><a href="config_file_chapter.html#id78002">User specific configuration data formats</a></li>
<li title="Examples of configuration data handling"><a href="config_file_chapter.html#id78216">Examples of configuration data handling</a></li>
<li title="Example of user specific configuration handler"><a href="config_file_chapter.html#id78268">Example of user specific configuration handler</a></li>
</ul>
</li>
<li id="no" title="Code Coverage Analysis" expanded="false">Code Coverage Analysis<ul>
<li><a href="cover_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="cover_chapter.html#id78384">General</a></li>
<li title="Usage"><a href="cover_chapter.html#id78404">Usage</a></li>
<li title="The cover specification file"><a href="cover_chapter.html#id78510">The cover specification file</a></li>
<li title="Logging"><a href="cover_chapter.html#id78563">Logging</a></li>
</ul>
</li>
<li id="no" title="Using Common Test for Large Scale Testing" expanded="false">Using Common Test for Large Scale Testing<ul>
<li><a href="ct_master_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_master_chapter.html#id78634">General</a></li>
<li title="Usage"><a href="ct_master_chapter.html#id78668">Usage</a></li>
<li title="Test Specifications"><a href="ct_master_chapter.html#id78815">Test Specifications</a></li>
<li title="Automatic startup of test target nodes"><a href="ct_master_chapter.html#id78987">Automatic startup of test target nodes</a></li>
</ul>
</li>
<li id="no" title="Event Handling" expanded="false">Event Handling<ul>
<li><a href="event_handler_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="event_handler_chapter.html#id79190">General</a></li>
<li title="Usage"><a href="event_handler_chapter.html#id79226">Usage</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Dependencies between Test Cases and Suites" expanded="true">Dependencies between Test Cases and Suites<ul>
<li><a href="dependencies_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="dependencies_chapter.html#id80143">General</a></li>
<li title="Saving configuration data"><a href="dependencies_chapter.html#id80265">Saving configuration data</a></li>
<li title="Sequences"><a href="dependencies_chapter.html#id80430">Sequences</a></li>
</ul>
</li>
<li id="no" title="Common Test Hooks" expanded="false">Common Test Hooks<ul>
<li><a href="ct_hooks_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_hooks_chapter.html#id80622">General</a></li>
<li title="Installing a CTH"><a href="ct_hooks_chapter.html#id80676">Installing a CTH</a></li>
<li title="CTH Scope"><a href="ct_hooks_chapter.html#id80835">CTH Scope</a></li>
<li title="Manipulating tests"><a href="ct_hooks_chapter.html#id81189">Manipulating tests</a></li>
<li title="Example CTH"><a href="ct_hooks_chapter.html#id81482">Example CTH</a></li>
<li title="Built-in CTHs"><a href="ct_hooks_chapter.html#id81535">Built-in CTHs</a></li>
</ul>
</li>
<li id="no" title="Some thoughts about testing" expanded="false">Some thoughts about testing<ul>
<li><a href="why_test_chapter.html">
Top of chapter
</a></li>
<li title="Goals"><a href="why_test_chapter.html#id81722">Goals</a></li>
<li title="What to test?"><a href="why_test_chapter.html#id81742">What to test?</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>12 Dependencies between Test Cases and Suites</h1>
<h3><a name="id80143">12.1
General</a></h3>
<p>When creating test suites, it is strongly recommended to not
create dependencies between test cases, i.e. letting test cases
depend on the result of previous test cases. There are various
reasons for this, for example:</p>
<ul>
<li>It makes it impossible to run test cases individually.</li>
<li>It makes it impossible to run test cases in different order.</li>
<li>It makes debugging very difficult (since a fault could be
the result of a problem in a different test case than the one failing).</li>
<li>There exists no good and explicit ways to declare dependencies, so
it may be very difficult to see and understand these in test suite
code and in test logs.</li>
<li>Extending, restructuring and maintaining test suites with
test case dependencies is difficult.</li>
</ul>
<p>There are often sufficient means to work around the need for test
case dependencies. Generally, the problem is related to the state of
the system under test (SUT). The action of one test case may alter the state
of the system and for some other test case to run properly, the new state
must be known.</p>
<p>Instead of passing data between test cases, it is recommended
that the test cases read the state from the SUT and perform assertions
(i.e. let the test case run if the state is as expected, otherwise reset or fail)
and/or use the state to set variables necessary for the test case to execute
properly. Common actions can often be implemented as library functions for
test cases to call to set the SUT in a required state. (Such common actions
may of course also be separately tested if necessary, to ensure they are
working as expected). It is sometimes also possible, but not always desirable,
to group tests together in one test case, i.e. let a test case perform a
"scenario" test (a test that consists of subtests).</p>
<p>Consider for example a server application under test. The following
functionality is to be tested:</p>
<ul>
<li>Starting the server.</li>
<li>Configuring the server.</li>
<li>Connecting a client to the server.</li>
<li>Disconnecting a client from the server.</li>
<li>Stopping the server.</li>
</ul>
<p>There are obvious dependencies between the listed functions. We can't configure
the server if it hasn't first been started, we can't connect a client until
the server has been properly configured, etc. If we want to have one test
case for each of the functions, we might be tempted to try to always run the
test cases in the stated order and carry possible data (identities, handles,
etc) between the cases and therefore introduce dependencies between them.
To avoid this we could consider starting and stopping the server for every test.
We would implement the start and stop action as common functions that may be
called from init_per_testcase and end_per_testcase. (We would of course test
the start and stop functionality separately). The configuration could perhaps also
be implemented as a common function, maybe grouped with the start function.
Finally the testing of connecting and disconnecting a client may be grouped into
one test case. The resulting suite would look something like this:</p>
<div class="example"><pre>
-module(my_server_SUITE).
-compile(export_all).
-include_lib("ct.hrl").
%%% init and end functions...
suite() -> [{require,my_server_cfg}].
init_per_testcase(start_and_stop, Config) ->
Config;
init_per_testcase(config, Config) ->
[{server_pid,start_server()} | Config];
init_per_testcase(_, Config) ->
ServerPid = start_server(),
configure_server(),
[{server_pid,ServerPid} | Config].
end_per_testcase(start_and_stop, _) ->
ok;
end_per_testcase(_, _) ->
ServerPid = ?config(server_pid),
stop_server(ServerPid).
%%% test cases...
all() -> [start_and_stop, config, connect_and_disconnect].
%% test that starting and stopping works
start_and_stop(_) ->
ServerPid = start_server(),
stop_server(ServerPid).
%% configuration test
config(Config) ->
ServerPid = ?config(server_pid, Config),
configure_server(ServerPid).
%% test connecting and disconnecting client
connect_and_disconnect(Config) ->
ServerPid = ?config(server_pid, Config),
{ok,SessionId} = my_server:connect(ServerPid),
ok = my_server:disconnect(ServerPid, SessionId).
%%% common functions...
start_server() ->
{ok,ServerPid} = my_server:start(),
ServerPid.
stop_server(ServerPid) ->
ok = my_server:stop(),
ok.
configure_server(ServerPid) ->
ServerCfgData = ct:get_config(my_server_cfg),
ok = my_server:configure(ServerPid, ServerCfgData),
ok.
</pre></div>
<h3><a name="id80265">12.2
Saving configuration data</a></h3>
<a name="save_config"></a>
<p>There might be situations where it is impossible, or infeasible at least, to
implement independent test cases. Maybe it is simply not possible to read the
SUT state. Maybe resetting the SUT is impossible and it takes much too long
to restart the system. In situations where test case dependency is necessary,
CT offers a structured way to carry data from one test case to the next. The
same mechanism may also be used to carry data from one test suite to the next.</p>
<p>The mechanism for passing data is called <span class="code">save_config</span>. The idea is that
one test case (or suite) may save the current value of Config - or any list of
key-value tuples - so that it can be read by the next executing test case
(or test suite). The configuration data is not saved permanently but can only
be passed from one case (or suite) to the next.</p>
<p>To save <span class="code">Config</span> data, return the tuple:</p>
<p><span class="code">{save_config,ConfigList}</span></p>
<p>from <span class="code">end_per_testcase</span> or from the main test case function. To read data
saved by a previous test case, use the <span class="code">config</span> macro with a
<span class="code">saved_config</span> key:</p>
<p><span class="code">{Saver,ConfigList} = ?config(saved_config, Config)</span></p>
<p><span class="code">Saver</span> (<span class="code">atom()</span>) is the name of the previous test case (where the
data was saved). The <span class="code">config</span> macro may be used to extract particular data
also from the recalled <span class="code">ConfigList</span>. It is strongly recommended that
<span class="code">Saver</span> is always matched to the expected name of the saving test case.
This way problems due to restructuring of the test suite may be avoided. Also it
makes the dependency more explicit and the test suite easier to read and maintain.</p>
<p>To pass data from one test suite to another, the same mechanism is used. The data
should be saved by the <span class="code">end_per_suite</span> function and read by <span class="code">init_per_suite</span>
in the suite that follows. When passing data between suites, <span class="code">Saver</span> carries the
name of the test suite.</p>
<p>Example:</p>
<div class="example"><pre>
-module(server_b_SUITE).
-compile(export_all).
-include_lib("ct.hrl").
%%% init and end functions...
init_per_suite(Config) ->
%% read config saved by previous test suite
{server_a_SUITE,OldConfig} = ?config(saved_config, Config),
%% extract server identity (comes from server_a_SUITE)
ServerId = ?config(server_id, OldConfig),
SessionId = connect_to_server(ServerId),
[{ids,{ServerId,SessionId}} | Config].
end_per_suite(Config) ->
%% save config for server_c_SUITE (session_id and server_id)
{save_config,Config}
%%% test cases...
all() -> [allocate, deallocate].
allocate(Config) ->
{ServerId,SessionId} = ?config(ids, Config),
{ok,Handle} = allocate_resource(ServerId, SessionId),
%% save handle for deallocation test
NewConfig = [{handle,Handle}],
{save_config,NewConfig}.
deallocate(Config) ->
{ServerId,SessionId} = ?config(ids, Config),
{allocate,OldConfig} = ?config(saved_config, Config),
Handle = ?config(handle, OldConfig),
ok = deallocate_resource(ServerId, SessionId, Handle).
</pre></div>
<p>It is also possible to save <span class="code">Config</span> data from a test case that is to be
skipped. To accomplish this, return the following tuple:</p>
<p><span class="code">{skip_and_save,Reason,ConfigList}</span></p>
<p>The result will be that the test case is skipped with <span class="code">Reason</span> printed to
the log file (as described in previous chapters), and <span class="code">ConfigList</span> is saved
for the next test case. <span class="code">ConfigList</span> may be read by means of
<span class="code">?config(saved_config, Config)</span>, as described above. <span class="code">skip_and_save</span>
may also be returned from <span class="code">init_per_suite</span>, in which case the saved data can
be read by <span class="code">init_per_suite</span> in the suite that follows.</p>
<h3><a name="id80430">12.3
Sequences</a></h3>
<a name="sequences"></a>
<p>It is possible that test cases depend on each other so that
if one case fails, the following test(s) should not be executed.
Typically, if the <span class="code">save_config</span> facility is used and a test
case that is expected to save data crashes, the following
case can not run. CT offers a way to declare such dependencies,
called sequences.</p>
<p>A sequence of test cases is defined as a test case group
with a <span class="code">sequence</span> property. Test case groups are defined by
means of the <span class="code">groups/0</span> function in the test suite (see the
<span class="bold_code"><a href="write_test_chapter.html#test_case_groups">Test case groups</a></span>
chapter for details).</p>
<p>For example, if we would like to make sure that if <span class="code">allocate</span>
in <span class="code">server_b_SUITE</span> (above) crashes, <span class="code">deallocate</span> is skipped,
we may define a sequence like this:</p>
<div class="example"><pre>
groups() -> [{alloc_and_dealloc, [sequence], [alloc,dealloc]}].</pre></div>
<p>Let's also assume the suite contains the test case <span class="code">get_resource_status</span>,
which is independent of the other two cases, then the <span class="code">all</span> function could
look like this:</p>
<div class="example"><pre>
all() -> [{group,alloc_and_dealloc}, get_resource_status].</pre></div>
<p>If <span class="code">alloc</span> succeeds, <span class="code">dealloc</span> is also executed. If <span class="code">alloc</span> fails
however, <span class="code">dealloc</span> is not executed but marked as SKIPPED in the html log.
<span class="code">get_resource_status</span> will run no matter what happens to the <span class="code">alloc_and_dealloc</span>
cases.</p>
<p>Test cases in a sequence will be executed in order until they have all succeeded or
until one case fails. If one fails, all following cases in the sequence are skipped.
The cases in the sequence that have succeeded up to that point are reported as successful
in the log. An arbitrary number of sequences may be specified. Example:</p>
<div class="example"><pre>
groups() -> [{scenarioA, [sequence], [testA1, testA2]},
{scenarioB, [sequence], [testB1, testB2, testB3]}].
all() -> [test1,
test2,
{group,scenarioA},
test3,
{group,scenarioB},
test4].</pre></div>
<p>It is possible to have sub-groups in a sequence group. Such sub-groups can have
any property, i.e. they are not required to also be sequences. If you want the status
of the sub-group to affect the sequence on the level above, return
<span class="code">{return_group_result,Status}</span> from <span class="code">end_per_group/2</span>, as described in the
<span class="bold_code"><a href="write_test_chapter.html#repeated_groups">Repeated groups</a></span>
chapter. A failed sub-group (<span class="code">Status == failed</span>) will cause the execution of a
sequence to fail in the same way a test case does.</p>
</div>
<div class="footer">
<hr>
<p>Copyright © 2003-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
17
>
i386
>
media
>
updates
>
by-pkgid
>
675c8c8167236dfcf8d66da674f931e8
>
files
>
249
