<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Writing Test Suites</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/common_test-1.6.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Common Test</strong><br><strong>User's Guide</strong><br><small>Version 1.6.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Common Test Basics" expanded="false">Common Test Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id66143">Introduction</a></li>
<li title="Test Suite Organisation"><a href="basics_chapter.html#id61412">Test Suite Organisation</a></li>
<li title="Support Libraries"><a href="basics_chapter.html#id67203">Support Libraries</a></li>
<li title="Suites and Test Cases"><a href="basics_chapter.html#id61632">Suites and Test Cases</a></li>
<li title="External Interfaces"><a href="basics_chapter.html#id63930">External Interfaces</a></li>
</ul>
</li>
<li id="no" title="Getting Started" expanded="false">Getting Started<ul>
<li><a href="getting_started_chapter.html">
Top of chapter
</a></li>
<li title="Are you new around here?"><a href="getting_started_chapter.html#id61356">Are you new around here?</a></li>
<li title="Test case execution"><a href="getting_started_chapter.html#id63050">Test case execution</a></li>
<li title="A simple test suite"><a href="getting_started_chapter.html#id63909">A simple test suite</a></li>
<li title="A test suite with configuration functions"><a href="getting_started_chapter.html#id68094">A test suite with configuration functions</a></li>
<li title="What happens next?"><a href="getting_started_chapter.html#id62545">What happens next?</a></li>
</ul>
</li>
<li id="no" title="Installation" expanded="false">Installation<ul>
<li><a href="install_chapter.html">
Top of chapter
</a></li>
<li title="General information"><a href="install_chapter.html#id66699">General information</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Writing Test Suites" expanded="true">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id71900">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id70761">Test suites</a></li>
<li title="Init and end per suite"><a href="write_test_chapter.html#id70806">Init and end per suite</a></li>
<li title="Init and end per test case"><a href="write_test_chapter.html#id70917">Init and end per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id71162">Test cases</a></li>
<li title="Test case info function"><a href="write_test_chapter.html#id71322">Test case info function</a></li>
<li title="Test suite info function"><a href="write_test_chapter.html#id71521">Test suite info function</a></li>
<li title="Test case groups"><a href="write_test_chapter.html#id71611">Test case groups</a></li>
<li title="The parallel property and nested groups"><a href="write_test_chapter.html#id72823">The parallel property and nested groups</a></li>
<li title="Parallel test cases and IO"><a href="write_test_chapter.html#id72860">Parallel test cases and IO</a></li>
<li title="Repeated groups"><a href="write_test_chapter.html#id72908">Repeated groups</a></li>
<li title="Shuffled test case order"><a href="write_test_chapter.html#id73046">Shuffled test case order</a></li>
<li title="Group info function"><a href="write_test_chapter.html#id73105">Group info function</a></li>
<li title="Info functions for init- and end-configuration"><a href="write_test_chapter.html#id73139">Info functions for init- and end-configuration</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id73202">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id73327">Execution environment</a></li>
<li title="Timetrap timeouts"><a href="write_test_chapter.html#id73364">Timetrap timeouts</a></li>
<li title="Logging - categories and verbosity levels"><a href="write_test_chapter.html#id73549">Logging - categories and verbosity levels</a></li>
<li title="Illegal dependencies"><a href="write_test_chapter.html#id73756">Illegal dependencies</a></li>
</ul>
</li>
<li id="no" title="Test Structure" expanded="false">Test Structure<ul>
<li><a href="test_structure_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_structure_chapter.html#id73930">Test structure</a></li>
<li title="Skipping test cases"><a href="test_structure_chapter.html#id73944">Skipping test cases</a></li>
<li title="Definition of terms"><a href="test_structure_chapter.html#id74020">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Examples and Templates" expanded="false">Examples and Templates<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite example"><a href="example_chapter.html#id74246">Test suite example</a></li>
<li title="Test suite templates"><a href="example_chapter.html#id74298">Test suite templates</a></li>
</ul>
</li>
<li id="no" title="Running Tests" expanded="false">Running Tests<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the Common Test Framework"><a href="run_test_chapter.html#id74488">Using the Common Test Framework</a></li>
<li title="Automatic compilation of test suites and help modules"><a href="run_test_chapter.html#id74528">Automatic compilation of test suites and help modules</a></li>
<li title="Running tests from the OS command line"><a href="run_test_chapter.html#id74645">Running tests from the OS command line</a></li>
<li title="Running tests from the Erlang shell or from an Erlang program"><a href="run_test_chapter.html#id75142">Running tests from the Erlang shell or from an Erlang program</a></li>
<li title="Test case group execution"><a href="run_test_chapter.html#id75304">Test case group execution</a></li>
<li title="Running the interactive shell mode"><a href="run_test_chapter.html#id75821">Running the interactive shell mode</a></li>
<li title="Step by step execution of test cases with the Erlang Debugger"><a href="run_test_chapter.html#id76003">Step by step execution of test cases with the Erlang Debugger</a></li>
<li title="Test Specifications"><a href="run_test_chapter.html#id76102">Test Specifications</a></li>
<li title="Running tests from the Web based GUI"><a href="run_test_chapter.html#id76643">Running tests from the Web based GUI</a></li>
<li title="Log files"><a href="run_test_chapter.html#id76735">Log files</a></li>
<li title="HTML Style Sheets"><a href="run_test_chapter.html#id77053">HTML Style Sheets</a></li>
<li title="Repeating tests"><a href="run_test_chapter.html#id77201">Repeating tests</a></li>
<li title="Silent Connections"><a href="run_test_chapter.html#id77412">Silent Connections</a></li>
</ul>
</li>
<li id="no" title="External Configuration Data" expanded="false">External Configuration Data<ul>
<li><a href="config_file_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="config_file_chapter.html#id77631">General</a></li>
<li title="Syntax"><a href="config_file_chapter.html#id77671">Syntax</a></li>
<li title="Requiring and reading configuration data"><a href="config_file_chapter.html#id77695">Requiring and reading configuration data</a></li>
<li title="Using configuration variables defined in multiple files"><a href="config_file_chapter.html#id77839">Using configuration variables defined in multiple files</a></li>
<li title="Encrypted configuration files"><a href="config_file_chapter.html#id77870">Encrypted configuration files</a></li>
<li title="Opening connections by using configuration data"><a href="config_file_chapter.html#id77934">Opening connections by using configuration data</a></li>
<li title="User specific configuration data formats"><a href="config_file_chapter.html#id78002">User specific configuration data formats</a></li>
<li title="Examples of configuration data handling"><a href="config_file_chapter.html#id78216">Examples of configuration data handling</a></li>
<li title="Example of user specific configuration handler"><a href="config_file_chapter.html#id78268">Example of user specific configuration handler</a></li>
</ul>
</li>
<li id="no" title="Code Coverage Analysis" expanded="false">Code Coverage Analysis<ul>
<li><a href="cover_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="cover_chapter.html#id78384">General</a></li>
<li title="Usage"><a href="cover_chapter.html#id78404">Usage</a></li>
<li title="The cover specification file"><a href="cover_chapter.html#id78510">The cover specification file</a></li>
<li title="Logging"><a href="cover_chapter.html#id78563">Logging</a></li>
</ul>
</li>
<li id="no" title="Using Common Test for Large Scale Testing" expanded="false">Using Common Test for Large Scale Testing<ul>
<li><a href="ct_master_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_master_chapter.html#id78634">General</a></li>
<li title="Usage"><a href="ct_master_chapter.html#id78668">Usage</a></li>
<li title="Test Specifications"><a href="ct_master_chapter.html#id78815">Test Specifications</a></li>
<li title="Automatic startup of test target nodes"><a href="ct_master_chapter.html#id78987">Automatic startup of test target nodes</a></li>
</ul>
</li>
<li id="no" title="Event Handling" expanded="false">Event Handling<ul>
<li><a href="event_handler_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="event_handler_chapter.html#id79190">General</a></li>
<li title="Usage"><a href="event_handler_chapter.html#id79226">Usage</a></li>
</ul>
</li>
<li id="no" title="Dependencies between Test Cases and Suites" expanded="false">Dependencies between Test Cases and Suites<ul>
<li><a href="dependencies_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="dependencies_chapter.html#id80143">General</a></li>
<li title="Saving configuration data"><a href="dependencies_chapter.html#id80265">Saving configuration data</a></li>
<li title="Sequences"><a href="dependencies_chapter.html#id80430">Sequences</a></li>
</ul>
</li>
<li id="no" title="Common Test Hooks" expanded="false">Common Test Hooks<ul>
<li><a href="ct_hooks_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_hooks_chapter.html#id80622">General</a></li>
<li title="Installing a CTH"><a href="ct_hooks_chapter.html#id80676">Installing a CTH</a></li>
<li title="CTH Scope"><a href="ct_hooks_chapter.html#id80835">CTH Scope</a></li>
<li title="Manipulating tests"><a href="ct_hooks_chapter.html#id81189">Manipulating tests</a></li>
<li title="Example CTH"><a href="ct_hooks_chapter.html#id81482">Example CTH</a></li>
<li title="Built-in CTHs"><a href="ct_hooks_chapter.html#id81535">Built-in CTHs</a></li>
</ul>
</li>
<li id="no" title="Some thoughts about testing" expanded="false">Some thoughts about testing<ul>
<li><a href="why_test_chapter.html">
Top of chapter
</a></li>
<li title="Goals"><a href="why_test_chapter.html#id81722">Goals</a></li>
<li title="What to test?"><a href="why_test_chapter.html#id81742">What to test?</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>4 Writing Test Suites</h1>
<h3><a name="id71900">4.1
Support for test suite authors</a></h3>
<a name="intro"></a>
<p>The <span class="code">ct</span> module provides the main interface for writing
test cases. This includes e.g:</p>
<ul>
<li>Functions for printing and logging</li>
<li>Functions for reading configuration data</li>
<li>Function for terminating a test case with error reason</li>
<li>Function for adding comments to the HTML overview page</li>
</ul>
<p>Please see the reference manual for the <span class="code">ct</span>
module for details about these functions.</p>
<p>The CT application also includes other modules named
<span class="code">ct_<component></span> that
provide various support, mainly simplified use of communication
protocols such as rpc, snmp, ftp, telnet, etc.</p>
<h3><a name="id70761">4.2
Test suites</a></h3>
<p>A test suite is an ordinary Erlang module that contains test
cases. It is recommended that the module has a name on the form
<span class="code">*_SUITE.erl</span>. Otherwise, the directory and auto compilation
function in CT will not be able to locate it (at least not per default).
</p>
<p>It is also recommended that the <span class="code">ct.hrl</span> header file is included
in all test suite modules.
</p>
<p>Each test suite module must export the function <span class="code">all/0</span>
which returns the list of all test case groups and test cases
to be executed in that module.
</p>
<p>The callback functions that the test suite should implement, and
which will be described in more detail below, are
all listed in the <span class="bold_code"><a href="common_test.html">common_test
reference manual page</a></span>.
</p>
<h3><a name="id70806">4.3
Init and end per suite</a></h3>
<p>Each test suite module may contain the optional configuration functions
<span class="code">init_per_suite/1</span> and <span class="code">end_per_suite/1</span>. If the init function
is defined, so must the end function be.
</p>
<p>If it exists, <span class="code">init_per_suite</span> is called initially before the
test cases are executed. It typically contains initializations that are
common for all test cases in the suite, and that are only to be
performed once. It is recommended to be used for setting up and
verifying state and environment on the SUT (System Under Test) and/or
the CT host node, so that the test cases in the suite will execute
correctly. Examples of initial configuration operations: Opening a connection
to the SUT, initializing a database, running an installation script, etc.
</p>
<p><span class="code">end_per_suite</span> is called as the final stage of the test suite execution
(after the last test case has finished). The function is meant to be used
for cleaning up after <span class="code">init_per_suite</span>.
</p>
<p><span class="code">init_per_suite</span> and <span class="code">end_per_suite</span> will execute on dedicated
Erlang processes, just like the test cases do. The result of these functions
is however not included in the test run statistics of successful, failed and
skipped cases.
</p>
<p>The argument to <span class="code">init_per_suite</span> is <span class="code">Config</span>, the
same key-value list of runtime configuration data that each test case takes
as input argument. <span class="code">init_per_suite</span> can modify this parameter with
information that the test cases need. The possibly modified <span class="code">Config</span>
list is the return value of the function.
</p>
<p>If <span class="code">init_per_suite</span> fails, all test cases in the test
suite will be skipped automatically (so called <strong>auto skipped</strong>),
including <span class="code">end_per_suite</span>.
</p>
<p>Note that if <span class="code">init_per_suite</span> and <span class="code">end_per_suite</span> do not exist
in the suite, Common Test calls dummy functions (with the same names)
instead, so that output generated by hook functions may be saved to the log
files for these dummies
(see the <span class="bold_code"><a href="ct_hooks_chapter.html#manipulating">Common Test Hooks</a></span>
chapter for more information).
</p>
<a name="per_testcase"></a>
<h3><a name="id70917">4.4
Init and end per test case</a></h3>
<p>Each test suite module can contain the optional configuration functions
<span class="code">init_per_testcase/2</span> and <span class="code">end_per_testcase/2</span>. If the init function
is defined, so must the end function be.</p>
<p>If it exists, <span class="code">init_per_testcase</span> is called before each
test case in the suite. It typically contains initialization which
must be done for each test case (analogue to <span class="code">init_per_suite</span> for the
suite).</p>
<p><span class="code">end_per_testcase/2</span> is called after each test case has
finished, giving the opportunity to perform clean-up after
<span class="code">init_per_testcase</span>.</p>
<p>The first argument to these functions is the name of the test
case. This value can be used with pattern matching in function clauses
or conditional expressions to choose different initialization and cleanup
routines for different test cases, or perform the same routine for a number of,
or all, test cases.</p>
<p>The second argument is the <span class="code">Config</span> key-value list of runtime
configuration data, which has the same value as the list returned by
<span class="code">init_per_suite</span>. <span class="code">init_per_testcase/2</span> may modify this
parameter or return it as is. The return value of <span class="code">init_per_testcase/2</span>
is passed as the <span class="code">Config</span> parameter to the test case itself.</p>
<p>The return value of <span class="code">end_per_testcase/2</span> is ignored by the
test server, with exception of the
<span class="bold_code"><a href="dependencies_chapter.html#save_config">save_config</a></span>
and <span class="code">fail</span> tuple.</p>
<p>It is possible in <span class="code">end_per_testcase</span> to check if the
test case was successful or not (which consequently may determine
how cleanup should be performed). This is done by reading the value
tagged with <span class="code">tc_status</span> from <span class="code">Config</span>. The value is either
<span class="code">ok</span>, <span class="code">{failed,Reason}</span> (where <span class="code">Reason</span> is <span class="code">timetrap_timeout</span>,
info from <span class="code">exit/1</span>, or details of a run-time error), or
<span class="code">{skipped,Reason}</span> (where Reason is a user specific term).
</p>
<p>The <span class="code">end_per_testcase/2</span> function is called even after a
test case terminates due to a call to <span class="code"><span class="bold_code"><a href="ct.html#abort_current_testcase-1">ct:abort_current_testcase/1</a></span></span>,
or after a timetrap timeout. However, <span class="code">end_per_testcase</span>
will then execute on a different process than the test case
function, and in this situation, <span class="code">end_per_testcase</span> will
not be able to change the reason for test case termination by
returning <span class="code">{fail,Reason}</span>, nor will it be able to save data with
<span class="code">{save_config,Data}</span>.</p>
<p>If <span class="code">init_per_testcase</span> crashes, the test case itself gets skipped
automatically (so called <strong>auto skipped</strong>). If <span class="code">init_per_testcase</span>
returns a tuple <span class="code">{skip,Reason}</span>, also then the test case gets skipped
(so called <strong>user skipped</strong>). It is also possible, by returning a tuple
<span class="code">{fail,Reason}</span> from <span class="code">init_per_testcase</span>, to mark the test case
as failed without actually executing it.
</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>If <span class="code">init_per_testcase</span> crashes, or returns <span class="code">{skip,Reason}</span>
or <span class="code">{fail,Reason}</span>, the <span class="code">end_per_testcase</span> function is not called.
</p></p></div>
</div>
<p>If it is determined during execution of <span class="code">end_per_testcase</span> that
the status of a successful test case should be changed to failed,
<span class="code">end_per_testcase</span> may return the tuple: <span class="code">{fail,Reason}</span>
(where <span class="code">Reason</span> describes why the test case fails).</p>
<p><span class="code">init_per_testcase</span> and <span class="code">end_per_testcase</span> execute on the
same Erlang process as the test case and printouts from these
configuration functions can be found in the test case log file.</p>
<h3><a name="id71162">4.5
Test cases</a></h3>
<a name="test_cases"></a>
<p>The smallest unit that the test server is concerned with is a
test case. Each test case can actually test many things, for
example make several calls to the same interface function with
different parameters.
</p>
<p>It is possible to choose to put many or few tests into each test
case. What exactly each test case does is of course up to the
author, but here are some things to keep in mind:
</p>
<p>Having many small test cases tend to result in extra, and possibly
duplicated code, as well as slow test execution because of
large overhead for initializations and cleanups. Duplicated
code should be avoided, e.g. by means of common help functions, or
the resulting suite will be difficult to read and understand, and
expensive to maintain.
</p>
<p>Larger test cases make it harder to tell what went wrong if it
fails, and large portions of test code will potentially be skipped
when errors occur. Furthermore, readability and maintainability suffers
when test cases become too large and extensive. Also, the resulting log
files may not reflect very well the number of tests that have
actually been performed.
</p>
<p>The test case function takes one argument, <span class="code">Config</span>, which
contains configuration information such as <span class="code">data_dir</span> and
<span class="code">priv_dir</span>. (See <span class="bold_code"><a href="#data_priv_dir">Data and
Private Directories</a></span> for more information about these).
The value of <span class="code">Config</span> at the time of the call, is the same
as the return value from <span class="code">init_per_testcase</span>, see above.
</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>The test case function argument <span class="code">Config</span> should not be
confused with the information that can be retrieved from
configuration files (using <span class="code"><span class="bold_code"><a href="ct.html#get_config-1">
ct:get_config/1/2</a></span></span>). The Config argument
should be used for runtime configuration of the test suite and the
test cases, while configuration files should typically contain data
related to the SUT. These two types of configuration data are handled
differently!</p></p></div>
</div>
<p>Since the <span class="code">Config</span> parameter is a list of key-value tuples, i.e.
a data type generally called a property list, it can be handled by means of the
<span class="code">proplists</span> module in the OTP <span class="code">stdlib</span>. A value can for example
be searched for and returned with the <span class="code">proplists:get_value/2</span> function.
Also, or alternatively, you might want to look in the general <span class="code">lists</span> module,
also in <span class="code">stdlib</span>, for useful functions. Normally, the only operations you
ever perform on <span class="code">Config</span> is insert (adding a tuple to the head of the list)
and lookup. Common Test provides a simple macro named <span class="code">?config</span>, which returns
a value of an item in <span class="code">Config</span> given the key (exactly like
<span class="code">proplists:get_value</span>). Example: <span class="code">PrivDir = ?config(priv_dir, Config)</span>.
</p>
<p>If the test case function crashes or exits purposely, it is considered
<strong>failed</strong>. If it returns a value (no matter what actual value) it is
considered successful. An exception to this rule is the return value
<span class="code">{skip,Reason}</span>. If this tuple is returned, the test case is considered
skipped and gets logged as such.</p>
<p>If the test case returns the tuple <span class="code">{comment,Comment}</span>, the case
is considered successful and <span class="code">Comment</span> is printed out in the overview
log file. This is by the way equal to calling <span class="code">ct:comment(Comment)</span>.
</p>
<h3><a name="id71322">4.6
Test case info function</a></h3>
<a name="info_function"></a>
<p>For each test case function there can be an additional function
with the same name but with no arguments. This is the test case
info function. The test case info function is expected to return a
list of tagged tuples that specifies various properties regarding the
test case.
</p>
<p>The following tags have special meaning:</p>
<dl>
<dt><strong><strong><span class="code">timetrap</span></strong></strong></dt>
<dd>
<p>
Set the maximum time the test case is allowed to execute. If
the timetrap time is exceeded, the test case fails with
reason <span class="code">timetrap_timeout</span>. Note that <span class="code">init_per_testcase</span>
and <span class="code">end_per_testcase</span> are included in the timetrap time.
Please see the <span class="bold_code"><a href="write_test_chapter.html#timetraps">Timetrap</a></span>
section for more details.
</p>
</dd>
<dt><strong><strong><span class="code">userdata</span></strong></strong></dt>
<dd>
<p>
Use this to specify arbitrary data related to the testcase. This
data can be retrieved at any time using the <span class="code"><span class="bold_code"><a href="ct.html#userdata-3">ct:userdata/3</a></span></span>
utility function.
</p>
</dd>
<dt><strong><strong><span class="code">silent_connections</span></strong></strong></dt>
<dd>
<p>
Please see the
<span class="bold_code"><a href="run_test_chapter.html#silent_connections">Silent Connections</a></span>
chapter for details.
</p>
</dd>
<dt><strong><strong><span class="code">require</span></strong></strong></dt>
<dd>
<p>
Use this to specify configuration variables that are required by the
test case. If the required configuration variables are not
found in any of the test system configuration files, the test case is
skipped.</p>
<p>
It is also possible to give a required variable a default value that will
be used if the variable is not found in any configuration file. To specify
a default value, add a tuple on the form:
<span class="code">{default_config,ConfigVariableName,Value}</span> to the test case info list
(the position in the list is irrelevant).
Examples:</p>
<div class="example"><pre>
testcase1() ->
[{require, ftp},
{default_config, ftp, [{ftp, "my_ftp_host"},
{username, "aladdin"},
{password, "sesame"}]}}].</pre></div>
<div class="example"><pre>
testcase2() ->
[{require, unix_telnet, unix},
{require, {unix, [telnet, username, password]}},
{default_config, unix, [{telnet, "my_telnet_host"},
{username, "aladdin"},
{password, "sesame"}]}}].</pre></div>
</dd>
</dl>
<p>See the <span class="bold_code"><a href="config_file_chapter.html#require_config_data">Config files</a></span>
chapter and the <span class="code"><span class="bold_code"><a href="ct.html#require-1">
ct:require/1/2</a></span></span> function in the
<span class="bold_code"><a href="ct.html">ct</a></span> reference manual for more information about
<span class="code">require</span>.</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>Specifying a default value for a required variable can result
in a test case always getting executed. This might not be a desired behaviour!</p>
</p></div>
</div>
<p>If <span class="code">timetrap</span> and/or <span class="code">require</span> is not set specifically for
a particular test case, default values specified by the <span class="code">suite/0</span>
function are used.
</p>
<p>Other tags than the ones mentioned above will simply be ignored by
the test server.
</p>
<p>
Example of a test case info function:
</p>
<div class="example"><pre>
reboot_node() ->
[
{timetrap,{seconds,60}},
{require,interfaces},
{userdata,
[{description,"System Upgrade: RpuAddition Normal RebootNode"},
{fts,"http://someserver.ericsson.se/test_doc4711.pdf"}]}
].</pre></div>
<h3><a name="id71521">4.7
Test suite info function</a></h3>
<a name="suite"></a>
<p>The <span class="code">suite/0</span> function can be used in a test suite
module to e.g. set a default <span class="code">timetrap</span> value and to
<span class="code">require</span> external configuration data. If a test case-, or
group info function also specifies any of the info tags, it
overrides the default values set by <span class="code">suite/0</span>. See the test
case info function above, and group info function below, for more
details.
</p>
<p>Other options that may be specified with the suite info list are:</p>
<ul>
<li>
<span class="code">stylesheet</span>,
see <span class="bold_code"><a href="run_test_chapter.html#html_stylesheet">HTML Style Sheets</a></span>.</li>
<li>
<span class="code">userdata</span>,
see <span class="bold_code"><a href="#info_function">Test case info function</a></span>.</li>
<li>
<span class="code">silent_connections</span>,
see <span class="bold_code"><a href="run_test_chapter.html#silent_connections">Silent Connections</a></span>.</li>
</ul>
<p>
Example of the suite info function:
</p>
<div class="example"><pre>
suite() ->
[
{timetrap,{minutes,10}},
{require,global_names},
{userdata,[{info,"This suite tests database transactions."}]},
{silent_connections,[telnet]},
{stylesheet,"db_testing.css"}
].</pre></div>
<h3><a name="id71611">4.8
Test case groups</a></h3>
<a name="test_case_groups"></a>
<p>A test case group is a set of test cases that share configuration
functions and execution properties. Test case groups are defined by
means of the <span class="code">groups/0</span> function according to the following syntax:</p>
<div class="example"><pre>
groups() -> GroupDefs
Types:
GroupDefs = [GroupDef]
GroupDef = {GroupName,Properties,GroupsAndTestCases}
GroupName = atom()
GroupsAndTestCases = [GroupDef | {group,GroupName} | TestCase]
TestCase = atom()</pre></div>
<p><span class="code">GroupName</span> is the name of the group and should be unique within
the test suite module. Groups may be nested, and this is accomplished
simply by including a group definition within the <span class="code">GroupsAndTestCases</span>
list of another group. <span class="code">Properties</span> is the list of execution
properties for the group. The possible values are:</p>
<div class="example"><pre>
Properties = [parallel | sequence | Shuffle | {RepeatType,N}]
Shuffle = shuffle | {shuffle,Seed}
Seed = {integer(),integer(),integer()}
RepeatType = repeat | repeat_until_all_ok | repeat_until_all_fail |
repeat_until_any_ok | repeat_until_any_fail
N = integer() | forever</pre></div>
<p>If the <span class="code">parallel</span> property is specified, Common Test will execute
all test cases in the group in parallel. If <span class="code">sequence</span> is specified,
the cases will be executed in a sequence, as described in the chapter
<span class="bold_code"><a href="dependencies_chapter.html#sequences">Dependencies between
test cases and suites</a></span>. If <span class="code">shuffle</span> is specified, the cases
in the group will be executed in random order. The <span class="code">repeat</span> property
orders Common Test to repeat execution of the cases in the group a given
number of times, or until any, or all, cases fail or succeed.</p>
<p>Example:</p>
<div class="example"><pre>
groups() -> [{group1, [parallel], [test1a,test1b]},
{group2, [shuffle,sequence], [test2a,test2b,test2c]}].</pre></div>
<p>To specify in which order groups should be executed (also with respect
to test cases that are not part of any group), tuples on the form
<span class="code">{group,GroupName}</span> should be added to the <span class="code">all/0</span> list. Example:</p>
<div class="example"><pre>
all() -> [testcase1, {group,group1}, testcase2, {group,group2}].</pre></div>
<p>It is also possible to specify execution properties with a group
tuple in <span class="code">all/0</span>: <span class="code">{group,GroupName,Properties}</span>. These
properties will override those specified in the group definition (see
<span class="code">groups/0</span> above). This way, it's possible to run the same set of tests,
but with different properties, without having to make copies of the group
definition in question.</p>
<p>If a group contains sub-groups, the execution properties for these may
also be specified in the group tuple:
<span class="code">{group,GroupName,Properties,SubGroups}</span>, where <span class="code">SubGroups</span>
is a list of tuples, <span class="code">{GroupName,Properties}</span>, or
<span class="code">{GroupName,Properties,SubGroups}</span>, representing the sub-groups.
Any sub-groups defined in <span class="code">group/0</span> for a group, that are not specified
in the <span class="code">SubGroups</span> list, will simply execute with their pre-defined
properties.</p>
<p>Example:</p>
<div class="example"><pre>
groups() -> {tests1, [], [{tests2, [], [t2a,t2b]},
{tests3, [], [t31,t3b]}]}.</pre></div>
<p>To execute group 'tests1' twice with different properties for 'tests2'
each time:</p>
<div class="example"><pre>
all() ->
[{group, tests1, default, [{tests2, [parallel]}]},
{group, tests1, default, [{tests2, [shuffle,{repeat,10}]}]}].</pre></div>
<p>Note that this is equivalent to this specification:</p>
<div class="example"><pre>
all() ->
[{group, tests1, default, [{tests2, [parallel]},
{tests3, default}]},
{group, tests1, default, [{tests2, [shuffle,{repeat,10}]},
{tests3, default}]}].</pre></div>
<p>The value <span class="code">default</span> states that the pre-defined properties
should be used.</p>
<p>Here's an example of how to override properties in a scenario
with deeply nested groups:</p>
<div class="example"><pre>
groups() ->
[{tests1, [], [{group, tests2}]},
{tests2, [], [{group, tests3}]},
{tests3, [{repeat,2}], [t3a,t3b,t3c]}].
all() ->
[{group, tests1, default,
[{tests2, default,
[{tests3, [parallel,{repeat,100}]}]}]}].</pre></div>
<p>The syntax described above may also be used in Test Specifications
in order to change properties of groups at the time of execution,
without even having to edit the test suite (please see the
<span class="bold_code"><a href="run_test_chapter.html#test_specifications">Test
Specifications</a></span> chapter for more info).</p>
<p>As illustrated above, properties may be combined. If e.g.
<span class="code">shuffle</span>, <span class="code">repeat_until_any_fail</span> and <span class="code">sequence</span>
are all specified, the test cases in the group will be executed
repeatedly, and in random order, until a test case fails. Then
execution is immediately stopped and the rest of the cases skipped.</p>
<p>Before execution of a group begins, the configuration function
<span class="code">init_per_group(GroupName, Config)</span> is called. The list of tuples
returned from this function is passed to the test cases in the usual
manner by means of the <span class="code">Config</span> argument. <span class="code">init_per_group/2</span>
is meant to be used for initializations common for the test cases in the
group. After execution of the group is finished, the
<span class="code">end_per_group(GroupName, Config</span> function is called. This function
is meant to be used for cleaning up after <span class="code">init_per_group/2</span>.</p>
<p>Whenever a group is executed, if <span class="code">init_per_group</span> and
<span class="code">end_per_group</span> do not exist in the suite, Common Test calls
dummy functions (with the same names) instead. Output generated by
hook functions will be saved to the log files for these dummies
(see the <span class="bold_code"><a href="ct_hooks_chapter.html#manipulating">Common Test
Hooks</a></span> chapter for more information).
</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p><span class="code">init_per_testcase/2</span> and <span class="code">end_per_testcase/2</span>
are always called for each individual test case, no matter if the case
belongs to a group or not.</p></p></div>
</div>
<p>The properties for a group is always printed on the top of the HTML log
for <span class="code">init_per_group/2</span>. Also, the total execution time for a group
can be found at the bottom of the log for <span class="code">end_per_group/2</span>.</p>
<p>Test case groups may be nested so that sets of groups can be
configured with the same <span class="code">init_per_group/2</span> and <span class="code">end_per_group/2</span>
functions. Nested groups may be defined by including a group definition,
or a group name reference, in the test case list of another group. Example:</p>
<div class="example"><pre>
groups() -> [{group1, [shuffle], [test1a,
{group2, [], [test2a,test2b]},
test1b]},
{group3, [], [{group,group4},
{group,group5}]},
{group4, [parallel], [test4a,test4b]},
{group5, [sequence], [test5a,test5b,test5c]}].</pre></div>
<p>In the example above, if <span class="code">all/0</span> would return group name references
in this order: <span class="code">[{group,group1},{group,group3}]</span>, the order of the
configuration functions and test cases will be the following (note that
<span class="code">init_per_testcase/2</span> and <span class="code">end_per_testcase/2:</span> are also
always called, but not included in this example for simplification):</p>
<div class="example"><pre>
- init_per_group(group1, Config) -> Config1 (*)
-- test1a(Config1)
-- init_per_group(group2, Config1) -> Config2
--- test2a(Config2), test2b(Config2)
-- end_per_group(group2, Config2)
-- test1b(Config1)
- end_per_group(group1, Config1)
- init_per_group(group3, Config) -> Config3
-- init_per_group(group4, Config3) -> Config4
--- test4a(Config4), test4b(Config4) (**)
-- end_per_group(group4, Config4)
-- init_per_group(group5, Config3) -> Config5
--- test5a(Config5), test5b(Config5), test5c(Config5)
-- end_per_group(group5, Config5)
- end_per_group(group3, Config3)
(*) The order of test case test1a, test1b and group2 is not actually
defined since group1 has a shuffle property.
(**) These cases are not executed in order, but in parallel.</pre></div>
<p>Properties are not inherited from top level groups to nested
sub-groups. E.g, in the example above, the test cases in <span class="code">group2</span>
will not be executed in random order (which is the property of
<span class="code">group1</span>).</p>
<h3><a name="id72823">4.9
The parallel property and nested groups</a></h3>
<p>If a group has a parallel property, its test cases will be spawned
simultaneously and get executed in parallel. A test case is not allowed
to execute in parallel with <span class="code">end_per_group/2</span> however, which means
that the time it takes to execute a parallel group is equal to the
execution time of the slowest test case in the group. A negative side
effect of running test cases in parallel is that the HTML summary pages
are not updated with links to the individual test case logs until the
<span class="code">end_per_group/2</span> function for the group has finished.</p>
<p>A group nested under a parallel group will start executing in parallel
with previous (parallel) test cases (no matter what properties the nested
group has). Since, however, test cases are never executed in parallel with
<span class="code">init_per_group/2</span> or <span class="code">end_per_group/2</span> of the same group, it's
only after a nested group has finished that any remaining parallel cases
in the previous group get spawned.</p>
<h3><a name="id72860">4.10
Parallel test cases and IO</a></h3>
<p>A parallel test case has a private IO server as its group leader.
(Please see the Erlang Run-Time System Application documentation for
a description of the group leader concept). The
central IO server process that handles the output from regular test
cases and configuration functions, does not respond to IO messages
during execution of parallel groups. This is important to understand
in order to avoid certain traps, like this one:</p>
<p>If a process, <span class="code">P</span>, is spawned during execution of e.g.
<span class="code">init_per_suite/1</span>, it will inherit the group leader of the
<span class="code">init_per_suite</span> process. This group leader is the central IO server
process mentioned above. If, at a later time, <strong>during parallel test case
execution</strong>, some event triggers process <span class="code">P</span> to call
<span class="code">io:format/1/2</span>, that call will never return (since the group leader
is in a non-responsive state) and cause <span class="code">P</span> to hang.
</p>
<h3><a name="id72908">4.11
Repeated groups</a></h3>
<a name="repeated_groups"></a>
<p>A test case group may be repeated a certain number of times
(specified by an integer) or indefinitely (specified by <span class="code">forever</span>).
The repetition may also be stopped prematurely if any or all cases
fail or succeed, i.e. if the property <span class="code">repeat_until_any_fail</span>,
<span class="code">repeat_until_any_ok</span>, <span class="code">repeat_until_all_fail</span>, or
<span class="code">repeat_until_all_ok</span> is used. If the basic <span class="code">repeat</span>
property is used, status of test cases is irrelevant for the repeat
operation.</p>
<p>It is possible to return the status of a sub-group (ok or
failed), to affect the execution of the group on the level above.
This is accomplished by, in <span class="code">end_per_group/2</span>, looking up the value
of <span class="code">tc_group_properties</span> in the <span class="code">Config</span> list and checking the
result of the test cases in the group. If status <span class="code">failed</span> should be
returned from the group as a result, <span class="code">end_per_group/2</span> should return
the value <span class="code">{return_group_result,failed}</span>. The status of a sub-group
is taken into account by Common Test when evaluating if execution of a
group should be repeated or not (unless the basic <span class="code">repeat</span>
property is used).</p>
<p>The <span class="code">tc_group_properties</span> value is a list of status tuples,
each with the key <span class="code">ok</span>, <span class="code">skipped</span> and <span class="code">failed</span>. The
value of a status tuple is a list containing names of test cases
that have been executed with the corresponding status as result.</p>
<p>Here's an example of how to return the status from a group:</p>
<div class="example"><pre>
end_per_group(_Group, Config) ->
Status = ?config(tc_group_result, Config),
case proplists:get_value(failed, Status) of
[] -> % no failed cases
{return_group_result,ok};
_Failed -> % one or more failed
{return_group_result,failed}
end.</pre></div>
<p>It is also possible in <span class="code">end_per_group/2</span> to check the status of
a sub-group (maybe to determine what status the current group should also
return). This is as simple as illustrated in the example above, only the
name of the group is stored in a tuple <span class="code">{group_result,GroupName}</span>,
which can be searched for in the status lists. Example:</p>
<div class="example"><pre>
end_per_group(group1, Config) ->
Status = ?config(tc_group_result, Config),
Failed = proplists:get_value(failed, Status),
case lists:member({group_result,group2}, Failed) of
true ->
{return_group_result,failed};
false ->
{return_group_result,ok}
end;
...</pre></div>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>When a test case group is repeated, the configuration
functions, <span class="code">init_per_group/2</span> and <span class="code">end_per_group/2</span>, are
also always called with each repetition.</p></p></div>
</div>
<h3><a name="id73046">4.12
Shuffled test case order</a></h3>
<p>The order that test cases in a group are executed, is under normal
circumstances the same as the order specified in the test case list
in the group definition. With the <span class="code">shuffle</span> property set, however,
Common Test will instead execute the test cases in random order.</p>
<p>The user may provide a seed value (a tuple of three integers) with
the shuffle property: <span class="code">{shuffle,Seed}</span>. This way, the same shuffling
order can be created every time the group is executed. If no seed value
is given, Common Test creates a "random" seed for the shuffling operation
(using the return value of <span class="code">erlang:now()</span>). The seed value is always
printed to the <span class="code">init_per_group/2</span> log file so that it can be used to
recreate the same execution order in a subsequent test run.</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>If a shuffled test case group is repeated, the seed will not
be reset in between turns.</p></p></div>
</div>
<p>If a sub-group is specified in a group with a <span class="code">shuffle</span> property,
the execution order of this sub-group in relation to the test cases
(and other sub-groups) in the group, is also random. The order of the
test cases in the sub-group is however not random (unless, of course, the
sub-group also has a <span class="code">shuffle</span> property).</p>
<h3><a name="id73105">4.13
Group info function</a></h3>
<a name="group_info"></a>
<p>The test case group info function, <span class="code">group(GroupName)</span>,
serves the same purpose as the suite- and test case info
functions previously described in this chapter. The scope for
the group info, however, is all test cases and sub-groups in the
group in question (<span class="code">GroupName</span>).</p>
<p>Example:</p>
<div class="example"><pre>
group(connection_tests) ->
[{require,login_data},
{timetrap,1000}].</pre></div>
<p>The group info properties override those set with the
suite info function, and may in turn be overridden by test
case info properties. Please see the test case info
function above for a list of valid info properties and more
general information.</p>
<h3><a name="id73139">4.14
Info functions for init- and end-configuration</a></h3>
<p>It is possible to use info functions also for the <span class="code">init_per_suite</span>,
<span class="code">end_per_suite</span>, <span class="code">init_per_group</span>, and <span class="code">end_per_group</span>
functions, and it works the same way as with info functions
for test cases (see above). This is useful e.g. for setting
timetraps and requiring external configuration data relevant
only for the configuration function in question (without
affecting properties set for groups and test cases in the suite).</p>
<p>The info function <span class="code">init/end_per_suite()</span> is called for
<span class="code">init/end_per_suite(Config)</span>, and info function
<span class="code">init/end_per_group(GroupName)</span> is called for
<span class="code">init/end_per_group(GroupName,Config)</span>. Info functions
can not be used with <span class="code">init/end_per_testcase(TestCase, Config)</span>,
however, since these configuration functions execute on the test case process
and will use the same properties as the test case (i.e. the properties
set by the test case info function, <span class="code">TestCase()</span>). Please see the test case
info function above for a list of valid info properties and more
general information.
</p>
<h3><a name="id73202">4.15
Data and Private Directories</a></h3>
<a name="data_priv_dir"></a>
<p>The data directory, <span class="code">data_dir</span>, is the directory where the
test module has its own files needed for the testing. The name
of the <span class="code">data_dir</span> is the the name of the test suite followed
by <span class="code">"_data"</span>. For example,
<span class="code">"some_path/foo_SUITE.beam"</span> has the data directory
<span class="code">"some_path/foo_SUITE_data/"</span>. Use this directory for portability,
i.e. to avoid hardcoding directory names in your suite. Since the data
directory is stored in the same directory as your test suite, you should
be able to rely on its existence at runtime, even if the path to your
test suite directory has changed between test suite implementation and
execution.
</p>
<p>
<span class="code">priv_dir</span> is the private directory for the test cases.
This directory may be used whenever a test case (or configuration function)
needs to write something to file. The name of the private directory is
generated by Common Test, which also creates the directory.
</p>
<p>By default, Common Test creates one central private directory
per test run that all test cases share. This may not always be suitable,
especially if the same test cases are executed multiple times during
a test run (e.g. if they belong to a test case group with repeat
property), and there's a risk that files in the private directory get
overwritten. Under these circumstances, it's possible to configure
Common Test to create one dedicated private directory per
test case and execution instead. This is accomplished by means of
the flag/option: <span class="code">create_priv_dir</span> (to be used with the
<span class="code">ct_run</span> program, the <span class="code"><span class="bold_code"><a href="ct.html#run_test-1">ct:run_test/1</a></span></span> function, or
as test specification term). There are three possible values
for this option:
<ul>
<li><span class="code">auto_per_run</span></li>
<li><span class="code">auto_per_tc</span></li>
<li><span class="code">manual_per_tc</span></li>
</ul>
The first value indicates the default priv_dir behaviour, i.e.
one private directory created per test run. The two latter
values tell Common Test to generate a unique test directory name
per test case and execution. If the auto version is used, <strong>all</strong>
private directories will be created automatically. This can obviously
become very inefficient for test runs with many test cases and/or
repetitions. Therefore, in case the manual version is instead used, the
test case must tell Common Test to create priv_dir when it needs it.
It does this by calling the function <span class="code"><span class="bold_code"><a href="ct.html#make_priv_dir-0">ct:make_priv_dir/0</a></span></span>.
</p>
<div class="note">
<div class="label">Note</div>
<div class="content"><p><p>You should not depend on current working directory for
reading and writing data files since this is not portable. All
scratch files are to be written in the <span class="code">priv_dir</span> and all
data files should be located in <span class="code">data_dir</span>. Note also that
the Common Test server sets current working directory to the test case
log directory at the start of every case.
</p></p></div>
</div>
<h3><a name="id73327">4.16
Execution environment</a></h3>
<p>Each test case is executed by a dedicated Erlang process. The
process is spawned when the test case starts, and terminated when
the test case is finished. The configuration functions
<span class="code">init_per_testcase</span> and <span class="code">end_per_testcase</span> execute on the
same process as the test case.
</p>
<p>The configuration functions <span class="code">init_per_suite</span> and
<span class="code">end_per_suite</span> execute, like test cases, on dedicated Erlang
processes.
</p>
<h3><a name="id73364">4.17
Timetrap timeouts</a></h3>
<a name="timetraps"></a>
<p>The default time limit for a test case is 30 minutes, unless a
<span class="code">timetrap</span> is specified either by the suite-, group-,
or test case info function. The timetrap timeout value defined by
<span class="code">suite/0</span> is the value that will be used for each test case
in the suite (as well as for the configuration functions
<span class="code">init_per_suite/1</span>, <span class="code">end_per_suite/1</span>, <span class="code">init_per_group/2</span>,
and <span class="code">end_per_group/2</span>). A timetrap value defined by
<span class="code">group(GroupName)</span> overrides one defined by <span class="code">suite()</span>
and will be used for each test case in group <span class="code">GroupName</span>, and any
of its sub-groups. If a timetrap value is defined by <span class="code">group/1</span>
for a sub-group, it overrides that of its higher level groups. Timetrap
values set by individual test cases (by means of the test case info
function) override both group- and suite- level timetraps.</p>
<p>It is also possible to dynamically set/reset a timetrap during the
excution of a test case, or configuration function. This is done by calling
<span class="code"><span class="bold_code"><a href="ct.html#timetrap-1">ct:timetrap/1</a></span></span>. This function cancels the current timetrap
and starts a new one (that stays active until timeout, or end of the
current function).</p>
<p>Timetrap values can be extended with a multiplier value specified at
startup with the <span class="code">multiply_timetraps</span> option. It is also possible
to let the test server decide to scale up timetrap timeout values
automatically, e.g. if tools such as cover or trace are running during
the test. This feature is disabled by default and can be enabled with
the <span class="code">scale_timetraps</span> start option.</p>
<p>If a test case needs to suspend itself for a time that also gets
multipled by <span class="code">multiply_timetraps</span> (and possibly also scaled up if
<span class="code">scale_timetraps</span> is enabled), the function <span class="code"><span class="bold_code"><a href="ct.html#sleep-1">ct:sleep/1</a></span></span>
may be used (instead of e.g. <span class="code">timer:sleep/1</span>).</p>
<p>A function (<span class="code">fun/0</span> or <span class="code">MFA</span>) may be specified as
timetrap value in the suite-, group- and test case info function, as
well as argument to the <span class="code"><span class="bold_code"><a href="ct.html#timetrap-1">ct:timetrap/1</a></span></span> function. Examples:</p>
<p><span class="code">{timetrap,{my_test_utils,timetrap,[?MODULE,system_start]}}</span></p>
<p><span class="code">ct:timetrap(fun() -> my_timetrap(TestCaseName, Config) end)</span></p>
<p>The user timetrap function may be used for two things:</p>
<ul>
<li>To act as a timetrap - the timeout is triggered when the
function returns.</li>
<li>To return a timetrap time value (other than a function).</li>
</ul>
<p>Before execution of the timetrap function (which is performed
on a parallel, dedicated timetrap process), Common Test cancels
any previously set timer for the test case or configuration function.
When the timetrap function returns, the timeout is triggered, <strong>unless</strong>
the return value is a valid timetrap time, such as an integer,
or a <span class="code">{SecMinOrHourTag,Time}</span> tuple (see the
<span class="bold_code"><a href="common_test.html">common_test reference manual</a></span> for
details). If a time value is returned, a new timetrap is started
to generate a timeout after the specified time.</p>
<p>The user timetrap function may of course return a time value after a delay,
and if so, the effective timetrap time is the delay time <strong>plus</strong> the
returned time.</p>
<h3><a name="id73549">4.18
Logging - categories and verbosity levels</a></h3>
<a name="logging"></a>
<p>Common Test provides three main functions for printing strings:</p>
<ul>
<li><span class="code">ct:log(Category, Importance, Format, Args)</span></li>
<li><span class="code">ct:print(Category, Importance, Format, Args)</span></li>
<li><span class="code">ct:pal(Category, Importance, Format, Args)</span></li>
</ul>
<p>The <span class="code">log/1/2/3/4</span> function will print a string to the test case
log file. The <span class="code">print/1/2/3/4</span> function will print the string to screen,
and the <span class="code">pal/1/2/3/4</span> function will print the same string both to file and
screen. (The functions are documented in the <span class="code">ct</span> reference manual).</p>
<p>The optional <span class="code">Category</span> argument may be used to categorize the
log printout, and categories can be used for two things:</p>
<ul>
<li>To compare the importance of the printout to a specific
verbosity level, and</li>
<li>to format the printout according to a user specific HTML
Style Sheet (CSS).</li>
</ul>
<p>The <span class="code">Importance</span> argument specifies a level of importance
which, compared to a verbosity level (general and/or set per category),
determines if the printout should be visible or not. <span class="code">Importance</span>
is an arbitrary integer in the range 0..99. Pre-defined constants
exist in the <span class="code">ct.hrl</span> header file. The default importance level,
<span class="code">?STD_IMPORTANCE</span> (used if the <span class="code">Importance</span> argument is not
provided), is 50. This is also the importance used for standard IO, e.g.
from printouts made with <span class="code">io:format/2</span>, <span class="code">io:put_chars/1</span>, etc.</p>
<p><span class="code">Importance</span> is compared to a verbosity level set by means of the
<span class="code">verbosity</span> start flag/option. The verbosity level can be set per
category and/or generally. The default verbosity level, <span class="code">?STD_VERBOSITY</span>,
is 50, i.e. all standard IO gets printed. If a lower verbosity level is set,
standard IO printouts will be ignored. Common Test performs the following test:</p>
<div class="example"><pre>Importance >= (100-VerbosityLevel)</pre></div>
<p>This also means that verbosity level 0 effectively turns all logging off
(with the exception of printouts made by Common Test itself).</p>
<p>The general verbosity level is not associated with any particular
category. This level sets the threshold for the standard IO printouts,
uncategorized <span class="code">ct:log/print/pal</span> printouts, as well as
printouts for categories with undefined verbosity level.</p>
<p>Example:</p>
<div class="example"><pre>
Some printouts during test case execution:
io:format("1. Standard IO, importance = ~w~n", [?STD_IMPORTANCE]),
ct:log("2. Uncategorized, importance = ~w", [?STD_IMPORTANCE]),
ct:log(info, "3. Categorized info, importance = ~w", [?STD_IMPORTANCE]]),
ct:log(info, ?LOW_IMPORTANCE, "4. Categorized info, importance = ~w", [?LOW_IMPORTANCE]),
ct:log(error, "5. Categorized error, importance = ~w", [?HI_IMPORTANCE]),
ct:log(error, ?HI_IMPORTANCE, "6. Categorized error, importance = ~w", [?MAX_IMPORTANCE]),
If starting the test without specifying any verbosity levels:
$ ct_run ...
the following gets printed:
1. Standard IO, importance = 50
2. Uncategorized, importance = 50
3. Categorized info, importance = 50
5. Categorized error, importance = 75
6. Categorized error, importance = 99
If starting the test with:
$ ct_run -verbosity 1 and info 75
the following gets printed:
3. Categorized info, importance = 50
4. Categorized info, importance = 25
6. Categorized error, importance = 99
</pre></div>
<p>How categories can be mapped to CSS tags is documented in the
<span class="bold_code"><a href="run_test_chapter.html#html_stylesheet">Running Tests</a></span>
chapter.</p>
<p>The <span class="code">Format</span> and <span class="code">Args</span> arguments in <span class="code">ct:log/print/pal</span> are
always passed on to the <span class="code">io:format/3</span> function in <span class="code">stdlib</span>
(please see the <span class="code">io</span> manual page for details).</p>
<p>For more information about log files, please see the
<span class="bold_code"><a href="run_test_chapter.html#log_files">Running Tests</a></span> chapter.</p>
<h3><a name="id73756">4.19
Illegal dependencies</a></h3>
<p>Even though it is highly efficient to write test suites with
the Common Test framework, there will surely be mistakes made,
mainly due to illegal dependencies. Noted below are some of the
more frequent mistakes from our own experience with running the
Erlang/OTP test suites.</p>
<ul>
<li>Depending on current directory, and writing there:<br>
<p>This is a common error in test suites. It is assumed that
the current directory is the same as what the author used as
current directory when the test case was developed. Many test
cases even try to write scratch files to this directory. Instead
<span class="code">data_dir</span> and <span class="code">priv_dir</span> should be used to locate
data and for writing scratch files.
</p>
</li>
<li>Depending on execution order:<br>
<p>During development of test suites, no assumption should preferrably
be made about the execution order of the test cases or suites.
E.g. a test case should not assume that a server it depends on,
has already been started by a previous test case. There are
several reasons for this:
</p>
<p>Firstly, the user/operator may specify the order at will, and maybe
a different execution order is more relevant or efficient on
some particular occasion. Secondly, if the user specifies a whole
directory of test suites for his/her test, the order the suites are
executed will depend on how the files are listed by the operating
system, which varies between systems. Thirdly, if a user
wishes to run only a subset of a test suite, there is no way
one test case could successfully depend on another.
</p>
</li>
<li>Depending on Unix:<br>
<p>Running unix commands through <span class="code">os:cmd</span> are likely
not to work on non-unix platforms.
</p>
</li>
<li>Nested test cases:<br>
<p>Invoking a test case from another not only tests the same
thing twice, but also makes it harder to follow what exactly
is being tested. Also, if the called test case fails for some
reason, so will the caller. This way one error gives cause to
several error reports, which is less than ideal.
</p>
<p>Functionality common for many test case functions may be implemented
in common help functions. If these functions are useful for test cases
across suites, put the help functions into common help modules.
</p>
</li>
<li>Failure to crash or exit when things go wrong:<br>
<p>Making requests without checking that the return value
indicates success may be ok if the test case will fail at a
later stage, but it is never acceptable just to print an error
message (into the log file) and return successfully. Such test cases
do harm since they create a false sense of security when overviewing
the test results.
</p>
</li>
<li>Messing up for subsequent test cases:<br>
<p>Test cases should restore as much of the execution
environment as possible, so that the subsequent test cases will
not crash because of execution order of the test cases.
The function <span class="code">end_per_testcase</span> is suitable for this.
</p>
</li>
</ul>
</div>
<div class="footer">
<hr>
<p>Copyright © 2003-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
