<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Common Test Basics</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/common_test-1.6.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Common Test</strong><br><strong>User's Guide</strong><br><small>Version 1.6.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="loadscrollpos" title="Common Test Basics" expanded="true">Common Test Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id66143">Introduction</a></li>
<li title="Test Suite Organisation"><a href="basics_chapter.html#id61412">Test Suite Organisation</a></li>
<li title="Support Libraries"><a href="basics_chapter.html#id67203">Support Libraries</a></li>
<li title="Suites and Test Cases"><a href="basics_chapter.html#id61632">Suites and Test Cases</a></li>
<li title="External Interfaces"><a href="basics_chapter.html#id63930">External Interfaces</a></li>
</ul>
</li>
<li id="no" title="Getting Started" expanded="false">Getting Started<ul>
<li><a href="getting_started_chapter.html">
Top of chapter
</a></li>
<li title="Are you new around here?"><a href="getting_started_chapter.html#id61356">Are you new around here?</a></li>
<li title="Test case execution"><a href="getting_started_chapter.html#id63050">Test case execution</a></li>
<li title="A simple test suite"><a href="getting_started_chapter.html#id63909">A simple test suite</a></li>
<li title="A test suite with configuration functions"><a href="getting_started_chapter.html#id68094">A test suite with configuration functions</a></li>
<li title="What happens next?"><a href="getting_started_chapter.html#id62545">What happens next?</a></li>
</ul>
</li>
<li id="no" title="Installation" expanded="false">Installation<ul>
<li><a href="install_chapter.html">
Top of chapter
</a></li>
<li title="General information"><a href="install_chapter.html#id66699">General information</a></li>
</ul>
</li>
<li id="no" title="Writing Test Suites" expanded="false">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id71900">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id70761">Test suites</a></li>
<li title="Init and end per suite"><a href="write_test_chapter.html#id70806">Init and end per suite</a></li>
<li title="Init and end per test case"><a href="write_test_chapter.html#id70917">Init and end per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id71162">Test cases</a></li>
<li title="Test case info function"><a href="write_test_chapter.html#id71322">Test case info function</a></li>
<li title="Test suite info function"><a href="write_test_chapter.html#id71521">Test suite info function</a></li>
<li title="Test case groups"><a href="write_test_chapter.html#id71611">Test case groups</a></li>
<li title="The parallel property and nested groups"><a href="write_test_chapter.html#id72823">The parallel property and nested groups</a></li>
<li title="Parallel test cases and IO"><a href="write_test_chapter.html#id72860">Parallel test cases and IO</a></li>
<li title="Repeated groups"><a href="write_test_chapter.html#id72908">Repeated groups</a></li>
<li title="Shuffled test case order"><a href="write_test_chapter.html#id73046">Shuffled test case order</a></li>
<li title="Group info function"><a href="write_test_chapter.html#id73105">Group info function</a></li>
<li title="Info functions for init- and end-configuration"><a href="write_test_chapter.html#id73139">Info functions for init- and end-configuration</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id73202">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id73327">Execution environment</a></li>
<li title="Timetrap timeouts"><a href="write_test_chapter.html#id73364">Timetrap timeouts</a></li>
<li title="Logging - categories and verbosity levels"><a href="write_test_chapter.html#id73549">Logging - categories and verbosity levels</a></li>
<li title="Illegal dependencies"><a href="write_test_chapter.html#id73756">Illegal dependencies</a></li>
</ul>
</li>
<li id="no" title="Test Structure" expanded="false">Test Structure<ul>
<li><a href="test_structure_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_structure_chapter.html#id73930">Test structure</a></li>
<li title="Skipping test cases"><a href="test_structure_chapter.html#id73944">Skipping test cases</a></li>
<li title="Definition of terms"><a href="test_structure_chapter.html#id74020">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Examples and Templates" expanded="false">Examples and Templates<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite example"><a href="example_chapter.html#id74246">Test suite example</a></li>
<li title="Test suite templates"><a href="example_chapter.html#id74298">Test suite templates</a></li>
</ul>
</li>
<li id="no" title="Running Tests" expanded="false">Running Tests<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the Common Test Framework"><a href="run_test_chapter.html#id74488">Using the Common Test Framework</a></li>
<li title="Automatic compilation of test suites and help modules"><a href="run_test_chapter.html#id74528">Automatic compilation of test suites and help modules</a></li>
<li title="Running tests from the OS command line"><a href="run_test_chapter.html#id74645">Running tests from the OS command line</a></li>
<li title="Running tests from the Erlang shell or from an Erlang program"><a href="run_test_chapter.html#id75142">Running tests from the Erlang shell or from an Erlang program</a></li>
<li title="Test case group execution"><a href="run_test_chapter.html#id75304">Test case group execution</a></li>
<li title="Running the interactive shell mode"><a href="run_test_chapter.html#id75821">Running the interactive shell mode</a></li>
<li title="Step by step execution of test cases with the Erlang Debugger"><a href="run_test_chapter.html#id76003">Step by step execution of test cases with the Erlang Debugger</a></li>
<li title="Test Specifications"><a href="run_test_chapter.html#id76102">Test Specifications</a></li>
<li title="Running tests from the Web based GUI"><a href="run_test_chapter.html#id76643">Running tests from the Web based GUI</a></li>
<li title="Log files"><a href="run_test_chapter.html#id76735">Log files</a></li>
<li title="HTML Style Sheets"><a href="run_test_chapter.html#id77053">HTML Style Sheets</a></li>
<li title="Repeating tests"><a href="run_test_chapter.html#id77201">Repeating tests</a></li>
<li title="Silent Connections"><a href="run_test_chapter.html#id77412">Silent Connections</a></li>
</ul>
</li>
<li id="no" title="External Configuration Data" expanded="false">External Configuration Data<ul>
<li><a href="config_file_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="config_file_chapter.html#id77631">General</a></li>
<li title="Syntax"><a href="config_file_chapter.html#id77671">Syntax</a></li>
<li title="Requiring and reading configuration data"><a href="config_file_chapter.html#id77695">Requiring and reading configuration data</a></li>
<li title="Using configuration variables defined in multiple files"><a href="config_file_chapter.html#id77839">Using configuration variables defined in multiple files</a></li>
<li title="Encrypted configuration files"><a href="config_file_chapter.html#id77870">Encrypted configuration files</a></li>
<li title="Opening connections by using configuration data"><a href="config_file_chapter.html#id77934">Opening connections by using configuration data</a></li>
<li title="User specific configuration data formats"><a href="config_file_chapter.html#id78002">User specific configuration data formats</a></li>
<li title="Examples of configuration data handling"><a href="config_file_chapter.html#id78216">Examples of configuration data handling</a></li>
<li title="Example of user specific configuration handler"><a href="config_file_chapter.html#id78268">Example of user specific configuration handler</a></li>
</ul>
</li>
<li id="no" title="Code Coverage Analysis" expanded="false">Code Coverage Analysis<ul>
<li><a href="cover_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="cover_chapter.html#id78384">General</a></li>
<li title="Usage"><a href="cover_chapter.html#id78404">Usage</a></li>
<li title="The cover specification file"><a href="cover_chapter.html#id78510">The cover specification file</a></li>
<li title="Logging"><a href="cover_chapter.html#id78563">Logging</a></li>
</ul>
</li>
<li id="no" title="Using Common Test for Large Scale Testing" expanded="false">Using Common Test for Large Scale Testing<ul>
<li><a href="ct_master_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_master_chapter.html#id78634">General</a></li>
<li title="Usage"><a href="ct_master_chapter.html#id78668">Usage</a></li>
<li title="Test Specifications"><a href="ct_master_chapter.html#id78815">Test Specifications</a></li>
<li title="Automatic startup of test target nodes"><a href="ct_master_chapter.html#id78987">Automatic startup of test target nodes</a></li>
</ul>
</li>
<li id="no" title="Event Handling" expanded="false">Event Handling<ul>
<li><a href="event_handler_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="event_handler_chapter.html#id79190">General</a></li>
<li title="Usage"><a href="event_handler_chapter.html#id79226">Usage</a></li>
</ul>
</li>
<li id="no" title="Dependencies between Test Cases and Suites" expanded="false">Dependencies between Test Cases and Suites<ul>
<li><a href="dependencies_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="dependencies_chapter.html#id80143">General</a></li>
<li title="Saving configuration data"><a href="dependencies_chapter.html#id80265">Saving configuration data</a></li>
<li title="Sequences"><a href="dependencies_chapter.html#id80430">Sequences</a></li>
</ul>
</li>
<li id="no" title="Common Test Hooks" expanded="false">Common Test Hooks<ul>
<li><a href="ct_hooks_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_hooks_chapter.html#id80622">General</a></li>
<li title="Installing a CTH"><a href="ct_hooks_chapter.html#id80676">Installing a CTH</a></li>
<li title="CTH Scope"><a href="ct_hooks_chapter.html#id80835">CTH Scope</a></li>
<li title="Manipulating tests"><a href="ct_hooks_chapter.html#id81189">Manipulating tests</a></li>
<li title="Example CTH"><a href="ct_hooks_chapter.html#id81482">Example CTH</a></li>
<li title="Built-in CTHs"><a href="ct_hooks_chapter.html#id81535">Built-in CTHs</a></li>
</ul>
</li>
<li id="no" title="Some thoughts about testing" expanded="false">Some thoughts about testing<ul>
<li><a href="why_test_chapter.html">
Top of chapter
</a></li>
<li title="Goals"><a href="why_test_chapter.html#id81722">Goals</a></li>
<li title="What to test?"><a href="why_test_chapter.html#id81742">What to test?</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>1 Common Test Basics</h1>
<a name="basics"></a>
<h3><a name="id66143">1.1
Introduction</a></h3>
<p>The <strong>Common Test</strong> framework (CT) is a tool which supports
implementation and automated execution of test cases towards arbitrary
types of target systems. The CT framework is based on the OTP Test
Server and it's the main tool being used in all testing- and verification
activities that are part of Erlang/OTP system development- and maintenance.
</p>
<p>Test cases can be executed individually or in batches. Common Test
also features a distributed testing mode with central control and logging
(a feature that makes it possible to test multiple systems independently in
one common session, useful e.g. for running automated large-scale regression
tests).
</p>
<p>
The SUT (System Under Test) may consist of one or several target
nodes. CT contains a generic test server which, together with
other test utilities, is used to perform test case execution.
It is possible to start the tests from a GUI or from the OS- or
Erlang shell. <strong>Test suites</strong> are files (Erlang
modules) that contain the <strong>test cases</strong> (Erlang functions)
to be executed. <strong>Support modules</strong> provide functions
that the test cases utilize in order to carry out the tests.
</p>
<p>In a black-box testing scenario, CT based test programs connect to
the target system(s) via standard O&M and CLI protocols. CT
provides implementations of, and wrapper interfaces to, some of these
protocols (most of which exist as stand-alone components and
applications in OTP). The wrappers simplify configuration and add
verbosity for logging purposes. CT will be continously extended with
useful support modules. (Note however that it's
a straightforward task to use any arbitrary Erlang/OTP component
for testing purposes with Common Test, without needing a CT wrapper
for it. It's as simple as calling Erlang functions). There
are a number of target independent interfaces supported in CT, such as
Generic Telnet, FTP, etc, which can be specialized or used
directly for controlling instruments, traffic load generators, etc.
</p>
<p>Common Test is also a very useful tool for white-box testing Erlang
code (e.g. module testing), since the test programs can call exported Erlang
functions directly and there's very little overhead required for
implementing basic test suites and executing simple tests. For black-box
testing Erlang software, Erlang RPC as well as standard O&M interfaces
can for example be used.
</p>
<p>A test case can handle several connections towards one or
several target systems, instruments and traffic generators in
parallel in order to perform the necessary actions for a
test. The handling of many connections in parallel is one of
the major strengths of Common Test (thanks to the efficient
support for concurrency in the Erlang runtime system - which CT users
can take great advantage of!).
</p>
<h3><a name="id61412">1.2
Test Suite Organisation</a></h3>
<p>
The test suites are organized in test directories and each test suite
may have a separate data directory. Typically, these files and directories
are version controlled similarly to other forms of source code (possibly by
means of a version control system like GIT or Subversion). However, CT does
not itself put any requirements on (or has any form of awareness of)
possible file and directory versions.
</p>
<h3><a name="id67203">1.3
Support Libraries</a></h3>
<p>
Support libraries contain functions that are useful for all test suites,
or for test suites in a specific functional area or subsystem.
In addition to the general support libraries provided by the
CT framework, and the various libraries and applications provided by
Erlang/OTP, there might also be a need for customized (user specific)
support libraries.
</p>
<h3><a name="id61632">1.4
Suites and Test Cases</a></h3>
<p>
Testing is performed by running test suites (sets of test cases) or
individual test cases. A test suite is implemented as an Erlang module named
<span class="code"><suite_name>_SUITE.erl</span> which contains a number of test cases.
A test case is an Erlang function which tests one or more things.
The test case is the smallest unit that the CT test server deals with.
</p>
<p>
Subsets of test cases, called test case groups, may also be defined. A test case
group can have execution properties associated with it. Execution properties
specify whether the test cases in the group should be executed in
random order, in parallel, in sequence, and if the execution of the group
should be repeated. Test case groups may also be nested (i.e. a group may,
besides test cases, contain sub-groups).
</p>
<p>
Besides test cases and groups, the test suite may also contain configuration
functions. These functions are meant to be used for setting up (and verifying)
environment and state on the SUT (and/or the CT host node), required for
the tests to execute correctly. Examples of operations: Opening a connection
to the SUT, initializing a database, running an installation script, etc.
Configuration may be performed per suite, per test case group and per
individual test case.
</p>
<p>
The test suite module must conform to a
<span class="bold_code"><a href="common_test.html">callback interface</a></span>
specified by the CT test server. See the
<span class="bold_code"><a href="write_test_chapter.html#intro">Writing Test Suites</a></span> chapter
for more information.
</p>
<p>
A test case is considered successful if it returns to the caller, no matter
what the returned value is. A few return values have special meaning however
(such as <span class="code">{skip,Reason}</span> which indicates that the test case is skipped,
<span class="code">{comment,Comment}</span> which prints a comment in the log for the test case and
<span class="code">{save_config,Config}</span> which makes the CT test server pass <span class="code">Config</span> to
the next test case).
A test case failure is specified as a runtime error (a crash), no matter what
the reason for termination is. If you use Erlang pattern matching effectively,
you can take advantage of this property. The result will be concise and
readable test case functions that look much more like scripts than actual programs.
Simple example:
</p>
<div class="example"><pre>
session(_Config) ->
{started,ServerId} = my_server:start(),
{clients,[]} = my_server:get_clients(ServerId),
MyId = self(),
connected = my_server:connect(ServerId, MyId),
{clients,[MyId]} = my_server:get_clients(ServerId),
disconnected = my_server:disconnect(ServerId, MyId),
{clients,[]} = my_server:get_clients(ServerId),
stopped = my_server:stop(ServerId).
</pre></div>
<p>
As a test suite runs, all information (including output to <span class="code">stdout</span>) is
recorded in several different log files. A minimum of information is displayed
in the user console (only start and stop information, plus a note
for each failed test case).
</p>
<p>
The result from each test case is recorded in a dedicated HTML log file, created
for the particular test run. An overview page displays each test case represented
by row in a table showing total execution time, whether the case was successful,
failed or skipped, plus an optional user comment. (For a failed test case, the
reason for termination is also printed in the comment field). The overview page
has a link to each test case log file, providing simple navigation with any standard
HTML browser.
</p>
<h3><a name="id63930">1.5
External Interfaces</a></h3>
<p>
The CT test server requires that the test suite defines and exports the
following mandatory or optional callback functions:
</p>
<dl>
<dt><strong>all()</strong></dt>
<dd>Returns a list of all test cases and groups in the suite. (Mandatory)</dd>
<dt><strong>suite()</strong></dt>
<dd>Info function used to return properties for the suite. (Optional)</dd>
<dt><strong>groups()</strong></dt>
<dd>For declaring test case groups. (Optional)</dd>
<dt><strong>init_per_suite(Config)</strong></dt>
<dd>Suite level configuration function, executed before the first
test case. (Optional)</dd>
<dt><strong>end_per_suite(Config)</strong></dt>
<dd>Suite level configuration function, executed after the last
test case. (Optional)</dd>
<dt><strong>group(GroupName)</strong></dt>
<dd>Info function used to return properties for a test case group. (Optional)</dd>
<dt><strong>init_per_group(GroupName, Config)</strong></dt>
<dd>Configuration function for a group, executed before the first
test case. (Optional)</dd>
<dt><strong>end_per_group(GroupName, Config)</strong></dt>
<dd>Configuration function for a group, executed after the last
test case. (Optional)</dd>
<dt><strong>init_per_testcase(TestCase, Config)</strong></dt>
<dd>Configuration function for a testcase, executed before each
test case. (Optional)</dd>
<dt><strong>end_per_testcase(TestCase, Config)</strong></dt>
<dd>Configuration function for a testcase, executed after each
test case. (Optional)</dd>
</dl>
<p>
For each test case the CT test server expects these functions:
</p>
<dl>
<dt><strong>Testcasename()</strong></dt>
<dd>Info function that returns a list of test case properties. (Optional)</dd>
<dt><strong>Testcasename(Config)</strong></dt>
<dd>The actual test case function.</dd>
</dl>
</div>
<div class="footer">
<hr>
<p>Copyright © 2003-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
17
>
i386
>
media
>
updates
>
by-pkgid
>
675c8c8167236dfcf8d66da674f931e8
>
files
>
229
