<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Using Common Test for Large Scale Testing</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/common_test-1.6.3.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Common Test</strong><br><strong>User's Guide</strong><br><small>Version 1.6.3</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Common Test Basics" expanded="false">Common Test Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id66143">Introduction</a></li>
<li title="Test Suite Organisation"><a href="basics_chapter.html#id61412">Test Suite Organisation</a></li>
<li title="Support Libraries"><a href="basics_chapter.html#id67203">Support Libraries</a></li>
<li title="Suites and Test Cases"><a href="basics_chapter.html#id61632">Suites and Test Cases</a></li>
<li title="External Interfaces"><a href="basics_chapter.html#id63930">External Interfaces</a></li>
</ul>
</li>
<li id="no" title="Getting Started" expanded="false">Getting Started<ul>
<li><a href="getting_started_chapter.html">
Top of chapter
</a></li>
<li title="Are you new around here?"><a href="getting_started_chapter.html#id61356">Are you new around here?</a></li>
<li title="Test case execution"><a href="getting_started_chapter.html#id63050">Test case execution</a></li>
<li title="A simple test suite"><a href="getting_started_chapter.html#id63909">A simple test suite</a></li>
<li title="A test suite with configuration functions"><a href="getting_started_chapter.html#id68094">A test suite with configuration functions</a></li>
<li title="What happens next?"><a href="getting_started_chapter.html#id62545">What happens next?</a></li>
</ul>
</li>
<li id="no" title="Installation" expanded="false">Installation<ul>
<li><a href="install_chapter.html">
Top of chapter
</a></li>
<li title="General information"><a href="install_chapter.html#id66699">General information</a></li>
</ul>
</li>
<li id="no" title="Writing Test Suites" expanded="false">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id71900">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id70761">Test suites</a></li>
<li title="Init and end per suite"><a href="write_test_chapter.html#id70806">Init and end per suite</a></li>
<li title="Init and end per test case"><a href="write_test_chapter.html#id70917">Init and end per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id71162">Test cases</a></li>
<li title="Test case info function"><a href="write_test_chapter.html#id71322">Test case info function</a></li>
<li title="Test suite info function"><a href="write_test_chapter.html#id71521">Test suite info function</a></li>
<li title="Test case groups"><a href="write_test_chapter.html#id71611">Test case groups</a></li>
<li title="The parallel property and nested groups"><a href="write_test_chapter.html#id72823">The parallel property and nested groups</a></li>
<li title="Parallel test cases and IO"><a href="write_test_chapter.html#id72860">Parallel test cases and IO</a></li>
<li title="Repeated groups"><a href="write_test_chapter.html#id72908">Repeated groups</a></li>
<li title="Shuffled test case order"><a href="write_test_chapter.html#id73046">Shuffled test case order</a></li>
<li title="Group info function"><a href="write_test_chapter.html#id73105">Group info function</a></li>
<li title="Info functions for init- and end-configuration"><a href="write_test_chapter.html#id73139">Info functions for init- and end-configuration</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id73202">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id73327">Execution environment</a></li>
<li title="Timetrap timeouts"><a href="write_test_chapter.html#id73364">Timetrap timeouts</a></li>
<li title="Logging - categories and verbosity levels"><a href="write_test_chapter.html#id73549">Logging - categories and verbosity levels</a></li>
<li title="Illegal dependencies"><a href="write_test_chapter.html#id73756">Illegal dependencies</a></li>
</ul>
</li>
<li id="no" title="Test Structure" expanded="false">Test Structure<ul>
<li><a href="test_structure_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_structure_chapter.html#id73930">Test structure</a></li>
<li title="Skipping test cases"><a href="test_structure_chapter.html#id73944">Skipping test cases</a></li>
<li title="Definition of terms"><a href="test_structure_chapter.html#id74020">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Examples and Templates" expanded="false">Examples and Templates<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite example"><a href="example_chapter.html#id74246">Test suite example</a></li>
<li title="Test suite templates"><a href="example_chapter.html#id74298">Test suite templates</a></li>
</ul>
</li>
<li id="no" title="Running Tests" expanded="false">Running Tests<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the Common Test Framework"><a href="run_test_chapter.html#id74488">Using the Common Test Framework</a></li>
<li title="Automatic compilation of test suites and help modules"><a href="run_test_chapter.html#id74528">Automatic compilation of test suites and help modules</a></li>
<li title="Running tests from the OS command line"><a href="run_test_chapter.html#id74645">Running tests from the OS command line</a></li>
<li title="Running tests from the Erlang shell or from an Erlang program"><a href="run_test_chapter.html#id75142">Running tests from the Erlang shell or from an Erlang program</a></li>
<li title="Test case group execution"><a href="run_test_chapter.html#id75304">Test case group execution</a></li>
<li title="Running the interactive shell mode"><a href="run_test_chapter.html#id75821">Running the interactive shell mode</a></li>
<li title="Step by step execution of test cases with the Erlang Debugger"><a href="run_test_chapter.html#id76003">Step by step execution of test cases with the Erlang Debugger</a></li>
<li title="Test Specifications"><a href="run_test_chapter.html#id76102">Test Specifications</a></li>
<li title="Running tests from the Web based GUI"><a href="run_test_chapter.html#id76643">Running tests from the Web based GUI</a></li>
<li title="Log files"><a href="run_test_chapter.html#id76735">Log files</a></li>
<li title="HTML Style Sheets"><a href="run_test_chapter.html#id77053">HTML Style Sheets</a></li>
<li title="Repeating tests"><a href="run_test_chapter.html#id77201">Repeating tests</a></li>
<li title="Silent Connections"><a href="run_test_chapter.html#id77412">Silent Connections</a></li>
</ul>
</li>
<li id="no" title="External Configuration Data" expanded="false">External Configuration Data<ul>
<li><a href="config_file_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="config_file_chapter.html#id77631">General</a></li>
<li title="Syntax"><a href="config_file_chapter.html#id77671">Syntax</a></li>
<li title="Requiring and reading configuration data"><a href="config_file_chapter.html#id77695">Requiring and reading configuration data</a></li>
<li title="Using configuration variables defined in multiple files"><a href="config_file_chapter.html#id77839">Using configuration variables defined in multiple files</a></li>
<li title="Encrypted configuration files"><a href="config_file_chapter.html#id77870">Encrypted configuration files</a></li>
<li title="Opening connections by using configuration data"><a href="config_file_chapter.html#id77934">Opening connections by using configuration data</a></li>
<li title="User specific configuration data formats"><a href="config_file_chapter.html#id78002">User specific configuration data formats</a></li>
<li title="Examples of configuration data handling"><a href="config_file_chapter.html#id78216">Examples of configuration data handling</a></li>
<li title="Example of user specific configuration handler"><a href="config_file_chapter.html#id78268">Example of user specific configuration handler</a></li>
</ul>
</li>
<li id="no" title="Code Coverage Analysis" expanded="false">Code Coverage Analysis<ul>
<li><a href="cover_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="cover_chapter.html#id78384">General</a></li>
<li title="Usage"><a href="cover_chapter.html#id78404">Usage</a></li>
<li title="The cover specification file"><a href="cover_chapter.html#id78510">The cover specification file</a></li>
<li title="Logging"><a href="cover_chapter.html#id78563">Logging</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Using Common Test for Large Scale Testing" expanded="true">Using Common Test for Large Scale Testing<ul>
<li><a href="ct_master_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_master_chapter.html#id78634">General</a></li>
<li title="Usage"><a href="ct_master_chapter.html#id78668">Usage</a></li>
<li title="Test Specifications"><a href="ct_master_chapter.html#id78815">Test Specifications</a></li>
<li title="Automatic startup of test target nodes"><a href="ct_master_chapter.html#id78987">Automatic startup of test target nodes</a></li>
</ul>
</li>
<li id="no" title="Event Handling" expanded="false">Event Handling<ul>
<li><a href="event_handler_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="event_handler_chapter.html#id79190">General</a></li>
<li title="Usage"><a href="event_handler_chapter.html#id79226">Usage</a></li>
</ul>
</li>
<li id="no" title="Dependencies between Test Cases and Suites" expanded="false">Dependencies between Test Cases and Suites<ul>
<li><a href="dependencies_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="dependencies_chapter.html#id80143">General</a></li>
<li title="Saving configuration data"><a href="dependencies_chapter.html#id80265">Saving configuration data</a></li>
<li title="Sequences"><a href="dependencies_chapter.html#id80430">Sequences</a></li>
</ul>
</li>
<li id="no" title="Common Test Hooks" expanded="false">Common Test Hooks<ul>
<li><a href="ct_hooks_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_hooks_chapter.html#id80622">General</a></li>
<li title="Installing a CTH"><a href="ct_hooks_chapter.html#id80676">Installing a CTH</a></li>
<li title="CTH Scope"><a href="ct_hooks_chapter.html#id80835">CTH Scope</a></li>
<li title="Manipulating tests"><a href="ct_hooks_chapter.html#id81189">Manipulating tests</a></li>
<li title="Example CTH"><a href="ct_hooks_chapter.html#id81482">Example CTH</a></li>
<li title="Built-in CTHs"><a href="ct_hooks_chapter.html#id81535">Built-in CTHs</a></li>
</ul>
</li>
<li id="no" title="Some thoughts about testing" expanded="false">Some thoughts about testing<ul>
<li><a href="why_test_chapter.html">
Top of chapter
</a></li>
<li title="Goals"><a href="why_test_chapter.html#id81722">Goals</a></li>
<li title="What to test?"><a href="why_test_chapter.html#id81742">What to test?</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>10 Using Common Test for Large Scale Testing</h1>
<h3><a name="id78634">10.1
General</a></h3>
<a name="general"></a>
<p>Large scale automated testing requires running multiple independent
test sessions in parallel. This is accomplished by running
a number of Common Test nodes on one or more hosts, testing
different target systems. Configuring, starting and controlling the
test nodes independently can be a cumbersome operation. To aid
this kind of automated large scale testing, CT offers a master test
node component, CT Master, that handles central configuration and control
in a system of distributed CT nodes.</p>
<p>The CT Master server runs on one dedicated Erlang node and uses distributed
Erlang to communicate with any number of CT test nodes, each hosting a regular
CT server. Test specifications are used as input to specify what to test on which
test nodes, using what configuration.</p>
<p>The CT Master server writes progress information to HTML log files similarly
to the regular CT server. The logs contain test statistics and links to the
log files written by each independent CT server.</p>
<p>The CT master API is exported by the <span class="code">ct_master</span> module.</p>
<h3><a name="id78668">10.2
Usage</a></h3>
<p>CT Master requires all test nodes to be on the same network and share a common
file system. As of this date, CT Master can not start test nodes
automatically. The nodes must have been started in advance for CT Master to be
able to start test sessions on them.</p>
<p>Tests are started by calling:</p>
<p><span class="code">ct_master:run(TestSpecs)</span> or
<span class="code">ct_master:run(TestSpecs, InclNodes, ExclNodes)</span></p>
<p><span class="code">TestSpecs</span> is either the name of a test specification file (string) or a list
of test specifications. In case of a list, the specifications will be handled (and
the corresponding tests executed) in sequence. An element in a <span class="code">TestSpecs</span> list
can also be list of test specifications. The specifications in such a list will be
merged into one combined specification prior to test execution. For example:</p>
<p><span class="code">ct_master:run(["ts1","ts2",["ts3","ts4"]])</span></p>
<p>means first the tests specified by "ts1" will run, then the tests specified by "ts2"
and finally the tests specified by both "ts3" and "ts4".</p>
<p>The <span class="code">InclNodes</span> argument to <span class="code">run/3</span> is a list of node names. The <span class="code">run/3</span>
function runs the tests in <span class="code">TestSpecs</span> just like <span class="code">run/1</span> but will also
take any test in <span class="code">TestSpecs</span> that's not explicitly tagged with a particular
node name and execute it on the nodes listed in <span class="code">InclNodes</span>. By using <span class="code">run/3</span>
this way it is possible to use any test specification, with or without node information,
in a large scale test environment! <span class="code">ExclNodes</span> is a list of nodes that should be
excluded from the test. I.e. tests that have been specified in the test specification
to run on a particular node will not be performed if that node is at runtime
listed in <span class="code">ExclNodes</span>.</p>
<p>If CT Master fails initially to connect to any of the test nodes specified in a
test specification or in the <span class="code">InclNodes</span> list, the operator will be prompted with
the option to either start over again (after manually checking the status of the
node(s) in question), to run without the missing nodes, or to abort the operation.</p>
<p>When tests start, CT Master prints information to console about the nodes that are
involved. CT Master also reports when tests finish, successfully or unsuccessfully. If
connection is lost to a node, the test on that node is considered finished. CT Master
will not attempt to reestablish contact with the failing node. At any time to get the
current status of the test nodes, call the function:</p>
<p><span class="code">ct_master:progress()</span></p>
<p>To stop one or more tests, use:</p>
<p><span class="code">ct_master:abort()</span> (stop all) or <span class="code">ct_master:abort(Nodes)</span></p>
<p>For detailed information about the <span class="code">ct_master</span> API, please see the
<span class="bold_code"><a href="ct_master.html">manual page</a></span> for this module.</p>
<h3><a name="id78815">10.3
Test Specifications</a></h3>
<a name="test_specifications"></a>
<p>The test specifications used as input to CT Master are fully compatible with the
specifications used as input to the regular CT server. The syntax is described in the
<span class="bold_code"><a href="run_test_chapter.html#test_specifications">Running Test Suites</a></span>
chapter.</p>
<p>All test specification terms can have a <span class="code">NodeRefs</span> element. This element
specifies which node or nodes a configuration operation or a test is to be executed
on. <span class="code">NodeRefs</span> is defined as:</p>
<p><span class="code">NodeRefs = all_nodes | [NodeRef] | NodeRef</span></p>
<p>where</p>
<p><span class="code">NodeRef = NodeAlias | node() | master</span></p>
<p>A <span class="code">NodeAlias</span> (<span class="code">atom()</span>) is used in a test specification as a
reference to a node name (so the actual node name only needs to be declared once,
which can of course also be achieved using constants).
The alias is declared with a <span class="code">node</span> term:</p>
<p><span class="code">{node, NodeAlias, NodeName}</span></p>
<p>If <span class="code">NodeRefs</span> has the value <span class="code">all_nodes</span>, the operation or test will
be performed on all given test nodes. (Declaring a term without a <span class="code">NodeRefs</span>
element actually has the same effect). If <span class="code">NodeRefs</span> has the value
<span class="code">master</span>, the operation is only performed on the CT Master node (namely set
the log directory or install an event handler).</p>
<p>Consider the example in the
<span class="bold_code"><a href="run_test_chapter.html#test_specifications">Running Test Suites</a></span>
chapter, now extended with node information and intended to be executed by the
CT Master:</p>
<div class="example"><pre>
{define, 'Top', "/home/test"}.
{define, 'T1', "'Top'/t1"}.
{define, 'T2', "'Top'/t2"}.
{define, 'T3', "'Top'/t3"}.
{define, 'CfgFile', "config.cfg"}.
{define, 'Node', ct_node}.
{node, node1, 'Node@host_x'}.
{node, node2, 'Node@host_y'}.
{logdir, master, "'Top'/master_logs"}.
{logdir, "'Top'/logs"}.
{config, node1, "'T1'/'CfgFile'"}.
{config, node2, "'T2'/'CfgFile'"}.
{config, "'T3'/'CfgFile'"}.
{suites, node1, 'T1', all}.
{skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
{skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
{skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.
{suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
{cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.
{skip_suites, 'T3', all, "Not implemented"}.</pre></div>
<p>This example specifies the same tests as the original example. But
now if started with a call to <span class="code">ct_master:run(TestSpecName)</span>, the
t1 test will be executed on node <span class="code">ct_node@host_x</span> (node1), the
t2 test on <span class="code">ct_node@host_y</span> (node2) and the t3 test on both
node1 and node2. The t1 config file will only be read on
node1 and the t2 config file only on node2, while the t3 config file
will be read on both node1 and node2. Both test nodes will write log
files to the same directory. (The CT Master node will however use a
different log directory than the test nodes).</p>
<p>If the test session is instead started with a call to
<span class="code">ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x])</span>,
the result is that the t1 test does not run on
<span class="code">ct_node@host_x</span> (or any other node) while the t3 test runs on
<span class="code">ct_node@host_y</span> and <span class="code">ct_node@host_z</span>.</p>
<p>A nice feature is that a test specification that includes node
information can still be used as input to the regular Common Test server
(as described in the
<span class="bold_code"><a href="run_test_chapter.html#test_specifications">Running Test Suites</a></span>
chapter). The result is that any test specified to run on a node with the same
name as the Common Test node in question (typically <span class="code">ct@somehost</span> if started
with the <span class="code">ct_run</span> program), will be performed. Tests without explicit
node association will always be performed too of course!</p>
<h3><a name="id78987">10.4
Automatic startup of test target nodes</a></h3>
<a name="ct_slave"></a>
<p>Is is possible to automatically start, and perform initial actions, on
test target nodes by using the test specification term <span class="code">init</span>.</p>
<p>Currently, two sub-terms are supported, <span class="code">node_start</span> and <span class="code">eval</span>.</p>
<p>Example:</p>
<div class="example"><pre>
{node, node1, node1@host1}.
{node, node2, node1@host2}.
{node, node3, node2@host2}.
{node, node4, node1@host3}.
{init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
{init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
{init, node4, {eval, {module, function, []}}}.</pre></div>
<p>This test specification declares that <span class="code">node1@host1</span> is to be started using
the user callback function <span class="code">callback_module:my_slave_callback/0</span>, and nodes
<span class="code">node1@host2</span> and <span class="code">node2@host2</span> will be started with the default callback
module <span class="code">ct_slave</span>. The given user name and password is used to log into remote
host <span class="code">host2</span>. Also, the function <span class="code">module:function/0</span> will be evaluated on
<span class="code">node1@host3</span>, and the result of this call will be printed to the log.</p>
<p>The default <span class="bold_code"><a href="ct_slave.html">ct_slave</a></span> callback module,
which is part of the Common Test application, has the following features:
<ul>
<li>Starting Erlang target nodes on local or remote hosts
(ssh is used for communication).
</li>
<li>Ability to start an Erlang emulator with additional flags
(any flags supported by <span class="code">erl</span> are supported).
</li>
<li>Supervision of a node being started by means of internal callback
functions. Used to prevent hanging nodes. (Configurable).
</li>
<li>Monitoring of the master node by the slaves. A slave node may be
stopped in case the master node terminates. (Configurable).
</li>
<li>Execution of user functions after a slave node is started.
Functions can be given as a list of {Module, Function, Arguments} tuples.
</li>
</ul>
</p>
<p>Note that it is possible to specify an <span class="code">eval</span> term for the node as well
as <span class="code">startup_functions</span> in the <span class="code">node_start</span> options list. In this
case first the node will be started, then the <span class="code">startup_functions</span> are
executed, and finally functions specified with <span class="code">eval</span> are called.
</p>
</div>
<div class="footer">
<hr>
<p>Copyright © 2003-2012 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
17
>
i386
>
media
>
updates
>
by-pkgid
>
675c8c8167236dfcf8d66da674f931e8
>
files
>
241
