<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns:fn="http://www.w3.org/2005/02/xpath-functions">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css">
<title>Erlang -- Some thoughts about testing</title>
</head>
<body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container">
<script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript">
<!--
function getWinHeight() {
var myHeight = 0;
if( typeof( window.innerHeight ) == 'number' ) {
//Non-IE
myHeight = window.innerHeight;
} else if( document.documentElement && ( document.documentElement.clientWidth ||
document.documentElement.clientHeight ) ) {
//IE 6+ in 'standards compliant mode'
myHeight = document.documentElement.clientHeight;
} else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) {
//IE 4 compatible
myHeight = document.body.clientHeight;
}
return myHeight;
}
function setscrollpos() {
var objf=document.getElementById('loadscrollpos');
document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2;
}
function addEvent(obj, evType, fn){
if (obj.addEventListener){
obj.addEventListener(evType, fn, true);
return true;
} else if (obj.attachEvent){
var r = obj.attachEvent("on"+evType, fn);
return r;
} else {
return false;
}
}
addEvent(window, 'load', setscrollpos);
//--></script><div id="leftnav"><div class="innertube">
<img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="users_guide.html">User's Guide</a><br><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/common_test-1.4.7.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Common Test</strong><br><strong>User's Guide</strong><br><small>Version 1.4.7</small></p>
<br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Chapters</strong></small></p>
<ul class="flipMenu" imagepath="../../../../doc/js/flipmenu">
<li id="no" title="Common Test Basics" expanded="false">Common Test Basics<ul>
<li><a href="basics_chapter.html">
Top of chapter
</a></li>
<li title="Introduction"><a href="basics_chapter.html#id2257382">Introduction</a></li>
<li title="Test Suite Organisation"><a href="basics_chapter.html#id2262699">Test Suite Organisation</a></li>
<li title="Support Libraries"><a href="basics_chapter.html#id2262725">Support Libraries</a></li>
<li title="Suites and Test Cases"><a href="basics_chapter.html#id2252225">Suites and Test Cases</a></li>
<li title="External Interfaces"><a href="basics_chapter.html#id2252025">External Interfaces</a></li>
</ul>
</li>
<li id="no" title="Installation" expanded="false">Installation<ul>
<li><a href="install_chapter.html">
Top of chapter
</a></li>
<li title="General information"><a href="install_chapter.html#id2259791">General information</a></li>
<li title="Unix/Linux"><a href="install_chapter.html#id2259969">Unix/Linux</a></li>
<li title="Windows"><a href="install_chapter.html#id2252127">Windows</a></li>
</ul>
</li>
<li id="no" title="Writing Test Suites" expanded="false">Writing Test Suites<ul>
<li><a href="write_test_chapter.html">
Top of chapter
</a></li>
<li title="Support for test suite authors"><a href="write_test_chapter.html#id2263189">Support for test suite authors</a></li>
<li title="Test suites"><a href="write_test_chapter.html#id2263446">Test suites</a></li>
<li title="Init and end per suite"><a href="write_test_chapter.html#id2252033">Init and end per suite</a></li>
<li title="Init and end per test case"><a href="write_test_chapter.html#id2263397">Init and end per test case</a></li>
<li title="Test cases"><a href="write_test_chapter.html#id2252705">Test cases</a></li>
<li title="Test case info function"><a href="write_test_chapter.html#id2252952">Test case info function</a></li>
<li title="Test suite info function"><a href="write_test_chapter.html#id2264301">Test suite info function</a></li>
<li title="Test case groups"><a href="write_test_chapter.html#id2264393">Test case groups</a></li>
<li title="The parallel property and nested groups"><a href="write_test_chapter.html#id2264666">The parallel property and nested groups</a></li>
<li title="Repeated groups"><a href="write_test_chapter.html#id2264711">Repeated groups</a></li>
<li title="Shuffled test case order"><a href="write_test_chapter.html#id2264872">Shuffled test case order</a></li>
<li title="Data and Private Directories"><a href="write_test_chapter.html#id2264941">Data and Private Directories</a></li>
<li title="Execution environment"><a href="write_test_chapter.html#id2265020">Execution environment</a></li>
<li title="Illegal dependencies"><a href="write_test_chapter.html#id2265070">Illegal dependencies</a></li>
</ul>
</li>
<li id="no" title="Test Structure" expanded="false">Test Structure<ul>
<li><a href="test_structure_chapter.html">
Top of chapter
</a></li>
<li title="Test structure"><a href="test_structure_chapter.html#id2265289">Test structure</a></li>
<li title="Skipping test cases"><a href="test_structure_chapter.html#id2265306">Skipping test cases</a></li>
<li title="Definition of terms"><a href="test_structure_chapter.html#id2265390">Definition of terms</a></li>
</ul>
</li>
<li id="no" title="Examples and Templates" expanded="false">Examples and Templates<ul>
<li><a href="example_chapter.html">
Top of chapter
</a></li>
<li title="Test suite example"><a href="example_chapter.html#id2266177">Test suite example</a></li>
<li title="Test suite templates"><a href="example_chapter.html#id2266264">Test suite templates</a></li>
</ul>
</li>
<li id="no" title="Running Test Suites" expanded="false">Running Test Suites<ul>
<li><a href="run_test_chapter.html">
Top of chapter
</a></li>
<li title="Using the Common Test Framework"><a href="run_test_chapter.html#id2266562">Using the Common Test Framework</a></li>
<li title="Automatic compilation of test suites and help modules"><a href="run_test_chapter.html#id2266607">Automatic compilation of test suites and help modules</a></li>
<li title="Running tests from the UNIX command line"><a href="run_test_chapter.html#id2266741">Running tests from the UNIX command line</a></li>
<li title="Running tests from the Web based GUI"><a href="run_test_chapter.html#id2267056">Running tests from the Web based GUI</a></li>
<li title="Running tests from the Erlang shell or from an Erlang program"><a href="run_test_chapter.html#id2267143">Running tests from the Erlang shell or from an Erlang program</a></li>
<li title="Running the interactive shell mode"><a href="run_test_chapter.html#id2267199">Running the interactive shell mode</a></li>
<li title="Step by step execution of test cases with the Erlang Debugger"><a href="run_test_chapter.html#id2267375">Step by step execution of test cases with the Erlang Debugger</a></li>
<li title="Using test specifications"><a href="run_test_chapter.html#id2267476">Using test specifications</a></li>
<li title="Log files"><a href="run_test_chapter.html#id2267715">Log files</a></li>
<li title="HTML Style Sheets"><a href="run_test_chapter.html#id2267921">HTML Style Sheets</a></li>
<li title="Repeating tests"><a href="run_test_chapter.html#id2268107">Repeating tests</a></li>
<li title="Silent Connections"><a href="run_test_chapter.html#id2268354">Silent Connections</a></li>
</ul>
</li>
<li id="no" title="Config Files" expanded="false">Config Files<ul>
<li><a href="config_file_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="config_file_chapter.html#id2268593">General</a></li>
<li title="Syntax"><a href="config_file_chapter.html#id2268633">Syntax</a></li>
<li title="Requiring and reading configuration data"><a href="config_file_chapter.html#id2268657">Requiring and reading configuration data</a></li>
<li title="Using configuration variables defined in multiple files"><a href="config_file_chapter.html#id2268807">Using configuration variables defined in multiple files</a></li>
<li title="Encrypted configuration files"><a href="config_file_chapter.html#id2268839">Encrypted configuration files</a></li>
<li title="Opening connections by using configuration data"><a href="config_file_chapter.html#id2268904">Opening connections by using configuration data</a></li>
<li title="Examples"><a href="config_file_chapter.html#id2268977">Examples</a></li>
</ul>
</li>
<li id="no" title="Code Coverage Analysis" expanded="false">Code Coverage Analysis<ul>
<li><a href="cover_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="cover_chapter.html#id2269096">General</a></li>
<li title="Usage"><a href="cover_chapter.html#id2269123">Usage</a></li>
<li title="The cover specification file"><a href="cover_chapter.html#id2269250">The cover specification file</a></li>
<li title="Logging"><a href="cover_chapter.html#id2269317">Logging</a></li>
</ul>
</li>
<li id="no" title="Using Common Test for Large Scale Testing" expanded="false">Using Common Test for Large Scale Testing<ul>
<li><a href="ct_master_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="ct_master_chapter.html#id2269392">General</a></li>
<li title="Usage"><a href="ct_master_chapter.html#id2269436">Usage</a></li>
<li title="Test Specifications"><a href="ct_master_chapter.html#id2269594">Test Specifications</a></li>
</ul>
</li>
<li id="no" title="Event Handling" expanded="false">Event Handling<ul>
<li><a href="event_handler_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="event_handler_chapter.html#id2269865">General</a></li>
<li title="Usage"><a href="event_handler_chapter.html#id2269911">Usage</a></li>
</ul>
</li>
<li id="no" title="Dependencies between Test Cases and Suites" expanded="false">Dependencies between Test Cases and Suites<ul>
<li><a href="dependencies_chapter.html">
Top of chapter
</a></li>
<li title="General"><a href="dependencies_chapter.html#id2270845">General</a></li>
<li title="Saving configuration data"><a href="dependencies_chapter.html#id2271007">Saving configuration data</a></li>
<li title="Sequences"><a href="dependencies_chapter.html#id2271201">Sequences</a></li>
</ul>
</li>
<li id="loadscrollpos" title="Some thoughts about testing" expanded="true">Some thoughts about testing<ul>
<li><a href="why_test_chapter.html">
Top of chapter
</a></li>
<li title="Goals"><a href="why_test_chapter.html#id2271413">Goals</a></li>
<li title="What to test?"><a href="why_test_chapter.html#id2271438">What to test?</a></li>
</ul>
</li>
</ul>
</div></div>
<div id="content">
<div class="innertube">
<h1>12 Some thoughts about testing</h1>
<h3><a name="id2271413">12.1
Goals</a></h3>
<p>It's not possible to prove that a program is correct by
testing. On the contrary, it has been formally proven that it is
impossible to prove programs in general by testing. Theoretical
program proofs or plain examination of code may be viable options
for those that wish to certify that a program is correct. The test
server, as it is based on testing, cannot be used for
certification. Its intended use is instead to (cost effectively)
<strong>find bugs</strong>. A successful test suite is one that reveals a
bug. If a test suite results in Ok, then we know very little that
we didn't know before.</p>
<h3><a name="id2271438">12.2
What to test?</a></h3>
<p>
There are many kinds of test suites. Some concentrate on
calling every function or command (in the documented way) in
a certain interface.
Some other do the same, but uses all kinds of illegal
parameters, and verifies that the server stays alive and rejects
the requests with reasonable error codes. Some test suites
simulate an application (typically consisting of a few modules of
an application), some try to do tricky requests in general, some
test suites even test internal functions with help of special
load-modules on target.</p>
<p>Another interesting category of test suites are the ones that
check that fixed bugs don't reoccur. When a bugfix is introduced,
a test case that checks for that specific bug should be written
and submitted to the affected test suite(s).</p>
<p>Aim for finding bugs. Write whatever test that has the highest
probability of finding a bug, now or in the future. Concentrate
more on the critical parts. Bugs in critical subsystems are a lot
more expensive than others.</p>
<p>Aim for functionality testing rather than implementation
details. Implementation details change quite often, and the test
suites should be long lived. Often implementation details differ
on different platforms and versions. If implementation details
have to be tested, try to factor them out into separate test
cases. Later on these test cases may be rewritten, or just
skipped.</p>
<p>Also, aim for testing everything once, no less, no more. It's
not effective having every test case fail just because one
function in the interface changed.</p>
</div>
<div class="footer">
<hr>
<p>Copyright © 2003-2010 Ericsson AB. All Rights Reserved.</p>
</div>
</div>
</div></body>
</html>
distrib
>
Fedora
>
13
>
i386
>
media
>
os
>
by-pkgid
>
f806c0f24240b25bde21a53f71766070
>
files
>
253
