-*- emacs-wiki -*-
Structure of a test
Each test consists of a set of directories under testing/. There are
directories for klips, pluto, packaging and libraries. Each directory
has a list of tests to run is stored in a file called TESTLIST in that
directory. e.g. testing/klips/TESTLIST.
The TESTLIST
This isn't actually a shell script. It just looks like one.
(Tragically Hip: "She's not pretty, she just looks that way.")
Some tools other than /bin/sh process it. Lines that start with # are
comments.
# test-kind directory-containing-test expectation [PR#]
The first word provides the test type, detailed below.
The second word is the name of the test to run. This the directory in
which the test case is to be found..
The third word may be one of:
blank/good
the test is believed to function, report failure
bad
the test is known to fail, report unexpected success
suspended
the test should not be run
The fourth word may be a number, which is a PR# if the test is
failing.
Test kinds
The test types are:
skiptest
means run no test.
ctltest
means run a single system without input/output.
klipstest
means run a single system with input/output networks
umlplutotest
means run a pair of systems
umlXhost
run an arbitrary number of systems
mkinsttest
a test of the "make install" machinery.
kernel_test_patch
a test of the "make kernelpatch" machinery.
Each test directory has a file in it called testparams.sh. This file
sets a number of environment variables to define the parameters of the
test.
Common parameters
TESTNAME
the name of the test (repeated for checking purposes)
TEST_TYPE
the type of the test (repeat of type type above)
TESTHOST
the name of the UML machine to run for the test, typically
"east" or "west"
TEST_PURPOSE
The purpose of the test is one of:
goal
The goal purpose is where a test is defined for code that
is not yet finished. The test indicates when the goals
have in fact been reached.
regress
This is a test to determine that a previously existing
bug has been repaired. This test will initially be
created to reproduce the bug in isolation, and then the
bug will be fixed.
exploit
This is a set of packets/programs that causes a
vulnerability to be exposed. It is a specific variation
of the regress option.
TEST_GOAL_ITEM
in the case of a goal test, this is a reference to the
requirements document
TEST_PROB_REPORT
in the case of regression test, this the problem report number
from GNATS
TEST_EXPLOIT_URL
in the case of an exploit, this is a URL referencing the paper
explaining the origin of the test and the origin of exploit
software
REF_CONSOLE_OUTPUT
a file in the test directory that contains the sanitized
console output against which to compare the output of the
actual test.
REF_CONSOLE_FIXUPS
a list of scripts (found in klips/test/fixups) to apply to
sanitize the console output of the machine under test. These
are typically perl, awk or sed scripts that remove things in
the kernel output that change each time the test is run and/or
compiled.
INIT_SCRIPT
a file of commands that is fed into the virtual machine's
console in single user mode prior to starting the tests. This
file will usually set up any eroute's and SADB entries that are
required for the test.
Lines beginning with # are skipped. Blank lines are skipped.
Otherwise, a shell prompted is waited for each time (consisting
of \n#) and then the command is sent. Note that the prompt is
waited for before the command and not after, so completion of
the last command in the script is not required. This is often
used to invoke a program to monitor the system, e.g. ipsec
pf_key.
RUN_SCRIPT
a file of commands that is fed into the virtual machine's
console in single user mode, before the packets are sent. On
single machine tests, this script doesn't provide any more
power than INIT_SCRIPT, but is implemented for consistency's
sake.
FINAL_SCRIPT
a file of commands that is fed into the virtual machine's
console in single user mode after the final packet is sent.
Similar to INIT_SCRIPT, above. If not specified, then the
single command "halt" is sent. If specified, then the script
should end with a halt command to nicely shutdown the UML.
CONSOLEDIFFDEBUG
If set to "true" then the series of console fixups (see
REF_CONSOLE_FIXUPS) will be output after it is constructed. (It
should be set to "false", or unset otherwise)
NETJIGDEBUG
If set to "true" then the series of console fixups (see
REF_CONSOLE_FIXUPS) will be output after it is constructed. (It
should be set to "false", or unset otherwise)
NETJIGTESTDEBUG
If set to "netjig", then the results of talking to the
uml_netjig will be printed to stderr during the test. In
addition, the jig will be invoked with --debug, which causes it
to log its process ID, and wait 60 seconds before continuing.
This can be used if you are trying to debug the uml_netjig
program itself.
HOSTTESTDEBUG
If set to "hosttest", then the results of taling to the
consoles of the UMLs will be printed to stderr during the test.
NETJIGWAITUSER
If set to "waituser", then the scripts will wait forever for
user input before they shut the tests down. Use this is if you
are debugging through the kernel.
PACKETRATE
A number, in miliseconds (default is 500ms) at which packets
will be replayed by the netjig.
- [[KLIPSTestParameters][KLIPStest parameters]]
- [[MKInstTestParameters][mkinsttest parameters]]
- [[RPMBuildInstallParameters][rpm_build_install_test parameters]]
- [[LibTestParameters][libtest parameters]]
- [[UMLPlutoTestParameters][umlplutotest parameters]]
- [[UMLXHostParameters][umlXhost parameters]]
- [[KernelPatchTestParameters][kernel_patch_test paramaters]]
- [[ModuleCompileParameters][module_compile paramaters]]
- [[UnitTestParameters][Unit Test parameters]]
