332 lines
9.3 KiB
Groff
332 lines
9.3 KiB
Groff
|
.\" @(#)tpkg.1 0.1 12-Oct-2005 OF;
|
||
|
|
.TH tpkg 1 "12 Oct 2005"
|
||
|
|
.SH NAME
|
||
|
|
tpkg \- test package - manipulate tests
|
||
|
|
.SH SYNOPSIS
|
||
|
|
.B tpkg
|
||
|
|
[
|
||
|
|
.IR OPTIONS
|
||
|
|
]
|
||
|
|
[
|
||
|
|
.IR c|e|t|f
|
||
|
|
]
|
||
|
|
[
|
||
|
|
.IR exe|create|extract|tmpl|fake
|
||
|
|
]
|
||
|
|
test.tpkg
|
||
|
|
|
||
|
|
.B tpkg
|
||
|
|
[
|
||
|
|
.IR OPTIONS
|
||
|
|
]
|
||
|
|
[
|
||
|
|
.IR r|l|d|h|n
|
||
|
|
]
|
||
|
|
[
|
||
|
|
.IR report|clean|cleandir|list|desc|help
|
||
|
|
]
|
||
|
|
test.tpkg
|
||
|
|
|
||
|
|
.B tpkg
|
||
|
|
[
|
||
|
|
.IR OPTIONS
|
||
|
|
]
|
||
|
|
.IR clone
|
||
|
|
test1.tpkg
|
||
|
|
test2.tpkg
|
||
|
|
|
||
|
|
.SH DESCRIPTION
|
||
|
|
\fBtpkg\fR is used to manipulate test packages. A test package
|
||
|
|
is nothing more than a tar archive which holds a number of
|
||
|
|
scripts. The general idea is that by using \fBtpkg\fR you can
|
||
|
|
keep your tests organized.
|
||
|
|
.PP
|
||
|
|
All errors are send to stderr, the output of the scripts is
|
||
|
|
stored in a result. file.
|
||
|
|
.PP
|
||
|
|
When no action is given a test is \fIexecuted\fR.
|
||
|
|
.PP
|
||
|
|
When a test is executed it is first unpacked, after unpacking
|
||
|
|
the pre-script is run. If this script executes successfully
|
||
|
|
the actual test is run. The test may read the \fI.tpkg.var\fR
|
||
|
|
file to get a hold of shared variables. During the test run
|
||
|
|
a \fIresult.\fR file is written with the output of the test.
|
||
|
|
After the test the post-script is run.
|
||
|
|
.PP
|
||
|
|
All files created by \fBtpkg\fR are removed when \fBtpkg\fR \fIclean\fR is run.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBexe\fR -- \fBexe\fR
|
||
|
|
Execute a test. This will create a \fIresult.\fR file which looks
|
||
|
|
like this:
|
||
|
|
|
||
|
|
** PASSED ** ** PASSED **
|
||
|
|
DateRunEnd: 1000023
|
||
|
|
BaseName: svn-build
|
||
|
|
Description: Check out and build libdns/trunk
|
||
|
|
DateRunStart: 1000021
|
||
|
|
--------------- Test Output ------------------
|
||
|
|
|
||
|
|
The first line indicates if a test was failed or passed (or unknown
|
||
|
|
in some cases). The second line shows when the tests was finished
|
||
|
|
in seconds since epoch. The third line is the basename as given in the
|
||
|
|
description file.
|
||
|
|
The fourth line is the test's description, and the fifth line tells when
|
||
|
|
the test was started in seconds after the epoch.
|
||
|
|
Everything after "----- Test Output ----" is the captured output of the
|
||
|
|
test script.
|
||
|
|
.PP
|
||
|
|
Note that a test is executed it runs in a temporary directory, so anything
|
||
|
|
you reference with a relative path should have an extra '../' in front of
|
||
|
|
it.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBcreate\fR -- \fBc\fR
|
||
|
|
Create a tpkg file out of the test files.
|
||
|
|
.TP
|
||
|
|
\fBextract\fR -- \fBe\fR
|
||
|
|
Extract a tpkg file. A user can then edit these files and
|
||
|
|
recreate the tpkg by running \fBtpkg create\fR.
|
||
|
|
.TP
|
||
|
|
\fBtmpl\fR -- \fBt\fR
|
||
|
|
Create new test template files. A \fIdsc\fR and \fItest\fR file are created.
|
||
|
|
.TP
|
||
|
|
\fBfake\fR -- \fBf\fR
|
||
|
|
Fake the test run, but do create a \fI.done\fR, usefull when writing
|
||
|
|
tests that depend on (lenghty) other tests.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBlist\fR -- \fBl\fR
|
||
|
|
Print the files in the test package to standard output.
|
||
|
|
.TP
|
||
|
|
\fBreport\fR -- \fBr\fR
|
||
|
|
Create a report by gathering information from the \fIresult\fR files.
|
||
|
|
Nothing is deleted when you run \fBtpkg report\fR.
|
||
|
|
A typical example of a result. file is this:
|
||
|
|
|
||
|
|
ldns-splint: 5 s : ** PASSED ** : Run a make lint test on the ldns...
|
||
|
|
svn-build: 15 s : ** PASSED ** : Check out and build libdns/trunk
|
||
|
|
|
||
|
|
PASSED: 2 (100.00 %) FAILED: 0 (0 %) UNKNOWN: 0 (0 %)
|
||
|
|
|
||
|
|
It shows the name of the test, the time it took while executing (in
|
||
|
|
seconds), whether it passed, failed or was unknown and the description of
|
||
|
|
the test.
|
||
|
|
.PP
|
||
|
|
Essentially \fBreport\fR takes a bunch a \fIresult.\fR files and creates a
|
||
|
|
nice test report of it. When the -q switch is given only failed tests are
|
||
|
|
shown.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBclean\fR -- \fBclean\fR
|
||
|
|
Clean up all auxiliary files, that is remove: \fI.done*\fR, \fIresult*\fR and
|
||
|
|
the \fI.tpkg.var\fR file.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBcleandir\fR -- \fBcd\fR
|
||
|
|
Clean up all testname.\fBdir\fR directories.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBdesc\fR -- \fBd\fR
|
||
|
|
Print the description of the test to standard output.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBhelp\fR -- \fBh\fR
|
||
|
|
If a .help file is found its contents are displayed. This file can be
|
||
|
|
used to document some test specific things, like how to call it or
|
||
|
|
describe its arguments (if any). Together with the Description line this
|
||
|
|
should fully document the test.
|
||
|
|
|
||
|
|
.TP
|
||
|
|
\fBclone\fR -- \fBcl\fR
|
||
|
|
Clone \fItest1.tpkg\fR to \fItest2.tpkg\fR. Care is taken to also rename
|
||
|
|
the member files and to update the \fIdsc\fR file. You should always check
|
||
|
|
the new test.
|
||
|
|
|
||
|
|
.SH OPTIONS
|
||
|
|
.TP
|
||
|
|
\fB\-h\fR
|
||
|
|
Show a short help message.
|
||
|
|
.TP
|
||
|
|
\fB\-v\fR
|
||
|
|
Show the version of \fItpkg\fR.
|
||
|
|
.TP
|
||
|
|
\fB\-q\fR
|
||
|
|
Quiet, only print errors to stdout.
|
||
|
|
.TP
|
||
|
|
\fB\-l\fR
|
||
|
|
Log the name of the test to syslog when starting the test.
|
||
|
|
.TP
|
||
|
|
\fB\-p\fR \fIPRI\fR
|
||
|
|
Use \fIPRI\fR as priority, for example: daemon.notice.
|
||
|
|
.TP
|
||
|
|
\fB\-k\fR
|
||
|
|
Do not remove the test directory when doing a \fItpkg\fR \fBcreate\fR
|
||
|
|
or \fItpkg\fR \fBexe\fR.
|
||
|
|
.TP
|
||
|
|
\fB\-d\fR \fIDIR\fR
|
||
|
|
Use \fIDIR\fR as a basedirectory. This is also the place where files
|
||
|
|
like \fI.tpkg.var\fR will be kept and where the \fIresult.\fR files are written.
|
||
|
|
.TP
|
||
|
|
\fB-a\fR \fIARGS\fR
|
||
|
|
Pass the unmodified string \fIARGS\fR through to the test being run.
|
||
|
|
This way options and/or arguments can be given to the test. The pre,
|
||
|
|
test and post script will all be given this string as an argument.
|
||
|
|
.TP
|
||
|
|
\fB-n\fR \fINUM\fR
|
||
|
|
When running \fItpkg report\fR, \fItpkg\fR will return with
|
||
|
|
a non-zero exit code if less than NUM of the tests passed.
|
||
|
|
Otherwise it will return with zero.
|
||
|
|
|
||
|
|
.SH FORMAT OF THE .DSC FILE
|
||
|
|
The most important file in each test is the description
|
||
|
|
file, the one with the .dsc extension. The file is line based,
|
||
|
|
so options spanning multiple lines are not recognized and worse,
|
||
|
|
they probably make tpkg fail in unexpected ways. Also note that
|
||
|
|
currently now consistency checking is done. Non existing files listed under
|
||
|
|
AuxFiles: are not detected for instance.
|
||
|
|
|
||
|
|
.PP
|
||
|
|
A typical example looks like this:
|
||
|
|
|
||
|
|
\fBBaseName:\fR test1
|
||
|
|
\fBVersion:\fR 0.1
|
||
|
|
\fBDescription:\fR Test negative SOA TTL, added to test BUG #6
|
||
|
|
\fBCreationDate:\fR 05-10-2005
|
||
|
|
\fBMaintainer:\fR Miek Gieben
|
||
|
|
\fBCategory:\fR
|
||
|
|
\fBComponent:\fR
|
||
|
|
\fBCmdDepends:\fR
|
||
|
|
\fBDepends:\fR
|
||
|
|
\fBHelp:\fR test1.help
|
||
|
|
\fBPre:\fR test1.pre
|
||
|
|
\fBPost:\fR test1.post
|
||
|
|
\fBTest:\fR test1.test
|
||
|
|
\fBAuxFiles:\fRtest1.a, test2.b
|
||
|
|
\fBPassed:\fR
|
||
|
|
\fBFailure:\fR
|
||
|
|
|
||
|
|
.PP
|
||
|
|
Details of each line follow:
|
||
|
|
.TP
|
||
|
|
\fBBaseName\fR
|
||
|
|
The basename of the test. This is the name of the .dsc
|
||
|
|
file without the .dsc extension
|
||
|
|
.TP
|
||
|
|
\fBVersion\fR
|
||
|
|
Currently nothing is done with this. It could either
|
||
|
|
evolve in a version for the .dsc file or it could
|
||
|
|
be used to version your tests.
|
||
|
|
.TP
|
||
|
|
\fBDescription\fR
|
||
|
|
A short explanation of what this test is supposed to test.
|
||
|
|
'tpkg report' uses this. \fBtpkg\fR uses ':' (colon) as a
|
||
|
|
delimiter, so it cannot be used in the description of a test.
|
||
|
|
.TP
|
||
|
|
\fBCreationDate \fR
|
||
|
|
Automatically set by 'tpkg tpml'. The original creation date
|
||
|
|
for this test.
|
||
|
|
.TP
|
||
|
|
\fBMaintainer\fR
|
||
|
|
Who created/maintains this test.
|
||
|
|
.TP
|
||
|
|
\fBCategory\fR
|
||
|
|
Under what category does this test fall. This is not
|
||
|
|
used by tpkg, but is a hint to the users of the tests.
|
||
|
|
This could something like: 'building', 'running', etc.
|
||
|
|
.TP
|
||
|
|
\fBComponent\fR
|
||
|
|
What software component are you testing, this could be
|
||
|
|
the name of the executable. User decides what to put
|
||
|
|
here, but it should be consisted for all tests.
|
||
|
|
.TP
|
||
|
|
\fBCmdDepends\fR
|
||
|
|
Depend on these commands. If the command cannot be found ($PATH
|
||
|
|
is search), the test is aborted. Usefull to check to the environment of a
|
||
|
|
test.
|
||
|
|
.TP
|
||
|
|
\fBDepends\fR
|
||
|
|
On what other test does this test depend. The full
|
||
|
|
package name should be given, with the .tpkg extension.
|
||
|
|
Multiple tests can be specified, with whitespace between.
|
||
|
|
.TP
|
||
|
|
\fBHelp\fR
|
||
|
|
Name of a file that has a few lines of usefull information
|
||
|
|
to the user of the test. The file must have a .help suffix.
|
||
|
|
.TP
|
||
|
|
\fBPre\fR
|
||
|
|
Name of a script that should be executed before the
|
||
|
|
test is run. If the pre-script fails the test fails.
|
||
|
|
.TP
|
||
|
|
\fBPost\fR
|
||
|
|
Name of a script that should be executed after the
|
||
|
|
test has run. If the post-script fails the test fails.
|
||
|
|
.TP
|
||
|
|
\fBTest\fR
|
||
|
|
Name of the main test script.
|
||
|
|
.TP
|
||
|
|
\fBAuxfiles\fR
|
||
|
|
Other files that are needed to run this test, i.e. to
|
||
|
|
compare against. Currently this list is limited to 8
|
||
|
|
other files. The list should use commas as delimiters:
|
||
|
|
fileA, fileB, FileC, ...
|
||
|
|
.TP
|
||
|
|
\fBPassed\fR
|
||
|
|
A regular expression that is matched against the output
|
||
|
|
of the main test script. If the expression matches the
|
||
|
|
test is a success.
|
||
|
|
.TP
|
||
|
|
\fBFailure\fR
|
||
|
|
A regular expression that is matched against the output
|
||
|
|
of the main test script. If the expression matches the
|
||
|
|
test is a failure.
|
||
|
|
|
||
|
|
.PP
|
||
|
|
All files used in one test must have the same basename, otherwise 'tpkg create'
|
||
|
|
will not pick them up.
|
||
|
|
|
||
|
|
.SH FAILED OR PASSED
|
||
|
|
.PP
|
||
|
|
A test is \fIfailed\fR when:
|
||
|
|
.TP
|
||
|
|
o
|
||
|
|
The test script returns with a non-zero value.
|
||
|
|
.TP
|
||
|
|
o
|
||
|
|
The test script returns with zero, and the 'Failure' regexp matches.
|
||
|
|
|
||
|
|
.PP
|
||
|
|
A test is \fIpassed\fR when:
|
||
|
|
.TP
|
||
|
|
o
|
||
|
|
The test script returns with zero.
|
||
|
|
.TP
|
||
|
|
o
|
||
|
|
The test script returns with zero, and the 'Passed' regexp matches.
|
||
|
|
|
||
|
|
.PP
|
||
|
|
A test is \fIneither\fR failed nor passed when:
|
||
|
|
.TP
|
||
|
|
o
|
||
|
|
Test test script returns with zero, and neither the 'Passed' nor 'Failure'
|
||
|
|
regular expressions matched.
|
||
|
|
If this happens you should rewrite your test.
|
||
|
|
|
||
|
|
.SH AUTHOR
|
||
|
|
Written by Miek Gieben, NLnet Labs.
|
||
|
|
|
||
|
|
.SH REPORTING BUGS
|
||
|
|
Report bugs to <miek@nlnetlabs.nl>
|
||
|
|
|
||
|
|
.SH BUGS
|
||
|
|
As of version 1.03 the internal consistency of a package is tested whenever
|
||
|
|
a test is executed.
|
||
|
|
|
||
|
|
.SH COPYRIGHT
|
||
|
|
Copyright (C) 2005, 2006 NLnet Labs. This is free software. There is NO
|
||
|
|
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
|
||
|
|
Licensed under the GPL version 2.
|
||
|
|
|
||
|
|
.SH SEE ALSO
|
||
|
|
\fBREADME\fR for information about how to actually write tests.
|