.\" @(#)tpkg.1 0.1 12-Oct-2005 OF; 
.TH tpkg 1 "12 Oct 2005"
.SH NAME
tpkg \- test package - manipulate tests
.SH SYNOPSIS
.B tpkg
[
.IR OPTIONS
]
[
.IR c|e|t|f
]
[
.IR exe|create|extract|tmpl|fake
]
test.tpkg

.B tpkg
[
.IR OPTIONS
]
[
.IR r|l|d|h|n
]
[
.IR report|clean|cleandir|list|desc|help
]
test.tpkg

.B tpkg
[
.IR OPTIONS
]
.IR clone
test1.tpkg
test2.tpkg

.SH DESCRIPTION
\fBtpkg\fR is used to manipulate test packages. A test package
is nothing more than a tar archive which holds a number of
scripts. The general idea is that by using \fBtpkg\fR you can
keep your tests organized.
.PP
All errors are send to stderr, the output of the scripts is
stored in a result. file.
.PP
When no action is given a test is \fIexecuted\fR.
.PP
When a test is executed it is first unpacked, after unpacking 
the pre-script is run. If this script executes successfully
the actual test is run. The test may read the \fI.tpkg.var\fR
file to get a hold of shared variables. During the test run
a \fIresult.\fR file is written with the output of the test. 
After the test the post-script is run.
.PP
All files created by \fBtpkg\fR are removed when \fBtpkg\fR \fIclean\fR is run.

.TP
\fBexe\fR -- \fBexe\fR
Execute a test. This will create a \fIresult.\fR file which looks
like this:

        ** PASSED **     ** PASSED **
        DateRunEnd: 1000023
        BaseName: svn-build
        Description: Check out and build libdns/trunk
        DateRunStart: 1000021
        --------------- Test Output ------------------

The first line indicates if a test was failed or passed (or unknown
in some cases). The second line shows when the tests was finished
in seconds since epoch. The third line is the basename as given in the 
description file.
The fourth line is the test's description, and the fifth line tells when
the test was started in seconds after the epoch.
Everything after "----- Test Output ----" is the captured output of the
test script.
.PP
Note that a test is executed it runs in a temporary directory, so anything
you reference with a relative path should have an extra '../' in front of
it.

.TP
\fBcreate\fR -- \fBc\fR
Create a tpkg file out of the test files.
.TP
\fBextract\fR -- \fBe\fR
Extract a tpkg file. A user can then edit these files and
recreate the tpkg by running \fBtpkg create\fR.
.TP
\fBtmpl\fR -- \fBt\fR
Create new test template files. A \fIdsc\fR and \fItest\fR file are created.
.TP
\fBfake\fR -- \fBf\fR
Fake the test run, but do create a \fI.done\fR, usefull when writing 
tests that depend on (lenghty) other tests.

.TP
\fBlist\fR -- \fBl\fR
Print the files in the test package to standard output.
.TP
\fBreport\fR -- \fBr\fR
Create a report by gathering information from the \fIresult\fR files.
Nothing is deleted when you run \fBtpkg report\fR.
A typical example of a result. file is this:
        
        ldns-splint: 5 s : ** PASSED ** : Run a make lint test on the ldns...
        svn-build: 15 s : ** PASSED ** : Check out and build libdns/trunk

        PASSED: 2 (100.00 %)     FAILED: 0 (0 %)    UNKNOWN: 0 (0 %)

It shows the name of the test, the time it took while executing (in
seconds), whether it passed, failed or was unknown and the description of
the test.
.PP
Essentially \fBreport\fR takes a bunch a \fIresult.\fR files and creates a
nice test report of it. When the -q switch is given only failed tests are
shown.

.TP
\fBclean\fR -- \fBclean\fR
Clean up all auxiliary files, that is remove: \fI.done*\fR, \fIresult*\fR and
the \fI.tpkg.var\fR file.

.TP
\fBcleandir\fR -- \fBcd\fR
Clean up all testname.\fBdir\fR directories.

.TP
\fBdesc\fR -- \fBd\fR
Print the description of the test to standard output.

.TP
\fBhelp\fR -- \fBh\fR
If a .help file is found its contents are displayed. This file can be
used to document some test specific things, like how to call it or 
describe its arguments (if any). Together with the Description line this
should fully document the test.

.TP
\fBclone\fR -- \fBcl\fR
Clone \fItest1.tpkg\fR to \fItest2.tpkg\fR. Care is taken to also rename
the member files and to update the \fIdsc\fR file. You should always check
the new test.

.SH OPTIONS
.TP
\fB\-h\fR
Show a short help message.
.TP
\fB\-v\fR
Show the version of \fItpkg\fR.
.TP
\fB\-q\fR
Quiet, only print errors to stdout.
.TP
\fB\-l\fR 
Log the name of the test to syslog when starting the test.
.TP
\fB\-p\fR \fIPRI\fR
Use \fIPRI\fR as priority, for example: daemon.notice.
.TP
\fB\-k\fR
Do not remove the test directory when doing a \fItpkg\fR \fBcreate\fR
or \fItpkg\fR \fBexe\fR.
.TP
\fB\-d\fR \fIDIR\fR
Use \fIDIR\fR as a basedirectory. This is also the place where files
like \fI.tpkg.var\fR will be kept and where the \fIresult.\fR files are written.
.TP
\fB-a\fR \fIARGS\fR
Pass the unmodified string \fIARGS\fR through to the test being run.
This way options and/or arguments can be given to the test. The pre,
test and post script will all be given this string as an argument.
.TP
\fB-n\fR \fINUM\fR
When running \fItpkg report\fR, \fItpkg\fR will return with
a non-zero exit code if less than NUM of the tests passed.
Otherwise it will return with zero.

.SH FORMAT OF THE .DSC FILE
The most important file in each test is the description
file, the one with the .dsc extension. The file is line based,
so options spanning multiple lines are not recognized and worse,
they probably make tpkg fail in unexpected ways. Also note that
currently now consistency checking is done. Non existing files listed under
AuxFiles: are not detected for instance.

.PP
A typical example looks like this:
        
        \fBBaseName:\fR test1
        \fBVersion:\fR 0.1
        \fBDescription:\fR  Test negative SOA TTL, added to test BUG #6
        \fBCreationDate:\fR 05-10-2005
        \fBMaintainer:\fR Miek Gieben
        \fBCategory:\fR 
        \fBComponent:\fR
        \fBCmdDepends:\fR
        \fBDepends:\fR
        \fBHelp:\fR test1.help
        \fBPre:\fR test1.pre
        \fBPost:\fR test1.post
        \fBTest:\fR test1.test
        \fBAuxFiles:\fRtest1.a, test2.b
        \fBPassed:\fR
        \fBFailure:\fR 

.PP
Details of each line follow:
.TP 
\fBBaseName\fR
The basename of the test. This is the name of the .dsc
file without the .dsc extension
.TP 
\fBVersion\fR
Currently nothing is done with this. It could either
evolve in a version for the .dsc file or it could
be used to version your tests.
.TP 
\fBDescription\fR
A short explanation of what this test is supposed to test.
 'tpkg report' uses this. \fBtpkg\fR uses ':' (colon) as a
delimiter, so it cannot be used in the description of a test.
.TP 
\fBCreationDate \fR
Automatically set by 'tpkg tpml'. The original creation date
for this test.
.TP 
\fBMaintainer\fR
Who created/maintains this test.
.TP 
\fBCategory\fR
Under what category does this test fall. This is not
used by tpkg, but is a hint to the users of the tests.
This could something like: 'building', 'running', etc.
.TP 
\fBComponent\fR
What software component are you testing, this could be
the name of the executable. User decides what to put
here, but it should be consisted for all tests.
.TP 
\fBCmdDepends\fR
Depend on these commands. If the command cannot be found ($PATH
is search), the test is aborted. Usefull to check to the environment of a
test.
.TP 
\fBDepends\fR
On what other test does this test depend. The full
package name should be given, with the .tpkg extension.
Multiple tests can be specified, with whitespace between.
.TP 
\fBHelp\fR
Name of a file that has a few lines of usefull information
to the user of the test. The file must have a .help suffix.
.TP 
\fBPre\fR
Name of a script that should be executed before the
test is run. If the pre-script fails the test fails.
.TP 
\fBPost\fR
Name of a script that should be executed after the 
test has run. If the post-script fails the test fails.
.TP 
\fBTest\fR
Name of the main test script.
.TP 
\fBAuxfiles\fR
Other files that are needed to run this test, i.e. to 
compare against. Currently this list is limited to 8
other files. The list should use commas as delimiters:
fileA, fileB, FileC, ...
.TP 
\fBPassed\fR
A regular expression that is matched against the output
of the main test script. If the expression matches the
test is a success.
.TP 
\fBFailure\fR
A regular expression that is matched against the output
of the main test script. If the expression matches the
test is a failure.

.PP
All files used in one test must have the same basename, otherwise 'tpkg create' 
will not pick them up.

.SH FAILED OR PASSED
.PP
A test is \fIfailed\fR when:
.TP 
o  
The test script returns with a non-zero value.
.TP
o
The test script returns with zero, and the 'Failure' regexp matches.
                
.PP
A test is \fIpassed\fR when:
.TP
o
The test script returns with zero.
.TP
o
The test script returns with zero, and the 'Passed' regexp matches.
                
.PP                
A test is \fIneither\fR failed nor passed when:
.TP
o
Test test script returns with zero, and neither the 'Passed' nor 'Failure'
regular expressions matched.
If this happens you should rewrite your test.

.SH AUTHOR
Written by Miek Gieben, NLnet Labs.

.SH REPORTING BUGS
Report bugs to <miek@nlnetlabs.nl>

.SH BUGS
As of version 1.03 the internal consistency of a package is tested whenever
a test is executed.

.SH COPYRIGHT
Copyright (C) 2005, 2006 NLnet Labs. This is free software. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Licensed under the GPL version 2.

.SH SEE ALSO
\fBREADME\fR for information about how to actually write tests.
