.\" @(#)tpkg.1 0.1 12-Oct-2005 OF; .TH tpkg 1 "12 Oct 2005" .SH NAME tpkg \- test package - manipulate tests .SH SYNOPSIS .B tpkg [ .IR OPTIONS ] [ .IR c|e|t|f ] [ .IR exe|create|extract|tmpl|fake ] test.tpkg .B tpkg [ .IR OPTIONS ] [ .IR r|l|d|h|n ] [ .IR report|clean|cleandir|list|desc|help ] test.tpkg .B tpkg [ .IR OPTIONS ] .IR clone test1.tpkg test2.tpkg .SH DESCRIPTION \fBtpkg\fR is used to manipulate test packages. A test package is nothing more than a tar archive which holds a number of scripts. The general idea is that by using \fBtpkg\fR you can keep your tests organized. .PP All errors are send to stderr, the output of the scripts is stored in a result. file. .PP When no action is given a test is \fIexecuted\fR. .PP When a test is executed it is first unpacked, after unpacking the pre-script is run. If this script executes successfully the actual test is run. The test may read the \fI.tpkg.var\fR file to get a hold of shared variables. During the test run a \fIresult.\fR file is written with the output of the test. After the test the post-script is run. .PP All files created by \fBtpkg\fR are removed when \fBtpkg\fR \fIclean\fR is run. .TP \fBexe\fR -- \fBexe\fR Execute a test. This will create a \fIresult.\fR file which looks like this: ** PASSED ** ** PASSED ** DateRunEnd: 1000023 BaseName: svn-build Description: Check out and build libdns/trunk DateRunStart: 1000021 --------------- Test Output ------------------ The first line indicates if a test was failed or passed (or unknown in some cases). The second line shows when the tests was finished in seconds since epoch. The third line is the basename as given in the description file. The fourth line is the test's description, and the fifth line tells when the test was started in seconds after the epoch. Everything after "----- Test Output ----" is the captured output of the test script. .PP Note that a test is executed it runs in a temporary directory, so anything you reference with a relative path should have an extra '../' in front of it. .TP \fBcreate\fR -- \fBc\fR Create a tpkg file out of the test files. .TP \fBextract\fR -- \fBe\fR Extract a tpkg file. A user can then edit these files and recreate the tpkg by running \fBtpkg create\fR. .TP \fBtmpl\fR -- \fBt\fR Create new test template files. A \fIdsc\fR and \fItest\fR file are created. .TP \fBfake\fR -- \fBf\fR Fake the test run, but do create a \fI.done\fR, usefull when writing tests that depend on (lenghty) other tests. .TP \fBlist\fR -- \fBl\fR Print the files in the test package to standard output. .TP \fBreport\fR -- \fBr\fR Create a report by gathering information from the \fIresult\fR files. Nothing is deleted when you run \fBtpkg report\fR. A typical example of a result. file is this: ldns-splint: 5 s : ** PASSED ** : Run a make lint test on the ldns... svn-build: 15 s : ** PASSED ** : Check out and build libdns/trunk PASSED: 2 (100.00 %) FAILED: 0 (0 %) UNKNOWN: 0 (0 %) It shows the name of the test, the time it took while executing (in seconds), whether it passed, failed or was unknown and the description of the test. .PP Essentially \fBreport\fR takes a bunch a \fIresult.\fR files and creates a nice test report of it. When the -q switch is given only failed tests are shown. .TP \fBclean\fR -- \fBclean\fR Clean up all auxiliary files, that is remove: \fI.done*\fR, \fIresult*\fR and the \fI.tpkg.var\fR file. .TP \fBcleandir\fR -- \fBcd\fR Clean up all testname.\fBdir\fR directories. .TP \fBdesc\fR -- \fBd\fR Print the description of the test to standard output. .TP \fBhelp\fR -- \fBh\fR If a .help file is found its contents are displayed. This file can be used to document some test specific things, like how to call it or describe its arguments (if any). Together with the Description line this should fully document the test. .TP \fBclone\fR -- \fBcl\fR Clone \fItest1.tpkg\fR to \fItest2.tpkg\fR. Care is taken to also rename the member files and to update the \fIdsc\fR file. You should always check the new test. .SH OPTIONS .TP \fB\-h\fR Show a short help message. .TP \fB\-v\fR Show the version of \fItpkg\fR. .TP \fB\-q\fR Quiet, only print errors to stdout. .TP \fB\-l\fR Log the name of the test to syslog when starting the test. .TP \fB\-p\fR \fIPRI\fR Use \fIPRI\fR as priority, for example: daemon.notice. .TP \fB\-k\fR Do not remove the test directory when doing a \fItpkg\fR \fBcreate\fR or \fItpkg\fR \fBexe\fR. .TP \fB\-d\fR \fIDIR\fR Use \fIDIR\fR as a basedirectory. This is also the place where files like \fI.tpkg.var\fR will be kept and where the \fIresult.\fR files are written. .TP \fB-a\fR \fIARGS\fR Pass the unmodified string \fIARGS\fR through to the test being run. This way options and/or arguments can be given to the test. The pre, test and post script will all be given this string as an argument. .TP \fB-n\fR \fINUM\fR When running \fItpkg report\fR, \fItpkg\fR will return with a non-zero exit code if less than NUM of the tests passed. Otherwise it will return with zero. .SH FORMAT OF THE .DSC FILE The most important file in each test is the description file, the one with the .dsc extension. The file is line based, so options spanning multiple lines are not recognized and worse, they probably make tpkg fail in unexpected ways. Also note that currently now consistency checking is done. Non existing files listed under AuxFiles: are not detected for instance. .PP A typical example looks like this: \fBBaseName:\fR test1 \fBVersion:\fR 0.1 \fBDescription:\fR Test negative SOA TTL, added to test BUG #6 \fBCreationDate:\fR 05-10-2005 \fBMaintainer:\fR Miek Gieben \fBCategory:\fR \fBComponent:\fR \fBCmdDepends:\fR \fBDepends:\fR \fBHelp:\fR test1.help \fBPre:\fR test1.pre \fBPost:\fR test1.post \fBTest:\fR test1.test \fBAuxFiles:\fRtest1.a, test2.b \fBPassed:\fR \fBFailure:\fR .PP Details of each line follow: .TP \fBBaseName\fR The basename of the test. This is the name of the .dsc file without the .dsc extension .TP \fBVersion\fR Currently nothing is done with this. It could either evolve in a version for the .dsc file or it could be used to version your tests. .TP \fBDescription\fR A short explanation of what this test is supposed to test. 'tpkg report' uses this. \fBtpkg\fR uses ':' (colon) as a delimiter, so it cannot be used in the description of a test. .TP \fBCreationDate \fR Automatically set by 'tpkg tpml'. The original creation date for this test. .TP \fBMaintainer\fR Who created/maintains this test. .TP \fBCategory\fR Under what category does this test fall. This is not used by tpkg, but is a hint to the users of the tests. This could something like: 'building', 'running', etc. .TP \fBComponent\fR What software component are you testing, this could be the name of the executable. User decides what to put here, but it should be consisted for all tests. .TP \fBCmdDepends\fR Depend on these commands. If the command cannot be found ($PATH is search), the test is aborted. Usefull to check to the environment of a test. .TP \fBDepends\fR On what other test does this test depend. The full package name should be given, with the .tpkg extension. Multiple tests can be specified, with whitespace between. .TP \fBHelp\fR Name of a file that has a few lines of usefull information to the user of the test. The file must have a .help suffix. .TP \fBPre\fR Name of a script that should be executed before the test is run. If the pre-script fails the test fails. .TP \fBPost\fR Name of a script that should be executed after the test has run. If the post-script fails the test fails. .TP \fBTest\fR Name of the main test script. .TP \fBAuxfiles\fR Other files that are needed to run this test, i.e. to compare against. Currently this list is limited to 8 other files. The list should use commas as delimiters: fileA, fileB, FileC, ... .TP \fBPassed\fR A regular expression that is matched against the output of the main test script. If the expression matches the test is a success. .TP \fBFailure\fR A regular expression that is matched against the output of the main test script. If the expression matches the test is a failure. .PP All files used in one test must have the same basename, otherwise 'tpkg create' will not pick them up. .SH FAILED OR PASSED .PP A test is \fIfailed\fR when: .TP o The test script returns with a non-zero value. .TP o The test script returns with zero, and the 'Failure' regexp matches. .PP A test is \fIpassed\fR when: .TP o The test script returns with zero. .TP o The test script returns with zero, and the 'Passed' regexp matches. .PP A test is \fIneither\fR failed nor passed when: .TP o Test test script returns with zero, and neither the 'Passed' nor 'Failure' regular expressions matched. If this happens you should rewrite your test. .SH AUTHOR Written by Miek Gieben, NLnet Labs. .SH REPORTING BUGS Report bugs to .SH BUGS As of version 1.03 the internal consistency of a package is tested whenever a test is executed. .SH COPYRIGHT Copyright (C) 2005, 2006 NLnet Labs. This is free software. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Licensed under the GPL version 2. .SH SEE ALSO \fBREADME\fR for information about how to actually write tests.