TEST PACKAGE -- TPKG

(c) NLnet Labs, 2005, 2006. Miek Gieben
Licensed under the GPL version 2.

Test Package

        The most important file in each test is the description
        file, the one with the .dsc extension. The file is line based,
        so options spanning multiple lines are not recognized or worse,
        make tpkg fail.

        A typical example looks like this:
        
                BaseName: test1
                Version: 0.1
                Description: Test negative SOA TTL, added to test BUG #6
                CreationDate: 05-10-2005
                Maintainer: Miek Gieben
                Category: 
                Component:
                Depends: 
                Help: test1.help
                Pre: test1.pre
                Post: test1.post
                Test: test1.test
                AuxFiles: test1.a, test2.b
                Passed:
                Failure: 

        Details of each line follow:

        BaseName:
                The basename of the test. This is the name of the .dsc
                file without the .dsc extension

        Version:
                Currently nothing is done with this. It could either
                evolve in a version for the .dsc file or it could
                be used to version your tests.

        Description:
                A short explanation of what this test is supposed to test.
                'tpkg report' uses this.

        CreationDate: 
                Automaticly set by 'tpkg tpml'. The original creation date
                for this test.

        Maintainer:
                Who created/maintains this test

        Category:
                Under what category does this test fall. This is not
                used by tpkg, but is a hint to the users of the tests.
                This could something like: 'building', 'running', etc.
                
        Component:
                What software component are you testing, this could be
                the name of the executeable. User decides what to put
                here, but it should be consistend for all tests.

        Depends:
                On what other test does this test depend. The full
                package name should be given, with the .tpkg extension.

                Currently this is limited to one test. 
        
        Help:
               Name of a file that has a few lines of usefull information
               to the user of the test. The file must have a .help suffix.

        Pre:
                Name of a script that should be executed before the
                test is run. If the pre-script fails the test fails.

        Post:
                Name of a script that should be executed after the 
                test has run. If the post-script fails the test fails.

        Test:
                Name fo the main test script.

        Auxfiles:
                Other files that are needed to run this test, i.e. to 
                compare against. Currently this list is limited to 8
                other files. The list should use commas as delimiters:
                fileA, fileB, FileC, ...

        Passed:
                A regular expression that is matched against the output
                of the main test script. If the expression matches the
                test is a succes.

        Failure:
                A regular expression that is matched against the output
                of the main test script. If the expression matches the
                test is a failure.

        All files used in 1 test must have the same basename, otherwise
        'tpkg create' will not pick them up.


When is a Test Successfull and When Not

        A test is failed when:
           *   The test script returns with a non-zero value.

           *   The test script returns with zero, and
               the 'Failure' regexp matches.
                

        A test is passed when:
           *   The test script returns with zero.

           *   The test script returns with zero, and
               the 'Passed' regexp matches.
                
        A test is neither failed nor passed when:
           *   Test test script returns with zero, and
               neither the 'Passed' nor 'Failure' regular
               expressions matched.

        If this happens you should rewrite your test.

                
Writting Test Packages

        .test file
                Start each test with a source of the .tpkg.var.master, like
                this:

                [ -f ../.tpkg.var.master ] && source ../.tpkg.var.master

                This is automaticly added when =tpkg tmpl TEST=
                is called.

		We use '../' here cause '.' is cleaned up after
		the test is run

        It might also be prudent to source the .tpkg.var.test file for test
        specific enviroment variables.

        Each test is executed in a temporary directory, keep this
        in mind when using relative pathnames. ../exe must become
        ../../exe will the test be able to find it.

        Test your test before creating a .tpkg file, this can 
        be as simple as: bash ./testX.test


Test Output
        
        result. files

                A typical example of a result. file is this:
                
                ** PASSED **     ** PASSED **
                BaseName: svn-build
                Description: Check out and build libdns/trunk
                DateRun: Fri Oct  7 20:28:23 CEST 2005
                --------------- Test Output ------------------

        A summery of result files of test can be seen by running
        tpkg report.


Running Tests:
        Each test is executed in the new temporary directory.
             
        tpkg exe test1
        tpkg exe test2
        ...
        tpkg exe testN

        tpkg report
                generate a report of all the tests

        tpkg clean
                remove all the result. files

Reading Reports:
        Typical example:

        ** PASSED ** : 4 s : ldns-splint : Run a make lint test on the libdns sources
        ** PASSED ** : 2 s : svn-build: : Check out and build libdns/trunk

        PASSED: 2 (100.00 %)            FAILED: 0 (0 %)         UNKNOWN: 0 (0 %)

Sample Output For NSD
        I've written a few tests for unit testing NSD, running these tests
        looks like this:

        % tpkg exe rfc3597-good.tpkg && tpkg exe rfc3597-bad.tpkg
        [rfc3597-good]  [log] Extracting...
        [rfc3597-good]  [log] executing dependency test: nsd-compile.tpkg
        [nsd-compile]  [log] Extracting...
        [nsd-compile]  [log] Executing test
        [nsd-compile]  [log] ** PASSED **
        [nsd-compile]  [log] Cleaning up
        [rfc3597-good]  [log] Executing test
        [rfc3597-good]  [log] ** PASSED **
        [rfc3597-good]  [log] Cleaning up
        [rfc3597-bad]  [log] Extracting...
        [rfc3597-bad]  [log] executing dependency test: nsd-compile.tpkg
        [nsd-compile]  [log] Extracting...
        [nsd-compile]  [log] Found .done-nsd-compile. Not executing this test.
        [nsd-compile]  [log] Cleaning up
        [rfc3597-bad]  [log] Executing test
        [rfc3597-bad]  [log] ** PASSED **
        [rfc3597-bad]  [log] Cleaning up

        % tpkg report
        nsd-compile: 1 s : ** PASSED ** : Compile NSD
        rfc3597-bad: 3 s : ** PASSED ** : Compile a zone with unknown rr data (the zone is BAD)
        rfc3597-good: 2 s : ** PASSED ** : Compile a zone with unknown rr data (the zone is good)

        PASSED: 3 (100.00 %)            FAILED: 0 (0 %)         UNKNOWN: 0 (0 %)
        
        % tpkg clean
          [log] rm result.nsd-compile
          [log] rm result.rfc3597-bad
          [log] rm result.rfc3597-good
          [log] rm .tpkg.var
          [log] rm .done files
