System testing ============== Before a new version of Zonemaster can be released it must pass three levels of system testing: Installation Testing, Smoke Testing and Acceptance Testing. Each test level is described in a separate section below. A separate section is reserved for the concept of Configurations, which is used by other sections. ## 1. Configurations The purpose of configurations is to provide different contexts for the tests to run in. A configuration is a tuple of the following parameters: * architecture (e.g. x86_64) * operating system version (e.g. Ubuntu 14.04) * database engine version (e.g. PostgreSQL 9.3) * system locale (e.g. en_US.UTF-8) * Perl version (e.g. 5.20) The set of possible values for the following configuration parameters (except for system locale) are declared in the README.md of the top-level zonemaster repository. The set of possible values for the system locale configuration parameter is: * C * en_US.UTF-8 Regarding minimum hardware requirements, the following defaults should be chosen: * CPU: 2 cores * RAM: 2GB * Storage: 20GB ## 2. Installation testing This test level validates that all components of Zonemaster can be installed according to their respective installation instructions. ### Configurations The set of configurations must include at least: * One configuration for each supported architecture. * One configuration for each supported operating system version. * One configuration for each supported database engine version. ### Procedure 1. Make sure you have the correct preliminary distribution tarball for each component. See [Create Test Distribution]. 2. For each configuration: 1. Prepare a machine 1. Allocate a machine with the architecture specified by the configuration. 2. Follow the preparation document for the operating system specified by the configuration. * [Debian-Preparation] * [FreeBSD-Preparation] * [Ubuntu-Preparation] * Rocky-Linux-Preparation (TBD) 2. Install Zonemaster LDNS 1. Install dependencies according to the [LDNS installation] instructions. 2. Install the preliminary distribution tarball for zonemaster-ldns. ```sh sudo cpanm Zonemaster-LDNS-${LDNS_VERSION}.tar.gz ``` 3. Follow the [post-installation sanity check][LDNS sanity check] section of the installation guide to the letter. 3. Install Zonemaster Engine 1. Install dependencies according to the [Engine installation] instructions. 2. Install the preliminary distribution tarball for zonemaster-engine. ```sh sudo cpanm Zonemaster-Engine-${ENGINE_VERSION}.tar.gz ``` 3. Follow the [post-installation sanity check][Engine sanity check] section of the installation guide to the letter. 4. Install Zonemaster Backend 1. Install dependencies according to the [Backend installation] instructions. 2. Install the preliminary distribution tarball for zonemaster-backend. ```sh sudo cpanm Zonemaster-Backend-${BACKEND_VERSION}.tar.gz ``` 3. Follow the configuration section of the [Backend installation] instructions to the letter. 4. Follow the startup section of the [Backend installation] instructions to the letter. 5. Follow the [smoke test] section of the installation guide to the letter. 5. Install Zonemaster GUI 1. Follow the prerequisites section of the [GUI installation] instructions to the letter. 2. Follow the installation section of the [GUI installation] instructions to the letter. 3. Follow the configuration section of the [GUI installation] instructions to the letter. 4. Follow the startup section of the [GUI installation] instructions to the letter. 5. Follow the [post-installation sanity check][GUI sanity check] section of the installation guide to the letter. 6. Install Zonemaster CLI 1. Follow the prerequisites section of the [CLI installation] installation to the letter. 2. Install the preliminary distribution tarball for zonemaster-backend. ```sh sudo cpanm Zonemaster-CLI-${CLI_VERSION}.tar.gz ``` 3. Follow the [post-installation sanity check][CLI sanity check] section of the installation guide to the letter. ## 3. Smoke testing This test level validates that fundamental use cases: * work independent of specific configurations * weren't broken by new changes ### Configurations ... To be specified. ### Procedure For each configuration: 1. Prepare an installation 2. Smoke test of Zonemaster CLI 1. Test a working domain (e.g. `afnic.fr`) 2. Test working IDN (e.g. `президент.рф`) 3. Test a broken domain (e.g. `error.com`) 4. Working domains not signed with DNSSEC (e.g. `google.com`) 5. Working domains signed with DNSSEC (e.g. `iis.se`) 6. Broken DNSSEC domains (e.g. `broken.dnssec.ee`) 3. Smoke test of Zonemaster GUI 1. Test progress bar 2. Test presentation of results 3. Test export 4. Test history ## 4. Acceptance testing This test level validates that each change since last release: * works independent of specific configurations * weren't broken by other new changes ### Configurations ... To be specified. ### Procedure 1. Prepare an installation 2. Acceptance testing through Zonemaster CLI Test each new entry in the Changes files of *Zonemaster CLI* and *Zonemaster Engine*. 4. Acceptance testing through Zonemaster Backend Test each new entry in the Changes file of *Zonemaster Backend*. 4. Acceptance testing through Zonemaster GUI Test each new entry in the Changes file of *Zonemaster GUI*. [Backend installation]: ../../public/installation/zonemaster-backend.md [CLI installation]: ../../public/installation/zonemaster-cli.md [CLI sanity check]: ../../public/installation/zonemaster-cli.md#post-installation-sanity-check [Create Test Distribution]: ../maintenance/ReleaseProcess-create-test-distribution.md [Debian-Preparation]: ../distrib-testing/Debian-build-environment.md [Engine installation]: ../../public/installation/zonemaster-engine.md [Engine sanity check]: ../../public/installation/zonemaster-engine.md#post-installation-sanity-check [FreeBSD-Preparation]: ../distrib-testing/FreeBSD-build-environment.md [GUI installation]: ../../public/installation/zonemaster-gui.md [GUI sanity check]: ../../public/installation/zonemaster-gui.md#post-installation-sanity-check [LDNS installation]: ../../public/installation/zonemaster-ldns.md#recommended-installation [LDNS sanity check]: ../../public/installation/zonemaster-ldns.md#post-installation-sanity-check [Requirements for IDN support]: https://github.com/zonemaster/zonemaster-ldns/blob/master/README.md#idn [Smoke test]: ../../public/installation/zonemaster-backend.md#61-smoke-test [Ubuntu-Preparation]: ../distrib-testing/Ubuntu-build-environment.md