CloudHost/zonemaster.es
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add full Zonemaster stack with Docker and Spanish UI

Browse Source 
- Clone all 5 Zonemaster component repos (LDNS, Engine, CLI, Backend, GUI)
- Dockerfile.backend: 8-stage multi-stage build LDNS→Engine→CLI→Backend
- Dockerfile.gui: Astro static build served via nginx
- docker-compose.yml: backend (internal) + frontend (port 5353)
- nginx.conf: root redirects to /es/, /api/ proxied to backend
- zonemaster-gui/config.ts: defaultLanguage set to 'es' (Spanish)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Malin
2026-04-21 08:19:24 +02:00
commit 8d4eaa1489
1567 changed files with 204155 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
zonemaster/docs/public/using/README.md Normal file
View File

@@ -0,0 +1,30 @@
# Zonemaster users guide
The purpose of the Zonemaster project is to develop a DNS validation software.
There are three ways one can use the Zonemaster software:
* Using the Web user Interface: In this case, there is no installation of
the Zonemaster software needed. A user types the URL
[https://zonemaster.net][zonemaster.net], provides as input a domain name,
and then can view the results.
* From the Command Line Interface (CLI): In this case, the user has to
install two components of the Zonemaster software, i.e. the
[Zonemaster-Engine] and the [Zonemaster-CLI].
* There are cases, wherein an organisation or a user, that wants to have the
results of their DNS validation stored and can be viewed later. In that
case, in addition to the Zonemaster-Engine and Zonemaster-CLI, the user
has to install the [Zonemaster-Backend] and the [Zonemaster-GUI].
User guides:
* [Use Zonemaster-CLI]
* [Use Zonemaster Backend]
* [Use Zonemaster-GUI]
[Use Zonemaster Backend]: backend/README.md
[Use Zonemaster-CLI]: cli.md
[Use Zonemaster-GUI]: gui/README.md
[Zonemaster-Backend]: ../installation/zonemaster-backend.md
[Zonemaster-CLI]: ../installation/zonemaster-cli.md
[Zonemaster-Engine]: ../installation/zonemaster-engine.md
[Zonemaster-GUI]: ../installation/zonemaster-gui.md
[zonemaster.net]: https://zonemaster.net/

10
zonemaster/docs/public/using/backend/README.md Normal file
View File

@@ -0,0 +1,10 @@
# Using the Backend
* [Using Zonemaster-Backend JSON-RPC API](Using-Zonemaster-Backend-JSON-RPC-API.md)
* [Using Zonemaster-Backend for batch testing](Using-Zonemaster-Backend-for-batch-testing.md)
* [RPCAPI reference]
* [Collecting metrics][Telemetry]
* [Using Zonemaster-Backend Docker container](Using-Zonemaster-Backend-Docker.md)
[RPCAPI reference]: rpcapi-reference.md
[Telemetry]: telemetry.md

127
zonemaster/docs/public/using/backend/Using-Zonemaster-Backend-Docker.md Normal file
View File

@@ -0,0 +1,127 @@
# Using Zonemaster-Backend Docker container
## Table of contents
* [Introduction](#introduction)
* [Invoking `zmtest` using Docker](#invoking-zmtest-using-docker)
* [Invoking `zmb` using Docker](#invoking-zmb-using-docker)
* [Invoking the command line tool using Docker]
* [IPv6 support](#ipv6-support)
## Introduction
This Docker image lets you run a working instance of Zonemaster-Backend.
It also contains a working version of zonemaster-cli which can be used
as a substitute for [zonemaster-cli],
how to use this version of cli is described at
[the end of this document][Invoking the command line tool using Docker].
The container is configured to use an SQLite database and run
all needed processes (rpcapi and testagent).
This image is designed to be used by one single user and
is therefore not suitable for production.
There is a limitation in Docker regarding IPv6.
Unless IPv6 has been enabled in the Docker daemon, there is no support for IPv6.
To avoid unnecessary errors,
use the corresponding option below if IPv6 support is not available.
See also the "IPv6 Support" section.
- zonemaster-cli: `--no-ipv6`
- zmtest: `--noipv6`
- zmb: `--ipv6 false`
## Invoking `zmtest` using Docker
The most basic use of the `zonemaster-backend` command is to just test a domain, e.g.
"zonemaster.net". To do so you first need to start `zonemaster/backend`
container and expose the port 5000
to be able to access the JSON RPC API.
```
docker run --rm -p 5000:5000 --name zm -d zonemaster/backend full
```
You can stop the Docker container with `docker stop zm`.
Once the zonemaster/backend container is started you can interact with it on
`localhost:5000` using JSON RPCAPI.
You can use `zmtest` embedded inside the backend image by using this command:
```
docker run -ti --rm --net host zonemaster/backend zmtest zonemaster.net
```
Run
```sh
docker run -ti --rm --net host zonemaster/backend zmtest zonemaster.net
```
to get usage for `zmtest`.
## Invoking `zmb` using Docker
You can also use the `zmb` command to interact with the `zonemaster/backend` container.
```
docker run -ti --rm --net host zonemaster/backend zmb start_domain_test --domain zonemaster.net
docker run -ti --rm --net host zonemaster/backend zmb get_test_results --test-id ed38765834e45b6e --lang en
```
When piping the output of `docker run` directly into `jq`,
you might encounter display issues due to the way Docker handles output.
These issues are caused by carriage return characters (^M) in the output.
To filter out these characters,
you should pipe `docker run`’s output into `tr -d '^M'` before piping it into `jq`.
```
docker run -ti --rm --net host zonemaster/backend zmb get_test_results (...) |tr -d '^M' | jq
```
Run
```sh
docker run -ti --rm --net host zonemaster/backend zmb
```
or
```
docker run -ti --rm --net host zonemaster/backend zmb man
```
to get usage and man page, respectively, for `zmb`.
## Invoking the command line tool using Docker
`zonemaster/backend` also contains the CLI tool shipped in
the [zonemaster-cli] container image.
You can invoke zonemaster-cli in the Zonemaster-Backend Docker image
by running the following command:
```
docker run -ti --rm zonemaster/backend:local cli zonemaster.net
```
## IPv6 support
On a Linux system IPv6 support can be enabled by creating or updating
`/etc/docker/daemon.json`. This is a minimal file that enables IPv6 support:
```json
{
"ipv6": true,
"fixed-cidr-v6": "2001:db8:1::/64"
}
```
Restart the docker daemon:
```sh
sudo systemctl restart docker
```
Also see the official Docker documentation "[Enable IPv6 support]".
[Docker Image Creation]: https://github.com/zonemaster/zonemaster/blob/master/docs/internal/maintenance/ReleaseProcess-create-docker-image.md
[Enable IPv6 support]: https://docs.docker.com/config/daemon/ipv6/
[Get started]: https://www.docker.com/get-started/
[IPv6 support]: #ipv6-support
[Invoking the command line tool using Docker]: #invoking-the-command-line-tool-using-docker
[Zonemaster-cli]: ../cli.md

311
zonemaster/docs/public/using/backend/Using-Zonemaster-Backend-JSON-RPC-API.md Normal file
View File

@@ -0,0 +1,311 @@
# Using Zonemaster-Backend JSON-RPC API
## Table of contents
* [Introduction](#introduction)
* [Check that the RPC API daemon is running and answering properly](#check-that-the-json-rpc-api-service-is-running-and-answering-properly)
* [Run a test of the zonemaster.net zone using zmtest](#run-a-test-of-the-zonemasternet-zone-using-zmtest)
* [Run a test of the zonemaster.net zone using zmb](#run-a-test-of-the-zonemasternet-zone-using-zmb)
* [Enqueue a test](#enqueue-a-test)
* [Check the progress](#check-the-progress)
* [Fetch the results](#fetch-the-results)
* [Find available languages](#find-available-languages)
* [Find previous tests](#find-previous-tests)
* [Other APIs](#other-apis)
* [IDN domain names](#idn-domain-names)
* [Using curl instead](#using-curl-instead)
## Introduction
This is a guide for getting started with the Zonemaster [JSON-RPC API] service
(running as a daemon).
Note that this guide makes a number of assumptions about your setup:
* That it is a Unix-like environment.
* That you have Zonemaster-Backend installed according to the
[installation guide]
* That Zonemaster-Backend is running. See [installation guide] for how to start
Zonemaster-Backend.
* If you have changed IP address (default 127.0.0.1) or port (default 5000) then
you must specify that in the commands below. Here we assume default.
For this guide we will use the two Zonemaster-Backend tools `zmb` and `zmtest`.
Both are installed when Zonemaster-Backend is installed. To get more information
what parameters they expect, run `zmb -h` and `zmtest -h`.
With `zmtest` you can start a domain test and get the result directly. With `zmb`
you can also do that, but in several steps, but on the other hand, `zmb` offers
many more possibilities. Actually, `zmtest` uses `zmb` behind the scen.
The `zmb` tool uses the JSON-RPC API to interact with Zonemaster-Backend.
Zonemaster-GUI also uses the same JSON-RPC API to start tests and fetch the
test results.
## Check that the JSON-RPC API service is running and answering properly
```sh
zmb version_info
```
To get a more readable output, pipe the command through `jq`:
```sh
zmb version_info | jq
```
If there is no response from the JSON-RPC API service, go to the
[installation guide] for how to start or restart it.
Below most commands will be piped through `jq`, but for processing you might want
to have the JSON on one line, or you might want to look at the options for `jq`
to extract fields (see `jq -h`).
## Run a test of the zonemaster.net zone using zmtest
To just run a test `zmtest` is the simple tool to use:
```sh
zmtest zonemaster.net
```
The tool will also display the result, which will be a large JSON object (here
truncated).
```
testid: 879d13569db70fde
100% done
{
"id": 1,
"jsonrpc": "2.0",
"result": {
"created_at": "2024-10-25T09:28:38Z",
"hash_id": "879d13569db70fde",
"params": {
"domain": "zonemaster.net",
"ds_info": [],
"ipv4": true,
"ipv6": true,
"nameservers": [],
"priority": 10,
"profile": "default",
"queue": 0
},
"results": [
{
"level": "INFO",
"message": "Using version v6.0.0 of the Zonemaster engine.\n",
"module": "System",
"testcase": "Unspecified"
},
{
"level": "INFO",
"message": "The parent zone is \"net\" as returned from name servers \"a.gtld-servers.net/192.5.6.30; a.gtld-servers.net/2001:503:a83e::2:30; b.gtld-servers.net/192.33.14.30; b.gtld-servers.net/2001:503:231d::2:30; c.gtld-servers.net/192.26.92.30; c.gtld-servers.net/2001:503:83eb::30; d.gtld-servers.net/192.31.80.30; d.gtld-servers.net/2001:500:856e::30; e.gtld-servers.net/192.12.94.30; e.gtld-servers.net/2001:502:1ca1::30; f.gtld-servers.net/192.35.51.30; f.gtld-servers.net/2001:503:d414::30; g.gtld-servers.net/192.42.93.30; g.gtld-servers.net/2001:503:eea3::30; h.gtld-servers.net/192.54.112.30; h.gtld-servers.net/2001:502:8cc::30; i.gtld-servers.net/192.43.172.30; i.gtld-servers.net/2001:503:39c1::30; j.gtld-servers.net/192.48.79.30; j.gtld-servers.net/2001:502:7094::30; k.gtld-servers.net/192.52.178.30; k.gtld-servers.net/2001:503:d2d::30; l.gtld-servers.net/192.41.162.30; l.gtld-servers.net/2001:500:d937::30; m.gtld-servers.net/192.55.83.30; m.gtld-servers.net/2001:501:b1f9::30\".\n",
"module": "Basic",
"testcase": "Basic01"
},
{
"level": "INFO",
"message": "The zone \"zonemaster.net\" is found.\n",
"module": "Basic",
"testcase": "Basic01"
},
{
"level": "INFO",
"message": "Authoritative answer on SOA query for \"zonemaster.net\" is returned by name servers \"ns2.nic.fr/192.93.0.4; ns2.nic.fr/2001:660:3005:1::1:2; nsa.dnsnode.net/194.58.192.46; nsa.dnsnode.net/2a01:3f1:46::53; nsp.dnsnode.net/194.58.198.32; nsp.dnsnode.net/2a01:3f1:3032::53; nsu.dnsnode.net/185.42.137.98; nsu.dnsnode.net/2a01:3f0:400::32\".\n",
"module": "Basic",
"testcase": "Basic02"
},
(...)
```
You will find the meaning of all fields in the outputs in
[Zonemaster-Backend JSON-RPC API reference][JSON-RPC API].
## Run a test of the zonemaster.net zone using zmb
If we instead use `zmb` then this will be done in three steps (that happen
behind the scen when using `zmtest`):
1. Enqueue test.
2. Check if testing has completed (progress) - maybe several times.
3. Fetch result (maybe several times with different translations).
### Enqueue a test
To enqueue a test of zonemaster.net we run
`zmb start_domain_test --domain zonemaster.net | jq` and immediately get the
response
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": "879d13569db70fde"
}
```
The `results` field holds the test ID which we need for further steps. It
always consists of 16 hexadecimal digits.
### Check the progress
To check the progress we run `zmb test_progress --test-id 879d13569db70fde | jq`
with the test ID from enqueueing, and get the response
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": 8
}
```
Now the `results` field holds the progress of testing, an integer between 0 and
100 to mean 0% to 100%. 0% means that testing has not started (maybe many tests
in the queue) and 100% means that testing has completed.
We have to stay in the check step until the test has reached 100%. In our
example above, it is 8%, so we run the command again and now we get the
response
```json
{
"result": 100,
"jsonrpc": "2.0",
"id": 1
}
```
### Fetch the results
When the test has completed the test results can be fetched. The test ID is found
above and then the language has to be chosen. Here `en` is used. More about
languages below. Run
`zmb get_test_results --test-id 879d13569db70fde --lang en | jq` and get a JSON
structure with data (as with `zmtest`). Again, the JSON structure is long and is
truncated here.
```
{
"id": 1,
"result": {
"results": [
{
"message": "Using version v6.0.0 of the Zonemaster engine.\n",
"testcase": "Unspecified",
"level": "INFO",
"module": "System"
},
{
"module": "Basic",
"level": "INFO",
"testcase": "Basic01",
"message": "The parent zone is \"net\" as returned from name servers \"a.gtld-servers.net/192.5.6.30; a.gtld-servers.net/2001:503:a83e::2:30; b.gtld-servers.net/192.33.14.30; b.gtld-servers.net/2001:503:231d::2:30; c.gtld-servers.net/192.26.92.30; c.gtld-servers.net/2001:503:83eb::30; d.gtld-servers.net/192.31.80.30; d.gtld-servers.net/2001:500:856e::30; e.gtld-servers.net/192.12.94.30; e.gtld-servers.net/2001:502:1ca1::30; f.gtld-servers.net/192.35.51.30; f.gtld-servers.net/2001:503:d414::30; g.gtld-servers.net/192.42.93.30; g.gtld-servers.net/2001:503:eea3::30; h.gtld-servers.net/192.54.112.30; h.gtld-servers.net/2001:502:8cc::30; i.gtld-servers.net/192.43.172.30; i.gtld-servers.net/2001:503:39c1::30; j.gtld-servers.net/192.48.79.30; j.gtld-servers.net/2001:502:7094::30; k.gtld-servers.net/192.52.178.30; k.gtld-servers.net/2001:503:d2d::30; l.gtld-servers.net/192.41.162.30; l.gtld-servers.net/2001:500:d937::30; m.gtld-servers.net/192.55.83.30; m.gtld-servers.net/2001:501:b1f9::30\".\n"
},
{
"message": "The zone \"zonemaster.net\" is found.\n",
"module": "Basic",
"testcase": "Basic01",
"level": "INFO"
},
{
"level": "INFO",
"testcase": "Basic02",
"module": "Basic",
"message": "Authoritative answer on SOA query for \"zonemaster.net\" is returned by name servers \"ns2.nic.fr/192.93.0.4; ns2.nic.fr/2001:660:3005:1::1:2; nsa.dnsnode.net/194.58.192.46; nsa.dnsnode.net/2a01:3f1:46::53; nsp.dnsnode.net/194.58.198.32; nsp.dnsnode.net/2a01:3f1:3032::53; nsu.dnsnode.net/185.42.137.98; nsu.dnsnode.net/2a01:3f0:400::32\".\n"
},
(...)
```
## Find available languages
The result can be presented in several languages, depending on installation. To
find out what languages that are available, run `zmb get_language_tags | jq`.
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": [
"da",
"en",
"es",
"fi",
"fr",
"nb",
"sl",
"sv"
]
}
```
## Find previous tests
It is possible to find previous tests of a specific domain (zone). Run
`zmb get_test_history --domain zonemaster.net |jq` to get all tests of
`zonemaster.net`. The output looks like the following:
```
{
"jsonrpc": "2.0",
"result": [
{
"undelegated": false,
"created_at": "2024-10-25T09:54:38Z",
"id": "00669309ac7887c3",
"overall_result": "ok"
},
{
"overall_result": "ok",
"id": "879d13569db70fde",
"undelegated": false,
"created_at": "2024-10-25T09:28:38Z"
}
],
"id": 1
}
```
Now you can get the results of any of the listed tests by running
`get_test_results` with selected test ID and language as above.
## Other APIs
There are a few other APIs that can be used throught `zmb`, and can be found by
`zmb -h` and in [Zonemaster-Backend JSON-RPC API reference][JSON-RPC API]. The APIs for batch
testing are covered in [Using Zonemaster Backend for batch testing].
## IDN domain names
When enqueuing a test of an IDN domain name with `zmb start_domain_test` it is
possible to provide the domain name as either A-label or U-label. E.g. both
"räksmörgås.se" or "xn--rksmrgs-5wao1o.se" (the same domain name) work as well.
The same is also true if using Curl instead (see below).
## Using Curl instead
Instead of using `zmb` it is possible to run the commands by `curl`, which
requires that the JSON structure is created, but can give greater flexibility.
The method can also be copied into different programming languages when
scripting.
The same enqueuing of the zonemaster.net zone as above will then be:
```sh
curl -sS -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 2, "method": "start_domain_test", "params": {"domain": "zonemaster.net"}}' http://localhost:5000/ | jq .
```
And fetching the result, when completed, will be:
```sh
curl -sS -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 5, "method": "get_test_results", "params": {"id": "879d13569db70fde", "language": "en"}}' http://localhost:5000/ | jq .
```
The format of the JSON structure for every method is found in the
[Zonemaster-Backend JSON-RPC API reference][JSON-RPC API].
[installation guide]: ../../installation/zonemaster-backend.md
[JSON-RPC API]: rpcapi-reference.md
[Using Zonemaster Backend for batch testing]: Using-Zonemaster-Backend-for-batch-testing.md

402
zonemaster/docs/public/using/backend/Using-Zonemaster-Backend-for-batch-testing.md Normal file
View File

@@ -0,0 +1,402 @@
# Using Zonemaster-Backend for batch testing
## Table of contents
* [Introduction](#introduction)
* [Combining GUI and batch](#combining-gui-and-batch)
* [Configuration to run batches](#configuration-to-run-batches)
* [Create batch user](#create-batch-user)
* [Run a batch job](#run-a-batch-job)
* [Start a batch job](#start-a-batch-job)
* [Status of a batch job](#status-of-a-batch-job)
* [Scale out](#scale-out)
* [Enable global cache](#enable-global-cache)
* [Increase the worker pool size](#increase-the-worker-pool-size)
* [Add more Test Agent services](#add-more-test-agent-services)
* [Add more test servers](#add-more-test-servers)
* [Queues](#queues)
* [Appendix: Add more JSON-RPC API services]
## Introduction
In [Using the Backend JSON-RPC API] it is shown how a test of a single domain name
(zone) can be started using the JSON-RPC API to Zonemaster-Backend. It is also
possible to start a batch of domain names at the same time. In principle such
a batch is the same thing as running the `start_domain_test` individually for each
domain name in the batch.
This document describes how batches of domain name tests can be run and what
to think about when running batches, especially large ones.
Zonemaster-Backend uses a database to store enqueued tests and
results from tests. It is possible to use SQLite as the database backend for
small installations and test installations, but that is not appropriate for running
real batches on. Therefore in this document it is assumed that Backend has been
installed with [MariaDB/MySQL] or [PostgreSQL].
## Combining GUI and batch
Both GUI and batch uses Zonemaster-Backend that stores the data in the database.
If the same database is used, then all tests run, regardless if they are
started via GUI or a batch, will be available in the GUI interface
using the `get_test_history` (see relevant section in
[Using the Backend JSON-RPC API][Using#find-previous-tests]). Depending on the
situation, running batches on the same Backend as GUI may be desirable or not
recommended. If the results are expected to be available through GUI then the batches must be
run on the same Backend. If the batches are run with a reduced set of test cases
then it can be confusing since the results are not fully comparable with
those from tests started from GUI. If the results from batch tests should not be
mixed with tests from normal GUI tests, a separate installation of Backend is
required, and that might not need any GUI at all.
Test prioritization is another aspect of mixing tests from batches and GUI on the same Backend. By default Backend will run
tests from GUI (or rather by `start_domain_test`) before test from a batch. In
this way interactive GUI users run tests do not have to wait for a
batch to complete. A large batch might run for days.
## Configuration to run batches
To start a batch job, setting `enable_add_batch_job` in [backend_config.ini] must
be enabled (*enabled* by default). Moreover a special user must also be available.
To create that user setting `enable_add_api_user` must be enabled
(*disabled* by default). (See below for how to create the user.) Note that this setting
only needs to be enabled while creating the user(s), not when starting a batch job.
After changing these settings, the JSON-RPC API service (running as a daemon)
must be restarted.
It is not recommended to enable `enable_add_api_user` for an JSON-RPC API service
that can be used by untrusted users, e.g. via GUI. If the installation for the
batch jobs is a dedicated installation, then it is not an issue. If it is used by
a public GUI that could be closed while the batch user or users are created, then
that could be an option. Creating the user only takes as long time as it takes to
type the command in, i.e. seconds.
Another option is to set up an additional JSON-RPC API service using the same
database. Zonemaster-Backend can have several parallel JSON-RPC API services with
different configuration, see [Appendix: Add more JSON-RPC API services].
## Create batch user
The [Using the Backend JSON-RPC API] guide shows how `zmb` can be used for
several JSON-RPC API methods. It can also be used for the creation of the batch
user. It is here assumed that `enable_add_api_user` has been enabled or else the
following will be included in the response to the `zmb add_api_user` command:
```
"message": "Procedure 'add_api_user' not found"
```
Run `zmb add_api_user --username myuser --api-key mykey | jq` where `myuser` is
a user name of your choice and `mykey` the password, which should probably be
longer and more complex. The output is simply
```json
{
"jsonrpc": "2.0",
"result": 1,
"id": 1
}
```
Now `enable_add_api_user` can be disabled.
## Run a batch job
The `zmb` tool is used for that purpose. A batch job can be started with just a few
domain names or a list of millions of domain names.
### Start a batch job
Batches are created using `add_batch_job`:
```
$ zmb add_batch_job --username myuser --api-key mykey --domain zonemaster.net --domain zonemaster.se --domain zonemaster.fr | jq
{
"jsonrpc": "2.0",
"result": 2,
"id": 1
}
```
Options that are mandatory are `--username`, `--api-key` and `--domain`. Option
`--domain` is repeated for every domain name in the batch. The other options are
optional. For all options see `zmb man`.
The batch ID is found in "result", which is "2" in this case.
Listing the domain names one by one is however cumbersome unless it is a very
small batch as above. A better alternative for a larger batch is to create text
file of the domain names to test, one domain name per line. Comments starting
with `#` is permitted, which makes it possible to add meta information to the
file.
```
cat batchfile.txt
# Example batch
zonemaster.net
zonemaster.se
zonemaster.fr
```
Now we can run the same batch as above, but from the file:
```
$ zmb add_batch_job --username myuser --api-key mykey --file batchfile | jq
{
"jsonrpc": "2.0",
"result": 3,
"id": 1
}
```
Starting the batch with the domain names one by one or in the file gives exactly
the same result. Also in this case, IDN domain names can be entered both with
A-label and U-label (UTF-8 is assumed).
### Status of a batch job
To check the status of a batch job the batch ID is required ("2" in the
case above), using `batch_status`:
```
$ zmb batch_status --batch-id 2 | jq
{
"id": 1,
"result": {
"finished_count": 0,
"running_count": 2,
"waiting_count": 1
},
"jsonrpc": "2.0"
}
```
Two tests are running, one is waiting to be started and none has been completed
yet.
To get the test IDs of the the tests in each of the categories, add one or more
of the "--lw" (list waiting), "--lr" (list running) or "--lf" (list finished)
option to the previous call:
```
$ zmb batch_status --batch-id 2 --lw --lr --lf| jq
{
"id": 1,
"result": {
"finished_count": 0,
"running_count": 2,
"waiting_count": 1,
"waiting_tests": [
"b26b874493ed4b57"
],
"running_tests": [
"9af528070fa2551a",
"708f82b5176261dc"
]
},
"jsonrpc": "2.0"
}
```
There is no "finished_tests" since that category is empty.
After some time running the command again (with "--lf") gives:
```json
{
"id": 1,
"result": {
"finished_count": 3,
"running_count": 0,
"waiting_count": 0,
"finished_tests": [
"9af528070fa2551a",
"708f82b5176261dc",
"b26b874493ed4b57"
]
},
"jsonrpc": "2.0"
}
```
Or without "--lf":
```json
{
"id": 1,
"result": {
"finished_count": 3,
"running_count": 0,
"waiting_count": 0
},
"jsonrpc": "2.0"
}
```
Now all tests in the batch are completed. The test IDs in `finished_tests`
can be used to get the results with `get_test_results`.
## Scale out
Running a batch can take a long time and if that is a problem then it is possible
to increase the performance by:
* Enabling global cache
* Increasing `number_of_processes_for_batch_testing`
* Adding one or more Test Agent service
* Adding one or more test servers
### Enable global cache
For each domain name in a batch job, a test run is executed.
Each test run keeps its own cache to avoid sending the same query twice.
However these caches are never shared between test runs.
This may result in large numbers of redundant queries for large batches where
the domain name directly (or indirectly) share data with other domain names.
To share responses between test runs, you can enable the global cache feature
which works like a second level cache.
For interactive tests via GUI the global cache is probably not desired. If a user
changes the DNS configuration of a domain then the following tests results should
not be based on old data.
To enable global cache a custom profile must be used. See the instructions in
[Configuration: Global cache in Zonemaster-Engine][Global-cache].
The custom profile is assumed to be `/etc/zonemaster/profile-batch.json`
(Linux) or `/etc/local/zonemaster/profile-batch.json` (FreeBSD).
The custom profile file, say `profile-batch.json`, must be added to the Backend
configuration to be made available for the tests. In `backend_config.ini` look
for the section `[PRIVATE PROFILES]`. Under that add the following line for
Linux:
```
batch=/etc/zonemaster/profile-batch.json
```
For FreeBSD add:
```
batch=/usr/local/etc/zonemaster/profile-batch.json
```
After updating the Backend configuration, restart both JSON-RPC API service and
Test Agent service (all daemons of both services if there are more than one of
each).
Now you can start a batch -- using the same batch file as above -- with:
```
$ zmb add_batch_job --profile batch --username myuser --api-key mykey --file batchfile | jq
{
"jsonrpc": "2.0",
"result": 4,
"id": 1
}
```
### Increase the worker pool size
A standard Backend installation has a single Test Agent service. Each Test Agent
service maintains a pool of workers. Test Agents monitor the Backend database for
tests and assigns them to workers.
Increasing the size of the worker pool can improve the testing performance. The
improvement is limited by CPU, I/O, memory and the ability of the Test Agent
service to keep its workers busy, so monitor these factors to find the best
configuration.
To adjust the worker pool size, set `number_of_processes_for_batch_testing` in
the [backend_config.ini] configuration file.
### Add more Test Agent services
If the increase of `number_of_processes_for_batch_testing` does not help anymore
and the server has free resources (IO, CPU and RAM) then adding another Test
Agent services (with its pool of workers) can help.
The default Test Agent service is started by the start script
`/etc/systemd/system/zm-testagent.service` (Rocky Linux),
`/etc/init.d/zm-testagent` (Debian/Ubuntu) or `/usr/local/etc/rc.d/zm_testagent`
(FreeBSD). Create a copy so that the additional Test Agent service has a new
name, new PID file and new log file. Also see the
[Backend installation instructions]. Each Test Agent service will be its own
daemon.
Start the new daemon and now a second Test Agent service is running.
Additional Test Agent services can be added by repeating the steps above.
Unless the different Test Agent services should process different
[queues](#queues) the same configuration file can be used.
### Add more test servers
Adding more servers is a generalization of adding more Test Agent daemons.
* Configure the database to listen on an external interface. The database can
also be moved to a dedicated server.
* Install Backend on the new Test Agent server, but disable JSON-RPC API service.
* Update configuration to use the external database server.
* Consider adding more Test Agent services.
* If global cache is used, configure Redis to listen on an external interface.
The Redis server can also be moved to a dedicated server.
* Consider moving JSON-RPC API service to a dedicated server. If so, update the
configuration to use the external database server.
* If applicable, configure all parts to use external database server and external
Redis server.
## Queues
Every enqueued test has a [queue tag][JSON-RPC API#queue]. In the
[configuration][Config#lock_on_queue] file it is set which queue should be
processed by the Test Agent service. Only tests with matching queue value
(default 0) will be processed. Queue tag is set by the JSON-RPC API methods
`start_domain_test` and `add_batch_job`, respectively (default 0 in both cases).
If desirable it is possible to create a batch with another queue value and then
let a dedicated Test Agent service, possibly on a dedicated server, process the
tests in the batch, while another Test Agent service processes tests from GUI.
It is important to note that if a test is tagged with a queue value that no
Test Agent service uses it will never be processed.
In the end, all completed tests will end up in the same database and will be
mixed when looking for completed tests for the same domain name
(`get_test_history`).
## Appendix: Add more JSON-RPC API services
The JSON-RPC API service listens by default on 127.0.0.1 on port 5000. That is
governed by the start script for the JSON-RPC API service which is
`/etc/systemd/system/zm-rpcapi.service` (Rocky Linux), `/etc/init.d/zm-rpcapi`
(Ubuntu and Debian) or `/usr/local/etc/rc.d/zm_rpcapi` (FreeBSD). Create a copy
so that the additional service (daemon) has a new name, new PID file, new log
file and listens to a new port. Also see the [Backend installation instructions].
Start the new daemon and now a second JSON-RPC API service is running.
To be useful, the second JSON-RPC API service should probably have a different
configuration file too. The configuration file is found as
`/etc/zonemaster/backend_config.ini` (Linux) or as
`/usr/local/etc/zonemaster/backend_config.ini` (FreeBSD). The database settings
must not be changed, or else the two JSON-RPC API services will not be part of
the same Zonemaster Backend. Examples of settings that could be different between
the two JSON-RPC API services are:
* enable_add_api_user
* Available profiles
Restart the JSON-RPC API services to use the updated configuration files.
[Appendix: Add more JSON-RPC API services]: #appendix-add-more-json-rpc-api-services
[Backend installation instructions]: ../../installation/zonemaster-backend.md
[Backend_config.ini]: ../../configuration/backend.md
[Global-cache]: ../../configuration/global-cache.md
[MariaDB/MySQL]: ../../installation/zonemaster-backend.md#7-installation-with-mariadb
[PostgreSQL]: ../../installation/zonemaster-backend.md#8-installation-with-postgresql
[Using the Backend JSON-RPC API]: Using-Zonemaster-Backend-JSON-RPC-API.md
[Using#find-previous-tests]: Using-Zonemaster-Backend-JSON-RPC-API.md#find-previous-tests
[JSON-RPC API#queue]: rpcapi-reference.md#queue
[Config#lock_on_queue]: ../../configuration/backend.md#lock_on_queue

1581
zonemaster/docs/public/using/backend/rpcapi-reference.md Normal file
View File

File diff suppressed because it is too large Load Diff

73
zonemaster/docs/public/using/backend/telemetry.md Normal file
View File

@@ -0,0 +1,73 @@
# Telemetry
## Metrics
If enabled in the [Metrics section][metrics feature] in the configuration file,
[Statsd][statsd] compatible metrics are available to use:
| Name | Type | Description |
| ------------------------------------------------ | ------- | ----------- |
| zonemaster.rpcapi.requests.\<METHOD>.\<STATUS> | Counter | Number of times the JSON RPC method \<METHOD> resulted in JSON RPC status \<STATUS>. The status is represented in string, possible values are: `RPC_PARSE_ERROR`, `RPC_INVALID_REQUEST`, `RPC_METHOD_NOT_FOUND`, `RPC_INVALID_PARAMS`, `RPC_INTERNAL_ERROR`. |
| zonemaster.testagent.tests_started | Counter | Number of tests that have started. |
| zonemaster.testagent.tests_completed | Counter | Number of tests that have been completed successfully. |
| zonemaster.testagent.tests_died | Counter | Number of tests that have died. |
| zonemaster.testagent.tests_duration_seconds | Timing | The duration of a test, emitted for each test. |
| zonemaster.testagent.cleanup_duration_seconds | Timing | Time spent to kill timed out processes. |
| zonemaster.testagent.fetchtests_duration_seconds | Timing | Time spent selecting the next text to run and processing unfinished tests. |
| zonemaster.testagent.running_processes | Gauge | Number of running processes in a test agent. |
| zonemaster.testagent.maximum_processes | Gauge | Maximum number of running processes in a test agent. |
### Usage
Testing the metrics feature can be as easy as running a listening UDP server like
```sh
ns -lup 8125
```
This should be enough to see the metrics emitted by Zonemaster.
More complex setups are required for the metrics to be used in alerts and dashboards.
StatsD metrics can be integrated to a number of metrics backend like Prometheus (using the [StatsD exporter]), InfluxDB (using Telegraf and the [StatsD plugin]), Graphite ([integration guide]) and others.
#### StatsD Exporter (Prometheus)
1. Download the binary (tar file) corresponding to your environment https://github.com/prometheus/statsd_exporter/releases
2. Untar the file
3. `cd` into the statsd_exporter directory
4. Create a `statsd_mapping.yml` file with content as below
```yml
mappings:
- match: zonemaster.rpcapi.requests.*.*
name: zonemaster_rpcapi_requests_total
labels:
method: $1
status: $2
- match: zonemaster.testagent.tests_duration_seconds
observer_type: histogram
buckets: [ 1, 2.5, 5, 10, 15, 30, 45, 60, 75, 90, 105, 120, 150, 180]
name: zonemaster_testagent_tests_duration_seconds
- match: zonemaster.testagent.cleanup_duration_seconds
observer_type: histogram
buckets: [ 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2]
name: zonemaster_testagent_cleanup_duration_seconds
- match: zonemaster.testagent.fetchtests_duration_seconds
observer_type: histogram
buckets: [ 0.001, 0.0025, 0.005, 0.0075, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 3, 5, 10]
name: zonemaster_testagent_fetchtests_duration_seconds
```
5. Run it like
```
./statsd_exporter --statsd.mapping-config=./statsd_mapping.yml --statsd.listen-udp=:8125 --statsd.listen-tcp=:8125
```
6. Run the following to see Zonemaster metrics:
```
curl localhost:9102/metrics | grep zonemaster
```
[metrics feature]: ../../installation/zonemaster-backend.md#101-metrics
[statsd]: https://github.com/statsd/statsd
[StatsD exporter]: https://github.com/prometheus/statsd_exporter
[StatsD plugin]: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd
[integration guide]: https://github.com/statsd/statsd/blob/master/docs/graphite.md

348
zonemaster/docs/public/using/cli.md Normal file
View File

@@ -0,0 +1,348 @@
# Using the CLI
## Table of contents
* [Docker or local installation](#docker-or-local-installation)
* [Invoking the command line tool using Docker](#invoking-the-command-line-tool-using-docker)
* [Invoking the command line tool using local installation](#invoking-the-command-line-tool-using-local-installation)
* [More details on the command line tool invocation](#more-details-on-the-command-line-tool-invocation)
* [Test reports](#test-reports)
* [Translation]
* [Advanced use](#advanced-use)
* [Docker on Mac with M1 chip](#docker-on-mac-with-m1-chip)
* [IPv6 support]
## Docker or local installation
The `zonemaster-cli` tool can be run from the command line of any computer that
meets one of the following requirements:
* Docker is installed on the computer, or
* Zonemaster-CLI has been installed on the computer.
### Using Docker
To run Zonemaster-CLI on Docker you have to make sure that Docker is installed
on the computer and that you can run Docker on it.
* Instructions for installation are found on Docker [get started] page.
* Run the command `docker ps` on the command line to verify that you can run
Docker on the computer.
When Docker has been correctly installed, no more installation is needed to run
`zonemaster-cli`. Just follow the examples below.
There is a limitation in Docker regarding IPv6. Unless IPv6 has been enabled in
the Docker daemon, there is no support for IPv6. To avoid meaningless errors,
use `--no-ipv6` if there is no IPv6 support. Also see section "[IPv6 support]".
### Local installation
To have an local installation of Zonemaster-CLI follow the
[installation instruction]. When installed, the examples below can be followed.
If the network has no IPv6 support (common in home networks) then turn off IPv6.
Use `--no-ipv6` to avoid meaningless errors if there is no IPv6 support.
## Invoking the command line tool using Docker
The most basic use of the `zonemaster-cli` command is to just test a domain, e.g.
"zonemaster.net".
There are two usable Docker images to run `zonemaster-cli`:
`zonemaster/cli` and `zonemaster/backend`.
For more information on how to use `zonemaster/backend` image to invoke
the cli, see the [Docker Zonemaster-Backend] documentation.
```sh
docker run -t --rm zonemaster/cli zonemaster.net --no-ipv6
```
or
```sh
docker run -t --rm zonemaster/cli zonemaster.net
```
To make sure that Docker uses the latest version, add `--pull always`, e.g.
```sh
docker run -t --rm --pull always zonemaster/cli zonemaster.net --no-ipv6
```
If `--pull always` is skipped, the invocation is quicker. The recommendation is
to include `--pull always` in the first command of a session to make sure latest
version is used, and then to exclude it to improve performance.
## Invoking the command line tool using local installation
The most basic use of the `zonemaster-cli` command is to just test a domain, e.g.
"zonemaster.net".
```sh
zonemaster-cli zonemaster.net
```
or
```sh
zonemaster-cli zonemaster.net --no-ipv6
```
## More details on the command line tool invocation
The output of any of the commands above comes continuously as the tests (test
cases) are performed.
```
Seconds Level Message
======= ========= =======
21.39 WARNING The DNSKEY with tag 54636 uses an algorithm number 5 (RSA/SHA1) which is not recommended to be used.
21.80 WARNING DNSKEY with tag 26280 and using algorithm 5 (RSA/SHA1) has a size (1024) smaller than the recommended one (2048).
23.61 NOTICE SOA 'refresh' value (10800) is less than the recommended one (14400).
```
The test and output can be modified with different options:
* If your machine is not configured for use with IPv6 you want to disable the
use of IPv6 with the `--no-ipv6` option.
* If you want to have the test case from which the message is printed then
include the `--show-testcase` option.
* If you want to see the messages translated into another language (see
"[Translation]" section below) then include e.g. `--locale da` (Docker) or
`--locale da_DK.UTF-8` (local installation).
The same test as above with the three options included:
```sh
docker run -t --rm zonemaster/cli zonemaster.net --no-ipv6 --show-testcase --locale=da
```
```sh
zonemaster-cli zonemaster.net --no-ipv6 --show-testcase --locale=da_DK.UTF-8
```
To get brief descriptions of a selection of the most important command line
options run:
```sh
zonemaster-cli --help
```
For complete reference documentation, see the manual page.
This includes an exhaustive list of options and in-depth documentation for each one,
as well as examples and additional context.
```sh
man zonemaster-cli
```
### Using Docker or local installation
The difference between running `zonemaster-cli` on Docker or local installation
is the invocation string, `docker run -t --rm zonemaster/cli` vs.
`zonemaster-cli`. To simplify this document, from now on the shorter
`zonemaster-cli` will be used and for Docker the longer string will be assumed.
To simplify repeated invocation on Docker an alias can be created for the shell.
## Test reports
The severity level of the different messages is CRITICAL, ERROR, WARNING, NOTICE,
INFO, DEBUG, DEBUG2 or DEBUG3. The default reporting level is NOTICE and higher.
To change the level of reporting you can use a command line option, e.g
`--level=INFO` includes level INFO and higher in the report. See
"[Severity Level Definitions]" for more information on the levels.
By default the output is formatted as plain text in English (or some other
language), but other more "technical" output formats are also available with
options `--raw` and `json`, respectively.
## Translation
### In Docker
By default all messages are in English. By using the `--locale=LANG` option
another language can be selected. Select "LANG" from the table below to have
Zonemaster translated into that language.
LANG | Language
-----|---------
da | Danish
en | English
fi | Finnish
fr | French
nb | Norwegian
es | Spanish
sl | Slovenian
sv | Swedish
E.g.:
```sh
docker run -t --rm zonemaster/cli zonemaster.net --locale=da
```
An alternative is to set the `LC_ALL` environment variable with correct language
value when the command is invoked, which can be useful if a shell alias is
created. E.g.
```sh
docker run -e LC_ALL=da -t --rm zonemaster/cli zonemaster.net
```
If environment variable `LC_ALL` is set in the local shell with the correct
"LANG" or with the equivalent "LOCALE" in from next section, then the following
command will export `LC_ALL` with the that value to the docker container.
```sh
docker run -e LC_ALL -t --rm zonemaster/cli zonemaster.net
```
Environment variables `LANG` and `LC_MESSAGES` can be used in the same way as
`LC_ALL`.
### In local installation
By default all messages are in the language set in the local environment (if
available in Zonemaster) or else in English. By using the `--locale=LOCALE`
option another language can be selected. Select "LOCALE" from the table below to
have Zonemaster translated into that language.
LOCALE | Language
------------|---------
da_DK.UTF-8 | Danish
en_US.UTF-8 | English
fi_FI.UTF-8 | Finnish
fr_FR.UTF-8 | French
nb_NO.UTF-8 | Norwegian
es_ES.UTF-8 | Spanish
sv_SE.UTF-8 | Swedish
sl_SI.UTF-8 | Slovenian
E.g.:
```sh
docker run -t --rm zonemaster/cli zonemaster.net --locale=da_DK.UTF-8
```
If the environment variable `LANGUAGE` is set with correct LOCALE then no option
is needed, e.g. `LANGUAGE=da_DK.UTF-8`. `zonemaster-cli` also respects `LC_ALL`,
`LC_MESSAGES` and `LANG`. `LANGUAGE` takes precedence over the other, and then
the order is `LC_ALL`, `LC_MESSAGES` and last `LANG`.
## Advanced use
There are some nice features available that can be of some use for advanced
users.
### Only run specific test cases
If you only want to run a specific test case rather than the whole suite of
tests, you can do that as well. E.g. test only test case [Connectivity03]:
```sh
zonemaster-cli --test Connectivity/connectivity03 example.com
```
Or all test case in the Connectivity test level:
```sh
zonemaster-cli --test Connectivity example.com
```
For more information on the available tests, you can list them right from
the command line tool:
```sh
zonemaster-cli --list_tests
```
### Use custom root hints
You can override the built-in list of root servers that `zonemaster-cli` uses
by providing a path to a custom root hints file with the `--hints HINTS-FILE`
option. For example:
```sh
zonemaster-cli --hints /path/to/custom.hints example.com
```
If you are running `zonemaster-cli` using Docker, you must mount your custom
root hints file inside the container using a volume so that `zonemaster-cli`
can access it, like so:
```sh
docker run -t --rm \
-v /path/to/custom.hints:/hints \
zonemaster/cli --hints /hints example.com
```
## Undelegated test
Before you do any delegation change at the parent, either changing the NS
records, glue address records or DS records, you might want to perform a
check of your new child zone configuration so that everything you plan to
change is in order. Zonemaster can do this for you. All you have to do
is give Zonemaster all the parent data you plan to have for your new
configuration. Any DNS lookups going for the parent will instead be
answered by the data you entered. E.g.
```sh
zonemaster-cli --ns ns1.example.com/192.168.23.23 \
--ns ns2.example.com/192.168.24.24 \
--ds 12345,3,1,123456789abcdef67890123456789abcdef67890
```
Any number of NS records and DS records can be given multiple times.
The syntax of the NS records is name/address, and the address can be
both IPv4 and IPv6. The DS syntax is keytag,algorithm,type,digest.
You can also choose to do a undelegated test using only the new DS
record, but keep the NS records from the parent by only specifying the
DS record and no NS records on the command line.
## Docker on Mac with M1 chip
If you run the Docker commands above on a Mac computer with the M1 chip, then you
will get the following warning:
> WARNING: The requested image's platform (linux/amd64) does not match the
> detected host platform (linux/arm64/v8) and no specific platform was requested
The warning says that the image is created for Intel/AMD64 architecture, and that
is not what your computer has. To get rid of the warning, add
`--platform linux/amd64` to `docker run`, e.g.
```sh
docker run --platform linux/amd64 -t --rm zonemaster/cli zonemaster.net --no-ipv6
```
If you search for the error messages you will get suggestions for how to
automatically include the `--platform linux/amd64` option every time you run
`docker run`.
## IPv6 support
On a Linux system IPv6 support can be enabled by creating or updating
`/etc/docker/daemon.json`. This is a minimal file that enables IPv6 support:
```json
{
"ipv6": true,
"fixed-cidr-v6": "2001:db8:1::/64"
}
```
Restart the docker daemon:
```sh
sudo systemctl restart docker
```
Also see the official Docker documentation "[Enable IPv6 support]".
[Connectivity03]: ../specifications/tests/Connectivity-TP/connectivity03.md
[Get started]: https://www.docker.com/get-started/
[Installation instruction]: ../installation/zonemaster-cli.md
[IPv6 support]: #ipv6-support
[Severity Level Definitions]: ../specifications/tests/SeverityLevelDefinitions.md
[Translation]: #translation
[Enable IPv6 support]: https://docs.docker.com/config/daemon/ipv6/
[Docker Zonemaster-Backend]: backend/Using-Zonemaster-Backend-Docker.md#invoking-the-command-line-tool-using-docker

5
zonemaster/docs/public/using/gui/README.md Normal file
View File

@@ -0,0 +1,5 @@
# Using the GUI
* The [GUI API][API]
[API]: api.md

56
zonemaster/docs/public/using/gui/api.md Normal file
View File

@@ -0,0 +1,56 @@
# API
The Zonemaster GUI provides an API to access specific resource and perform
specific tasks by adding a path to the Zonemaster GUI base URL, e.g.
`https://zonemaster.net/`.
For the paths the following keys are used:
* `<lang>`: A [Language Code][Language Codes]
* `<test-id>`: A string of 16 characters in `0-9a-f` (hexadecimal)
* `<domain>`: A domain name to be tested (temporarily not supported)
The supported paths are the following:
* `/<lang>/result/<test-id>`: access the result page with results for the test
with `<test-id>` in the selected language, e.g.
`https://zonemaster.net/en/result/7e363fb606343397`.
* `/<lang>/faq`: access the FAQ in the selected language, e.g.
`https://zonemaster.net/en/faq`.
* `/api`: With JSON RPC call access the Zonemaster [Backend RPC-API], e.g.
`https://zonemaster.net/api`.
The GUI will redirect from `<path>` (e.g. `https://zonemaster.net/`) to
`<lang>/<path>` (`https://zonemaster.net/en/`) automatically using the
user browser language, if supported and configured by the installation.
Else it uses the default language defined in Zonemaster-GUI.
The following paths are temporarily not supported:
* `<lang>/run-test`: access the input form.
* `<lang>/run-test/<domain>`: populate the input form with `<domain>`.
## Language Codes
Zonemaster uses [ISO 639-1] two-letter language codes, in lower case. A default
installation of Zonemaster-GUI currently has support for the following language
codes and languages:
* `da` for Danish language
* `en` for English language, the default language in a default installation
* `es` for Spanish language
* `fi` for Finnish language
* `fr` for French language
* `nb` for Norwegian language
* `sl` for Slovenian language
* `sv` for Swedish language
A non-default installation may have support for only some of the languages or
maybe other languages.
[ISO 639-1]: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes
[Language Codes]: #language-codes
[Backend RPC-API]: ../backend/rpcapi-reference.md