feat: add full Zonemaster stack with Docker and Spanish UI
- Clone all 5 Zonemaster component repos (LDNS, Engine, CLI, Backend, GUI) - Dockerfile.backend: 8-stage multi-stage build LDNS→Engine→CLI→Backend - Dockerfile.gui: Astro static build served via nginx - docker-compose.yml: backend (internal) + frontend (port 5353) - nginx.conf: root redirects to /es/, /api/ proxied to backend - zonemaster-gui/config.ts: defaultLanguage set to 'es' (Spanish) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
10
zonemaster/docs/public/using/backend/README.md
Normal file
10
zonemaster/docs/public/using/backend/README.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# Using the Backend
|
||||
|
||||
* [Using Zonemaster-Backend JSON-RPC API](Using-Zonemaster-Backend-JSON-RPC-API.md)
|
||||
* [Using Zonemaster-Backend for batch testing](Using-Zonemaster-Backend-for-batch-testing.md)
|
||||
* [RPCAPI reference]
|
||||
* [Collecting metrics][Telemetry]
|
||||
* [Using Zonemaster-Backend Docker container](Using-Zonemaster-Backend-Docker.md)
|
||||
|
||||
[RPCAPI reference]: rpcapi-reference.md
|
||||
[Telemetry]: telemetry.md
|
||||
@@ -0,0 +1,127 @@
|
||||
# Using Zonemaster-Backend Docker container
|
||||
|
||||
## Table of contents
|
||||
|
||||
* [Introduction](#introduction)
|
||||
* [Invoking `zmtest` using Docker](#invoking-zmtest-using-docker)
|
||||
* [Invoking `zmb` using Docker](#invoking-zmb-using-docker)
|
||||
* [Invoking the command line tool using Docker]
|
||||
* [IPv6 support](#ipv6-support)
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
This Docker image lets you run a working instance of Zonemaster-Backend.
|
||||
It also contains a working version of zonemaster-cli which can be used
|
||||
as a substitute for [zonemaster-cli],
|
||||
how to use this version of cli is described at
|
||||
[the end of this document][Invoking the command line tool using Docker].
|
||||
|
||||
The container is configured to use an SQLite database and run
|
||||
all needed processes (rpcapi and testagent).
|
||||
This image is designed to be used by one single user and
|
||||
is therefore not suitable for production.
|
||||
|
||||
There is a limitation in Docker regarding IPv6.
|
||||
Unless IPv6 has been enabled in the Docker daemon, there is no support for IPv6.
|
||||
To avoid unnecessary errors,
|
||||
use the corresponding option below if IPv6 support is not available.
|
||||
See also the "IPv6 Support" section.
|
||||
|
||||
- zonemaster-cli: `--no-ipv6`
|
||||
- zmtest: `--noipv6`
|
||||
- zmb: `--ipv6 false`
|
||||
|
||||
|
||||
## Invoking `zmtest` using Docker
|
||||
|
||||
The most basic use of the `zonemaster-backend` command is to just test a domain, e.g.
|
||||
"zonemaster.net". To do so you first need to start `zonemaster/backend`
|
||||
container and expose the port 5000
|
||||
to be able to access the JSON RPC API.
|
||||
|
||||
```
|
||||
docker run --rm -p 5000:5000 --name zm -d zonemaster/backend full
|
||||
```
|
||||
You can stop the Docker container with `docker stop zm`.
|
||||
|
||||
Once the zonemaster/backend container is started you can interact with it on
|
||||
`localhost:5000` using JSON RPCAPI.
|
||||
|
||||
You can use `zmtest` embedded inside the backend image by using this command:
|
||||
```
|
||||
docker run -ti --rm --net host zonemaster/backend zmtest zonemaster.net
|
||||
```
|
||||
Run
|
||||
```sh
|
||||
docker run -ti --rm --net host zonemaster/backend zmtest zonemaster.net
|
||||
```
|
||||
to get usage for `zmtest`.
|
||||
|
||||
|
||||
## Invoking `zmb` using Docker
|
||||
|
||||
You can also use the `zmb` command to interact with the `zonemaster/backend` container.
|
||||
|
||||
```
|
||||
docker run -ti --rm --net host zonemaster/backend zmb start_domain_test --domain zonemaster.net
|
||||
docker run -ti --rm --net host zonemaster/backend zmb get_test_results --test-id ed38765834e45b6e --lang en
|
||||
```
|
||||
|
||||
When piping the output of `docker run` directly into `jq`,
|
||||
you might encounter display issues due to the way Docker handles output.
|
||||
These issues are caused by carriage return characters (^M) in the output.
|
||||
To filter out these characters,
|
||||
you should pipe `docker run`’s output into `tr -d '^M'` before piping it into `jq`.
|
||||
|
||||
```
|
||||
docker run -ti --rm --net host zonemaster/backend zmb get_test_results (...) |tr -d '^M' | jq
|
||||
```
|
||||
Run
|
||||
```sh
|
||||
docker run -ti --rm --net host zonemaster/backend zmb
|
||||
```
|
||||
or
|
||||
```
|
||||
docker run -ti --rm --net host zonemaster/backend zmb man
|
||||
```
|
||||
to get usage and man page, respectively, for `zmb`.
|
||||
|
||||
## Invoking the command line tool using Docker
|
||||
|
||||
`zonemaster/backend` also contains the CLI tool shipped in
|
||||
the [zonemaster-cli] container image.
|
||||
You can invoke zonemaster-cli in the Zonemaster-Backend Docker image
|
||||
by running the following command:
|
||||
|
||||
```
|
||||
docker run -ti --rm zonemaster/backend:local cli zonemaster.net
|
||||
```
|
||||
|
||||
|
||||
## IPv6 support
|
||||
|
||||
On a Linux system IPv6 support can be enabled by creating or updating
|
||||
`/etc/docker/daemon.json`. This is a minimal file that enables IPv6 support:
|
||||
|
||||
```json
|
||||
{
|
||||
"ipv6": true,
|
||||
"fixed-cidr-v6": "2001:db8:1::/64"
|
||||
}
|
||||
```
|
||||
|
||||
Restart the docker daemon:
|
||||
```sh
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
Also see the official Docker documentation "[Enable IPv6 support]".
|
||||
|
||||
|
||||
[Docker Image Creation]: https://github.com/zonemaster/zonemaster/blob/master/docs/internal/maintenance/ReleaseProcess-create-docker-image.md
|
||||
[Enable IPv6 support]: https://docs.docker.com/config/daemon/ipv6/
|
||||
[Get started]: https://www.docker.com/get-started/
|
||||
[IPv6 support]: #ipv6-support
|
||||
[Invoking the command line tool using Docker]: #invoking-the-command-line-tool-using-docker
|
||||
[Zonemaster-cli]: ../cli.md
|
||||
@@ -0,0 +1,311 @@
|
||||
# Using Zonemaster-Backend JSON-RPC API
|
||||
|
||||
## Table of contents
|
||||
|
||||
* [Introduction](#introduction)
|
||||
* [Check that the RPC API daemon is running and answering properly](#check-that-the-json-rpc-api-service-is-running-and-answering-properly)
|
||||
* [Run a test of the zonemaster.net zone using zmtest](#run-a-test-of-the-zonemasternet-zone-using-zmtest)
|
||||
* [Run a test of the zonemaster.net zone using zmb](#run-a-test-of-the-zonemasternet-zone-using-zmb)
|
||||
* [Enqueue a test](#enqueue-a-test)
|
||||
* [Check the progress](#check-the-progress)
|
||||
* [Fetch the results](#fetch-the-results)
|
||||
* [Find available languages](#find-available-languages)
|
||||
* [Find previous tests](#find-previous-tests)
|
||||
* [Other APIs](#other-apis)
|
||||
* [IDN domain names](#idn-domain-names)
|
||||
* [Using curl instead](#using-curl-instead)
|
||||
|
||||
## Introduction
|
||||
|
||||
This is a guide for getting started with the Zonemaster [JSON-RPC API] service
|
||||
(running as a daemon).
|
||||
|
||||
Note that this guide makes a number of assumptions about your setup:
|
||||
|
||||
* That it is a Unix-like environment.
|
||||
* That you have Zonemaster-Backend installed according to the
|
||||
[installation guide]
|
||||
* That Zonemaster-Backend is running. See [installation guide] for how to start
|
||||
Zonemaster-Backend.
|
||||
* If you have changed IP address (default 127.0.0.1) or port (default 5000) then
|
||||
you must specify that in the commands below. Here we assume default.
|
||||
|
||||
For this guide we will use the two Zonemaster-Backend tools `zmb` and `zmtest`.
|
||||
Both are installed when Zonemaster-Backend is installed. To get more information
|
||||
what parameters they expect, run `zmb -h` and `zmtest -h`.
|
||||
|
||||
With `zmtest` you can start a domain test and get the result directly. With `zmb`
|
||||
you can also do that, but in several steps, but on the other hand, `zmb` offers
|
||||
many more possibilities. Actually, `zmtest` uses `zmb` behind the scen.
|
||||
|
||||
The `zmb` tool uses the JSON-RPC API to interact with Zonemaster-Backend.
|
||||
Zonemaster-GUI also uses the same JSON-RPC API to start tests and fetch the
|
||||
test results.
|
||||
|
||||
## Check that the JSON-RPC API service is running and answering properly
|
||||
|
||||
```sh
|
||||
zmb version_info
|
||||
```
|
||||
|
||||
To get a more readable output, pipe the command through `jq`:
|
||||
|
||||
```sh
|
||||
zmb version_info | jq
|
||||
```
|
||||
|
||||
If there is no response from the JSON-RPC API service, go to the
|
||||
[installation guide] for how to start or restart it.
|
||||
|
||||
Below most commands will be piped through `jq`, but for processing you might want
|
||||
to have the JSON on one line, or you might want to look at the options for `jq`
|
||||
to extract fields (see `jq -h`).
|
||||
|
||||
## Run a test of the zonemaster.net zone using zmtest
|
||||
|
||||
To just run a test `zmtest` is the simple tool to use:
|
||||
|
||||
```sh
|
||||
zmtest zonemaster.net
|
||||
```
|
||||
|
||||
The tool will also display the result, which will be a large JSON object (here
|
||||
truncated).
|
||||
|
||||
```
|
||||
testid: 879d13569db70fde
|
||||
100% done
|
||||
{
|
||||
"id": 1,
|
||||
"jsonrpc": "2.0",
|
||||
"result": {
|
||||
"created_at": "2024-10-25T09:28:38Z",
|
||||
"hash_id": "879d13569db70fde",
|
||||
"params": {
|
||||
"domain": "zonemaster.net",
|
||||
"ds_info": [],
|
||||
"ipv4": true,
|
||||
"ipv6": true,
|
||||
"nameservers": [],
|
||||
"priority": 10,
|
||||
"profile": "default",
|
||||
"queue": 0
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"level": "INFO",
|
||||
"message": "Using version v6.0.0 of the Zonemaster engine.\n",
|
||||
"module": "System",
|
||||
"testcase": "Unspecified"
|
||||
},
|
||||
{
|
||||
"level": "INFO",
|
||||
"message": "The parent zone is \"net\" as returned from name servers \"a.gtld-servers.net/192.5.6.30; a.gtld-servers.net/2001:503:a83e::2:30; b.gtld-servers.net/192.33.14.30; b.gtld-servers.net/2001:503:231d::2:30; c.gtld-servers.net/192.26.92.30; c.gtld-servers.net/2001:503:83eb::30; d.gtld-servers.net/192.31.80.30; d.gtld-servers.net/2001:500:856e::30; e.gtld-servers.net/192.12.94.30; e.gtld-servers.net/2001:502:1ca1::30; f.gtld-servers.net/192.35.51.30; f.gtld-servers.net/2001:503:d414::30; g.gtld-servers.net/192.42.93.30; g.gtld-servers.net/2001:503:eea3::30; h.gtld-servers.net/192.54.112.30; h.gtld-servers.net/2001:502:8cc::30; i.gtld-servers.net/192.43.172.30; i.gtld-servers.net/2001:503:39c1::30; j.gtld-servers.net/192.48.79.30; j.gtld-servers.net/2001:502:7094::30; k.gtld-servers.net/192.52.178.30; k.gtld-servers.net/2001:503:d2d::30; l.gtld-servers.net/192.41.162.30; l.gtld-servers.net/2001:500:d937::30; m.gtld-servers.net/192.55.83.30; m.gtld-servers.net/2001:501:b1f9::30\".\n",
|
||||
"module": "Basic",
|
||||
"testcase": "Basic01"
|
||||
},
|
||||
{
|
||||
"level": "INFO",
|
||||
"message": "The zone \"zonemaster.net\" is found.\n",
|
||||
"module": "Basic",
|
||||
"testcase": "Basic01"
|
||||
},
|
||||
{
|
||||
"level": "INFO",
|
||||
"message": "Authoritative answer on SOA query for \"zonemaster.net\" is returned by name servers \"ns2.nic.fr/192.93.0.4; ns2.nic.fr/2001:660:3005:1::1:2; nsa.dnsnode.net/194.58.192.46; nsa.dnsnode.net/2a01:3f1:46::53; nsp.dnsnode.net/194.58.198.32; nsp.dnsnode.net/2a01:3f1:3032::53; nsu.dnsnode.net/185.42.137.98; nsu.dnsnode.net/2a01:3f0:400::32\".\n",
|
||||
"module": "Basic",
|
||||
"testcase": "Basic02"
|
||||
},
|
||||
(...)
|
||||
```
|
||||
|
||||
You will find the meaning of all fields in the outputs in
|
||||
[Zonemaster-Backend JSON-RPC API reference][JSON-RPC API].
|
||||
|
||||
## Run a test of the zonemaster.net zone using zmb
|
||||
|
||||
If we instead use `zmb` then this will be done in three steps (that happen
|
||||
behind the scen when using `zmtest`):
|
||||
|
||||
1. Enqueue test.
|
||||
2. Check if testing has completed (progress) - maybe several times.
|
||||
3. Fetch result (maybe several times with different translations).
|
||||
|
||||
### Enqueue a test
|
||||
|
||||
To enqueue a test of zonemaster.net we run
|
||||
`zmb start_domain_test --domain zonemaster.net | jq` and immediately get the
|
||||
response
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"result": "879d13569db70fde"
|
||||
}
|
||||
```
|
||||
The `results` field holds the test ID which we need for further steps. It
|
||||
always consists of 16 hexadecimal digits.
|
||||
|
||||
### Check the progress
|
||||
|
||||
To check the progress we run `zmb test_progress --test-id 879d13569db70fde | jq`
|
||||
with the test ID from enqueueing, and get the response
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"result": 8
|
||||
}
|
||||
```
|
||||
|
||||
Now the `results` field holds the progress of testing, an integer between 0 and
|
||||
100 to mean 0% to 100%. 0% means that testing has not started (maybe many tests
|
||||
in the queue) and 100% means that testing has completed.
|
||||
|
||||
We have to stay in the check step until the test has reached 100%. In our
|
||||
example above, it is 8%, so we run the command again and now we get the
|
||||
response
|
||||
|
||||
```json
|
||||
{
|
||||
"result": 100,
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
### Fetch the results
|
||||
|
||||
When the test has completed the test results can be fetched. The test ID is found
|
||||
above and then the language has to be chosen. Here `en` is used. More about
|
||||
languages below. Run
|
||||
`zmb get_test_results --test-id 879d13569db70fde --lang en | jq` and get a JSON
|
||||
structure with data (as with `zmtest`). Again, the JSON structure is long and is
|
||||
truncated here.
|
||||
|
||||
```
|
||||
{
|
||||
"id": 1,
|
||||
"result": {
|
||||
"results": [
|
||||
{
|
||||
"message": "Using version v6.0.0 of the Zonemaster engine.\n",
|
||||
"testcase": "Unspecified",
|
||||
"level": "INFO",
|
||||
"module": "System"
|
||||
},
|
||||
{
|
||||
"module": "Basic",
|
||||
"level": "INFO",
|
||||
"testcase": "Basic01",
|
||||
"message": "The parent zone is \"net\" as returned from name servers \"a.gtld-servers.net/192.5.6.30; a.gtld-servers.net/2001:503:a83e::2:30; b.gtld-servers.net/192.33.14.30; b.gtld-servers.net/2001:503:231d::2:30; c.gtld-servers.net/192.26.92.30; c.gtld-servers.net/2001:503:83eb::30; d.gtld-servers.net/192.31.80.30; d.gtld-servers.net/2001:500:856e::30; e.gtld-servers.net/192.12.94.30; e.gtld-servers.net/2001:502:1ca1::30; f.gtld-servers.net/192.35.51.30; f.gtld-servers.net/2001:503:d414::30; g.gtld-servers.net/192.42.93.30; g.gtld-servers.net/2001:503:eea3::30; h.gtld-servers.net/192.54.112.30; h.gtld-servers.net/2001:502:8cc::30; i.gtld-servers.net/192.43.172.30; i.gtld-servers.net/2001:503:39c1::30; j.gtld-servers.net/192.48.79.30; j.gtld-servers.net/2001:502:7094::30; k.gtld-servers.net/192.52.178.30; k.gtld-servers.net/2001:503:d2d::30; l.gtld-servers.net/192.41.162.30; l.gtld-servers.net/2001:500:d937::30; m.gtld-servers.net/192.55.83.30; m.gtld-servers.net/2001:501:b1f9::30\".\n"
|
||||
},
|
||||
{
|
||||
"message": "The zone \"zonemaster.net\" is found.\n",
|
||||
"module": "Basic",
|
||||
"testcase": "Basic01",
|
||||
"level": "INFO"
|
||||
},
|
||||
{
|
||||
"level": "INFO",
|
||||
"testcase": "Basic02",
|
||||
"module": "Basic",
|
||||
"message": "Authoritative answer on SOA query for \"zonemaster.net\" is returned by name servers \"ns2.nic.fr/192.93.0.4; ns2.nic.fr/2001:660:3005:1::1:2; nsa.dnsnode.net/194.58.192.46; nsa.dnsnode.net/2a01:3f1:46::53; nsp.dnsnode.net/194.58.198.32; nsp.dnsnode.net/2a01:3f1:3032::53; nsu.dnsnode.net/185.42.137.98; nsu.dnsnode.net/2a01:3f0:400::32\".\n"
|
||||
},
|
||||
(...)
|
||||
```
|
||||
|
||||
## Find available languages
|
||||
|
||||
The result can be presented in several languages, depending on installation. To
|
||||
find out what languages that are available, run `zmb get_language_tags | jq`.
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"result": [
|
||||
"da",
|
||||
"en",
|
||||
"es",
|
||||
"fi",
|
||||
"fr",
|
||||
"nb",
|
||||
"sl",
|
||||
"sv"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Find previous tests
|
||||
|
||||
It is possible to find previous tests of a specific domain (zone). Run
|
||||
`zmb get_test_history --domain zonemaster.net |jq` to get all tests of
|
||||
`zonemaster.net`. The output looks like the following:
|
||||
|
||||
```
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"result": [
|
||||
{
|
||||
"undelegated": false,
|
||||
"created_at": "2024-10-25T09:54:38Z",
|
||||
"id": "00669309ac7887c3",
|
||||
"overall_result": "ok"
|
||||
},
|
||||
{
|
||||
"overall_result": "ok",
|
||||
"id": "879d13569db70fde",
|
||||
"undelegated": false,
|
||||
"created_at": "2024-10-25T09:28:38Z"
|
||||
}
|
||||
],
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
Now you can get the results of any of the listed tests by running
|
||||
`get_test_results` with selected test ID and language as above.
|
||||
|
||||
## Other APIs
|
||||
|
||||
There are a few other APIs that can be used throught `zmb`, and can be found by
|
||||
`zmb -h` and in [Zonemaster-Backend JSON-RPC API reference][JSON-RPC API]. The APIs for batch
|
||||
testing are covered in [Using Zonemaster Backend for batch testing].
|
||||
|
||||
## IDN domain names
|
||||
|
||||
When enqueuing a test of an IDN domain name with `zmb start_domain_test` it is
|
||||
possible to provide the domain name as either A-label or U-label. E.g. both
|
||||
"räksmörgås.se" or "xn--rksmrgs-5wao1o.se" (the same domain name) work as well.
|
||||
|
||||
The same is also true if using Curl instead (see below).
|
||||
|
||||
## Using Curl instead
|
||||
|
||||
Instead of using `zmb` it is possible to run the commands by `curl`, which
|
||||
requires that the JSON structure is created, but can give greater flexibility.
|
||||
The method can also be copied into different programming languages when
|
||||
scripting.
|
||||
|
||||
The same enqueuing of the zonemaster.net zone as above will then be:
|
||||
|
||||
```sh
|
||||
curl -sS -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 2, "method": "start_domain_test", "params": {"domain": "zonemaster.net"}}' http://localhost:5000/ | jq .
|
||||
```
|
||||
|
||||
And fetching the result, when completed, will be:
|
||||
|
||||
```sh
|
||||
curl -sS -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 5, "method": "get_test_results", "params": {"id": "879d13569db70fde", "language": "en"}}' http://localhost:5000/ | jq .
|
||||
```
|
||||
|
||||
The format of the JSON structure for every method is found in the
|
||||
[Zonemaster-Backend JSON-RPC API reference][JSON-RPC API].
|
||||
|
||||
|
||||
[installation guide]: ../../installation/zonemaster-backend.md
|
||||
[JSON-RPC API]: rpcapi-reference.md
|
||||
[Using Zonemaster Backend for batch testing]: Using-Zonemaster-Backend-for-batch-testing.md
|
||||
@@ -0,0 +1,402 @@
|
||||
# Using Zonemaster-Backend for batch testing
|
||||
|
||||
## Table of contents
|
||||
|
||||
* [Introduction](#introduction)
|
||||
* [Combining GUI and batch](#combining-gui-and-batch)
|
||||
* [Configuration to run batches](#configuration-to-run-batches)
|
||||
* [Create batch user](#create-batch-user)
|
||||
* [Run a batch job](#run-a-batch-job)
|
||||
* [Start a batch job](#start-a-batch-job)
|
||||
* [Status of a batch job](#status-of-a-batch-job)
|
||||
* [Scale out](#scale-out)
|
||||
* [Enable global cache](#enable-global-cache)
|
||||
* [Increase the worker pool size](#increase-the-worker-pool-size)
|
||||
* [Add more Test Agent services](#add-more-test-agent-services)
|
||||
* [Add more test servers](#add-more-test-servers)
|
||||
* [Queues](#queues)
|
||||
* [Appendix: Add more JSON-RPC API services]
|
||||
|
||||
## Introduction
|
||||
|
||||
In [Using the Backend JSON-RPC API] it is shown how a test of a single domain name
|
||||
(zone) can be started using the JSON-RPC API to Zonemaster-Backend. It is also
|
||||
possible to start a batch of domain names at the same time. In principle such
|
||||
a batch is the same thing as running the `start_domain_test` individually for each
|
||||
domain name in the batch.
|
||||
|
||||
This document describes how batches of domain name tests can be run and what
|
||||
to think about when running batches, especially large ones.
|
||||
|
||||
Zonemaster-Backend uses a database to store enqueued tests and
|
||||
results from tests. It is possible to use SQLite as the database backend for
|
||||
small installations and test installations, but that is not appropriate for running
|
||||
real batches on. Therefore in this document it is assumed that Backend has been
|
||||
installed with [MariaDB/MySQL] or [PostgreSQL].
|
||||
|
||||
## Combining GUI and batch
|
||||
|
||||
Both GUI and batch uses Zonemaster-Backend that stores the data in the database.
|
||||
If the same database is used, then all tests run, regardless if they are
|
||||
started via GUI or a batch, will be available in the GUI interface
|
||||
using the `get_test_history` (see relevant section in
|
||||
[Using the Backend JSON-RPC API][Using#find-previous-tests]). Depending on the
|
||||
situation, running batches on the same Backend as GUI may be desirable or not
|
||||
recommended. If the results are expected to be available through GUI then the batches must be
|
||||
run on the same Backend. If the batches are run with a reduced set of test cases
|
||||
then it can be confusing since the results are not fully comparable with
|
||||
those from tests started from GUI. If the results from batch tests should not be
|
||||
mixed with tests from normal GUI tests, a separate installation of Backend is
|
||||
required, and that might not need any GUI at all.
|
||||
|
||||
Test prioritization is another aspect of mixing tests from batches and GUI on the same Backend. By default Backend will run
|
||||
tests from GUI (or rather by `start_domain_test`) before test from a batch. In
|
||||
this way interactive GUI users run tests do not have to wait for a
|
||||
batch to complete. A large batch might run for days.
|
||||
|
||||
## Configuration to run batches
|
||||
|
||||
To start a batch job, setting `enable_add_batch_job` in [backend_config.ini] must
|
||||
be enabled (*enabled* by default). Moreover a special user must also be available.
|
||||
To create that user setting `enable_add_api_user` must be enabled
|
||||
(*disabled* by default). (See below for how to create the user.) Note that this setting
|
||||
only needs to be enabled while creating the user(s), not when starting a batch job.
|
||||
|
||||
After changing these settings, the JSON-RPC API service (running as a daemon)
|
||||
must be restarted.
|
||||
|
||||
It is not recommended to enable `enable_add_api_user` for an JSON-RPC API service
|
||||
that can be used by untrusted users, e.g. via GUI. If the installation for the
|
||||
batch jobs is a dedicated installation, then it is not an issue. If it is used by
|
||||
a public GUI that could be closed while the batch user or users are created, then
|
||||
that could be an option. Creating the user only takes as long time as it takes to
|
||||
type the command in, i.e. seconds.
|
||||
|
||||
Another option is to set up an additional JSON-RPC API service using the same
|
||||
database. Zonemaster-Backend can have several parallel JSON-RPC API services with
|
||||
different configuration, see [Appendix: Add more JSON-RPC API services].
|
||||
|
||||
## Create batch user
|
||||
|
||||
The [Using the Backend JSON-RPC API] guide shows how `zmb` can be used for
|
||||
several JSON-RPC API methods. It can also be used for the creation of the batch
|
||||
user. It is here assumed that `enable_add_api_user` has been enabled or else the
|
||||
following will be included in the response to the `zmb add_api_user` command:
|
||||
```
|
||||
"message": "Procedure 'add_api_user' not found"
|
||||
```
|
||||
|
||||
Run `zmb add_api_user --username myuser --api-key mykey | jq` where `myuser` is
|
||||
a user name of your choice and `mykey` the password, which should probably be
|
||||
longer and more complex. The output is simply
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"result": 1,
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
Now `enable_add_api_user` can be disabled.
|
||||
|
||||
## Run a batch job
|
||||
|
||||
The `zmb` tool is used for that purpose. A batch job can be started with just a few
|
||||
domain names or a list of millions of domain names.
|
||||
|
||||
### Start a batch job
|
||||
|
||||
Batches are created using `add_batch_job`:
|
||||
|
||||
```
|
||||
$ zmb add_batch_job --username myuser --api-key mykey --domain zonemaster.net --domain zonemaster.se --domain zonemaster.fr | jq
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"result": 2,
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
Options that are mandatory are `--username`, `--api-key` and `--domain`. Option
|
||||
`--domain` is repeated for every domain name in the batch. The other options are
|
||||
optional. For all options see `zmb man`.
|
||||
|
||||
The batch ID is found in "result", which is "2" in this case.
|
||||
|
||||
Listing the domain names one by one is however cumbersome unless it is a very
|
||||
small batch as above. A better alternative for a larger batch is to create text
|
||||
file of the domain names to test, one domain name per line. Comments starting
|
||||
with `#` is permitted, which makes it possible to add meta information to the
|
||||
file.
|
||||
|
||||
```
|
||||
cat batchfile.txt
|
||||
# Example batch
|
||||
zonemaster.net
|
||||
zonemaster.se
|
||||
zonemaster.fr
|
||||
```
|
||||
|
||||
Now we can run the same batch as above, but from the file:
|
||||
|
||||
```
|
||||
$ zmb add_batch_job --username myuser --api-key mykey --file batchfile | jq
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"result": 3,
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
Starting the batch with the domain names one by one or in the file gives exactly
|
||||
the same result. Also in this case, IDN domain names can be entered both with
|
||||
A-label and U-label (UTF-8 is assumed).
|
||||
|
||||
### Status of a batch job
|
||||
|
||||
To check the status of a batch job the batch ID is required ("2" in the
|
||||
case above), using `batch_status`:
|
||||
|
||||
```
|
||||
$ zmb batch_status --batch-id 2 | jq
|
||||
{
|
||||
"id": 1,
|
||||
"result": {
|
||||
"finished_count": 0,
|
||||
"running_count": 2,
|
||||
"waiting_count": 1
|
||||
},
|
||||
"jsonrpc": "2.0"
|
||||
}
|
||||
```
|
||||
|
||||
Two tests are running, one is waiting to be started and none has been completed
|
||||
yet.
|
||||
|
||||
To get the test IDs of the the tests in each of the categories, add one or more
|
||||
of the "--lw" (list waiting), "--lr" (list running) or "--lf" (list finished)
|
||||
option to the previous call:
|
||||
|
||||
```
|
||||
$ zmb batch_status --batch-id 2 --lw --lr --lf| jq
|
||||
{
|
||||
"id": 1,
|
||||
"result": {
|
||||
"finished_count": 0,
|
||||
"running_count": 2,
|
||||
"waiting_count": 1,
|
||||
"waiting_tests": [
|
||||
"b26b874493ed4b57"
|
||||
],
|
||||
"running_tests": [
|
||||
"9af528070fa2551a",
|
||||
"708f82b5176261dc"
|
||||
]
|
||||
},
|
||||
"jsonrpc": "2.0"
|
||||
}
|
||||
```
|
||||
|
||||
There is no "finished_tests" since that category is empty.
|
||||
|
||||
After some time running the command again (with "--lf") gives:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"result": {
|
||||
"finished_count": 3,
|
||||
"running_count": 0,
|
||||
"waiting_count": 0,
|
||||
"finished_tests": [
|
||||
"9af528070fa2551a",
|
||||
"708f82b5176261dc",
|
||||
"b26b874493ed4b57"
|
||||
]
|
||||
},
|
||||
"jsonrpc": "2.0"
|
||||
}
|
||||
```
|
||||
|
||||
Or without "--lf":
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"result": {
|
||||
"finished_count": 3,
|
||||
"running_count": 0,
|
||||
"waiting_count": 0
|
||||
},
|
||||
"jsonrpc": "2.0"
|
||||
}
|
||||
```
|
||||
|
||||
Now all tests in the batch are completed. The test IDs in `finished_tests`
|
||||
can be used to get the results with `get_test_results`.
|
||||
|
||||
## Scale out
|
||||
|
||||
Running a batch can take a long time and if that is a problem then it is possible
|
||||
to increase the performance by:
|
||||
|
||||
* Enabling global cache
|
||||
* Increasing `number_of_processes_for_batch_testing`
|
||||
* Adding one or more Test Agent service
|
||||
* Adding one or more test servers
|
||||
|
||||
### Enable global cache
|
||||
|
||||
For each domain name in a batch job, a test run is executed.
|
||||
Each test run keeps its own cache to avoid sending the same query twice.
|
||||
However these caches are never shared between test runs.
|
||||
This may result in large numbers of redundant queries for large batches where
|
||||
the domain name directly (or indirectly) share data with other domain names.
|
||||
To share responses between test runs, you can enable the global cache feature
|
||||
which works like a second level cache.
|
||||
|
||||
For interactive tests via GUI the global cache is probably not desired. If a user
|
||||
changes the DNS configuration of a domain then the following tests results should
|
||||
not be based on old data.
|
||||
|
||||
To enable global cache a custom profile must be used. See the instructions in
|
||||
[Configuration: Global cache in Zonemaster-Engine][Global-cache].
|
||||
|
||||
The custom profile is assumed to be `/etc/zonemaster/profile-batch.json`
|
||||
(Linux) or `/etc/local/zonemaster/profile-batch.json` (FreeBSD).
|
||||
|
||||
The custom profile file, say `profile-batch.json`, must be added to the Backend
|
||||
configuration to be made available for the tests. In `backend_config.ini` look
|
||||
for the section `[PRIVATE PROFILES]`. Under that add the following line for
|
||||
Linux:
|
||||
|
||||
```
|
||||
batch=/etc/zonemaster/profile-batch.json
|
||||
```
|
||||
|
||||
For FreeBSD add:
|
||||
|
||||
```
|
||||
batch=/usr/local/etc/zonemaster/profile-batch.json
|
||||
```
|
||||
|
||||
After updating the Backend configuration, restart both JSON-RPC API service and
|
||||
Test Agent service (all daemons of both services if there are more than one of
|
||||
each).
|
||||
|
||||
Now you can start a batch -- using the same batch file as above -- with:
|
||||
|
||||
```
|
||||
$ zmb add_batch_job --profile batch --username myuser --api-key mykey --file batchfile | jq
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"result": 4,
|
||||
"id": 1
|
||||
}
|
||||
```
|
||||
|
||||
### Increase the worker pool size
|
||||
|
||||
A standard Backend installation has a single Test Agent service. Each Test Agent
|
||||
service maintains a pool of workers. Test Agents monitor the Backend database for
|
||||
tests and assigns them to workers.
|
||||
|
||||
Increasing the size of the worker pool can improve the testing performance. The
|
||||
improvement is limited by CPU, I/O, memory and the ability of the Test Agent
|
||||
service to keep its workers busy, so monitor these factors to find the best
|
||||
configuration.
|
||||
|
||||
To adjust the worker pool size, set `number_of_processes_for_batch_testing` in
|
||||
the [backend_config.ini] configuration file.
|
||||
|
||||
### Add more Test Agent services
|
||||
|
||||
If the increase of `number_of_processes_for_batch_testing` does not help anymore
|
||||
and the server has free resources (IO, CPU and RAM) then adding another Test
|
||||
Agent services (with its pool of workers) can help.
|
||||
|
||||
The default Test Agent service is started by the start script
|
||||
`/etc/systemd/system/zm-testagent.service` (Rocky Linux),
|
||||
`/etc/init.d/zm-testagent` (Debian/Ubuntu) or `/usr/local/etc/rc.d/zm_testagent`
|
||||
(FreeBSD). Create a copy so that the additional Test Agent service has a new
|
||||
name, new PID file and new log file. Also see the
|
||||
[Backend installation instructions]. Each Test Agent service will be its own
|
||||
daemon.
|
||||
|
||||
Start the new daemon and now a second Test Agent service is running.
|
||||
Additional Test Agent services can be added by repeating the steps above.
|
||||
|
||||
Unless the different Test Agent services should process different
|
||||
[queues](#queues) the same configuration file can be used.
|
||||
|
||||
### Add more test servers
|
||||
|
||||
Adding more servers is a generalization of adding more Test Agent daemons.
|
||||
|
||||
* Configure the database to listen on an external interface. The database can
|
||||
also be moved to a dedicated server.
|
||||
* Install Backend on the new Test Agent server, but disable JSON-RPC API service.
|
||||
* Update configuration to use the external database server.
|
||||
* Consider adding more Test Agent services.
|
||||
* If global cache is used, configure Redis to listen on an external interface.
|
||||
The Redis server can also be moved to a dedicated server.
|
||||
* Consider moving JSON-RPC API service to a dedicated server. If so, update the
|
||||
configuration to use the external database server.
|
||||
* If applicable, configure all parts to use external database server and external
|
||||
Redis server.
|
||||
|
||||
## Queues
|
||||
|
||||
Every enqueued test has a [queue tag][JSON-RPC API#queue]. In the
|
||||
[configuration][Config#lock_on_queue] file it is set which queue should be
|
||||
processed by the Test Agent service. Only tests with matching queue value
|
||||
(default 0) will be processed. Queue tag is set by the JSON-RPC API methods
|
||||
`start_domain_test` and `add_batch_job`, respectively (default 0 in both cases).
|
||||
|
||||
If desirable it is possible to create a batch with another queue value and then
|
||||
let a dedicated Test Agent service, possibly on a dedicated server, process the
|
||||
tests in the batch, while another Test Agent service processes tests from GUI.
|
||||
|
||||
It is important to note that if a test is tagged with a queue value that no
|
||||
Test Agent service uses it will never be processed.
|
||||
|
||||
In the end, all completed tests will end up in the same database and will be
|
||||
mixed when looking for completed tests for the same domain name
|
||||
(`get_test_history`).
|
||||
|
||||
## Appendix: Add more JSON-RPC API services
|
||||
|
||||
The JSON-RPC API service listens by default on 127.0.0.1 on port 5000. That is
|
||||
governed by the start script for the JSON-RPC API service which is
|
||||
`/etc/systemd/system/zm-rpcapi.service` (Rocky Linux), `/etc/init.d/zm-rpcapi`
|
||||
(Ubuntu and Debian) or `/usr/local/etc/rc.d/zm_rpcapi` (FreeBSD). Create a copy
|
||||
so that the additional service (daemon) has a new name, new PID file, new log
|
||||
file and listens to a new port. Also see the [Backend installation instructions].
|
||||
|
||||
Start the new daemon and now a second JSON-RPC API service is running.
|
||||
|
||||
To be useful, the second JSON-RPC API service should probably have a different
|
||||
configuration file too. The configuration file is found as
|
||||
`/etc/zonemaster/backend_config.ini` (Linux) or as
|
||||
`/usr/local/etc/zonemaster/backend_config.ini` (FreeBSD). The database settings
|
||||
must not be changed, or else the two JSON-RPC API services will not be part of
|
||||
the same Zonemaster Backend. Examples of settings that could be different between
|
||||
the two JSON-RPC API services are:
|
||||
|
||||
* enable_add_api_user
|
||||
* Available profiles
|
||||
|
||||
Restart the JSON-RPC API services to use the updated configuration files.
|
||||
|
||||
|
||||
[Appendix: Add more JSON-RPC API services]: #appendix-add-more-json-rpc-api-services
|
||||
[Backend installation instructions]: ../../installation/zonemaster-backend.md
|
||||
[Backend_config.ini]: ../../configuration/backend.md
|
||||
[Global-cache]: ../../configuration/global-cache.md
|
||||
[MariaDB/MySQL]: ../../installation/zonemaster-backend.md#7-installation-with-mariadb
|
||||
[PostgreSQL]: ../../installation/zonemaster-backend.md#8-installation-with-postgresql
|
||||
[Using the Backend JSON-RPC API]: Using-Zonemaster-Backend-JSON-RPC-API.md
|
||||
[Using#find-previous-tests]: Using-Zonemaster-Backend-JSON-RPC-API.md#find-previous-tests
|
||||
[JSON-RPC API#queue]: rpcapi-reference.md#queue
|
||||
[Config#lock_on_queue]: ../../configuration/backend.md#lock_on_queue
|
||||
1581
zonemaster/docs/public/using/backend/rpcapi-reference.md
Normal file
1581
zonemaster/docs/public/using/backend/rpcapi-reference.md
Normal file
File diff suppressed because it is too large
Load Diff
73
zonemaster/docs/public/using/backend/telemetry.md
Normal file
73
zonemaster/docs/public/using/backend/telemetry.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Telemetry
|
||||
|
||||
## Metrics
|
||||
|
||||
If enabled in the [Metrics section][metrics feature] in the configuration file,
|
||||
[Statsd][statsd] compatible metrics are available to use:
|
||||
|
||||
| Name | Type | Description |
|
||||
| ------------------------------------------------ | ------- | ----------- |
|
||||
| zonemaster.rpcapi.requests.\<METHOD>.\<STATUS> | Counter | Number of times the JSON RPC method \<METHOD> resulted in JSON RPC status \<STATUS>. The status is represented in string, possible values are: `RPC_PARSE_ERROR`, `RPC_INVALID_REQUEST`, `RPC_METHOD_NOT_FOUND`, `RPC_INVALID_PARAMS`, `RPC_INTERNAL_ERROR`. |
|
||||
| zonemaster.testagent.tests_started | Counter | Number of tests that have started. |
|
||||
| zonemaster.testagent.tests_completed | Counter | Number of tests that have been completed successfully. |
|
||||
| zonemaster.testagent.tests_died | Counter | Number of tests that have died. |
|
||||
| zonemaster.testagent.tests_duration_seconds | Timing | The duration of a test, emitted for each test. |
|
||||
| zonemaster.testagent.cleanup_duration_seconds | Timing | Time spent to kill timed out processes. |
|
||||
| zonemaster.testagent.fetchtests_duration_seconds | Timing | Time spent selecting the next text to run and processing unfinished tests. |
|
||||
| zonemaster.testagent.running_processes | Gauge | Number of running processes in a test agent. |
|
||||
| zonemaster.testagent.maximum_processes | Gauge | Maximum number of running processes in a test agent. |
|
||||
|
||||
|
||||
### Usage
|
||||
|
||||
Testing the metrics feature can be as easy as running a listening UDP server like
|
||||
|
||||
```sh
|
||||
ns -lup 8125
|
||||
```
|
||||
|
||||
This should be enough to see the metrics emitted by Zonemaster.
|
||||
|
||||
More complex setups are required for the metrics to be used in alerts and dashboards.
|
||||
StatsD metrics can be integrated to a number of metrics backend like Prometheus (using the [StatsD exporter]), InfluxDB (using Telegraf and the [StatsD plugin]), Graphite ([integration guide]) and others.
|
||||
|
||||
#### StatsD Exporter (Prometheus)
|
||||
|
||||
1. Download the binary (tar file) corresponding to your environment https://github.com/prometheus/statsd_exporter/releases
|
||||
2. Untar the file
|
||||
3. `cd` into the statsd_exporter directory
|
||||
4. Create a `statsd_mapping.yml` file with content as below
|
||||
```yml
|
||||
mappings:
|
||||
- match: zonemaster.rpcapi.requests.*.*
|
||||
name: zonemaster_rpcapi_requests_total
|
||||
labels:
|
||||
method: $1
|
||||
status: $2
|
||||
- match: zonemaster.testagent.tests_duration_seconds
|
||||
observer_type: histogram
|
||||
buckets: [ 1, 2.5, 5, 10, 15, 30, 45, 60, 75, 90, 105, 120, 150, 180]
|
||||
name: zonemaster_testagent_tests_duration_seconds
|
||||
- match: zonemaster.testagent.cleanup_duration_seconds
|
||||
observer_type: histogram
|
||||
buckets: [ 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2]
|
||||
name: zonemaster_testagent_cleanup_duration_seconds
|
||||
- match: zonemaster.testagent.fetchtests_duration_seconds
|
||||
observer_type: histogram
|
||||
buckets: [ 0.001, 0.0025, 0.005, 0.0075, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 3, 5, 10]
|
||||
name: zonemaster_testagent_fetchtests_duration_seconds
|
||||
```
|
||||
5. Run it like
|
||||
```
|
||||
./statsd_exporter --statsd.mapping-config=./statsd_mapping.yml --statsd.listen-udp=:8125 --statsd.listen-tcp=:8125
|
||||
```
|
||||
6. Run the following to see Zonemaster metrics:
|
||||
```
|
||||
curl localhost:9102/metrics | grep zonemaster
|
||||
```
|
||||
|
||||
[metrics feature]: ../../installation/zonemaster-backend.md#101-metrics
|
||||
[statsd]: https://github.com/statsd/statsd
|
||||
[StatsD exporter]: https://github.com/prometheus/statsd_exporter
|
||||
[StatsD plugin]: https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd
|
||||
[integration guide]: https://github.com/statsd/statsd/blob/master/docs/graphite.md
|
||||
Reference in New Issue
Block a user