- **Backend / API**: serves the UI, exposes the REST API, holds the scheduler, runs the route cycles, manages config.
- **Scheduler**: in-process (APScheduler). **One cron trigger per enabled route**, plus a daily history-prune job (armed independently, so history is trimmed even while every route is paused). Re-armed whenever config changes. GC and verify have no triggers of their own — they are options of a route, or a manual action on a box.
- **Run queue + power lease**: one run is ever in flight; the rest wait in a FIFO queue. Each PBS a run needs is held under a refcounted lease that wakes it on first acquire and powers it off on last release. See below.
-`pbs` — PBS API client (datastore status, GC, verify, remotes and sync jobs). TLS-pinned per device to the fingerprint stored at setup (rejects a changed cert).
A **route** is one scheduled flow of backup data between devices: *sources → target*, on its own schedule, with its own retention and options. Devices live in `pves[]` and `pbss[]` and are referenced by id; the id is also the name the UI shows.
The **kind** follows from which devices the route names:
| Kind | Sources | Target | What runs |
|---|---|---|---|
| `backup` | one or more PVEs (`sources[].pve`, with a per-source guest selection) | a PBS | `vzdump` on each source, one task per cluster node |
| `sync` | one PBS (`source_pbs`) | another PBS | a PBS remote + sync job, `pull` or `push` |
| `external` | none | a PBS | nothing of its own — it watches the tasks PVE/PBS start on their own schedules |
| `verify` | none | a PBS | a verification pass over the target's snapshots |
Guests are selected **per source** (`sources[].guests`), because vmids collide between PVEs. Mode is `all` or `include`; in `include` mode a newly created guest is *not* picked up automatically.
The per-guest last-backup cache is filled by listing the target datastore's snapshots, and that listing covers the datastore's **root namespace only** — no `ns` parameter is sent, and a PBS namespace is configured on the Proxmox storage entry, where Joulenap never sees it. A namespaced setup therefore backs up, prunes and collects garbage correctly while every one of its guests reads *never backed up*. PBS groups are `ct/<vmid>` / `vm/<vmid>` with no record of which host wrote them, so two PVEs sharing a datastore and a vmid also share a group and prune each other's snapshots — use non-overlapping vmid ranges across hosts.
A route's `schedule` is a time plus seven weekday flags. `schedule.cron` is an escape hatch for anything richer (day-of-month, steps, ranges) and **wins over `time`/`days`** when set; the UI then shows it read-only.
`options` carries the per-route knobs: `mode` / `bwlimit` / `min_free_percent` (backup only, they are vzdump's), `gc` and `verify_after` (run on the target after the data lands), and `reverify_days` for a verify route. `retention` is vzdump's `prune-backups`, per route.
Cross-references are validated at load: ids are unique, every referenced device exists, a **backup** route's target must appear in every source PVE's `storages` map, and an **external** route's target must have `managed_power: true`.
## Queue and power lease
**One run at a time.** A route firing while another run is in flight joins a FIFO queue rather than being dropped; the same route already queued or running is refused (`AlreadyQueued`). The queue key is the route id, or `pbs:<id>:gc` / `pbs:<id>:verify` for an ad-hoc maintenance run.
**Each PBS a run needs is leased.** The lease is refcounted:
- the first holder wakes the box (WoL, then poll until it answers) or finds it already awake — `wol_retries + 1` attempts, each waiting up to `wait_timeout`;
- every holder releases when its run is done, and only the **last** release powers the box off;
- a **sync** route takes two leases (source and target) and releases them independently.
The release records a `poweroff` step whose detail says what actually happened:
| Outcome | Meaning |
|---|---|
| `powered off` | last holder, nothing queued needs it, run succeeded → SSH poweroff |
| `left on: still needed by another run` | a queued run needs this box |
| `left on: Joulenap does not manage this box's power` | `managed_power: false` |
| `left powered on` | the run failed (left up for inspection), the user asked to keep it on, or the box was busy |
Only the last of those is worth a warning; the other three are the correct outcome and are recorded as *skipped*, not failed.
`managed_power: false` describes an always-on PBS. The lease is the single place that knows: acquiring degrades to a reachability check and releasing does nothing.
**The lease is keyed per device, not per machine.** A backup server serving two datastores is two devices, so each holds its own lease and neither knows about the other: a run that finishes with the first can power the machine off while a queued run on the second still wants it, and that run then wakes it again. Runs are serialised by the single-run lock, so this costs one extra sleep/wake cycle rather than correctness — but on one physical box, prefer a single datastore, or expect the extra cycle.
- **backup** — `[wake + wait, per lease] → precheck (only when min_free_percent > 0; aborts rather than filling the datastore) → vzdump per source PVE, one task per cluster node, with the route's retention/mode/bwlimit → [GC if enabled] → [verify if enabled] → record → [power-off, per lease]`. A source that fails is recorded and the loop continues; the run ends `failure` naming the sources that broke.
- **sync** — wake both boxes, then on the executing box (`pull` → the target fetches, `push` → the source sends): delete any stale sync job, ensure the remote, ensure the sync job, run it and wait. Then GC/verify on the target. **Order matters**: PBS refuses to delete a remote a job still references, so the job goes first. A task ending in warnings is reported naming the direction, both boxes and the first WARN/ERROR line of its log.
- **external** — wake, then watch: poll up to `external.first_task_wait` for the *first* task to appear, then wait for `external.idle_wait` seconds of continuous silence, restarting that countdown whenever a new task shows up (so a chained backup → prune → GC → sync is not cut short). Joulenap starts nothing itself. A wake where no task ever appears still powers the box back off and says so, so a misfiring PVE/PBS schedule is noticed instead of silently missed.
- **verify** — wake, verify (`reverify_days` keeps it incremental; `0` re-verifies everything), power off.
- **ad-hoc GC / verify on a box** — the homepage's per-PBS buttons. The same steps, so the history reads identically; only the route column is empty.
All steps are recorded in the DB and exposed on the run itself via `/api/runs/{id}` (the activity log — one line per event, not per step — is `/api/logs`); while a run is in progress the raw PVE/PBS task output is tailed into `/api/tasklog` for the UI's task-log panel.
0.9 modelled exactly one PVE, one PBS and one backup job (`pve:` / `pbs:` / `backup:`). On the first start after the upgrade an existing `config.yaml` is converted in place:
-`pve:` → one `pves[]` entry, its single `storage_id` becoming `storages: {<pbs id>: <storage>}`;
-`pbs:` → one `pbss[]` entry, with `managed_power` set from whether a MAC was configured, and 0.9's global external-mode timeouts moved onto the device;
- the backup job → one route named **Backup** (kind `external` if 0.9's external-schedules mode was on, otherwise `backup`), carrying the schedule, guest selection, retention, and `maintenance.gc` / `maintenance.verify.after_backup` as route options;
- a scheduled verification → a second route named **Verify**;
- a plain "at HH:MM on these weekdays" cron becomes `time` + `days`; anything richer is kept verbatim as `schedule.cron`.
Two rules keep it from ever bricking a boot: the original is copied to **`config.yaml.pre-overhaul.bak`** first (an existing `.bak` is never overwritten, and the copy is chmod'd `0600` because it holds every secret), and the converted config is validated before it is adopted — if it fails, the file on disk is left untouched, but nothing in 1.0 reads the 0.9 sections, so the app boots with **no devices and no routes and nothing scheduled**. The reason is surfaced as `config_error` on `/api/status` precisely so the UI says why instead of looking like a fresh install; the user can rewrite `config.yaml` from the Advanced tab while running empty.
One conversion is lossy and deliberately widens rather than narrows: **0.9's `exclude` guest mode no longer exists**. Inverting the list would need a live guest list that is not available at load time, so the route becomes `all` and a warning is logged — a route that backs up more than before, never less. Narrow it from the UI if that is not what you want.
Everything is served under `/api`. Auth is a signed **session cookie** started by `/api/login`; every endpoint requires it except `/api/health`, `/api/auth/status`, `/api/auth/setup`, `/api/login` and `/api/logout` (which only clears a cookie, idempotently) — plus `/api/dashboard` and `/metrics`, which are deliberately outside the session and authenticated by the read-only API key instead.
| GET | `/api/dashboard` | flat, read-only status for external dashboards — **API-key auth** (`X-API-Key` header or `?key=`), not the session cookie. Payload below; snippets in [`INTEGRATIONS.md`](INTEGRATIONS.md) |
| GET | `/metrics` | Prometheus exposition for Grafana — same API key. The **one route outside `/api`**, because `/metrics` is Prometheus's default `metrics_path` |
| GET | `/api/devices/pves/{pve_id}/storages` | this PVE's PBS-backed storages as it reports them, nothing written (502 on a connector failure) |
| POST | `/api/devices/pves/{pve_id}/storages` | re-read them and relink to registered devices, replacing the map (502 on a connector failure; 422 if the result would orphan a route) |
| POST | `/api/devices/pbss/{pbs_id}/power` | `{action: "wake" \| "poweroff"}` |
| POST | `/api/devices/pbss/{pbs_id}/gc` | queue an ad-hoc GC on this box (optional `{keep_on}`) |
| POST | `/api/devices/pbss/{pbs_id}/verify` | queue an ad-hoc verification on this box |
**Runs, history & logs**
| Method | Path | Purpose |
|---|---|---|
| POST | `/api/runs/{run_id}/stop` | stop the run in flight (optional `{power_off}`); also stops the PVE/PBS task behind it. 202 = accepted, not finished — cancellation is cooperative. 409 if it is not the run in flight |
| GET | `/api/runs?limit=&route=` | run history (summaries), optionally filtered to one route |
**Setup wizard** — all stateless: they return discovered values for the frontend to assemble and save with `POST /api/devices/{kind}`. Only `ssh/keygen` and `ssh/trust` write to disk (the shared keypair, and the confirmed host key in `data/known_hosts`).
**Changed in 1.0.0**: the flat single-PBS fields became these two lists, one entry per route and one per PBS. There is no longer a single "next run" or "datastore" to report, so a widget built on 0.9's shape picks a list entry (`.routes[0].next_run`) instead. The same applies to `/metrics`, whose series are now labelled by `route=`, `pbs=` and `datastore=`.
Per device — each PVE and each PBS gets its own token.
- **PVE token**: `VM.Audit` (list guests) + `VM.Backup` + `Datastore.Audit` + `Datastore.AllocateSpace`**and `Datastore.Allocate`** on the PBS-backed storage (the last is required for vzdump's retention/prune, which deletes old backups). Root-mode setup creates a `Joulenap` role with exactly these privileges (`connectors/provision.py`).
- **PBS token**: `DatastoreAdmin` on the datastore (status, GC, verify) plus `Audit` on `/system` (read-only node CPU/RAM/network for the dashboard). PBS has no API to create custom roles, so root-mode setup grants these built-ins scoped by path. The token is named **`joulenap-<datastore>`**: a device is a *(host, datastore)* pair, so one machine can hold two, and a shared name would mean provisioning the second deleted and recreated the first one's token. Deleting a token also drops its ACL entries, so that would have left the first device unable to connect *and* unable to be repaired by re-entering a secret.
- **PBS token, additionally, for sync routes**: `RemoteAdmin`**and**`RemoteSyncPushOperator` on `/remote`, so Joulenap can create the remote and the sync job. PBS refuses ACL writes from a token, so these can only be granted from a root login — the wizard does it while it still holds the root ticket, and a box set up before 1.0 gets them from the device editor's **Grant sync permissions** action (`POST /api/wizard/pbs/grant-sync`), which asks for root once and stores nothing. See [`CONFIG-WIZARD.md`](CONFIG-WIZARD.md#sync-routes-need-one-extra-grant).
