joulenap/docs/ARCHITECTURE.md

# Joulenap — Architecture & API


## Goals

- Run scheduled Proxmox VE backups to **backup servers that are normally powered off**: wake → wait → back up → prune → (GC) → power off → notify.
- Support **any number of PVE and PBS boxes**, wired together by explicit **routes**, including PBS→PBS off-site sync.
- Be **config-driven** and distributable (Docker image / LXC), nothing hard-coded.
- Modify **nothing** on the Proxmox host: Joulenap owns its own scheduler and acts via APIs + one SSH command.


## Components

- **Web UI** (frontend): single-page app. Talks to the backend over the REST API below.
- **Backend / API**: serves the UI, exposes the REST API, holds the scheduler, runs the route cycles, manages config.
- **Scheduler**: in-process (APScheduler). **One cron trigger per enabled route**, plus a daily history-prune job and a one-minute liveness heartbeat (both armed independently, so history is trimmed and liveness recorded even while every route is paused). Re-armed whenever config changes. GC and verify have no triggers of their own — they are options of a route, or a manual action on a box.
- **Startup catch-up**: the jobstore is in memory, so a fire due while the container was stopped is simply lost. At startup each armed route is checked for a slot that came due *while the process was not running*, and one is reported as a missed run (logged, and notified under `on_failure` — never auto-run: a restart must not kick off a heavy PBS-waking backup). "Not running" is a fact, not an inference: the heartbeat touches `data/.heartbeat` every minute and its mtime bounds the window, so a schedule edited to earlier in the day — or a route disabled and re-enabled, or the kill-switch — cannot be mistaken for downtime. No heartbeat on record (first boot, unwritable data dir) reports nothing.
- **Run queue + power lease**: one run is ever in flight; the rest wait in a FIFO queue. Each PBS a run needs is held under a refcounted lease that wakes it on first acquire and powers it off on last release. See below.
- **Connectors**:
  - `pve` — PVE API client (list guests, trigger `vzdump`, read task status).
  - `pbs` — PBS API client (datastore status, GC, verify, remotes and sync jobs). TLS-pinned per device to the fingerprint stored at setup (rejects a changed cert).
  - `wol` — sends the Wake-on-LAN magic packet on the LAN.
  - `power` — SSH to a PBS for `poweroff`, verified against `data/known_hosts` (host key confirmed in the wizard).
  - `notify` — Apprise / Telegram / ntfy / Discord / email senders.
  - `update` — asks GitHub once a day whether a newer release exists (opt-in via `app.update_check`; no outbound call when off).
- **Store**: `config.yaml` for settings; a small SQLite DB (`data/`) for run history and logs.


## The route model

A **route** is one scheduled flow of backup data between devices: *sources → target*, on its own schedule, with its own retention and options. Devices live in `pves[]` and `pbss[]` and are referenced by id; the id is also the name the UI shows.

The **kind** follows from which devices the route names:

| Kind | Sources | Target | What runs |
|---|---|---|---|
| `backup` | one or more PVEs (`sources[].pve`, with a per-source guest selection) | a PBS | `vzdump` on each source, one task per cluster node |
| `sync` | one PBS (`source_pbs`) | another PBS | a PBS remote + sync job, `pull` or `push` |
| `external` | none | a PBS | nothing of its own — it watches the tasks PVE/PBS start on their own schedules |
| `verify` | none | a PBS | a verification pass over the target's snapshots |

Guests are selected **per source** (`sources[].guests`), because vmids collide between PVEs. Mode is `all` or `include`; in `include` mode a newly created guest is *not* picked up automatically.

The per-guest last-backup cache is filled by listing the target datastore's snapshots, and that listing covers the datastore's **root namespace only** — no `ns` parameter is sent, and a PBS namespace is configured on the Proxmox storage entry, where Joulenap never sees it. A namespaced setup therefore backs up, prunes and collects garbage correctly while every one of its guests reads *never backed up*. PBS groups are `ct/<vmid>` / `vm/<vmid>` with no record of which host wrote them, so two PVEs sharing a datastore and a vmid also share a group and prune each other's snapshots — use non-overlapping vmid ranges across hosts.

A route's `schedule` is a time plus seven weekday flags. `schedule.cron` is an escape hatch for anything richer (day-of-month, steps, ranges) and **wins over `time`/`days`** when set; the UI then shows it read-only.

`options` carries the per-route knobs: `mode` / `bwlimit` / `min_free_percent` (backup only, they are vzdump's), `gc` and `verify_after` (run on the target after the data lands), and `reverify_days` for a verify route. `retention` is vzdump's `prune-backups`, per route.

Cross-references are validated at load: ids are unique, every referenced device exists, a **backup** route's target must appear in every source PVE's `storages` map, and an **external** route's target must have `managed_power: true`.


## Queue and power lease

**One run at a time.** A route firing while another run is in flight joins a FIFO queue rather than being dropped; the same route already queued or running is refused (`AlreadyQueued`). The queue key is the route id, or `pbs:<id>:gc` / `pbs:<id>:verify` for an ad-hoc maintenance run.

**Each machine a run needs is leased.** The lease is refcounted, and keyed on the device's **host** rather than its id — power is physical, and an SSH poweroff takes down every PBS instance on the box, so two datastores on one machine share one lease and one power decision:

- the first holder wakes the box (WoL, then poll until it answers) or finds it already awake — `wol_retries + 1` attempts, each waiting up to `wait_timeout`;
- every holder releases when its run is done, and only the **last** release powers the box off;
- a **sync** route between two machines takes two leases and releases them independently; between two datastores of one machine it takes one, so the timeline shows a single wake and a single power-off.

The release records a `poweroff` step whose detail says what actually happened:

| Outcome | Meaning |
|---|---|
| `powered off` | last holder, nothing queued needs it, run succeeded → SSH poweroff |
| `left on: still needed by another run` | a queued run needs this box |
| `left on: Joulenap does not manage this box's power` | `managed_power: false` |
| `left powered on` | the run failed (left up for inspection), the user asked to keep it on, or the box was busy |

Only the last of those is worth a warning; the other three are the correct outcome and are recorded as *skipped*, not failed.

`managed_power: false` describes an always-on PBS. The lease is the single place that knows: acquiring degrades to a reachability check and releasing does nothing.

Because the key is the machine, a run holding one datastore of a box reports every device on that box as busy — which is what disables the ⏻ button on its siblings, since an SSH power-off would take the running server down with them.


## What each kind does

- **backup** — `[wake + wait, per lease] → precheck (only when min_free_percent > 0; aborts rather than filling the datastore) → vzdump per source PVE, one task per cluster node, with the route's retention/mode/bwlimit → [GC if enabled] → [verify if enabled] → record → [power-off, per lease]`. A source that fails is recorded and the loop continues; the run ends `failure` naming the sources that broke.
- **sync** — wake both boxes, then on the executing box (`pull` → the target fetches, `push` → the source sends): delete any stale sync job, ensure the remote, ensure the sync job, run it and wait. Then GC/verify on the target. **Order matters**: PBS refuses to delete a remote a job still references, so the job goes first. A task ending in warnings is reported naming the direction, both boxes and the first WARN/ERROR line of its log.
- **external** — wake, then watch: poll up to `external.first_task_wait` for the *first* task to appear, then wait for `external.idle_wait` seconds of continuous silence, restarting that countdown whenever a new task shows up (so a chained backup → prune → GC → sync is not cut short). Joulenap starts nothing itself. A wake where no task ever appears still powers the box back off and says so, so a misfiring PVE/PBS schedule is noticed instead of silently missed.
- **verify** — wake, verify (`reverify_days` keeps it incremental; `0` re-verifies everything), power off.
- **ad-hoc GC / verify on a box** — the homepage's per-PBS buttons. The same steps, so the history reads identically; only the route column is empty.

All steps are recorded in the DB and exposed on the run itself via `/api/runs/{id}` (the activity log — one line per event, not per step — is `/api/logs`); while a run is in progress the raw PVE/PBS task output is tailed into `/api/tasklog` for the UI's task-log panel.


## Upgrading from 0.9 (config migration)

0.9 modelled exactly one PVE, one PBS and one backup job (`pve:` / `pbs:` / `backup:`). On the first start after the upgrade an existing `config.yaml` is converted in place:

- `pve:` → one `pves[]` entry, its single `storage_id` becoming `storages: {<pbs id>: <storage>}`;
- `pbs:` → one `pbss[]` entry, with `managed_power` set from whether a MAC was configured, and 0.9's global external-mode timeouts moved onto the device;
- the backup job → one route named **Backup** (kind `external` if 0.9's external-schedules mode was on, otherwise `backup`), carrying the schedule, guest selection, retention, and `maintenance.gc` / `maintenance.verify.after_backup` as route options;
- a scheduled verification → a second route named **Verify**;
- a plain "at HH:MM on these weekdays" cron becomes `time` + `days`; anything richer is kept verbatim as `schedule.cron`.

Two rules keep it from ever bricking a boot: the original is copied to **`config.yaml.pre-overhaul.bak`** first (an existing `.bak` is never overwritten, and the copy is chmod'd `0600` because it holds every secret), and the converted config is validated before it is adopted — if it fails, the file on disk is left untouched, but nothing in 1.0 reads the 0.9 sections, so the app boots with **no devices and no routes and nothing scheduled**. The reason is surfaced as `config_error` on `/api/status` precisely so the UI says why instead of looking like a fresh install; the user can rewrite `config.yaml` from the Advanced tab while running empty.

One conversion is lossy and deliberately widens rather than narrows: **0.9's `exclude` guest mode no longer exists**. Inverting the list would need a live guest list that is not available at load time, so the route becomes `all` and a warning is logged — a route that backs up more than before, never less. Narrow it from the UI if that is not what you want.


## REST API

Everything is served under `/api`. Auth is a signed **session cookie** started by `/api/login`; every endpoint requires it except `/api/health`, `/api/auth/status`, `/api/auth/setup`, `/api/login` and `/api/logout` (which only clears a cookie, idempotently) — plus `/api/dashboard` and `/metrics`, which are deliberately outside the session and authenticated by the read-only API key instead.

**Health & meta**

| Method | Path | Purpose |
|---|---|---|
| GET | `/api/health` | version + liveness (used by the Docker healthcheck) |
| GET | `/api/update` | running version, plus the latest GitHub release when `app.update_check` is on (cached 24h; no outbound call when off) |
| GET | `/api/dashboard` | flat, read-only status for external dashboards — **API-key auth** (`X-API-Key` header or `?key=`), not the session cookie. Payload below; snippets in [`INTEGRATIONS.md`](INTEGRATIONS.md) |
| GET | `/metrics` | Prometheus exposition for Grafana — same API key. The **one route outside `/api`**, because `/metrics` is Prometheus's default `metrics_path` |

**Auth & account**

| Method | Path | Purpose |
|---|---|---|
| GET | `/api/auth/status` | whether first-run setup is still needed / already signed in |
| POST | `/api/auth/setup` | first run: create the admin account |
| POST | `/api/login` | authenticate, start session |
| POST | `/api/logout` | end session |
| GET | `/api/auth/me` | current user |
| PUT | `/api/account` | change username / password (requires `current_password`) |

**Status & config**

| Method | Path | Purpose |
|---|---|---|
| GET | `/api/status` | the homepage poll: `state`, `scheduler_enabled`, the `running` run, the `queued[]` list, every route's `next_runs[]`, per-device `pves[]` / `pbss[]` (online, lease holders, datastore, load), `last_run`, and `config_error` |
| GET | `/api/config` | current config (secrets redacted) |
| PUT | `/api/config` | validate + save config, re-arm the scheduler |
| GET | `/api/config/yaml` | the redacted config serialised as YAML, for the Advanced tab's editor |
| PUT | `/api/config/yaml` | apply an edited YAML document (same validation and merge as `PUT /api/config`) |
| POST | `/api/config/api-key` | generate/rotate the dashboard-integration API key (returned once) |
| DELETE | `/api/config/api-key` | clear the key, disabling `/api/dashboard` and `/metrics` |
| GET | `/api/guests?pve=` | one PVE's CTs/VMs (id, name, type, node) with each guest's cached last backup and the PBSs holding it |
| POST | `/api/scheduler/toggle` | the global kill-switch across every route; returns the re-armed `next_runs` |

**Routes**

| Method | Path | Purpose |
|---|---|---|
| GET | `/api/routes` | every route |
| POST | `/api/routes` | create one (409 on a duplicate id) |
| PUT | `/api/routes/{route_id}` | replace one |
| DELETE | `/api/routes/{route_id}` | delete one. The snapshots on the PBS and the run history survive |
| POST | `/api/routes/{route_id}/run` | run it now — optional `{keep_on}` to leave the boxes awake. 202 with how many runs are ahead of it |

**Devices**

| Method | Path | Purpose |
|---|---|---|
| GET | `/api/devices` | `{pves, pbss}`, secrets redacted |
| POST | `/api/devices/{kind}` | create a `pves` / `pbss` entry (409 on a duplicate id) |
| PUT | `/api/devices/{kind}/{device_id}` | update one |
| DELETE | `/api/devices/{kind}/{device_id}` | delete one — **409 naming the routes** still using it |
| POST | `/api/devices/{kind}/{device_id}/test` | live connection test (502 with the reason on failure) |
| GET | `/api/devices/pves/{pve_id}/storages` | this PVE's PBS-backed storages as it reports them, nothing written (502 on a connector failure) |
| POST | `/api/devices/pves/{pve_id}/storages` | re-read them and relink to registered devices, replacing the map (502 on a connector failure; 422 if the result would orphan a route) |
| POST | `/api/devices/pbss/{pbs_id}/power` | `{action: "wake" \| "poweroff"}` |
| POST | `/api/devices/pbss/{pbs_id}/gc` | queue an ad-hoc GC on this box (optional `{keep_on}`) |
| POST | `/api/devices/pbss/{pbs_id}/verify` | queue an ad-hoc verification on this box |

**Runs, history & logs**

| Method | Path | Purpose |
|---|---|---|
| POST | `/api/runs/{run_id}/stop` | stop the run in flight (optional `{power_off}`); also stops the PVE/PBS task behind it. 202 = accepted, not finished — cancellation is cooperative. 409 if it is not the run in flight |
| GET | `/api/runs?limit=&route=` | run history (summaries), optionally filtered to one route |
| GET | `/api/runs/{id}` | one run with its steps + logs |
| GET | `/api/logs?limit=` | recent activity-log lines |
| GET | `/api/tasklog?after=&run=` | PVE/PBS task output — the live tail, or one past run's by id |
| POST | `/api/notify/test` | send a test notification; always 200, with a per-channel outcome |

**Setup wizard** — all stateless: they return discovered values for the frontend to assemble and save with `POST /api/devices/{kind}`. Only `ssh/keygen` and `ssh/trust` write to disk (the shared keypair, and the confirmed host key in `data/known_hosts`).

| Method | Path | Purpose |
|---|---|---|
| POST | `/api/wizard/pve/connect` | connect to a PVE, list its nodes + PBS-backed storages (root mode also mints a scoped token) |
| POST | `/api/wizard/storage/derive` | derive PBS host/port/datastore/fingerprint from one storage |
| POST | `/api/wizard/pbs/check` | reach a PBS, read its fingerprint |
| POST | `/api/wizard/pbs/provision` | root mode: auto-create a scoped PBS token |
| POST | `/api/wizard/pbs/grant-sync` | root mode: add the `/remote` roles a sync route needs to an existing token |
| GET | `/api/wizard/interfaces` | local NICs, to pick the WoL broadcast interface |
| POST | `/api/wizard/wol/detect-mac` | detect a PBS MAC by connecting to it, then reading ARP |
| POST | `/api/wizard/wol/test` | send a test magic packet before the device exists |
| POST | `/api/wizard/ssh/keygen` | **get-or-create** the poweroff SSH keypair |
| POST | `/api/wizard/ssh/hostkey` | scan a PBS SSH host key + fingerprint (to confirm before a root password is sent) |
| POST | `/api/wizard/ssh/trust` | persist the user-confirmed host key to `data/known_hosts` |
| POST | `/api/wizard/ssh/install` | root mode: install the public key on the PBS over SSH |


### `GET /api/dashboard` payload

A deliberately separate, additive-only contract for third-party widgets, with machine-style enum values that are never localized.

```json
{
  "state": "idle | running | paused",
  "routes": [
    {
      "id": "nightly", "name": "Nightly", "kind": "backup | sync | external | verify",
      "enabled": true, "next_run": "2026-07-10T02:00:00Z",
      "last_run_status": "success | failed | never", "last_run_time": "2026-07-09T02:00:00Z"
    }
  ],
  "pbss": [
    {
      "id": "pbs-01", "state": "sleeping | online | backing_up",
      "datastore_used_pct": 62,
      "datastore_used_bytes": 1900000000000, "datastore_total_bytes": 3100000000000
    }
  ]
}
```

**Changed in 1.0.0**: the flat single-PBS fields became these two lists, one entry per route and one per PBS. There is no longer a single "next run" or "datastore" to report, so a widget built on 0.9's shape picks a list entry (`.routes[0].next_run`) instead. The same applies to `/metrics`, whose series are now labelled by `route=`, `pbs=` and `datastore=`.


## Permissions cheat-sheet

Per device — each PVE and each PBS gets its own token.

- **PVE token**: `VM.Audit` (list guests) + `VM.Backup` + `Datastore.Audit` + `Datastore.AllocateSpace` **and `Datastore.Allocate`** on the PBS-backed storage (the last is required for vzdump's retention/prune, which deletes old backups). Root-mode setup creates a `Joulenap` role with exactly these privileges (`connectors/provision.py`).
- **PBS token**: `DatastoreAdmin` on the datastore (status, GC, verify) plus `Audit` on `/system` (read-only node CPU/RAM/network for the dashboard). PBS has no API to create custom roles, so root-mode setup grants these built-ins scoped by path. The token is named **`joulenap-<datastore>`**: a device is a *(host, datastore)* pair, so one machine can hold two, and a shared name would mean provisioning the second deleted and recreated the first one's token. Deleting a token also drops its ACL entries, so that would have left the first device unable to connect *and* unable to be repaired by re-entering a secret.
- **PBS token, additionally, for sync routes**: `RemoteAdmin` **and** `RemoteSyncPushOperator` on `/remote`, so Joulenap can create the remote and the sync job. PBS refuses ACL writes from a token, so these can only be granted from a root login — the wizard does it while it still holds the root ticket, and a box set up before 1.0 gets them from the device editor's **Grant sync permissions** action (`POST /api/wizard/pbs/grant-sync`), which asks for root once and stores nothing. See [`CONFIG-WIZARD.md`](CONFIG-WIZARD.md#sync-routes-need-one-extra-grant).
- **SSH to a PBS**: one dedicated key, shared by every managed box, ideally installed with a forced command that only allows `poweroff`.