mirror of
https://github.com/Joulenap/joulenap.git
synced 2026-08-11 13:21:43 +02:00
Merge pull request #24 from Joulenap/release-0.6.0-history-cancel-metrics
release: 0.6.0 — run history, job cancellation, Prometheus metrics
This commit is contained in:
@@ -13,7 +13,7 @@ body:
|
||||
attributes:
|
||||
label: Joulenap version
|
||||
description: Shown in the UI footer.
|
||||
placeholder: "0.5.0"
|
||||
placeholder: "0.6.0"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
|
||||
+50
-4
@@ -7,6 +7,51 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.6.0]
|
||||
|
||||
### Added
|
||||
|
||||
- **Run history in the interface** — the activity card now has two tabs. Alongside the familiar
|
||||
activity log there is a Run history table listing every run with its job type, what triggered it,
|
||||
the result, how long it took and how many guests it covered. Clicking a run expands it in place to
|
||||
show each step (wake, wait, backup, garbage collection, verify, power-off) with its own duration
|
||||
and outcome, plus that run's log lines. Failures show their error without expanding. The history
|
||||
is kept for as long as `maintenance.history.retention_days`, which the panel states.
|
||||
- **Stop a running job** — the Run backup and Run GC buttons turn into a Stop button while a job is
|
||||
in progress, with a confirmation dialog that can also power the PBS off once the job has stopped.
|
||||
Stopping also cancels the underlying task on the Proxmox side, so a cancelled backup does not keep
|
||||
running on the server after Joulenap has stopped watching it. A scheduled verify can be stopped
|
||||
the same way. Previously a stuck backup or an unreachable PBS blocked every later job — and manual
|
||||
power-off — until the container was restarted.
|
||||
- **Prometheus metrics for Grafana** — a new `/metrics` endpoint, protected by the same read-only
|
||||
API key as the dashboard integration, exposing PBS power state, scheduler state, the last run's
|
||||
result and duration, datastore usage, run counts, and the last backup time of every individual
|
||||
guest. Scraping never wakes the PBS, and cached values keep reporting while it sleeps. This makes
|
||||
it possible to alert on a guest quietly dropping out of your backup set; `docs/INTEGRATIONS.md`
|
||||
has the scrape configuration, the full metric list, example queries and ready-made alert rules.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Notifications name the job that ran.** A scheduled verification or a garbage-collection run that
|
||||
failed used to notify "backup failed"; each job type now reports its own outcome, in English and
|
||||
Italian.
|
||||
- **Removed the `backup.guests.auto_include_new` setting.** It never had any effect, while its name
|
||||
and default implied newly created guests were picked up automatically. Existing configuration
|
||||
files keep working — the key is ignored and dropped on the next save. The behaviour it seemed to
|
||||
promise is what "all" and "exclude" mode already do; "include" mode is, and always was, an
|
||||
explicit list. The documentation now says so.
|
||||
- **Documentation accuracy pass.** Corrected the guest-selection and garbage-collection
|
||||
descriptions, the Proxmox VE token privilege list (which omitted `Datastore.Audit` and
|
||||
`Datastore.Allocate`, so a manually created token would fail at prune time), the Proxmox Backup
|
||||
Server token privileges, the supported-versions table, and the API reference, which was missing
|
||||
several endpoints. Added a walkthrough of the Settings tabs to the install guide.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Toggle switches are now announced correctly by screen readers, and can no longer submit a
|
||||
surrounding form by accident.
|
||||
- Repaired three changelog comparison links that pointed at version tags which were never published.
|
||||
|
||||
## [0.5.0]
|
||||
|
||||
### Added
|
||||
@@ -263,13 +308,14 @@ Backup Server, all from a web UI.
|
||||
- Config-driven via `config.yaml` (pydantic-validated); secrets stay in `config.yaml` and are
|
||||
redacted from API responses.
|
||||
|
||||
[Unreleased]: https://github.com/Joulenap/joulenap/compare/v0.5.0...HEAD
|
||||
[Unreleased]: https://github.com/Joulenap/joulenap/compare/v0.6.0...HEAD
|
||||
[0.6.0]: https://github.com/Joulenap/joulenap/compare/v0.5.0...v0.6.0
|
||||
[0.5.0]: https://github.com/Joulenap/joulenap/compare/v0.4.4...v0.5.0
|
||||
[0.4.4]: https://github.com/Joulenap/joulenap/compare/v0.4.3...v0.4.4
|
||||
[0.4.3]: https://github.com/Joulenap/joulenap/compare/v0.4.2...v0.4.3
|
||||
[0.4.2]: https://github.com/Joulenap/joulenap/compare/v0.4.1...v0.4.2
|
||||
[0.4.1]: https://github.com/Joulenap/joulenap/compare/v0.4.0...v0.4.1
|
||||
[0.4.0]: https://github.com/Joulenap/joulenap/compare/v0.3.1...v0.4.0
|
||||
[0.4.2]: https://github.com/Joulenap/joulenap/compare/3f94413...v0.4.2
|
||||
[0.4.1]: https://github.com/Joulenap/joulenap/compare/340646b...3f94413
|
||||
[0.4.0]: https://github.com/Joulenap/joulenap/compare/v0.3.1...340646b
|
||||
[0.3.1]: https://github.com/Joulenap/joulenap/compare/v0.3.0...v0.3.1
|
||||
[0.3.0]: https://github.com/Joulenap/joulenap/compare/v0.2.0...v0.3.0
|
||||
[0.2.0]: https://github.com/Joulenap/joulenap/compare/v0.1.1...v0.2.0
|
||||
|
||||
@@ -40,20 +40,22 @@ Joulenap **owns the schedule** itself (internal scheduler), so nothing on the Pr
|
||||
|
||||
- ⏰ Web UI scheduler: choose backup time, enable/disable, see next/last run
|
||||
- 🔌 Wake-on-LAN of the PBS, with readiness wait and timeout
|
||||
- 🗂️ Per-guest selection (toggle which CTs/VMs to back up, or "all + auto-include new")
|
||||
- ♻️ Retention (daily/weekly/monthly) and scheduled Garbage Collection
|
||||
- 🗂️ Per-guest selection: back up **all** guests, all **except** a list, or an explicit **include** list (new guests are covered automatically in the first two)
|
||||
- ♻️ Retention (daily/weekly/monthly/yearly), Garbage Collection after each backup, and a separately scheduled verify
|
||||
- 🔔 Notifications: Apprise, Telegram, ntfy, Discord, email — on success and/or failure
|
||||
- 📜 Live log viewer and manual "Run backup now" / "Run GC now"
|
||||
- 📊 Dashboard integration: expose backup status to Homepage, Homarr, Dashy or Glance — see [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md)
|
||||
- 📜 Live log viewer, run history with per-step detail, live PVE/PBS task output, and manual "Run backup now" / "Run GC now" (optionally leaving the PBS awake) — stoppable mid-run
|
||||
- ⚙️ Advanced settings tab with a built-in `config.yaml` editor, plus an opt-in update check
|
||||
- 📊 Integrations: backup status for Homepage, Homarr, Dashy or Glance, plus a Prometheus `/metrics` endpoint for Grafana (alert when a guest stops being backed up) — see [`docs/INTEGRATIONS.md`](docs/INTEGRATIONS.md)
|
||||
- 🌍 Multi-language UI
|
||||
- 🔒 Login-protected; secrets kept out of the repo
|
||||
|
||||
## Status
|
||||
|
||||
**v0.5.0.** Feature-complete: scheduler + Wake-on-LAN + vzdump + retention + GC + verify +
|
||||
**v0.6.0.** Feature-complete: scheduler + Wake-on-LAN + vzdump + retention + GC + verify +
|
||||
notifications + setup wizard, packaged as a Docker image — with transport hardening (PBS TLS
|
||||
pinning + SSH host-key verification) and auth hardening (login rate-limit, session hardening).
|
||||
Includes a read-only [dashboard integration](docs/INTEGRATIONS.md) (Homepage/Homarr/Dashy/Glance),
|
||||
Includes run history with per-step detail, the ability to stop a job mid-run,
|
||||
[integrations](docs/INTEGRATIONS.md) for dashboards (Homepage/Homarr/Dashy/Glance) and Prometheus,
|
||||
persistent datastore usage shown even while the PBS is powered off, a per-channel notification
|
||||
test report, and a responsive UI that works on a phone.
|
||||
See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the design and API.
|
||||
@@ -95,7 +97,7 @@ All settings live in `config.yaml` (see [`config.example.yaml`](config.example.y
|
||||
|
||||
Joulenap can trigger backups and power machines on/off, so treat it as privileged:
|
||||
|
||||
- Use **scoped API tokens** for PVE (Audit + Backup) and PBS, not root passwords.
|
||||
- Use **scoped API tokens** for PVE and PBS, not root passwords — the exact privileges each one needs are listed in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md#permissions-cheat-sheet).
|
||||
- The SSH key to PBS should be dedicated and, ideally, restricted to the power-off command.
|
||||
- **PBS API is TLS-pinned**: calls to PBS are pinned to its certificate fingerprint (captured at setup), so a swapped/MITM cert is rejected; a legitimately renewed cert is accepted after you re-run PBS detection in the wizard.
|
||||
- **PBS SSH host key is verified**: confirmed once during setup and stored in `data/known_hosts`; later power-off/GC connections verify against it. Details in [`docs/CONFIG-WIZARD.md`](docs/CONFIG-WIZARD.md#security).
|
||||
@@ -109,7 +111,7 @@ Joulenap can trigger backups and power machines on/off, so treat it as privilege
|
||||
## Roadmap
|
||||
|
||||
- [✅] v0.1: scheduler + WoL + vzdump + retention + notifications + web UI
|
||||
- [✅] Garbage Collection scheduling and verify jobs
|
||||
- [✅] Garbage Collection after each backup, and scheduled verify jobs
|
||||
- [✅] Per-guest last-backup status from PBS
|
||||
- [ ] RTC-wake option (BIOS alarm) as an alternative to WoL
|
||||
- [ ] Multiple PBS targets / off-site sync
|
||||
|
||||
+2
-2
@@ -9,8 +9,8 @@ Joulenap is pre-1.0 and moves fast; only the latest release line receives securi
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | --------- |
|
||||
| 0.1.x | ✅ |
|
||||
| < 0.1 | ❌ |
|
||||
| 0.5.x | ✅ |
|
||||
| < 0.5 | ❌ |
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Joulenap — web UI + scheduler for energy-saving Proxmox backups to a normally-off PBS."""
|
||||
|
||||
__version__ = "0.5.0"
|
||||
__version__ = "0.6.0"
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Shared API-key check for the read-only, session-less endpoints.
|
||||
|
||||
Both `/api/dashboard` (dashboard widgets) and `/metrics` (Prometheus) are polled by
|
||||
machines that can't hold a login session, so they authenticate with the single
|
||||
``app.api_key`` instead — sent as an ``X-API-Key`` header, or as a ``?key=`` query
|
||||
param for clients that can't set custom headers (Prometheus's ``params:``, older
|
||||
dashboard widgets).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
|
||||
from ..core.config_store import ConfigStore
|
||||
|
||||
|
||||
def authorize_api_key(request: Request, store: ConfigStore) -> None:
|
||||
"""Raise unless the request carries the configured API key.
|
||||
|
||||
403 when no key is configured (the integration is off) vs 401 for a wrong key, so a
|
||||
scraper can tell "not enabled here" from "my credential is wrong".
|
||||
"""
|
||||
key = store.config.app.api_key
|
||||
if not key:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Read-only API access is disabled (no API key configured)",
|
||||
)
|
||||
provided = request.headers.get("X-API-Key") or request.query_params.get("key") or ""
|
||||
if not secrets.compare_digest(provided.encode("utf-8"), key.encode("utf-8")):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or missing API key"
|
||||
)
|
||||
@@ -9,10 +9,9 @@ session cookie — so this router is deliberately outside require_auth.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -20,6 +19,7 @@ from ..core.config_store import ConfigStore
|
||||
from ..db import get_session
|
||||
from ..db.models import RunStatus
|
||||
from . import _probe
|
||||
from ._apikey import authorize_api_key
|
||||
from .deps import JobService, Scheduler, get_config_store, get_job_service, get_scheduler
|
||||
|
||||
router = APIRouter(tags=["dashboard"])
|
||||
@@ -35,20 +35,6 @@ class DashboardResponse(BaseModel):
|
||||
datastore_total_bytes: int | None
|
||||
|
||||
|
||||
def _authorize(request: Request, store: ConfigStore) -> None:
|
||||
key = store.config.app.api_key
|
||||
if not key:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Dashboard integration is disabled (no API key configured)",
|
||||
)
|
||||
provided = request.headers.get("X-API-Key") or request.query_params.get("key") or ""
|
||||
if not secrets.compare_digest(provided.encode("utf-8"), key.encode("utf-8")):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid or missing API key"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/dashboard", response_model=DashboardResponse)
|
||||
def get_dashboard(
|
||||
request: Request,
|
||||
@@ -57,7 +43,7 @@ def get_dashboard(
|
||||
job_service: JobService = Depends(get_job_service),
|
||||
session: Session = Depends(get_session),
|
||||
) -> DashboardResponse:
|
||||
_authorize(request, store)
|
||||
authorize_api_key(request, store)
|
||||
|
||||
config = store.config
|
||||
last = _probe.latest_finished_cycle_run(session)
|
||||
|
||||
@@ -47,3 +47,30 @@ def run_gc(
|
||||
opts: RunOptions | None = None, job_service: JobService = Depends(get_job_service)
|
||||
) -> RunStarted:
|
||||
return _start(job_service.submit_gc, (opts or RunOptions()).keep_on)
|
||||
|
||||
|
||||
class CancelRequest(BaseModel):
|
||||
# Which run to stop. Required so a click landing as one run ends and another begins
|
||||
# can't cancel the wrong job.
|
||||
run_id: int
|
||||
# Power the PBS off after stopping (the toggle in the stop dialog). Default: leave it
|
||||
# on, since cancelling often means the user wants to work on the box.
|
||||
power_off: bool = False
|
||||
|
||||
|
||||
@router.post("/jobs/cancel", status_code=status.HTTP_202_ACCEPTED)
|
||||
def cancel_job(
|
||||
body: CancelRequest, job_service: JobService = Depends(get_job_service)
|
||||
) -> dict[str, int]:
|
||||
"""Ask the in-flight run to stop (11.2).
|
||||
|
||||
202 means the request was accepted, not that the run has ended: cancellation is
|
||||
cooperative, so the worker stops the remote vzdump/GC task and unwinds within a poll
|
||||
interval. Poll GET /api/runs/{id} for the final status.
|
||||
"""
|
||||
if not job_service.cancel(body.run_id, power_off=body.power_off):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail="That run is not the one currently in progress",
|
||||
)
|
||||
return {"run_id": body.run_id}
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""GET /metrics — Prometheus exposition for Grafana homelabs (11.11).
|
||||
|
||||
Mounted at the root, not under ``/api``: ``/metrics`` is Prometheus's default
|
||||
``metrics_path``, so a scrape config needs no extra setting. Auth is the same read-only
|
||||
API key as ``/api/dashboard`` (header or ``?key=``), because a scraper can't hold a
|
||||
session cookie.
|
||||
|
||||
The exposition format is plain text, so we write it directly rather than take on
|
||||
``prometheus_client`` — a registry and its multiprocess machinery for what is a formatted
|
||||
string here.
|
||||
|
||||
Everything below is read from the DB and the same status probe the dashboard uses: a
|
||||
scrape never wakes the PBS, and reports cached datastore/last-backup values while it
|
||||
sleeps (which is the normal state — that's the whole point of Joulenap).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, Request, Response
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from .. import __version__
|
||||
from ..core.config_store import ConfigStore
|
||||
from ..db import get_session
|
||||
from ..db.guest_backups import get_last_backups
|
||||
from ..db.models import Run, RunStatus
|
||||
from . import _probe
|
||||
from ._apikey import authorize_api_key
|
||||
from .deps import JobService, Scheduler, get_config_store, get_job_service, get_scheduler
|
||||
|
||||
router = APIRouter(tags=["metrics"])
|
||||
|
||||
# The classic text format. Prometheus content-negotiates, but this is what every scraper
|
||||
# understands and what `promtool check metrics` validates against.
|
||||
CONTENT_TYPE = "text/plain; version=0.0.4; charset=utf-8"
|
||||
|
||||
_PREFIX = "joulenap_"
|
||||
|
||||
|
||||
def _escape(value: str) -> str:
|
||||
"""Escape a label value per the exposition format (backslash, quote, newline)."""
|
||||
return value.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
|
||||
|
||||
|
||||
def _number(value: float) -> str:
|
||||
"""Render a sample value without losing precision.
|
||||
|
||||
Deliberately not ``f"{value:g}"``: %g keeps 6 significant digits, which rounds a Unix
|
||||
timestamp (10 digits) to the nearest ~1000 seconds — every ``*_timestamp_seconds``
|
||||
series would have been silently wrong by up to a quarter of an hour. Integral values
|
||||
print as integers; anything else takes repr's shortest round-trip form.
|
||||
"""
|
||||
f = float(value)
|
||||
if f.is_integer():
|
||||
return str(int(f))
|
||||
return repr(f)
|
||||
|
||||
|
||||
class _Writer:
|
||||
"""Accumulates metric families in exposition order."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lines: list[str] = []
|
||||
|
||||
def metric(
|
||||
self,
|
||||
name: str,
|
||||
help_text: str,
|
||||
kind: str,
|
||||
samples: list[tuple[dict[str, str], float]],
|
||||
) -> None:
|
||||
"""Emit one metric family. ``samples`` empty -> nothing at all is written.
|
||||
|
||||
Omitting is deliberate: a missing last-run time must not be published as 0, which
|
||||
graphs as January 1970 and quietly poisons every average. Prometheus's answer to
|
||||
"no value" is an absent series (query it with ``absent()``), not a sentinel.
|
||||
"""
|
||||
if not samples:
|
||||
return
|
||||
full = _PREFIX + name
|
||||
self._lines.append(f"# HELP {full} {help_text}")
|
||||
self._lines.append(f"# TYPE {full} {kind}")
|
||||
for labels, value in samples:
|
||||
rendered = ",".join(f'{k}="{_escape(v)}"' for k, v in labels.items())
|
||||
suffix = f"{{{rendered}}}" if rendered else ""
|
||||
self._lines.append(f"{full}{suffix} {_number(value)}")
|
||||
|
||||
def gauge(self, name: str, help_text: str, value: float | None) -> None:
|
||||
"""Single unlabelled gauge; ``None`` omits the series."""
|
||||
self.metric(name, help_text, "gauge", [] if value is None else [({}, value)])
|
||||
|
||||
def render(self) -> str:
|
||||
return "\n".join(self._lines) + "\n"
|
||||
|
||||
|
||||
def _epoch(dt: datetime | None) -> float | None:
|
||||
return dt.timestamp() if dt else None
|
||||
|
||||
|
||||
@router.get("/metrics", include_in_schema=False)
|
||||
def get_metrics(
|
||||
request: Request,
|
||||
store: ConfigStore = Depends(get_config_store),
|
||||
scheduler: Scheduler = Depends(get_scheduler),
|
||||
job_service: JobService = Depends(get_job_service),
|
||||
session: Session = Depends(get_session),
|
||||
) -> Response:
|
||||
authorize_api_key(request, store)
|
||||
|
||||
config = store.config
|
||||
w = _Writer()
|
||||
|
||||
w.metric(
|
||||
"build_info",
|
||||
"Joulenap build information; the value is always 1.",
|
||||
"gauge",
|
||||
[({"version": __version__}, 1)],
|
||||
)
|
||||
|
||||
# --- power + scheduler state ---
|
||||
pbs_online, live_ds, load = _probe.probe_pbs(config, job_service.deps.build_pbs)
|
||||
w.gauge("pbs_online", "1 if the PBS answers on its API port, 0 while asleep.", int(pbs_online))
|
||||
w.gauge(
|
||||
"scheduler_enabled",
|
||||
"1 if the scheduled backup job is armed.",
|
||||
int(config.backup.enabled),
|
||||
)
|
||||
w.gauge(
|
||||
"job_running",
|
||||
"1 while a backup, GC or verify run is in flight.",
|
||||
int(job_service.is_running),
|
||||
)
|
||||
w.gauge(
|
||||
"next_run_timestamp_seconds",
|
||||
"Unix time of the next scheduled backup; absent when the scheduler is off.",
|
||||
_epoch(scheduler.next_run_time),
|
||||
)
|
||||
|
||||
# PBS load is only meaningful while the box is awake, so those series come and go.
|
||||
w.gauge("pbs_cpu_percent", "PBS CPU usage percent (only while the PBS is awake).",
|
||||
load.cpu if load else None)
|
||||
w.gauge("pbs_memory_percent", "PBS memory usage percent (only while the PBS is awake).",
|
||||
load.mem if load else None)
|
||||
w.gauge("pbs_uptime_seconds", "PBS uptime in seconds (only while the PBS is awake).",
|
||||
load.uptime if load else None)
|
||||
|
||||
# --- last completed backup cycle ---
|
||||
last = _probe.latest_finished_cycle_run(session)
|
||||
w.gauge(
|
||||
"last_run_timestamp_seconds",
|
||||
"Unix time the last finished backup cycle started; absent until one has run.",
|
||||
_epoch(last.started_at) if last else None,
|
||||
)
|
||||
w.gauge(
|
||||
"last_run_success",
|
||||
"1 if the last finished backup cycle succeeded, 0 if it failed or was aborted.",
|
||||
int(last.status == RunStatus.SUCCESS) if last else None,
|
||||
)
|
||||
w.gauge(
|
||||
"last_run_duration_seconds",
|
||||
"Wall-clock duration of the last finished backup cycle.",
|
||||
(last.finished_at - last.started_at).total_seconds()
|
||||
if last and last.finished_at
|
||||
else None,
|
||||
)
|
||||
w.gauge(
|
||||
"last_run_guests",
|
||||
"Number of guests backed up by the last finished backup cycle.",
|
||||
last.guests_ok if last and last.guests_ok is not None else None,
|
||||
)
|
||||
|
||||
# --- datastore (cached, so it still reports while the PBS sleeps) ---
|
||||
ds = _probe.resolve_datastore(config.pbs.datastore, live_ds)
|
||||
w.gauge("datastore_used_bytes", "PBS datastore bytes used (last known value).",
|
||||
ds.used if ds else None)
|
||||
w.gauge("datastore_total_bytes", "PBS datastore total bytes (last known value).",
|
||||
ds.total if ds else None)
|
||||
|
||||
# --- per-guest freshness: the series worth alerting on ---
|
||||
# Labelled by vmid only. Guest *names* live on PVE, and fetching them would put an API
|
||||
# call on every scrape just to decorate a label.
|
||||
guests = get_last_backups(session)
|
||||
w.metric(
|
||||
"guest_last_backup_timestamp_seconds",
|
||||
"Unix time of each guest's most recent snapshot on the PBS, from Joulenap's cache.",
|
||||
"gauge",
|
||||
[({"vmid": str(vmid)}, ts.timestamp()) for vmid, ts in sorted(guests.items())],
|
||||
)
|
||||
|
||||
# --- run history ---
|
||||
# A gauge, NOT a `_total` counter: the daily prune job deletes runs older than
|
||||
# maintenance.history.retention_days, so this legitimately goes down and would break
|
||||
# every rate()/increase() over a counter.
|
||||
rows = session.execute(
|
||||
select(Run.kind, Run.status, func.count(Run.id))
|
||||
.where(Run.status != RunStatus.RUNNING)
|
||||
.group_by(Run.kind, Run.status)
|
||||
).all()
|
||||
w.metric(
|
||||
"runs_recent",
|
||||
"Finished runs currently in the history window, by kind and status.",
|
||||
"gauge",
|
||||
[({"kind": kind, "status": status}, count) for kind, status, count in rows],
|
||||
)
|
||||
|
||||
return Response(content=w.render(), media_type=CONTENT_TYPE)
|
||||
@@ -18,6 +18,8 @@ class RunSummary(BaseModel):
|
||||
status: str
|
||||
started_at: datetime
|
||||
finished_at: datetime | None
|
||||
# Nullable while a run is in flight, and for kinds that don't touch guests (GC/verify).
|
||||
guests_ok: int | None = None
|
||||
error: str | None = None
|
||||
|
||||
@classmethod
|
||||
|
||||
@@ -42,6 +42,7 @@ class StatusResponse(BaseModel):
|
||||
next_run: datetime | None
|
||||
job_running: bool
|
||||
running_kind: str | None = None # "cycle" | "gc" | "verify" while a run is in flight
|
||||
running_run_id: int | None = None # the in-flight run, so the UI can cancel exactly it
|
||||
pbs_online: bool
|
||||
last_run: RunSummary | None
|
||||
datastore: DatastoreInfo | None = None
|
||||
@@ -72,6 +73,7 @@ def get_status(
|
||||
next_run=scheduler.next_run_time,
|
||||
job_running=job_service.is_running,
|
||||
running_kind=running.kind if running else None,
|
||||
running_run_id=running.id if running else None,
|
||||
pbs_online=pbs_online,
|
||||
last_run=RunSummary.of(last) if last else None,
|
||||
datastore=datastore,
|
||||
|
||||
+18
-1
@@ -116,7 +116,6 @@ class PbsConfig(_Base):
|
||||
|
||||
class GuestsConfig(_Base):
|
||||
mode: Literal["all", "include", "exclude"] = "all"
|
||||
auto_include_new: bool = True
|
||||
# Field name matches the YAML key. ``typing.List`` (not ``list[int]``) avoids the
|
||||
# field name shadowing the builtin during Python 3.14 deferred annotation eval.
|
||||
list: List[int] = Field(default_factory=list) # noqa: UP006
|
||||
@@ -232,6 +231,23 @@ class Config(_Base):
|
||||
# --- load / save / redact ----------------------------------------------------
|
||||
|
||||
|
||||
def _drop_legacy_keys(raw: dict[str, Any]) -> None:
|
||||
"""Strip keys removed in a later version, in place, before validation.
|
||||
|
||||
``_Base`` forbids extra keys, so a field we delete from the schema would make every
|
||||
existing ``config.yaml`` fail at startup — the app wouldn't boot after a container pull
|
||||
(the BE-B1 lesson: never brick the app on load). Dropping the key here keeps the old file
|
||||
valid; the next save rewrites it without the key.
|
||||
|
||||
Removed in 0.6.0: ``backup.guests.auto_include_new`` — it never had any effect. Include
|
||||
mode is an explicit list; ``all`` and ``exclude`` already cover newly created guests.
|
||||
"""
|
||||
backup = raw.get("backup")
|
||||
guests = backup.get("guests") if isinstance(backup, dict) else None
|
||||
if isinstance(guests, dict):
|
||||
guests.pop("auto_include_new", None)
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> Config:
|
||||
"""Read and validate ``config.yaml``. Raises with a clear message if missing/invalid."""
|
||||
p = path or paths.config_path()
|
||||
@@ -243,6 +259,7 @@ def load_config(path: Path | None = None) -> Config:
|
||||
raw = yaml.safe_load(fh) or {}
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"Config at {p} must be a YAML mapping, got {type(raw).__name__}.")
|
||||
_drop_legacy_keys(raw)
|
||||
return Config.model_validate(raw)
|
||||
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ import time
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
|
||||
from .errors import TaskError
|
||||
from .errors import TaskCancelled, TaskError
|
||||
|
||||
# One task-log line as returned by the tailer: (line number, text). The line number is
|
||||
# the task's own 1-based ``n``; the tailer uses it as the offset cursor for the next fetch.
|
||||
@@ -28,12 +28,18 @@ def poll_task(
|
||||
*,
|
||||
log_fn: Callable[[int], list[LogLine]] | None = None,
|
||||
on_lines: Callable[[list[LogLine]], None] | None = None,
|
||||
should_cancel: Callable[[], bool] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Poll ``status_fn(upid)`` until the task stops; return its final status.
|
||||
|
||||
Raises :class:`TaskError` if the task finishes with a non-OK exit status or does
|
||||
not finish within ``timeout`` seconds.
|
||||
|
||||
``should_cancel`` makes the wait interruptible: it is consulted once per poll, and a
|
||||
True raises :class:`TaskCancelled` — the caller decides whether to also stop the remote
|
||||
task. This is the only way out of a long wait, since a blocking thread can't be
|
||||
interrupted from outside; the poll interval is therefore the cancel latency.
|
||||
|
||||
If both ``log_fn`` and ``on_lines`` are given, each poll also drains any new task-log
|
||||
lines: ``log_fn(offset)`` returns lines numbered greater than ``offset`` (empty once
|
||||
caught up) and ``on_lines`` is handed each new batch. The final poll (task stopped)
|
||||
@@ -54,6 +60,11 @@ def poll_task(
|
||||
seen = max(n for n, _ in batch)
|
||||
|
||||
while True:
|
||||
if should_cancel is not None and should_cancel():
|
||||
# Drain first so the task-log panel keeps the last lines the task managed to
|
||||
# write before we walked away.
|
||||
drain()
|
||||
raise TaskCancelled(f"Wait for task {upid} cancelled")
|
||||
status = status_fn(upid)
|
||||
drain() # pull whatever's been logged since the last tick (tail after stop)
|
||||
if status.get("status") == "stopped":
|
||||
|
||||
@@ -33,3 +33,11 @@ class TaskError(ConnectorError):
|
||||
def __init__(self, message: str, exit_status: str | None = None):
|
||||
super().__init__(message)
|
||||
self.exit_status = exit_status
|
||||
|
||||
|
||||
class TaskCancelled(ConnectorError):
|
||||
"""The caller asked to stop waiting on a task (user-requested cancellation).
|
||||
|
||||
Deliberately *not* a :class:`TaskError`: the remote task didn't fail, we chose to stop
|
||||
waiting for it — the two lead to different run outcomes (aborted vs failed).
|
||||
"""
|
||||
|
||||
@@ -33,10 +33,18 @@ def wait_until_reachable(
|
||||
interval: float = 2.0,
|
||||
connect_timeout: float = 3.0,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
should_cancel: Callable[[], bool] | None = None,
|
||||
) -> bool:
|
||||
"""Poll ``host:port`` until reachable or ``timeout`` elapses. Returns success."""
|
||||
"""Poll ``host:port`` until reachable or ``timeout`` elapses. Returns success.
|
||||
|
||||
``should_cancel`` lets a user cancel abandon the wait early — it returns False like a
|
||||
timeout would, since "the PBS never came up" is exactly the state the caller must handle.
|
||||
Without it a cancelled run would sit here for the full wake timeout.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
if should_cancel is not None and should_cancel():
|
||||
return False
|
||||
if tcp_reachable(host, port, connect_timeout):
|
||||
return True
|
||||
if time.monotonic() >= deadline:
|
||||
|
||||
@@ -195,6 +195,11 @@ class PbsClient:
|
||||
)
|
||||
return [(int(e["n"]), e.get("t") or "") for e in (data or [])]
|
||||
|
||||
def stop_task(self, upid: str) -> None:
|
||||
"""Ask PBS to stop a running task (the GC/verify behind a cancelled run), so it
|
||||
doesn't keep the datastore busy after Joulenap stopped watching it."""
|
||||
self._api.request("DELETE", f"/nodes/{self.node}/tasks/{upid}")
|
||||
|
||||
def wait_task(
|
||||
self,
|
||||
upid: str,
|
||||
@@ -203,16 +208,18 @@ class PbsClient:
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
*,
|
||||
on_log: Callable[[list[LogLine]], None] | None = None,
|
||||
should_cancel: Callable[[], bool] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Poll a task until it stops. Returns the final status; raises on non-OK exit.
|
||||
|
||||
Pass ``on_log`` to also tail the task log — each new batch of ``(line_no, text)``
|
||||
pairs is handed to it as the task runs.
|
||||
pairs is handed to it as the task runs. ``should_cancel`` makes the wait
|
||||
interruptible (raises ``TaskCancelled``); see :func:`poll_task`.
|
||||
"""
|
||||
log_fn = (lambda start: self.task_log(upid, start)) if on_log else None
|
||||
return poll_task(
|
||||
self.task_status, upid, poll_interval, timeout, sleep,
|
||||
log_fn=log_fn, on_lines=on_log,
|
||||
log_fn=log_fn, on_lines=on_log, should_cancel=should_cancel,
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
|
||||
@@ -155,6 +155,14 @@ class PveClient:
|
||||
)
|
||||
return [(int(e["n"]), e.get("t") or "") for e in (data or [])]
|
||||
|
||||
def stop_task(self, upid: str) -> None:
|
||||
"""Ask PVE to stop a running task (the vzdump behind a cancelled run).
|
||||
|
||||
Without this a cancelled backup would keep running on the PVE host after Joulenap
|
||||
stopped watching it, and the next run would collide with it.
|
||||
"""
|
||||
self._api.request("DELETE", f"/nodes/{self.node}/tasks/{upid}")
|
||||
|
||||
def wait_task(
|
||||
self,
|
||||
upid: str,
|
||||
@@ -163,16 +171,18 @@ class PveClient:
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
*,
|
||||
on_log: Callable[[list[LogLine]], None] | None = None,
|
||||
should_cancel: Callable[[], bool] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Poll a task until it stops. Returns the final status; raises on non-OK exit.
|
||||
|
||||
Pass ``on_log`` to also tail the task log — each new batch of ``(line_no, text)``
|
||||
pairs is handed to it as the task runs.
|
||||
pairs is handed to it as the task runs. ``should_cancel`` makes the wait
|
||||
interruptible (raises ``TaskCancelled``); see :func:`poll_task`.
|
||||
"""
|
||||
log_fn = (lambda start: self.task_log(upid, start)) if on_log else None
|
||||
return poll_task(
|
||||
self.task_status, upid, poll_interval, timeout, sleep,
|
||||
log_fn=log_fn, on_lines=on_log,
|
||||
log_fn=log_fn, on_lines=on_log, should_cancel=should_cancel,
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
|
||||
@@ -10,7 +10,10 @@ power-off step simply never runs.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Protocol
|
||||
|
||||
from ..config import Config
|
||||
from ..connectors.errors import TaskCancelled
|
||||
from ..connectors.pbs import DatastoreStatus
|
||||
from ..connectors.pve import PveClient, build_prune_string
|
||||
from ..db import session_scope
|
||||
@@ -45,6 +48,56 @@ class CycleAbort(Exception):
|
||||
"""Raised when the PBS doesn't come up — the cycle aborts without powering off."""
|
||||
|
||||
|
||||
class CycleCancelled(Exception):
|
||||
"""Raised when the user stopped the run from the UI (11.2).
|
||||
|
||||
Separate from :class:`CycleAbort` so the run records *why* it ended and the power-off
|
||||
decision follows what the user chose in the stop dialog rather than the abort rules.
|
||||
"""
|
||||
|
||||
|
||||
class _TaskClient(Protocol):
|
||||
"""The slice of PveClient/PbsClient that :func:`_wait_or_stop` needs."""
|
||||
|
||||
def wait_task(self, upid: str, *args: Any, **kwargs: Any) -> dict[str, Any]: ...
|
||||
def stop_task(self, upid: str) -> None: ...
|
||||
|
||||
|
||||
def _wait_or_stop(
|
||||
client: _TaskClient,
|
||||
upid: str,
|
||||
recorder: RunRecorder,
|
||||
deps: CycleDeps,
|
||||
step: StepName,
|
||||
source: str,
|
||||
) -> None:
|
||||
"""Wait for a PVE/PBS task, stopping it remotely if the user cancels the run.
|
||||
|
||||
Abandoning the wait isn't enough: the vzdump/GC would keep running on the far side while
|
||||
Joulenap considers itself idle, and the next run would collide with it. The stop is
|
||||
best-effort — if the API refuses, the run still ends cancelled and the reason is logged,
|
||||
because leaving the lock held would be the worse failure (that is the whole point of 11.2).
|
||||
"""
|
||||
try:
|
||||
client.wait_task(
|
||||
upid,
|
||||
poll_interval=_TAIL_INTERVAL,
|
||||
on_log=_tailer(recorder, step, source),
|
||||
should_cancel=deps.cancelled,
|
||||
)
|
||||
except TaskCancelled as exc:
|
||||
try:
|
||||
client.stop_task(upid)
|
||||
recorder.log(LogLevel.WARN, f"cancelled: asked {source.upper()} to stop task {upid}")
|
||||
except Exception as stop_exc:
|
||||
recorder.log(
|
||||
LogLevel.ERROR,
|
||||
f"cancelled, but could not stop {source.upper()} task {upid}: {stop_exc} "
|
||||
"— check it on the server",
|
||||
)
|
||||
raise CycleCancelled("Run cancelled") from exc
|
||||
|
||||
|
||||
def select_vmids(config: Config, pve: PveClient) -> tuple[list[int] | None, bool]:
|
||||
"""Resolve the configured guest selection into vzdump arguments.
|
||||
|
||||
@@ -79,11 +132,7 @@ def _run_backup_step(config: Config, recorder: RunRecorder, deps: CycleDeps) ->
|
||||
bwlimit=config.backup.bwlimit,
|
||||
)
|
||||
step.detail = upid
|
||||
pve.wait_task(
|
||||
upid,
|
||||
poll_interval=_TAIL_INTERVAL,
|
||||
on_log=_tailer(recorder, StepName.BACKUP, "pve"),
|
||||
)
|
||||
_wait_or_stop(pve, upid, recorder, deps, StepName.BACKUP, "pve")
|
||||
# Record the count only once the task succeeded, so a failed run doesn't
|
||||
# advertise guests as backed up.
|
||||
recorder.run.guests_ok = guest_count
|
||||
@@ -120,11 +169,7 @@ def run_gc_step(config: Config, recorder: RunRecorder, deps: CycleDeps) -> None:
|
||||
with deps.build_pbs(config) as pbs:
|
||||
upid = pbs.start_gc()
|
||||
step.detail = upid
|
||||
pbs.wait_task(
|
||||
upid,
|
||||
poll_interval=_TAIL_INTERVAL,
|
||||
on_log=_tailer(recorder, StepName.GC, "pbs"),
|
||||
)
|
||||
_wait_or_stop(pbs, upid, recorder, deps, StepName.GC, "pbs")
|
||||
|
||||
|
||||
def run_verify_step(
|
||||
@@ -143,11 +188,7 @@ def run_verify_step(
|
||||
else:
|
||||
upid = pbs.start_verify(ignore_verified=True, outdated_after=outdated_after)
|
||||
step.detail = upid
|
||||
pbs.wait_task(
|
||||
upid,
|
||||
poll_interval=_TAIL_INTERVAL,
|
||||
on_log=_tailer(recorder, StepName.VERIFY, "pbs"),
|
||||
)
|
||||
_wait_or_stop(pbs, upid, recorder, deps, StepName.VERIFY, "pbs")
|
||||
|
||||
|
||||
def _wait_for_pbs(config: Config, recorder: RunRecorder, deps: CycleDeps) -> bool:
|
||||
@@ -161,8 +202,12 @@ def _wait_for_pbs(config: Config, recorder: RunRecorder, deps: CycleDeps) -> boo
|
||||
p = config.pbs
|
||||
attempts = p.wol_retries + 1
|
||||
for attempt in range(1, attempts + 1):
|
||||
if deps.wait_reachable(config):
|
||||
if deps.wait_reachable(config, deps.cancelled):
|
||||
return True
|
||||
# A cancelled wait returns False like a timeout does, so check *why* before
|
||||
# burning the remaining wake attempts on a run the user already stopped.
|
||||
if deps.cancelled():
|
||||
raise CycleCancelled("Run cancelled while waiting for the PBS")
|
||||
if attempt < attempts:
|
||||
recorder.log(
|
||||
LogLevel.WARN,
|
||||
@@ -256,6 +301,25 @@ def _poweroff(config: Config, recorder: RunRecorder, deps: CycleDeps) -> None:
|
||||
recorder.log(LogLevel.WARN, f"power-off failed, PBS left on: {exc}")
|
||||
|
||||
|
||||
def _finish_cancelled(config: Config, recorder: RunRecorder, deps: CycleDeps) -> None:
|
||||
"""Close out a run the user stopped: record it ABORTED, and honour the power-off choice
|
||||
from the stop dialog.
|
||||
|
||||
Power off only when the user asked *and* the WAIT step actually succeeded — cancelling
|
||||
during the wake wait means the box may never have come up, so an SSH power-off would just
|
||||
fail and muddy the run with a spurious failed step. ``_poweroff`` waits for the PBS to go
|
||||
idle first, which also gives the task we just stopped time to unwind.
|
||||
"""
|
||||
woke = any(
|
||||
s.name == StepName.WAIT and s.status == StepStatus.SUCCESS for s in recorder.run.steps
|
||||
)
|
||||
if deps.cancel_power_off() and woke:
|
||||
_poweroff(config, recorder, deps)
|
||||
elif woke:
|
||||
recorder.skip_step(StepName.POWEROFF, "cancelled; PBS left on")
|
||||
recorder.finish(RunStatus.ABORTED, error="Cancelled by user")
|
||||
|
||||
|
||||
def _finish_power(
|
||||
config: Config, recorder: RunRecorder, deps: CycleDeps, *, power_off: bool
|
||||
) -> None:
|
||||
@@ -290,11 +354,19 @@ def run_backup_cycle(
|
||||
_preflight_step(config, recorder, deps)
|
||||
_run_backup_step(config, recorder, deps)
|
||||
|
||||
# A cancel that lands between steps must not start the next one — the task waits
|
||||
# check the flag themselves, this covers the gaps between them.
|
||||
if deps.cancelled():
|
||||
raise CycleCancelled("Run cancelled")
|
||||
|
||||
if config.maintenance.gc.enabled:
|
||||
run_gc_step(config, recorder, deps)
|
||||
else:
|
||||
recorder.skip_step(StepName.GC, "GC disabled")
|
||||
|
||||
if deps.cancelled():
|
||||
raise CycleCancelled("Run cancelled")
|
||||
|
||||
# Quick verify of just this run's new snapshots, while the PBS is still awake.
|
||||
if config.maintenance.verify.after_backup:
|
||||
run_verify_step(config, recorder, deps, outdated_after=None)
|
||||
@@ -308,6 +380,11 @@ def run_backup_cycle(
|
||||
_finish_power(config, recorder, deps, power_off=power_off)
|
||||
|
||||
recorder.finish(RunStatus.SUCCESS)
|
||||
except CycleCancelled:
|
||||
# No notification: the user pressed Stop and is standing at the UI — a "backup
|
||||
# aborted" push would just be noise about their own click.
|
||||
_finish_cancelled(config, recorder, deps)
|
||||
return
|
||||
except CycleAbort as exc:
|
||||
recorder.finish(RunStatus.ABORTED, error=str(exc))
|
||||
except Exception as exc: # connector/task failures: leave PBS on, mark failed
|
||||
@@ -343,6 +420,11 @@ def run_verify_cycle(
|
||||
_finish_power(config, recorder, deps, power_off=power_off)
|
||||
|
||||
recorder.finish(RunStatus.SUCCESS)
|
||||
except CycleCancelled:
|
||||
# No notification: the user pressed Stop and is standing at the UI — a "backup
|
||||
# aborted" push would just be noise about their own click.
|
||||
_finish_cancelled(config, recorder, deps)
|
||||
return
|
||||
except CycleAbort as exc:
|
||||
recorder.finish(RunStatus.ABORTED, error=str(exc))
|
||||
except Exception as exc: # connector/task failures: leave PBS on, mark failed
|
||||
@@ -377,6 +459,11 @@ def run_gc_cycle(
|
||||
_finish_power(config, recorder, deps, power_off=power_off)
|
||||
|
||||
recorder.finish(RunStatus.SUCCESS)
|
||||
except CycleCancelled:
|
||||
# No notification: the user pressed Stop and is standing at the UI — a "backup
|
||||
# aborted" push would just be noise about their own click.
|
||||
_finish_cancelled(config, recorder, deps)
|
||||
return
|
||||
except CycleAbort as exc:
|
||||
recorder.finish(RunStatus.ABORTED, error=str(exc))
|
||||
except Exception as exc: # connector/task failures: leave PBS on, mark failed
|
||||
|
||||
@@ -67,9 +67,11 @@ def _send_wol(config: Config) -> None:
|
||||
send_magic_packet(p.mac, broadcast=dest, source_ip=source_ip)
|
||||
|
||||
|
||||
def _wait_reachable(config: Config) -> bool:
|
||||
def _wait_reachable(config: Config, should_cancel: Callable[[], bool] | None = None) -> bool:
|
||||
p = config.pbs
|
||||
return net.wait_until_reachable(p.host, p.port, timeout=p.wait_timeout)
|
||||
return net.wait_until_reachable(
|
||||
p.host, p.port, timeout=p.wait_timeout, should_cancel=should_cancel
|
||||
)
|
||||
|
||||
|
||||
def _wait_pbs_idle(config: Config) -> bool:
|
||||
@@ -89,9 +91,16 @@ class CycleDeps:
|
||||
build_pbs: Callable[[Config], PbsClient]
|
||||
build_power: Callable[[Config], PbsPower]
|
||||
send_wol: Callable[[Config], None]
|
||||
wait_reachable: Callable[[Config], bool]
|
||||
wait_reachable: Callable[..., bool]
|
||||
wait_pbs_idle: Callable[[Config], bool]
|
||||
notify: Callable[[Config, Run, DatastoreStatus | None], None]
|
||||
# True once the user has asked to stop the in-flight run. Wired by JobService to its own
|
||||
# cancel event and read live, so the cycle can check it without knowing about the service.
|
||||
# Default: nothing ever cancels (tests and direct callers that don't care).
|
||||
cancelled: Callable[[], bool] = lambda: False
|
||||
# Whether that cancel asked for the PBS to be powered off afterwards (the toggle in the
|
||||
# stop dialog). Only meaningful once ``cancelled()`` is True.
|
||||
cancel_power_off: Callable[[], bool] = lambda: False
|
||||
|
||||
@classmethod
|
||||
def default(cls) -> CycleDeps:
|
||||
|
||||
@@ -32,11 +32,34 @@ class JobService:
|
||||
self._store = config_store
|
||||
self.deps = deps or CycleDeps.default()
|
||||
self._lock = threading.Lock()
|
||||
# Cancellation (11.2). The cycle can't be interrupted from outside — a blocking
|
||||
# thread never yields — so it polls these through `deps`, which we point at our own
|
||||
# state here. Guarded by `_lock` only for the run id; the Event is already atomic.
|
||||
self._cancel = threading.Event()
|
||||
self._cancel_power_off = False
|
||||
self._current_run_id: int | None = None
|
||||
self.deps.cancelled = self._cancel.is_set
|
||||
self.deps.cancel_power_off = lambda: self._cancel_power_off
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
return self._lock.locked()
|
||||
|
||||
def cancel(self, run_id: int, *, power_off: bool = False) -> bool:
|
||||
"""Ask the in-flight run to stop. Returns False if it isn't the one running.
|
||||
|
||||
The run id is required rather than "cancel whatever is running": a click that lands
|
||||
just as one run finishes and the next begins would otherwise stop the wrong job. The
|
||||
flag is cooperative — the worker notices within a poll interval, stops the remote
|
||||
task, and releases the lock on its way out.
|
||||
"""
|
||||
if self._current_run_id != run_id or not self.is_running:
|
||||
return False
|
||||
self._cancel_power_off = power_off
|
||||
self._cancel.set()
|
||||
log.info("Cancellation requested for run %d (power_off=%s)", run_id, power_off)
|
||||
return True
|
||||
|
||||
@contextmanager
|
||||
def exclusive(self) -> Iterator[None]:
|
||||
"""Hold the single-run lock for a non-job operation (e.g. a manual power-off) so it
|
||||
@@ -116,8 +139,12 @@ class JobService:
|
||||
if not self._lock.acquire(blocking=False):
|
||||
raise AlreadyRunningError("A backup or GC run is already in progress")
|
||||
try:
|
||||
# Clear any cancel left over from the previous run before this one can observe it.
|
||||
self._cancel.clear()
|
||||
self._cancel_power_off = False
|
||||
config = self._store.config # read live config at run time
|
||||
recorder = RunRecorder(kind, trigger)
|
||||
self._current_run_id = recorder.run_id
|
||||
except BaseException:
|
||||
self._lock.release()
|
||||
raise
|
||||
|
||||
@@ -20,6 +20,7 @@ from starlette.middleware.sessions import SessionMiddleware
|
||||
|
||||
from . import __version__
|
||||
from .api import api_router
|
||||
from .api import metrics as metrics_api
|
||||
from .config import Config
|
||||
from .core.config_store import ConfigStore
|
||||
from .core.ratelimit import LoginRateLimiter
|
||||
@@ -143,6 +144,9 @@ def create_app() -> FastAPI:
|
||||
return JSONResponse({"status": "ok", "version": __version__})
|
||||
|
||||
app.include_router(api_router)
|
||||
# Outside /api on purpose: /metrics is Prometheus's default metrics_path, so a scrape
|
||||
# config needs no extra setting. Registered before the SPA mount so it isn't shadowed.
|
||||
app.include_router(metrics_api.router)
|
||||
_mount_frontend(app)
|
||||
return app
|
||||
|
||||
|
||||
@@ -23,6 +23,19 @@ _MESSAGES: dict[str, dict[str, dict[str, str]]] = {
|
||||
"success": {"title": "✅ Joulenap — backup succeeded"},
|
||||
"failure": {"title": "❌ Joulenap — backup failed"},
|
||||
"aborted": {"title": "⚠️ Joulenap — backup aborted"},
|
||||
# Per-kind titles for the non-backup cycles (see _title_for). Full strings rather
|
||||
# than a "{noun} succeeded" template: Italian needs gender agreement (backup
|
||||
# riuscito / verifica riuscita), which a noun slot can't express.
|
||||
"gc": {
|
||||
"success": "✅ Joulenap — garbage collection succeeded",
|
||||
"failure": "❌ Joulenap — garbage collection failed",
|
||||
"aborted": "⚠️ Joulenap — garbage collection aborted",
|
||||
},
|
||||
"verify": {
|
||||
"success": "✅ Joulenap — verification succeeded",
|
||||
"failure": "❌ Joulenap — verification failed",
|
||||
"aborted": "⚠️ Joulenap — verification aborted",
|
||||
},
|
||||
"missed": {
|
||||
"title": "⚠️ Joulenap — missed scheduled backup",
|
||||
"intro": "A scheduled backup was skipped because Joulenap was offline when it "
|
||||
@@ -53,6 +66,16 @@ _MESSAGES: dict[str, dict[str, dict[str, str]]] = {
|
||||
"success": {"title": "✅ Joulenap — backup riuscito"},
|
||||
"failure": {"title": "❌ Joulenap — backup fallito"},
|
||||
"aborted": {"title": "⚠️ Joulenap — backup interrotto"},
|
||||
"gc": {
|
||||
"success": "✅ Joulenap — garbage collection riuscita",
|
||||
"failure": "❌ Joulenap — garbage collection fallita",
|
||||
"aborted": "⚠️ Joulenap — garbage collection interrotta",
|
||||
},
|
||||
"verify": {
|
||||
"success": "✅ Joulenap — verifica riuscita",
|
||||
"failure": "❌ Joulenap — verifica fallita",
|
||||
"aborted": "⚠️ Joulenap — verifica interrotta",
|
||||
},
|
||||
"missed": {
|
||||
"title": "⚠️ Joulenap — backup pianificato mancato",
|
||||
"intro": "Un backup pianificato è stato saltato perché Joulenap era offline "
|
||||
@@ -93,6 +116,17 @@ def _pack(language: str) -> dict[str, dict[str, str]]:
|
||||
return _MESSAGES.get(language, _MESSAGES["en"])
|
||||
|
||||
|
||||
def _title_for(pack: dict[str, dict[str, str]], kind: str, event: str) -> str:
|
||||
"""Title for a finished run, worded for the kind of cycle it was.
|
||||
|
||||
A GC or verify cycle reports its own outcome instead of borrowing the backup wording (a
|
||||
scheduled verify failure used to notify "backup failed"). Anything without its own block
|
||||
— a normal backup cycle, or a kind added later — falls back to the backup title, so a new
|
||||
``RunKind`` degrades to today's behaviour instead of raising.
|
||||
"""
|
||||
return pack.get(kind, {}).get(event) or pack[event]["title"]
|
||||
|
||||
|
||||
def _format_duration(seconds: float) -> str:
|
||||
total = int(seconds)
|
||||
minutes, secs = divmod(total, 60)
|
||||
@@ -165,7 +199,7 @@ def build_run_message(
|
||||
if _pbs_left_on(run):
|
||||
lines.append(labels["pbs_left_on"])
|
||||
|
||||
return pack[event]["title"], "\n".join(lines)
|
||||
return _title_for(pack, run.kind, event), "\n".join(lines)
|
||||
|
||||
|
||||
def _format_dt(dt: datetime | None) -> str:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "joulenap"
|
||||
version = "0.5.0"
|
||||
version = "0.6.0"
|
||||
description = "Self-hosted web UI + scheduler for energy-saving Proxmox backups to a normally-off PBS."
|
||||
readme = "../README.md"
|
||||
requires-python = ">=3.12"
|
||||
|
||||
+23
-4
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from app.connectors.errors import ConnectorError, TaskError
|
||||
from app.connectors.errors import ConnectorError, TaskCancelled, TaskError
|
||||
from app.connectors.pbs import DatastoreStatus, NodeLoad
|
||||
from app.connectors.pve import Guest
|
||||
from app.jobs.deps import CycleDeps
|
||||
@@ -34,6 +34,7 @@ class FakePve:
|
||||
self.fail_task = fail_task
|
||||
self.log_lines = log_lines or []
|
||||
self.vzdump_args: dict | None = None
|
||||
self.stopped: list[str] = [] # upids passed to stop_task
|
||||
|
||||
def __enter__(self) -> FakePve:
|
||||
return self
|
||||
@@ -64,13 +65,22 @@ class FakePve:
|
||||
}
|
||||
return "UPID:pve:backup"
|
||||
|
||||
def wait_task(self, upid: str, poll_interval=None, on_log=None, **_) -> dict:
|
||||
def wait_task(
|
||||
self, upid: str, poll_interval=None, on_log=None, should_cancel=None, **_
|
||||
) -> dict:
|
||||
if on_log and self.log_lines:
|
||||
on_log(list(enumerate(self.log_lines, start=1)))
|
||||
# Mirror poll_task: the cancel probe is checked before reporting a result, so a
|
||||
# cycle test can cancel mid-task exactly like the real client would.
|
||||
if should_cancel is not None and should_cancel():
|
||||
raise TaskCancelled(f"Wait for task {upid} cancelled")
|
||||
if self.fail_task:
|
||||
raise TaskError("vzdump failed", exit_status="job errors")
|
||||
return {"status": "stopped", "exitstatus": "OK"}
|
||||
|
||||
def stop_task(self, upid: str) -> None:
|
||||
self.stopped.append(upid)
|
||||
|
||||
|
||||
class FakePbs:
|
||||
def __init__(
|
||||
@@ -88,6 +98,7 @@ class FakePbs:
|
||||
self.fail_task = fail_task
|
||||
self.gc_started = False
|
||||
self.verify_started = False
|
||||
self.stopped: list[str] = [] # upids passed to stop_task
|
||||
self.verify_args: dict | None = None
|
||||
self.log_lines = log_lines or []
|
||||
self.fail_datastore = fail_datastore
|
||||
@@ -115,7 +126,9 @@ class FakePbs:
|
||||
self.verify_args = {"ignore_verified": ignore_verified, "outdated_after": outdated_after}
|
||||
return "UPID:pbs:verify"
|
||||
|
||||
def wait_task(self, upid: str, poll_interval=None, on_log=None, **_) -> dict:
|
||||
def wait_task(
|
||||
self, upid: str, poll_interval=None, on_log=None, should_cancel=None, **_
|
||||
) -> dict:
|
||||
lines = self.log_lines
|
||||
if upid.endswith(":gc") and self.gc_log_lines is not None:
|
||||
lines = self.gc_log_lines
|
||||
@@ -123,10 +136,15 @@ class FakePbs:
|
||||
lines = self.verify_log_lines
|
||||
if on_log and lines:
|
||||
on_log(list(enumerate(lines, start=1)))
|
||||
if should_cancel is not None and should_cancel():
|
||||
raise TaskCancelled(f"Wait for task {upid} cancelled")
|
||||
if self.fail_task:
|
||||
raise TaskError("gc failed", exit_status="error")
|
||||
return {"status": "stopped", "exitstatus": "OK"}
|
||||
|
||||
def stop_task(self, upid: str) -> None:
|
||||
self.stopped.append(upid)
|
||||
|
||||
def datastore_status(self) -> DatastoreStatus:
|
||||
if self.fail_datastore:
|
||||
raise ConnectorError("datastore read failed")
|
||||
@@ -173,7 +191,8 @@ def make_deps(
|
||||
build_pbs=lambda _c: pbs,
|
||||
build_power=lambda _c: power,
|
||||
send_wol=wol or (lambda _c: None),
|
||||
wait_reachable=lambda _c: wait(),
|
||||
# Second arg is the cancel probe the real _wait_reachable takes; fakes ignore it.
|
||||
wait_reachable=lambda _c, _cancel=None: wait(),
|
||||
wait_pbs_idle=lambda _c: idle(),
|
||||
notify=notify or (lambda _c, _r, _d=None: None),
|
||||
)
|
||||
|
||||
@@ -352,6 +352,40 @@ def test_gc_run_records(app_ctx):
|
||||
assert _wait_run(client, run_id)["kind"] == "gc"
|
||||
|
||||
|
||||
def test_cancel_asks_the_service_to_stop_that_run(app_ctx):
|
||||
client, app = app_ctx
|
||||
seen = {}
|
||||
|
||||
def fake_cancel(run_id, *, power_off=False):
|
||||
seen["args"] = (run_id, power_off)
|
||||
return True
|
||||
|
||||
app.state.job_service.cancel = fake_cancel
|
||||
|
||||
r = client.post("/api/jobs/cancel", json={"run_id": 7, "power_off": True})
|
||||
assert r.status_code == 202
|
||||
assert r.json() == {"run_id": 7}
|
||||
assert seen["args"] == (7, True)
|
||||
|
||||
|
||||
def test_cancel_defaults_to_leaving_the_pbs_on(app_ctx):
|
||||
client, app = app_ctx
|
||||
seen = {}
|
||||
app.state.job_service.cancel = lambda run_id, *, power_off=False: seen.update(
|
||||
power_off=power_off
|
||||
) or True
|
||||
|
||||
client.post("/api/jobs/cancel", json={"run_id": 7})
|
||||
assert seen["power_off"] is False
|
||||
|
||||
|
||||
def test_cancel_conflicts_when_that_run_is_not_in_flight(app_ctx):
|
||||
# e.g. the click landed just after the run finished — 409, not a silent no-op.
|
||||
client, app = app_ctx
|
||||
app.state.job_service.cancel = lambda run_id, *, power_off=False: False
|
||||
assert client.post("/api/jobs/cancel", json={"run_id": 7}).status_code == 409
|
||||
|
||||
|
||||
def test_backup_run_conflict_when_busy(app_ctx):
|
||||
client, app = app_ctx
|
||||
|
||||
@@ -370,6 +404,19 @@ def test_run_not_found(app_ctx):
|
||||
assert client.get("/api/runs/999999").status_code == 404
|
||||
|
||||
|
||||
def test_run_summary_carries_guests_ok(app_ctx):
|
||||
# The history table shows how many guests a run backed up, so the summary (not just the
|
||||
# detail) has to carry it — /api/runs is the only request that view makes per poll.
|
||||
client, app = app_ctx
|
||||
_inject(app, reachable=True)
|
||||
run_id = client.post("/api/backup/run").json()["run_id"]
|
||||
_wait_run(client, run_id)
|
||||
|
||||
summary = next(r for r in client.get("/api/runs").json() if r["id"] == run_id)
|
||||
assert summary["guests_ok"] == client.get(f"/api/runs/{run_id}").json()["guests_ok"]
|
||||
assert summary["guests_ok"] is not None
|
||||
|
||||
|
||||
def test_tasklog_empty_when_nothing_ran(app_ctx):
|
||||
client, _app = app_ctx
|
||||
assert client.get("/api/tasklog").json() == {"run_id": None, "lines": []}
|
||||
|
||||
@@ -6,6 +6,7 @@ from fakes import FakePbs, FakePower, FakePve, make_deps
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import Config
|
||||
from app.connectors.errors import ConnectorError
|
||||
from app.connectors.pve import Guest
|
||||
from app.db import session_scope
|
||||
from app.db.models import (
|
||||
@@ -541,3 +542,88 @@ def test_failed_cycle_notifies_with_failure_content(temp_db):
|
||||
# Failure title (not the success one); body surfaces the recorded error.
|
||||
assert captured["title"] == _pack(cfg.app.language)["failure"]["title"]
|
||||
assert captured["run"].error in captured["body"]
|
||||
|
||||
|
||||
# --- cancellation (11.2) -----------------------------------------------------
|
||||
|
||||
|
||||
def _cancelled_deps(*, power_off: bool = False, **kw):
|
||||
"""Deps whose cancel flag is already set, so the first task wait bails out."""
|
||||
deps, pve, pbs, power = make_deps(**kw)
|
||||
deps.cancelled = lambda: True
|
||||
deps.cancel_power_off = lambda: power_off
|
||||
return deps, pve, pbs, power
|
||||
|
||||
|
||||
def test_cancel_stops_the_remote_vzdump_task(temp_config, temp_db):
|
||||
# Abandoning the wait isn't enough: vzdump would keep running on PVE while Joulenap
|
||||
# considers itself idle, and the next run would collide with it.
|
||||
deps, pve, _pbs, power = _cancelled_deps()
|
||||
run_id = _run(_config(), deps)
|
||||
|
||||
assert pve.stopped == ["UPID:pve:backup"]
|
||||
with session_scope() as session:
|
||||
run = session.get(Run, run_id)
|
||||
assert run.status == RunStatus.ABORTED
|
||||
assert run.error == "Cancelled by user"
|
||||
assert power.powered_off is False # default: leave the box on
|
||||
|
||||
|
||||
def test_cancel_powers_off_when_the_stop_dialog_asked_for_it(temp_config, temp_db):
|
||||
deps, _pve, _pbs, power = _cancelled_deps(power_off=True)
|
||||
_run(_config(), deps)
|
||||
assert power.powered_off is True
|
||||
|
||||
|
||||
def test_cancel_before_the_pbs_wakes_does_not_try_to_power_off(temp_config, temp_db):
|
||||
# Cancelling during the wake wait: the box may never have come up, so an SSH poweroff
|
||||
# would just fail and leave a spurious failed step on the run.
|
||||
deps, _pve, _pbs, power = _cancelled_deps(power_off=True, reachable=False)
|
||||
run_id = _run(_config(), deps)
|
||||
|
||||
assert power.powered_off is False
|
||||
with session_scope() as session:
|
||||
run = session.get(Run, run_id)
|
||||
assert run.status == RunStatus.ABORTED
|
||||
steps = {s.name: s for s in run.steps}
|
||||
assert StepName.POWEROFF not in steps
|
||||
assert steps[StepName.WAIT].status != StepStatus.SUCCESS
|
||||
|
||||
|
||||
def test_cancel_does_not_notify(temp_config, temp_db):
|
||||
# The user pressed Stop and is standing at the UI; a "backup aborted" push is noise.
|
||||
sent = []
|
||||
deps, _pve, _pbs, _power = _cancelled_deps(notify=lambda c, r, d=None: sent.append(r))
|
||||
_run(_config(), deps)
|
||||
assert sent == []
|
||||
|
||||
|
||||
def test_cancel_records_the_stop_in_the_activity_log(temp_config, temp_db):
|
||||
deps, _pve, _pbs, _power = _cancelled_deps()
|
||||
run_id = _run(_config(), deps)
|
||||
with session_scope() as session:
|
||||
messages = [
|
||||
e.message
|
||||
for e in session.scalars(select(LogEvent).where(LogEvent.run_id == run_id)).all()
|
||||
]
|
||||
assert any("stop task" in m for m in messages)
|
||||
|
||||
|
||||
def test_cancel_still_ends_the_run_when_the_remote_stop_fails(temp_config, temp_db):
|
||||
# A refused stop must not leave the lock held — that failure mode *is* finding 11.2.
|
||||
deps, pve, _pbs, _power = _cancelled_deps()
|
||||
|
||||
def boom(_upid):
|
||||
raise ConnectorError("403 no permission")
|
||||
|
||||
pve.stop_task = boom
|
||||
run_id = _run(_config(), deps)
|
||||
|
||||
with session_scope() as session:
|
||||
run = session.get(Run, run_id)
|
||||
assert run.status == RunStatus.ABORTED
|
||||
messages = [
|
||||
e.message
|
||||
for e in session.scalars(select(LogEvent).where(LogEvent.run_id == run_id)).all()
|
||||
]
|
||||
assert any("could not stop" in m for m in messages)
|
||||
|
||||
@@ -46,6 +46,23 @@ def test_unknown_key_rejected(tmp_path: Path):
|
||||
load_config(p)
|
||||
|
||||
|
||||
def test_legacy_auto_include_new_is_dropped_not_rejected(tmp_path: Path):
|
||||
# BE-C5: the field was removed from the schema, but every pre-0.6.0 config.yaml still has
|
||||
# it on disk. extra="forbid" would turn that into a startup failure after a container
|
||||
# pull, so the loader strips it instead — and the next save writes the file without it.
|
||||
p = tmp_path / "config.yaml"
|
||||
p.write_text(
|
||||
"backup:\n guests:\n mode: include\n auto_include_new: true\n list: [100]\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
cfg = load_config(p)
|
||||
assert cfg.backup.guests.mode == "include"
|
||||
assert cfg.backup.guests.list == [100]
|
||||
assert not hasattr(cfg.backup.guests, "auto_include_new")
|
||||
save_config(cfg, p)
|
||||
assert "auto_include_new" not in p.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_roundtrip_save_load(tmp_path: Path):
|
||||
cfg = load_config(EXAMPLE)
|
||||
cfg.app.auth.password_hash = "$2b$12$abcdefghijklmnopqrstuv"
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""GET /metrics — Prometheus exposition (11.11): auth, format, and value mapping."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
from fakes import make_deps
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.db import session_scope
|
||||
from app.db.guest_backups import upsert_last_backups
|
||||
from app.db.models import Run, RunKind, RunStatus, RunTrigger
|
||||
from app.jobs import JobService
|
||||
from app.main import create_app
|
||||
|
||||
KEY = "metrics-key-123"
|
||||
|
||||
# name{label="v",...} value — the exposition format's sample line.
|
||||
SAMPLE = re.compile(r'^[a-zA-Z_:][a-zA-Z0-9_:]*(\{[^}]*\})? -?[0-9.eE+-]+$')
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(temp_config, temp_db, monkeypatch):
|
||||
monkeypatch.setattr("app.connectors.net.tcp_reachable", lambda *a, **k: False)
|
||||
app = create_app()
|
||||
app.state.config_store.update(lambda c: setattr(c.app, "api_key", KEY))
|
||||
deps, _pve, _pbs, _power = make_deps()
|
||||
app.state.job_service = JobService(app.state.config_store, deps=deps)
|
||||
with TestClient(app) as c:
|
||||
yield c
|
||||
|
||||
|
||||
def _scrape(client) -> str:
|
||||
r = client.get("/metrics", params={"key": KEY})
|
||||
assert r.status_code == 200
|
||||
return r.text
|
||||
|
||||
|
||||
def _value(text: str, name: str) -> float | None:
|
||||
"""The value of the first sample of ``name`` (exact family match), or None if absent."""
|
||||
for line in text.splitlines():
|
||||
if line.startswith("#"):
|
||||
continue
|
||||
head, _, value = line.rpartition(" ")
|
||||
if head == name or head.startswith(name + "{"):
|
||||
return float(value)
|
||||
return None
|
||||
|
||||
|
||||
def _add_run(**kw) -> int:
|
||||
with session_scope() as session:
|
||||
run = Run(
|
||||
kind=kw.get("kind", RunKind.CYCLE),
|
||||
trigger=RunTrigger.SCHEDULED,
|
||||
status=kw.get("status", RunStatus.SUCCESS),
|
||||
started_at=kw.get("started_at", datetime(2026, 6, 28, 4, 0, tzinfo=UTC)),
|
||||
finished_at=kw.get("finished_at", datetime(2026, 6, 28, 4, 1, 23, tzinfo=UTC)),
|
||||
guests_ok=kw.get("guests_ok", 4),
|
||||
)
|
||||
session.add(run)
|
||||
session.flush()
|
||||
return run.id
|
||||
|
||||
|
||||
# --- auth --------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_403_when_no_api_key_is_configured(client):
|
||||
# Same contract as /api/dashboard: 403 = the integration is off, not a bad credential.
|
||||
client.app.state.config_store.update(lambda c: setattr(c.app, "api_key", ""))
|
||||
assert client.get("/metrics").status_code == 403
|
||||
|
||||
|
||||
def test_401_without_a_key(client):
|
||||
assert client.get("/metrics").status_code == 401
|
||||
|
||||
|
||||
def test_401_with_the_wrong_key(client):
|
||||
assert client.get("/metrics", params={"key": "nope"}).status_code == 401
|
||||
assert client.get("/metrics", headers={"X-API-Key": "nope"}).status_code == 401
|
||||
|
||||
|
||||
def test_accepts_the_key_as_a_header_too(client):
|
||||
# Prometheus scrape configs use ?key= (params:), dashboards tend to use the header.
|
||||
assert client.get("/metrics", headers={"X-API-Key": KEY}).status_code == 200
|
||||
|
||||
|
||||
# --- format ------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_content_type_is_the_prometheus_text_format(client):
|
||||
r = client.get("/metrics", params={"key": KEY})
|
||||
assert r.headers["content-type"].startswith("text/plain")
|
||||
assert "version=0.0.4" in r.headers["content-type"]
|
||||
|
||||
|
||||
def test_every_line_is_a_valid_help_type_or_sample(client):
|
||||
_add_run()
|
||||
upsert_last_backups_now({101: datetime(2026, 6, 28, 4, 1, tzinfo=UTC)})
|
||||
text = _scrape(client)
|
||||
|
||||
families = set()
|
||||
for line in text.splitlines():
|
||||
if not line:
|
||||
continue
|
||||
if line.startswith("# HELP ") or line.startswith("# TYPE "):
|
||||
families.add(line.split()[2])
|
||||
continue
|
||||
assert SAMPLE.match(line), f"not a valid sample line: {line!r}"
|
||||
assert line.startswith("joulenap_"), f"unprefixed metric: {line!r}"
|
||||
assert families, "no metric families emitted"
|
||||
|
||||
|
||||
def test_every_sample_family_is_declared(client):
|
||||
_add_run()
|
||||
text = _scrape(client)
|
||||
declared = {ln.split()[2] for ln in text.splitlines() if ln.startswith("# TYPE ")}
|
||||
for line in text.splitlines():
|
||||
if line.startswith("#") or not line:
|
||||
continue
|
||||
name = line.split("{")[0].split(" ")[0]
|
||||
assert name in declared, f"{name} emitted without a # TYPE line"
|
||||
|
||||
|
||||
# --- values ------------------------------------------------------------------
|
||||
|
||||
|
||||
def upsert_last_backups_now(latest: dict[int, datetime]) -> None:
|
||||
with session_scope() as session:
|
||||
upsert_last_backups(session, {v: int(ts.timestamp()) for v, ts in latest.items()})
|
||||
|
||||
|
||||
def test_reports_build_info_and_state(client):
|
||||
text = _scrape(client)
|
||||
assert "joulenap_build_info{version=" in text
|
||||
assert _value(text, "joulenap_pbs_online") == 0 # stubbed unreachable
|
||||
assert _value(text, "joulenap_scheduler_enabled") == 1
|
||||
assert _value(text, "joulenap_job_running") == 0
|
||||
|
||||
|
||||
def test_last_run_series_track_the_run_history(client):
|
||||
_add_run()
|
||||
text = _scrape(client)
|
||||
assert _value(text, "joulenap_last_run_success") == 1
|
||||
assert _value(text, "joulenap_last_run_duration_seconds") == 83
|
||||
assert _value(text, "joulenap_last_run_guests") == 4
|
||||
assert _value(text, "joulenap_last_run_timestamp_seconds") == datetime(
|
||||
2026, 6, 28, 4, 0, tzinfo=UTC
|
||||
).timestamp()
|
||||
|
||||
|
||||
def test_timestamps_keep_full_precision(client):
|
||||
# %g formatting would round a 10-digit Unix timestamp to the nearest ~1000s, so every
|
||||
# *_timestamp_seconds series would drift by up to a quarter of an hour.
|
||||
started = datetime(2026, 6, 28, 4, 0, 43, tzinfo=UTC)
|
||||
_add_run(started_at=started, finished_at=started + timedelta(seconds=83))
|
||||
text = _scrape(client)
|
||||
assert _value(text, "joulenap_last_run_timestamp_seconds") == started.timestamp()
|
||||
assert f"joulenap_last_run_timestamp_seconds {int(started.timestamp())}" in text
|
||||
|
||||
|
||||
def test_a_failed_last_run_reports_zero_not_absent(client):
|
||||
_add_run(status=RunStatus.FAILURE, guests_ok=None)
|
||||
text = _scrape(client)
|
||||
assert _value(text, "joulenap_last_run_success") == 0
|
||||
# guests_ok is null on a failed run -> the series is omitted rather than reported as 0.
|
||||
assert _value(text, "joulenap_last_run_guests") is None
|
||||
|
||||
|
||||
def test_absent_values_are_omitted_not_zeroed(client):
|
||||
# With no history at all, publishing 0 would graph the last backup as January 1970.
|
||||
text = _scrape(client)
|
||||
for name in (
|
||||
"joulenap_last_run_timestamp_seconds",
|
||||
"joulenap_last_run_success",
|
||||
"joulenap_last_run_duration_seconds",
|
||||
"joulenap_datastore_used_bytes",
|
||||
"joulenap_guest_last_backup_timestamp_seconds",
|
||||
):
|
||||
assert _value(text, name) is None, f"{name} should be absent, not zero"
|
||||
|
||||
|
||||
def test_per_guest_series_come_from_the_backup_cache(client):
|
||||
stamp = datetime(2026, 6, 28, 2, 30, tzinfo=UTC)
|
||||
upsert_last_backups_now({101: stamp, 102: stamp - timedelta(days=3)})
|
||||
text = _scrape(client)
|
||||
assert (
|
||||
f'joulenap_guest_last_backup_timestamp_seconds{{vmid="101"}} {int(stamp.timestamp())}'
|
||||
in text
|
||||
)
|
||||
assert 'joulenap_guest_last_backup_timestamp_seconds{vmid="102"}' in text
|
||||
|
||||
|
||||
def test_run_counts_are_grouped_by_kind_and_status(client):
|
||||
_add_run()
|
||||
_add_run()
|
||||
_add_run(kind=RunKind.GC, status=RunStatus.FAILURE)
|
||||
text = _scrape(client)
|
||||
assert 'joulenap_runs_recent{kind="cycle",status="success"} 2' in text
|
||||
assert 'joulenap_runs_recent{kind="gc",status="failure"} 1' in text
|
||||
|
||||
|
||||
def test_in_flight_runs_are_not_counted_as_history(client):
|
||||
# A RUNNING row has no outcome yet; counting it would show a phantom status bucket.
|
||||
_add_run(status=RunStatus.RUNNING, finished_at=None)
|
||||
text = _scrape(client)
|
||||
assert "status=\"running\"" not in text
|
||||
@@ -36,6 +36,22 @@ def test_wait_until_reachable_succeeds_after_retries():
|
||||
assert calls["n"] == 3
|
||||
|
||||
|
||||
def test_wait_until_reachable_gives_up_immediately_when_cancelled():
|
||||
# A cancelled run must not sit through the full wake timeout (11.2). Reported as False,
|
||||
# same as a timeout, because "the PBS isn't up" is the state the caller has to handle.
|
||||
with mock.patch("socket.create_connection", side_effect=OSError("down")) as sock:
|
||||
ok = wait_until_reachable(
|
||||
"10.0.0.12",
|
||||
8007,
|
||||
timeout=600,
|
||||
interval=0,
|
||||
sleep=lambda _s: None,
|
||||
should_cancel=lambda: True,
|
||||
)
|
||||
assert ok is False
|
||||
assert sock.call_count == 0 # bailed before even trying to connect
|
||||
|
||||
|
||||
def test_wait_until_reachable_times_out():
|
||||
with mock.patch("socket.create_connection", side_effect=OSError("down")):
|
||||
ok = wait_until_reachable(
|
||||
|
||||
@@ -162,8 +162,10 @@ def test_ntfy_http_uses_insecure_scheme():
|
||||
# --- messages ----------------------------------------------------------------
|
||||
|
||||
|
||||
def _run(status: RunStatus, *, error: str | None = None) -> Run:
|
||||
run = Run(kind=RunKind.CYCLE, trigger=RunTrigger.MANUAL, status=status, error=error)
|
||||
def _run(
|
||||
status: RunStatus, *, error: str | None = None, kind: RunKind = RunKind.CYCLE
|
||||
) -> Run:
|
||||
run = Run(kind=kind, trigger=RunTrigger.MANUAL, status=status, error=error)
|
||||
run.started_at = datetime(2026, 6, 28, 4, 0, 0, tzinfo=UTC)
|
||||
run.finished_at = datetime(2026, 6, 28, 4, 1, 23, tzinfo=UTC)
|
||||
return run
|
||||
@@ -202,6 +204,35 @@ def test_run_message_failure_includes_error_and_locale():
|
||||
assert "vzdump failed" in body
|
||||
|
||||
|
||||
def test_run_message_title_names_the_kind_that_ran():
|
||||
# A verify or GC cycle must not report itself as a backup (doc-gap #7): a scheduled
|
||||
# verify failure used to notify "backup failed".
|
||||
verify = build_run_message(Config(), _run(RunStatus.FAILURE, kind=RunKind.VERIFY))[0]
|
||||
assert "verification failed" in verify
|
||||
assert "backup" not in verify
|
||||
gc = build_run_message(Config(), _run(RunStatus.SUCCESS, kind=RunKind.GC))[0]
|
||||
assert "garbage collection succeeded" in gc
|
||||
assert "backup" not in gc
|
||||
|
||||
|
||||
def test_run_message_kind_titles_are_localized():
|
||||
cfg = Config()
|
||||
cfg.app.language = "it"
|
||||
# Italian agrees in gender with the noun — "verifica fallita", not "fallito".
|
||||
assert "verifica fallita" in build_run_message(
|
||||
cfg, _run(RunStatus.FAILURE, kind=RunKind.VERIFY)
|
||||
)[0]
|
||||
|
||||
|
||||
def test_run_message_unmapped_kind_falls_back_to_the_backup_title():
|
||||
# A backup cycle keeps today's wording, and a kind with no block of its own degrades to
|
||||
# it rather than raising.
|
||||
assert "backup succeeded" in build_run_message(Config(), _run(RunStatus.SUCCESS))[0]
|
||||
assert "backup succeeded" in build_run_message(
|
||||
Config(), _run(RunStatus.SUCCESS, kind=RunKind.BACKUP)
|
||||
)[0]
|
||||
|
||||
|
||||
def _woke() -> RunStep:
|
||||
"""A completed WAIT step — the PBS came up, so 'left on' hinges only on power-off."""
|
||||
return RunStep(name=StepName.WAIT, status=StepStatus.SUCCESS)
|
||||
|
||||
@@ -7,7 +7,7 @@ from urllib.parse import parse_qs
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.connectors.errors import ApiError, TaskError
|
||||
from app.connectors.errors import ApiError, TaskCancelled, TaskError
|
||||
from app.connectors.pve import PveClient, build_prune_string
|
||||
|
||||
|
||||
@@ -123,6 +123,51 @@ def test_wait_task_timeout():
|
||||
)
|
||||
|
||||
|
||||
def test_wait_task_cancels_without_waiting_for_the_task():
|
||||
"""A cancel flag breaks the wait even though the task is still running (11.2)."""
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return json_data({"status": "running"})
|
||||
|
||||
with pytest.raises(TaskCancelled):
|
||||
make_client(handler).wait_task(
|
||||
"UPID:x",
|
||||
poll_interval=0,
|
||||
sleep=lambda _s: None,
|
||||
should_cancel=lambda: True,
|
||||
)
|
||||
|
||||
|
||||
def test_wait_task_keeps_waiting_while_cancel_is_false():
|
||||
# The probe must be consulted per poll, not once — a False mustn't end the wait.
|
||||
polls = {"n": 0}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
polls["n"] += 1
|
||||
if polls["n"] < 3:
|
||||
return json_data({"status": "running"})
|
||||
return json_data({"status": "stopped", "exitstatus": "OK"})
|
||||
|
||||
status = make_client(handler).wait_task(
|
||||
"UPID:x", poll_interval=0, sleep=lambda _s: None, should_cancel=lambda: False
|
||||
)
|
||||
assert status["exitstatus"] == "OK"
|
||||
assert polls["n"] == 3
|
||||
|
||||
|
||||
def test_stop_task_deletes_the_task():
|
||||
seen = {}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen["method"] = request.method
|
||||
seen["path"] = request.url.path
|
||||
return json_data(None)
|
||||
|
||||
make_client(handler).stop_task("UPID:x")
|
||||
assert seen["method"] == "DELETE"
|
||||
assert seen["path"].endswith("/nodes/pve/tasks/UPID:x")
|
||||
|
||||
|
||||
def test_task_log_parses_offset_lines():
|
||||
seen = {}
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from fakes import make_deps
|
||||
@@ -119,3 +120,83 @@ def test_submit_releases_lock_if_thread_fails_to_start(temp_config, temp_db, mon
|
||||
run = s.scalars(select(Run)).one()
|
||||
assert run.status == RunStatus.FAILURE
|
||||
assert run.finished_at is not None
|
||||
|
||||
|
||||
# --- cancellation (11.2) -----------------------------------------------------
|
||||
|
||||
|
||||
def test_cancel_ends_the_run_aborted_and_frees_the_lock(temp_config, temp_db):
|
||||
# The point of 11.2: a stuck run blocks every later job *and* manual power-off until
|
||||
# restart. After a cancel the lock must be free and a new run must start immediately.
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def blocking_wol(_config):
|
||||
started.set()
|
||||
assert release.wait(timeout=5)
|
||||
|
||||
deps, _pve, _pbs, _power = make_deps(wol=blocking_wol)
|
||||
service = JobService(ConfigStore.load_or_create(), deps=deps)
|
||||
|
||||
run_id = service.submit_backup()
|
||||
assert started.wait(timeout=5)
|
||||
assert service.cancel(run_id) is True
|
||||
release.set()
|
||||
|
||||
deadline = time.monotonic() + 5
|
||||
while service.is_running and time.monotonic() < deadline:
|
||||
time.sleep(0.02)
|
||||
assert service.is_running is False
|
||||
|
||||
with session_scope() as session:
|
||||
run = session.get(Run, run_id)
|
||||
assert run.status == RunStatus.ABORTED
|
||||
assert "ancelled" in (run.error or "")
|
||||
# The lock is genuinely free: another run starts rather than raising.
|
||||
assert service.run_gc() != run_id
|
||||
|
||||
|
||||
def test_cancel_refuses_a_run_that_is_not_the_one_in_flight(temp_config, temp_db):
|
||||
# A click landing as one run ends and the next begins must not stop the new run.
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def blocking_wol(_config):
|
||||
started.set()
|
||||
assert release.wait(timeout=5)
|
||||
|
||||
deps, _pve, _pbs, _power = make_deps(wol=blocking_wol)
|
||||
service = JobService(ConfigStore.load_or_create(), deps=deps)
|
||||
|
||||
run_id = service.submit_backup()
|
||||
assert started.wait(timeout=5)
|
||||
try:
|
||||
assert service.cancel(run_id + 999) is False
|
||||
finally:
|
||||
release.set()
|
||||
deadline = time.monotonic() + 5
|
||||
while service.is_running and time.monotonic() < deadline:
|
||||
time.sleep(0.02)
|
||||
|
||||
with session_scope() as session:
|
||||
assert session.get(Run, run_id).status == RunStatus.SUCCESS # ran to completion
|
||||
|
||||
|
||||
def test_cancel_is_refused_when_nothing_is_running(temp_config, temp_db):
|
||||
deps, _pve, _pbs, _power = make_deps()
|
||||
service = JobService(ConfigStore.load_or_create(), deps=deps)
|
||||
run_id = service.run_backup()
|
||||
assert service.cancel(run_id) is False
|
||||
|
||||
|
||||
def test_a_stale_cancel_does_not_kill_the_next_run(temp_config, temp_db):
|
||||
# Cancel arrives moments before the run ends on its own; the flag must not leak into
|
||||
# the run that starts next.
|
||||
deps, _pve, _pbs, _power = make_deps()
|
||||
service = JobService(ConfigStore.load_or_create(), deps=deps)
|
||||
first = service.run_backup()
|
||||
service.cancel(first) # refused (nothing running), but prove the state is clean anyway
|
||||
|
||||
second = service.run_backup()
|
||||
with session_scope() as session:
|
||||
assert session.get(Run, second).status == RunStatus.SUCCESS
|
||||
|
||||
+13
-7
@@ -6,14 +6,15 @@
|
||||
|
||||
app:
|
||||
language: en # UI language: en, it, ...
|
||||
theme: dark # dark | light
|
||||
theme: dark # reserved — the UI is dark-only today; this value is not read
|
||||
port: 8080 # web UI port
|
||||
timezone: "" # IANA tz for the scheduler, e.g. "Europe/Rome".
|
||||
# Empty -> the TZ env var, then UTC. Set this (or TZ) or
|
||||
# "backup at 02:00" runs at 02:00 UTC in a container.
|
||||
secret_key: "CHANGE_ME" # random string for session signing
|
||||
# Read-only key for the dashboard integration endpoint (GET /api/dashboard).
|
||||
# Leave empty to disable. Generate/rotate it from Settings -> Integrations in the UI.
|
||||
# Read-only key for the machine-readable endpoints: GET /api/dashboard (dashboard
|
||||
# widgets) and GET /metrics (Prometheus). Leave empty to disable both.
|
||||
# Generate/rotate it from Settings -> Integrations in the UI.
|
||||
api_key: ""
|
||||
# Ask GitHub once a day whether a newer Joulenap release exists, and show a footer
|
||||
# badge if so. Off by default: the app makes no outbound internet call unless you
|
||||
@@ -38,7 +39,10 @@ pve:
|
||||
port: 8006
|
||||
node: "" # node name, e.g. pve
|
||||
verify_tls: false # self-signed cert on most homelabs
|
||||
api_token_id: "" # e.g. "root@pam!joulenap" — token with PVEAuditor (VM.Audit) + VM.Backup + Datastore.AllocateSpace
|
||||
# Token privileges: VM.Audit, VM.Backup, Datastore.Audit, Datastore.AllocateSpace and
|
||||
# Datastore.Allocate on the PBS storage. The last one is required for vzdump's
|
||||
# retention/prune (it deletes old backups) — without it, backups run but pruning fails.
|
||||
api_token_id: "" # e.g. "root@pam!joulenap"
|
||||
api_token_secret: ""
|
||||
storage_id: "" # the PBS storage as named in PVE (Datacenter > Storage), e.g. pbs
|
||||
|
||||
@@ -49,7 +53,9 @@ pbs:
|
||||
port: 8007
|
||||
datastore: "" # e.g. backup
|
||||
fingerprint: "" # PBS dashboard > Show Fingerprint, e.g. "aa:bb:cc:..."
|
||||
api_token_id: "" # e.g. "root@pam!joulenap" — token for GC / status (DatastoreAdmin or similar)
|
||||
# Token privileges: DatastoreAdmin on the datastore (status + start GC) plus Audit on
|
||||
# /system (read-only node CPU/RAM/network for the dashboard).
|
||||
api_token_id: "" # e.g. "root@pam!joulenap"
|
||||
api_token_secret: ""
|
||||
# Wake-on-LAN
|
||||
mac: "" # PBS NIC MAC, e.g. "00:11:22:33:44:55"
|
||||
@@ -70,8 +76,8 @@ backup:
|
||||
min_free_percent: 0 # abort backup if PBS datastore free space is below this % (0 = disabled)
|
||||
guests:
|
||||
mode: all # all | include | exclude
|
||||
# auto_include_new applies when mode = all
|
||||
auto_include_new: true
|
||||
# A newly created VM/CT is backed up automatically in "all" and "exclude" mode, but NOT
|
||||
# in "include" mode — that list is explicit, so add new guests to it yourself.
|
||||
# used when mode = include / exclude
|
||||
list: [] # e.g. [104, 106, 202]
|
||||
retention:
|
||||
|
||||
+21
-8
@@ -12,13 +12,14 @@
|
||||
|
||||
- **Web UI** (frontend): single-page app. Talks to the backend over the REST API below.
|
||||
- **Backend / API**: serves the UI, exposes the REST API, holds the scheduler, runs the backup cycle, manages config.
|
||||
- **Scheduler**: in-process (APScheduler). Cron-style triggers for the backup job and the scheduled verify; GC has no trigger of its own — it runs as a step of the backup cycle. Re-armed whenever config changes.
|
||||
- **Scheduler**: in-process (APScheduler). Three jobs: cron-style triggers for the backup job and the scheduled verify, plus a daily history-prune job (armed independently of the backup config, so history is trimmed even while backups are disabled). GC has no trigger of its own — it runs as a step of the backup cycle. Re-armed whenever config changes.
|
||||
- **Connectors**:
|
||||
- `pve` — PVE API client (list guests, trigger `vzdump`, read task status).
|
||||
- `pbs` — PBS API client (datastore status, start/poll Garbage Collection, verify). TLS-pinned to the fingerprint stored at setup (rejects a changed cert).
|
||||
- `wol` — sends the Wake-on-LAN magic packet on the LAN.
|
||||
- `power` — SSH to PBS for `poweroff`, verified against `data/known_hosts` (host key confirmed in the wizard).
|
||||
- `notify` — Apprise / Telegram / ntfy / Discord / email senders.
|
||||
- `update` — asks GitHub once a day whether a newer release exists (opt-in via `app.update_check`; no outbound call when off).
|
||||
- **Store**: `config.yaml` for settings; a small SQLite DB (`data/`) for run history and logs.
|
||||
|
||||
|
||||
@@ -29,23 +30,30 @@
|
||||
3. **Backup**: trigger `vzdump` via PVE API for the selected guests, to `pve.storage_id`, with `mode` and `retention` (prune-backups). Poll the task to completion.
|
||||
4. **Maintenance** (if due): start PBS **GC** via PBS API and **wait** for it to finish; optional verify.
|
||||
5. **Power-off**: on success, SSH `poweroff` to PBS. On failure, leave it on for inspection.
|
||||
6. **Notify**: send result (success/failure, durations, sizes) on the enabled channels.
|
||||
6. **Notify**: send result (success/failure, duration, guest count, datastore usage) on the enabled channels.
|
||||
|
||||
All steps are logged to the DB and exposed via `/api/logs`.
|
||||
Two sibling cycles reuse the same wake/power-off machinery: a **GC cycle** (wake → GC → power-off, run on demand from the dashboard) and a **verify cycle** (wake → verify → power-off, on its own cron schedule). Either can be asked to leave the PBS awake afterwards — the `{keep_on}` flag on the manual endpoints.
|
||||
|
||||
All steps are logged to the DB and exposed via `/api/logs`; while a run is in progress the raw PVE/PBS task output is tailed into `/api/tasklog` for the UI's task-log panel.
|
||||
|
||||
|
||||
## REST API
|
||||
|
||||
Everything is served under `/api`. Auth is a signed **session cookie** started by `/api/login`; every endpoint requires it except `/api/health`, `/api/auth/status`, `/api/auth/setup` and `/api/login`.
|
||||
Everything is served under `/api`. Auth is a signed **session cookie** started by `/api/login`; every endpoint requires it except `/api/health`, `/api/auth/status`, `/api/auth/setup` and `/api/login` — plus `/api/dashboard`, which is deliberately outside the session and authenticated by its own read-only API key instead.
|
||||
|
||||
**Auth & account**
|
||||
**Health & meta**
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/health` | version + liveness (used by the Docker healthcheck) |
|
||||
| GET | `/api/update` | running version, plus the latest GitHub release when `app.update_check` is on (cached 24h; no outbound call when off) |
|
||||
| GET | `/api/config/yaml` | the redacted config serialised as YAML, for the Advanced tab's editor |
|
||||
| PUT | `/api/config/yaml` | apply an edited YAML document (same validation and merge as `PUT /api/config`) |
|
||||
| GET | `/api/dashboard` | flat, read-only status for external dashboards — **API-key auth** (`X-API-Key` header or `?key=`), not the session cookie. See [`INTEGRATIONS.md`](INTEGRATIONS.md) |
|
||||
| GET | `/metrics` | Prometheus exposition for Grafana — same API key. The **one route outside `/api`**, because `/metrics` is Prometheus's default `metrics_path` |
|
||||
|
||||
**Auth & account**
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/api/auth/status` | whether first-run setup is still needed / already signed in |
|
||||
| POST | `/api/auth/setup` | first run: create the admin account |
|
||||
| POST | `/api/login` | authenticate, start session |
|
||||
@@ -60,6 +68,10 @@ Everything is served under `/api`. Auth is a signed **session cookie** started b
|
||||
| GET | `/api/status` | scheduler state, next/last run, PBS power, datastore + node load |
|
||||
| GET | `/api/config` | current config (secrets redacted) |
|
||||
| PUT | `/api/config` | validate + save config, re-arm scheduler (the "Apply changes" action) |
|
||||
| GET | `/api/config/yaml` | the redacted config serialised as YAML, for the Advanced tab's editor |
|
||||
| PUT | `/api/config/yaml` | apply an edited YAML document (same validation and merge as `PUT /api/config`) |
|
||||
| POST | `/api/config/api-key` | generate/rotate the dashboard-integration API key (returned once) |
|
||||
| DELETE | `/api/config/api-key` | clear the key, disabling `/api/dashboard` |
|
||||
| GET | `/api/guests` | list CTs/VMs from PVE (id, name, type) for the selection panel |
|
||||
| POST | `/api/scheduler/toggle` | enable/disable the backup job (atomic switch) |
|
||||
|
||||
@@ -69,6 +81,7 @@ Everything is served under `/api`. Auth is a signed **session cookie** started b
|
||||
|---|---|---|
|
||||
| POST | `/api/backup/run` | run a backup cycle now (optional `{keep_on}` to leave the PBS on) |
|
||||
| POST | `/api/gc/run` | run a GC cycle now: wake → GC → power-off (optional `{keep_on}` to leave the PBS on) |
|
||||
| POST | `/api/jobs/cancel` | stop the run in flight (`{run_id}`, optional `{power_off}`); also stops the PVE/PBS task behind it. 202 = accepted, not finished — cancellation is cooperative |
|
||||
| POST | `/api/power/on` | wake the PBS (Wake-on-LAN) |
|
||||
| POST | `/api/power/off` | power the PBS off (SSH) |
|
||||
| POST | `/api/wol/test` | send a test magic packet |
|
||||
@@ -103,6 +116,6 @@ UI convention: text fields are saved with an explicit **Apply changes** (`PUT /a
|
||||
|
||||
## Permissions cheat-sheet
|
||||
|
||||
- **PVE token**: `VM.Audit` (list guests) + `VM.Backup` + `Datastore.AllocateSpace` **and `Datastore.Allocate`** on the PBS storage (the latter is required for vzdump's retention/prune, which deletes old backups). Quick setup creates a `Joulenap` role with exactly these privileges.
|
||||
- **PVE token**: `VM.Audit` (list guests) + `VM.Backup` + `Datastore.Audit` + `Datastore.AllocateSpace` **and `Datastore.Allocate`** on the PBS storage (the last is required for vzdump's retention/prune, which deletes old backups). Quick setup creates a `Joulenap` role with exactly these privileges (`connectors/provision.py`).
|
||||
- **PBS token**: `DatastoreAdmin` on the datastore (status + start GC) plus `Audit` on `/system` (read-only node CPU/RAM/network for the dashboard). PBS has no API to create custom roles, so quick setup grants these built-ins scoped by path.
|
||||
- **SSH to PBS**: dedicated key; ideally a forced command on PBS that only allows `poweroff`.
|
||||
|
||||
@@ -251,6 +251,35 @@ Once the UI is up at `http://<host>:8080` (same for every install path):
|
||||
**notifications** (Telegram / ntfy / email / Discord) under Settings if you want them. Use **Run
|
||||
backup now** to test the full wake → backup → power-off cycle end-to-end.
|
||||
|
||||
## The Settings tabs
|
||||
|
||||
Day-to-day you'll live on the Dashboard (schedule, guests, retention, manual runs). Everything else
|
||||
sits behind **Settings**:
|
||||
|
||||
- **Setup** — the connection wizard from step 2 above: PVE, the PBS-backed storage, Wake-on-LAN MAC
|
||||
and the poweroff SSH key. Re-run it whenever the PBS certificate is renewed or its address changes.
|
||||
- **Backup safety** — guardrails around the cycle: a minimum-free-space check that aborts rather than
|
||||
backing up onto a nearly-full datastore, how long to wait for a busy PBS to finish its own task
|
||||
before powering it off, and the verify options (a quick verify of new snapshots after each backup,
|
||||
or a full verification on its own schedule). *(The wake timeout and Wake-on-LAN retries live on the
|
||||
Dashboard, next to the schedule.)*
|
||||
- **Notifications** — Telegram, ntfy, email (SMTP) and Discord with friendly forms, plus a
|
||||
catch-all list for any other [Apprise](https://github.com/caronc/apprise) URL or plain webhook.
|
||||
Choose whether to notify on success, on failure, or both; **Send test** reports per channel.
|
||||
- **Integrations** — generate the read-only API key for the dashboard endpoint and copy a
|
||||
ready-made snippet for Homepage / Homarr / Dashy / Glance (details in
|
||||
[`INTEGRATIONS.md`](INTEGRATIONS.md)). The opt-in **update check** lives here too: off by default,
|
||||
and while it's off Joulenap makes no outbound internet call at all.
|
||||
- **Localization** — interface language and the timezone your schedule is interpreted in.
|
||||
- **Account** — change the admin username or password. Changing the password signs out every
|
||||
existing session immediately.
|
||||
- **Advanced** — the knobs with no home on the other screens: backup mode (snapshot / suspend /
|
||||
stop), a vzdump bandwidth cap, the `keep_last` and `keep_yearly` retention buckets, how long run
|
||||
history is kept, and the web port / session settings (these last ones need a restart). At the
|
||||
bottom is a **`config.yaml` editor** with syntax highlighting: it saves through exactly the same
|
||||
validation as the forms, so a bad value is rejected instead of persisted, and its **Copy** button
|
||||
gives you the whole config with the secrets redacted — handy for a bug report.
|
||||
|
||||
## Updating
|
||||
|
||||
Your `config.yaml` and data live in the mounted `data/` directory (or your native data dir), so they
|
||||
|
||||
+158
-23
@@ -1,15 +1,14 @@
|
||||
# Dashboard integration
|
||||
# Integrations
|
||||
|
||||
Joulenap exposes a small, read-only, API-key-protected JSON endpoint —
|
||||
`GET /api/dashboard` — so a homelab dashboard (Homepage, Homarr, Dashy,
|
||||
Glance, …) can poll it and show your backup status alongside your other
|
||||
services: whether the PBS is asleep/awake/backing up, when the next run is
|
||||
scheduled, how the last run went, and how full the datastore is.
|
||||
Joulenap exposes two read-only, API-key-protected endpoints for other tools:
|
||||
|
||||
This endpoint is intentionally separate from the internal `/api/status` used
|
||||
by Joulenap's own UI: it's a stable, additive-only public contract with
|
||||
plain machine-readable values (no localization, no session cookie), guarded
|
||||
by its own API key instead of a login session.
|
||||
- **`GET /api/dashboard`** — a flat JSON summary for homelab dashboards
|
||||
(Homepage, Homarr, Dashy, Glance). See below.
|
||||
- **`GET /metrics`** — Prometheus metrics for Grafana. See
|
||||
[Prometheus & Grafana](#prometheus--grafana).
|
||||
|
||||
Both use the **same API key**, generated once under **Settings →
|
||||
Integrations**; enabling the integration enables both.
|
||||
|
||||
## Enabling it
|
||||
|
||||
@@ -20,25 +19,39 @@ by its own API key instead of a login session.
|
||||
again.
|
||||
3. Pick your dashboard in the picker on that page to get a ready-to-paste
|
||||
config snippet with the key and endpoint URL already filled in.
|
||||
4. Disabling the integration (the **Disable** button) clears the key and the
|
||||
endpoint immediately starts rejecting requests again.
|
||||
4. Disabling the integration (the **Disable** button) clears the key, and
|
||||
both endpoints immediately start rejecting requests again.
|
||||
|
||||
Regenerating the key invalidates the previous one immediately — update any
|
||||
dashboard config that used the old key.
|
||||
dashboard or scrape config that used the old key.
|
||||
|
||||
## Authentication
|
||||
|
||||
Every request to `/api/dashboard` must include the API key, either as:
|
||||
Every request to `/api/dashboard` and `/metrics` must include the API key,
|
||||
either as:
|
||||
|
||||
- an **`X-API-Key` header** (preferred, wherever the dashboard supports
|
||||
custom request headers), or
|
||||
- an **`X-API-Key` header** (preferred, wherever the client supports custom
|
||||
request headers), or
|
||||
- a **`?key=<your-api-key>` query parameter** appended to the URL, for
|
||||
dashboards/widgets that can't set custom headers.
|
||||
dashboards/widgets that can't set custom headers — and for Prometheus,
|
||||
whose `params:` setting works on every version.
|
||||
|
||||
No key configured → `403 Forbidden` (integration disabled). Wrong or missing
|
||||
key → `401 Unauthorized`.
|
||||
|
||||
## Response reference
|
||||
## Dashboard integration
|
||||
|
||||
`GET /api/dashboard` lets a homelab dashboard poll Joulenap and show your
|
||||
backup status alongside your other services: whether the PBS is
|
||||
asleep/awake/backing up, when the next run is scheduled, how the last run
|
||||
went, and how full the datastore is.
|
||||
|
||||
This endpoint is intentionally separate from the internal `/api/status` used
|
||||
by Joulenap's own UI: it's a stable, additive-only public contract with
|
||||
plain machine-readable values (no localization, no session cookie), guarded
|
||||
by its own API key instead of a login session.
|
||||
|
||||
### Response reference
|
||||
|
||||
`GET /api/dashboard` returns a flat JSON object:
|
||||
|
||||
@@ -52,13 +65,13 @@ key → `401 Unauthorized`.
|
||||
| `datastore_used_bytes` | number \| null | Bytes used |
|
||||
| `datastore_total_bytes` | number \| null | Total bytes (free = total − used) |
|
||||
|
||||
## Per-dashboard setup
|
||||
### Per-dashboard setup
|
||||
|
||||
The endpoint URL is your Joulenap instance's origin plus `/api/dashboard`,
|
||||
e.g. `http://192.168.1.50:8080/api/dashboard`. Replace `<your-api-key>` with
|
||||
the key from step 2 above in every snippet below.
|
||||
|
||||
### Homepage
|
||||
#### Homepage
|
||||
|
||||
Homepage's built-in `customapi` widget maps JSON response fields directly
|
||||
onto labelled rows:
|
||||
@@ -85,7 +98,7 @@ onto labelled rows:
|
||||
format: percent
|
||||
```
|
||||
|
||||
### Glance
|
||||
#### Glance
|
||||
|
||||
Glance's `custom-api` widget fetches the JSON and renders it through a Go
|
||||
template:
|
||||
@@ -103,7 +116,7 @@ template:
|
||||
<div>Datastore: {{ .JSON.Int "datastore_used_pct" }}%</div>
|
||||
```
|
||||
|
||||
### Homarr
|
||||
#### Homarr
|
||||
|
||||
> **Note:** Homarr's widget system changed significantly in 2026. Older
|
||||
> Homarr releases only offered a generic iframe/link-style widget with no
|
||||
@@ -131,7 +144,7 @@ template:
|
||||
> fallback for the URL field instead:
|
||||
> `http://192.168.1.50:8080/api/dashboard?key=<your-api-key>`
|
||||
|
||||
### Dashy
|
||||
#### Dashy
|
||||
|
||||
> **Note:** Dashy's generic JSON widget is called `customapi` (it was
|
||||
> explicitly modeled after Homepage's widget of the same name), not a plain
|
||||
@@ -163,3 +176,125 @@ instead of from the browser. If your Dashy version predates the `customapi`
|
||||
widget, use the query-string fallback
|
||||
(`http://192.168.1.50:8080/api/dashboard?key=<your-api-key>`) with whatever
|
||||
generic widget your version offers.
|
||||
|
||||
## Prometheus & Grafana
|
||||
|
||||
`GET /metrics` exposes Joulenap's state in the Prometheus text format, so a
|
||||
homelab Prometheus can scrape it and Grafana can graph it — and, more to the
|
||||
point, so Alertmanager can tell you **when a guest stops being backed up**.
|
||||
|
||||
It's served at `/metrics` (not under `/api`) because that's Prometheus's
|
||||
default `metrics_path`.
|
||||
|
||||
A scrape reads the database and does the same one-second TCP probe the
|
||||
dashboard uses. **It never wakes the PBS**, and datastore usage and per-guest
|
||||
backup times come from Joulenap's cache, so they keep reporting while the
|
||||
box is asleep — which is most of the time, by design.
|
||||
|
||||
### Scrape config
|
||||
|
||||
Prometheus's `params:` works on every version, unlike custom scrape headers:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: joulenap
|
||||
metrics_path: /metrics
|
||||
params:
|
||||
key: ["<your-api-key>"]
|
||||
static_configs:
|
||||
- targets: ["192.168.1.50:8080"]
|
||||
```
|
||||
|
||||
A 60s `scrape_interval` is plenty — nothing here changes faster than a
|
||||
backup cycle.
|
||||
|
||||
### Metric reference
|
||||
|
||||
All metrics are gauges prefixed `joulenap_`.
|
||||
|
||||
| Metric | Labels | Meaning |
|
||||
|--------|--------|---------|
|
||||
| `joulenap_build_info` | `version` | Always 1; the label carries the running version |
|
||||
| `joulenap_pbs_online` | — | 1 if the PBS answers on its API port, 0 while asleep |
|
||||
| `joulenap_scheduler_enabled` | — | 1 if the scheduled backup job is armed |
|
||||
| `joulenap_job_running` | — | 1 while a backup, GC or verify run is in flight |
|
||||
| `joulenap_next_run_timestamp_seconds` | — | Unix time of the next scheduled backup |
|
||||
| `joulenap_last_run_timestamp_seconds` | — | When the last finished backup cycle started |
|
||||
| `joulenap_last_run_success` | — | 1 if the last finished cycle succeeded, else 0 |
|
||||
| `joulenap_last_run_duration_seconds` | — | How long that cycle took |
|
||||
| `joulenap_last_run_guests` | — | Guests backed up by that cycle |
|
||||
| `joulenap_datastore_used_bytes` | — | Datastore bytes used (last known value) |
|
||||
| `joulenap_datastore_total_bytes` | — | Datastore size in bytes (last known value) |
|
||||
| `joulenap_guest_last_backup_timestamp_seconds` | `vmid` | Each guest's most recent snapshot |
|
||||
| `joulenap_pbs_cpu_percent` | — | PBS CPU %, **only present while the PBS is awake** |
|
||||
| `joulenap_pbs_memory_percent` | — | PBS memory %, only while awake |
|
||||
| `joulenap_pbs_uptime_seconds` | — | PBS uptime, only while awake |
|
||||
| `joulenap_runs_recent` | `kind`, `status` | Finished runs in the history window |
|
||||
|
||||
Two things worth knowing before you write queries:
|
||||
|
||||
- **A value Joulenap doesn't have is an absent series, not a zero.** Before
|
||||
the first backup there is no `joulenap_last_run_timestamp_seconds` at all,
|
||||
because publishing `0` would graph your last backup as January 1970. Use
|
||||
`absent()` to alert on "never ran".
|
||||
- **`joulenap_runs_recent` is a gauge, not a counter.** The daily prune job
|
||||
deletes runs older than `maintenance.history.retention_days`, so the number
|
||||
legitimately goes *down* — `rate()` and `increase()` would be nonsense on
|
||||
it. It answers "how many failures are in my retention window", not "how
|
||||
many ever".
|
||||
|
||||
### Useful queries
|
||||
|
||||
```promql
|
||||
# Hours since each guest was last backed up
|
||||
(time() - joulenap_guest_last_backup_timestamp_seconds) / 3600
|
||||
|
||||
# Datastore usage percent
|
||||
100 * joulenap_datastore_used_bytes / joulenap_datastore_total_bytes
|
||||
|
||||
# Days until the datastore is full, from the last week's growth
|
||||
(joulenap_datastore_total_bytes - joulenap_datastore_used_bytes)
|
||||
/ (deriv(joulenap_datastore_used_bytes[7d]) * 86400)
|
||||
|
||||
# Share of recent backup cycles that succeeded
|
||||
joulenap_runs_recent{kind="cycle",status="success"}
|
||||
/ sum by () (joulenap_runs_recent{kind="cycle"})
|
||||
```
|
||||
|
||||
### Alerting rules
|
||||
|
||||
The one that justifies wiring this up at all — a guest quietly falling out
|
||||
of your backup set:
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: joulenap
|
||||
rules:
|
||||
- alert: JoulenapGuestBackupStale
|
||||
expr: time() - joulenap_guest_last_backup_timestamp_seconds > 172800
|
||||
for: 1h
|
||||
annotations:
|
||||
summary: "Guest {{ $labels.vmid }} has no backup in over 48h"
|
||||
|
||||
- alert: JoulenapLastBackupFailed
|
||||
expr: joulenap_last_run_success == 0
|
||||
for: 15m
|
||||
annotations:
|
||||
summary: "The last Joulenap backup cycle did not succeed"
|
||||
|
||||
- alert: JoulenapNeverRan
|
||||
expr: absent(joulenap_last_run_timestamp_seconds)
|
||||
for: 24h
|
||||
annotations:
|
||||
summary: "Joulenap has never completed a backup cycle"
|
||||
|
||||
- alert: JoulenapDatastoreFilling
|
||||
expr: 100 * joulenap_datastore_used_bytes / joulenap_datastore_total_bytes > 85
|
||||
for: 1h
|
||||
annotations:
|
||||
summary: "PBS datastore is over 85% full"
|
||||
```
|
||||
|
||||
Set the staleness threshold to comfortably more than your backup interval —
|
||||
`172800` (48h) suits a nightly schedule; a run that starts late or takes a
|
||||
while shouldn't page you.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "joulenap-frontend",
|
||||
"private": true,
|
||||
"version": "0.5.0",
|
||||
"version": "0.6.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -151,6 +151,8 @@ export const api = {
|
||||
req<{ enabled: boolean; next_run: string | null }>('POST', '/scheduler/toggle', { enabled }),
|
||||
runBackup: (keepOn: boolean) => req<{ run_id: number }>('POST', '/backup/run', { keep_on: keepOn }),
|
||||
runGc: (keepOn: boolean) => req<{ run_id: number }>('POST', '/gc/run', { keep_on: keepOn }),
|
||||
cancelJob: (runId: number, powerOff: boolean) =>
|
||||
req<{ run_id: number }>('POST', '/jobs/cancel', { run_id: runId, power_off: powerOff }),
|
||||
powerOn: () => req<{ ok: boolean }>('POST', '/power/on'),
|
||||
powerOff: () => req<{ ok: boolean }>('POST', '/power/off'),
|
||||
wolTest: () => req<{ sent: boolean; mac: string }>('POST', '/wol/test'),
|
||||
|
||||
@@ -17,6 +17,7 @@ export interface RunSummary {
|
||||
status: string
|
||||
started_at: string
|
||||
finished_at: string | null
|
||||
guests_ok: number | null
|
||||
error: string | null
|
||||
}
|
||||
|
||||
@@ -72,6 +73,7 @@ export interface StatusResponse {
|
||||
next_run: string | null
|
||||
job_running: boolean
|
||||
running_kind?: 'cycle' | 'gc' | 'verify' | null
|
||||
running_run_id?: number | null
|
||||
pbs_online: boolean
|
||||
last_run: RunSummary | null
|
||||
datastore: DatastoreInfo | null
|
||||
@@ -90,7 +92,6 @@ export interface GuestInfo {
|
||||
|
||||
export interface GuestsConfig {
|
||||
mode: 'all' | 'include' | 'exclude'
|
||||
auto_include_new: boolean
|
||||
list: number[]
|
||||
}
|
||||
|
||||
|
||||
@@ -12,6 +12,9 @@ export function Toggle({ on, onClick, size = 'lg' }: ToggleProps) {
|
||||
const knob = size === 'lg' ? 16 : 12
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={on}
|
||||
onClick={onClick}
|
||||
style={{
|
||||
width: w,
|
||||
|
||||
+100
-5
@@ -12,6 +12,8 @@ import type {
|
||||
NetInterface,
|
||||
PbsDerive,
|
||||
PveConnectResult,
|
||||
RunDetail,
|
||||
RunSummary,
|
||||
StatusResponse,
|
||||
TaskLogResponse,
|
||||
UserInfo,
|
||||
@@ -79,7 +81,7 @@ const CONFIG: Config = {
|
||||
mode: 'snapshot',
|
||||
bwlimit: 0,
|
||||
min_free_percent: 10,
|
||||
guests: { mode: 'include', auto_include_new: false, list: [100, 102] },
|
||||
guests: { mode: 'include', list: [100, 102] },
|
||||
retention: { keep_last: 0, keep_daily: 7, keep_weekly: 4, keep_monthly: 6, keep_yearly: 0 },
|
||||
},
|
||||
maintenance: {
|
||||
@@ -120,6 +122,7 @@ const STATUS: StatusResponse = {
|
||||
status: 'ok',
|
||||
started_at: '2026-07-08T02:30:00Z',
|
||||
finished_at: '2026-07-08T02:41:12Z',
|
||||
guests_ok: 4,
|
||||
error: null,
|
||||
},
|
||||
datastore: { used: 1_800_000_000_000, total: 2_800_000_000_000, used_pct: 63 },
|
||||
@@ -153,6 +156,88 @@ const LOGS: LogLine[] = [
|
||||
|
||||
const TASKLOG: TaskLogResponse = { lines: [], run_id: null }
|
||||
|
||||
// Run history: one of each outcome, so the table's badges, the inline error and the
|
||||
// still-running row (no finished_at -> elapsed duration) can all be seen without a backend.
|
||||
const RUNS: RunSummary[] = [
|
||||
{
|
||||
id: 45,
|
||||
kind: 'cycle',
|
||||
trigger: 'manual',
|
||||
status: 'running',
|
||||
started_at: '2026-07-09T21:28:40Z',
|
||||
finished_at: null,
|
||||
guests_ok: null,
|
||||
error: null,
|
||||
},
|
||||
{
|
||||
id: 44,
|
||||
kind: 'verify',
|
||||
trigger: 'scheduled',
|
||||
status: 'success',
|
||||
started_at: '2026-07-09T04:00:00Z',
|
||||
finished_at: '2026-07-09T04:06:31Z',
|
||||
guests_ok: null,
|
||||
error: null,
|
||||
},
|
||||
{
|
||||
id: 43,
|
||||
kind: 'cycle',
|
||||
trigger: 'scheduled',
|
||||
status: 'failure',
|
||||
started_at: '2026-07-08T22:30:00Z',
|
||||
finished_at: '2026-07-08T22:30:46Z',
|
||||
guests_ok: null,
|
||||
error: 'vzdump exited with code 255: no space left on device',
|
||||
},
|
||||
{
|
||||
id: 42,
|
||||
kind: 'cycle',
|
||||
trigger: 'scheduled',
|
||||
status: 'success',
|
||||
started_at: '2026-07-08T02:30:00Z',
|
||||
finished_at: '2026-07-08T02:41:12Z',
|
||||
guests_ok: 4,
|
||||
error: null,
|
||||
},
|
||||
{
|
||||
id: 41,
|
||||
kind: 'gc',
|
||||
trigger: 'manual',
|
||||
status: 'aborted',
|
||||
started_at: '2026-07-07T19:05:00Z',
|
||||
finished_at: '2026-07-07T19:08:02Z',
|
||||
guests_ok: null,
|
||||
error: 'PBS did not come up within 180s',
|
||||
},
|
||||
]
|
||||
|
||||
const RUN_DETAIL: Record<number, RunDetail> = {
|
||||
43: {
|
||||
...RUNS[2],
|
||||
steps: [
|
||||
{ name: 'wake', status: 'success', started_at: '2026-07-08T22:30:00Z', finished_at: '2026-07-08T22:30:02Z', detail: null },
|
||||
{ name: 'wait', status: 'success', started_at: '2026-07-08T22:30:02Z', finished_at: '2026-07-08T22:30:43Z', detail: 'reachable after 41s' },
|
||||
{ name: 'backup', status: 'failure', started_at: '2026-07-08T22:30:43Z', finished_at: '2026-07-08T22:30:46Z', detail: 'vzdump exit 255' },
|
||||
],
|
||||
logs: [
|
||||
{ id: 91, run_id: 43, ts: '2026-07-08T22:30:43Z', level: 'INFO', message: 'Starting vzdump for 4 guests' },
|
||||
{ id: 92, run_id: 43, ts: '2026-07-08T22:30:46Z', level: 'ERROR', message: 'vzdump failed: no space left on device' },
|
||||
],
|
||||
},
|
||||
42: {
|
||||
...RUNS[3],
|
||||
steps: [
|
||||
{ name: 'wake', status: 'success', started_at: '2026-07-08T02:30:00Z', finished_at: '2026-07-08T02:30:02Z', detail: null },
|
||||
{ name: 'backup', status: 'success', started_at: '2026-07-08T02:30:40Z', finished_at: '2026-07-08T02:40:05Z', detail: '4 guests' },
|
||||
{ name: 'gc', status: 'success', started_at: '2026-07-08T02:40:05Z', finished_at: '2026-07-08T02:41:00Z', detail: null },
|
||||
{ name: 'poweroff', status: 'success', started_at: '2026-07-08T02:41:00Z', finished_at: '2026-07-08T02:41:12Z', detail: null },
|
||||
],
|
||||
logs: [
|
||||
{ id: 80, run_id: 42, ts: '2026-07-08T02:41:12Z', level: 'OK', message: 'Backup finished, 4 guests' },
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
// --- setup wizard fixtures ---------------------------------------------------
|
||||
// Lets the wizard advance card-by-card with no backend: connecting PVE returns a node
|
||||
// and a PBS-backed storage, confirming that storage seeds the PBS card, checking the
|
||||
@@ -263,7 +348,6 @@ backup:
|
||||
min_free_percent: 10
|
||||
guests:
|
||||
mode: all
|
||||
auto_include_new: true
|
||||
list: []
|
||||
retention:
|
||||
keep_last: 0
|
||||
@@ -288,10 +372,10 @@ notifications:
|
||||
`
|
||||
|
||||
const ROUTES: Record<string, unknown> = {
|
||||
'GET /health': { status: 'ok', version: '0.5.0-stub' },
|
||||
'GET /health': { status: 'ok', version: '0.6.0-stub' },
|
||||
'GET /update': {
|
||||
current: '0.5.0-stub',
|
||||
latest: '0.5.0',
|
||||
current: '0.6.0-stub',
|
||||
latest: '0.6.0',
|
||||
update_available: true,
|
||||
url: 'https://github.com/Joulenap/joulenap/releases',
|
||||
},
|
||||
@@ -304,6 +388,8 @@ const ROUTES: Record<string, unknown> = {
|
||||
'PUT /config/yaml': CONFIG,
|
||||
'GET /guests': GUESTS,
|
||||
'GET /tasklog': TASKLOG,
|
||||
'GET /runs': RUNS,
|
||||
'POST /jobs/cancel': { run_id: 45 },
|
||||
'POST /wizard/pve/connect': WIZARD_PVE_CONNECT,
|
||||
'POST /wizard/storage/derive': WIZARD_STORAGE_DERIVE,
|
||||
'POST /wizard/pbs/check': WIZARD_PBS_CHECK,
|
||||
@@ -331,6 +417,15 @@ globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
let body: unknown = ROUTES[key]
|
||||
if (body === undefined && key.startsWith('GET /logs')) body = LOGS
|
||||
if (body === undefined && bare === '/logs') body = LOGS
|
||||
// ROUTES is keyed on exact paths, so /runs/{id} needs its own match.
|
||||
const runId = method === 'GET' ? /^\/runs\/(\d+)$/.exec(bare)?.[1] : undefined
|
||||
if (runId !== undefined) {
|
||||
body = RUN_DETAIL[Number(runId)] ?? {
|
||||
...(RUNS.find((r) => r.id === Number(runId)) ?? RUNS[0]),
|
||||
steps: [],
|
||||
logs: [],
|
||||
}
|
||||
}
|
||||
if (body === undefined) body = { ok: true }
|
||||
|
||||
return new Response(JSON.stringify(body), {
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import { api } from '../api/client'
|
||||
import type { RunSummary } from '../api/types'
|
||||
|
||||
// Polls GET /api/runs for the Run history view. Only runs while the view is showing, so the
|
||||
// default (activity log) view costs no extra requests; the same 8s cadence as the activity
|
||||
// log keeps a finished run from lingering as "running" for long.
|
||||
export function useRuns(active: boolean) {
|
||||
const [runs, setRuns] = useState<RunSummary[]>([])
|
||||
const [error, setError] = useState(false)
|
||||
// Same guard as useTaskLog: a slow response must not overlap the next tick.
|
||||
const inFlightRef = useRef(false)
|
||||
|
||||
const poll = useCallback(async () => {
|
||||
if (inFlightRef.current) return
|
||||
inFlightRef.current = true
|
||||
try {
|
||||
setRuns(await api.runs(50))
|
||||
setError(false)
|
||||
} catch {
|
||||
// Transient failure: keep the list we have (blanking it reads as "no runs ever", which
|
||||
// is a very different message) and flag it so the view can say so.
|
||||
setError(true)
|
||||
} finally {
|
||||
inFlightRef.current = false
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => {
|
||||
if (!active) return
|
||||
poll()
|
||||
const id = setInterval(poll, 8000)
|
||||
return () => clearInterval(id)
|
||||
}, [active, poll])
|
||||
|
||||
return { runs, error, refresh: poll }
|
||||
}
|
||||
@@ -56,6 +56,9 @@
|
||||
"manualJobs": "Manual jobs",
|
||||
"runBackup": "Run backup now",
|
||||
"runGc": "Run GC now",
|
||||
"stopBackup": "Stop backup",
|
||||
"stopGc": "Stop GC",
|
||||
"stopVerify": "Stop verify",
|
||||
"power": "Power",
|
||||
"powerOn": "Power on PBS",
|
||||
"powerOff": "Power off PBS",
|
||||
@@ -109,6 +112,30 @@
|
||||
"colLevel": "Level",
|
||||
"colMessage": "Message",
|
||||
"noLogs": "No activity yet",
|
||||
"historyViews": "Activity views",
|
||||
"runHistory": "Run history",
|
||||
"runsCount": "{{n}} runs",
|
||||
"colStarted": "Started",
|
||||
"colKind": "Job",
|
||||
"colTrigger": "Trigger",
|
||||
"colStatus": "Result",
|
||||
"colDuration": "Duration",
|
||||
"colGuests": "Guests",
|
||||
"kindBackup": "Backup",
|
||||
"kindGc": "GC",
|
||||
"kindVerify": "Verify",
|
||||
"trigger_scheduled": "scheduled",
|
||||
"trigger_manual": "manual",
|
||||
"runSuccess": "OK",
|
||||
"runFailure": "Failed",
|
||||
"runAborted": "Aborted",
|
||||
"runRunning": "Running",
|
||||
"noRuns": "No runs recorded yet.",
|
||||
"runsError": "Couldn't load the run history — retrying.",
|
||||
"runSteps": "Steps",
|
||||
"runLog": "Log",
|
||||
"noSteps": "No steps recorded.",
|
||||
"historyRetention": "History is kept for {{n}} days.",
|
||||
"taskLog": "Task log",
|
||||
"taskLogLive": "live",
|
||||
"taskLines": "{{n}} lines",
|
||||
@@ -130,6 +157,10 @@
|
||||
"gcMsg": "Garbage collection will start. If the PBS is asleep it will be woken first (up to {{timeout}}s); it can temporarily reduce datastore performance while running.",
|
||||
"gcYes": "Run GC",
|
||||
"keepOn": "Keep PBS powered on after the job",
|
||||
"stopTitle": "Stop the running job?",
|
||||
"stopMsg": "The {{job}} in progress will be told to stop, and the task running on the server is cancelled with it. Data already written is kept; the run is recorded as aborted.",
|
||||
"stopYes": "Stop job",
|
||||
"stopPowerOff": "Power the PBS off after stopping",
|
||||
"onTitle": "Power on the PBS?",
|
||||
"onMsg": "A Wake-on-LAN packet will be sent to the server. Startup can take up to {{timeout}} seconds.",
|
||||
"onYes": "Power on PBS",
|
||||
|
||||
@@ -56,6 +56,9 @@
|
||||
"manualJobs": "Job manuali",
|
||||
"runBackup": "Esegui backup ora",
|
||||
"runGc": "Esegui GC ora",
|
||||
"stopBackup": "Interrompi backup",
|
||||
"stopGc": "Interrompi GC",
|
||||
"stopVerify": "Interrompi verifica",
|
||||
"power": "Alimentazione",
|
||||
"powerOn": "Accendi PBS",
|
||||
"powerOff": "Spegni PBS",
|
||||
@@ -109,6 +112,30 @@
|
||||
"colLevel": "Livello",
|
||||
"colMessage": "Messaggio",
|
||||
"noLogs": "Nessuna attività",
|
||||
"historyViews": "Viste attività",
|
||||
"runHistory": "Cronologia",
|
||||
"runsCount": "{{n}} esecuzioni",
|
||||
"colStarted": "Inizio",
|
||||
"colKind": "Operazione",
|
||||
"colTrigger": "Avvio",
|
||||
"colStatus": "Esito",
|
||||
"colDuration": "Durata",
|
||||
"colGuests": "Guest",
|
||||
"kindBackup": "Backup",
|
||||
"kindGc": "GC",
|
||||
"kindVerify": "Verifica",
|
||||
"trigger_scheduled": "pianificato",
|
||||
"trigger_manual": "manuale",
|
||||
"runSuccess": "OK",
|
||||
"runFailure": "Fallita",
|
||||
"runAborted": "Interrotta",
|
||||
"runRunning": "In corso",
|
||||
"noRuns": "Nessuna esecuzione registrata.",
|
||||
"runsError": "Impossibile caricare la cronologia — nuovo tentativo in corso.",
|
||||
"runSteps": "Fasi",
|
||||
"runLog": "Log",
|
||||
"noSteps": "Nessuna fase registrata.",
|
||||
"historyRetention": "La cronologia viene conservata per {{n}} giorni.",
|
||||
"taskLog": "Log operazioni",
|
||||
"taskLogLive": "in diretta",
|
||||
"taskLines": "{{n}} righe",
|
||||
@@ -130,6 +157,10 @@
|
||||
"gcMsg": "Verrà avviata la garbage collection. Se il PBS è spento verrà prima acceso (fino a {{timeout}}s); può ridurre temporaneamente le prestazioni del datastore.",
|
||||
"gcYes": "Esegui GC",
|
||||
"keepOn": "Mantieni il PBS acceso dopo il job",
|
||||
"stopTitle": "Interrompere il job in corso?",
|
||||
"stopMsg": "Verrà richiesta l'interruzione del {{job}} in corso, insieme all'operazione in esecuzione sul server. I dati già scritti vengono mantenuti; l'esecuzione viene registrata come interrotta.",
|
||||
"stopYes": "Interrompi",
|
||||
"stopPowerOff": "Spegni il PBS dopo l'interruzione",
|
||||
"onTitle": "Accendere il PBS?",
|
||||
"onMsg": "Verrà inviato un pacchetto Wake-on-LAN al server. L'avvio può richiedere fino a {{timeout}} secondi.",
|
||||
"onYes": "Accendi PBS",
|
||||
|
||||
@@ -8,8 +8,9 @@ import { useRegisterDirty } from '../shell/UnsavedGuard'
|
||||
import { useTaskLog } from '../hooks/useTaskLog'
|
||||
import { buildCron, isAdvancedSchedule, parseCron } from '../utils/cron'
|
||||
import { guestsSelectionError } from '../utils/guests'
|
||||
import { ActivityLog } from './dashboard/ActivityLog'
|
||||
import { runKindLabelKey } from '../utils/status'
|
||||
import { GuestsPanel } from './dashboard/GuestsPanel'
|
||||
import { HistoryCard } from './dashboard/HistoryCard'
|
||||
import { ManualPanel } from './dashboard/ManualPanel'
|
||||
import { type SchedulerDraft, SchedulerCard } from './dashboard/SchedulerCard'
|
||||
import { StatTiles } from './dashboard/StatTiles'
|
||||
@@ -256,6 +257,32 @@ export function Dashboard({ status, refreshStatus }: DashboardProps) {
|
||||
})
|
||||
}
|
||||
|
||||
// Stop the in-flight run (11.2). Reuses the same confirm as the run actions, with the
|
||||
// toggle repurposed: default OFF, because cancelling usually means "I want the box now".
|
||||
const stopAction = () => {
|
||||
const runId = status?.running_run_id
|
||||
if (typeof runId !== 'number') return
|
||||
const kind = status?.running_kind ?? 'cycle'
|
||||
setKeepOn(false)
|
||||
setConfirm({
|
||||
title: t('dashboard.confirm.stopTitle'),
|
||||
message: t('dashboard.confirm.stopMsg', { job: t(runKindLabelKey(kind)) }),
|
||||
confirmLabel: t('dashboard.confirm.stopYes'),
|
||||
danger: true,
|
||||
icon: '■',
|
||||
toggle: { label: t('dashboard.confirm.stopPowerOff'), value: false, onChange: setKeepOn },
|
||||
onConfirm: async () => {
|
||||
setActionErr(null)
|
||||
try {
|
||||
await api.cancelJob(runId, keepOnRef.current)
|
||||
} catch (e) {
|
||||
setActionErr(e instanceof ApiError ? e.message : t('dashboard.actionFailed'))
|
||||
}
|
||||
pollAfterAction()
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
const toggleGuest = (vmid: number) => {
|
||||
setSavedNote(false)
|
||||
setErr(null)
|
||||
@@ -275,6 +302,7 @@ export function Dashboard({ status, refreshStatus }: DashboardProps) {
|
||||
error={actionErr}
|
||||
onBackup={() => runAction('backup', (k) => api.runBackup(k), false, '▶')}
|
||||
onGc={() => runAction('gc', (k) => api.runGc(k), false, '⟳')}
|
||||
onStop={stopAction}
|
||||
onPowerOn={() => runAction('on', () => api.powerOn(), false, '⏻')}
|
||||
onPowerOff={() => runAction('off', () => api.powerOff(), true, '⏻')}
|
||||
/>
|
||||
@@ -305,7 +333,7 @@ export function Dashboard({ status, refreshStatus }: DashboardProps) {
|
||||
refreshing={refreshing}
|
||||
error={guestsErr}
|
||||
/>
|
||||
<ActivityLog logs={logs} />
|
||||
<HistoryCard logs={logs} />
|
||||
</div>
|
||||
|
||||
<TaskLog lines={taskLog.lines} running={jobRunning} runId={taskLog.runId} />
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import type { LogLine } from '../../api/types'
|
||||
import { c, mono, panelStyle } from '../../theme'
|
||||
import { c, mono } from '../../theme'
|
||||
import { fmtClock } from '../../utils/format'
|
||||
|
||||
const LEVELS: Record<string, { color: string; bg: string }> = {
|
||||
// Shared with RunHistory, which renders the same log lines nested inside an expanded run.
|
||||
export const LEVELS: Record<string, { color: string; bg: string }> = {
|
||||
INFO: { color: '#8aa6c0', bg: 'rgba(138,166,192,.13)' },
|
||||
OK: { color: c.green, bg: 'rgba(63,178,127,.14)' },
|
||||
WARN: { color: c.amber, bg: 'rgba(224,169,43,.14)' },
|
||||
ERROR: { color: c.red, bg: 'rgba(229,103,91,.14)' },
|
||||
}
|
||||
|
||||
const colHead: React.CSSProperties = {
|
||||
export const colHead: React.CSSProperties = {
|
||||
fontSize: 10,
|
||||
fontWeight: 600,
|
||||
letterSpacing: '.06em',
|
||||
@@ -18,14 +19,12 @@ const colHead: React.CSSProperties = {
|
||||
color: '#6f7884',
|
||||
}
|
||||
|
||||
// The flat activity stream. The surrounding panel, the view switch and the scroll container
|
||||
// live in HistoryCard, which renders either this or RunHistory.
|
||||
export function ActivityLog({ logs }: { logs: LogLine[] }) {
|
||||
const { t } = useTranslation()
|
||||
return (
|
||||
<div style={{ ...panelStyle, padding: '8px 0 6px', alignSelf: 'stretch' }}>
|
||||
<div style={{ display: 'flex', alignItems: 'center', justifyContent: 'space-between', padding: '6px 18px 12px' }}>
|
||||
<span style={{ fontSize: 13, fontWeight: 700, letterSpacing: '.04em' }}>{t('dashboard.activityLog')}</span>
|
||||
<span style={{ fontFamily: mono, fontSize: 11, color: '#6f7884' }}>{t('dashboard.events', { n: logs.length })}</span>
|
||||
</div>
|
||||
<>
|
||||
<div
|
||||
className="jn-log-head"
|
||||
style={{ padding: '0 18px 7px', borderBottom: `1px solid ${c.border}` }}
|
||||
@@ -34,7 +33,7 @@ export function ActivityLog({ logs }: { logs: LogLine[] }) {
|
||||
<span style={colHead}>{t('dashboard.colLevel')}</span>
|
||||
<span style={colHead}>{t('dashboard.colMessage')}</span>
|
||||
</div>
|
||||
<div style={{ maxHeight: 250, overflowY: 'auto' }}>
|
||||
<div>
|
||||
{logs.length === 0 && (
|
||||
<div style={{ padding: '14px 18px', fontSize: 13, color: c.textFaint }}>{t('dashboard.noLogs')}</div>
|
||||
)}
|
||||
@@ -74,6 +73,6 @@ export function ActivityLog({ logs }: { logs: LogLine[] }) {
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
</div>
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { useId, useState } from 'react'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import type { LogLine } from '../../api/types'
|
||||
import { useRuns } from '../../hooks/useRuns'
|
||||
import { c, mono, panelStyle } from '../../theme'
|
||||
import { ActivityLog } from './ActivityLog'
|
||||
import { RunHistory } from './RunHistory'
|
||||
|
||||
type View = 'log' | 'runs'
|
||||
|
||||
// The activity stream and the run history are the same data at two zoom levels (every log
|
||||
// line belongs to a run), so they share one card and a two-tab switch instead of costing the
|
||||
// dashboard another full-width panel.
|
||||
export function HistoryCard({ logs }: { logs: LogLine[] }) {
|
||||
const { t } = useTranslation()
|
||||
const [view, setView] = useState<View>('log')
|
||||
const panelId = useId()
|
||||
// Only polls while the history tab is open, so the default view costs no extra requests.
|
||||
const { runs, error } = useRuns(view === 'runs')
|
||||
|
||||
const tabStyle = (active: boolean): React.CSSProperties => ({
|
||||
fontSize: 12,
|
||||
fontWeight: 700,
|
||||
letterSpacing: '.04em',
|
||||
textTransform: 'uppercase',
|
||||
padding: '5px 11px',
|
||||
borderRadius: 6,
|
||||
border: 'none',
|
||||
cursor: 'pointer',
|
||||
background: active ? c.inputBorder : 'transparent',
|
||||
color: active ? c.text : c.textFaint,
|
||||
})
|
||||
|
||||
const tab = (id: View, labelKey: string) => (
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected={view === id}
|
||||
aria-controls={panelId}
|
||||
onClick={() => setView(id)}
|
||||
style={tabStyle(view === id)}
|
||||
>
|
||||
{t(labelKey)}
|
||||
</button>
|
||||
)
|
||||
|
||||
return (
|
||||
<div style={{ ...panelStyle, padding: '8px 0 6px', alignSelf: 'stretch' }}>
|
||||
<div
|
||||
style={{
|
||||
display: 'flex',
|
||||
alignItems: 'center',
|
||||
justifyContent: 'space-between',
|
||||
gap: 10,
|
||||
padding: '6px 14px 12px',
|
||||
}}
|
||||
>
|
||||
<div role="tablist" aria-label={t('dashboard.historyViews')} style={{ display: 'flex', gap: 4 }}>
|
||||
{tab('log', 'dashboard.activityLog')}
|
||||
{tab('runs', 'dashboard.runHistory')}
|
||||
</div>
|
||||
<span style={{ fontFamily: mono, fontSize: 11, color: '#6f7884', flex: '0 0 auto' }}>
|
||||
{view === 'log'
|
||||
? t('dashboard.events', { n: logs.length })
|
||||
: t('dashboard.runsCount', { n: runs.length })}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Runs get more room: an expanded row carries its steps and its log lines. */}
|
||||
<div id={panelId} role="tabpanel" style={{ maxHeight: view === 'runs' ? 340 : 250, overflowY: 'auto' }}>
|
||||
{view === 'log' ? <ActivityLog logs={logs} /> : <RunHistory runs={runs} error={error} />}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -7,6 +7,7 @@ interface Props {
|
||||
error: string | null
|
||||
onBackup: () => void
|
||||
onGc: () => void
|
||||
onStop: () => void
|
||||
onPowerOn: () => void
|
||||
onPowerOff: () => void
|
||||
}
|
||||
@@ -44,7 +45,7 @@ function actionBtn(variant: 'primary' | 'ghost' | 'green' | 'red', enabled: bool
|
||||
return { ...base, background: 'transparent', color: c.text, border: '1px solid #3a434d' }
|
||||
}
|
||||
|
||||
export function ManualPanel({ status, error, onBackup, onGc, onPowerOn, onPowerOff }: Props) {
|
||||
export function ManualPanel({ status, error, onBackup, onGc, onStop, onPowerOn, onPowerOff }: Props) {
|
||||
const { t } = useTranslation()
|
||||
const online = !!status?.pbs_online
|
||||
const busy = !!status?.job_running
|
||||
@@ -53,27 +54,56 @@ export function ManualPanel({ status, error, onBackup, onGc, onPowerOn, onPowerO
|
||||
const canJob = !busy
|
||||
const canPower = online && !busy
|
||||
|
||||
// While a run is in flight, its own button becomes Stop (11.2). A scheduled *verify* has
|
||||
// no button of its own, so it borrows the primary slot — otherwise a hung verify would be
|
||||
// unstoppable, which is exactly the deadlock this feature exists to break. Cancelling
|
||||
// needs the run id; without it (a run started before this build, say) we only disable.
|
||||
const runningKind = busy ? (status?.running_kind ?? 'cycle') : null
|
||||
const canStop = busy && typeof status?.running_run_id === 'number'
|
||||
const stopSlot = runningKind === 'gc' ? 'gc' : 'primary'
|
||||
const stopLabel = t(
|
||||
runningKind === 'gc'
|
||||
? 'dashboard.stopGc'
|
||||
: runningKind === 'verify'
|
||||
? 'dashboard.stopVerify'
|
||||
: 'dashboard.stopBackup',
|
||||
)
|
||||
|
||||
const stopButton = (
|
||||
<button type="button" style={actionBtn('red', canStop)} disabled={!canStop} onClick={onStop}>
|
||||
■ {stopLabel}
|
||||
</button>
|
||||
)
|
||||
|
||||
return (
|
||||
<div style={{ ...panelStyle, padding: '16px 18px', height: '100%' }}>
|
||||
<div style={{ display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 22 }}>
|
||||
<div>
|
||||
<span style={colHead}>{t('dashboard.manualJobs')}</span>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 9 }}>
|
||||
<button style={actionBtn('primary', canJob)} disabled={!canJob} onClick={onBackup}>
|
||||
▶ {t('dashboard.runBackup')}
|
||||
</button>
|
||||
<button style={actionBtn('ghost', canJob)} disabled={!canJob} onClick={onGc}>
|
||||
⟳ {t('dashboard.runGc')}
|
||||
</button>
|
||||
{stopSlot === 'primary' && busy ? (
|
||||
stopButton
|
||||
) : (
|
||||
<button type="button" style={actionBtn('primary', canJob)} disabled={!canJob} onClick={onBackup}>
|
||||
▶ {t('dashboard.runBackup')}
|
||||
</button>
|
||||
)}
|
||||
{stopSlot === 'gc' && busy ? (
|
||||
stopButton
|
||||
) : (
|
||||
<button type="button" style={actionBtn('ghost', canJob)} disabled={!canJob} onClick={onGc}>
|
||||
⟳ {t('dashboard.runGc')}
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<span style={colHead}>{t('dashboard.power')}</span>
|
||||
<div style={{ display: 'flex', flexDirection: 'column', gap: 9 }}>
|
||||
<button style={actionBtn('green', !online)} disabled={online} onClick={onPowerOn}>
|
||||
<button type="button" style={actionBtn('green', !online)} disabled={online} onClick={onPowerOn}>
|
||||
⏻ {t('dashboard.powerOn')}
|
||||
</button>
|
||||
<button style={actionBtn('red', canPower)} disabled={!canPower} onClick={onPowerOff}>
|
||||
<button type="button" style={actionBtn('red', canPower)} disabled={!canPower} onClick={onPowerOff}>
|
||||
⏻ {t('dashboard.powerOff')}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
import { useState } from 'react'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import { api } from '../../api/client'
|
||||
import type { RunDetail, RunSummary } from '../../api/types'
|
||||
import { useConfig } from '../../config/ConfigContext'
|
||||
import { c, mono } from '../../theme'
|
||||
import { fmtClock, fmtDuration, fmtShort } from '../../utils/format'
|
||||
import { runDurationMs, runKindLabelKey, runStatusStyle } from '../../utils/status'
|
||||
import { LEVELS, colHead } from './ActivityLog'
|
||||
|
||||
const badge: React.CSSProperties = {
|
||||
display: 'inline-block',
|
||||
fontFamily: mono,
|
||||
fontSize: 10,
|
||||
fontWeight: 600,
|
||||
padding: '2px 7px',
|
||||
borderRadius: 5,
|
||||
textTransform: 'uppercase',
|
||||
}
|
||||
|
||||
const cell: React.CSSProperties = { fontFamily: mono, fontSize: 12, color: c.textDim }
|
||||
|
||||
function StatusBadge({ status }: { status: string }) {
|
||||
const { t } = useTranslation()
|
||||
const s = runStatusStyle(status)
|
||||
return <span style={{ ...badge, color: s.color, background: s.bg }}>{t(s.labelKey)}</span>
|
||||
}
|
||||
|
||||
// One run's steps + its own log lines, fetched on expand. Rendered inside the expanded row.
|
||||
function RunDetailPanel({ detail }: { detail: RunDetail | null }) {
|
||||
const { t } = useTranslation()
|
||||
if (!detail) {
|
||||
return (
|
||||
<div style={{ padding: '10px 18px 12px 34px', fontSize: 12, color: c.textFaint }}>
|
||||
{t('common.loading')}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
return (
|
||||
<div style={{ padding: '4px 18px 12px 34px', borderLeft: `2px solid ${c.border}`, marginLeft: 16 }}>
|
||||
<div style={{ ...colHead, margin: '8px 0 5px' }}>{t('dashboard.runSteps')}</div>
|
||||
{detail.steps.length === 0 && (
|
||||
<div style={{ fontSize: 12, color: c.textFaint }}>{t('dashboard.noSteps')}</div>
|
||||
)}
|
||||
{detail.steps.map((s) => {
|
||||
const ms = runDurationMs(s)
|
||||
return (
|
||||
<div
|
||||
key={`${s.name}-${s.started_at}`}
|
||||
style={{ display: 'flex', alignItems: 'center', gap: 10, padding: '3px 0', flexWrap: 'wrap' }}
|
||||
>
|
||||
<span style={{ ...cell, minWidth: 74, color: c.textMid }}>{s.name}</span>
|
||||
<StatusBadge status={s.status} />
|
||||
<span style={{ ...cell, minWidth: 44 }}>{ms === null ? '—' : fmtDuration(ms)}</span>
|
||||
{s.detail && (
|
||||
<span style={{ fontSize: 12, color: c.textFaint, overflowWrap: 'anywhere' }}>{s.detail}</span>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
|
||||
<div style={{ ...colHead, margin: '12px 0 5px' }}>{t('dashboard.runLog')}</div>
|
||||
{detail.logs.length === 0 && (
|
||||
<div style={{ fontSize: 12, color: c.textFaint }}>{t('dashboard.noLogs')}</div>
|
||||
)}
|
||||
{detail.logs.map((l) => {
|
||||
const lvl = LEVELS[l.level] ?? LEVELS.INFO
|
||||
return (
|
||||
<div key={l.id} style={{ display: 'flex', gap: 10, padding: '2px 0', alignItems: 'baseline' }}>
|
||||
<span style={{ ...cell, flex: '0 0 auto' }}>{fmtClock(new Date(l.ts))}</span>
|
||||
<span style={{ ...badge, color: lvl.color, background: lvl.bg, flex: '0 0 auto' }}>
|
||||
{l.level}
|
||||
</span>
|
||||
<span style={{ fontSize: 12, color: '#c8cdd4', minWidth: 0, overflowWrap: 'anywhere' }}>
|
||||
{l.message}
|
||||
</span>
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
export function RunHistory({ runs, error }: { runs: RunSummary[]; error: boolean }) {
|
||||
const { t } = useTranslation()
|
||||
const { config } = useConfig()
|
||||
const [openId, setOpenId] = useState<number | null>(null)
|
||||
const [details, setDetails] = useState<Record<number, RunDetail>>({})
|
||||
|
||||
const toggle = async (run: RunSummary) => {
|
||||
if (openId === run.id) {
|
||||
setOpenId(null)
|
||||
return
|
||||
}
|
||||
setOpenId(run.id)
|
||||
// Cached detail is reused, except for a run still in flight — its steps are still being
|
||||
// written, so a cached copy would freeze mid-cycle.
|
||||
if (details[run.id] && run.status !== 'running') return
|
||||
try {
|
||||
const detail = await api.run(run.id)
|
||||
setDetails((d) => ({ ...d, [run.id]: detail }))
|
||||
} catch {
|
||||
// Leave it in the loading state; the next expand retries.
|
||||
}
|
||||
}
|
||||
|
||||
const retentionDays = config?.maintenance.history.retention_days ?? 0
|
||||
|
||||
return (
|
||||
<>
|
||||
<div
|
||||
className="jn-run-head"
|
||||
style={{ padding: '0 18px 7px', borderBottom: `1px solid ${c.border}` }}
|
||||
>
|
||||
<span style={colHead}>{t('dashboard.colStarted')}</span>
|
||||
<span style={colHead}>{t('dashboard.colKind')}</span>
|
||||
<span style={colHead}>{t('dashboard.colTrigger')}</span>
|
||||
<span style={colHead}>{t('dashboard.colStatus')}</span>
|
||||
<span style={colHead}>{t('dashboard.colDuration')}</span>
|
||||
<span style={colHead}>{t('dashboard.colGuests')}</span>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
{runs.length === 0 && (
|
||||
<div style={{ padding: '14px 18px', fontSize: 13, color: c.textFaint }}>
|
||||
{error ? t('dashboard.runsError') : t('dashboard.noRuns')}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{runs.map((run) => {
|
||||
const open = openId === run.id
|
||||
const ms = runDurationMs(run)
|
||||
return (
|
||||
<div key={run.id} style={{ borderBottom: '1px solid #1b212880' }}>
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => void toggle(run)}
|
||||
aria-expanded={open}
|
||||
className="jn-run-row"
|
||||
style={{
|
||||
width: '100%',
|
||||
alignItems: 'center',
|
||||
padding: '7px 18px',
|
||||
background: 'transparent',
|
||||
border: 'none',
|
||||
cursor: 'pointer',
|
||||
textAlign: 'left',
|
||||
color: c.text,
|
||||
}}
|
||||
>
|
||||
<span style={{ ...cell, color: c.textFaint }}>
|
||||
<span style={{ color: c.textFaint, marginRight: 6 }}>{open ? '▾' : '▸'}</span>
|
||||
{fmtShort(new Date(run.started_at))}
|
||||
</span>
|
||||
<span style={{ ...cell, color: c.textMid }}>{t(runKindLabelKey(run.kind))}</span>
|
||||
<span style={cell}>{t(`dashboard.trigger_${run.trigger}`)}</span>
|
||||
<span>
|
||||
<StatusBadge status={run.status} />
|
||||
</span>
|
||||
<span style={cell}>{ms === null ? '—' : fmtDuration(ms)}</span>
|
||||
<span style={cell}>{run.guests_ok ?? '—'}</span>
|
||||
</button>
|
||||
|
||||
{/* An error is worth seeing without expanding — it's why you opened this view. */}
|
||||
{!open && run.error && (
|
||||
<div
|
||||
style={{
|
||||
padding: '0 18px 8px 34px',
|
||||
fontSize: 12,
|
||||
color: c.red,
|
||||
overflowWrap: 'anywhere',
|
||||
}}
|
||||
>
|
||||
{run.error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{open && <RunDetailPanel detail={details[run.id] ?? null} />}
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
|
||||
{retentionDays > 0 && runs.length > 0 && (
|
||||
<div style={{ padding: '9px 18px 2px', fontSize: 11, color: c.textMuted }}>
|
||||
{t('dashboard.historyRetention', { n: retentionDays })}
|
||||
</div>
|
||||
)}
|
||||
</>
|
||||
)
|
||||
}
|
||||
@@ -88,6 +88,14 @@
|
||||
grid-template-columns: 84px 96px 1fr;
|
||||
}
|
||||
|
||||
/* Run history: started | kind | trigger | status | duration | guests */
|
||||
.jn-run-head,
|
||||
.jn-run-row {
|
||||
display: grid;
|
||||
grid-template-columns: 116px 90px 92px 92px 72px 1fr;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
/* Settings */
|
||||
.jn-settings {
|
||||
display: grid;
|
||||
@@ -187,6 +195,19 @@
|
||||
grid-column: 1 / -1;
|
||||
}
|
||||
|
||||
/* Same treatment for run rows: drop the six column labels, then two lines per run —
|
||||
* started + kind + status on the first, trigger/duration/guests on the second. */
|
||||
.jn-run-head {
|
||||
display: none;
|
||||
}
|
||||
.jn-run-row {
|
||||
grid-template-columns: 106px 1fr auto;
|
||||
row-gap: 3px;
|
||||
}
|
||||
.jn-run-row > :nth-child(n + 4) {
|
||||
grid-row: 2;
|
||||
}
|
||||
|
||||
.jn-settings {
|
||||
grid-template-columns: 1fr;
|
||||
gap: 14px;
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import { fmtBytesTB, fmtUptime, rel } from './format.ts'
|
||||
import { fmtBytesTB, fmtDuration, fmtUptime, rel } from './format.ts'
|
||||
|
||||
test('rel: sub-minute vs rounds-to-a-minute', () => {
|
||||
assert.equal(rel(0), '<1m')
|
||||
@@ -26,6 +26,16 @@ test('fmtBytesTB: TB at/above 1e12, GB below', () => {
|
||||
assert.equal(fmtBytesTB(5e11), '500.00 GB')
|
||||
})
|
||||
|
||||
test('fmtDuration: keeps seconds below an hour (rel() would say "1m" for 41s)', () => {
|
||||
assert.equal(fmtDuration(2_000), '2s')
|
||||
assert.equal(fmtDuration(41_000), '41s')
|
||||
assert.equal(rel(41_000), '1m') // why fmtDuration exists
|
||||
assert.equal(fmtDuration(83_000), '1m 23s')
|
||||
assert.equal(fmtDuration(600_000), '10m 00s')
|
||||
assert.equal(fmtDuration(7_500_000), '2h 05m')
|
||||
assert.equal(fmtDuration(-1), '0s')
|
||||
})
|
||||
|
||||
test('fmtUptime: compact d/h/m with rollovers', () => {
|
||||
assert.equal(fmtUptime(90), '1m')
|
||||
assert.equal(fmtUptime(3660), '1h 1m')
|
||||
|
||||
@@ -34,6 +34,21 @@ export function rel(ms: number): string {
|
||||
return `${d}d${hh ? ' ' + hh + 'h' : ''}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Elapsed time for a run or one of its steps: `43s`, `1m 23s`, `2h 5m`.
|
||||
*
|
||||
* Not `rel()` — that rounds to whole minutes for "next run in 3h", which renders a 41-second
|
||||
* wake-wait as "1m" and a 2-second step as "<1m". Step timings are the diagnostic the run
|
||||
* detail exists for, so below an hour they keep their seconds.
|
||||
*/
|
||||
export function fmtDuration(ms: number): string {
|
||||
const s = Math.max(0, Math.round(ms / 1000))
|
||||
if (s < 60) return `${s}s`
|
||||
const m = Math.floor(s / 60)
|
||||
if (m < 60) return `${m}m ${pad(s % 60)}s`
|
||||
return `${Math.floor(m / 60)}h ${pad(m % 60)}m`
|
||||
}
|
||||
|
||||
export function fmtBytesTB(n: number): string {
|
||||
const tb = n / 1e12
|
||||
if (tb >= 1) return tb.toFixed(2) + ' TB'
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { test } from 'node:test'
|
||||
import { runningLabelKey } from './status.ts'
|
||||
import { runDurationMs, runKindLabelKey, runStatusStyle, runningLabelKey } from './status.ts'
|
||||
|
||||
test('runningLabelKey maps each run kind to its own label', () => {
|
||||
assert.equal(runningLabelKey('cycle'), 'status.running')
|
||||
@@ -12,3 +12,41 @@ test('runningLabelKey falls back to the backup label for null/unknown', () => {
|
||||
assert.equal(runningLabelKey(null), 'status.running')
|
||||
assert.equal(runningLabelKey(undefined), 'status.running')
|
||||
})
|
||||
|
||||
test('runKindLabelKey reads a backup cycle as a backup', () => {
|
||||
// The backup cycle is stored as "cycle"; only gc/verify get their own label.
|
||||
assert.equal(runKindLabelKey('cycle'), 'dashboard.kindBackup')
|
||||
assert.equal(runKindLabelKey('gc'), 'dashboard.kindGc')
|
||||
assert.equal(runKindLabelKey('verify'), 'dashboard.kindVerify')
|
||||
assert.equal(runKindLabelKey('something-new'), 'dashboard.kindBackup')
|
||||
})
|
||||
|
||||
test('runStatusStyle gives each outcome its own colour, unknown reads as running', () => {
|
||||
assert.equal(runStatusStyle('success').labelKey, 'dashboard.runSuccess')
|
||||
assert.equal(runStatusStyle('failure').labelKey, 'dashboard.runFailure')
|
||||
assert.equal(runStatusStyle('aborted').labelKey, 'dashboard.runAborted')
|
||||
assert.notEqual(runStatusStyle('success').color, runStatusStyle('failure').color)
|
||||
assert.equal(runStatusStyle('nonsense').labelKey, 'dashboard.runRunning')
|
||||
})
|
||||
|
||||
test('runDurationMs measures a finished run and elapsed time for one still going', () => {
|
||||
const started = '2026-06-28T04:00:00Z'
|
||||
assert.equal(
|
||||
runDurationMs({ started_at: started, finished_at: '2026-06-28T04:01:23Z' }),
|
||||
83_000,
|
||||
)
|
||||
// Unfinished: measured against the injected clock, not wall time.
|
||||
assert.equal(
|
||||
runDurationMs({ started_at: started, finished_at: null }, Date.parse('2026-06-28T04:00:30Z')),
|
||||
30_000,
|
||||
)
|
||||
})
|
||||
|
||||
test('runDurationMs returns null on an unparseable timestamp and never goes negative', () => {
|
||||
assert.equal(runDurationMs({ started_at: 'not-a-date', finished_at: null }), null)
|
||||
// Clock skew (finished before started) must not render as a negative duration.
|
||||
assert.equal(
|
||||
runDurationMs({ started_at: '2026-06-28T04:01:00Z', finished_at: '2026-06-28T04:00:00Z' }),
|
||||
0,
|
||||
)
|
||||
})
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { StatusResponse } from '../api/types'
|
||||
import type { RunSummary, StatusResponse } from '../api/types'
|
||||
|
||||
/**
|
||||
* i18n key for the header pill label while a run is in flight.
|
||||
@@ -17,3 +17,58 @@ export function runningLabelKey(kind: StatusResponse['running_kind']): string {
|
||||
return 'status.running'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* i18n key for a run's kind in the history table.
|
||||
*
|
||||
* A backup cycle is stored as `cycle` (RunKind.BACKUP is never used as a run kind), so
|
||||
* anything unrecognised — including a kind added by a newer backend — reads as a backup
|
||||
* rather than rendering a raw enum value.
|
||||
*/
|
||||
export function runKindLabelKey(kind: string): string {
|
||||
switch (kind) {
|
||||
case 'gc':
|
||||
return 'dashboard.kindGc'
|
||||
case 'verify':
|
||||
return 'dashboard.kindVerify'
|
||||
default:
|
||||
return 'dashboard.kindBackup'
|
||||
}
|
||||
}
|
||||
|
||||
export interface RunStatusStyle {
|
||||
labelKey: string
|
||||
color: string
|
||||
bg: string
|
||||
}
|
||||
|
||||
// Palette hexes are inlined rather than imported from theme.ts so this module stays free of
|
||||
// React/CSS imports and can be unit-tested under `node --test`.
|
||||
const RUN_STATUS: Record<string, RunStatusStyle> = {
|
||||
success: { labelKey: 'dashboard.runSuccess', color: '#3fb27f', bg: 'rgba(63,178,127,.14)' },
|
||||
failure: { labelKey: 'dashboard.runFailure', color: '#e5675b', bg: 'rgba(229,103,91,.14)' },
|
||||
aborted: { labelKey: 'dashboard.runAborted', color: '#e0a92b', bg: 'rgba(224,169,43,.14)' },
|
||||
running: { labelKey: 'dashboard.runRunning', color: '#3b82f6', bg: 'rgba(59,130,246,.14)' },
|
||||
}
|
||||
|
||||
/** Badge styling + label key for a run or step status; unknown values read as running. */
|
||||
export function runStatusStyle(status: string): RunStatusStyle {
|
||||
return RUN_STATUS[status] ?? RUN_STATUS.running
|
||||
}
|
||||
|
||||
/**
|
||||
* How long a run took, in ms — or how long it has been going if it hasn't finished.
|
||||
*
|
||||
* `now` is injected so the caller (and the tests) control the clock. Returns null when the
|
||||
* result would be meaningless, so the caller renders a dash instead of "0m".
|
||||
*/
|
||||
export function runDurationMs(
|
||||
run: Pick<RunSummary, 'started_at' | 'finished_at'>,
|
||||
now: number = Date.now(),
|
||||
): number | null {
|
||||
const started = Date.parse(run.started_at)
|
||||
if (Number.isNaN(started)) return null
|
||||
const end = run.finished_at ? Date.parse(run.finished_at) : now
|
||||
if (Number.isNaN(end)) return null
|
||||
return Math.max(0, end - started)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user