backups: record that a config was checked, not only that it changed
Some checks failed
CI / backend (push) Failing after 8s
CI / naming (push) Successful in 2s
CI / frontend (push) Successful in 10s
CI / migrations-mysql (push) Failing after 7s

The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.

Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.

lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.

The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.

The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.

Threshold is backups_staledays, default 3, and 0 disables the card.
This commit is contained in:
cproudlock
2026-08-11 13:52:07 -04:00
parent 1ca8a9b8e8
commit 6c975a107c
8 changed files with 325 additions and 2 deletions

View File

@@ -10,6 +10,22 @@ ADR-007 and ADR-002.
## [Unreleased]
### Added
- A backup now records that it was CHECKED, not only that it changed. Dedup
means an unchanged configuration writes no revision, so the stored timestamps
moved only on a change: a machine whose settings had been stable for six
months was indistinguishable from a machine whose backup died six months ago.
The only evidence a backup still ran was a line in a log file on the PC.
`lastseenat` is touched on every matching post, including the no-op, and is
backfilled from the timestamps that already exist.
- Dashboard cards for enforcement failures, PCs not reporting, and backups that
have stopped. The dashboard now renders cards plugins declare, which it never
did before - five plugins had been declaring widgets into a void, pointing at
components nobody wrote. Cards are ordered by severity, hide themselves when
there is nothing to report, gate on a permission, and each fetches
independently so one broken endpoint cannot blank the board.
## [0.9.0] - 2026-08-11
Driven by a fleet that had been failing quietly. A bay had been returning 500