backups: record that a config was checked, not only that it changed
Some checks failed
CI / backend (push) Failing after 8s
CI / naming (push) Successful in 2s
CI / frontend (push) Successful in 10s
CI / migrations-mysql (push) Failing after 7s

The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.

Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.

lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.

The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.

The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.

Threshold is backups_staledays, default 3, and 0 disables the card.
This commit is contained in:
cproudlock
2026-08-11 13:52:07 -04:00
parent 1ca8a9b8e8
commit 6c975a107c
8 changed files with 325 additions and 2 deletions

View File

@@ -46,8 +46,10 @@ CUTOVER_PLUGINS = (
# stamp '<plugin>0001anchor'; measuringtools stamps its real baseline id.
EXPECTED_HEAD_REVISION = {plugin: f'{plugin}0001anchor' for plugin in CUTOVER_PLUGINS}
EXPECTED_HEAD_REVISION['measuringtools'] = 'measuringtools0001baseline'
# backups is also post-cutover: its 0001 really creates backuprevisions.
EXPECTED_HEAD_REVISION['backups'] = 'backups0001baseline'
# backups is also post-cutover: its 0001 really creates backuprevisions, and
# 0002 adds lastseenat - when a config was last CONFIRMED unchanged, which dedup
# otherwise throws away.
EXPECTED_HEAD_REVISION['backups'] = 'backups0002lastseenat'
# geenforce adds the content-addressed blob store (manifestblobs) on top of its
# baseline.
EXPECTED_HEAD_REVISION['geenforce'] = 'geenforce0002blobs'