backups: record that a config was checked, not only that it changed
Some checks failed
CI / backend (push) Failing after 8s
CI / naming (push) Successful in 2s
CI / frontend (push) Successful in 10s
CI / migrations-mysql (push) Failing after 7s

The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.

Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.

lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.

The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.

The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.

Threshold is backups_staledays, default 3, and 0 disables the card.
This commit is contained in:
cproudlock
2026-08-11 13:52:07 -04:00
parent 1ca8a9b8e8
commit 6c975a107c
8 changed files with 325 additions and 2 deletions

View File

@@ -161,6 +161,17 @@ class BackupsPlugin(BasePlugin):
'description': 'UNC root that opaque (non-JSON) backups are '
'written under by the collecting PC. Site-specific.',
},
{
'key': 'backups_staledays',
'value': '3',
'valuetype': 'integer',
'category': 'backups',
'description': 'Days without a CONFIRMED check before a backup '
'is listed as stopped on the dashboard. Measured '
'from the last check, not the last change - an '
'unchanged config writes no revision. 0 disables '
'the card.',
},
{
'key': 'backups_retentioncount',
'value': '50',
@@ -180,6 +191,36 @@ class BackupsPlugin(BasePlugin):
},
]
def get_dashboard_widgets(self) -> List[Dict]:
"""Dashboard card: backups that have stopped running.
Keyed on the last CONFIRMED check, never on the last change. Dedup means
an unchanged config writes no revision, so a card keyed on revision age
would flag most of a healthy fleet.
"""
return [
{
'id': 'backups-stale',
'title': 'Backups stopped',
'endpoint': '/api/backups/dashboard/stale',
'render': 'exceptions',
'severity': 'warning',
'permission': 'backups.view',
'empty': 'hide',
'position': 30,
'map': {
'title': 'assetnumber',
'detail': 'backupkind',
'meta': [
{'key': 'sourcehostname', 'label': 'from'},
{'key': 'quietdays', 'label': 'last checked',
'suffix': ' days ago'},
],
'link': '/backups/asset/{assetid}',
},
},
]
# ---- ADR-006 collector contract -------------------------------------
def get_collector_schema(self) -> Optional[dict]:
@@ -299,6 +340,13 @@ class BackupsPlugin(BasePlugin):
.first())
if latest is not None and latest.contenthash == contenthash:
# Unchanged, so no revision - but RECORD THE CHECK. Without this the
# only evidence a backup still runs is a line in a log on the PC,
# and a config stable for six months is indistinguishable from a
# backup that died six months ago. Cheap: one column on a row that
# already exists, no new history.
latest.lastseenat = datetime.utcnow()
db.session.flush()
return {
'action': 'noop',
'assetid': assetid,
@@ -330,6 +378,7 @@ class BackupsPlugin(BasePlugin):
bytesize=bytesize,
sourcehostname=payload.get('sourcehostname'),
collectedat=collectedat or datetime.utcnow(),
lastseenat=datetime.utcnow(),
)
revision.payload = projection
db.session.add(revision)