backups: record that a config was checked, not only that it changed

The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.

Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.

lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.

The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.

The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.

Threshold is backups_staledays, default 3, and 0 disables the card.

plugins/backups/services/staleness.py

@@ -0,0 +1,85 @@
+"""Which backup chains have stopped running.
+
+Split from the route for the same reason retention.py is: the rule here is the
+interesting part and it should be testable without an auth layer in the way.
+
+THE RULE, and it is not the obvious one. Dedup means an unchanged configuration
+writes no revision, so `collectedat` moves only when something CHANGES. A
+machine whose settings have been stable for six months has a six-month-old
+newest revision and is entirely healthy. Staleness is therefore measured from
+`lastseenat` - when the config was last CONFIRMED current, recorded on every
+matching post including the no-op that writes nothing else.
+
+Keyed on the CHAIN (asset, kind, source PC), because that is the unit that fails
+independently. A machine with two part markers can have one still reporting
+while the other stopped, and a per-asset view would report the machine as fine.
+"""
+
+from datetime import datetime, timedelta
+
+from shopdb.api import db, Asset
+
+from ..models import BackupRevision
+
+DEFAULTSTALEDAYS = 3
+
+
+def staledays():
+    """Configured threshold; 0 or negative disables the card entirely."""
+    from shopdb.api import Setting
+
+    setting = Setting.query.filter_by(key='backups_staledays').first()
+    if setting and (setting.value or '').strip():
+        try:
+            return int(setting.value)
+        except (TypeError, ValueError):
+            pass
+    return DEFAULTSTALEDAYS
+
+
+def latestperchain():
+    """Newest revision of every (asset, kind, source) chain."""
+    latest = {}
+    for revision in (BackupRevision.query
+                     .order_by(BackupRevision.backuprevisionid.asc()).all()):
+        latest[(revision.assetid, revision.backupkind,
+                revision.sourcehostname)] = revision
+    return latest
+
+
+def stalechains(days=None, limit=50):
+    """Chains whose last confirmed check is older than the threshold.
+
+    Deliberately silent about assets that have NEVER been backed up. Whether one
+    SHOULD be is a question only the manifest can answer - most machines carry
+    no NTLARS at all - and guessing would list a hundred healthy assets, which
+    is the noise that makes a board worth ignoring. That is the
+    desired-versus-observed card, which needs the manifest resolver.
+    """
+    days = staledays() if days is None else days
+    if days <= 0:
+        return []
+
+    now = datetime.utcnow()
+    cutoff = now - timedelta(days=days)
+
+    rows = []
+    for revision in latestperchain().values():
+        # Rows written before lastseenat existed fall back to the timestamps
+        # that do exist, so an old install reports something sane on day one
+        # rather than every chain at once.
+        seen = revision.lastseenat or revision.collectedat or revision.createdat
+        if seen is None or seen >= cutoff:
+            continue
+        asset = db.session.get(Asset, revision.assetid)
+        rows.append({
+            'assetid': revision.assetid,
+            'assetnumber': asset.assetnumber if asset else str(revision.assetid),
+            'backupkind': revision.backupkind,
+            'sourcehostname': revision.sourcehostname,
+            'lastseenat': seen.isoformat() + 'Z',
+            'quietdays': (now - seen).days,
+        })
+
+    rows.sort(key=lambda r: -r['quietdays'])
+    return rows[:limit]