6c975a107c
The stale-backup card could not be built as designed, and the reason is more important than the card. Dedup means an unchanged configuration writes no revision, so collectedat moves only on a CHANGE. A machine stable for six months has a six-month-old newest revision and is perfectly healthy. Keying a staleness card on revision age would have flagged most of the fleet - exactly the noise that makes a board worth ignoring. Underneath that: ShopDB could not distinguish those cases at all. On a no-op the server returned "unchanged" and wrote nothing, so "we checked yesterday and it matched" was discarded. That fact is the one thing a backup system must be able to prove, and the only record of it was a line in a log file on the PC. lastseenat records the check rather than the change. Touched on every matching post including the no-op; set on creation, since a new revision has by definition just been seen; backfilled from collectedat or createdat so existing rows start from the last moment the config can be PROVEN current, rather than from now - claiming a check that never happened would be worse than silence. The card keys on it, one row per CHAIN rather than per asset: a machine with two part markers can have one still reporting while the other stopped, and a per-asset view would report the machine as fine. It stays deliberately silent about assets never backed up, because whether one SHOULD be is a question only the manifest can answer, and guessing would list a hundred healthy machines. The rule lives in services/staleness.py rather than the route, so it is testable without an auth layer in the way - the same split retention.py uses. Threshold is backups_staledays, default 3, and 0 disables the card.
86 lines
3.2 KiB
Python
86 lines
3.2 KiB
Python
"""Which backup chains have stopped running.
|
|
|
|
Split from the route for the same reason retention.py is: the rule here is the
|
|
interesting part and it should be testable without an auth layer in the way.
|
|
|
|
THE RULE, and it is not the obvious one. Dedup means an unchanged configuration
|
|
writes no revision, so `collectedat` moves only when something CHANGES. A
|
|
machine whose settings have been stable for six months has a six-month-old
|
|
newest revision and is entirely healthy. Staleness is therefore measured from
|
|
`lastseenat` - when the config was last CONFIRMED current, recorded on every
|
|
matching post including the no-op that writes nothing else.
|
|
|
|
Keyed on the CHAIN (asset, kind, source PC), because that is the unit that fails
|
|
independently. A machine with two part markers can have one still reporting
|
|
while the other stopped, and a per-asset view would report the machine as fine.
|
|
"""
|
|
|
|
from datetime import datetime, timedelta
|
|
|
|
from shopdb.api import db, Asset
|
|
|
|
from ..models import BackupRevision
|
|
|
|
DEFAULTSTALEDAYS = 3
|
|
|
|
|
|
def staledays():
|
|
"""Configured threshold; 0 or negative disables the card entirely."""
|
|
from shopdb.api import Setting
|
|
|
|
setting = Setting.query.filter_by(key='backups_staledays').first()
|
|
if setting and (setting.value or '').strip():
|
|
try:
|
|
return int(setting.value)
|
|
except (TypeError, ValueError):
|
|
pass
|
|
return DEFAULTSTALEDAYS
|
|
|
|
|
|
def latestperchain():
|
|
"""Newest revision of every (asset, kind, source) chain."""
|
|
latest = {}
|
|
for revision in (BackupRevision.query
|
|
.order_by(BackupRevision.backuprevisionid.asc()).all()):
|
|
latest[(revision.assetid, revision.backupkind,
|
|
revision.sourcehostname)] = revision
|
|
return latest
|
|
|
|
|
|
def stalechains(days=None, limit=50):
|
|
"""Chains whose last confirmed check is older than the threshold.
|
|
|
|
Deliberately silent about assets that have NEVER been backed up. Whether one
|
|
SHOULD be is a question only the manifest can answer - most machines carry
|
|
no NTLARS at all - and guessing would list a hundred healthy assets, which
|
|
is the noise that makes a board worth ignoring. That is the
|
|
desired-versus-observed card, which needs the manifest resolver.
|
|
"""
|
|
days = staledays() if days is None else days
|
|
if days <= 0:
|
|
return []
|
|
|
|
now = datetime.utcnow()
|
|
cutoff = now - timedelta(days=days)
|
|
|
|
rows = []
|
|
for revision in latestperchain().values():
|
|
# Rows written before lastseenat existed fall back to the timestamps
|
|
# that do exist, so an old install reports something sane on day one
|
|
# rather than every chain at once.
|
|
seen = revision.lastseenat or revision.collectedat or revision.createdat
|
|
if seen is None or seen >= cutoff:
|
|
continue
|
|
asset = db.session.get(Asset, revision.assetid)
|
|
rows.append({
|
|
'assetid': revision.assetid,
|
|
'assetnumber': asset.assetnumber if asset else str(revision.assetid),
|
|
'backupkind': revision.backupkind,
|
|
'sourcehostname': revision.sourcehostname,
|
|
'lastseenat': seen.isoformat() + 'Z',
|
|
'quietdays': (now - seen).days,
|
|
})
|
|
|
|
rows.sort(key=lambda r: -r['quietdays'])
|
|
return rows[:limit]
|