e0e4cce8bd
Three faults, visible only once the board ran against production data. BACKUPS SAID THE WHOLE FLEET HAD STOPPED. The lastseenat backfill was wrong. It seeded from collectedat, reasoning that the last change was the last provable moment - but an unchanged config writes no revision, so a machine whose settings last changed nine months ago got a nine-month-old lastseenat and was instantly reported as a dead backup. Every chain lit up at once, which is worse than no card: it says the site is broken when it is fine. The honest value is NULL. Before the column existed nothing recorded when a config was last confirmed, and inventing a date does not change that. Migration 0003 clears the backfill, and staleness now IGNORES a NULL chain rather than substituting timestamps that mean something else. A chain becomes measurable the first time its PC posts, which for NTLARS is within a day. TONER READ "None%". The supply dict has no 'percent' key - it is 'remaining'. Supply names are also shortened, because "Black Toner Level 4%" spends three words saying what the card already says. THE CARDS READ AS WALLS OF TEXT. Rows wrapped into paragraphs and a card with forty PCs pushed everything below it off the screen. Now: at most five rows with "and N more", one line per row that truncates rather than wraps, meta pushed right and dropped first since it matters least, and severity reduced to a small dot beside an uppercase label instead of a coloured card - six severity-painted cards read as a crisis, which is how a board stops being read. Worth recording that none of this could fail in a test. Every one needed real data on a real fleet.
90 lines
3.5 KiB
Python
90 lines
3.5 KiB
Python
"""Which backup chains have stopped running.
|
|
|
|
Split from the route for the same reason retention.py is: the rule here is the
|
|
interesting part and it should be testable without an auth layer in the way.
|
|
|
|
THE RULE, and it is not the obvious one. Dedup means an unchanged configuration
|
|
writes no revision, so `collectedat` moves only when something CHANGES. A
|
|
machine whose settings have been stable for six months has a six-month-old
|
|
newest revision and is entirely healthy. Staleness is therefore measured from
|
|
`lastseenat` - when the config was last CONFIRMED current, recorded on every
|
|
matching post including the no-op that writes nothing else.
|
|
|
|
Keyed on the CHAIN (asset, kind, source PC), because that is the unit that fails
|
|
independently. A machine with two part markers can have one still reporting
|
|
while the other stopped, and a per-asset view would report the machine as fine.
|
|
"""
|
|
|
|
from datetime import datetime, timedelta
|
|
|
|
from shopdb.api import db, Asset
|
|
|
|
from ..models import BackupRevision
|
|
|
|
DEFAULTSTALEDAYS = 3
|
|
|
|
|
|
def staledays():
|
|
"""Configured threshold; 0 or negative disables the card entirely."""
|
|
from shopdb.api import Setting
|
|
|
|
setting = Setting.query.filter_by(key='backups_staledays').first()
|
|
if setting and (setting.value or '').strip():
|
|
try:
|
|
return int(setting.value)
|
|
except (TypeError, ValueError):
|
|
pass
|
|
return DEFAULTSTALEDAYS
|
|
|
|
|
|
def latestperchain():
|
|
"""Newest revision of every (asset, kind, source) chain."""
|
|
latest = {}
|
|
for revision in (BackupRevision.query
|
|
.order_by(BackupRevision.backuprevisionid.asc()).all()):
|
|
latest[(revision.assetid, revision.backupkind,
|
|
revision.sourcehostname)] = revision
|
|
return latest
|
|
|
|
|
|
def stalechains(days=None, limit=50):
|
|
"""Chains whose last confirmed check is older than the threshold.
|
|
|
|
Deliberately silent about assets that have NEVER been backed up. Whether one
|
|
SHOULD be is a question only the manifest can answer - most machines carry
|
|
no NTLARS at all - and guessing would list a hundred healthy assets, which
|
|
is the noise that makes a board worth ignoring. That is the
|
|
desired-versus-observed card, which needs the manifest resolver.
|
|
"""
|
|
days = staledays() if days is None else days
|
|
if days <= 0:
|
|
return []
|
|
|
|
now = datetime.utcnow()
|
|
cutoff = now - timedelta(days=days)
|
|
|
|
rows = []
|
|
for revision in latestperchain().values():
|
|
# NULL means never confirmed since the column existed, and it is NOT
|
|
# substituted with collectedat. That substitution is what the first
|
|
# version did, and on a real fleet it reported every machine whose
|
|
# config had simply been stable for months as a stopped backup - the
|
|
# whole board lit up and said the site was broken when it was fine.
|
|
# A chain becomes measurable the first time its PC posts; until then
|
|
# this card says nothing about it, which is the truth.
|
|
seen = revision.lastseenat
|
|
if seen is None or seen >= cutoff:
|
|
continue
|
|
asset = db.session.get(Asset, revision.assetid)
|
|
rows.append({
|
|
'assetid': revision.assetid,
|
|
'assetnumber': asset.assetnumber if asset else str(revision.assetid),
|
|
'backupkind': revision.backupkind,
|
|
'sourcehostname': revision.sourcehostname,
|
|
'lastseenat': seen.isoformat() + 'Z',
|
|
'quietdays': (now - seen).days,
|
|
})
|
|
|
|
rows.sort(key=lambda r: -r['quietdays'])
|
|
return rows[:limit]
|