Three faults, visible only once the board ran against production data.
BACKUPS SAID THE WHOLE FLEET HAD STOPPED. The lastseenat backfill was wrong. It
seeded from collectedat, reasoning that the last change was the last provable
moment - but an unchanged config writes no revision, so a machine whose settings
last changed nine months ago got a nine-month-old lastseenat and was instantly
reported as a dead backup. Every chain lit up at once, which is worse than no
card: it says the site is broken when it is fine.
The honest value is NULL. Before the column existed nothing recorded when a
config was last confirmed, and inventing a date does not change that. Migration
0003 clears the backfill, and staleness now IGNORES a NULL chain rather than
substituting timestamps that mean something else. A chain becomes measurable the
first time its PC posts, which for NTLARS is within a day.
TONER READ "None%". The supply dict has no 'percent' key - it is 'remaining'.
Supply names are also shortened, because "Black Toner Level 4%" spends three
words saying what the card already says.
THE CARDS READ AS WALLS OF TEXT. Rows wrapped into paragraphs and a card with
forty PCs pushed everything below it off the screen. Now: at most five rows with
"and N more", one line per row that truncates rather than wraps, meta pushed
right and dropped first since it matters least, and severity reduced to a small
dot beside an uppercase label instead of a coloured card - six severity-painted
cards read as a crisis, which is how a board stops being read.
Worth recording that none of this could fail in a test. Every one needed real
data on a real fleet.
The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.
Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.
lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.
The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.
The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.
Threshold is backups_staledays, default 3, and 0 disables the card.
