dashboard: fix what a real fleet showed, which tests could not
Three faults, visible only once the board ran against production data. BACKUPS SAID THE WHOLE FLEET HAD STOPPED. The lastseenat backfill was wrong. It seeded from collectedat, reasoning that the last change was the last provable moment - but an unchanged config writes no revision, so a machine whose settings last changed nine months ago got a nine-month-old lastseenat and was instantly reported as a dead backup. Every chain lit up at once, which is worse than no card: it says the site is broken when it is fine. The honest value is NULL. Before the column existed nothing recorded when a config was last confirmed, and inventing a date does not change that. Migration 0003 clears the backfill, and staleness now IGNORES a NULL chain rather than substituting timestamps that mean something else. A chain becomes measurable the first time its PC posts, which for NTLARS is within a day. TONER READ "None%". The supply dict has no 'percent' key - it is 'remaining'. Supply names are also shortened, because "Black Toner Level 4%" spends three words saying what the card already says. THE CARDS READ AS WALLS OF TEXT. Rows wrapped into paragraphs and a card with forty PCs pushed everything below it off the screen. Now: at most five rows with "and N more", one line per row that truncates rather than wraps, meta pushed right and dropped first since it matters least, and severity reduced to a small dot beside an uppercase label instead of a coloured card - six severity-painted cards read as a crisis, which is how a board stops being read. Worth recording that none of this could fail in a test. Every one needed real data on a real fleet.
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
"""backups: clear the backfilled lastseenat - it was a guess, and it showed.
|
||||
|
||||
0002 added lastseenat and backfilled it from collectedat, reasoning that the
|
||||
last change was the last moment the config could be PROVEN current. On a fleet
|
||||
that was wrong in practice: an unchanged config writes no revision, so a machine
|
||||
whose settings last changed nine months ago got a nine-month-old lastseenat and
|
||||
was immediately reported as a stopped backup. Every chain lit up at once, which
|
||||
is worse than no card - it says the fleet is broken when it is fine.
|
||||
|
||||
The honest value is NULL: before this column existed, nothing recorded when a
|
||||
config was last confirmed, and inventing a date does not change that. A chain
|
||||
becomes measurable the first time its PC posts after the upgrade, which for
|
||||
NTLARS is within a day.
|
||||
|
||||
So staleness now IGNORES a chain whose lastseenat is NULL, rather than falling
|
||||
back to timestamps that mean something else.
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'backups0003clearlastseen'
|
||||
down_revision = 'backups0002lastseenat'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
columns = {c['name'] for c in
|
||||
sa.inspect(op.get_bind()).get_columns('backuprevisions')}
|
||||
if 'lastseenat' in columns:
|
||||
op.execute('UPDATE backuprevisions SET lastseenat = NULL')
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Nothing to restore: the backfilled values were derived, not recorded.
|
||||
pass
|
||||
@@ -65,10 +65,14 @@ def stalechains(days=None, limit=50):
|
||||
|
||||
rows = []
|
||||
for revision in latestperchain().values():
|
||||
# Rows written before lastseenat existed fall back to the timestamps
|
||||
# that do exist, so an old install reports something sane on day one
|
||||
# rather than every chain at once.
|
||||
seen = revision.lastseenat or revision.collectedat or revision.createdat
|
||||
# NULL means never confirmed since the column existed, and it is NOT
|
||||
# substituted with collectedat. That substitution is what the first
|
||||
# version did, and on a real fleet it reported every machine whose
|
||||
# config had simply been stable for months as a stopped backup - the
|
||||
# whole board lit up and said the site was broken when it was fine.
|
||||
# A chain becomes measurable the first time its PC posts; until then
|
||||
# this card says nothing about it, which is the truth.
|
||||
seen = revision.lastseenat
|
||||
if seen is None or seen >= cutoff:
|
||||
continue
|
||||
asset = db.session.get(Asset, revision.assetid)
|
||||
|
||||
Reference in New Issue
Block a user