Core's Alembic env got connection.commit() when the stamp bug was found; the
per-plugin template did not. MySQL commits DDL implicitly, which flushes
everything queued before it including the previous migration's version stamp,
and the LAST migration of a run has no DDL after it - so its stamp rolled back
at close while its schema change survived. flask plugin upgrade-all then exited
0 having silently re-run that migration, and re-ran it again on every deploy
after. Invisible for exactly as long as every plugin head happened to be
idempotent.
Two were not.
backups 0003 cleared lastseenat for EVERY row, which is correct once and
destroys evidence on each repeat. It is now scoped to the backfill's actual
signature, COALESCE(collectedat, createdat) - the expression 0002 wrote - plus a
date bound. Both conditions are needed. Matching on collectedat alone misses
every row whose collectedat is NULL, so precisely the rows carrying the most
invented value would have kept it forever; and value equality is not a signature
on MySQL, where db.DateTime is second-precision and the collector writes both
stamps in one statement, so a genuinely fresh revision would read as a backfill
and be wiped. SQLite keeps microseconds, which is why no test could show it.
geenforce 0003 added a column unconditionally, so it failed on a fresh database
built from the models and on any re-run. Guarded like network0003prefix.
Three faults, visible only once the board ran against production data.
BACKUPS SAID THE WHOLE FLEET HAD STOPPED. The lastseenat backfill was wrong. It
seeded from collectedat, reasoning that the last change was the last provable
moment - but an unchanged config writes no revision, so a machine whose settings
last changed nine months ago got a nine-month-old lastseenat and was instantly
reported as a dead backup. Every chain lit up at once, which is worse than no
card: it says the site is broken when it is fine.
The honest value is NULL. Before the column existed nothing recorded when a
config was last confirmed, and inventing a date does not change that. Migration
0003 clears the backfill, and staleness now IGNORES a NULL chain rather than
substituting timestamps that mean something else. A chain becomes measurable the
first time its PC posts, which for NTLARS is within a day.
TONER READ "None%". The supply dict has no 'percent' key - it is 'remaining'.
Supply names are also shortened, because "Black Toner Level 4%" spends three
words saying what the card already says.
THE CARDS READ AS WALLS OF TEXT. Rows wrapped into paragraphs and a card with
forty PCs pushed everything below it off the screen. Now: at most five rows with
"and N more", one line per row that truncates rather than wraps, meta pushed
right and dropped first since it matters least, and severity reduced to a small
dot beside an uppercase label instead of a coloured card - six severity-painted
cards read as a crisis, which is how a board stops being read.
Worth recording that none of this could fail in a test. Every one needed real
data on a real fleet.
The stale-backup card could not be built as designed, and the reason is more
important than the card. Dedup means an unchanged configuration writes no
revision, so collectedat moves only on a CHANGE. A machine stable for six
months has a six-month-old newest revision and is perfectly healthy. Keying a
staleness card on revision age would have flagged most of the fleet - exactly
the noise that makes a board worth ignoring.
Underneath that: ShopDB could not distinguish those cases at all. On a no-op
the server returned "unchanged" and wrote nothing, so "we checked yesterday and
it matched" was discarded. That fact is the one thing a backup system must be
able to prove, and the only record of it was a line in a log file on the PC.
lastseenat records the check rather than the change. Touched on every matching
post including the no-op; set on creation, since a new revision has by
definition just been seen; backfilled from collectedat or createdat so existing
rows start from the last moment the config can be PROVEN current, rather than
from now - claiming a check that never happened would be worse than silence.
The card keys on it, one row per CHAIN rather than per asset: a machine with
two part markers can have one still reporting while the other stopped, and a
per-asset view would report the machine as fine. It stays deliberately silent
about assets never backed up, because whether one SHOULD be is a question only
the manifest can answer, and guessing would list a hundred healthy machines.
The rule lives in services/staleness.py rather than the route, so it is
testable without an auth layer in the way - the same split retention.py uses.
Threshold is backups_staledays, default 3, and 0 disables the card.
Adds a kind-pluggable backups plugin. Configuration captured from a PC is
filed against the MACHINE it controls, with a revision history and download
back to the native format.
NTLARS/DNC is the first kind. Settings live in the controlling PC's registry
but describe the machine, so revisions attach to the machine's asset and carry
no foreign key to the PC: history survives a PC being replaced or deleted, and
sourcehostname records the handover.
Storage splits by kind. Parseable kinds store a dialect-neutral JSON
projection in ShopDB and re-render on download; opaque vendor formats (part
marker and similar) keep their bytes on the SFLD share with ShopDB holding
metadata and the UNC pointer.
Two .reg dialects exist in the wild: NTLARS's own Save... export omits the
WOW6432Node path segment, scripted exports include it. Parsing strips whichever
root matched, so a stored revision commits to neither and download offers both
(NTLARS Load... by default, WOW6432Node for direct reg import). Getting this
backwards is silent, so the dedup hash deliberately excludes sourcedialect and
both dialects of one config dedup to a single revision.
Dedup is load-bearing: the collector runs every GE-Enforce cycle across the
fleet, so a revision is inserted only when the content hash differs from that
asset's latest for that kind.
A freshly imaged PC opens NTLARS with a blank General tab. Recording that would
make an empty config the newest revision exactly when someone needs the last
good one, so a blank MachineNo is rejected rather than accepted as a change.
Two of the 320 known-good backups on the share already have that shape.
DNC Info card summarises the latest revision on the machine page: General
(Cnc, NcIF, HostType), eFocas, Serial, NTSHR when populated (only 18 of 147
machines), and MARK when the machine is a marker. MARK is gated on Cnc=MARKER
or the ShopDB machine type, not on the MARK key having content: MARK carries
serial defaults on 145 of 147 machines and DncPatterns reads YES on 103
including ordinary lathes, so neither identifies a marker.
The info card is owned by the kind (BackupKind.infopanel/buildinfo) and served
by a generic endpoint, so the expected successor to DNC ships its own card by
adding a class rather than changing the plugin or the panel wiring.
Also: schedule and retention settings with a prune that never drops the newest
or the oldest revision, and scripts/import_ntlars_backups.py to seed history
from the existing per-machine .reg files (144 of 147 resolve to assets).
Codec verified against all 320 real backups: round-trips clean through both
dialects. Bay-side generation verified on Windows against reg.exe export.
