backups: a revision chain belongs to a PC, not just a machine

Dedup compared a posted config against the latest revision for the ASSET,
which is only correct when a machine number means one PC. Several PCs share one
here: the part markers on 0613, 0615 and WJPRT are separate devices, differing
by COM port, filed under one machine number. Each marker's post therefore
differed from whichever marker had posted last, nothing ever deduped, and the
table grew by one row per PC per collection cycle.

A chain is now (asset, kind, source hostname). An unchanged config is a no-op
again, and each PC keeps its own history against the machine. NULL sources -
rows written before the column was populated, and hand-loaded ones - form their
own chain via IS NULL; `column == None` never matches in SQL, so without that
those rows would have re-posted forever.

Two consumers assumed the old key and are fixed with it. Retention pruned per
asset, so a busy marker's revisions could evict a quiet marker's only backup;
it now prunes each chain separately, protecting the newest and oldest of each.
The revision diff compared against the previous revision on the machine, which
across two markers reported one device's COM port as a change on the other; it
now compares within the source's own chain.

scripts/collapse_duplicate_backup_revisions.py cleans up what the old rule
wrote. It removes only a revision whose hash repeats the one before it in the
same chain - rows the fixed code would never have written - and keeps every
genuine change, every chain's newest and oldest, and every source. Dry run by
default. Its --report mode explains what grew each chain, which separates a
legitimately shared machine number from two PCs wrongly carrying the same one,
and from a value inside the config that changes on its own.
This commit is contained in:
cproudlock
2026-08-10 15:02:38 -04:00
parent 5108ba8aaa
commit d429c882b4
6 changed files with 376 additions and 16 deletions

View File

@@ -2,18 +2,23 @@
Dedup already keeps growth low - a revision appears only when a setting really
changed - so retention exists for the pathological case, not the normal one: a
value that flaps (or two PCs alternately claiming one machine number) would
otherwise append a revision every collection cycle forever.
value that flaps would otherwise append a revision every collection cycle
forever.
Two independent limits, both off by default at 0:
retentioncount keep at most N revisions per asset per kind
retentioncount keep at most N revisions per chain
retentiondays additionally drop anything older than N days
The NEWEST and the OLDEST revision are never pruned. The newest is the one a
tech restores from; the oldest is the earliest known-good baseline, which is
usually the most valuable row in the table and the one a naive "keep last N"
would delete first.
A CHAIN is one asset, one kind, one source hostname. Several PCs can share a
machine number, and each keeps its own revision history against that machine;
pruning per asset would let a busy marker's revisions push a quiet marker's
only backup out of the table.
The NEWEST and the OLDEST revision of each chain are never pruned. The newest
is the one a tech restores from; the oldest is the earliest known-good
baseline, which is usually the most valuable row in the table and the one a
naive "keep last N" would delete first.
"""
from datetime import datetime, timedelta
@@ -23,11 +28,45 @@ from shopdb.api import db
from ..models import BackupRevision
def samesource(sourcehostname):
"""Filter matching one source's chain, treating NULL as its own source.
`sourcehostname == None` never matches in SQL, so a NULL-sourced row - every
row written before the column was populated, and any hand-loaded one - needs
IS NULL. Without this those rows form no chain at all and re-post forever.
"""
if sourcehostname is None or sourcehostname == '':
return BackupRevision.sourcehostname.is_(None)
return BackupRevision.sourcehostname == sourcehostname
def sourcesfor(assetid, backupkind):
"""Every distinct source hostname with a revision for this asset+kind."""
rows = (db.session.query(BackupRevision.sourcehostname)
.filter(BackupRevision.assetid == assetid,
BackupRevision.backupkind == backupkind)
.distinct()
.all())
return [row[0] for row in rows]
def prune(assetid, backupkind, retentioncount=0, retentiondays=0):
"""Delete surplus revisions for one asset+kind. Returns the number removed.
Caller commits. Returns 0 when both limits are disabled.
Prunes each source's chain independently. Caller commits. Returns 0 when
both limits are disabled.
"""
if int(retentioncount or 0) <= 0 and int(retentiondays or 0) <= 0:
return 0
return sum(prunechain(assetid, backupkind, source,
retentioncount=retentioncount,
retentiondays=retentiondays)
for source in sourcesfor(assetid, backupkind))
def prunechain(assetid, backupkind, sourcehostname, retentioncount=0,
retentiondays=0):
"""Delete surplus revisions for one asset+kind+source chain."""
retentioncount = int(retentioncount or 0)
retentiondays = int(retentiondays or 0)
if retentioncount <= 0 and retentiondays <= 0:
@@ -35,7 +74,8 @@ def prune(assetid, backupkind, retentioncount=0, retentiondays=0):
revisions = (db.session.query(BackupRevision)
.filter(BackupRevision.assetid == assetid,
BackupRevision.backupkind == backupkind)
BackupRevision.backupkind == backupkind,
samesource(sourcehostname))
.order_by(BackupRevision.backuprevisionid.desc())
.all())
if len(revisions) <= 2: