backups: a revision chain belongs to a PC, not just a machine

Dedup compared a posted config against the latest revision for the ASSET,
which is only correct when a machine number means one PC. Several PCs share one
here: the part markers on 0613, 0615 and WJPRT are separate devices, differing
by COM port, filed under one machine number. Each marker's post therefore
differed from whichever marker had posted last, nothing ever deduped, and the
table grew by one row per PC per collection cycle.

A chain is now (asset, kind, source hostname). An unchanged config is a no-op
again, and each PC keeps its own history against the machine. NULL sources -
rows written before the column was populated, and hand-loaded ones - form their
own chain via IS NULL; `column == None` never matches in SQL, so without that
those rows would have re-posted forever.

Two consumers assumed the old key and are fixed with it. Retention pruned per
asset, so a busy marker's revisions could evict a quiet marker's only backup;
it now prunes each chain separately, protecting the newest and oldest of each.
The revision diff compared against the previous revision on the machine, which
across two markers reported one device's COM port as a change on the other; it
now compares within the source's own chain.

scripts/collapse_duplicate_backup_revisions.py cleans up what the old rule
wrote. It removes only a revision whose hash repeats the one before it in the
same chain - rows the fixed code would never have written - and keeps every
genuine change, every chain's newest and oldest, and every source. Dry run by
default. Its --report mode explains what grew each chain, which separates a
legitimately shared machine number from two PCs wrongly carrying the same one,
and from a value inside the config that changes on its own.
This commit is contained in:
cproudlock
2026-08-10 15:02:38 -04:00
parent 5108ba8aaa
commit d429c882b4
6 changed files with 376 additions and 16 deletions

View File

@@ -25,7 +25,7 @@ from .api import backups_bp
from .models import BackupRevision
from .services.registry import (REGISTRY, getkind, canonicalhash,
DEFAULTSHAREROOT)
from .services.retention import prune
from .services.retention import prune, samesource as _samesource
logger = logging.getLogger(__name__)
@@ -266,15 +266,26 @@ class BackupsPlugin(BasePlugin):
"contenthash and sharepath are required for kind '{}'".format(
kindkey))
# with_for_update serializes concurrent posts for the same asset+kind.
# The whole fleet collects on the same GE-Enforce cycle, so a retry or
# two PCs claiming one machine number can otherwise both pass the hash
# check and insert duplicate identical revisions. No unique constraint
# can cover this: dedup is against the LATEST row only, because a config
# Dedup is against the LATEST row of THIS SOURCE's chain, not the
# asset's. Several PCs legitimately share one machine number here - the
# part markers on 0613, 0615 and WJPRT do, and their configs genuinely
# differ, typically by COM port. Keyed on
# asset alone, each marker's post differed from whichever marker posted
# last, so nothing ever deduped and the chain grew by one revision per
# PC per collection cycle. Keyed on the source, an unchanged config is a
# no-op again and each marker keeps its own history against the machine.
#
# Dedup is against the latest row only, not any row, because a config
# reverting to an earlier state is a legitimate new revision.
#
# with_for_update serializes concurrent posts for the same chain: the
# fleet collects on one GE-Enforce cycle, so a retry could otherwise
# pass the hash check twice and insert the same revision.
sourcehostname = payload.get('sourcehostname')
latest = (db.session.query(BackupRevision)
.filter(BackupRevision.assetid == assetid,
BackupRevision.backupkind == kindkey)
BackupRevision.backupkind == kindkey,
_samesource(sourcehostname))
.order_by(BackupRevision.backuprevisionid.desc())
.with_for_update()
.first())