backups: a revision chain belongs to a PC, not just a machine
Dedup compared a posted config against the latest revision for the ASSET, which is only correct when a machine number means one PC. Several PCs share one here: the part markers on 0613, 0615 and WJPRT are separate devices, differing by COM port, filed under one machine number. Each marker's post therefore differed from whichever marker had posted last, nothing ever deduped, and the table grew by one row per PC per collection cycle. A chain is now (asset, kind, source hostname). An unchanged config is a no-op again, and each PC keeps its own history against the machine. NULL sources - rows written before the column was populated, and hand-loaded ones - form their own chain via IS NULL; `column == None` never matches in SQL, so without that those rows would have re-posted forever. Two consumers assumed the old key and are fixed with it. Retention pruned per asset, so a busy marker's revisions could evict a quiet marker's only backup; it now prunes each chain separately, protecting the newest and oldest of each. The revision diff compared against the previous revision on the machine, which across two markers reported one device's COM port as a change on the other; it now compares within the source's own chain. scripts/collapse_duplicate_backup_revisions.py cleans up what the old rule wrote. It removes only a revision whose hash repeats the one before it in the same chain - rows the fixed code would never have written - and keeps every genuine change, every chain's newest and oldest, and every source. Dry run by default. Its --report mode explains what grew each chain, which separates a legitimately shared machine number from two PCs wrongly carrying the same one, and from a value inside the config that changes on its own.
This commit is contained in:
@@ -25,7 +25,7 @@ from .api import backups_bp
|
||||
from .models import BackupRevision
|
||||
from .services.registry import (REGISTRY, getkind, canonicalhash,
|
||||
DEFAULTSHAREROOT)
|
||||
from .services.retention import prune
|
||||
from .services.retention import prune, samesource as _samesource
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -266,15 +266,26 @@ class BackupsPlugin(BasePlugin):
|
||||
"contenthash and sharepath are required for kind '{}'".format(
|
||||
kindkey))
|
||||
|
||||
# with_for_update serializes concurrent posts for the same asset+kind.
|
||||
# The whole fleet collects on the same GE-Enforce cycle, so a retry or
|
||||
# two PCs claiming one machine number can otherwise both pass the hash
|
||||
# check and insert duplicate identical revisions. No unique constraint
|
||||
# can cover this: dedup is against the LATEST row only, because a config
|
||||
# Dedup is against the LATEST row of THIS SOURCE's chain, not the
|
||||
# asset's. Several PCs legitimately share one machine number here - the
|
||||
# part markers on 0613, 0615 and WJPRT do, and their configs genuinely
|
||||
# differ, typically by COM port. Keyed on
|
||||
# asset alone, each marker's post differed from whichever marker posted
|
||||
# last, so nothing ever deduped and the chain grew by one revision per
|
||||
# PC per collection cycle. Keyed on the source, an unchanged config is a
|
||||
# no-op again and each marker keeps its own history against the machine.
|
||||
#
|
||||
# Dedup is against the latest row only, not any row, because a config
|
||||
# reverting to an earlier state is a legitimate new revision.
|
||||
#
|
||||
# with_for_update serializes concurrent posts for the same chain: the
|
||||
# fleet collects on one GE-Enforce cycle, so a retry could otherwise
|
||||
# pass the hash check twice and insert the same revision.
|
||||
sourcehostname = payload.get('sourcehostname')
|
||||
latest = (db.session.query(BackupRevision)
|
||||
.filter(BackupRevision.assetid == assetid,
|
||||
BackupRevision.backupkind == kindkey)
|
||||
BackupRevision.backupkind == kindkey,
|
||||
_samesource(sourcehostname))
|
||||
.order_by(BackupRevision.backuprevisionid.desc())
|
||||
.with_for_update()
|
||||
.first())
|
||||
|
||||
Reference in New Issue
Block a user