d429c882b4
Dedup compared a posted config against the latest revision for the ASSET, which is only correct when a machine number means one PC. Several PCs share one here: the part markers on 0613, 0615 and WJPRT are separate devices, differing by COM port, filed under one machine number. Each marker's post therefore differed from whichever marker had posted last, nothing ever deduped, and the table grew by one row per PC per collection cycle. A chain is now (asset, kind, source hostname). An unchanged config is a no-op again, and each PC keeps its own history against the machine. NULL sources - rows written before the column was populated, and hand-loaded ones - form their own chain via IS NULL; `column == None` never matches in SQL, so without that those rows would have re-posted forever. Two consumers assumed the old key and are fixed with it. Retention pruned per asset, so a busy marker's revisions could evict a quiet marker's only backup; it now prunes each chain separately, protecting the newest and oldest of each. The revision diff compared against the previous revision on the machine, which across two markers reported one device's COM port as a change on the other; it now compares within the source's own chain. scripts/collapse_duplicate_backup_revisions.py cleans up what the old rule wrote. It removes only a revision whose hash repeats the one before it in the same chain - rows the fixed code would never have written - and keeps every genuine change, every chain's newest and oldest, and every source. Dry run by default. Its --report mode explains what grew each chain, which separates a legitimately shared machine number from two PCs wrongly carrying the same one, and from a value inside the config that changes on its own.
109 lines
4.2 KiB
Python
109 lines
4.2 KiB
Python
"""Revision retention.
|
|
|
|
Dedup already keeps growth low - a revision appears only when a setting really
|
|
changed - so retention exists for the pathological case, not the normal one: a
|
|
value that flaps would otherwise append a revision every collection cycle
|
|
forever.
|
|
|
|
Two independent limits, both off by default at 0:
|
|
|
|
retentioncount keep at most N revisions per chain
|
|
retentiondays additionally drop anything older than N days
|
|
|
|
A CHAIN is one asset, one kind, one source hostname. Several PCs can share a
|
|
machine number, and each keeps its own revision history against that machine;
|
|
pruning per asset would let a busy marker's revisions push a quiet marker's
|
|
only backup out of the table.
|
|
|
|
The NEWEST and the OLDEST revision of each chain are never pruned. The newest
|
|
is the one a tech restores from; the oldest is the earliest known-good
|
|
baseline, which is usually the most valuable row in the table and the one a
|
|
naive "keep last N" would delete first.
|
|
"""
|
|
|
|
from datetime import datetime, timedelta
|
|
|
|
from shopdb.api import db
|
|
|
|
from ..models import BackupRevision
|
|
|
|
|
|
def samesource(sourcehostname):
|
|
"""Filter matching one source's chain, treating NULL as its own source.
|
|
|
|
`sourcehostname == None` never matches in SQL, so a NULL-sourced row - every
|
|
row written before the column was populated, and any hand-loaded one - needs
|
|
IS NULL. Without this those rows form no chain at all and re-post forever.
|
|
"""
|
|
if sourcehostname is None or sourcehostname == '':
|
|
return BackupRevision.sourcehostname.is_(None)
|
|
return BackupRevision.sourcehostname == sourcehostname
|
|
|
|
|
|
def sourcesfor(assetid, backupkind):
|
|
"""Every distinct source hostname with a revision for this asset+kind."""
|
|
rows = (db.session.query(BackupRevision.sourcehostname)
|
|
.filter(BackupRevision.assetid == assetid,
|
|
BackupRevision.backupkind == backupkind)
|
|
.distinct()
|
|
.all())
|
|
return [row[0] for row in rows]
|
|
|
|
|
|
def prune(assetid, backupkind, retentioncount=0, retentiondays=0):
|
|
"""Delete surplus revisions for one asset+kind. Returns the number removed.
|
|
|
|
Prunes each source's chain independently. Caller commits. Returns 0 when
|
|
both limits are disabled.
|
|
"""
|
|
if int(retentioncount or 0) <= 0 and int(retentiondays or 0) <= 0:
|
|
return 0
|
|
return sum(prunechain(assetid, backupkind, source,
|
|
retentioncount=retentioncount,
|
|
retentiondays=retentiondays)
|
|
for source in sourcesfor(assetid, backupkind))
|
|
|
|
|
|
def prunechain(assetid, backupkind, sourcehostname, retentioncount=0,
|
|
retentiondays=0):
|
|
"""Delete surplus revisions for one asset+kind+source chain."""
|
|
retentioncount = int(retentioncount or 0)
|
|
retentiondays = int(retentiondays or 0)
|
|
if retentioncount <= 0 and retentiondays <= 0:
|
|
return 0
|
|
|
|
revisions = (db.session.query(BackupRevision)
|
|
.filter(BackupRevision.assetid == assetid,
|
|
BackupRevision.backupkind == backupkind,
|
|
samesource(sourcehostname))
|
|
.order_by(BackupRevision.backuprevisionid.desc())
|
|
.all())
|
|
if len(revisions) <= 2:
|
|
return 0 # newest + oldest are both protected
|
|
|
|
newest = revisions[0]
|
|
oldest = revisions[-1]
|
|
protected = {newest.backuprevisionid, oldest.backuprevisionid}
|
|
|
|
doomed = []
|
|
|
|
if retentioncount > 0 and len(revisions) > retentioncount:
|
|
# Walk from the oldest END of the middle, so the surplus dropped is
|
|
# always the least recent, never the newest.
|
|
for revision in revisions[retentioncount:]:
|
|
if revision.backuprevisionid not in protected:
|
|
doomed.append(revision)
|
|
|
|
if retentiondays > 0:
|
|
cutoff = datetime.utcnow() - timedelta(days=retentiondays)
|
|
for revision in revisions:
|
|
if revision.backuprevisionid in protected:
|
|
continue
|
|
stamp = revision.collectedat or revision.createdat
|
|
if stamp and stamp < cutoff and revision not in doomed:
|
|
doomed.append(revision)
|
|
|
|
for revision in doomed:
|
|
db.session.delete(revision)
|
|
return len(doomed)
|