Let a share-backed backup say what is in it

A 'share' kind keeps its bytes on the file share and gives ShopDB only a hash
and a UNC path. That is right for an opaque vendor file, but it means the app
cannot answer the question a tech actually arrives with - what does this part
marker hold, and which pattern disappeared last month. Reading the share from
the app server is not an option: the download routes refuse to mount it on
purpose, because a permissions slip there becomes an unexplained empty panel.

So the collector now accepts an optional manifest alongside the hash: the
index, not the content. Names, sizes, per-file hashes, and the marking system
each pattern was authored for. Advisory throughout - a client that sends none
behaves exactly as before, and dedup still keys on contenthash.

The marker asset gains a Marker Contents card built from it: counts by type,
the controller family, and a diff against the previous snapshot in the same
source chain. Removed files are named rather than counted, because "1 removed"
only tells someone to go and look. Files edited in place are named too - the
file set can be identical while the content hash moved, and reporting no
change there would contradict the reason the revision exists.

The manifest is deliberately NOT covered by contenthash, which attests the
bytes on the share. A client that starts describing itself therefore posts an
unchanged hash, so a no-op backfills a manifest the row does not yet have -
otherwise the card would appear only on markers that had recently changed, and
stay hidden on every healthy one. The same no-op refreshes sharepath, so a
snapshot that moved on the share stops being pointed at where it used to be.

Manifests are validated at ingest rather than at render: a malformed one would
otherwise become the latest revision and fail every view of the asset page
until the content next changed, outliving the client bug that caused it.

sharedir() for the marker kind is overridden to the layout the site actually
uses, so what ShopDB displays and validates matches what the collector writes.
This commit is contained in:
cproudlock
2026-08-25 13:06:49 -04:00
parent 9d61e52ca7
commit 512cc9be76
6 changed files with 712 additions and 12 deletions

View File

@@ -67,7 +67,10 @@ class BackupRevision(db.Model):
# (assetid, backupkind).
contenthash = db.Column(db.String(64), nullable=False, index=True)
# Populated for storagebackend='shopdb' only. MEDIUMTEXT holding serialized
# Populated for storagebackend='shopdb', and optionally for 'share' kinds
# that send a manifest describing what their snapshot contains (the bytes
# still live on the share; this is the index, not the content).
# MEDIUMTEXT holding serialized
# JSON rather than a native JSON column, matching geenforce.manifestjson.
# Text also means the bytes come back exactly as written, so the canonical
# key ordering the codec produces survives the round trip - a native JSON
@@ -105,7 +108,18 @@ class BackupRevision(db.Model):
@property
def payload(self):
"""Decoded payloadjson, or None for opaque (share-backed) kinds."""
"""Decoded payloadjson, or None when the row stores nothing.
A 'shopdb' kind always has one: the parsed projection ShopDB is
authoritative for. A 'share' kind has one only when its client sends a
manifest describing the snapshot's contents, and None otherwise - which
includes every share revision written before manifests existed.
Note the manifest is NOT covered by contenthash: the hash attests the
bytes on the share, so two posts with identical content but different
manifests dedup to one row. apply_collector_payload backfills a missing
manifest on a no-op for that reason.
"""
if not self.payloadjson:
return None
try: