Let a share-backed backup say what is in it

A 'share' kind keeps its bytes on the file share and gives ShopDB only a hash
and a UNC path. That is right for an opaque vendor file, but it means the app
cannot answer the question a tech actually arrives with - what does this part
marker hold, and which pattern disappeared last month. Reading the share from
the app server is not an option: the download routes refuse to mount it on
purpose, because a permissions slip there becomes an unexplained empty panel.

So the collector now accepts an optional manifest alongside the hash: the
index, not the content. Names, sizes, per-file hashes, and the marking system
each pattern was authored for. Advisory throughout - a client that sends none
behaves exactly as before, and dedup still keys on contenthash.

The marker asset gains a Marker Contents card built from it: counts by type,
the controller family, and a diff against the previous snapshot in the same
source chain. Removed files are named rather than counted, because "1 removed"
only tells someone to go and look. Files edited in place are named too - the
file set can be identical while the content hash moved, and reporting no
change there would contradict the reason the revision exists.

The manifest is deliberately NOT covered by contenthash, which attests the
bytes on the share. A client that starts describing itself therefore posts an
unchanged hash, so a no-op backfills a manifest the row does not yet have -
otherwise the card would appear only on markers that had recently changed, and
stay hidden on every healthy one. The same no-op refreshes sharepath, so a
snapshot that moved on the share stops being pointed at where it used to be.

Manifests are validated at ingest rather than at render: a malformed one would
otherwise become the latest revision and fail every view of the asset page
until the content next changed, outliving the client bug that caused it.

sharedir() for the marker kind is overridden to the layout the site actually
uses, so what ShopDB displays and validates matches what the collector writes.

plugins/backups/api/routes.py

@@ -156,9 +156,30 @@ def asset_info(assetid):
     if revision is None:
         return success_response({'fields': [], 'sectioncount': 0})
 
+    try:
+        projection = revision.payload
+    except ValueError:
+        # payload deliberately raises on unreadable payloadjson so a corrupt row
+        # is named rather than mistaken for an opaque one. That is right for a
+        # download; for an at-a-glance card it would take the whole asset page
+        # down over one bad row, so the card degrades to empty instead.
+        current_app.logger.warning(
+            'backuprevision %s has unreadable payloadjson; info card empty',
+            revision.backuprevisionid)
+        return success_response({'fields': [], 'sectioncount': 0})
+
     data = kind.buildinfo(
-        revision.payload, assetid,
-        partmarkertypes=current_app.config.get('BACKUPS_PARTMARKER_TYPES'))
+        projection, assetid,
+        partmarkertypes=current_app.config.get('BACKUPS_PARTMARKER_TYPES'),
+        # A kind that diffs against its own previous revision needs to know
+        # which revision it is looking at. Passed generically rather than
+        # queried inside the kind, so the endpoint stays the single place that
+        # decides "latest".
+        revisionid=revision.backuprevisionid,
+        # The card names the PC a snapshot came from, and chains its diff on the
+        # same value. Passed from the column rather than read out of the stored
+        # manifest, which carries its own copy that can disagree.
+        sourcehostname=revision.sourcehostname)
     data['backuprevisionid'] = revision.backuprevisionid
     data['collectedat'] = _utciso(revision.collectedat)
     return success_response(data)

plugins/backups/models/backup.py

@@ -67,7 +67,10 @@ class BackupRevision(db.Model):
     # (assetid, backupkind).
     contenthash = db.Column(db.String(64), nullable=False, index=True)
 
-    # Populated for storagebackend='shopdb' only. MEDIUMTEXT holding serialized
+    # Populated for storagebackend='shopdb', and optionally for 'share' kinds
+    # that send a manifest describing what their snapshot contains (the bytes
+    # still live on the share; this is the index, not the content).
+    # MEDIUMTEXT holding serialized
     # JSON rather than a native JSON column, matching geenforce.manifestjson.
     # Text also means the bytes come back exactly as written, so the canonical
     # key ordering the codec produces survives the round trip - a native JSON
@@ -105,7 +108,18 @@ class BackupRevision(db.Model):
 
     @property
     def payload(self):
-        """Decoded payloadjson, or None for opaque (share-backed) kinds."""
+        """Decoded payloadjson, or None when the row stores nothing.
+
+        A 'shopdb' kind always has one: the parsed projection ShopDB is
+        authoritative for. A 'share' kind has one only when its client sends a
+        manifest describing the snapshot's contents, and None otherwise - which
+        includes every share revision written before manifests existed.
+
+        Note the manifest is NOT covered by contenthash: the hash attests the
+        bytes on the share, so two posts with identical content but different
+        manifests dedup to one row. apply_collector_payload backfills a missing
+        manifest on a no-op for that reason.
+        """
         if not self.payloadjson:
             return None
         try:

plugins/backups/plugin.py

@@ -263,6 +263,15 @@ class BackupsPlugin(BasePlugin):
                         'type': 'string',
                         'description': "Full UNC path. Required for 'share' kinds.",
                     },
+                    'manifest': {
+                        'type': 'object',
+                        'description': "Optional description of a 'share' kind's "
+                                       "contents - what is in the snapshot, not "
+                                       "the bytes themselves. Lets an asset panel "
+                                       "list a snapshot and diff it against the "
+                                       "previous one without the app server "
+                                       "mounting the share.",
+                    },
                     'sourcefilename': {'type': 'string'},
                     'sourcehostname': {
                         'type': 'string',
@@ -325,6 +334,42 @@ class BackupsPlugin(BasePlugin):
                     "contenthash and sharepath are required for kind '{}'".format(
                         kindkey))
 
+            # A share kind may also describe itself. The bytes stay on the
+            # share - that part is unchanged - but a manifest of what the
+            # snapshot CONTAINS is small, structured, and the only way an asset
+            # panel can list a part marker's patterns or say which one
+            # disappeared last month. The alternative is the app server
+            # mounting SFLD, which the read routes deliberately refuse to do.
+            #
+            # It is advisory: dedup still keys on contenthash, so a client that
+            # sends no manifest behaves exactly as before.
+            manifest = payload.get('manifest')
+            if manifest is not None:
+                # Validated at the door, not at render time. A shallow check
+                # here would let a malformed manifest become the LATEST
+                # revision, and every view of the asset panel would then fail
+                # until the content next changed - a client bug turning into a
+                # broken page that outlives it.
+                if not isinstance(manifest, dict):
+                    raise ValueError('manifest must be an object')
+                files = manifest.get('files')
+                if files is not None:
+                    if not isinstance(files, list):
+                        raise ValueError('manifest files must be a list')
+                    for entry in files:
+                        if not isinstance(entry, dict):
+                            raise ValueError(
+                                'each manifest file must be an object')
+                        if not isinstance(entry.get('name'), str) or not entry['name']:
+                            raise ValueError(
+                                'each manifest file needs a name')
+                        size = entry.get('bytes')
+                        if size is not None and not isinstance(size, int):
+                            raise ValueError(
+                                'manifest file {!r} has a non-integer size'.format(
+                                    entry['name']))
+                projection = manifest
+
         # Dedup is against the LATEST row of THIS SOURCE's chain, not the
         # asset's. Several PCs legitimately share one machine number here - the
         # part markers on 0613, 0615 and WJPRT do, and their configs genuinely
@@ -356,6 +401,26 @@ class BackupsPlugin(BasePlugin):
             # backup that died six months ago. Cheap: one column on a row that
             # already exists, no new history.
             latest.lastseenat = datetime.utcnow()
+
+            # BACKFILL, on a no-op. The contenthash attests the BYTES, not the
+            # manifest, so a client that starts describing itself posts the same
+            # hash as before and would otherwise no-op forever - leaving the
+            # index unstored on exactly the markers that are healthy and
+            # unchanging, which is most of them. A revision that already has a
+            # manifest is left alone; this only fills a hole.
+            #
+            # Honest by construction: the hash proves the content is identical,
+            # so indexing it now describes the same snapshot. lastseenat
+            # semantics are untouched.
+            if projection is not None and latest.payload is None:
+                latest.payload = projection
+            # Same reasoning for the pointer: a snapshot that moved on the share
+            # (a changed layout, a renamed asset directory) reports a new
+            # sharepath with unchanged bytes, and the stored path would
+            # otherwise go on pointing at a directory nobody writes to.
+            if sharepath and latest.sharepath != sharepath:
+                latest.sharepath = sharepath
+
             db.session.flush()
             return {
                 'action': 'noop',

plugins/backups/services/partmarkerinfo.py

@@ -0,0 +1,176 @@
+"""Marker Contents card.
+
+What a part marker holds right now, on the marker's own asset page, plus what
+changed since the previous snapshot.
+
+WHY THIS EXISTS AT ALL
+
+    A 'share' kind keeps its bytes on SFLD and only a UNC pointer in ShopDB, so
+    the obvious way to list a snapshot's contents - read the directory - is
+    exactly what the read routes refuse to do: mounting the share in the app
+    server turns a permissions slip into an unexplained empty panel. Instead the
+    collector sends a MANIFEST alongside the hash, and this builds the card from
+    that. Small, structured, and it works from a server that cannot see the
+    share at all.
+
+WHAT IT ANSWERS
+
+    "What patterns are on this marker" - the question a tech arrives with, and
+    today the only way to answer it is to walk to the bay and read the panel.
+
+    "What changed" - the added/removed diff against the previous snapshot. This
+    is the reason the snapshot layout exists: a pattern deleted from a
+    controller is invisible everywhere else until someone needs it. Here it is
+    dated, and the snapshot that still holds it is one row above in the history.
+
+The manifest is written by Backup-PartMarker.ps1 and carries, per file: name,
+bytes, sha256, and the marking system it was authored for.
+"""
+
+
+def _previousmanifest(assetid, currentid):
+    """The manifest of the snapshot before this one, or None.
+
+    Scoped to the same source host, matching how dedup chains revisions: two
+    PCs reporting the same marker keep separate histories, and diffing across
+    them would report every file as added and removed at once.
+
+    The host comes from the REVISION's own column, not from a field inside the
+    manifest. Those can disagree - the column is what the collector recorded and
+    what retention chains on - and using the manifest's copy would silently diff
+    against the wrong chain, or against nothing at all.
+    """
+    from shopdb.api import db
+    from ..models import BackupRevision
+    from .retention import samesource
+
+    current = db.session.get(BackupRevision, currentid)
+    if current is None:
+        return None, None
+    sourcehostname = current.sourcehostname
+
+    row = (db.session.query(BackupRevision)
+           .filter(BackupRevision.assetid == assetid,
+                   BackupRevision.backupkind == 'partmarker',
+                   BackupRevision.backuprevisionid < currentid,
+                   samesource(sourcehostname))
+           .order_by(BackupRevision.backuprevisionid.desc())
+           .first())
+    if row is None:
+        return None, None
+    when = row.collectedat or row.createdat
+    try:
+        return row.payload, when
+    except ValueError:
+        # A corrupt neighbour must not take the card down - the current
+        # snapshot is still worth showing without its diff.
+        return None, None
+
+
+def _hashes(manifest):
+    """name -> sha256, for spotting a file edited in place."""
+    return {entry.get('name'): entry.get('sha256')
+            for entry in (manifest or {}).get('files', [])
+            if entry.get('name') and entry.get('sha256')}
+
+
+def _names(manifest):
+    return {entry.get('name') for entry in (manifest or {}).get('files', [])
+            if entry.get('name')}
+
+
+def _summarise(files):
+    """Counts by extension, so 34 patterns and 12 settings read as two facts."""
+    groups = {}
+    for entry in files:
+        name = entry.get('name') or ''
+        ext = name.rsplit('.', 1)[-1].upper() if '.' in name else 'OTHER'
+        groups[ext] = groups.get(ext, 0) + 1
+    labels = {'PAT': 'Patterns', 'DAT': 'Settings files', 'TFF': 'Fonts'}
+    return [(labels.get(ext, ext), count) for ext, count in sorted(groups.items())]
+
+
+def build(projection, assetid, revisionid=None, sourcehostname=None,
+          **options):
+    """Build the card from a stored partmarker manifest.
+
+    Returns the keyvalue shape. An older revision predating manifests has no
+    projection at all, which renders as no card rather than an error - the
+    revision itself is still listed in the history panel.
+    """
+    # "No manifest" and "a manifest listing nothing" are different facts. The
+    # first is an older revision predating manifests, which has no card. The
+    # second is a controller that has been wiped or replaced - precisely the
+    # disaster the Removed line exists to report - so it must render, with the
+    # diff intact.
+    if not projection:
+        return {'fields': [], 'sectioncount': 0}
+    manifest = projection
+    files = manifest.get('files') or []
+
+    fields = []
+
+    for label, count in _summarise(files):
+        fields.append({'label': label, 'value': str(count)})
+
+    total = sum(int(entry.get('bytes') or 0) for entry in files)
+    fields.append({'label': 'Total size', 'value': '{:,} bytes'.format(total)})
+
+    if manifest.get('family'):
+        fields.append({'label': 'Controller', 'value': manifest['family']})
+
+    # The heads patterns were authored for. Worth showing because they can
+    # DIFFER from the controller holding them - a TMC420 in the reference site
+    # holds both a TMP3200/470 and a TMP4210/420 pattern - and pushing a pattern
+    # back onto a system with a different head is a mistake worth catching
+    # before the restore, not after.
+    systems = manifest.get('systems') or []
+    if systems:
+        fields.append({'label': 'Authored for', 'value': ', '.join(systems)})
+
+    # From the revision's own column where the caller passes it: the manifest
+    # carries its own copy, and the two can disagree. The column is what the
+    # collector recorded and what the diff above is chained on, so naming a
+    # different PC here would contradict the comparison shown beside it.
+    capturedfrom = sourcehostname or manifest.get('sourcehost')
+    if capturedfrom:
+        fields.append({'label': 'Captured from', 'value': capturedfrom})
+
+    # The diff. Only meaningful with a predecessor: on a marker's first snapshot
+    # every file would otherwise read as "added", which says nothing.
+    if revisionid:
+        previous, when = _previousmanifest(assetid, revisionid)
+        if previous is not None:
+            before, now = _names(previous), _names(manifest)
+            added = sorted(now - before)
+            removed = sorted(before - now)
+
+            # Same name, different bytes. A revision exists only because the
+            # content hash moved, so if the file SET is identical then something
+            # was edited in place - and saying "no files added or removed" would
+            # be a flat contradiction of why this row exists. The per-file
+            # sha256 is in the manifest, so the edited files can be named.
+            beforehashes = _hashes(previous)
+            changed = sorted(name for name, digest in _hashes(manifest).items()
+                             if name in beforehashes and beforehashes[name] != digest)
+
+            # Date the window. Retention prunes middle revisions, so "previous"
+            # can be months older than the row before it in the history, and an
+            # undated diff would quietly attribute an old removal to this
+            # capture.
+            label = 'Since {}'.format(when.strftime('%Y-%m-%d')) if when else 'Since previous'
+
+            if added:
+                fields.append({'label': 'Added', 'value': ', '.join(added)})
+            if removed:
+                # The one the whole design exists for. Named, not counted -
+                # "1 removed" would send someone to the share to find out which.
+                fields.append({'label': 'Removed', 'value': ', '.join(removed)})
+            if changed:
+                fields.append({'label': 'Changed', 'value': ', '.join(changed)})
+            if not (added or removed or changed):
+                fields.append({'label': label, 'value': 'No files added, removed or changed'})
+            else:
+                fields.append({'label': 'Compared with', 'value': label.replace('Since ', '')})
+
+    return {'fields': fields, 'sectioncount': 1}

plugins/backups/services/registry.py

@@ -279,11 +279,60 @@ class PartMarkerKind(BackupKind):
     key = 'partmarker'
     displayname = 'Part Marker Configuration'
     storagebackend = 'share'
-    # NO asset panel. The marker's DNC settings are a tab on the single DNC Info
-    # card, and a separate card that is empty on 145 of 147 machines earns
-    # nobody anything. The kind is still fully live - it stores, dedupes and
-    # serves revisions, which are listed on the backup history page.
-    assettypes = []
+    # Marker assets are core type 'machine', like the other 144 machines - but
+    # only a marker ever has a partmarker revision, and the list renderer hides
+    # a panel with no rows and no empty text. So declaring 'machine' shows the
+    # panel on markers and nowhere else, with no special-casing.
+    #
+    # This was deliberately EMPTY while backups were filed against the operation
+    # number, when a partmarker card would have appeared on every machine and
+    # said nothing. Since a marker PC's backup is filed against its own marker
+    # asset (see markerforsource), the panel has a home of its own.
+    assettypes = ['machine']
+
+    def sharedir(self, machinetype, identifier, shareroot=None):
+        """<root>\\partmarker\\<marker asset>\\ - the SITE's layout, which
+        inverts the base class's <root>\\<machinetype>\\<identifier>\\<kind>.
+
+        Overridden rather than left to the default so what ShopDB displays and
+        validates matches what the collector actually writes. Still advisory:
+        the authoritative path is the sharepath on each revision, which is what
+        keeps older snapshots resolvable after a PC swap renames the asset.
+
+        Snapshots live in timestamped subdirectories BELOW this - one per
+        capture, so a pattern deleted from the controller stays recoverable
+        from the snapshot that still holds it.
+        """
+        root = shareroot or DEFAULTSHAREROOT
+        if not root:
+            return ''
+        return '{}\\{}\\{}'.format(root, self.key, identifier or 'unknown')
+
+    def infopanel(self):
+        """Contents of the latest snapshot, on the marker's own asset page.
+
+        A 'share' kind stores no bytes in ShopDB, so this is built from the
+        manifest the collector sends alongside the hash. Without it the app
+        server would have to mount SFLD to say what a snapshot contains, which
+        the read routes refuse to do on purpose.
+        """
+        return {
+            'id': 'backups-partmarker-contents',
+            'title': 'Marker Contents',
+            'assettypes': ['machine'],
+            'endpoint': '/api/backups/asset/{assetid}/info?kind=partmarker',
+            'render': 'keyvalue',
+            # Above the history panel: "what is on this marker" is the question
+            # a tech arrives with; the history is for the rarer restore.
+            'position': 39,
+        }
+
+    def buildinfo(self, projection, assetid, **options):
+        from . import partmarkerinfo
+        return partmarkerinfo.build(
+            projection, assetid,
+            revisionid=options.get('revisionid'),
+            sourcehostname=options.get('sourcehostname'))
 
     def resolveassetid(self, payload):
         from shopdb.api import db, Asset