Import the wilm endpoint telemetry, PC Detail rework and backup overlay

Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at 7905917c. wilm rides the publication lineage and shares no
ancestor with wj, so this is a filtered tree patch rather than a merge:
94 files, and every path the publication export strips (docs/, tools/, mcp/,
CLAUDE.md, start-*.sh, .gitea/, the three excluded tests) is held back, because
a raw diff would delete each of them from this tree.

What comes across:

* Endpoint telemetry, phase 2. Physical disk health from two sources, NIC link
  state, boot type, Defender health, processor/memory/chassis inventory, the
  listening TCP set, mapped network drives and Windows Update history. Five
  additive migrations, 0012 through 0016, off the 0011 head.
* Mapped drives are collected by a probe running in the interactive user's
  session, because they do not exist in the agent's session 0. Two Windows
  sources can disagree about one drive and the worse status wins, so a
  half-broken share is never rendered as healthy.
* Windows events are reported once each, keyed on the channel's own record id,
  instead of being re-sent whenever a report overlapped the last one.
* PC Detail rework: Details cards flow, the hand-written Network card gives way
  to a full address history, Storage and disk health merge into one card with
  two sections, Operator Reports and Corporate Tools get their own tabs.
* An opt-in restic backup overlay with a third container, restore gated on the
  dump's own data revision rather than the code head, and backup/restore history.
* The agent's version is read from the binary's PE version rather than trusted
  from the csproj, and the Fleet view gains an Agent Version column.
* The enqueue route now asks caller_holds instead of the owner's permissions,
  which closes the scoped-token elevation on run_script, update_app and
  system_restart.

Held back deliberately:

* agent/installer/install-shopdbagent.ps1 stays deleted, as upstream deleted it.
  The installer that reaches a bay is generated by _install_script() in
  plugins/endpoints/api/siteagent.py; the standalone script had stopped being
  shipped and was collecting fixes no endpoint ever ran.
* The .gitignore hunk adding docs/*.md. Correct on the publication lineage,
  where docs/ becomes the wiki and its absence is how the gates detect a
  published checkout. Here it would silently ignore every new doc.
* Their deletions of docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md. Upstream
  those were stubs; here PLUGIN-HOOKS.md is the 31KB document CLAUDE.md points
  readers at.
* Five zero-byte files committed by an accidental `git add -A`, a run log from
  the author's own machine, and docs/notes/, which is working scratch that says
  so in its own header and belongs in devnotes/ on this side.

Carried with the import so it lands green: the 17 new routes are described in
docs/api-inventory.json with the spec and project map regenerated, and two new
frontend specs read fixtures through fileURLToPath(URL) which node rejects under
jsdom's own URL global.

Suite 2591 passed / 9 skipped; frontend 391 passed; naming, style and doc gates
clean.

Scope: product

scripts/backup/.gitattributes

@@ -0,0 +1,4 @@
+# Keep shell scripts LF-only so they run inside Linux containers regardless of
+# the host checkout. A CRLF shebang makes the kernel look for "bash\r" and the
+# container dies with: env: 'bash\r': No such file or directory.
+*.sh text eol=lf

scripts/backup/Dockerfile.backup

@@ -0,0 +1,38 @@
+# Backup sidecar image: mysql:8.4 (for a version-matched mysqldump) plus restic
+# (for deduped, encrypted, destination-agnostic file snapshots) and a small
+# Python for the manifest writer.
+#
+# WHY mysql:8.4 as the base: the dump tool must match the server major/minor so
+# the SQL it emits restores cleanly onto the same engine. Reusing the exact
+# image the db service runs removes version drift as a failure mode.
+FROM mysql:8.4
+
+# Track 1 local build unblock: trust the corporate proxy (Zscaler/GE) root CA so
+# the build-time fetches below (Oracle yum, github.com, PyPI) succeed behind TLS
+# interception. This is a local convenience only; the shipping build vendors
+# these deps instead of fetching them (pinned restic + SHA256, vendored
+# bzip2/python3/alembic or a base image that carries them) so the release image
+# needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
+# and must be staged at scripts/backup/zscaler-root-ca.pem before building.
+COPY scripts/backup/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/zscaler-root-ca.pem
+RUN update-ca-trust
+
+# restic from the distro is often stale; pull the pinned release binary so the
+# snapshot format and retention flags are predictable across sites.
+ARG RESTIC_VERSION=0.17.3
+RUN microdnf install -y bzip2 python3 python3-pip ca-certificates \
+	&& curl -fsSL -o /tmp/restic.bz2 \
+		"https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2" \
+	&& bunzip2 /tmp/restic.bz2 \
+	&& install -m 0755 /tmp/restic /usr/local/bin/restic \
+	&& rm -f /tmp/restic \
+	&& microdnf clean all
+
+# alembic so the manifest writer can read the current schema head. pip does not
+# read the system trust store, so point it at the extracted system CA bundle
+# (now including the proxy root) to get through TLS interception.
+RUN python3 -m pip install --no-cache-dir \
+	--cert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
+	alembic
+
+ENTRYPOINT ["/opt/backup/backup.sh"]

scripts/backup/SIMULATE-RESTORE.README.md

@@ -0,0 +1,44 @@
+# Restore Simulation (FOR DEVELOPER USE ONLY)
+
+`simulate-restore.sh` is a developer diagnostic that proves the backup/restore
+pair round-trips the live database. It is NOT an operator tool and NOT part of
+the shipped feature. It mutates a live database on purpose - never run it
+against production or anything you cannot disturb.
+
+## Testing cadence
+
+- Before merging any change to `scripts/backup/*.sh`, `docker-compose.backup.yml`,
+  or `shopdb/core/api/shopdbbackups.py`.
+- Before cutting a release that touches the backup/restore surface.
+- After a schema migration that adds or drops tables (new table archetype).
+- Ad hoc when investigating a restore-fidelity concern.
+
+## What it does
+
+1. Requests a backup through the real `.backup-requested` sentinel.
+2. Fingerprints every base table with `CHECKSUM TABLE` (the restore target).
+3. Quiesces the `api` service, mutates the table archetypes:
+   - core standalone (vendor rename, setting flip)
+   - consumable ledger (`modelsupplies.quantityonhand` bump)
+4. Requests a restore through the real `.restore-requested` sentinel.
+5. Re-fingerprints and asserts every table returned to backup-time state.
+6. Writes `simulate-restore-results.txt` next to the script.
+
+## Testing parameters
+
+- Stack must be up with the backup profile:
+  `docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup up -d`
+- Run from git bash on the host: `bash scripts/backup/simulate-restore.sh`
+- Coverage: all `BASE TABLE` rows in `shopdb_flask` (110+ tables), one pass.
+
+## Expected drift (NOT failures)
+
+- `apitokens`: the app rewrites token last-used continuously; restore is
+  correct at backup time.
+- Endpoint-agent tables (`computers`, `endpointstates`, `endpointevents`,
+  `endpointapplications`): external fleet agents keep writing after restore and
+  cannot be quiesced from here.
+- Asset ROW deletion is not permitted in a live/CMMC environment (RESTRICT +
+  compliance). The cascade edge is reported informationally only. A missing or
+  corrupted asset is still fully restored because the dump drops and recreates
+  the table - restore fidelity does not depend on a live delete.

scripts/backup/backup.sh

@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# ShopDB backup one-shot.
+#
+# Produces one recoverable set per run and stores it two ways: a plain
+# db.sql.gz + manifest.json on the local /backups volume for a fast, repo-free
+# restore, and a restic snapshot (db dump + instance tree + manifest) at
+# RESTIC_REPOSITORY for a deduped, encrypted, offsite copy.
+#
+# Sequence:
+#   1. mysqldump --single-transaction (InnoDB-consistent, non-blocking)
+#   2. manifest.py stamps core+plugin Alembic revisions + image tag
+#   3. restic backup the dump + /app/instance + manifest as ONE snapshot
+#   4. restic forget --keep-daily 5 --keep-weekly 4 --keep-monthly 6 --prune
+#   5. restic check - fail loudly if the repo is unhealthy, so a bad repo is
+#      caught on backup night, not restore night.
+#
+# ASCII only, no emojis - matches the repo style gate.
+set -euo pipefail
+
+BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
+SITE="${SHOPDB_SITE:-shopdb}"
+STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
+SETNAME="${SITE}-${STAMP}"
+SETDIR="${BACKUPS_ROOT}/${SETNAME}"
+
+# shellcheck source=scripts/backup/status.sh
+. /opt/backup/status.sh
+
+log() { echo "[backup] $*"; }
+
+# Record the run outcome on any exit so a failed backup is visible in the UI
+# history, not just a silent gap.
+BACKUP_DONE=false
+_on_exit() {
+	code=$?
+	if [ "${BACKUP_DONE}" = true ]; then
+		status_write backup done "${SETNAME}" "Backup complete"
+		history_append backup success "${SETNAME}" "Backup complete"
+	else
+		status_write backup failed "${SETNAME}" "Backup failed (exit ${code})"
+		history_append backup failed "${SETNAME}" "Backup failed (exit ${code})"
+	fi
+}
+trap _on_exit EXIT
+
+mkdir -p "${SETDIR}"
+status_write backup database "${SETNAME}" "Dumping database"
+
+# 1. Database: whole-DB logical dump, consistent and non-blocking on InnoDB.
+log "dumping database ${MYSQL_DATABASE} from ${MYSQL_HOST}"
+mysqldump \
+	--host="${MYSQL_HOST}" \
+	--user="${MYSQL_USER}" \
+	--password="${MYSQL_PASSWORD}" \
+	--single-transaction \
+	--routines \
+	--triggers \
+	--databases "${MYSQL_DATABASE}" \
+	| gzip -c > "${SETDIR}/db.sql.gz"
+log "database dump written ($(du -h "${SETDIR}/db.sql.gz" | cut -f1))"
+
+# 2. Manifest: the schema stamp that makes a restore a checked operation.
+# Capture the revision actually applied to the DATABASE (its alembic_version),
+# not just the code head. A restore must gate on the DATA revision; stamping
+# only the code head hides an image that has outrun its own database and lets a
+# behind-schema dump restore under ahead-schema code (1054 Unknown column on
+# every affected table). Query it straight from the dumped DB.
+log "reading applied database revision (alembic_version)"
+DATAREV="$(mysql \
+	--host="${MYSQL_HOST}" \
+	--user="${MYSQL_USER}" \
+	--password="${MYSQL_PASSWORD}" \
+	--skip-column-names --batch \
+	-e "SELECT version_num FROM alembic_version LIMIT 1;" \
+	"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
+[ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty"
+
+log "writing manifest"
+python3 /opt/backup/manifest.py \
+	--out "${SETDIR}/manifest.json" \
+	--takenat "${STAMP}" \
+	--site "${SITE}" \
+	--imagetag "${SHOPDB_IMAGE_TAG:-source}" \
+	--datarev "${DATAREV}" \
+	--migrations /app/migrations \
+	--plugins /app/plugins
+
+# Consistency guard: if the DATA revision does not match the CODE head, this
+# backup captures a database the running image has outrun. It is still stored
+# (losing it would be worse) but flagged loudly so a restore of it is a
+# deliberate choice, not a surprise.
+CODEREV="$(python3 -c \
+	'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \
+	"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
+if [ -n "${DATAREV}" ] && [ "${DATAREV}" != "${CODEREV}" ]; then
+	log "WARN schema drift: database is at '${DATAREV}' but code head is '${CODEREV}'"
+	log "WARN this DB has not run 'flask db upgrade' to the running image; restore will upgrade it"
+	history_append backup warning "${SETNAME}" "Schema drift: data ${DATAREV} vs code ${CODEREV}"
+fi
+
+# 3. restic snapshot: db dump + instance tree + manifest, one point in time.
+#    Skips gracefully when restic/repo are not configured (local-only sites).
+IS_CHECK_PASSED=false
+status_write backup snapshot "${SETNAME}" "Snapshotting to backup store"
+if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
+	if ! restic snapshots >/dev/null 2>&1; then
+		log "initializing restic repository at ${RESTIC_REPOSITORY}"
+		restic init
+	fi
+	log "restic backup (db dump + instance tree + manifest)"
+	restic backup \
+		--tag "${SITE}" \
+		--tag "${SETNAME}" \
+		"${SETDIR}/db.sql.gz" \
+		"${SETDIR}/manifest.json" \
+		/app/instance
+
+	# 4. Retention: any day in the last 5, plus weekly/monthly tail.
+	log "pruning per retention policy"
+	restic forget \
+		--tag "${SITE}" \
+		--keep-daily 5 \
+		--keep-weekly 4 \
+		--keep-monthly 6 \
+		--prune
+
+	# 5. Integrity: learn about corruption now, not at restore time.
+	log "verifying repository integrity"
+	if restic check; then
+		IS_CHECK_PASSED=true
+		log "restic check passed"
+	else
+		log "restic check FAILED - repository may be corrupt"
+	fi
+else
+	log "restic not configured - local db.sql.gz + manifest only"
+fi
+
+# Record the verification result into the manifest so the UI can badge the set.
+python3 /opt/backup/manifest.py \
+	--amend "${SETDIR}/manifest.json" \
+	--ischeckpassed "${IS_CHECK_PASSED}"
+
+# Clear the request sentinel the UI drops when an operator clicks "Back up now".
+rm -f "${BACKUPS_ROOT}/.backup-requested" 2>/dev/null || true
+
+BACKUP_DONE=true
+log "backup set ${SETNAME} complete"

scripts/backup/manifest.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Backup manifest writer.
+
+The manifest is the stamp that turns "we have dumps" into "we can actually
+restore." It records, at capture time:
+  - takenat   : ISO-8601 UTC timestamp of the run
+  - site      : the site tag
+  - imagetag  : the app image tag that produced the data
+  - schemarev : the core Alembic head revision(s) the running CODE is at, plus
+                each plugin chain's heads - the versions a restore target must
+                be at least, or restored rows reference columns the live schema
+                has not applied yet.
+  - datarev   : the Alembic revision actually applied to the DATABASE that was
+                dumped (its alembic_version), which is the number a restore must
+                gate on. schemarev stamps the code on disk; datarev stamps the
+                data in the dump. They diverge when an image is ahead of a DB
+                that has not run 'flask db upgrade' yet, and that divergence is
+                exactly what a code-to-code check cannot see.
+
+Two modes:
+  --out ...      write a fresh manifest for a new set
+  --amend ...    patch an existing manifest (used to record ischeckpassed after
+                 restic check runs, or to backfill datarev)
+
+ASCII only, no emojis - matches the repo style gate.
+"""
+
+import argparse
+import json
+import os
+import sys
+
+
+def _heads_for(chain_dir):
+    """Return the Alembic head revision id(s) under a versions directory.
+
+    We read the revision graph the cheap way: a head is a revision no other
+    revision names as its down_revision. Done by text scan so this stays
+    dependency-light and works for both the core chain and each plugin chain.
+    """
+    versions = os.path.join(chain_dir, 'versions')
+    if not os.path.isdir(versions):
+        return []
+    revisions = {}
+    down_refs = set()
+    for name in os.listdir(versions):
+        if not name.endswith('.py'):
+            continue
+        path = os.path.join(versions, name)
+        revid = None
+        downid = None
+        try:
+            with open(path, 'r', encoding='utf-8') as handle:
+                for line in handle:
+                    stripped = line.strip()
+                    if stripped.startswith('revision =') and revid is None:
+                        revid = stripped.split('=', 1)[1].strip().strip('"\'')
+                    elif stripped.startswith('down_revision ='):
+                        value = stripped.split('=', 1)[1].strip()
+                        downid = None if value in ('None', 'null') else value.strip('"\'')
+        except OSError:
+            continue
+        if revid:
+            revisions[revid] = downid
+            if downid:
+                down_refs.add(downid)
+    return sorted(rev for rev in revisions if rev not in down_refs)
+
+
+def _collect_schema(migrations_dir, plugins_dir):
+    """Core heads plus each plugin's chain heads, keyed for compatibility checks."""
+    schema = {'core': _heads_for(migrations_dir), 'plugins': {}}
+    if plugins_dir and os.path.isdir(plugins_dir):
+        for name in sorted(os.listdir(plugins_dir)):
+            chain = os.path.join(plugins_dir, name, 'migrations')
+            heads = _heads_for(chain)
+            if heads:
+                schema['plugins'][name] = heads
+    return schema
+
+
+def write_manifest(args):
+    manifest = {
+        'takenat': args.takenat,
+        'site': args.site,
+        'imagetag': args.imagetag,
+        'schemarev': _collect_schema(args.migrations, args.plugins),
+        'datarev': args.datarev or '',
+        'ischeckpassed': False,
+    }
+    with open(args.out, 'w', encoding='utf-8') as handle:
+        json.dump(manifest, handle, indent=2, sort_keys=True)
+    return 0
+
+
+def amend_manifest(args):
+    try:
+        with open(args.amend, 'r', encoding='utf-8') as handle:
+            manifest = json.load(handle)
+    except (OSError, ValueError):
+        manifest = {}
+    if args.ischeckpassed is not None:
+        manifest['ischeckpassed'] = args.ischeckpassed.lower() == 'true'
+    if args.datarev is not None:
+        manifest['datarev'] = args.datarev
+    with open(args.amend, 'w', encoding='utf-8') as handle:
+        json.dump(manifest, handle, indent=2, sort_keys=True)
+    return 0
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description='ShopDB backup manifest writer')
+    parser.add_argument('--out')
+    parser.add_argument('--amend')
+    parser.add_argument('--takenat', default='')
+    parser.add_argument('--site', default='shopdb')
+    parser.add_argument('--imagetag', default='source')
+    parser.add_argument('--migrations', default='/app/migrations')
+    parser.add_argument('--plugins', default='/app/plugins')
+    parser.add_argument('--datarev')
+    parser.add_argument('--ischeckpassed')
+    args = parser.parse_args(argv)
+
+    if args.amend:
+        return amend_manifest(args)
+    if args.out:
+        return write_manifest(args)
+    parser.error('one of --out or --amend is required')
+
+
+if __name__ == '__main__':
+    sys.exit(main())

scripts/backup/restore.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# ShopDB restore.
+#
+# The DANGEROUS half of the pair, so it is deliberately guarded. A restore is
+# only safe onto code whose schema is at least the set's - otherwise restored
+# rows reference columns the live schema has not applied yet and every query on
+# those tables 500s. This script refuses an incompatible restore rather than
+# leaving the site half-broken.
+#
+# Usage (from the backup container):
+#   docker compose -f docker-compose.yml -f docker-compose.backup.yml \
+#     run --rm --entrypoint /opt/backup/restore.sh backup <snapshot-set-name>
+#
+# It does NOT roll back code - that is git's job (git checkout <tag> && build).
+# Restore code FIRST, bring the schema up (flask db upgrade + plugin
+# upgrade-all), THEN run this to load the data.
+#
+# ASCII only, no emojis - matches the repo style gate.
+set -euo pipefail
+
+BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
+SETNAME="${1:-}"
+
+# shellcheck source=scripts/backup/status.sh
+. /opt/backup/status.sh
+
+log() { echo "[restore] $*"; }
+
+# On any exit, stamp the terminal phase and append a history record so the UI
+# reflects success or failure even when a step aborts early (set -e). A restore
+# is only "done" once the DATA has been upgraded to the running CODE, and that
+# upgrade runs in the Flask-capable api container, not here (this sidecar has no
+# app). So the happy path ends by handing off to the api: this script stamps an
+# 'upgrading' phase and drops an .upgrade-requested sentinel, and the api runs
+# 'flask db upgrade' + 'flask plugin upgrade-all' and writes the final 'done'.
+RESTORE_LOADED=false
+_on_exit() {
+	code=$?
+	if [ "${RESTORE_LOADED}" = true ]; then
+		# Data is loaded; the api takes it from 'upgrading' to 'done'. Do not
+		# stamp success here or the UI would call the restore complete before
+		# the schema is actually brought up to code.
+		status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
+	else
+		status_write restore failed "${SETNAME}" "Restore failed (exit ${code})"
+		history_append restore failed "${SETNAME}" "Restore failed (exit ${code})"
+	fi
+}
+trap _on_exit EXIT
+
+die() { echo "[restore] ERROR: $*" >&2; exit 1; }
+
+[ -n "${SETNAME}" ] || die "usage: restore.sh <snapshot-set-name>"
+
+status_write restore checking "${SETNAME}" "Checking schema compatibility"
+
+SETDIR="${BACKUPS_ROOT}/${SETNAME}"
+DBDUMP="${SETDIR}/db.sql.gz"
+
+# Prefer the local fast path; fall back to pulling the set out of restic.
+if [ ! -f "${DBDUMP}" ]; then
+	log "local dump not found; restoring set from restic"
+	command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
+	restic restore latest --target / --include "${SETDIR}" \
+		|| die "restic restore of ${SETNAME} failed"
+fi
+[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
+
+# Compatibility gate: the restore must gate on the revision the DUMP'S DATA is
+# at (datarev), not on a code-to-code comparison. Loading data that sits at an
+# OLDER revision than the running code is fine - we upgrade it afterwards - but
+# loading data NEWER than the code (a revision the code has never heard of)
+# would leave rows referencing columns the code cannot migrate to, so that is
+# refused.
+log "checking schema compatibility"
+
+# The set's data revision: prefer the manifest field; for older manifests that
+# predate datarev, parse the real alembic_version straight out of the dump so a
+# legacy set is still gated on its true data, not on a code-derived stamp.
+SET_DATAREV="$(python3 -c \
+	'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
+	"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
+if [ -z "${SET_DATAREV}" ]; then
+	log "manifest has no datarev (legacy set) - reading it from the dump"
+	SET_DATAREV="$(gunzip -c "${DBDUMP}" \
+		| grep -A2 -i 'INSERT INTO `alembic_version`' \
+		| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
+fi
+
+if [ -z "${SET_DATAREV}" ]; then
+	die "cannot determine the set's data revision - refusing an unverified restore"
+fi
+
+# Does the running code know this revision at all? If a migration file with that
+# revision id exists in the core chain, the code is at-or-past it (the head is
+# the tip of the same chain), so a post-load upgrade can carry the data forward.
+# If it is absent, the dump is NEWER than the code - refuse.
+if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
+	log "set datarev   : ${SET_DATAREV}"
+	die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
+fi
+log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
+
+# Files first, then DB, so the two land together.
+if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
+	status_write restore files "${SETNAME}" "Restoring instance files"
+	log "restoring instance files from restic"
+	restic restore latest --target / --include /app/instance \
+		|| die "restic restore of instance files failed"
+fi
+
+status_write restore database "${SETNAME}" "Loading database dump"
+log "loading database dump"
+gunzip -c "${DBDUMP}" | mysql \
+	--host="${MYSQL_HOST}" \
+	--user="${MYSQL_USER}" \
+	--password="${MYSQL_PASSWORD}" \
+	|| die "database load failed"
+
+# Hand off to the api container for the schema upgrade. The dump may sit at an
+# older revision than this image (allowed and expected); without the upgrade the
+# data stays behind the code and every query on a table the newer migrations
+# changed 500s with 1054 Unknown column. This sidecar has no Flask, so it cannot
+# run 'flask db upgrade' itself - it records the intent and lets the api (which
+# owns the app context and already runs these exact idempotent commands on every
+# deploy) finalize. The api clears .upgrade-requested and stamps the final
+# 'done' once the upgrade succeeds.
+status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
+printf '%s' "${SETNAME}" > "${BACKUPS_ROOT}/.upgrade-requested"
+log "database loaded at revision ${SET_DATAREV}; handed off to api for schema upgrade"
+
+rm -f "${BACKUPS_ROOT}/.restore-requested" 2>/dev/null || true
+RESTORE_LOADED=true

scripts/backup/simulate-restore.sh

@@ -0,0 +1,235 @@
+#!/usr/bin/env bash
+# ShopDB restore simulation harness.  ***FOR DEVELOPER USE ONLY***
+#
+# This is a developer diagnostic, NOT an operator tool and NOT part of the
+# shipped backup feature. It mutates a live database on purpose. Do not run it
+# against production or any environment you are not prepared to disturb.
+#
+# WHY: the restore path reloads the WHOLE database in one pass (mysqldump
+# --databases emits DROP TABLE + CREATE TABLE + full INSERTs, so restore is a
+# full replace, not a row-level merge). The strongest real-world check is a
+# full-fidelity round trip: fingerprint every table, take a backup, mutate
+# across the table archetypes (core standalone, asset + plugin subtype, and
+# consumable ledgers), restore, then prove every table's fingerprint returns to
+# the backup-time state.
+#
+# It drives the exact sentinel path the UI uses (.backup-requested /
+# .restore-requested on the shared volume), not a private code path, so a pass
+# here is a pass for the operator-facing feature.
+#
+# KNOWN, EXPECTED, NOT-A-FAILURE behaviors (see LiveDatabaseTest notes below):
+#   - apitokens does not round-trip bit-for-bit: the app rewrites token
+#     last-used timestamps continuously, so restore reproduces backup-time
+#     state, which differs from any later fingerprint. This is correct.
+#   - endpoint-agent tables (computers heartbeat, endpointstates/events/
+#     applications) churn from the EXTERNAL fleet agents, which cannot be
+#     quiesced. Post-restore writes from them are expected drift, not a
+#     restore defect.
+#   - Asset ROWS cannot be deleted in a live/CMMC environment (RESTRICT and
+#     compliance controls). The cascade probe is therefore informational only;
+#     restore fidelity does not depend on it. A missing/corrupted asset is
+#     still fully restored because the dump drops and recreates the table.
+#
+# Usage (from a host with the stack up, git bash):
+#   bash scripts/backup/simulate-restore.sh
+#
+# ASCII only, no emojis - matches the repo style gate.
+set -uo pipefail
+
+# Silence the docker CLI "What's next" hints - they bleed into command
+# substitution and corrupt captured SQL output.
+export DOCKER_CLI_HINTS=false
+
+COMPOSE="docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup"
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+# Tables that legitimately differ after restore and MUST NOT be counted as
+# failures:
+#   - App-churn: token last-used, sessions, audit trail.
+#   - Endpoint-agent churn: the external fleet agents keep writing to these
+#     after restore (heartbeats, state, events, inventory) and cannot be
+#     quiesced from here.
+# Restore reproduces their state at dump time; later fingerprints differ by
+# design, so they are reported informationally.
+VOLATILE="apitokens sessions auditlogs computers endpointstates endpointevents endpointapplications"
+WORKDIR="$(mktemp -d)"
+BASELINE="${WORKDIR}/baseline.txt"
+MUTATED="${WORKDIR}/mutated.txt"
+RESTORED="${WORKDIR}/restored.txt"
+MARKER="simrestore-$(date -u +%Y%m%dT%H%M%SZ)"
+
+log() { echo "[sim] $*"; }
+die() { echo "[sim] ERROR: $*" >&2; exit 1; }
+
+# Run a SQL statement, returning tab-separated rows with no column headers.
+sql() {
+	printf '%s\n' "$1" | ${COMPOSE} exec -T db sh -c \
+		'mysql -ushopdb -p"$MYSQL_PASSWORD" shopdb_flask -N 2>/dev/null'
+}
+
+# Fingerprint every base table with CHECKSUM TABLE into the given file.
+fingerprint() {
+	out="$1"
+	tables="$(sql "SELECT table_name FROM information_schema.tables \
+		WHERE table_schema='shopdb_flask' AND table_type='BASE TABLE' \
+		ORDER BY table_name;" | tr -d '\r')"
+	: > "${out}"
+	for table in ${tables}; do
+		# CHECKSUM TABLE returns "dbname.table<TAB>checksum"; keep table+checksum.
+		line="$(sql "CHECKSUM TABLE \`${table}\`;" | tr -d '\r')"
+		checksum="$(printf '%s' "${line}" | awk '{print $2}')"
+		printf '%s\t%s\n' "${table}" "${checksum}" >> "${out}"
+	done
+}
+
+wait_for_phase() {
+	kind="$1"; want="$2"; timeout="${3:-120}"
+	waited=0
+	while [ "${waited}" -lt "${timeout}" ]; do
+		phase="$(${COMPOSE} exec -T backup-watcher sh -c \
+			"cat /backups/.${kind}-status.json 2>/dev/null" \
+			| sed -n 's/.*\"phase\":\"\([^\"]*\)\".*/\1/p')"
+		if [ "${phase}" = "${want}" ]; then return 0; fi
+		if [ "${phase}" = "failed" ]; then
+			die "${kind} reported failed status"
+		fi
+		sleep 3
+		waited=$((waited + 3))
+	done
+	die "${kind} did not reach '${want}' within ${timeout}s (last: ${phase:-none})"
+}
+
+# --- 0. Preconditions ------------------------------------------------------
+${COMPOSE} exec -T db sh -c 'true' >/dev/null 2>&1 \
+	|| die "db container not reachable - is the stack up?"
+
+# --- 0b. Datarev-gate preflight (external tester) --------------------------
+# This live round trip runs one codebase against itself, so it can never
+# reproduce the "data revision behind the running code" case the restore gate
+# now handles. Run the standalone datarev tester first so a pass here also
+# covers that offline contract (manifest datarev, known-vs-unknown revision
+# refusal, legacy fallback, and the api upgrade handoff).
+log "running datarev gate preflight (test-restore-datarev.sh)"
+bash "${SCRIPT_DIR}/test-restore-datarev.sh" \
+	|| die "datarev gate preflight failed - see output above"
+log "datarev gate preflight passed"
+
+# --- 1. Backup the current state, then fingerprint the RESTORE TARGET ------
+# The restore point is the database as it existed WHEN THE DUMP RAN, not some
+# earlier moment. Live-churn tables (apitokens last-used, sessions, auditlogs)
+# move under the running app, so the baseline must be captured immediately
+# after the backup completes - that is the exact state restore must reproduce.
+log "requesting a backup (sentinel .backup-requested)"
+${COMPOSE} exec -T backup-watcher sh -c 'echo sim > /backups/.backup-requested'
+wait_for_phase backup done 300
+SETNAME="$(${COMPOSE} exec -T backup-watcher sh -c \
+	'cat /backups/.backup-status.json 2>/dev/null' \
+	| sed -n 's/.*\"setname\":\"\([^\"]*\)\".*/\1/p')"
+[ -n "${SETNAME}" ] || die "could not read the completed backup set name"
+log "backup complete: ${SETNAME}"
+
+log "baseline fingerprint of all tables (restore target)"
+fingerprint "${BASELINE}"
+baseline_count="$(wc -l < "${BASELINE}")"
+log "fingerprinted ${baseline_count} tables"
+
+# Quiesce the app for the mutate/restore/verify window. A real DR runbook stops
+# serving traffic before a restore; here it also stops the endpoint-agent write
+# path (computers heartbeat, endpointstates/events/applications) that would
+# otherwise churn the DB concurrently and defeat a bit-for-bit comparison.
+log "stopping api to quiesce writers"
+${COMPOSE} stop api >/dev/null 2>&1
+resume_api() { log "resuming api"; ${COMPOSE} start api >/dev/null 2>&1; }
+trap resume_api EXIT
+
+# --- 2. Mutate across the table archetypes ---------------------------------
+# Core standalone: rename a vendor and flip a setting.
+# Ledger/consumable: bump a modelsupplies quantityonhand.
+# NOTE: asset ROW deletion is intentionally NOT performed. In a live/CMMC
+# environment assets cannot be removed (RESTRICT + compliance controls), which
+# is the app behaving as designed. Restore fidelity does not depend on it: the
+# dump drops and recreates every table, so a missing or corrupted asset is
+# fully reconstructed regardless. The cascade edge is reported informationally.
+log "mutating rows (marker ${MARKER})"
+
+sql "UPDATE vendors SET vendorname=CONCAT('${MARKER}-', vendorname) \
+	ORDER BY vendorid LIMIT 1;"
+sql "UPDATE settings SET settingvalue='${MARKER}' \
+	WHERE settingkey=(SELECT settingkey FROM (SELECT settingkey FROM settings \
+	ORDER BY settingkey LIMIT 1) AS s);"
+
+# Cascade edge (informational only - no destructive delete): report the FK
+# delete rule that would remove computer subtype rows if an asset were deleted.
+CASCADE_RULE="$(sql "SELECT delete_rule FROM information_schema.referential_constraints \
+	WHERE constraint_schema='shopdb_flask' AND referenced_table_name='assets' \
+	AND table_name='computers' LIMIT 1;" | tr -d '\r')"
+log "cascade edge computers->assets delete_rule=${CASCADE_RULE:-unknown} (live asset delete not permitted; not exercised)"
+
+# Ledger: move a cached quantityonhand so the restore must revert it.
+sql "UPDATE modelsupplies SET quantityonhand=quantityonhand+9999 \
+	ORDER BY id LIMIT 1;" 2>/dev/null \
+	|| sql "UPDATE modelsupplies SET quantityonhand=quantityonhand+9999 \
+	ORDER BY modelsupplyid LIMIT 1;"
+
+log "post-mutation fingerprint"
+fingerprint "${MUTATED}"
+changed="$(diff <(sort "${BASELINE}") <(sort "${MUTATED}") | grep -c '^>')"
+log "mutation changed ${changed} table fingerprint(s)"
+
+# --- 3. Restore the backup we took -----------------------------------------
+log "requesting restore of ${SETNAME}"
+${COMPOSE} exec -T backup-watcher sh -c "printf '%s' '${SETNAME}' > /backups/.restore-requested"
+wait_for_phase restore done 600
+log "restore complete"
+
+# --- 4. Verify every table returned to baseline ----------------------------
+log "post-restore fingerprint"
+fingerprint "${RESTORED}"
+
+fails=0
+volatile_diffs=0
+while IFS="$(printf '\t')" read -r table base; do
+	restored="$(grep -P "^${table}\t" "${RESTORED}" | awk '{print $2}')"
+	if [ "${base}" != "${restored}" ]; then
+		case " ${VOLATILE} " in
+			*" ${table} "*)
+				echo "[sim] (volatile) ${table} differs as expected: baseline=${base} restored=${restored}"
+				volatile_diffs=$((volatile_diffs + 1))
+				;;
+			*)
+				echo "[sim] MISMATCH ${table}: baseline=${base} restored=${restored}"
+				fails=$((fails + 1))
+				;;
+		esac
+	fi
+done < "${BASELINE}"
+
+echo "=================================================="
+RESULTS_FILE="${SCRIPT_DIR}/simulate-restore-results.txt"
+{
+	echo "ShopDB restore simulation results (FOR DEVELOPER USE ONLY)"
+	echo "run (UTC)      : $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+	echo "marker         : ${MARKER}"
+	echo "backup set     : ${SETNAME}"
+	echo "tables checked : ${baseline_count}"
+	echo "cascade edge   : computers->assets delete_rule=${CASCADE_RULE:-unknown} (not exercised - live asset delete not permitted)"
+	echo ""
+	echo "Expected, not-a-failure drift (excluded from strict equality):"
+	echo "  - apitokens: app rewrites token last-used; restore is correct at backup time."
+	echo "  - endpoint-agent tables (computers, endpointstates/events/applications):"
+	echo "    external fleet agents keep writing post-restore and cannot be quiesced."
+	echo "  - asset row deletion is not permitted live (RESTRICT + CMMC); the table is"
+	echo "    still fully restored because the dump drops and recreates it."
+	echo ""
+	if [ "${fails}" -eq 0 ]; then
+		echo "RESULT: PASS - all ${baseline_count} tables restored to backup-time state"
+		echo "        (${volatile_diffs} volatile table(s) excluded from strict equality)"
+	else
+		echo "RESULT: FAIL - ${fails} table(s) did not match after restore"
+	fi
+} | tee "${RESULTS_FILE}"
+echo "=================================================="
+echo "[sim] results written to ${RESULTS_FILE}"
+echo "[sim] work dir: ${WORKDIR}"
+echo "=================================================="
+[ "${fails}" -eq 0 ]

scripts/backup/status.sh

@@ -0,0 +1,100 @@
+# ShopDB backup/restore status + history helper.
+#
+# WHY this exists: the restore and backup runs happen in the watcher container,
+# but the operator watches the UI, which only talks to the api. This helper is
+# the bridge: it writes two kinds of state onto the shared /backups volume that
+# the api can read.
+#
+#   1. A LIVE status file (.restore-status.json / .backup-status.json) rewritten
+#      at each phase so the UI can show an indeterminate spinner with a phase
+#      label (queued, restoring files, loading database, done, failed).
+#   2. An append-only HISTORY file (history/restores.json / history/backups.json)
+#      that keeps the record of past runs so the UI can list them in a card.
+#
+# Both writes are atomic (write temp, rename) so the api never reads a half
+# written file. History is bounded to the most recent HISTORY_MAX entries.
+#
+# This file is SOURCED, not executed. It exposes:
+#   status_write   <kind> <phase> <setname> [message]
+#   history_append <kind> <outcome> <setname> [message]
+#
+# kind is "restore" or "backup". ASCII only, no emojis - matches the style gate.
+
+STATUS_BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
+HISTORY_MAX="${BACKUP_HISTORY_MAX:-50}"
+
+_status_now() { date -u +%Y-%m-%dT%H%M%SZ; }
+
+_status_json_escape() {
+	# Escape backslashes and double quotes for embedding in a JSON string.
+	printf '%s' "${1:-}" | sed 's/\\/\\\\/g; s/"/\\"/g'
+}
+
+# status_write <kind> <phase> <setname> [message]
+# Rewrites the live status file for this run kind.
+status_write() {
+	kind="$1"
+	phase="$2"
+	setname="${3:-}"
+	message="${4:-}"
+	statusfile="${STATUS_BACKUPS_ROOT}/.${kind}-status.json"
+	tmpfile="${statusfile}.tmp.$$"
+	{
+		printf '{'
+		printf '"kind":"%s",' "$(_status_json_escape "${kind}")"
+		printf '"phase":"%s",' "$(_status_json_escape "${phase}")"
+		printf '"setname":"%s",' "$(_status_json_escape "${setname}")"
+		printf '"message":"%s",' "$(_status_json_escape "${message}")"
+		printf '"updatedat":"%s"' "$(_status_now)"
+		printf '}\n'
+	} > "${tmpfile}" 2>/dev/null || return 0
+	mv -f "${tmpfile}" "${statusfile}" 2>/dev/null || rm -f "${tmpfile}" 2>/dev/null
+	return 0
+}
+
+# history_append <kind> <outcome> <setname> [message]
+# Appends a completed run record and trims to HISTORY_MAX newest entries.
+history_append() {
+	kind="$1"
+	outcome="$2"
+	setname="${3:-}"
+	message="${4:-}"
+	historydir="${STATUS_BACKUPS_ROOT}/history"
+	historyfile="${historydir}/${kind}s.json"
+	tmpfile="${historyfile}.tmp.$$"
+	mkdir -p "${historydir}" 2>/dev/null || return 0
+
+	record="$(printf '{"setname":"%s","outcome":"%s","message":"%s","finishedat":"%s"}' \
+		"$(_status_json_escape "${setname}")" \
+		"$(_status_json_escape "${outcome}")" \
+		"$(_status_json_escape "${message}")" \
+		"$(_status_now)")"
+
+	if command -v python3 >/dev/null 2>&1; then
+		HISTORY_FILE="${historyfile}" HISTORY_RECORD="${record}" \
+			HISTORY_MAX="${HISTORY_MAX}" python3 - "${tmpfile}" <<'PY' 2>/dev/null || return 0
+import json, os, sys
+
+path = os.environ["HISTORY_FILE"]
+record = json.loads(os.environ["HISTORY_RECORD"])
+limit = int(os.environ.get("HISTORY_MAX", "50"))
+tmp = sys.argv[1]
+
+try:
+	with open(path, "r", encoding="utf-8") as handle:
+		entries = json.load(handle)
+	if not isinstance(entries, list):
+		entries = []
+except (OSError, ValueError):
+	entries = []
+
+entries.append(record)
+entries = entries[-limit:]
+
+with open(tmp, "w", encoding="utf-8") as handle:
+	json.dump(entries, handle)
+os.replace(tmp, path)
+PY
+	fi
+	return 0
+}

scripts/backup/test-restore-datarev.sh

@@ -0,0 +1,172 @@
+#!/usr/bin/env bash
+# ShopDB restore datarev-gate tester.  ***FOR DEVELOPER USE ONLY***
+#
+# WHY THIS EXISTS SEPARATELY FROM simulate-restore.sh:
+# simulate-restore.sh is a full-fidelity ROUND TRIP against a live stack: it
+# backs up, mutates, restores, and proves every table returns to backup-time
+# state. By construction it runs ONE codebase against ITSELF, so the database it
+# dumps is always at the SAME revision as the code that restores it. It can
+# therefore never exercise the case the FBJXBDB4 outage was about: a dump whose
+# DATA revision is BEHIND (or ahead of) the running CODE.
+#
+# This tester fills that gap offline. It builds throwaway migration trees and
+# fake gzipped dumps and drives the exact decision logic restore.sh uses:
+#   1. manifest.py records datarev (the DB's real alembic_version), not just the
+#      code head (schemarev).
+#   2. A dump whose datarev the code KNOWS (an ancestor in the chain) is allowed
+#      - the post-load upgrade rolls it forward.
+#   3. A dump whose datarev the code does NOT know (newer than this build) is
+#      REFUSED - restoring it would 500 on 1054 Unknown column.
+#   4. A legacy manifest with no datarev falls back to parsing alembic_version
+#      straight out of the dump.
+#   5. restore.sh hands the loaded restore off to the api via the
+#      .upgrade-requested sentinel instead of marking it done in the sidecar.
+#
+# No Docker, no MySQL, no network: it is a pure logic gate test, safe to run in
+# CI and from inside simulate-restore.sh as a preflight.
+#
+# ASCII only, no emojis - matches the repo style gate.
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+MANIFEST_PY="${SCRIPT_DIR}/manifest.py"
+RESTORE_SH="${SCRIPT_DIR}/restore.sh"
+WORKDIR="$(mktemp -d)"
+trap 'rm -rf "${WORKDIR}"' EXIT
+
+# The backup image ships python3; a dev host may only have `python` (and on
+# Windows `python3` can be a non-functional Store alias that resolves on PATH
+# but errors when run). Pick the first interpreter that actually EXECUTES, so
+# this tester runs anywhere without changing the shipped scripts.
+PY=""
+for candidate in python3 python py; do
+	if "${candidate}" -c 'pass' >/dev/null 2>&1; then
+		PY="${candidate}"
+		break
+	fi
+done
+[ -n "${PY}" ] || { echo "[test] ERROR: no working python interpreter found"; exit 1; }
+
+passes=0
+fails=0
+pass() { echo "[test] PASS: $*"; passes=$((passes + 1)); }
+fail() { echo "[test] FAIL: $*"; fails=$((fails + 1)); }
+
+# Build a throwaway core migration chain: baseline -> mid -> head. The gate in
+# restore.sh only needs each version file to declare `revision = '<id>'`, so
+# that is all we write.
+build_chain() {
+	root="$1"
+	versions="${root}/versions"
+	mkdir -p "${versions}"
+	printf "revision = '%s'\ndown_revision = None\n" \
+		'0001_baseline' > "${versions}/0001_baseline.py"
+	printf "revision = '%s'\ndown_revision = '%s'\n" \
+		'0002_mid' '0001_baseline' > "${versions}/0002_mid.py"
+	printf "revision = '%s'\ndown_revision = '%s'\n" \
+		'0003_head' '0002_mid' > "${versions}/0003_head.py"
+}
+
+# Fake a gzipped mysqldump that stamps a given alembic_version, matching the
+# INSERT shape restore.sh greps for when a legacy manifest lacks datarev.
+build_dump() {
+	out="$1"; datarev="$2"
+	{
+		echo 'DROP TABLE IF EXISTS `alembic_version`;'
+		echo 'CREATE TABLE `alembic_version` (`version_num` varchar(32));'
+		echo 'INSERT INTO `alembic_version` VALUES'
+		echo "('${datarev}');"
+	} | gzip -c > "${out}"
+}
+
+# Mirror of the restore.sh predicate: does the code chain know this revision?
+# Kept identical to restore.sh so this test tracks the shipped decision.
+code_knows_revision() {
+	versions_dir="$1"; datarev="$2"
+	grep -rqE "^revision = ['\"]${datarev}['\"]" "${versions_dir}" 2>/dev/null
+}
+
+# Mirror of the restore.sh datarev extraction: manifest field first, else parse
+# the dump.
+set_datarev() {
+	setdir="$1"; dbdump="$2"
+	value="$(${PY} -c \
+		'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
+		"${setdir}/manifest.json" 2>/dev/null || echo "")"
+	if [ -z "${value}" ]; then
+		value="$(gunzip -c "${dbdump}" \
+			| grep -A2 -i 'INSERT INTO `alembic_version`' \
+			| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
+	fi
+	printf '%s' "${value}"
+}
+
+CHAIN="${WORKDIR}/migrations"
+build_chain "${CHAIN}"
+VERSIONS="${CHAIN}/versions"
+
+# --- 1. manifest.py records datarev distinct from the code head -------------
+SET1="${WORKDIR}/set1"; mkdir -p "${SET1}"
+${PY} "${MANIFEST_PY}" --out "${SET1}/manifest.json" \
+	--migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \
+	--datarev '0002_mid' --takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
+recorded="$(${PY} -c \
+	'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
+	"${SET1}/manifest.json" 2>/dev/null)"
+head_stamp="$(${PY} -c \
+	'import json,sys;print(json.load(open(sys.argv[1])).get("schemarev",{}).get("core",[]))' \
+	"${SET1}/manifest.json" 2>/dev/null)"
+if [ "${recorded}" = '0002_mid' ]; then
+	pass "manifest.py records datarev (${recorded}) separate from code head ${head_stamp}"
+else
+	fail "manifest.py did not record datarev (got '${recorded}')"
+fi
+
+# --- 2. datarev the code KNOWS (behind head) is allowed --------------------
+DBDUMP1="${SET1}/db.sql.gz"; build_dump "${DBDUMP1}" '0002_mid'
+rev="$(set_datarev "${SET1}" "${DBDUMP1}")"
+if [ "${rev}" = '0002_mid' ] && code_knows_revision "${VERSIONS}" "${rev}"; then
+	pass "data behind code (${rev}) is known and allowed to restore"
+else
+	fail "data behind code (${rev}) was not accepted"
+fi
+
+# --- 3. datarev the code does NOT know (newer than build) is refused -------
+SET2="${WORKDIR}/set2"; mkdir -p "${SET2}"
+${PY} "${MANIFEST_PY}" --out "${SET2}/manifest.json" \
+	--migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \
+	--datarev '0004_future' --takenat '2024-02-01T00:00:00Z' >/dev/null 2>&1
+DBDUMP2="${SET2}/db.sql.gz"; build_dump "${DBDUMP2}" '0004_future'
+rev="$(set_datarev "${SET2}" "${DBDUMP2}")"
+if [ "${rev}" = '0004_future' ] && ! code_knows_revision "${VERSIONS}" "${rev}"; then
+	pass "data ahead of code (${rev}) is unknown and correctly refused"
+else
+	fail "data ahead of code (${rev}) was not refused"
+fi
+
+# --- 4. legacy manifest (no datarev) falls back to the dump ----------------
+SET3="${WORKDIR}/set3"; mkdir -p "${SET3}"
+# A pre-datarev manifest: schemarev present, datarev absent/empty.
+printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \
+	> "${SET3}/manifest.json"
+DBDUMP3="${SET3}/db.sql.gz"; build_dump "${DBDUMP3}" '0001_baseline'
+rev="$(set_datarev "${SET3}" "${DBDUMP3}")"
+if [ "${rev}" = '0001_baseline' ] && code_knows_revision "${VERSIONS}" "${rev}"; then
+	pass "legacy set with no datarev falls back to dump revision (${rev})"
+else
+	fail "legacy set fallback failed (got '${rev}')"
+fi
+
+# --- 5. restore.sh hands off to the api instead of self-completing ---------
+if grep -q '\.upgrade-requested' "${RESTORE_SH}" \
+	&& grep -q "status_write restore upgrading" "${RESTORE_SH}" \
+	&& ! grep -qE 'status_write restore done' "${RESTORE_SH}"; then
+	pass "restore.sh parks in 'upgrading' and hands off via .upgrade-requested"
+else
+	fail "restore.sh does not hand off the upgrade to the api"
+fi
+
+echo "=================================================="
+echo "[test] datarev gate: ${passes} passed, ${fails} failed"
+echo "=================================================="
+[ "${fails}" -eq 0 ]

scripts/backup/watch.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# ShopDB backup WATCHER - the resident half of the in-app backup utility.
+#
+# WHY this exists: the api container cannot run docker, so when an operator
+# clicks "Back up now" the api only drops a sentinel file (.backup-requested)
+# on the shared backups volume. This watcher is the thing that actuates that
+# request: it polls for the sentinel and runs the one-shot backup.sh, which
+# ships a restic snapshot to the backup-store container and clears the sentinel
+# on success. That turns the UI button into a real, self-serve backup without
+# giving the api container any docker or host privileges.
+#
+# ASCII only, no emojis - matches the repo style gate.
+set -uo pipefail
+
+BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
+SENTINEL="${BACKUPS_ROOT}/.backup-requested"
+RESTORE_SENTINEL="${BACKUPS_ROOT}/.restore-requested"
+INTERVAL="${BACKUP_POLL_SECONDS:-10}"
+
+log() { echo "[backup-watch] $*"; }
+
+# shellcheck source=scripts/backup/status.sh
+. /opt/backup/status.sh
+
+mkdir -p "${BACKUPS_ROOT}"
+# The api container runs unprivileged and only drops sentinel files here; it
+# cannot write into a root-owned 0755 directory. Make the backups root
+# writable so "Back up now" and "Restore" can queue their sentinels.
+chmod 0777 "${BACKUPS_ROOT}" 2>/dev/null || true
+log "watcher up; polling ${SENTINEL} and ${RESTORE_SENTINEL} every ${INTERVAL}s"
+
+while true; do
+	if [ -f "${SENTINEL}" ]; then
+		log "backup requested - running one-shot"
+		status_write backup queued "" "Backup queued"
+		if /opt/backup/backup.sh; then
+			log "backup completed"
+		else
+			# backup.sh only clears the sentinel on success, so a failed run
+			# would otherwise loop forever. Clear it here and wait for the next
+			# explicit request rather than hammering a broken backup.
+			log "backup FAILED - clearing request, awaiting next"
+			rm -f "${SENTINEL}" 2>/dev/null || true
+		fi
+	fi
+	if [ -f "${RESTORE_SENTINEL}" ]; then
+		# The api drops the target set name into the restore sentinel. Read it,
+		# hand it to restore.sh, and clear the request whether it succeeds or
+		# fails so a bad set cannot pin the watcher in a tight retry loop.
+		setname="$(head -n1 "${RESTORE_SENTINEL}" 2>/dev/null | tr -d '\r\n')"
+		log "restore requested - running restore for ${setname}"
+		status_write restore queued "${setname}" "Restore queued"
+		if /opt/backup/restore.sh "${setname}"; then
+			log "restore completed"
+		else
+			log "restore FAILED - clearing request, awaiting next"
+		fi
+		rm -f "${RESTORE_SENTINEL}" 2>/dev/null || true
+	fi
+	sleep "${INTERVAL}"
+done

scripts/check-naming-and-style.sh

@@ -35,6 +35,10 @@ EXCLUDES=(
     --exclude-dir=.git
     --exclude-dir=versions
     --exclude-dir=staticdocs
+    # Local, gitignored scratch notes (see .gitignore devnotes/). Not product
+    # source, never committed, and free to hold pasted logs full of em-dashes
+    # and arrows - grep does not consult .gitignore, so exclude it explicitly.
+    --exclude-dir=devnotes
 )
 
 # Root-level gitignored scratch: review writeups and chat handoffs, free to hold

scripts/uninstall-shopdbagent.ps1

@@ -1,10 +1,11 @@
 # uninstall-shopdbagent.ps1
 # ShopDB Endpoint Agent uninstaller. Run from an ELEVATED (admin) PowerShell on
-# the endpoint. Reverses everything install-shopdbagent.ps1 and the site agent
-# generator's install.ps1 create:
+# the endpoint. Reverses everything the site agent generator's install.ps1
+# creates:
 #   - stops and deletes the Windows service
 #   - kills any running agent/tray processes
 #   - removes the tray logon Run key (64-bit and 32-bit views)
+#   - unregisters the ShopDB user-context probe scheduled task
 #   - removes the credential hive HKLM\SOFTWARE\GE\ShopDB (both views)
 #   - removes the Defender path exclusion for the install directory
 #   - deletes the install directory and the ProgramData data/log tree
@@ -20,6 +21,10 @@ param(
 	[string] $ServiceName = "ShopDbEndpointAgent",
 	[string] $TrayExeName = "shopdb-endpoint-tray.exe",
 
+	# Scheduled task that runs the user-context probe in the operator's session.
+	# Must match the installer's -UserContextTaskName.
+	[string] $UserContextTaskName = "ShopDB user-context probe",
+
 	# Also remove any server TLS certificate the site-agent installer imported
 	# into the machine Trusted Root store. Off by default because a root cert is
 	# shared trust material: only pull it when you are sure no other service on
@@ -113,6 +118,34 @@ Start-Sleep -Seconds 1
 # 3. Remove the tray logon Run key entry (both views) and the credential hive.
 Write-Host "[3/6] Removing registry entries ..."
 Remove-HklmBothViews -SubKey "SOFTWARE\Microsoft\Windows\CurrentVersion\Run" -Name "ShopDbEndpointTray"
+
+# Unregister the probe task. Left behind it would keep firing at every logon
+# against a deleted exe, which shows up as a recurring task-failure history on a
+# machine nobody believes has the agent on it any more.
+try {
+	if (Get-ScheduledTask -TaskName $UserContextTaskName -ErrorAction SilentlyContinue) {
+		Unregister-ScheduledTask -TaskName $UserContextTaskName -Confirm:$false -ErrorAction Stop
+		Write-Host "  removed scheduled task '$UserContextTaskName'"
+	}
+	else {
+		Write-Host "  scheduled task '$UserContextTaskName' not present."
+	}
+}
+catch {
+	Write-Warning "  could not remove scheduled task '$UserContextTaskName': $($_.Exception.Message)"
+}
+
+# The install's one-shot tray launcher is transient and normally removes itself,
+# but an install that failed partway can leave it behind.
+try {
+	if (Get-ScheduledTask -TaskName "ShopDB tray first launch" -ErrorAction SilentlyContinue) {
+		Unregister-ScheduledTask -TaskName "ShopDB tray first launch" -Confirm:$false -ErrorAction Stop
+		Write-Host "  removed leftover scheduled task 'ShopDB tray first launch'"
+	}
+}
+catch {
+	Write-Warning "  could not remove scheduled task 'ShopDB tray first launch': $($_.Exception.Message)"
+}
 Remove-HklmBothViews -SubKey "SOFTWARE\GE\ShopDB"
 
 # Remove the All-Users Start Menu shortcut the site-agent installer created.