ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Endpoint telemetry, PC Detail rework and a checked backup/restore path

Browse Source
This commit is contained in:
Cameron Proudlock
2026-09-06 18:53:11 -04:00
parent 32b8745d10
commit 993353dfc0
97 changed files with 13340 additions and 756 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
scripts/backup/.gitattributes vendored Normal file
View File

@@ -0,0 +1,4 @@
# Keep shell scripts LF-only so they run inside Linux containers regardless of
# the host checkout. A CRLF shebang makes the kernel look for "bash\r" and the
# container dies with: env: 'bash\r': No such file or directory.
*.sh text eol=lf

53
scripts/backup/Dockerfile.backup Normal file
View File

@@ -0,0 +1,53 @@
# Backup sidecar image: mysql:8.4 (for a version-matched mysqldump) plus restic
# (for deduped, encrypted, destination-agnostic file snapshots) and a small
# Python for the manifest writer.
#
# WHY mysql:8.4 as the base: the dump tool must match the server major/minor so
# the SQL it emits restores cleanly onto the same engine. Reusing the exact
# image the db service runs removes version drift as a failure mode.
FROM mysql:8.4
# Track 1 local build unblock: trust the corporate proxy (Zscaler/GE) root CA so
# the build-time fetches below (Oracle yum, github.com, PyPI) succeed behind TLS
# interception. This is a local convenience only; the shipping build vendors
# these deps instead of fetching them (pinned restic + SHA256, vendored
# bzip2/python3/alembic or a base image that carries them) so the release image
# needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
# and must be staged at scripts/backup/zscaler-root-ca.pem before building.
#
# Copied via the directory rather than by name, because the cert is OPTIONAL: it
# is gitignored, so a plain `COPY scripts/backup/zscaler-root-ca.pem` fails the
# build outright and nobody outside a TLS-intercepting network - another site,
# or our own CI - could build this image at all. A trailing glob does not fix
# that either: BuildKit fails a COPY whose pattern matches nothing, which is
# precisely the case being handled. Copying a directory that always exists and
# testing for the file inside it is the form that works both ways.
COPY scripts/backup/ /tmp/buildctx/
RUN if [ -f /tmp/buildctx/zscaler-root-ca.pem ]; then \
echo "staging corporate root CA"; \
cp /tmp/buildctx/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/; \
update-ca-trust; \
else \
echo "no corporate root CA staged - using the public trust store"; \
fi \
&& rm -rf /tmp/buildctx
# restic from the distro is often stale; pull the pinned release binary so the
# snapshot format and retention flags are predictable across sites.
ARG RESTIC_VERSION=0.17.3
RUN microdnf install -y bzip2 python3 python3-pip ca-certificates \
&& curl -fsSL -o /tmp/restic.bz2 \
"https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2" \
&& bunzip2 /tmp/restic.bz2 \
&& install -m 0755 /tmp/restic /usr/local/bin/restic \
&& rm -f /tmp/restic \
&& microdnf clean all
# alembic so the manifest writer can read the current schema head. pip does not
# read the system trust store, so point it at the extracted system CA bundle
# (now including the proxy root) to get through TLS interception.
RUN python3 -m pip install --no-cache-dir \
--cert /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
alembic
ENTRYPOINT ["/opt/backup/backup.sh"]

44
scripts/backup/SIMULATE-RESTORE.README.md Normal file
View File

@@ -0,0 +1,44 @@
# Restore Simulation (FOR DEVELOPER USE ONLY)
`simulate-restore.sh` is a developer diagnostic that proves the backup/restore
pair round-trips the live database. It is NOT an operator tool and NOT part of
the shipped feature. It mutates a live database on purpose - never run it
against production or anything you cannot disturb.
## Testing cadence
- Before merging any change to `scripts/backup/*.sh`, `docker-compose.backup.yml`,
or `shopdb/core/api/shopdbbackups.py`.
- Before cutting a release that touches the backup/restore surface.
- After a schema migration that adds or drops tables (new table archetype).
- Ad hoc when investigating a restore-fidelity concern.
## What it does
1. Requests a backup through the real `.backup-requested` sentinel.
2. Fingerprints every base table with `CHECKSUM TABLE` (the restore target).
3. Quiesces the `api` service, mutates the table archetypes:
- core standalone (vendor rename, setting flip)
- consumable ledger (`modelsupplies.quantityonhand` bump)
4. Requests a restore through the real `.restore-requested` sentinel.
5. Re-fingerprints and asserts every table returned to backup-time state.
6. Writes `simulate-restore-results.txt` next to the script.
## Testing parameters
- Stack must be up with the backup profile:
`docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup up -d`
- Run from git bash on the host: `bash scripts/backup/simulate-restore.sh`
- Coverage: all `BASE TABLE` rows in `shopdb_flask` (110+ tables), one pass.
## Expected drift (NOT failures)
- `apitokens`: the app rewrites token last-used continuously; restore is
correct at backup time.
- Endpoint-agent tables (`computers`, `endpointstates`, `endpointevents`,
`endpointapplications`): external fleet agents keep writing after restore and
cannot be quiesced from here.
- Asset ROW deletion is not permitted in a live/CMMC environment (RESTRICT +
compliance). The cascade edge is reported informationally only. A missing or
corrupted asset is still fully restored because the dump drops and recreates
the table - restore fidelity does not depend on a live delete.

197
scripts/backup/backup.sh Executable file
View File

@@ -0,0 +1,197 @@
#!/usr/bin/env bash
# ShopDB backup one-shot.
#
# Produces one recoverable set per run and stores it two ways: a plain
# db.sql.gz + manifest.json on the local /backups volume for a fast, repo-free
# restore, and a restic snapshot (db dump + instance tree + manifest) at
# RESTIC_REPOSITORY for a deduped, encrypted, offsite copy.
#
# Sequence:
# 1. mysqldump --single-transaction (InnoDB-consistent, non-blocking)
# 2. manifest.py stamps core+plugin Alembic revisions + image tag
# 3. restic backup the dump + /app/instance + manifest as ONE snapshot
# 4. restic forget --keep-daily 5 --keep-weekly 4 --keep-monthly 6 --prune
# 5. restic check - fail loudly if the repo is unhealthy, so a bad repo is
# caught on backup night, not restore night.
#
# ASCII only, no emojis - matches the repo style gate.
set -euo pipefail
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
SITE="${SHOPDB_SITE:-shopdb}"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
SETNAME="${SITE}-${STAMP}"
SETDIR="${BACKUPS_ROOT}/${SETNAME}"
# shellcheck source=scripts/backup/status.sh
. /opt/backup/status.sh
log() { echo "[backup] $*"; }
# Record the run outcome on any exit so a failed backup is visible in the UI
# history, not just a silent gap.
BACKUP_DONE=false
_on_exit() {
code=$?
if [ "${BACKUP_DONE}" = true ]; then
status_write backup done "${SETNAME}" "Backup complete"
history_append backup success "${SETNAME}" "Backup complete"
else
status_write backup failed "${SETNAME}" "Backup failed (exit ${code})"
history_append backup failed "${SETNAME}" "Backup failed (exit ${code})"
fi
}
trap _on_exit EXIT
mkdir -p "${SETDIR}"
status_write backup database "${SETNAME}" "Dumping database"
# 1. Database: whole-DB logical dump, consistent and non-blocking on InnoDB.
# Dumped WITHOUT --databases on purpose. That flag bakes CREATE DATABASE and USE
# <name> into the file, which pins the dump to the schema name it was taken
# from: restoring onto a site whose MYSQL_DATABASE differs loaded the rows into
# the dump's schema, left the live one untouched, and reported success.
# --no-tablespaces because dumping tablespace metadata needs the PROCESS
# privilege, which the app user does not have on MySQL 8.
log "dumping database ${MYSQL_DATABASE} from ${MYSQL_HOST}"
mysqldump \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--single-transaction \
--routines \
--triggers \
--no-tablespaces \
"${MYSQL_DATABASE}" \
| gzip -c > "${SETDIR}/db.sql.gz"
log "database dump written ($(du -h "${SETDIR}/db.sql.gz" | cut -f1))"
# 2. Manifest: the schema stamp that makes a restore a checked operation.
# Capture the revision actually applied to the DATABASE (its alembic_version),
# not just the code head. A restore must gate on the DATA revision; stamping
# only the code head hides an image that has outrun its own database and lets a
# behind-schema dump restore under ahead-schema code (1054 Unknown column on
# every affected table). Query it straight from the dumped DB.
log "reading applied database revision (alembic_version)"
DATAREV="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT version_num FROM alembic_version LIMIT 1;" \
"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
if [ -z "${DATAREV}" ]; then
# Recorded loudly rather than shrugged off: a set with no data revision
# cannot be gated on restore, and the gate treats an empty stamp as "no
# evidence" and refuses it. Better a visible warning now than a set that
# looks restorable and is not.
log "WARN could not read alembic_version - this set will NOT be restorable"
history_append backup warning "${SETNAME}" \
"Could not read alembic_version; set has no data revision and cannot be restored"
fi
# The same question for every PLUGIN chain. Each plugin owns an
# alembic_version_<plugin> table, and those move independently of the core one:
# code rolled back to an older plugin revision can sit at an identical core
# head while the dump carries plugin tables it has never heard of. Stamping only
# the core revision let that combination pass the restore gate, load, and then
# fail during the plugin upgrade with the data already replaced. Read straight
# from the dumped DB, same as the core revision above.
log "reading applied plugin revisions (alembic_version_*)"
DATAPLUGINS="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT table_name FROM information_schema.tables
WHERE table_schema = '${MYSQL_DATABASE}'
AND table_name LIKE 'alembic\\_version\\_%';" 2>/dev/null \
| tr -d '\r' \
| while read -r table; do
[ -n "${table}" ] || continue
plugin="${table#alembic_version_}"
rev="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT version_num FROM \`${table}\` LIMIT 1;" \
"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
[ -n "${rev}" ] && printf '%s\t%s\n' "${plugin}" "${rev}"
done \
| python3 -c 'import json,sys; print(json.dumps(dict(l.rstrip("\n").split("\t",1) for l in sys.stdin if "\t" in l)))')"
[ -n "${DATAPLUGINS}" ] || DATAPLUGINS='{}'
log "plugin revisions: ${DATAPLUGINS}"
log "writing manifest"
python3 /opt/backup/manifest.py \
--out "${SETDIR}/manifest.json" \
--takenat "${STAMP}" \
--site "${SITE}" \
--imagetag "${SHOPDB_IMAGE_TAG:-source}" \
--datarev "${DATAREV}" \
--dataplugins "${DATAPLUGINS}" \
--migrations /app/migrations \
--plugins /app/plugins
# Consistency guard: if the DATA revision does not match the CODE head, this
# backup captures a database the running image has outrun. It is still stored
# (losing it would be worse) but flagged loudly so a restore of it is a
# deliberate choice, not a surprise.
CODEREV="$(python3 -c \
'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -n "${DATAREV}" ] && [ "${DATAREV}" != "${CODEREV}" ]; then
log "WARN schema drift: database is at '${DATAREV}' but code head is '${CODEREV}'"
log "WARN this DB has not run 'flask db upgrade' to the running image; restore will upgrade it"
history_append backup warning "${SETNAME}" "Schema drift: data ${DATAREV} vs code ${CODEREV}"
fi
# 3. restic snapshot: db dump + instance tree + manifest, one point in time.
# Skips gracefully when restic/repo are not configured (local-only sites).
IS_CHECK_PASSED=false
status_write backup snapshot "${SETNAME}" "Snapshotting to backup store"
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
if ! restic snapshots >/dev/null 2>&1; then
log "initializing restic repository at ${RESTIC_REPOSITORY}"
restic init
fi
log "restic backup (db dump + instance tree + manifest)"
restic backup \
--tag "${SITE}" \
--tag "${SETNAME}" \
"${SETDIR}/db.sql.gz" \
"${SETDIR}/manifest.json" \
/app/instance
# 4. Retention: any day in the last 5, plus weekly/monthly tail.
log "pruning per retention policy"
restic forget \
--tag "${SITE}" \
--keep-daily 5 \
--keep-weekly 4 \
--keep-monthly 6 \
--prune
# 5. Integrity: learn about corruption now, not at restore time.
log "verifying repository integrity"
if restic check; then
IS_CHECK_PASSED=true
log "restic check passed"
else
log "restic check FAILED - repository may be corrupt"
fi
else
log "restic not configured - local db.sql.gz + manifest only"
fi
# Record the verification result into the manifest so the UI can badge the set.
python3 /opt/backup/manifest.py \
--amend "${SETDIR}/manifest.json" \
--ischeckpassed "${IS_CHECK_PASSED}"
# Clear the request sentinel the UI drops when an operator clicks "Back up now".
rm -f "${BACKUPS_ROOT}/.backup-requested" 2>/dev/null || true
BACKUP_DONE=true
log "backup set ${SETNAME} complete"

163
scripts/backup/manifest.py Normal file
View File

@@ -0,0 +1,163 @@
#!/usr/bin/env python3
"""Backup manifest writer.
The manifest is the stamp that turns "we have dumps" into "we can actually
restore." It records, at capture time:
- takenat : ISO-8601 UTC timestamp of the run
- site : the site tag
- imagetag : the app image tag that produced the data
- schemarev : the core Alembic head revision(s) the running CODE is at, plus
each plugin chain's heads - the versions a restore target must
be at least, or restored rows reference columns the live schema
has not applied yet.
- datarev : the CORE Alembic revision actually applied to the DATABASE that
was dumped (its alembic_version), which is the number a restore
must gate on. schemarev stamps the code on disk; datarev stamps
the data in the dump. They diverge when an image is ahead of a DB
that has not run 'flask db upgrade' yet, and that divergence is
exactly what a code-to-code check cannot see.
- dataplugins: the same thing for every PLUGIN chain - the applied revision of
each alembic_version_<plugin> table. The core chain alone cannot
answer compatibility: plugin schemas move independently, so code
rolled back to an older plugin revision can sit at an identical
core head while the dump carries plugin tables it has never heard
of. Gating on core only let that pass, load, and then fail during
the plugin upgrade with the data already replaced.
Two modes:
--out ... write a fresh manifest for a new set
--amend ... patch an existing manifest (used to record ischeckpassed after
restic check runs, or to backfill datarev)
ASCII only, no emojis - matches the repo style gate.
"""
import argparse
import json
import os
import sys
def _heads_for(chain_dir):
"""Return the Alembic head revision id(s) under a versions directory.
We read the revision graph the cheap way: a head is a revision no other
revision names as its down_revision. Done by text scan so this stays
dependency-light and works for both the core chain and each plugin chain.
"""
versions = os.path.join(chain_dir, 'versions')
if not os.path.isdir(versions):
return []
revisions = {}
down_refs = set()
for name in os.listdir(versions):
if not name.endswith('.py'):
continue
path = os.path.join(versions, name)
revid = None
downid = None
try:
with open(path, 'r', encoding='utf-8') as handle:
for line in handle:
stripped = line.strip()
if stripped.startswith('revision =') and revid is None:
revid = stripped.split('=', 1)[1].strip().strip('"\'')
elif stripped.startswith('down_revision ='):
value = stripped.split('=', 1)[1].strip()
downid = None if value in ('None', 'null') else value.strip('"\'')
except OSError:
continue
if revid:
revisions[revid] = downid
if downid:
down_refs.add(downid)
return sorted(rev for rev in revisions if rev not in down_refs)
def _parse_dataplugins(raw):
"""The applied per-plugin revisions, as a {plugin: revision} map.
Bad or absent JSON becomes an empty map rather than an error: a manifest
that cannot say what the plugin chains were at is exactly the legacy case
the restore gate already knows how to treat conservatively, and failing the
backup here would lose the set over a stamp.
"""
if not raw:
return {}
try:
value = json.loads(raw)
except ValueError:
return {}
if not isinstance(value, dict):
return {}
return {str(k): str(v) for k, v in value.items() if k and v}
def _collect_schema(migrations_dir, plugins_dir):
"""Core heads plus each plugin's chain heads, keyed for compatibility checks."""
schema = {'core': _heads_for(migrations_dir), 'plugins': {}}
if plugins_dir and os.path.isdir(plugins_dir):
for name in sorted(os.listdir(plugins_dir)):
chain = os.path.join(plugins_dir, name, 'migrations')
heads = _heads_for(chain)
if heads:
schema['plugins'][name] = heads
return schema
def write_manifest(args):
manifest = {
'takenat': args.takenat,
'site': args.site,
'imagetag': args.imagetag,
'schemarev': _collect_schema(args.migrations, args.plugins),
'datarev': args.datarev or '',
'dataplugins': _parse_dataplugins(args.dataplugins),
'ischeckpassed': False,
}
with open(args.out, 'w', encoding='utf-8') as handle:
json.dump(manifest, handle, indent=2, sort_keys=True)
return 0
def amend_manifest(args):
try:
with open(args.amend, 'r', encoding='utf-8') as handle:
manifest = json.load(handle)
except (OSError, ValueError):
manifest = {}
if args.ischeckpassed is not None:
manifest['ischeckpassed'] = args.ischeckpassed.lower() == 'true'
if args.datarev is not None:
manifest['datarev'] = args.datarev
if args.dataplugins is not None:
manifest['dataplugins'] = _parse_dataplugins(args.dataplugins)
with open(args.amend, 'w', encoding='utf-8') as handle:
json.dump(manifest, handle, indent=2, sort_keys=True)
return 0
def main(argv=None):
parser = argparse.ArgumentParser(description='ShopDB backup manifest writer')
parser.add_argument('--out')
parser.add_argument('--amend')
parser.add_argument('--takenat', default='')
parser.add_argument('--site', default='shopdb')
parser.add_argument('--imagetag', default='source')
parser.add_argument('--migrations', default='/app/migrations')
parser.add_argument('--plugins', default='/app/plugins')
parser.add_argument('--datarev')
parser.add_argument('--dataplugins',
help='JSON object of {plugin: applied revision}')
parser.add_argument('--ischeckpassed')
args = parser.parse_args(argv)
if args.amend:
return amend_manifest(args)
if args.out:
return write_manifest(args)
parser.error('one of --out or --amend is required')
if __name__ == '__main__':
sys.exit(main())

290
scripts/backup/restore.sh Executable file
View File

@@ -0,0 +1,290 @@
#!/usr/bin/env bash
# ShopDB restore.
#
# The DANGEROUS half of the pair, so it is deliberately guarded. A restore is
# only safe onto code whose schema is at least the set's - otherwise restored
# rows reference columns the live schema has not applied yet and every query on
# those tables 500s. This script refuses an incompatible restore rather than
# leaving the site half-broken.
#
# Usage (from the backup container):
# docker compose -f docker-compose.yml -f docker-compose.backup.yml \
# run --rm --entrypoint /opt/backup/restore.sh backup <snapshot-set-name>
#
# It does NOT roll back code - that is git's job (git checkout <tag> && build).
# Restore code FIRST, bring the schema up (flask db upgrade + plugin
# upgrade-all), THEN run this to load the data.
#
# ASCII only, no emojis - matches the repo style gate.
set -euo pipefail
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
SETNAME="${1:-}"
# Overridable so the gate can be exercised outside a container against throwaway
# migration trees. In the image these are exactly the mounted paths, so the
# shipped behaviour is unchanged; test-restore-datarev.sh drives THIS script
# rather than a copy of its logic, which is the only way the test can notice
# this script drifting.
BACKUP_LIB_DIR="${BACKUP_LIB_DIR:-/opt/backup}"
APP_MIGRATIONS_DIR="${APP_MIGRATIONS_DIR:-/app/migrations}"
APP_PLUGINS_DIR="${APP_PLUGINS_DIR:-/app/plugins}"
APP_INSTANCE_DIR="${APP_INSTANCE_DIR:-/app/instance}"
# When set, stop after the compatibility gate and report the verdict as the exit
# code, touching nothing. This is what makes the gate testable.
RESTORE_CHECK_ONLY="${RESTORE_CHECK_ONLY:-}"
# shellcheck source=scripts/backup/status.sh
. "${BACKUP_LIB_DIR}/status.sh"
log() { echo "[restore] $*"; }
# On any exit, stamp the terminal phase and append a history record so the UI
# reflects success or failure even when a step aborts early (set -e). A restore
# is only "done" once the DATA has been upgraded to the running CODE, and that
# upgrade runs in the Flask-capable api container, not here (this sidecar has no
# app). So the happy path ends by handing off to the api: this script stamps an
# 'upgrading' phase and drops an .upgrade-requested sentinel, and the api runs
# 'flask db upgrade' + 'flask plugin upgrade-all' and writes the final 'done'.
RESTORE_LOADED=false
_on_exit() {
code=$?
if [ "${RESTORE_LOADED}" = true ]; then
# Data is loaded; the api takes it from 'upgrading' to 'done'. Do not
# stamp success here or the UI would call the restore complete before
# the schema is actually brought up to code.
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
else
status_write restore failed "${SETNAME}" "Restore failed (exit ${code})"
history_append restore failed "${SETNAME}" "Restore failed (exit ${code})"
fi
}
trap _on_exit EXIT
die() { echo "[restore] ERROR: $*" >&2; exit 1; }
[ -n "${SETNAME}" ] || die "usage: restore.sh <snapshot-set-name>"
status_write restore checking "${SETNAME}" "Checking schema compatibility"
SETDIR="${BACKUPS_ROOT}/${SETNAME}"
DBDUMP="${SETDIR}/db.sql.gz"
# Prefer the local fast path; fall back to pulling the set out of restic.
if [ ! -f "${DBDUMP}" ]; then
log "local dump not found; restoring set from restic"
command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
restic restore latest --tag "${SETNAME}" --target / --include "${SETDIR}" \
|| die "restic restore of ${SETNAME} failed"
fi
[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
# Compatibility gate: the restore must gate on the revision the DUMP'S DATA is
# at (datarev), not on a code-to-code comparison. Loading data that sits at an
# OLDER revision than the running code is fine - we upgrade it afterwards - but
# loading data NEWER than the code (a revision the code has never heard of)
# would leave rows referencing columns the code cannot migrate to, so that is
# refused.
log "checking schema compatibility"
# The set's data revision: prefer the manifest field; for older manifests that
# predate datarev, parse the real alembic_version straight out of the dump so a
# legacy set is still gated on its true data, not on a code-derived stamp.
SET_DATAREV="$(python3 -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${SET_DATAREV}" ]; then
log "manifest has no datarev (legacy set) - reading it from the dump"
SET_DATAREV="$(gunzip -c "${DBDUMP}" \
| grep -A2 -i 'INSERT INTO `alembic_version`' \
| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
fi
if [ -z "${SET_DATAREV}" ]; then
die "cannot determine the set's data revision - refusing an unverified restore"
fi
# Does the running code know this revision at all? If a migration file with that
# revision id exists in the core chain, the code is at-or-past it (the head is
# the tip of the same chain), so a post-load upgrade can carry the data forward.
# If it is absent, the dump is NEWER than the code - refuse.
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" "${APP_MIGRATIONS_DIR}/versions" 2>/dev/null; then
log "set datarev : ${SET_DATAREV}"
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
fi
log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
# The same gate for every PLUGIN chain, because the core chain alone cannot
# answer this. Plugin schemas move independently of core: roll the code back to
# an older plugin revision and the core head can be identical while the dump
# carries plugin tables the code has never heard of. Gating on core only let
# that pass, load the dump, and then fail in 'flask plugin upgrade-all' with
# "Can't locate revision" and the data already replaced - a failed restore that
# has destroyed what it was restoring over.
#
# A manifest with no dataplugins is a legacy set taken before this was stamped.
# Its plugin revisions are unknowable, so it is allowed through on the core gate
# alone (the behaviour before this check existed) rather than being made
# unrestorable retroactively.
SET_DATAPLUGINS="$(python3 -c \
'import json,sys
try:
data = json.load(open(sys.argv[1])).get("dataplugins") or {}
except Exception:
data = {}
for name, rev in sorted(data.items()):
print(name + " " + rev)' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${SET_DATAPLUGINS}" ]; then
log "manifest stamps no plugin revisions (legacy set) - core gate only"
else
while read -r plugin rev; do
[ -n "${plugin}" ] || continue
chain="${APP_PLUGINS_DIR}/${plugin}/migrations/versions"
if [ ! -d "${chain}" ]; then
die "the backup carries data for plugin '${plugin}', which this build does not have - deploy a build that includes it before restoring"
fi
if ! grep -rqE "^revision = ['\"]${rev}['\"]" "${chain}" 2>/dev/null; then
log "plugin ${plugin} data revision: ${rev}"
die "plugin '${plugin}' data is at a revision this code does not have - deploy code at or past '${rev}' before restoring"
fi
log "plugin ${plugin} at ${rev} - known to this code"
done <<PLUGINREVS
${SET_DATAPLUGINS}
PLUGINREVS
fi
# Last gate: which schema does this dump target? A dump taken with --databases
# carries CREATE DATABASE and USE <name>, pinning it to the schema it came from.
# Loaded into a site whose MYSQL_DATABASE differs, the rows land in the dump's
# schema, the live database is untouched, and the restore reports success. New
# sets are dumped without it and load into whatever schema is named here; older
# sets are still bound, so refuse rather than restore beside the live data.
# Checked HERE, with the other gates, so it costs nothing and stops before the
# safety copy rather than after it.
DUMP_SCHEMA="$(gunzip -c "${DBDUMP}" \
| grep -m1 -oE '^USE `[^`]+`' \
| sed -E 's/^USE `(.*)`/\1/' || true)"
if [ -n "${DUMP_SCHEMA}" ]; then
if [ "${DUMP_SCHEMA}" != "${MYSQL_DATABASE}" ]; then
die "this set is bound to schema '${DUMP_SCHEMA}' but this site's database is '${MYSQL_DATABASE}' - restoring it would load the data beside the live database and report success"
fi
log "legacy schema-bound dump for '${DUMP_SCHEMA}' - matches this site"
fi
if [ -n "${RESTORE_CHECK_ONLY}" ]; then
log "check-only: every gate passed, stopping before anything destructive"
RESTORE_LOADED=false
trap - EXIT
exit 0
fi
# Safety set FIRST. Everything below this point is destructive and none of it
# is transactional: the dump carries DROP TABLE per table, and the instance tree
# is replaced wholesale. Without a copy taken here, a restore of the wrong set -
# or a failure part-way through the load - leaves nothing to go back to. This is
# a local copy on the same volume, so it is cheap and it is NOT a substitute for
# a real backup; it exists to make THIS operation reversible.
SAFETY_STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
SAFETY_DIR="${BACKUPS_ROOT}/pre-restore-${SAFETY_STAMP}"
status_write restore safety "${SETNAME}" "Copying the current state aside"
log "taking a pre-restore safety copy into ${SAFETY_DIR}"
mkdir -p "${SAFETY_DIR}" || die "could not create ${SAFETY_DIR}"
mysqldump \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--single-transaction \
--routines \
--triggers \
--no-tablespaces \
"${MYSQL_DATABASE}" \
| gzip -c > "${SAFETY_DIR}/db.sql.gz" \
|| die "pre-restore dump failed - refusing to touch the live database"
if [ -d /app/instance ]; then
tar -czf "${SAFETY_DIR}/instance.tar.gz" -C /app instance \
|| die "pre-restore instance copy failed - refusing to touch the live files"
fi
# Named so an operator finding this directory months later knows what it is and
# how to use it, without reading this script.
cat > "${SAFETY_DIR}/README.txt" <<SAFETYNOTE
Automatic pre-restore copy of the state this site was in immediately before
restoring set '${SETNAME}' at ${SAFETY_STAMP}.
db.sql.gz mysqldump of ${MYSQL_DATABASE} as it was
instance.tar.gz /app/instance as it was (absent if there was no such tree)
To go back to this state, load the dump the same way a restore does:
gunzip -c db.sql.gz | mysql --host=... --user=... --password=... \
--database=${MYSQL_DATABASE}
tar -xzf instance.tar.gz -C /app
This is a local copy on the backups volume, taken for reversibility of one
restore. It is not a backup set: it has no manifest and no restic snapshot, and
nothing prunes it.
SAFETYNOTE
log "safety copy written ($(du -sh "${SAFETY_DIR}" | cut -f1))"
# Files first, then DB, so the two land together.
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
status_write restore files "${SETNAME}" "Restoring instance files"
# Scoped to THIS set's snapshot. A bare 'latest' restores whatever was
# snapshotted most recently, so restoring anything other than the newest set
# paired that set's DATABASE with the NEWEST snapshot's uploads and
# plugins.json - a mismatch nothing downstream could detect.
log "restoring instance files from the ${SETNAME} snapshot"
STAGE="$(mktemp -d)"
restic restore latest --tag "${SETNAME}" --target "${STAGE}" \
--include /app/instance \
|| die "restic restore of instance files for ${SETNAME} failed"
# Replace the CONTENTS of the instance tree rather than restoring over it,
# so files created after the snapshot are gone afterwards - a restore has to
# put the site back to the snapshot, not merge into it. Done as an explicit
# staged swap rather than 'restic restore --delete --target /', because that
# aims a delete pass at the container root and the blast radius of getting
# its filter wrong is the whole filesystem. /app/instance is a mount, so the
# directory itself has to survive; only what is inside it is replaced.
if [ -d "${STAGE}/app/instance" ]; then
find /app/instance -mindepth 1 -delete \
|| die "could not clear the instance tree before restoring it"
cp -a "${STAGE}/app/instance/." /app/instance/ \
|| die "could not lay down the restored instance tree"
else
log "WARN snapshot for ${SETNAME} carried no /app/instance - leaving files alone"
fi
rm -rf "${STAGE}"
fi
status_write restore database "${SETNAME}" "Loading database dump"
log "loading database dump into ${MYSQL_DATABASE}"
gunzip -c "${DBDUMP}" | mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--database="${MYSQL_DATABASE}" \
|| die "database load failed"
# Hand off to the api container for the schema upgrade. The dump may sit at an
# older revision than this image (allowed and expected); without the upgrade the
# data stays behind the code and every query on a table the newer migrations
# changed 500s with 1054 Unknown column. This sidecar has no Flask, so it cannot
# run 'flask db upgrade' itself - it records the intent and lets the api (which
# owns the app context and already runs these exact idempotent commands on every
# deploy) finalize. The api clears .upgrade-requested and stamps the final
# 'done' once the upgrade succeeds.
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
printf '%s' "${SETNAME}" > "${BACKUPS_ROOT}/.upgrade-requested"
log "database loaded at revision ${SET_DATAREV}; handed off to api for schema upgrade"
rm -f "${BACKUPS_ROOT}/.restore-requested" 2>/dev/null || true
RESTORE_LOADED=true

235
scripts/backup/simulate-restore.sh Executable file
View File

@@ -0,0 +1,235 @@
#!/usr/bin/env bash
# ShopDB restore simulation harness. ***FOR DEVELOPER USE ONLY***
#
# This is a developer diagnostic, NOT an operator tool and NOT part of the
# shipped backup feature. It mutates a live database on purpose. Do not run it
# against production or any environment you are not prepared to disturb.
#
# WHY: the restore path reloads the WHOLE database in one pass (mysqldump
# --databases emits DROP TABLE + CREATE TABLE + full INSERTs, so restore is a
# full replace, not a row-level merge). The strongest real-world check is a
# full-fidelity round trip: fingerprint every table, take a backup, mutate
# across the table archetypes (core standalone, asset + plugin subtype, and
# consumable ledgers), restore, then prove every table's fingerprint returns to
# the backup-time state.
#
# It drives the exact sentinel path the UI uses (.backup-requested /
# .restore-requested on the shared volume), not a private code path, so a pass
# here is a pass for the operator-facing feature.
#
# KNOWN, EXPECTED, NOT-A-FAILURE behaviors (see LiveDatabaseTest notes below):
# - apitokens does not round-trip bit-for-bit: the app rewrites token
# last-used timestamps continuously, so restore reproduces backup-time
# state, which differs from any later fingerprint. This is correct.
# - endpoint-agent tables (computers heartbeat, endpointstates/events/
# applications) churn from the EXTERNAL fleet agents, which cannot be
# quiesced. Post-restore writes from them are expected drift, not a
# restore defect.
# - Asset ROWS cannot be deleted in a live/CMMC environment (RESTRICT and
# compliance controls). The cascade probe is therefore informational only;
# restore fidelity does not depend on it. A missing/corrupted asset is
# still fully restored because the dump drops and recreates the table.
#
# Usage (from a host with the stack up, git bash):
# bash scripts/backup/simulate-restore.sh
#
# ASCII only, no emojis - matches the repo style gate.
set -uo pipefail
# Silence the docker CLI "What's next" hints - they bleed into command
# substitution and corrupt captured SQL output.
export DOCKER_CLI_HINTS=false
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.backup.yml --profile backup"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# Tables that legitimately differ after restore and MUST NOT be counted as
# failures:
# - App-churn: token last-used, sessions, audit trail.
# - Endpoint-agent churn: the external fleet agents keep writing to these
# after restore (heartbeats, state, events, inventory) and cannot be
# quiesced from here.
# Restore reproduces their state at dump time; later fingerprints differ by
# design, so they are reported informationally.
VOLATILE="apitokens sessions auditlogs computers endpointstates endpointevents endpointapplications"
WORKDIR="$(mktemp -d)"
BASELINE="${WORKDIR}/baseline.txt"
MUTATED="${WORKDIR}/mutated.txt"
RESTORED="${WORKDIR}/restored.txt"
MARKER="simrestore-$(date -u +%Y%m%dT%H%M%SZ)"
log() { echo "[sim] $*"; }
die() { echo "[sim] ERROR: $*" >&2; exit 1; }
# Run a SQL statement, returning tab-separated rows with no column headers.
sql() {
printf '%s\n' "$1" | ${COMPOSE} exec -T db sh -c \
'mysql -ushopdb -p"$MYSQL_PASSWORD" shopdb_flask -N 2>/dev/null'
}
# Fingerprint every base table with CHECKSUM TABLE into the given file.
fingerprint() {
out="$1"
tables="$(sql "SELECT table_name FROM information_schema.tables \
WHERE table_schema='shopdb_flask' AND table_type='BASE TABLE' \
ORDER BY table_name;" | tr -d '\r')"
: > "${out}"
for table in ${tables}; do
# CHECKSUM TABLE returns "dbname.table<TAB>checksum"; keep table+checksum.
line="$(sql "CHECKSUM TABLE \`${table}\`;" | tr -d '\r')"
checksum="$(printf '%s' "${line}" | awk '{print $2}')"
printf '%s\t%s\n' "${table}" "${checksum}" >> "${out}"
done
}
wait_for_phase() {
kind="$1"; want="$2"; timeout="${3:-120}"
waited=0
while [ "${waited}" -lt "${timeout}" ]; do
phase="$(${COMPOSE} exec -T backup-watcher sh -c \
"cat /backups/.${kind}-status.json 2>/dev/null" \
| sed -n 's/.*\"phase\":\"\([^\"]*\)\".*/\1/p')"
if [ "${phase}" = "${want}" ]; then return 0; fi
if [ "${phase}" = "failed" ]; then
die "${kind} reported failed status"
fi
sleep 3
waited=$((waited + 3))
done
die "${kind} did not reach '${want}' within ${timeout}s (last: ${phase:-none})"
}
# --- 0. Preconditions ------------------------------------------------------
${COMPOSE} exec -T db sh -c 'true' >/dev/null 2>&1 \
|| die "db container not reachable - is the stack up?"
# --- 0b. Datarev-gate preflight (external tester) --------------------------
# This live round trip runs one codebase against itself, so it can never
# reproduce the "data revision behind the running code" case the restore gate
# now handles. Run the standalone datarev tester first so a pass here also
# covers that offline contract (manifest datarev, known-vs-unknown revision
# refusal, legacy fallback, and the api upgrade handoff).
log "running datarev gate preflight (test-restore-datarev.sh)"
bash "${SCRIPT_DIR}/test-restore-datarev.sh" \
|| die "datarev gate preflight failed - see output above"
log "datarev gate preflight passed"
# --- 1. Backup the current state, then fingerprint the RESTORE TARGET ------
# The restore point is the database as it existed WHEN THE DUMP RAN, not some
# earlier moment. Live-churn tables (apitokens last-used, sessions, auditlogs)
# move under the running app, so the baseline must be captured immediately
# after the backup completes - that is the exact state restore must reproduce.
log "requesting a backup (sentinel .backup-requested)"
${COMPOSE} exec -T backup-watcher sh -c 'echo sim > /backups/.backup-requested'
wait_for_phase backup done 300
SETNAME="$(${COMPOSE} exec -T backup-watcher sh -c \
'cat /backups/.backup-status.json 2>/dev/null' \
| sed -n 's/.*\"setname\":\"\([^\"]*\)\".*/\1/p')"
[ -n "${SETNAME}" ] || die "could not read the completed backup set name"
log "backup complete: ${SETNAME}"
log "baseline fingerprint of all tables (restore target)"
fingerprint "${BASELINE}"
baseline_count="$(wc -l < "${BASELINE}")"
log "fingerprinted ${baseline_count} tables"
# Quiesce the app for the mutate/restore/verify window. A real DR runbook stops
# serving traffic before a restore; here it also stops the endpoint-agent write
# path (computers heartbeat, endpointstates/events/applications) that would
# otherwise churn the DB concurrently and defeat a bit-for-bit comparison.
log "stopping api to quiesce writers"
${COMPOSE} stop api >/dev/null 2>&1
resume_api() { log "resuming api"; ${COMPOSE} start api >/dev/null 2>&1; }
trap resume_api EXIT
# --- 2. Mutate across the table archetypes ---------------------------------
# Core standalone: rename a vendor and flip a setting.
# Ledger/consumable: bump a modelsupplies quantityonhand.
# NOTE: asset ROW deletion is intentionally NOT performed. In a live/CMMC
# environment assets cannot be removed (RESTRICT + compliance controls), which
# is the app behaving as designed. Restore fidelity does not depend on it: the
# dump drops and recreates every table, so a missing or corrupted asset is
# fully reconstructed regardless. The cascade edge is reported informationally.
log "mutating rows (marker ${MARKER})"
sql "UPDATE vendors SET vendorname=CONCAT('${MARKER}-', vendorname) \
ORDER BY vendorid LIMIT 1;"
sql "UPDATE settings SET settingvalue='${MARKER}' \
WHERE settingkey=(SELECT settingkey FROM (SELECT settingkey FROM settings \
ORDER BY settingkey LIMIT 1) AS s);"
# Cascade edge (informational only - no destructive delete): report the FK
# delete rule that would remove computer subtype rows if an asset were deleted.
CASCADE_RULE="$(sql "SELECT delete_rule FROM information_schema.referential_constraints \
WHERE constraint_schema='shopdb_flask' AND referenced_table_name='assets' \
AND table_name='computers' LIMIT 1;" | tr -d '\r')"
log "cascade edge computers->assets delete_rule=${CASCADE_RULE:-unknown} (live asset delete not permitted; not exercised)"
# Ledger: move a cached quantityonhand so the restore must revert it.
sql "UPDATE modelsupplies SET quantityonhand=quantityonhand+9999 \
ORDER BY id LIMIT 1;" 2>/dev/null \
|| sql "UPDATE modelsupplies SET quantityonhand=quantityonhand+9999 \
ORDER BY modelsupplyid LIMIT 1;"
log "post-mutation fingerprint"
fingerprint "${MUTATED}"
changed="$(diff <(sort "${BASELINE}") <(sort "${MUTATED}") | grep -c '^>')"
log "mutation changed ${changed} table fingerprint(s)"
# --- 3. Restore the backup we took -----------------------------------------
log "requesting restore of ${SETNAME}"
${COMPOSE} exec -T backup-watcher sh -c "printf '%s' '${SETNAME}' > /backups/.restore-requested"
wait_for_phase restore done 600
log "restore complete"
# --- 4. Verify every table returned to baseline ----------------------------
log "post-restore fingerprint"
fingerprint "${RESTORED}"
fails=0
volatile_diffs=0
while IFS="$(printf '\t')" read -r table base; do
restored="$(grep -P "^${table}\t" "${RESTORED}" | awk '{print $2}')"
if [ "${base}" != "${restored}" ]; then
case " ${VOLATILE} " in
*" ${table} "*)
echo "[sim] (volatile) ${table} differs as expected: baseline=${base} restored=${restored}"
volatile_diffs=$((volatile_diffs + 1))
;;
*)
echo "[sim] MISMATCH ${table}: baseline=${base} restored=${restored}"
fails=$((fails + 1))
;;
esac
fi
done < "${BASELINE}"
echo "=================================================="
RESULTS_FILE="${SCRIPT_DIR}/simulate-restore-results.txt"
{
echo "ShopDB restore simulation results (FOR DEVELOPER USE ONLY)"
echo "run (UTC) : $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "marker : ${MARKER}"
echo "backup set : ${SETNAME}"
echo "tables checked : ${baseline_count}"
echo "cascade edge : computers->assets delete_rule=${CASCADE_RULE:-unknown} (not exercised - live asset delete not permitted)"
echo ""
echo "Expected, not-a-failure drift (excluded from strict equality):"
echo " - apitokens: app rewrites token last-used; restore is correct at backup time."
echo " - endpoint-agent tables (computers, endpointstates/events/applications):"
echo " external fleet agents keep writing post-restore and cannot be quiesced."
echo " - asset row deletion is not permitted live (RESTRICT + CMMC); the table is"
echo " still fully restored because the dump drops and recreates it."
echo ""
if [ "${fails}" -eq 0 ]; then
echo "RESULT: PASS - all ${baseline_count} tables restored to backup-time state"
echo " (${volatile_diffs} volatile table(s) excluded from strict equality)"
else
echo "RESULT: FAIL - ${fails} table(s) did not match after restore"
fi
} | tee "${RESULTS_FILE}"
echo "=================================================="
echo "[sim] results written to ${RESULTS_FILE}"
echo "[sim] work dir: ${WORKDIR}"
echo "=================================================="
[ "${fails}" -eq 0 ]

100
scripts/backup/status.sh Executable file
View File

@@ -0,0 +1,100 @@
# ShopDB backup/restore status + history helper.
#
# WHY this exists: the restore and backup runs happen in the watcher container,
# but the operator watches the UI, which only talks to the api. This helper is
# the bridge: it writes two kinds of state onto the shared /backups volume that
# the api can read.
#
# 1. A LIVE status file (.restore-status.json / .backup-status.json) rewritten
# at each phase so the UI can show an indeterminate spinner with a phase
# label (queued, restoring files, loading database, done, failed).
# 2. An append-only HISTORY file (history/restores.json / history/backups.json)
# that keeps the record of past runs so the UI can list them in a card.
#
# Both writes are atomic (write temp, rename) so the api never reads a half
# written file. History is bounded to the most recent HISTORY_MAX entries.
#
# This file is SOURCED, not executed. It exposes:
# status_write <kind> <phase> <setname> [message]
# history_append <kind> <outcome> <setname> [message]
#
# kind is "restore" or "backup". ASCII only, no emojis - matches the style gate.
STATUS_BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
HISTORY_MAX="${BACKUP_HISTORY_MAX:-50}"
_status_now() { date -u +%Y-%m-%dT%H%M%SZ; }
_status_json_escape() {
# Escape backslashes and double quotes for embedding in a JSON string.
printf '%s' "${1:-}" | sed 's/\\/\\\\/g; s/"/\\"/g'
}
# status_write <kind> <phase> <setname> [message]
# Rewrites the live status file for this run kind.
status_write() {
kind="$1"
phase="$2"
setname="${3:-}"
message="${4:-}"
statusfile="${STATUS_BACKUPS_ROOT}/.${kind}-status.json"
tmpfile="${statusfile}.tmp.$$"
{
printf '{'
printf '"kind":"%s",' "$(_status_json_escape "${kind}")"
printf '"phase":"%s",' "$(_status_json_escape "${phase}")"
printf '"setname":"%s",' "$(_status_json_escape "${setname}")"
printf '"message":"%s",' "$(_status_json_escape "${message}")"
printf '"updatedat":"%s"' "$(_status_now)"
printf '}\n'
} > "${tmpfile}" 2>/dev/null || return 0
mv -f "${tmpfile}" "${statusfile}" 2>/dev/null || rm -f "${tmpfile}" 2>/dev/null
return 0
}
# history_append <kind> <outcome> <setname> [message]
# Appends a completed run record and trims to HISTORY_MAX newest entries.
history_append() {
kind="$1"
outcome="$2"
setname="${3:-}"
message="${4:-}"
historydir="${STATUS_BACKUPS_ROOT}/history"
historyfile="${historydir}/${kind}s.json"
tmpfile="${historyfile}.tmp.$$"
mkdir -p "${historydir}" 2>/dev/null || return 0
record="$(printf '{"setname":"%s","outcome":"%s","message":"%s","finishedat":"%s"}' \
"$(_status_json_escape "${setname}")" \
"$(_status_json_escape "${outcome}")" \
"$(_status_json_escape "${message}")" \
"$(_status_now)")"
if command -v python3 >/dev/null 2>&1; then
HISTORY_FILE="${historyfile}" HISTORY_RECORD="${record}" \
HISTORY_MAX="${HISTORY_MAX}" python3 - "${tmpfile}" <<'PY' 2>/dev/null || return 0
import json, os, sys
path = os.environ["HISTORY_FILE"]
record = json.loads(os.environ["HISTORY_RECORD"])
limit = int(os.environ.get("HISTORY_MAX", "50"))
tmp = sys.argv[1]
try:
with open(path, "r", encoding="utf-8") as handle:
entries = json.load(handle)
if not isinstance(entries, list):
entries = []
except (OSError, ValueError):
entries = []
entries.append(record)
entries = entries[-limit:]
with open(tmp, "w", encoding="utf-8") as handle:
json.dump(entries, handle)
os.replace(tmp, path)
PY
fi
return 0
}

231
scripts/backup/test-restore-datarev.sh Executable file
View File

@@ -0,0 +1,231 @@
#!/usr/bin/env bash
# ShopDB restore datarev-gate tester. ***FOR DEVELOPER USE ONLY***
#
# WHY THIS EXISTS SEPARATELY FROM simulate-restore.sh:
# simulate-restore.sh is a full-fidelity ROUND TRIP against a live stack: it
# backs up, mutates, restores, and proves every table returns to backup-time
# state. By construction it runs ONE codebase against ITSELF, so the database it
# dumps is always at the SAME revision as the code that restores it. It can
# therefore never exercise the case the FBJXBDB4 outage was about: a dump whose
# DATA revision is BEHIND (or ahead of) the running CODE.
#
# This tester fills that gap offline. It builds throwaway migration trees and
# fake gzipped dumps and drives the exact decision logic restore.sh uses:
# 1. manifest.py records datarev (the DB's real alembic_version), not just the
# code head (schemarev).
# 2. A dump whose datarev the code KNOWS (an ancestor in the chain) is allowed
# - the post-load upgrade rolls it forward.
# 3. A dump whose datarev the code does NOT know (newer than this build) is
# REFUSED - restoring it would 500 on 1054 Unknown column.
# 4. A legacy manifest with no datarev falls back to parsing alembic_version
# straight out of the dump.
# 5. restore.sh hands the loaded restore off to the api via the
# .upgrade-requested sentinel instead of marking it done in the sidecar.
#
# No Docker, no MySQL, no network: it is a pure logic gate test, safe to run in
# CI and from inside simulate-restore.sh as a preflight.
#
# ASCII only, no emojis - matches the repo style gate.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
MANIFEST_PY="${SCRIPT_DIR}/manifest.py"
RESTORE_SH="${SCRIPT_DIR}/restore.sh"
WORKDIR="$(mktemp -d)"
trap 'rm -rf "${WORKDIR}"' EXIT
# The backup image ships python3; a dev host may only have `python` (and on
# Windows `python3` can be a non-functional Store alias that resolves on PATH
# but errors when run). Pick the first interpreter that actually EXECUTES, so
# this tester runs anywhere without changing the shipped scripts.
PY=""
for candidate in python3 python py; do
if "${candidate}" -c 'pass' >/dev/null 2>&1; then
PY="${candidate}"
break
fi
done
[ -n "${PY}" ] || { echo "[test] ERROR: no working python interpreter found"; exit 1; }
passes=0
fails=0
pass() { echo "[test] PASS: $*"; passes=$((passes + 1)); }
fail() { echo "[test] FAIL: $*"; fails=$((fails + 1)); }
# Build a throwaway core migration chain: baseline -> mid -> head. The gate in
# restore.sh only needs each version file to declare `revision = '<id>'`, so
# that is all we write.
build_chain() {
root="$1"
versions="${root}/versions"
mkdir -p "${versions}"
printf "revision = '%s'\ndown_revision = None\n" \
'0001_baseline' > "${versions}/0001_baseline.py"
printf "revision = '%s'\ndown_revision = '%s'\n" \
'0002_mid' '0001_baseline' > "${versions}/0002_mid.py"
printf "revision = '%s'\ndown_revision = '%s'\n" \
'0003_head' '0002_mid' > "${versions}/0003_head.py"
}
# Fake a gzipped mysqldump that stamps a given alembic_version, matching the
# INSERT shape restore.sh greps for when a legacy manifest lacks datarev.
build_dump() {
out="$1"; datarev="$2"
{
echo 'DROP TABLE IF EXISTS `alembic_version`;'
echo 'CREATE TABLE `alembic_version` (`version_num` varchar(32));'
echo 'INSERT INTO `alembic_version` VALUES'
echo "('${datarev}');"
} | gzip -c > "${out}"
}
# Build a throwaway plugin chain, shaped like plugins/<name>/migrations/versions.
build_plugin_chain() {
root="$1"; name="$2"; rev="$3"
versions="${root}/${name}/migrations/versions"
mkdir -p "${versions}"
printf "revision = '%s'\ndown_revision = None\n" \
"${rev}" > "${versions}/${rev}.py"
}
# Drive the REAL restore.sh gate. This is the point of the rewrite: the test
# used to carry its own copy of the predicate, so it stayed green no matter how
# far restore.sh drifted from it. RESTORE_CHECK_ONLY stops the script right
# after the gate, so the verdict here IS the shipped verdict.
run_gate() {
backups_root="$1"; setname="$2"; chain="$3"; plugins="$4"
RESTORE_CHECK_ONLY=1 \
SHOPDB_BACKUPS_ROOT="${backups_root}" \
BACKUP_LIB_DIR="${SCRIPT_DIR}" \
APP_MIGRATIONS_DIR="${chain}" \
APP_PLUGINS_DIR="${plugins}" \
bash "${RESTORE_SH}" "${setname}" >"${WORKDIR}/gate.log" 2>&1
}
CHAIN="${WORKDIR}/migrations"
build_chain "${CHAIN}"
PLUGINS="${WORKDIR}/plugins"
build_plugin_chain "${PLUGINS}" 'endpoints' 'endpoints0002known'
ROOT="${WORKDIR}/backups"
mkdir -p "${ROOT}"
# Lay out one set the way a real backup does, under the backups root.
make_set() {
setname="$1"; datarev="$2"; dumprev="$3"; dataplugins="$4"
setdir="${ROOT}/${setname}"
mkdir -p "${setdir}"
if [ "${datarev}" = "__legacy__" ]; then
printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \
> "${setdir}/manifest.json"
else
${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \
--migrations "${CHAIN}" --plugins "${PLUGINS}" \
--datarev "${datarev}" --dataplugins "${dataplugins}" \
--takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
fi
build_dump "${setdir}/db.sql.gz" "${dumprev}"
printf '%s' "${setdir}"
}
# --- 1. manifest.py records datarev separately from the code head ----------
make_set 'set-behind' '0002_mid' '0002_mid' '{}' >/dev/null
recorded="$(${PY} -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
"${ROOT}/set-behind/manifest.json" 2>/dev/null)"
head_stamp="$(${PY} -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("schemarev",{}).get("core",[]))' \
"${ROOT}/set-behind/manifest.json" 2>/dev/null)"
if [ "${recorded}" = '0002_mid' ]; then
pass "manifest.py records datarev (${recorded}) separate from code head ${head_stamp}"
else
fail "manifest.py did not record datarev (got '${recorded}')"
fi
# --- 2. datarev the code KNOWS (behind head) is allowed --------------------
if run_gate "${ROOT}" 'set-behind' "${CHAIN}" "${PLUGINS}"; then
pass "data behind code (0002_mid) is known and allowed to restore"
else
fail "data behind code was refused: $(tail -1 "${WORKDIR}/gate.log")"
fi
# --- 3. datarev the code does NOT know (newer than build) is refused -------
make_set 'set-ahead' '0004_future' '0004_future' '{}' >/dev/null
if run_gate "${ROOT}" 'set-ahead' "${CHAIN}" "${PLUGINS}"; then
fail "data ahead of code (0004_future) was NOT refused"
else
pass "data ahead of code (0004_future) is unknown and correctly refused"
fi
# --- 4. legacy manifest (no datarev) falls back to the dump ----------------
make_set 'set-legacy' '__legacy__' '0001_baseline' '{}' >/dev/null
if run_gate "${ROOT}" 'set-legacy' "${CHAIN}" "${PLUGINS}"; then
pass "legacy set with no datarev falls back to the dump revision"
else
fail "legacy set fallback failed: $(tail -1 "${WORKDIR}/gate.log")"
fi
# --- 5. restore.sh hands off to the api instead of self-completing ---------
if grep -q '\.upgrade-requested' "${RESTORE_SH}" \
&& grep -q "status_write restore upgrading" "${RESTORE_SH}" \
&& ! grep -qE 'status_write restore done' "${RESTORE_SH}"; then
pass "restore.sh parks in 'upgrading' and hands off via .upgrade-requested"
else
fail "restore.sh does not hand off the upgrade to the api"
fi
# --- 6. a PLUGIN chain the code does not know is refused -------------------
# The core head is identical in both of these; only the plugin revision differs.
# That is the whole failure: a core-only gate cannot tell them apart.
make_set 'set-plugin-known' '0002_mid' '0002_mid' '{"endpoints":"endpoints0002known"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-known' "${CHAIN}" "${PLUGINS}"; then
pass "a set whose plugin revision the code knows is allowed"
else
fail "known plugin revision was refused: $(tail -1 "${WORKDIR}/gate.log")"
fi
make_set 'set-plugin-future' '0002_mid' '0002_mid' '{"endpoints":"endpoints0099future"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-future' "${CHAIN}" "${PLUGINS}"; then
fail "plugin data ahead of code was NOT refused (core-only gate regression)"
else
pass "plugin data ahead of code is refused even though the core head matches"
fi
# --- 7. a plugin the build does not have at all is refused -----------------
make_set 'set-plugin-missing' '0002_mid' '0002_mid' '{"nosuchplugin":"whatever0001"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-missing' "${CHAIN}" "${PLUGINS}"; then
fail "a set carrying data for an absent plugin was NOT refused"
else
pass "a set carrying data for a plugin this build lacks is refused"
fi
# --- 8. a dump bound to another schema is refused --------------------------
# --databases bakes 'USE <name>' in, which silently loaded the rows beside the
# live database and reported success.
setdir="${ROOT}/set-otherschema"
mkdir -p "${setdir}"
${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \
--migrations "${CHAIN}" --plugins "${PLUGINS}" \
--datarev '0002_mid' --dataplugins '{}' \
--takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
{
echo 'CREATE DATABASE `someothersite`;'
echo 'USE `someothersite`;'
echo 'INSERT INTO `alembic_version` VALUES'
echo "('0002_mid');"
} | gzip -c > "${setdir}/db.sql.gz"
# Now a gate, so check-only reaches it without needing a database.
out="$(RESTORE_CHECK_ONLY=1 SHOPDB_BACKUPS_ROOT="${ROOT}" \
BACKUP_LIB_DIR="${SCRIPT_DIR}" \
APP_MIGRATIONS_DIR="${CHAIN}" APP_PLUGINS_DIR="${PLUGINS}" \
MYSQL_DATABASE='shopdb_flask' \
bash "${RESTORE_SH}" 'set-otherschema' 2>&1)" || true
if printf '%s' "${out}" | grep -q "bound to schema 'someothersite'"; then
pass "a dump bound to another site's schema is refused"
else
fail "a schema-bound dump was not caught (got: $(printf '%s' "${out}" | tail -1))"
fi
echo "=================================================="
echo "[test] datarev gate: ${passes} passed, ${fails} failed"
echo "=================================================="
[ "${fails}" -eq 0 ]

61
scripts/backup/watch.sh Executable file
View File

@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# ShopDB backup WATCHER - the resident half of the in-app backup utility.
#
# WHY this exists: the api container cannot run docker, so when an operator
# clicks "Back up now" the api only drops a sentinel file (.backup-requested)
# on the shared backups volume. This watcher is the thing that actuates that
# request: it polls for the sentinel and runs the one-shot backup.sh, which
# ships a restic snapshot to the backup-store container and clears the sentinel
# on success. That turns the UI button into a real, self-serve backup without
# giving the api container any docker or host privileges.
#
# ASCII only, no emojis - matches the repo style gate.
set -uo pipefail
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
SENTINEL="${BACKUPS_ROOT}/.backup-requested"
RESTORE_SENTINEL="${BACKUPS_ROOT}/.restore-requested"
INTERVAL="${BACKUP_POLL_SECONDS:-10}"
log() { echo "[backup-watch] $*"; }
# shellcheck source=scripts/backup/status.sh
. /opt/backup/status.sh
mkdir -p "${BACKUPS_ROOT}"
# The api container runs unprivileged and only drops sentinel files here; it
# cannot write into a root-owned 0755 directory. Make the backups root
# writable so "Back up now" and "Restore" can queue their sentinels.
chmod 0777 "${BACKUPS_ROOT}" 2>/dev/null || true
log "watcher up; polling ${SENTINEL} and ${RESTORE_SENTINEL} every ${INTERVAL}s"
while true; do
if [ -f "${SENTINEL}" ]; then
log "backup requested - running one-shot"
status_write backup queued "" "Backup queued"
if /opt/backup/backup.sh; then
log "backup completed"
else
# backup.sh only clears the sentinel on success, so a failed run
# would otherwise loop forever. Clear it here and wait for the next
# explicit request rather than hammering a broken backup.
log "backup FAILED - clearing request, awaiting next"
rm -f "${SENTINEL}" 2>/dev/null || true
fi
fi
if [ -f "${RESTORE_SENTINEL}" ]; then
# The api drops the target set name into the restore sentinel. Read it,
# hand it to restore.sh, and clear the request whether it succeeds or
# fails so a bad set cannot pin the watcher in a tight retry loop.
setname="$(head -n1 "${RESTORE_SENTINEL}" 2>/dev/null | tr -d '\r\n')"
log "restore requested - running restore for ${setname}"
status_write restore queued "${setname}" "Restore queued"
if /opt/backup/restore.sh "${setname}"; then
log "restore completed"
else
log "restore FAILED - clearing request, awaiting next"
fi
rm -f "${RESTORE_SENTINEL}" 2>/dev/null || true
fi
sleep "${INTERVAL}"
done

4
scripts/check-naming-and-style.sh
View File

@@ -35,6 +35,10 @@ EXCLUDES=(
--exclude-dir=.git
--exclude-dir=versions
--exclude-dir=staticdocs
# Local, gitignored scratch notes (see .gitignore devnotes/). Not product
# source, never committed, and free to hold pasted logs full of em-dashes
# and arrows - grep does not consult .gitignore, so exclude it explicitly.
--exclude-dir=devnotes
)
# Root-level gitignored scratch: review writeups and chat handoffs, free to hold

37
scripts/uninstall-shopdbagent.ps1
View File

@@ -1,10 +1,11 @@
# uninstall-shopdbagent.ps1
# ShopDB Endpoint Agent uninstaller. Run from an ELEVATED (admin) PowerShell on
# the endpoint. Reverses everything install-shopdbagent.ps1 and the site agent
# generator's install.ps1 create:
# the endpoint. Reverses everything the site agent generator's install.ps1
# creates:
# - stops and deletes the Windows service
# - kills any running agent/tray processes
# - removes the tray logon Run key (64-bit and 32-bit views)
# - unregisters the ShopDB user-context probe scheduled task
# - removes the credential hive HKLM\SOFTWARE\GE\ShopDB (both views)
# - removes the Defender path exclusion for the install directory
# - deletes the install directory and the ProgramData data/log tree
@@ -20,6 +21,10 @@ param(
[string] $ServiceName = "ShopDbEndpointAgent",
[string] $TrayExeName = "shopdb-endpoint-tray.exe",
# Scheduled task that runs the user-context probe in the operator's session.
# Must match the installer's -UserContextTaskName.
[string] $UserContextTaskName = "ShopDB user-context probe",
# Also remove any server TLS certificate the site-agent installer imported
# into the machine Trusted Root store. Off by default because a root cert is
# shared trust material: only pull it when you are sure no other service on
@@ -113,6 +118,34 @@ Start-Sleep -Seconds 1
# 3. Remove the tray logon Run key entry (both views) and the credential hive.
Write-Host "[3/6] Removing registry entries ..."
Remove-HklmBothViews -SubKey "SOFTWARE\Microsoft\Windows\CurrentVersion\Run" -Name "ShopDbEndpointTray"
# Unregister the probe task. Left behind it would keep firing at every logon
# against a deleted exe, which shows up as a recurring task-failure history on a
# machine nobody believes has the agent on it any more.
try {
if (Get-ScheduledTask -TaskName $UserContextTaskName -ErrorAction SilentlyContinue) {
Unregister-ScheduledTask -TaskName $UserContextTaskName -Confirm:$false -ErrorAction Stop
Write-Host " removed scheduled task '$UserContextTaskName'"
}
else {
Write-Host " scheduled task '$UserContextTaskName' not present."
}
}
catch {
Write-Warning " could not remove scheduled task '$UserContextTaskName': $($_.Exception.Message)"
}
# The install's one-shot tray launcher is transient and normally removes itself,
# but an install that failed partway can leave it behind.
try {
if (Get-ScheduledTask -TaskName "ShopDB tray first launch" -ErrorAction SilentlyContinue) {
Unregister-ScheduledTask -TaskName "ShopDB tray first launch" -Confirm:$false -ErrorAction Stop
Write-Host " removed leftover scheduled task 'ShopDB tray first launch'"
}
}
catch {
Write-Warning " could not remove scheduled task 'ShopDB tray first launch': $($_.Exception.Message)"
}
Remove-HklmBothViews -SubKey "SOFTWARE\GE\ShopDB"
# Remove the All-Users Start Menu shortcut the site-agent installer created.