6e36df841e
Ports 37 of Benjamin Baker's commits from the wilm branch, mirrored from the
corporate GitHub at 7905917c. wilm rides the publication lineage and shares no
ancestor with wj, so this is a filtered tree patch rather than a merge:
94 files, and every path the publication export strips (docs/, tools/, mcp/,
CLAUDE.md, start-*.sh, .gitea/, the three excluded tests) is held back, because
a raw diff would delete each of them from this tree.
What comes across:
* Endpoint telemetry, phase 2. Physical disk health from two sources, NIC link
state, boot type, Defender health, processor/memory/chassis inventory, the
listening TCP set, mapped network drives and Windows Update history. Five
additive migrations, 0012 through 0016, off the 0011 head.
* Mapped drives are collected by a probe running in the interactive user's
session, because they do not exist in the agent's session 0. Two Windows
sources can disagree about one drive and the worse status wins, so a
half-broken share is never rendered as healthy.
* Windows events are reported once each, keyed on the channel's own record id,
instead of being re-sent whenever a report overlapped the last one.
* PC Detail rework: Details cards flow, the hand-written Network card gives way
to a full address history, Storage and disk health merge into one card with
two sections, Operator Reports and Corporate Tools get their own tabs.
* An opt-in restic backup overlay with a third container, restore gated on the
dump's own data revision rather than the code head, and backup/restore history.
* The agent's version is read from the binary's PE version rather than trusted
from the csproj, and the Fleet view gains an Agent Version column.
* The enqueue route now asks caller_holds instead of the owner's permissions,
which closes the scoped-token elevation on run_script, update_app and
system_restart.
Held back deliberately:
* agent/installer/install-shopdbagent.ps1 stays deleted, as upstream deleted it.
The installer that reaches a bay is generated by _install_script() in
plugins/endpoints/api/siteagent.py; the standalone script had stopped being
shipped and was collecting fixes no endpoint ever ran.
* The .gitignore hunk adding docs/*.md. Correct on the publication lineage,
where docs/ becomes the wiki and its absence is how the gates detect a
published checkout. Here it would silently ignore every new doc.
* Their deletions of docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md. Upstream
those were stubs; here PLUGIN-HOOKS.md is the 31KB document CLAUDE.md points
readers at.
* Five zero-byte files committed by an accidental `git add -A`, a run log from
the author's own machine, and docs/notes/, which is working scratch that says
so in its own header and belongs in devnotes/ on this side.
Carried with the import so it lands green: the 17 new routes are described in
docs/api-inventory.json with the spec and project map regenerated, and two new
frontend specs read fixtures through fileURLToPath(URL) which node rejects under
jsdom's own URL global.
Suite 2591 passed / 9 skipped; frontend 391 passed; naming, style and doc gates
clean.
Scope: product
134 lines
6.0 KiB
Bash
134 lines
6.0 KiB
Bash
#!/usr/bin/env bash
|
|
# ShopDB restore.
|
|
#
|
|
# The DANGEROUS half of the pair, so it is deliberately guarded. A restore is
|
|
# only safe onto code whose schema is at least the set's - otherwise restored
|
|
# rows reference columns the live schema has not applied yet and every query on
|
|
# those tables 500s. This script refuses an incompatible restore rather than
|
|
# leaving the site half-broken.
|
|
#
|
|
# Usage (from the backup container):
|
|
# docker compose -f docker-compose.yml -f docker-compose.backup.yml \
|
|
# run --rm --entrypoint /opt/backup/restore.sh backup <snapshot-set-name>
|
|
#
|
|
# It does NOT roll back code - that is git's job (git checkout <tag> && build).
|
|
# Restore code FIRST, bring the schema up (flask db upgrade + plugin
|
|
# upgrade-all), THEN run this to load the data.
|
|
#
|
|
# ASCII only, no emojis - matches the repo style gate.
|
|
set -euo pipefail
|
|
|
|
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
|
|
SETNAME="${1:-}"
|
|
|
|
# shellcheck source=scripts/backup/status.sh
|
|
. /opt/backup/status.sh
|
|
|
|
log() { echo "[restore] $*"; }
|
|
|
|
# On any exit, stamp the terminal phase and append a history record so the UI
|
|
# reflects success or failure even when a step aborts early (set -e). A restore
|
|
# is only "done" once the DATA has been upgraded to the running CODE, and that
|
|
# upgrade runs in the Flask-capable api container, not here (this sidecar has no
|
|
# app). So the happy path ends by handing off to the api: this script stamps an
|
|
# 'upgrading' phase and drops an .upgrade-requested sentinel, and the api runs
|
|
# 'flask db upgrade' + 'flask plugin upgrade-all' and writes the final 'done'.
|
|
RESTORE_LOADED=false
|
|
_on_exit() {
|
|
code=$?
|
|
if [ "${RESTORE_LOADED}" = true ]; then
|
|
# Data is loaded; the api takes it from 'upgrading' to 'done'. Do not
|
|
# stamp success here or the UI would call the restore complete before
|
|
# the schema is actually brought up to code.
|
|
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
|
|
else
|
|
status_write restore failed "${SETNAME}" "Restore failed (exit ${code})"
|
|
history_append restore failed "${SETNAME}" "Restore failed (exit ${code})"
|
|
fi
|
|
}
|
|
trap _on_exit EXIT
|
|
|
|
die() { echo "[restore] ERROR: $*" >&2; exit 1; }
|
|
|
|
[ -n "${SETNAME}" ] || die "usage: restore.sh <snapshot-set-name>"
|
|
|
|
status_write restore checking "${SETNAME}" "Checking schema compatibility"
|
|
|
|
SETDIR="${BACKUPS_ROOT}/${SETNAME}"
|
|
DBDUMP="${SETDIR}/db.sql.gz"
|
|
|
|
# Prefer the local fast path; fall back to pulling the set out of restic.
|
|
if [ ! -f "${DBDUMP}" ]; then
|
|
log "local dump not found; restoring set from restic"
|
|
command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
|
|
restic restore latest --target / --include "${SETDIR}" \
|
|
|| die "restic restore of ${SETNAME} failed"
|
|
fi
|
|
[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
|
|
|
|
# Compatibility gate: the restore must gate on the revision the DUMP'S DATA is
|
|
# at (datarev), not on a code-to-code comparison. Loading data that sits at an
|
|
# OLDER revision than the running code is fine - we upgrade it afterwards - but
|
|
# loading data NEWER than the code (a revision the code has never heard of)
|
|
# would leave rows referencing columns the code cannot migrate to, so that is
|
|
# refused.
|
|
log "checking schema compatibility"
|
|
|
|
# The set's data revision: prefer the manifest field; for older manifests that
|
|
# predate datarev, parse the real alembic_version straight out of the dump so a
|
|
# legacy set is still gated on its true data, not on a code-derived stamp.
|
|
SET_DATAREV="$(python3 -c \
|
|
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
|
|
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
|
|
if [ -z "${SET_DATAREV}" ]; then
|
|
log "manifest has no datarev (legacy set) - reading it from the dump"
|
|
SET_DATAREV="$(gunzip -c "${DBDUMP}" \
|
|
| grep -A2 -i 'INSERT INTO `alembic_version`' \
|
|
| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
|
|
fi
|
|
|
|
if [ -z "${SET_DATAREV}" ]; then
|
|
die "cannot determine the set's data revision - refusing an unverified restore"
|
|
fi
|
|
|
|
# Does the running code know this revision at all? If a migration file with that
|
|
# revision id exists in the core chain, the code is at-or-past it (the head is
|
|
# the tip of the same chain), so a post-load upgrade can carry the data forward.
|
|
# If it is absent, the dump is NEWER than the code - refuse.
|
|
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
|
|
log "set datarev : ${SET_DATAREV}"
|
|
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
|
|
fi
|
|
log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
|
|
|
|
# Files first, then DB, so the two land together.
|
|
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
|
|
status_write restore files "${SETNAME}" "Restoring instance files"
|
|
log "restoring instance files from restic"
|
|
restic restore latest --target / --include /app/instance \
|
|
|| die "restic restore of instance files failed"
|
|
fi
|
|
|
|
status_write restore database "${SETNAME}" "Loading database dump"
|
|
log "loading database dump"
|
|
gunzip -c "${DBDUMP}" | mysql \
|
|
--host="${MYSQL_HOST}" \
|
|
--user="${MYSQL_USER}" \
|
|
--password="${MYSQL_PASSWORD}" \
|
|
|| die "database load failed"
|
|
|
|
# Hand off to the api container for the schema upgrade. The dump may sit at an
|
|
# older revision than this image (allowed and expected); without the upgrade the
|
|
# data stays behind the code and every query on a table the newer migrations
|
|
# changed 500s with 1054 Unknown column. This sidecar has no Flask, so it cannot
|
|
# run 'flask db upgrade' itself - it records the intent and lets the api (which
|
|
# owns the app context and already runs these exact idempotent commands on every
|
|
# deploy) finalize. The api clears .upgrade-requested and stamps the final
|
|
# 'done' once the upgrade succeeds.
|
|
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
|
|
printf '%s' "${SETNAME}" > "${BACKUPS_ROOT}/.upgrade-requested"
|
|
log "database loaded at revision ${SET_DATAREV}; handed off to api for schema upgrade"
|
|
|
|
rm -f "${BACKUPS_ROOT}/.restore-requested" 2>/dev/null || true
|
|
RESTORE_LOADED=true
|