Close the last three restore-gate gaps, and make the gate test the real gate
Some checks failed
CI / naming (push) Successful in 3s
CI / frontend (push) Successful in 12s
CI / migrations-mysql (push) Successful in 42s
CI / backend (push) Failing after 5m13s

Three defects that all shared a shape: the restore path looked checked and was
not.

An unreadable data revision was treated as a legacy set. backup.sh discards the
mysql error when reading alembic_version, so a set could be written with an
empty datarev; the compatibility check then fell through to the code-to-code
head comparison - the very check datarev was introduced to replace - and badged
the set restorable on no evidence at all. A manifest that CARRIES the key and
left it empty is now refused, while one with no key at all is still the genuine
legacy case and keeps the fallback. backup.sh records the failure in run history
rather than passing it over in a log line, since the set it just wrote cannot be
restored.

Dumps were bound to the schema they came from. mysqldump --databases bakes
CREATE DATABASE and USE <name> into the file, so restoring onto a site whose
MYSQL_DATABASE differs loaded every row into the dump's schema, left the live
database untouched, and reported success. New sets are dumped without it and
load into the schema named at restore time. Existing sets are still bound, so
the gate reads which schema they name and refuses a mismatch instead of
restoring beside the live data. Also dropped --no-tablespaces into the main
dump: it needs the PROCESS privilege the app user does not have on MySQL 8.

The gate's own test carried a copy of the gate. test-restore-datarev.sh
re-implemented the predicate in its own shell functions, so it would have stayed
green through any drift in restore.sh - including the plugin-chain hole fixed in
the previous commit, which it could never have caught. restore.sh now takes
RESTORE_CHECK_ONLY, which stops it immediately after the gates having touched
nothing, plus overridable paths for the migration and plugin trees. The test
drives the shipped script and asserts on its exit code, so the verdict under
test IS the verdict that ships. Confirmed non-vacuous by planting a violation:
disabling the plugin revision check turns the suite red.

Nine cases now, up from five, covering both plugin-chain directions, a plugin
the build does not have, and the schema binding.

Moving the schema check earlier was prompted by that test: it had been placed
after the safety copy, so it needed a live database to reach a verdict it can
reach for free. It sits with the other gates now and stops before anything is
written.

Round-tripped against a real stack after the dump format change: backup, mutate,
restore, data back to backup-time state, and the new dumps confirmed to carry no
schema name.

Suite 2607 passed / 9 skipped.

Scope: product
This commit is contained in:
cproudlock
2026-09-06 18:31:37 -04:00
parent e2705d732a
commit a46058a4dd
6 changed files with 216 additions and 58 deletions

View File

@@ -21,8 +21,22 @@ set -euo pipefail
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
SETNAME="${1:-}"
# Overridable so the gate can be exercised outside a container against throwaway
# migration trees. In the image these are exactly the mounted paths, so the
# shipped behaviour is unchanged; test-restore-datarev.sh drives THIS script
# rather than a copy of its logic, which is the only way the test can notice
# this script drifting.
BACKUP_LIB_DIR="${BACKUP_LIB_DIR:-/opt/backup}"
APP_MIGRATIONS_DIR="${APP_MIGRATIONS_DIR:-/app/migrations}"
APP_PLUGINS_DIR="${APP_PLUGINS_DIR:-/app/plugins}"
APP_INSTANCE_DIR="${APP_INSTANCE_DIR:-/app/instance}"
# When set, stop after the compatibility gate and report the verdict as the exit
# code, touching nothing. This is what makes the gate testable.
RESTORE_CHECK_ONLY="${RESTORE_CHECK_ONLY:-}"
# shellcheck source=scripts/backup/status.sh
. /opt/backup/status.sh
. "${BACKUP_LIB_DIR}/status.sh"
log() { echo "[restore] $*"; }
@@ -91,11 +105,12 @@ if [ -z "${SET_DATAREV}" ]; then
die "cannot determine the set's data revision - refusing an unverified restore"
fi
# Does the running code know this revision at all? If a migration file with that
# revision id exists in the core chain, the code is at-or-past it (the head is
# the tip of the same chain), so a post-load upgrade can carry the data forward.
# If it is absent, the dump is NEWER than the code - refuse.
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" "${APP_MIGRATIONS_DIR}/versions" 2>/dev/null; then
log "set datarev : ${SET_DATAREV}"
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
fi
@@ -128,7 +143,7 @@ if [ -z "${SET_DATAPLUGINS}" ]; then
else
while read -r plugin rev; do
[ -n "${plugin}" ] || continue
chain="/app/plugins/${plugin}/migrations/versions"
chain="${APP_PLUGINS_DIR}/${plugin}/migrations/versions"
if [ ! -d "${chain}" ]; then
die "the backup carries data for plugin '${plugin}', which this build does not have - deploy a build that includes it before restoring"
fi
@@ -142,6 +157,31 @@ ${SET_DATAPLUGINS}
PLUGINREVS
fi
# Last gate: which schema does this dump target? A dump taken with --databases
# carries CREATE DATABASE and USE <name>, pinning it to the schema it came from.
# Loaded into a site whose MYSQL_DATABASE differs, the rows land in the dump's
# schema, the live database is untouched, and the restore reports success. New
# sets are dumped without it and load into whatever schema is named here; older
# sets are still bound, so refuse rather than restore beside the live data.
# Checked HERE, with the other gates, so it costs nothing and stops before the
# safety copy rather than after it.
DUMP_SCHEMA="$(gunzip -c "${DBDUMP}" \
| grep -m1 -oE '^USE `[^`]+`' \
| sed -E 's/^USE `(.*)`/\1/' || true)"
if [ -n "${DUMP_SCHEMA}" ]; then
if [ "${DUMP_SCHEMA}" != "${MYSQL_DATABASE}" ]; then
die "this set is bound to schema '${DUMP_SCHEMA}' but this site's database is '${MYSQL_DATABASE}' - restoring it would load the data beside the live database and report success"
fi
log "legacy schema-bound dump for '${DUMP_SCHEMA}' - matches this site"
fi
if [ -n "${RESTORE_CHECK_ONLY}" ]; then
log "check-only: every gate passed, stopping before anything destructive"
RESTORE_LOADED=false
trap - EXIT
exit 0
fi
# Safety set FIRST. Everything below this point is destructive and none of it
# is transactional: the dump carries DROP TABLE per table, and the instance tree
# is replaced wholesale. Without a copy taken here, a restore of the wrong set -
@@ -162,7 +202,7 @@ mysqldump \
--routines \
--triggers \
--no-tablespaces \
--databases "${MYSQL_DATABASE}" \
"${MYSQL_DATABASE}" \
| gzip -c > "${SAFETY_DIR}/db.sql.gz" \
|| die "pre-restore dump failed - refusing to touch the live database"
@@ -182,7 +222,8 @@ restoring set '${SETNAME}' at ${SAFETY_STAMP}.
To go back to this state, load the dump the same way a restore does:
gunzip -c db.sql.gz | mysql --host=... --user=... --password=...
gunzip -c db.sql.gz | mysql --host=... --user=... --password=... \
--database=${MYSQL_DATABASE}
tar -xzf instance.tar.gz -C /app
This is a local copy on the backups volume, taken for reversibility of one
@@ -223,11 +264,14 @@ if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
fi
status_write restore database "${SETNAME}" "Loading database dump"
log "loading database dump"
log "loading database dump into ${MYSQL_DATABASE}"
gunzip -c "${DBDUMP}" | mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--database="${MYSQL_DATABASE}" \
|| die "database load failed"
# Hand off to the api container for the schema upgrade. The dump may sit at an