ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Gate restore on the dump's data revision, not the code head

Browse Source 
A restore could load a database dump whose schema was older than the running
code. The manifest stamped schemarev from the code image on disk, so the
compatibility check compared the code against itself and always passed. A set
captured from a database at 7d37_import_batches_and_journal restored under
code at 7d41_drop_guest_tools_secrets, leaving the data behind the code: every
query on a table the newer migrations touched failed with 1054 Unknown column,
which surfaced as empty configuration-policy dropdowns and a failure to load
data on edit PC.

The authoritative signal is the alembic_version of the database that was
dumped, so manifests now carry datarev alongside schemarev, backup.sh reads it
from the database and warns when it has drifted from the image, and the gate
asks whether the running code knows that revision. Data behind the code is
allowed because it can be rolled forward; data ahead of the code is refused
because it cannot.

Rolling it forward has to happen somewhere with an app context. The backup
sidecar is mysql plus restic plus alembic only and has no Flask, so restore.sh
loads the dump and parks the run in an upgrading phase behind an
.upgrade-requested sentinel rather than declaring a success it cannot verify.
The api container owns the Flask CLI and already runs these exact idempotent
commands on every deploy, so a finalize endpoint there runs db upgrade plus
plugin upgrade-all and stamps the terminal phase. The UI drives it on seeing
the upgrading phase, so recovery needs no operator intervention.

The live simulation runs one codebase against itself, so its dump is always at
the same revision as the code restoring it and it can never reproduce this
drift. The gate therefore gets a standalone tester that builds throwaway
migration chains and dumps offline, invoked as a simulation preflight. It
caught a real bug in the legacy dump-parse fallback, where the revision was
matched loosely enough to return the alembic_version table name instead of the
value it holds.
This commit is contained in:
benjamin.baker
2026-09-04 15:38:31 -04:00
parent 3c9321f16c
commit b973561cb7
7 changed files with 450 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
scripts/backup/restore.sh
View File

@@ -27,13 +27,20 @@ SETNAME="${1:-}"
log() { echo "[restore] $*"; }
# On any exit, stamp the terminal phase and append a history record so the UI
# reflects success or failure even when a step aborts early (set -e).
RESTORE_DONE=false
# reflects success or failure even when a step aborts early (set -e). A restore
# is only "done" once the DATA has been upgraded to the running CODE, and that
# upgrade runs in the Flask-capable api container, not here (this sidecar has no
# app). So the happy path ends by handing off to the api: this script stamps an
# 'upgrading' phase and drops an .upgrade-requested sentinel, and the api runs
# 'flask db upgrade' + 'flask plugin upgrade-all' and writes the final 'done'.
RESTORE_LOADED=false
_on_exit() {
code=$?
if [ "${RESTORE_DONE}" = true ]; then
status_write restore done "${SETNAME}" "Restore complete"
history_append restore success "${SETNAME}" "Restore complete"
if [ "${RESTORE_LOADED}" = true ]; then
# Data is loaded; the api takes it from 'upgrading' to 'done'. Do not
# stamp success here or the UI would call the restore complete before
# the schema is actually brought up to code.
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
else
status_write restore failed "${SETNAME}" "Restore failed (exit ${code})"
history_append restore failed "${SETNAME}" "Restore failed (exit ${code})"
@@ -59,26 +66,41 @@ if [ ! -f "${DBDUMP}" ]; then
fi
[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
# Compatibility gate: the running schema must cover the set's stamped heads.
# Compatibility gate: the restore must gate on the revision the DUMP'S DATA is
# at (datarev), not on a code-to-code comparison. Loading data that sits at an
# OLDER revision than the running code is fine - we upgrade it afterwards - but
# loading data NEWER than the code (a revision the code has never heard of)
# would leave rows referencing columns the code cannot migrate to, so that is
# refused.
log "checking schema compatibility"
CURRENT_HEADS="$(python3 /opt/backup/manifest.py \
--out /tmp/current-manifest.json \
--migrations /app/migrations --plugins /app/plugins \
>/dev/null 2>&1 && python3 -c \
'import json;print(",".join(sorted(json.load(open("/tmp/current-manifest.json"))["schemarev"]["core"])))')"
SET_HEADS="$(python3 -c \
'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${SET_HEADS}" ]; then
die "set ${SETNAME} has no readable manifest - refusing an unverified restore"
# The set's data revision: prefer the manifest field; for older manifests that
# predate datarev, parse the real alembic_version straight out of the dump so a
# legacy set is still gated on its true data, not on a code-derived stamp.
SET_DATAREV="$(python3 -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${SET_DATAREV}" ]; then
log "manifest has no datarev (legacy set) - reading it from the dump"
SET_DATAREV="$(gunzip -c "${DBDUMP}" \
| grep -A2 -i 'INSERT INTO `alembic_version`' \
| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
fi
if [ "${CURRENT_HEADS}" != "${SET_HEADS}" ]; then
log "current heads : ${CURRENT_HEADS}"
log "set heads : ${SET_HEADS}"
die "schema mismatch - bring code to the set's revision, run 'flask db upgrade' and 'flask plugin upgrade-all', then retry"
if [ -z "${SET_DATAREV}" ]; then
die "cannot determine the set's data revision - refusing an unverified restore"
fi
# Does the running code know this revision at all? If a migration file with that
# revision id exists in the core chain, the code is at-or-past it (the head is
# the tip of the same chain), so a post-load upgrade can carry the data forward.
# If it is absent, the dump is NEWER than the code - refuse.
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
log "set datarev : ${SET_DATAREV}"
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
fi
log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
# Files first, then DB, so the two land together.
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
status_write restore files "${SETNAME}" "Restoring instance files"
@@ -95,6 +117,17 @@ gunzip -c "${DBDUMP}" | mysql \
--password="${MYSQL_PASSWORD}" \
|| die "database load failed"
# Hand off to the api container for the schema upgrade. The dump may sit at an
# older revision than this image (allowed and expected); without the upgrade the
# data stays behind the code and every query on a table the newer migrations
# changed 500s with 1054 Unknown column. This sidecar has no Flask, so it cannot
# run 'flask db upgrade' itself - it records the intent and lets the api (which
# owns the app context and already runs these exact idempotent commands on every
# deploy) finalize. The api clears .upgrade-requested and stamps the final
# 'done' once the upgrade succeeds.
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
printf '%s' "${SETNAME}" > "${BACKUPS_ROOT}/.upgrade-requested"
log "database loaded at revision ${SET_DATAREV}; handed off to api for schema upgrade"
rm -f "${BACKUPS_ROOT}/.restore-requested" 2>/dev/null || true
RESTORE_DONE=true
log "restore of ${SETNAME} complete"
RESTORE_LOADED=true