Gate restore on the dump's data revision, not the code head

A restore could load a database dump whose schema was older than the running
code. The manifest stamped schemarev from the code image on disk, so the
compatibility check compared the code against itself and always passed. A set
captured from a database at 7d37_import_batches_and_journal restored under
code at 7d41_drop_guest_tools_secrets, leaving the data behind the code: every
query on a table the newer migrations touched failed with 1054 Unknown column,
which surfaced as empty configuration-policy dropdowns and a failure to load
data on edit PC.

The authoritative signal is the alembic_version of the database that was
dumped, so manifests now carry datarev alongside schemarev, backup.sh reads it
from the database and warns when it has drifted from the image, and the gate
asks whether the running code knows that revision. Data behind the code is
allowed because it can be rolled forward; data ahead of the code is refused
because it cannot.

Rolling it forward has to happen somewhere with an app context. The backup
sidecar is mysql plus restic plus alembic only and has no Flask, so restore.sh
loads the dump and parks the run in an upgrading phase behind an
.upgrade-requested sentinel rather than declaring a success it cannot verify.
The api container owns the Flask CLI and already runs these exact idempotent
commands on every deploy, so a finalize endpoint there runs db upgrade plus
plugin upgrade-all and stamps the terminal phase. The UI drives it on seeing
the upgrading phase, so recovery needs no operator intervention.

The live simulation runs one codebase against itself, so its dump is always at
the same revision as the code restoring it and it can never reproduce this
drift. The gate therefore gets a standalone tester that builds throwaway
migration chains and dumps offline, invoked as a simulation preflight. It
caught a real bug in the legacy dump-parse fallback, where the revision was
matched loosely enough to return the alembic_version table name instead of the
value it holds.
This commit is contained in:
benjamin.baker
2026-09-04 15:38:31 -04:00
parent 3c9321f16c
commit b973561cb7
7 changed files with 450 additions and 28 deletions

View File

@@ -60,15 +60,44 @@ mysqldump \
log "database dump written ($(du -h "${SETDIR}/db.sql.gz" | cut -f1))"
# 2. Manifest: the schema stamp that makes a restore a checked operation.
# Capture the revision actually applied to the DATABASE (its alembic_version),
# not just the code head. A restore must gate on the DATA revision; stamping
# only the code head hides an image that has outrun its own database and lets a
# behind-schema dump restore under ahead-schema code (1054 Unknown column on
# every affected table). Query it straight from the dumped DB.
log "reading applied database revision (alembic_version)"
DATAREV="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT version_num FROM alembic_version LIMIT 1;" \
"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
[ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty"
log "writing manifest"
python3 /opt/backup/manifest.py \
--out "${SETDIR}/manifest.json" \
--takenat "${STAMP}" \
--site "${SITE}" \
--imagetag "${SHOPDB_IMAGE_TAG:-source}" \
--datarev "${DATAREV}" \
--migrations /app/migrations \
--plugins /app/plugins
# Consistency guard: if the DATA revision does not match the CODE head, this
# backup captures a database the running image has outrun. It is still stored
# (losing it would be worse) but flagged loudly so a restore of it is a
# deliberate choice, not a surprise.
CODEREV="$(python3 -c \
'import json,sys;print(",".join(sorted(json.load(open(sys.argv[1]))["schemarev"]["core"])))' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -n "${DATAREV}" ] && [ "${DATAREV}" != "${CODEREV}" ]; then
log "WARN schema drift: database is at '${DATAREV}' but code head is '${CODEREV}'"
log "WARN this DB has not run 'flask db upgrade' to the running image; restore will upgrade it"
history_append backup warning "${SETNAME}" "Schema drift: data ${DATAREV} vs code ${CODEREV}"
fi
# 3. restic snapshot: db dump + instance tree + manifest, one point in time.
# Skips gracefully when restic/repo are not configured (local-only sites).
IS_CHECK_PASSED=false