fa6b99bf91
Closes the three defects in the imported restore path, at Ben's request. Each was proven against a real stack before and after: two backup sets with different data and different instance files, then a restore of the OLDER one, which is the case that was broken. A restore no longer depends on a tab staying open. The backup sidecar has no Flask, so it loads the dump and leaves a sentinel for the api container to run the migrations. The only thing that ever acted on that sentinel was the poll loop of the modal that queued the restore, so closing the tab - or letting a token expire - during the minutes a load takes left the data at the old revision under new code, every query on a newer column failing, and nothing retrying. A watcher in the api container now claims and runs it. The endpoint no longer runs the upgrade at all; it only makes sure something is pending, which is also how a failed upgrade is retried. That takes Alembic out of the request path, where it could outrun the gunicorn worker timeout and be killed mid-chain, leaving a half-applied revision the next attempt trips over. The claim is a rename, because rename(2) is atomic: several workers and a browser poll can race the same sentinel and exactly one wins. A failed upgrade parks itself as .upgrade-failed rather than re-arming, since putting it back would rerun the same broken chain on every poll forever. Instance files come from the set being restored. The restic call asked for 'latest' regardless of which set was named, so restoring anything but the newest paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json - a mismatch nothing downstream could detect. Now scoped by the set's own tag, which backup.sh has always written. The tree is replaced rather than restored over, so files created after the snapshot are gone afterwards: a restore has to put the site back to the snapshot, not merge into it. Done as a staged swap rather than 'restic restore --delete --target /', because that aims a delete pass at the container root and the cost of getting its filter wrong is the whole filesystem. Nothing destructive happens before a copy is taken. The dump carries DROP TABLE per table with no transaction around it, and the instance tree is replaced wholesale, so a wrong set or a failure part-way through left nothing to go back to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball and a README saying what it is and how to load it - and refuses to touch anything if that copy cannot be written. Two more found while proving the above, both of which stopped the overlay working at all: * Every script under scripts/backup/ was committed non-executable, while every other .sh in the repo is 755. The compose entrypoints are /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup and watcher containers could not start from a clean checkout on any machine, and nothing could fix it at runtime. * Dockerfile.backup could not build without the corporate CA staged. A plain COPY of the gitignored cert fails outright; the trailing-glob form does too, because BuildKit fails a COPY whose pattern matches nothing - which is exactly the case being handled. Copying the directory and testing for the file inside it is the form that works both ways. Verified by building the image with no cert present. Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic claim, the parked failure and the endpoint no longer running the upgrade. Scope: product
206 lines
9.3 KiB
Bash
Executable File
206 lines
9.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ShopDB restore.
|
|
#
|
|
# The DANGEROUS half of the pair, so it is deliberately guarded. A restore is
|
|
# only safe onto code whose schema is at least the set's - otherwise restored
|
|
# rows reference columns the live schema has not applied yet and every query on
|
|
# those tables 500s. This script refuses an incompatible restore rather than
|
|
# leaving the site half-broken.
|
|
#
|
|
# Usage (from the backup container):
|
|
# docker compose -f docker-compose.yml -f docker-compose.backup.yml \
|
|
# run --rm --entrypoint /opt/backup/restore.sh backup <snapshot-set-name>
|
|
#
|
|
# It does NOT roll back code - that is git's job (git checkout <tag> && build).
|
|
# Restore code FIRST, bring the schema up (flask db upgrade + plugin
|
|
# upgrade-all), THEN run this to load the data.
|
|
#
|
|
# ASCII only, no emojis - matches the repo style gate.
|
|
set -euo pipefail
|
|
|
|
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
|
|
SETNAME="${1:-}"
|
|
|
|
# shellcheck source=scripts/backup/status.sh
|
|
. /opt/backup/status.sh
|
|
|
|
log() { echo "[restore] $*"; }
|
|
|
|
# On any exit, stamp the terminal phase and append a history record so the UI
|
|
# reflects success or failure even when a step aborts early (set -e). A restore
|
|
# is only "done" once the DATA has been upgraded to the running CODE, and that
|
|
# upgrade runs in the Flask-capable api container, not here (this sidecar has no
|
|
# app). So the happy path ends by handing off to the api: this script stamps an
|
|
# 'upgrading' phase and drops an .upgrade-requested sentinel, and the api runs
|
|
# 'flask db upgrade' + 'flask plugin upgrade-all' and writes the final 'done'.
|
|
RESTORE_LOADED=false
|
|
_on_exit() {
|
|
code=$?
|
|
if [ "${RESTORE_LOADED}" = true ]; then
|
|
# Data is loaded; the api takes it from 'upgrading' to 'done'. Do not
|
|
# stamp success here or the UI would call the restore complete before
|
|
# the schema is actually brought up to code.
|
|
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
|
|
else
|
|
status_write restore failed "${SETNAME}" "Restore failed (exit ${code})"
|
|
history_append restore failed "${SETNAME}" "Restore failed (exit ${code})"
|
|
fi
|
|
}
|
|
trap _on_exit EXIT
|
|
|
|
die() { echo "[restore] ERROR: $*" >&2; exit 1; }
|
|
|
|
[ -n "${SETNAME}" ] || die "usage: restore.sh <snapshot-set-name>"
|
|
|
|
status_write restore checking "${SETNAME}" "Checking schema compatibility"
|
|
|
|
SETDIR="${BACKUPS_ROOT}/${SETNAME}"
|
|
DBDUMP="${SETDIR}/db.sql.gz"
|
|
|
|
# Prefer the local fast path; fall back to pulling the set out of restic.
|
|
if [ ! -f "${DBDUMP}" ]; then
|
|
log "local dump not found; restoring set from restic"
|
|
command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
|
|
restic restore latest --tag "${SETNAME}" --target / --include "${SETDIR}" \
|
|
|| die "restic restore of ${SETNAME} failed"
|
|
fi
|
|
[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
|
|
|
|
# Compatibility gate: the restore must gate on the revision the DUMP'S DATA is
|
|
# at (datarev), not on a code-to-code comparison. Loading data that sits at an
|
|
# OLDER revision than the running code is fine - we upgrade it afterwards - but
|
|
# loading data NEWER than the code (a revision the code has never heard of)
|
|
# would leave rows referencing columns the code cannot migrate to, so that is
|
|
# refused.
|
|
log "checking schema compatibility"
|
|
|
|
# The set's data revision: prefer the manifest field; for older manifests that
|
|
# predate datarev, parse the real alembic_version straight out of the dump so a
|
|
# legacy set is still gated on its true data, not on a code-derived stamp.
|
|
SET_DATAREV="$(python3 -c \
|
|
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
|
|
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
|
|
if [ -z "${SET_DATAREV}" ]; then
|
|
log "manifest has no datarev (legacy set) - reading it from the dump"
|
|
SET_DATAREV="$(gunzip -c "${DBDUMP}" \
|
|
| grep -A2 -i 'INSERT INTO `alembic_version`' \
|
|
| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
|
|
fi
|
|
|
|
if [ -z "${SET_DATAREV}" ]; then
|
|
die "cannot determine the set's data revision - refusing an unverified restore"
|
|
fi
|
|
|
|
# Does the running code know this revision at all? If a migration file with that
|
|
# revision id exists in the core chain, the code is at-or-past it (the head is
|
|
# the tip of the same chain), so a post-load upgrade can carry the data forward.
|
|
# If it is absent, the dump is NEWER than the code - refuse.
|
|
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
|
|
log "set datarev : ${SET_DATAREV}"
|
|
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
|
|
fi
|
|
log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
|
|
|
|
# Safety set FIRST. Everything below this point is destructive and none of it
|
|
# is transactional: the dump carries DROP TABLE per table, and the instance tree
|
|
# is replaced wholesale. Without a copy taken here, a restore of the wrong set -
|
|
# or a failure part-way through the load - leaves nothing to go back to. This is
|
|
# a local copy on the same volume, so it is cheap and it is NOT a substitute for
|
|
# a real backup; it exists to make THIS operation reversible.
|
|
SAFETY_STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
|
|
SAFETY_DIR="${BACKUPS_ROOT}/pre-restore-${SAFETY_STAMP}"
|
|
status_write restore safety "${SETNAME}" "Copying the current state aside"
|
|
log "taking a pre-restore safety copy into ${SAFETY_DIR}"
|
|
mkdir -p "${SAFETY_DIR}" || die "could not create ${SAFETY_DIR}"
|
|
|
|
mysqldump \
|
|
--host="${MYSQL_HOST}" \
|
|
--user="${MYSQL_USER}" \
|
|
--password="${MYSQL_PASSWORD}" \
|
|
--single-transaction \
|
|
--routines \
|
|
--triggers \
|
|
--no-tablespaces \
|
|
--databases "${MYSQL_DATABASE}" \
|
|
| gzip -c > "${SAFETY_DIR}/db.sql.gz" \
|
|
|| die "pre-restore dump failed - refusing to touch the live database"
|
|
|
|
if [ -d /app/instance ]; then
|
|
tar -czf "${SAFETY_DIR}/instance.tar.gz" -C /app instance \
|
|
|| die "pre-restore instance copy failed - refusing to touch the live files"
|
|
fi
|
|
|
|
# Named so an operator finding this directory months later knows what it is and
|
|
# how to use it, without reading this script.
|
|
cat > "${SAFETY_DIR}/README.txt" <<SAFETYNOTE
|
|
Automatic pre-restore copy of the state this site was in immediately before
|
|
restoring set '${SETNAME}' at ${SAFETY_STAMP}.
|
|
|
|
db.sql.gz mysqldump of ${MYSQL_DATABASE} as it was
|
|
instance.tar.gz /app/instance as it was (absent if there was no such tree)
|
|
|
|
To go back to this state, load the dump the same way a restore does:
|
|
|
|
gunzip -c db.sql.gz | mysql --host=... --user=... --password=...
|
|
tar -xzf instance.tar.gz -C /app
|
|
|
|
This is a local copy on the backups volume, taken for reversibility of one
|
|
restore. It is not a backup set: it has no manifest and no restic snapshot, and
|
|
nothing prunes it.
|
|
SAFETYNOTE
|
|
log "safety copy written ($(du -sh "${SAFETY_DIR}" | cut -f1))"
|
|
|
|
# Files first, then DB, so the two land together.
|
|
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
|
|
status_write restore files "${SETNAME}" "Restoring instance files"
|
|
# Scoped to THIS set's snapshot. A bare 'latest' restores whatever was
|
|
# snapshotted most recently, so restoring anything other than the newest set
|
|
# paired that set's DATABASE with the NEWEST snapshot's uploads and
|
|
# plugins.json - a mismatch nothing downstream could detect.
|
|
log "restoring instance files from the ${SETNAME} snapshot"
|
|
STAGE="$(mktemp -d)"
|
|
restic restore latest --tag "${SETNAME}" --target "${STAGE}" \
|
|
--include /app/instance \
|
|
|| die "restic restore of instance files for ${SETNAME} failed"
|
|
|
|
# Replace the CONTENTS of the instance tree rather than restoring over it,
|
|
# so files created after the snapshot are gone afterwards - a restore has to
|
|
# put the site back to the snapshot, not merge into it. Done as an explicit
|
|
# staged swap rather than 'restic restore --delete --target /', because that
|
|
# aims a delete pass at the container root and the blast radius of getting
|
|
# its filter wrong is the whole filesystem. /app/instance is a mount, so the
|
|
# directory itself has to survive; only what is inside it is replaced.
|
|
if [ -d "${STAGE}/app/instance" ]; then
|
|
find /app/instance -mindepth 1 -delete \
|
|
|| die "could not clear the instance tree before restoring it"
|
|
cp -a "${STAGE}/app/instance/." /app/instance/ \
|
|
|| die "could not lay down the restored instance tree"
|
|
else
|
|
log "WARN snapshot for ${SETNAME} carried no /app/instance - leaving files alone"
|
|
fi
|
|
rm -rf "${STAGE}"
|
|
fi
|
|
|
|
status_write restore database "${SETNAME}" "Loading database dump"
|
|
log "loading database dump"
|
|
gunzip -c "${DBDUMP}" | mysql \
|
|
--host="${MYSQL_HOST}" \
|
|
--user="${MYSQL_USER}" \
|
|
--password="${MYSQL_PASSWORD}" \
|
|
|| die "database load failed"
|
|
|
|
# Hand off to the api container for the schema upgrade. The dump may sit at an
|
|
# older revision than this image (allowed and expected); without the upgrade the
|
|
# data stays behind the code and every query on a table the newer migrations
|
|
# changed 500s with 1054 Unknown column. This sidecar has no Flask, so it cannot
|
|
# run 'flask db upgrade' itself - it records the intent and lets the api (which
|
|
# owns the app context and already runs these exact idempotent commands on every
|
|
# deploy) finalize. The api clears .upgrade-requested and stamps the final
|
|
# 'done' once the upgrade succeeds.
|
|
status_write restore upgrading "${SETNAME}" "Upgrading restored schema to code"
|
|
printf '%s' "${SETNAME}" > "${BACKUPS_ROOT}/.upgrade-requested"
|
|
log "database loaded at revision ${SET_DATAREV}; handed off to api for schema upgrade"
|
|
|
|
rm -f "${BACKUPS_ROOT}/.restore-requested" 2>/dev/null || true
|
|
RESTORE_LOADED=true
|