fa6b99bf91
Closes the three defects in the imported restore path, at Ben's request. Each was proven against a real stack before and after: two backup sets with different data and different instance files, then a restore of the OLDER one, which is the case that was broken. A restore no longer depends on a tab staying open. The backup sidecar has no Flask, so it loads the dump and leaves a sentinel for the api container to run the migrations. The only thing that ever acted on that sentinel was the poll loop of the modal that queued the restore, so closing the tab - or letting a token expire - during the minutes a load takes left the data at the old revision under new code, every query on a newer column failing, and nothing retrying. A watcher in the api container now claims and runs it. The endpoint no longer runs the upgrade at all; it only makes sure something is pending, which is also how a failed upgrade is retried. That takes Alembic out of the request path, where it could outrun the gunicorn worker timeout and be killed mid-chain, leaving a half-applied revision the next attempt trips over. The claim is a rename, because rename(2) is atomic: several workers and a browser poll can race the same sentinel and exactly one wins. A failed upgrade parks itself as .upgrade-failed rather than re-arming, since putting it back would rerun the same broken chain on every poll forever. Instance files come from the set being restored. The restic call asked for 'latest' regardless of which set was named, so restoring anything but the newest paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json - a mismatch nothing downstream could detect. Now scoped by the set's own tag, which backup.sh has always written. The tree is replaced rather than restored over, so files created after the snapshot are gone afterwards: a restore has to put the site back to the snapshot, not merge into it. Done as a staged swap rather than 'restic restore --delete --target /', because that aims a delete pass at the container root and the cost of getting its filter wrong is the whole filesystem. Nothing destructive happens before a copy is taken. The dump carries DROP TABLE per table with no transaction around it, and the instance tree is replaced wholesale, so a wrong set or a failure part-way through left nothing to go back to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball and a README saying what it is and how to load it - and refuses to touch anything if that copy cannot be written. Two more found while proving the above, both of which stopped the overlay working at all: * Every script under scripts/backup/ was committed non-executable, while every other .sh in the repo is 755. The compose entrypoints are /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup and watcher containers could not start from a clean checkout on any machine, and nothing could fix it at runtime. * Dockerfile.backup could not build without the corporate CA staged. A plain COPY of the gitignored cert fails outright; the trailing-glob form does too, because BuildKit fails a COPY whose pattern matches nothing - which is exactly the case being handled. Copying the directory and testing for the file inside it is the form that works both ways. Verified by building the image with no cert present. Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic claim, the parked failure and the endpoint no longer running the upgrade. Scope: product
62 lines
2.5 KiB
Bash
Executable File
62 lines
2.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# ShopDB backup WATCHER - the resident half of the in-app backup utility.
|
|
#
|
|
# WHY this exists: the api container cannot run docker, so when an operator
|
|
# clicks "Back up now" the api only drops a sentinel file (.backup-requested)
|
|
# on the shared backups volume. This watcher is the thing that actuates that
|
|
# request: it polls for the sentinel and runs the one-shot backup.sh, which
|
|
# ships a restic snapshot to the backup-store container and clears the sentinel
|
|
# on success. That turns the UI button into a real, self-serve backup without
|
|
# giving the api container any docker or host privileges.
|
|
#
|
|
# ASCII only, no emojis - matches the repo style gate.
|
|
set -uo pipefail
|
|
|
|
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
|
|
SENTINEL="${BACKUPS_ROOT}/.backup-requested"
|
|
RESTORE_SENTINEL="${BACKUPS_ROOT}/.restore-requested"
|
|
INTERVAL="${BACKUP_POLL_SECONDS:-10}"
|
|
|
|
log() { echo "[backup-watch] $*"; }
|
|
|
|
# shellcheck source=scripts/backup/status.sh
|
|
. /opt/backup/status.sh
|
|
|
|
mkdir -p "${BACKUPS_ROOT}"
|
|
# The api container runs unprivileged and only drops sentinel files here; it
|
|
# cannot write into a root-owned 0755 directory. Make the backups root
|
|
# writable so "Back up now" and "Restore" can queue their sentinels.
|
|
chmod 0777 "${BACKUPS_ROOT}" 2>/dev/null || true
|
|
log "watcher up; polling ${SENTINEL} and ${RESTORE_SENTINEL} every ${INTERVAL}s"
|
|
|
|
while true; do
|
|
if [ -f "${SENTINEL}" ]; then
|
|
log "backup requested - running one-shot"
|
|
status_write backup queued "" "Backup queued"
|
|
if /opt/backup/backup.sh; then
|
|
log "backup completed"
|
|
else
|
|
# backup.sh only clears the sentinel on success, so a failed run
|
|
# would otherwise loop forever. Clear it here and wait for the next
|
|
# explicit request rather than hammering a broken backup.
|
|
log "backup FAILED - clearing request, awaiting next"
|
|
rm -f "${SENTINEL}" 2>/dev/null || true
|
|
fi
|
|
fi
|
|
if [ -f "${RESTORE_SENTINEL}" ]; then
|
|
# The api drops the target set name into the restore sentinel. Read it,
|
|
# hand it to restore.sh, and clear the request whether it succeeds or
|
|
# fails so a bad set cannot pin the watcher in a tight retry loop.
|
|
setname="$(head -n1 "${RESTORE_SENTINEL}" 2>/dev/null | tr -d '\r\n')"
|
|
log "restore requested - running restore for ${setname}"
|
|
status_write restore queued "${setname}" "Restore queued"
|
|
if /opt/backup/restore.sh "${setname}"; then
|
|
log "restore completed"
|
|
else
|
|
log "restore FAILED - clearing request, awaiting next"
|
|
fi
|
|
rm -f "${RESTORE_SENTINEL}" 2>/dev/null || true
|
|
fi
|
|
sleep "${INTERVAL}"
|
|
done
|