Finish a restore without a browser, from the set that was asked for

Closes the three defects in the imported restore path, at Ben's request. Each
was proven against a real stack before and after: two backup sets with different
data and different instance files, then a restore of the OLDER one, which is the
case that was broken.

A restore no longer depends on a tab staying open. The backup sidecar has no
Flask, so it loads the dump and leaves a sentinel for the api container to run
the migrations. The only thing that ever acted on that sentinel was the poll
loop of the modal that queued the restore, so closing the tab - or letting a
token expire - during the minutes a load takes left the data at the old revision
under new code, every query on a newer column failing, and nothing retrying. A
watcher in the api container now claims and runs it. The endpoint no longer runs
the upgrade at all; it only makes sure something is pending, which is also how a
failed upgrade is retried. That takes Alembic out of the request path, where it
could outrun the gunicorn worker timeout and be killed mid-chain, leaving a
half-applied revision the next attempt trips over.

The claim is a rename, because rename(2) is atomic: several workers and a
browser poll can race the same sentinel and exactly one wins. A failed upgrade
parks itself as .upgrade-failed rather than re-arming, since putting it back
would rerun the same broken chain on every poll forever.

Instance files come from the set being restored. The restic call asked for
'latest' regardless of which set was named, so restoring anything but the newest
paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json -
a mismatch nothing downstream could detect. Now scoped by the set's own tag,
which backup.sh has always written. The tree is replaced rather than restored
over, so files created after the snapshot are gone afterwards: a restore has to
put the site back to the snapshot, not merge into it. Done as a staged swap
rather than 'restic restore --delete --target /', because that aims a delete
pass at the container root and the cost of getting its filter wrong is the whole
filesystem.

Nothing destructive happens before a copy is taken. The dump carries DROP TABLE
per table with no transaction around it, and the instance tree is replaced
wholesale, so a wrong set or a failure part-way through left nothing to go back
to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball
and a README saying what it is and how to load it - and refuses to touch
anything if that copy cannot be written.

Two more found while proving the above, both of which stopped the overlay
working at all:

* Every script under scripts/backup/ was committed non-executable, while every
  other .sh in the repo is 755. The compose entrypoints are
  /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup
  and watcher containers could not start from a clean checkout on any machine,
  and nothing could fix it at runtime.

* Dockerfile.backup could not build without the corporate CA staged. A plain
  COPY of the gitignored cert fails outright; the trailing-glob form does too,
  because BuildKit fails a COPY whose pattern matches nothing - which is exactly
  the case being handled. Copying the directory and testing for the file inside
  it is the form that works both ways. Verified by building the image with no
  cert present.

Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic
claim, the parked failure and the endpoint no longer running the upgrade.

Scope: product

scripts/backup/Dockerfile.backup

@@ -15,13 +15,22 @@ FROM mysql:8.4
 # needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
 # and must be staged at scripts/backup/zscaler-root-ca.pem before building.
 #
-# The trailing * makes the COPY optional: a plain COPY of a gitignored file
-# fails the build outright, so nobody outside a TLS-intercepting network -
-# another site, or our own CI - could build this image at all. Absent cert,
-# the glob matches nothing, the layer is a no-op and the fetches below go out
-# over the public trust store.
-COPY scripts/backup/zscaler-root-ca.pem* /etc/pki/ca-trust/source/anchors/
-RUN update-ca-trust
+# Copied via the directory rather than by name, because the cert is OPTIONAL: it
+# is gitignored, so a plain `COPY scripts/backup/zscaler-root-ca.pem` fails the
+# build outright and nobody outside a TLS-intercepting network - another site,
+# or our own CI - could build this image at all. A trailing glob does not fix
+# that either: BuildKit fails a COPY whose pattern matches nothing, which is
+# precisely the case being handled. Copying a directory that always exists and
+# testing for the file inside it is the form that works both ways.
+COPY scripts/backup/ /tmp/buildctx/
+RUN if [ -f /tmp/buildctx/zscaler-root-ca.pem ]; then \
+		echo "staging corporate root CA"; \
+		cp /tmp/buildctx/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/; \
+		update-ca-trust; \
+	else \
+		echo "no corporate root CA staged - using the public trust store"; \
+	fi \
+	&& rm -rf /tmp/buildctx
 
 # restic from the distro is often stale; pull the pinned release binary so the
 # snapshot format and retention flags are predictable across sites.

scripts/backup/restore.sh

@@ -61,7 +61,7 @@ DBDUMP="${SETDIR}/db.sql.gz"
 if [ ! -f "${DBDUMP}" ]; then
 	log "local dump not found; restoring set from restic"
 	command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
-	restic restore latest --target / --include "${SETDIR}" \
+	restic restore latest --tag "${SETNAME}" --target / --include "${SETDIR}" \
 		|| die "restic restore of ${SETNAME} failed"
 fi
 [ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
@@ -101,12 +101,84 @@ if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2
 fi
 log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
 
+# Safety set FIRST. Everything below this point is destructive and none of it
+# is transactional: the dump carries DROP TABLE per table, and the instance tree
+# is replaced wholesale. Without a copy taken here, a restore of the wrong set -
+# or a failure part-way through the load - leaves nothing to go back to. This is
+# a local copy on the same volume, so it is cheap and it is NOT a substitute for
+# a real backup; it exists to make THIS operation reversible.
+SAFETY_STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
+SAFETY_DIR="${BACKUPS_ROOT}/pre-restore-${SAFETY_STAMP}"
+status_write restore safety "${SETNAME}" "Copying the current state aside"
+log "taking a pre-restore safety copy into ${SAFETY_DIR}"
+mkdir -p "${SAFETY_DIR}" || die "could not create ${SAFETY_DIR}"
+
+mysqldump \
+	--host="${MYSQL_HOST}" \
+	--user="${MYSQL_USER}" \
+	--password="${MYSQL_PASSWORD}" \
+	--single-transaction \
+	--routines \
+	--triggers \
+	--no-tablespaces \
+	--databases "${MYSQL_DATABASE}" \
+	| gzip -c > "${SAFETY_DIR}/db.sql.gz" \
+	|| die "pre-restore dump failed - refusing to touch the live database"
+
+if [ -d /app/instance ]; then
+	tar -czf "${SAFETY_DIR}/instance.tar.gz" -C /app instance \
+		|| die "pre-restore instance copy failed - refusing to touch the live files"
+fi
+
+# Named so an operator finding this directory months later knows what it is and
+# how to use it, without reading this script.
+cat > "${SAFETY_DIR}/README.txt" <<SAFETYNOTE
+Automatic pre-restore copy of the state this site was in immediately before
+restoring set '${SETNAME}' at ${SAFETY_STAMP}.
+
+  db.sql.gz         mysqldump of ${MYSQL_DATABASE} as it was
+  instance.tar.gz   /app/instance as it was (absent if there was no such tree)
+
+To go back to this state, load the dump the same way a restore does:
+
+  gunzip -c db.sql.gz | mysql --host=... --user=... --password=...
+  tar -xzf instance.tar.gz -C /app
+
+This is a local copy on the backups volume, taken for reversibility of one
+restore. It is not a backup set: it has no manifest and no restic snapshot, and
+nothing prunes it.
+SAFETYNOTE
+log "safety copy written ($(du -sh "${SAFETY_DIR}" | cut -f1))"
+
 # Files first, then DB, so the two land together.
 if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
 	status_write restore files "${SETNAME}" "Restoring instance files"
-	log "restoring instance files from restic"
-	restic restore latest --target / --include /app/instance \
-		|| die "restic restore of instance files failed"
+	# Scoped to THIS set's snapshot. A bare 'latest' restores whatever was
+	# snapshotted most recently, so restoring anything other than the newest set
+	# paired that set's DATABASE with the NEWEST snapshot's uploads and
+	# plugins.json - a mismatch nothing downstream could detect.
+	log "restoring instance files from the ${SETNAME} snapshot"
+	STAGE="$(mktemp -d)"
+	restic restore latest --tag "${SETNAME}" --target "${STAGE}" \
+		--include /app/instance \
+		|| die "restic restore of instance files for ${SETNAME} failed"
+
+	# Replace the CONTENTS of the instance tree rather than restoring over it,
+	# so files created after the snapshot are gone afterwards - a restore has to
+	# put the site back to the snapshot, not merge into it. Done as an explicit
+	# staged swap rather than 'restic restore --delete --target /', because that
+	# aims a delete pass at the container root and the blast radius of getting
+	# its filter wrong is the whole filesystem. /app/instance is a mount, so the
+	# directory itself has to survive; only what is inside it is replaced.
+	if [ -d "${STAGE}/app/instance" ]; then
+		find /app/instance -mindepth 1 -delete \
+			|| die "could not clear the instance tree before restoring it"
+		cp -a "${STAGE}/app/instance/." /app/instance/ \
+			|| die "could not lay down the restored instance tree"
+	else
+		log "WARN snapshot for ${SETNAME} carried no /app/instance - leaving files alone"
+	fi
+	rm -rf "${STAGE}"
 fi
 
 status_write restore database "${SETNAME}" "Loading database dump"