Finish a restore without a browser, from the set that was asked for

Closes the three defects in the imported restore path, at Ben's request. Each
was proven against a real stack before and after: two backup sets with different
data and different instance files, then a restore of the OLDER one, which is the
case that was broken.

A restore no longer depends on a tab staying open. The backup sidecar has no
Flask, so it loads the dump and leaves a sentinel for the api container to run
the migrations. The only thing that ever acted on that sentinel was the poll
loop of the modal that queued the restore, so closing the tab - or letting a
token expire - during the minutes a load takes left the data at the old revision
under new code, every query on a newer column failing, and nothing retrying. A
watcher in the api container now claims and runs it. The endpoint no longer runs
the upgrade at all; it only makes sure something is pending, which is also how a
failed upgrade is retried. That takes Alembic out of the request path, where it
could outrun the gunicorn worker timeout and be killed mid-chain, leaving a
half-applied revision the next attempt trips over.

The claim is a rename, because rename(2) is atomic: several workers and a
browser poll can race the same sentinel and exactly one wins. A failed upgrade
parks itself as .upgrade-failed rather than re-arming, since putting it back
would rerun the same broken chain on every poll forever.

Instance files come from the set being restored. The restic call asked for
'latest' regardless of which set was named, so restoring anything but the newest
paired that set's DATABASE with the NEWEST snapshot's uploads and plugins.json -
a mismatch nothing downstream could detect. Now scoped by the set's own tag,
which backup.sh has always written. The tree is replaced rather than restored
over, so files created after the snapshot are gone afterwards: a restore has to
put the site back to the snapshot, not merge into it. Done as a staged swap
rather than 'restic restore --delete --target /', because that aims a delete
pass at the container root and the cost of getting its filter wrong is the whole
filesystem.

Nothing destructive happens before a copy is taken. The dump carries DROP TABLE
per table with no transaction around it, and the instance tree is replaced
wholesale, so a wrong set or a failure part-way through left nothing to go back
to. restore.sh now writes a pre-restore-<stamp>/ set - dump, instance tarball
and a README saying what it is and how to load it - and refuses to touch
anything if that copy cannot be written.

Two more found while proving the above, both of which stopped the overlay
working at all:

* Every script under scripts/backup/ was committed non-executable, while every
  other .sh in the repo is 755. The compose entrypoints are
  /opt/backup/watch.sh and backup.sh from a read-only bind mount, so the backup
  and watcher containers could not start from a clean checkout on any machine,
  and nothing could fix it at runtime.

* Dockerfile.backup could not build without the corporate CA staged. A plain
  COPY of the gitignored cert fails outright; the trailing-glob form does too,
  because BuildKit fails a COPY whose pattern matches nothing - which is exactly
  the case being handled. Copying the directory and testing for the file inside
  it is the form that works both ways. Verified by building the image with no
  cert present.

Suite 2600 passed / 9 skipped. The finalizer has unit tests covering the atomic
claim, the parked failure and the endpoint no longer running the upgrade.

Scope: product

shopdb/core/api/shopdbbackups.py

@@ -17,6 +17,8 @@ load rows that reference a column the live schema has not applied yet.
 
 import json
 import os
+import threading
+import time
 import subprocess
 from datetime import datetime, timezone
 
@@ -182,6 +184,106 @@ def _append_history(kind, outcome, setname, message):
         pass
 
 
+# The post-restore upgrade is claimed by renaming the sentinel. rename(2) is
+# atomic, so exactly one claimant wins even with several gunicorn workers and a
+# browser poll all racing - which matters because the loser would otherwise run
+# Alembic concurrently against the same database.
+UPGRADE_SENTINEL = '.upgrade-requested'
+UPGRADE_CLAIM = '.upgrade-inprogress'
+UPGRADE_FAILED = '.upgrade-failed'
+
+
+def _claim_upgrade():
+    """Take exclusive ownership of a pending upgrade, or return None.
+
+    Returns the set name on success. Losing the race is not an error: it means
+    another worker is already finalizing the same restore.
+    """
+    root = _backups_root()
+    sentinel = os.path.join(root, UPGRADE_SENTINEL)
+    claim = os.path.join(root, UPGRADE_CLAIM)
+    try:
+        os.rename(sentinel, claim)
+    except OSError:
+        return None
+    try:
+        with open(claim, 'r', encoding='utf-8') as handle:
+            return handle.read().strip() or 'unknown'
+    except OSError:
+        return 'unknown'
+
+
+def _finalize_claimed(setname):
+    """Run the upgrade for a claimed restore and record how it ended.
+
+    A failure parks the claim as .upgrade-failed rather than putting it back:
+    re-arming it here would retry a migration that just failed, every poll,
+    forever. The operator re-arms it explicitly through the finalize endpoint.
+    """
+    root = _backups_root()
+    claim = os.path.join(root, UPGRADE_CLAIM)
+    _write_status('restore', 'upgrading', setname,
+                  'Upgrading restored schema to code')
+    try:
+        _run_schema_upgrade()
+    except Exception as error:  # noqa: BLE001 - report any upgrade failure
+        message = 'Schema upgrade failed: ' + str(error)
+        try:
+            os.replace(claim, os.path.join(root, UPGRADE_FAILED))
+        except OSError:
+            pass
+        _write_status('restore', 'failed', setname, message)
+        _append_history('restore', 'failed', setname, message)
+        current_app.logger.error('Post-restore schema upgrade failed: %s', error)
+        return False
+    try:
+        os.remove(claim)
+    except OSError:
+        pass
+    _write_status('restore', 'done', setname, 'Restore complete')
+    _append_history('restore', 'success', setname, 'Restore complete')
+    current_app.logger.info('Restore finalized: %s', setname)
+    return True
+
+
+def start_restore_finalizer(app):
+    """Watch for a restore awaiting its schema upgrade, and run it.
+
+    WHY THIS EXISTS: the restore itself runs in the Flask-less backup sidecar,
+    which loads the dump and drops a sentinel because it cannot run
+    'flask db upgrade'. Until this watcher, the ONLY thing that acted on that
+    sentinel was the browser poll loop of the modal that queued the restore -
+    so closing the tab, or letting a token expire, during the minutes a load
+    takes left the data at the old revision under new code, with every query on
+    a newer column failing and nothing retrying.
+
+    Runs in the api container because that is where the app context and the
+    engine live, and out of band rather than in a request because Alembic can
+    outrun the gunicorn worker timeout and get killed mid-chain.
+    """
+    interval = max(1, int(os.environ.get('SHOPDB_FINALIZE_POLL_SECONDS', '10')))
+
+    def _loop():
+        while True:
+            time.sleep(interval)
+            try:
+                with app.app_context():
+                    if not _backups_configured():
+                        continue
+                    setname = _claim_upgrade()
+                    if setname:
+                        _finalize_claimed(setname)
+            except Exception:  # noqa: BLE001 - a watcher must not die
+                try:
+                    app.logger.exception('Restore finalizer pass failed')
+                except Exception:  # noqa: BLE001
+                    pass
+
+    thread = threading.Thread(target=_loop, name='restore-finalizer', daemon=True)
+    thread.start()
+    return thread
+
+
 def _run_schema_upgrade():
     """Bring the just-restored database up to the running code, in-process.
 
@@ -469,55 +571,53 @@ def restore_backup(snapshotid):
 @shopdbbackups_bp.route('/finalize-restore', methods=['POST'])
 @require_role('admin')
 def finalize_restore():
-    """Run the post-restore schema upgrade in the app context.
+    """Make sure a restore awaiting its schema upgrade is queued to be finalized.
 
-    The restore itself runs in the Flask-less backup sidecar, which loads the
-    dump and then drops an .upgrade-requested sentinel because it cannot run
-    'flask db upgrade' itself. This endpoint is the api-side half: it brings the
-    just-restored DATA up to the running CODE with the exact idempotent commands
-    the deploy path uses, then stamps the final 'done'. Without it a restored
-    dump that sits at an older revision leaves every newer-migration table
-    throwing 1054 Unknown column.
+    This used to RUN the upgrade inline, which made two things true that should
+    not have been: a restore only completed while a browser tab stayed open on
+    it, and the upgrade ran inside a request, where Alembic can outrun the
+    gunicorn worker timeout and be killed part-way through the chain, leaving a
+    half-applied revision the next attempt trips over.
 
-    Idempotent and safe to call when nothing is pending: if there is no
-    .upgrade-requested sentinel it reports there was nothing to finalize.
+    The work now belongs to the finalizer watcher in this container
+    (start_restore_finalizer). This endpoint only ensures something is pending
+    for it to claim, so it is safe to call repeatedly, and it is how an operator
+    retries an upgrade that failed.
     """
     if not _backups_configured():
         return _unavailable_response()
 
     root = _backups_root()
-    sentinel = os.path.join(root, '.upgrade-requested')
-    try:
-        with open(sentinel, 'r', encoding='utf-8') as handle:
-            setname = handle.read().strip()
-    except OSError:
+    sentinel = os.path.join(root, UPGRADE_SENTINEL)
+    claim = os.path.join(root, UPGRADE_CLAIM)
+    failed = os.path.join(root, UPGRADE_FAILED)
+
+    if os.path.exists(claim):
         return success_response(
-            data={'finalized': False},
-            message='No restore is awaiting schema upgrade.')
+            data={'queued': True, 'inprogress': True},
+            message='The schema upgrade is already running.')
 
-    _write_status('restore', 'upgrading', setname,
-                  'Upgrading restored schema to code')
-    try:
-        _run_schema_upgrade()
-    except Exception as error:  # noqa: BLE001 - report any upgrade failure
-        _write_status('restore', 'failed', setname,
-                      'Schema upgrade failed: ' + str(error))
-        _append_history('restore', 'failed', setname,
-                        'Schema upgrade failed: ' + str(error))
-        return error_response(ErrorCodes.INTERNAL_ERROR,
-                              'Post-restore schema upgrade failed: ' + str(error),
-                              http_code=500)
+    if os.path.exists(sentinel):
+        return success_response(
+            data={'queued': True, 'inprogress': False},
+            message='The schema upgrade is queued and will run shortly.')
+
+    # A previous attempt failed and parked itself. Re-arming is the retry.
+    if os.path.exists(failed):
+        try:
+            os.replace(failed, sentinel)
+        except OSError as error:
+            return error_response(ErrorCodes.INTERNAL_ERROR,
+                                  'Could not re-queue the schema upgrade: ' + str(error),
+                                  http_code=500)
+        AuditLog.log('restore-finalize-retried', 'Backup')
+        return success_response(
+            data={'queued': True, 'inprogress': False, 'retried': True},
+            message='The failed schema upgrade has been queued to run again.')
 
-    try:
-        os.remove(sentinel)
-    except OSError:
-        pass
-    _write_status('restore', 'done', setname, 'Restore complete')
-    _append_history('restore', 'success', setname, 'Restore complete')
-    AuditLog.log('restore-finalized', 'Backup', entityname=setname)
     return success_response(
-        data={'finalized': True, 'snapshotid': setname},
-        message='Restore finalized: schema upgraded to the running code.')
+        data={'queued': False},
+        message='No restore is awaiting schema upgrade.')
 
 
 @shopdbbackups_bp.route('/status', methods=['GET'])