Gate a restore on every schema chain, not just the core one

The compatibility gate read alembic_version and nothing else, but plugin schemas
move independently of core. Roll the code back to an older plugin revision and
the core head can be identical, so a dump carrying plugin tables the code has
never heard of passed the gate, loaded - destroying what was there - and only
then failed in 'flask plugin upgrade-all' with "Can't locate revision". A failed
restore that has already replaced what it was restoring over is the worst
outcome this feature can produce, and it was reachable from an ordinary rollback.

Backups now stamp the applied revision of every alembic_version_<plugin> table
alongside the core one, read from the dumped database the same way. restore.sh
checks each against that plugin's own versions directory and refuses before
touching anything, naming the plugin and the revision to deploy past. A plugin
in the dump that this build does not have at all is refused for the same reason.
The list endpoint's compatibility badge asks the same question, so the UI cannot
offer a set the restore will reject.

Sets taken before the stamp existed carry no dataplugins and keep their old
core-only verdict. Their plugin revisions are unknowable, and making every
existing set unrestorable to close this would be a worse trade than the gap.

Proven against a real stack, reproducing the exact case: core head identical,
endpoints data stamped at a revision the build does not have. The restore
refused, the live row was still there afterwards, and it stopped early enough
that the pre-restore safety copy was never needed. The same set with its true
revision restored normally.

Suite 2605 passed / 9 skipped, with the gate covered both ways plus the legacy
fallback.

Scope: product

docs/PROJECT-MAP.md

@@ -91,6 +91,6 @@ Manifest-less directories under `plugins/` are core frontend surface and always
 
 ## Size
 
-- test functions defined: **1847** (parametrised cases collect higher)
+- test functions defined: **1852** (parametrised cases collect higher)
 - documented API paths: **376** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)
 

scripts/backup/backup.sh

@@ -75,6 +75,39 @@ DATAREV="$(mysql \
 	"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
 [ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty"
 
+# The same question for every PLUGIN chain. Each plugin owns an
+# alembic_version_<plugin> table, and those move independently of the core one:
+# code rolled back to an older plugin revision can sit at an identical core
+# head while the dump carries plugin tables it has never heard of. Stamping only
+# the core revision let that combination pass the restore gate, load, and then
+# fail during the plugin upgrade with the data already replaced. Read straight
+# from the dumped DB, same as the core revision above.
+log "reading applied plugin revisions (alembic_version_*)"
+DATAPLUGINS="$(mysql \
+	--host="${MYSQL_HOST}" \
+	--user="${MYSQL_USER}" \
+	--password="${MYSQL_PASSWORD}" \
+	--skip-column-names --batch \
+	-e "SELECT table_name FROM information_schema.tables
+	     WHERE table_schema = '${MYSQL_DATABASE}'
+	       AND table_name LIKE 'alembic\\_version\\_%';" 2>/dev/null \
+	| tr -d '\r' \
+	| while read -r table; do
+		[ -n "${table}" ] || continue
+		plugin="${table#alembic_version_}"
+		rev="$(mysql \
+			--host="${MYSQL_HOST}" \
+			--user="${MYSQL_USER}" \
+			--password="${MYSQL_PASSWORD}" \
+			--skip-column-names --batch \
+			-e "SELECT version_num FROM \`${table}\` LIMIT 1;" \
+			"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
+		[ -n "${rev}" ] && printf '%s\t%s\n' "${plugin}" "${rev}"
+	 done \
+	| python3 -c 'import json,sys; print(json.dumps(dict(l.rstrip("\n").split("\t",1) for l in sys.stdin if "\t" in l)))')"
+[ -n "${DATAPLUGINS}" ] || DATAPLUGINS='{}'
+log "plugin revisions: ${DATAPLUGINS}"
+
 log "writing manifest"
 python3 /opt/backup/manifest.py \
 	--out "${SETDIR}/manifest.json" \
@@ -82,6 +115,7 @@ python3 /opt/backup/manifest.py \
 	--site "${SITE}" \
 	--imagetag "${SHOPDB_IMAGE_TAG:-source}" \
 	--datarev "${DATAREV}" \
+	--dataplugins "${DATAPLUGINS}" \
 	--migrations /app/migrations \
 	--plugins /app/plugins
 

scripts/backup/manifest.py

@@ -10,12 +10,19 @@ restore." It records, at capture time:
                 each plugin chain's heads - the versions a restore target must
                 be at least, or restored rows reference columns the live schema
                 has not applied yet.
-  - datarev   : the Alembic revision actually applied to the DATABASE that was
+  - datarev   : the CORE Alembic revision actually applied to the DATABASE that
-                dumped (its alembic_version), which is the number a restore must
+                was dumped (its alembic_version), which is the number a restore
-                gate on. schemarev stamps the code on disk; datarev stamps the
+                must gate on. schemarev stamps the code on disk; datarev stamps
-                data in the dump. They diverge when an image is ahead of a DB
+                the data in the dump. They diverge when an image is ahead of a DB
                 that has not run 'flask db upgrade' yet, and that divergence is
                 exactly what a code-to-code check cannot see.
+  - dataplugins: the same thing for every PLUGIN chain - the applied revision of
+                each alembic_version_<plugin> table. The core chain alone cannot
+                answer compatibility: plugin schemas move independently, so code
+                rolled back to an older plugin revision can sit at an identical
+                core head while the dump carries plugin tables it has never heard
+                of. Gating on core only let that pass, load, and then fail during
+                the plugin upgrade with the data already replaced.
 
 Two modes:
   --out ...      write a fresh manifest for a new set
@@ -67,6 +74,25 @@ def _heads_for(chain_dir):
     return sorted(rev for rev in revisions if rev not in down_refs)
 
 
+def _parse_dataplugins(raw):
+    """The applied per-plugin revisions, as a {plugin: revision} map.
+
+    Bad or absent JSON becomes an empty map rather than an error: a manifest
+    that cannot say what the plugin chains were at is exactly the legacy case
+    the restore gate already knows how to treat conservatively, and failing the
+    backup here would lose the set over a stamp.
+    """
+    if not raw:
+        return {}
+    try:
+        value = json.loads(raw)
+    except ValueError:
+        return {}
+    if not isinstance(value, dict):
+        return {}
+    return {str(k): str(v) for k, v in value.items() if k and v}
+
+
 def _collect_schema(migrations_dir, plugins_dir):
     """Core heads plus each plugin's chain heads, keyed for compatibility checks."""
     schema = {'core': _heads_for(migrations_dir), 'plugins': {}}
@@ -86,6 +112,7 @@ def write_manifest(args):
         'imagetag': args.imagetag,
         'schemarev': _collect_schema(args.migrations, args.plugins),
         'datarev': args.datarev or '',
+        'dataplugins': _parse_dataplugins(args.dataplugins),
         'ischeckpassed': False,
     }
     with open(args.out, 'w', encoding='utf-8') as handle:
@@ -103,6 +130,8 @@ def amend_manifest(args):
         manifest['ischeckpassed'] = args.ischeckpassed.lower() == 'true'
     if args.datarev is not None:
         manifest['datarev'] = args.datarev
+    if args.dataplugins is not None:
+        manifest['dataplugins'] = _parse_dataplugins(args.dataplugins)
     with open(args.amend, 'w', encoding='utf-8') as handle:
         json.dump(manifest, handle, indent=2, sort_keys=True)
     return 0
@@ -118,6 +147,8 @@ def main(argv=None):
     parser.add_argument('--migrations', default='/app/migrations')
     parser.add_argument('--plugins', default='/app/plugins')
     parser.add_argument('--datarev')
+    parser.add_argument('--dataplugins',
+                        help='JSON object of {plugin: applied revision}')
     parser.add_argument('--ischeckpassed')
     args = parser.parse_args(argv)
 

scripts/backup/restore.sh

@@ -101,6 +101,47 @@ if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2
 fi
 log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
 
+# The same gate for every PLUGIN chain, because the core chain alone cannot
+# answer this. Plugin schemas move independently of core: roll the code back to
+# an older plugin revision and the core head can be identical while the dump
+# carries plugin tables the code has never heard of. Gating on core only let
+# that pass, load the dump, and then fail in 'flask plugin upgrade-all' with
+# "Can't locate revision" and the data already replaced - a failed restore that
+# has destroyed what it was restoring over.
+#
+# A manifest with no dataplugins is a legacy set taken before this was stamped.
+# Its plugin revisions are unknowable, so it is allowed through on the core gate
+# alone (the behaviour before this check existed) rather than being made
+# unrestorable retroactively.
+SET_DATAPLUGINS="$(python3 -c \
+	'import json,sys
+try:
+    data = json.load(open(sys.argv[1])).get("dataplugins") or {}
+except Exception:
+    data = {}
+for name, rev in sorted(data.items()):
+    print(name + " " + rev)' \
+	"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
+
+if [ -z "${SET_DATAPLUGINS}" ]; then
+	log "manifest stamps no plugin revisions (legacy set) - core gate only"
+else
+	while read -r plugin rev; do
+		[ -n "${plugin}" ] || continue
+		chain="/app/plugins/${plugin}/migrations/versions"
+		if [ ! -d "${chain}" ]; then
+			die "the backup carries data for plugin '${plugin}', which this build does not have - deploy a build that includes it before restoring"
+		fi
+		if ! grep -rqE "^revision = ['\"]${rev}['\"]" "${chain}" 2>/dev/null; then
+			log "plugin ${plugin} data revision: ${rev}"
+			die "plugin '${plugin}' data is at a revision this code does not have - deploy code at or past '${rev}' before restoring"
+		fi
+		log "plugin ${plugin} at ${rev} - known to this code"
+	done <<PLUGINREVS
+${SET_DATAPLUGINS}
+PLUGINREVS
+fi
+
 # Safety set FIRST. Everything below this point is destructive and none of it
 # is transactional: the dump carries DROP TABLE per table, and the instance tree
 # is replaced wholesale. Without a copy taken here, a restore of the wrong set -

shopdb/core/api/shopdbbackups.py

@@ -383,6 +383,63 @@ def _known_core_revisions():
         return set()
 
 
+def _known_plugin_revisions():
+    """Every revision each plugin chain knows, as {plugin: {revisions}}.
+
+    Read by text scan rather than through Alembic. Each plugin chain has its own
+    env and version table, so loading them properly would mean building 17
+    ScriptDirectories on a request that only needs to answer "is this id in
+    there" - and manifest.py already reads the same trees the same way, so the
+    two agree by construction.
+    """
+    root = os.path.dirname(current_app.root_path)
+    plugins_dir = os.path.join(root, 'plugins')
+    known = {}
+    if not os.path.isdir(plugins_dir):
+        return known
+    for name in sorted(os.listdir(plugins_dir)):
+        versions = os.path.join(plugins_dir, name, 'migrations', 'versions')
+        if not os.path.isdir(versions):
+            continue
+        revisions = set()
+        for filename in os.listdir(versions):
+            if not filename.endswith('.py'):
+                continue
+            try:
+                with open(os.path.join(versions, filename), 'r',
+                          encoding='utf-8') as handle:
+                    for line in handle:
+                        stripped = line.strip()
+                        if stripped.startswith('revision ='):
+                            revisions.add(
+                                stripped.split('=', 1)[1].strip().strip('"\''))
+                            break
+            except OSError:
+                continue
+        if revisions:
+            known[name] = revisions
+    return known
+
+
+def _plugin_data_is_known(manifest):
+    """Whether this build knows every plugin revision the set's data sits at.
+
+    A set with no `dataplugins` predates the stamp; its plugin revisions are
+    unknowable, so it keeps the old core-only verdict rather than being marked
+    unrestorable retroactively.
+    """
+    stamped = manifest.get('dataplugins') or {}
+    if not isinstance(stamped, dict) or not stamped:
+        return True
+    known = _known_plugin_revisions()
+    for name, revision in stamped.items():
+        if not revision:
+            continue
+        if revision not in known.get(name, ()):
+            return False
+    return True
+
+
 def _is_compatible(manifest, current):
     """Decide whether a set restores onto THIS build.
 
@@ -395,6 +452,11 @@ def _is_compatible(manifest, current):
     revision, the dump is from a NEWER schema than this build and would 500 on
     restored rows, so we block until the code is upgraded.
 
+    The same question is asked of every PLUGIN chain, because plugin schemas
+    move independently: code rolled back to an older plugin revision can sit at
+    an identical core head while the dump carries plugin tables it cannot
+    migrate.
+
     Legacy sets predate `datarev`; fall back to the old head-subset check on
     `schemarev` so they stay listable and restorable as before.
     """
@@ -405,7 +467,10 @@ def _is_compatible(manifest, current):
         known = _known_core_revisions()
         if not known:
             return False
-        return datarev in known
+        # Core AND every plugin chain. Core alone said yes to a set whose plugin
+        # data the code had never heard of, which loads and then dies during the
+        # plugin upgrade with the old data already gone.
+        return datarev in known and _plugin_data_is_known(manifest)
     setrevs = set(manifest.get('schemarev', {}).get('core', []))
     currevs = set(current.get('core', []))
     if not setrevs or not currevs:

tests/test_core/test_shopdbbackups_finalize.py

@@ -140,3 +140,72 @@ def test_an_unconfigured_deployment_refuses(app, client, auth_headers,
                            headers=auth_headers)
 
     assert response.status_code == 404
+
+
+class TestPluginChainCompatibility:
+    """The restore gate has to see plugin chains, not just the core one.
+
+    The failure it exists for: roll code back to an older plugin revision and
+    the CORE head can be identical, so a core-only gate passes a dump whose
+    plugin tables the code has never heard of. It loads - destroying what was
+    there - and only then dies in 'flask plugin upgrade-all'.
+    """
+
+    def _manifest(self, core_rev, plugins=None):
+        manifest = {'datarev': core_rev, 'schemarev': {'core': [core_rev]}}
+        if plugins is not None:
+            manifest['dataplugins'] = plugins
+        return manifest
+
+    def _real_core_rev(self, app):
+        with app.app_context():
+            known = shopdbbackups._known_core_revisions()
+        assert known, 'expected this build to know its own core revisions'
+        return sorted(known)[0]
+
+    def _real_plugin_rev(self, app):
+        with app.app_context():
+            known = shopdbbackups._known_plugin_revisions()
+        assert known, 'expected this build to have plugin migration chains'
+        name = sorted(known)[0]
+        return name, sorted(known[name])[0]
+
+    def test_a_set_from_a_newer_plugin_chain_is_refused(self, app):
+        core = self._real_core_rev(app)
+        name, _ = self._real_plugin_rev(app)
+        manifest = self._manifest(core, {name: 'thisrevisiondoesnotexist'})
+
+        with app.app_context():
+            assert shopdbbackups._is_compatible(manifest, {}) is False, \
+                'a dump carrying unknown plugin data was called restorable'
+
+    def test_a_set_whose_plugin_chains_are_known_is_allowed(self, app):
+        core = self._real_core_rev(app)
+        name, rev = self._real_plugin_rev(app)
+        manifest = self._manifest(core, {name: rev})
+
+        with app.app_context():
+            assert shopdbbackups._is_compatible(manifest, {}) is True
+
+    def test_a_plugin_this_build_does_not_have_is_refused(self, app):
+        core = self._real_core_rev(app)
+        manifest = self._manifest(core, {'nosuchplugin': 'somerevision'})
+
+        with app.app_context():
+            assert shopdbbackups._is_compatible(manifest, {}) is False
+
+    def test_a_legacy_set_without_the_stamp_keeps_its_old_verdict(self, app):
+        """Sets taken before dataplugins existed must not become unrestorable."""
+        core = self._real_core_rev(app)
+
+        with app.app_context():
+            assert shopdbbackups._is_compatible(self._manifest(core), {}) is True
+            assert shopdbbackups._is_compatible(
+                self._manifest(core, {}), {}) is True
+
+    def test_the_core_gate_still_refuses_an_unknown_core_revision(self, app):
+        name, rev = self._real_plugin_rev(app)
+        manifest = self._manifest('unknowncorerevision', {name: rev})
+
+        with app.app_context():
+            assert shopdbbackups._is_compatible(manifest, {}) is False