ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ADR-014 Phase 2: flask plugin prune-schema for lean per-site DBs

Browse Source 
A lean site still gets every plugin's tables from the shared core Alembic
baseline. prune-schema drops the tables of plugins not installed on this
site, leaving core + chosen-plugin tables, with no edit to any released
migration (the relocate-into-plugin-baselines alternative would mean
rewriting ~15 released core migrations for a cosmetic gain - see ADR-014).

- shopdb/plugins/cli.py: prune-schema command. Dry-run by default; --yes to
  execute; refuses non-empty tables without --force. Drops by table name (no
  plugin import) so it works on a lean image. MySQL: private AUTOCOMMIT engine
  (db.engine's pooled connections sit idle-in-transaction in a CLI context and
  would deadlock the DROP on a metadata lock). SQLite: db.engine, restoring the
  prior foreign_keys pragma so the StaticPool connection is not left changed.
- tests/test_plugin_prune_schema.py: drop-only-not-installed, full no-op,
  refuse-non-empty, force-drops-non-empty.
- docs/DEPLOY.md: lean provisioning step after upgrade-all.
- ADR-014 ACCEPTED; index updated.

Verified on MySQL: full install then prune = no-op (86 tables); lean install
(machines+printers) then prune drops the other 19 plugin tables; second run
no-op. Full suite 1077 passed.
This commit is contained in:
cproudlock
2026-07-19 11:39:46 -04:00
parent 42ca8d75c3
commit c386e211df
6 changed files with 307 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CLAUDE.md
View File

@@ -23,7 +23,7 @@ Architecture decisions live in `docs/adr/`. Read those before making schema or c
- ADR-011: Machines rename + modeltypes retyping - ACCEPTED
- ADR-012: GE-Enforce manifest ownership in shopdb - ACCEPTED
- ADR-013: Plugin catalog, curated shelf, and lean per-site builds - PROPOSED
- ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables) - PROPOSED
- ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, prune not-installed plugin tables) - ACCEPTED
## Coding convention

18
docs/DEPLOY.md
View File

@@ -73,6 +73,24 @@ any plugin-specific migrations added after the ownership cutover. Both commands
are idempotent, so re-running them is safe. See ADR-008 for why plugin schema
splits into per-plugin chains from the cutover forward.
**Lean sites (ADR-014):** the core chain creates every bundled plugin's tables,
so a site that ships only some plugins still has the others' (empty) tables. To
carry only core + chosen-plugin tables, prune the rest once, at initial
provisioning, after the two commands above:
```bash
docker compose exec api flask plugin prune-schema # dry-run, review
docker compose exec api flask plugin prune-schema --yes --force
```
It drops the tables of every plugin not installed on this site. `--force` is
needed because the core chain seeds a few plugin reference tables (default
access protocols, etc.); at first provisioning those hold only seeded defaults,
before any site data. It refuses to drop a table that holds rows without
`--force`, so it is safe to leave out of routine upgrades - run it only when
provisioning a lean site or after deliberately removing a plugin. Installing a
pruned plugin later recreates its tables automatically.
**Charset:** the schema is utf8mb4 (`utf8mb4_unicode_ci`). The docker-compose `db` service sets `--character-set-server=utf8mb4`, so the auto-created `shopdb_flask` database is utf8mb4. If you point at an external MySQL instead of the bundled container, create the database as utf8mb4 first, or it inherits the server default (often latin1) and the schema silently drifts:
```sql

98
docs/adr/ADR-014-schema-lean-per-site.md
View File

@@ -1,6 +1,6 @@
# ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables)
- Status: PROPOSED
- Status: ACCEPTED
- Date: 2026-07-19
- Deciders: cproudlock
- Relates to: ADR-008 (per-plugin migration ownership), ADR-013 (plugin catalog + lean per-site builds), ADR-001 (asset model)
@@ -65,54 +65,78 @@ the pre-cutover core baseline. Correct and inert regardless of Phase 2 (no
current caller creates against a populated schema). Verified against the
plugin-migration suite.
### Phase 2 (load-bearing, dedicated pass): lift plugin tables into plugin baselines
### Phase 2 (executed): prune not-installed plugin tables after upgrade
With no cross-plugin FKs remaining, each plugin's tables can be created
independently. One coherent baseline edit:
Two mechanisms were weighed to make a lean site's database carry only
core + chosen-plugin tables:
- Remove the ~30 plugin-owned `create_table` blocks from the core baseline
(68b3947ae14f), plus the four dead-object blocks (machinerelationships,
printerdata, installedapps, communications.machineid) it creates only for
later migrations to drop. Core baseline then creates only core tables.
- Change each of the 14 plugin 0001 anchors from stamp-only `pass` to
`create_plugin_tables(<name>)` / `drop_plugin_tables(<name>)` (idempotent, per
above).
- **Relocate** (rejected): pull every plugin-table create/alter out of the core
chain into the plugin baselines, so the core chain never creates a
not-installed plugin's table. Measurement killed this: plugin tables are
created and altered across ~15 released core migrations (baseline plus 7c04,
7d05, 7d08, 7d13, 7d15, 7d16, 7d17, ...), not just the baseline. Because the
whole core chain runs before any plugin chain, removing a table's create from
core while a later core migration still alters it breaks FULL installs too, so
relocation means surgically rewriting ~15 released migrations - the highest
blast radius in the project - for a purely cosmetic gain (the omitted tables
are empty and the lean CODE build already never loads the plugin).
A fresh lean install then creates core tables plus only the chosen plugins'
tables. A fresh full install creates the identical table set it does today.
- **Prune-after-upgrade** (chosen): leave the entire core chain untouched. Add
`flask plugin prune-schema`, which drops the tables of every plugin in
PLUGIN_TABLE_OWNERS that is not installed on this site. Run once at deploy,
after `flask db upgrade` and `flask plugin upgrade-all`. Same end state
(core + chosen tables) with near-zero blast radius: no released migration is
edited, and an existing full site is unaffected because it never runs the
command.
Existing-database safety: an existing database is stamped past the baseline and
past each plugin's old stamp-anchor, so neither re-runs; it keeps its tables.
Editing the baseline's content only changes what a FRESH install creates. This
is the highest-blast-radius edit in the project (the released baseline every
site's DB derives from), so it is staged as its own pass gated on the full
verification matrix: fresh-full (== current schema), fresh-lean (strict subset),
existing-DB (no re-run, unchanged), and the migrations-mysql CI (fresh upgrade
from empty + per-plugin install + second-run no-op).
`prune-schema` drops by table name (no plugin-code import), so it works on a
lean image where the omitted plugin's directory is absent. It is a dry-run by
default and refuses to drop a table that holds rows unless `--force`, so a
misfire on a populated site cannot silently delete data. Because the core chain
seeds a few plugin reference tables (e.g. 7d05 inserts default access
protocols), initial lean provisioning uses `--force` - at that point the tables
hold only migration-seeded defaults, before any site data exists.
The idempotent `create_plugin_tables` (enabling change above) is what lets a
lean site later ADD an omitted plugin: its anchor recreates the pruned tables.
Verified end to end on MySQL: fresh full install (86 tables) then prune is a
no-op; fresh lean install (machines + printers) then prune drops the other 19
plugin tables, leaving core + chosen; second prune is a no-op; the non-empty
guard refuses without `--force`. Four SQLite regression tests pin the behavior
(tests/test_plugin_prune_schema.py), running in the backend CI job via the real
CLI runner: drop-only-not-installed, full-site no-op, refuse-non-empty, and
force-drops-non-empty.
## Consequences
### Positive
- Removes every cross-plugin foreign key; plugin schemas become independent, as
ADR-013 requires.
- Deletes dead legacy tables/columns every database has carried since the
cutover (real cleanup, not just lean).
- After Phase 2, a lean site's database contains only core + chosen-plugin
tables.
- A lean site's database contains only core + chosen-plugin tables, with no edit
to any released migration (near-zero blast radius).
- The cross-plugin FK blocker ADR-013 cited is gone (dead cruft, dropped by
existing migrations), so plugin schemas are already FK-independent.
- Adding an omitted plugin to a lean site later just works: the idempotent
anchor recreates its tables.
### Negative / risk
- Phase 2 edits the released baseline's content. It is safe because existing
databases never re-run a stamped revision, but it demands the full fresh +
existing + CI verification and is therefore staged separately.
- Dropping tables is destructive; the migration downgrade recreates them empty
(structure only) - acceptable because they hold no live data.
- prune-schema is destructive by nature; the row-count guard + dry-run default +
required `--force` for non-empty tables contain that. It is a deploy-time
provisioning step, not something to run casually on a live populated site.
- A lean fresh install still transiently creates then drops the omitted plugins'
tables (the core chain builds them, prune removes them). Harmless and one-time
at provisioning; the trade for not touching the released baseline.
## Implementation
- Phase 1: core migration `7d27_retire_legacy_machine_fk_tables` + drop the dead
`communications.machineid` column from the model. Verified: fresh upgrade,
idempotent re-run, and a scratch database that had the tables drops them.
- Phase 2: baseline edit + 14 plugin anchor rewrites + idempotent-create guards,
gated by the migrations-mysql CI, in a dedicated pass.
- Phase 1: nothing to do - the dead cross-boundary FK objects were already
dropped by existing migrations `7a01_adr001_position_contract` and
`7c01_drop_legacy_machine`; verified absent on a fresh full MySQL upgrade.
- Enabling change: `create_plugin_tables` made idempotent
(`shopdb/plugins/alembic_template.py`).
- Phase 2: `flask plugin prune-schema` (`shopdb/plugins/cli.py`), dry-run by
default, `--yes` to execute, `--force` for non-empty tables. Deploy order:
`flask db upgrade` -> `flask plugin upgrade-all` -> `flask plugin prune-schema
--yes --force`. Regression tests in `tests/test_plugin_prune_schema.py` (run in
the backend CI job).

2
docs/adr/README.md
View File

@@ -26,7 +26,7 @@ Each ADR captures a single architectural decision: the context, the decision its
| [011](ADR-011-machines-rename.md) | Machines rename + modeltypes retyping | ACCEPTED |
| [012](ADR-012-geenforce-manifest-ownership.md) | GE-Enforce manifest ownership in shopdb | ACCEPTED |
| [013](ADR-013-plugin-catalog-and-lean-builds.md) | Plugin catalog, curated shelf, and lean per-site builds | PROPOSED |
| [014](ADR-014-schema-lean-per-site.md) | Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables) | PROPOSED |
| [014](ADR-014-schema-lean-per-site.md) | Schema-lean per-site builds (retire cross-plugin FKs, prune not-installed plugin tables) | ACCEPTED |
## Authoring

139
shopdb/plugins/cli.py
View File

@@ -886,3 +886,142 @@ def upgrade_all_plugins():
click.echo(click.style(f" {name:20} no migrations", fg='yellow'))
else:
click.echo(click.style(f" {name:20} {status}", fg='red'))
@plugin_cli.command('prune-schema')
@click.option('--yes', is_flag=True,
help='Actually drop the tables (default is a dry-run preview)')
@click.option('--force', is_flag=True,
help='Drop even tables that hold rows (DATA LOSS); default refuses')
@with_appcontext
def prune_schema(yes: bool, force: bool):
"""Drop tables owned by plugins this site did NOT install (ADR-014).
Schema-lean per-site DBs: the shared core Alembic baseline creates every
plugin's tables, so a lean site that omits a plugin still carries that
plugin's (empty) tables. This drops the tables of every plugin in
PLUGIN_TABLE_OWNERS that is not installed here, leaving core + chosen-plugin
tables only. Run once at deploy AFTER `flask db upgrade` and
`flask plugin upgrade-all`.
Dry-run by default; pass --yes to execute. Drops by table name (no plugin
code import) so it works on a lean image where the omitted plugin's
directory is absent. Refuses to drop a non-empty table unless --force, so a
misfire on a populated full site cannot silently delete data.
"""
from sqlalchemy import create_engine, inspect, text
from shopdb.extensions import db
from shopdb.plugins.alembic_template import PLUGIN_TABLE_OWNERS
pm = current_app.extensions.get('plugin_manager')
if not pm:
click.echo(click.style("Plugin manager not initialized", fg='red'))
raise SystemExit(1)
installed = {p['name'] for p in pm.discover_available() if p['installed']}
engine = db.engine
existing = set(inspect(engine).get_table_names())
# A table is a prune candidate when its owning plugin is not installed here
# AND the table actually exists in this database.
victims = []
for plugin, tables in PLUGIN_TABLE_OWNERS.items():
if plugin in installed:
continue
for tablename in tables:
if tablename in existing:
victims.append((plugin, tablename))
if not victims:
click.echo("Nothing to prune: every plugin-owned table belongs to an "
"installed plugin.")
return
# On MySQL, use a SEPARATE engine in AUTOCOMMIT, not db.engine. Two reasons,
# both of which burned real debugging time:
# 1. db.engine's pool keeps connections idle-in-transaction (Flask-
# SQLAlchemy has no request-teardown in a CLI context); a DROP sharing
# that pool waits on their locks.
# 2. Without engine-level AUTOCOMMIT, SQLAlchemy runs the COUNT probes in an
# open transaction (SET AUTOCOMMIT=0). Those reads hold shared metadata
# locks on every victim table, so the later DROP blocks on the metadata
# lock forever. Setting isolation_level on the CONNECTION (not the
# engine) silently did NOT take effect - it must be on the engine.
# With engine-level AUTOCOMMIT every statement commits on its own, so no read
# holds a lock into the DROP phase. lock_wait_timeout makes any residual
# contention fail fast. SQLite has none of this (no metadata locks), and a
# separate engine to an in-memory database would be a different, empty DB, so
# there we just use db.engine. db.session is only used above, for pluginstate.
dialect = engine.dialect.name
db.session.remove()
if dialect == 'mysql':
prune_engine = create_engine(
current_app.config['SQLALCHEMY_DATABASE_URI'],
isolation_level='AUTOCOMMIT')
own_engine = True
else:
prune_engine = engine
own_engine = False
try:
with prune_engine.connect() as conn:
if dialect == 'mysql':
conn.execute(text('SET SESSION lock_wait_timeout=15'))
# Row counts so a non-empty table is never dropped by accident.
nonempty = []
for plugin, tablename in victims:
count = conn.execute(
text(f'SELECT COUNT(*) FROM `{tablename}`')).scalar()
if count:
nonempty.append((plugin, tablename, count))
click.echo(click.style(
f"Plugins not installed here: "
f"{', '.join(sorted({p for p, _ in victims}))}", fg='cyan'))
click.echo(f"Tables to prune ({len(victims)}):")
for plugin, tablename in victims:
click.echo(f" {tablename:32} ({plugin})")
if nonempty and not force:
click.echo("")
click.echo(click.style(
"REFUSING: the following tables hold rows. Re-run with "
"--force to drop them anyway (this deletes data), or install "
"the owning plugin instead.", fg='red'))
for plugin, tablename, count in nonempty:
click.echo(click.style(f" {tablename:32} {count} rows",
fg='red'))
raise SystemExit(1)
if not yes:
click.echo("")
click.echo("Dry-run. Re-run with --yes to drop the tables above.")
return
# Intra-plugin foreign keys mean drop order matters; disable the
# checks for the batch rather than topologically sorting the tables
# without the models. Restore the PRIOR foreign-key setting after,
# not a hard ON: on SQLite the same connection is reused (StaticPool),
# so forcing ON would leak into whatever ran next.
fk_prev = None
if dialect == 'mysql':
conn.execute(text('SET FOREIGN_KEY_CHECKS=0'))
elif dialect == 'sqlite':
fk_prev = conn.execute(text('PRAGMA foreign_keys')).scalar()
conn.execute(text('PRAGMA foreign_keys=OFF'))
for _plugin, tablename in victims:
conn.execute(text(f'DROP TABLE IF EXISTS `{tablename}`'))
if dialect == 'mysql':
conn.execute(text('SET FOREIGN_KEY_CHECKS=1'))
elif dialect == 'sqlite':
conn.execute(text(f'PRAGMA foreign_keys={int(fk_prev or 0)}'))
conn.commit()
finally:
if own_engine:
prune_engine.dispose()
click.echo(click.style(f"Pruned {len(victims)} table(s). This database now "
f"carries core + installed-plugin tables only.",
fg='green'))

87
tests/test_plugin_prune_schema.py Normal file
View File

@@ -0,0 +1,87 @@
"""Tests for `flask plugin prune-schema` (ADR-014 schema-lean per-site).
The shared core Alembic baseline creates every plugin's tables, so a lean site
that omits a plugin still carries that plugin's (empty) tables. prune-schema
drops the tables of every not-installed plugin, leaving core + chosen-plugin
tables. These tests pin the three behaviors that matter: only not-installed
plugins are dropped, the non-empty guard refuses without --force, and a full
site (every plugin installed) is a no-op.
"""
import pytest
from sqlalchemy import inspect, text
from shopdb.extensions import db as _db
from shopdb.plugins.cli import plugin_cli
from shopdb.plugins.alembic_template import PLUGIN_TABLE_OWNERS
def _install_only(app, monkeypatch, installed_names):
"""Force discover_available to report exactly `installed_names` installed."""
pm = app.extensions['plugin_manager']
listing = [{'name': name, 'installed': name in installed_names}
for name in PLUGIN_TABLE_OWNERS]
monkeypatch.setattr(pm, 'discover_available', lambda: listing)
def test_prune_drops_only_not_installed(app, db, runner, monkeypatch):
"""A lean site keeps its plugins' tables and drops every other plugin's."""
_install_only(app, monkeypatch, {'machines', 'printers'})
before = set(inspect(_db.engine).get_table_names())
assert 'usbdevices' in before and 'machines' in before # baseline sanity
result = runner.invoke(plugin_cli, ['prune-schema', '--yes'])
assert result.exit_code == 0, result.output
after = set(inspect(_db.engine).get_table_names())
# Kept: machines + printers owned tables.
for keep in PLUGIN_TABLE_OWNERS['machines'] + PLUGIN_TABLE_OWNERS['printers']:
assert keep in after, f'{keep} should have been kept'
# Dropped: every other plugin's tables.
for plugin, tables in PLUGIN_TABLE_OWNERS.items():
if plugin in ('machines', 'printers'):
continue
for tablename in tables:
assert tablename not in after, f'{tablename} should have been dropped'
# Core tables untouched.
for core in ('assets', 'users', 'settings'):
assert core in after
def test_prune_full_site_is_noop(app, db, runner, monkeypatch):
"""Every plugin installed -> nothing to prune, schema unchanged."""
_install_only(app, monkeypatch, set(PLUGIN_TABLE_OWNERS))
before = set(inspect(_db.engine).get_table_names())
result = runner.invoke(plugin_cli, ['prune-schema', '--yes', '--force'])
assert result.exit_code == 0, result.output
assert 'Nothing to prune' in result.output
assert set(inspect(_db.engine).get_table_names()) == before
def test_prune_refuses_non_empty_without_force(app, db, runner, monkeypatch):
"""A not-installed table that holds rows is not dropped without --force."""
_install_only(app, monkeypatch, {'machines'})
_db.session.execute(text(
"INSERT INTO usbdevicetypes (typename, createddate, modifieddate, "
"isactive) VALUES ('kept-by-guard', '2026-01-01', '2026-01-01', 1)"))
_db.session.commit()
result = runner.invoke(plugin_cli, ['prune-schema', '--yes'])
assert result.exit_code == 1
assert 'REFUSING' in result.output
# The row-bearing table survives the refusal.
assert 'usbdevicetypes' in set(inspect(_db.engine).get_table_names())
def test_prune_force_drops_non_empty(app, db, runner, monkeypatch):
"""--force drops even a table that holds rows."""
_install_only(app, monkeypatch, {'machines'})
_db.session.execute(text(
"INSERT INTO usbdevicetypes (typename, createddate, modifieddate, "
"isactive) VALUES ('dropped-by-force', '2026-01-01', '2026-01-01', 1)"))
_db.session.commit()
result = runner.invoke(plugin_cli, ['prune-schema', '--yes', '--force'])
assert result.exit_code == 0, result.output
assert 'usbdevicetypes' not in set(inspect(_db.engine).get_table_names())