Let a baseline meet a table that is already there
Some checks failed
CI / backend (push) Failing after 11s
CI / naming (push) Successful in 2s
CI / frontend (push) Successful in 11s
CI / migrations-mysql (push) Failing after 7s

Reported: `flask plugin upgrade-all` failed the tools plugin with "table
securesendsecrets already exists".

Every post-cutover baseline emitted a plain op.create_table. That is correct
only while a table's existence and its plugin's Alembic stamp can never
disagree, and they can. Anything that builds tables from the models -
db.create_all(), which `flask db-utils create-all` calls - creates them and
stamps nothing. A run that creates a table and then fails before its stamp
commits leaves the same split, and re-running is the obvious thing to try. In
both cases upgrade-all meets existing tables with an unstamped chain and the
create is fatal.

Nothing caught it because the migration test DROPS the post-cutover tables
db.create_all() built before running upgrade-all. That arranges the one state
where an unguarded create works, and never the state the report came from. The
new test is the same sequence with the drop removed for the plugin under test.

create_table_if_missing and create_index_if_missing are in alembic_template.py
next to create_plugin_tables, which had the idempotency rule right and turns out
to be called by nobody - its docstring claimed every 0001 used it. Fixed the
docstring rather than rewiring the baselines onto it, because the reason they
hand-write their DDL still stands: the helper's per-plugin MetaData cannot
resolve a foreign key to a core table at CreateTable-compile time.

INDEXES ARE MATCHED ON COLUMNS, NOT NAME, and that distinction matters. The two
creators disagree about names - db.create_all() takes SQLAlchemy's default
convention (ix_securesendsecrets_token) and a baseline spells it the way this
repo's naming rules require (ixsecuresendsecretstoken). A name-keyed guard would
have called the index missing and added a second one over the same column, which
MySQL permits with a warning, so every affected site would have quietly grown
duplicates. The test asserts no table ends up with two indexes over the same
columns.

Applied to all nine table-creating plugin migrations, not just the tools
baseline that was reported: the four later migrations that create tables have
the same exposure, and geenforce, measuringtools, printedparts and backups are
the same defect waiting for a different site.

WHAT THIS DOES NOT FIX. It makes a baseline idempotent, not a whole chain. A
longer chain can still meet an ALTER whose column is already present -
printedparts adding gagelabtag to a table create_all built at head is the case
that showed up while testing. Guarding every op one at a time is the wrong
answer to that; a database built from the models is already AT head and its
chains want stamping rather than replaying, and there is no `flask plugin stamp`
today. Left alone deliberately rather than half-built, and the new test is
scoped to single-baseline plugins so it does not pretend otherwise.
This commit is contained in:
cproudlock
2026-08-24 09:28:37 -04:00
parent de3c5aa562
commit b96637e1a1
12 changed files with 285 additions and 74 deletions

View File

@@ -276,3 +276,89 @@ def test_upgrade_all_on_fresh_db_is_clean_and_idempotent(tmp_path, monkeypatch):
(plugin_manager._app, plugin_manager._db, plugin_manager.registry,
plugin_manager.loader, plugin_manager.migration_manager,
plugin_manager._registered_prefixes) = saved
# Plugins whose whole chain is a single baseline. Their tables existing already
# is the ONLY divergence possible between schema and stamp, so upgrade-all must
# handle it end to end. A longer chain can also meet an ALTER whose column is
# already present, which no amount of guarding a create_table fixes - see the
# test below for why that is a different problem.
SINGLE_BASELINE_PLUGINS = tuple(
plugin for plugin in POST_CUTOVER_PLUGINS
if len(list((PLUGINS_DIR / plugin / 'migrations' / 'versions').glob('0*.py'))) == 1
)
@pytest.mark.parametrize('plugin', SINGLE_BASELINE_PLUGINS)
def test_baseline_skips_tables_that_already_exist(plugin, tmp_path, monkeypatch):
"""A baseline must stamp, not fail, when its table is already there.
Reported: `flask plugin upgrade-all` failed the tools plugin with "table
securesendsecrets already exists". Every post-cutover baseline was a plain
op.create_table, correct only while a table's existence and its plugin's
Alembic stamp can never disagree. They can - anything that builds tables
from the models (db.create_all) stamps nothing, and a run that creates a
table then fails before its stamp commits leaves the same split.
The other upgrade-all test DROPS these tables before running, so it only
ever covered the state where they are absent. This is the other state.
"""
from sqlalchemy import inspect, text
from shopdb.config import TestingConfig
from shopdb import create_app
from shopdb.extensions import db
from shopdb.plugins import plugin_manager
monkeypatch.setattr(TestingConfig, 'SQLALCHEMY_DATABASE_URI',
f'sqlite:///{tmp_path / "preexisting.db"}')
saved = (plugin_manager._app, plugin_manager._db, plugin_manager.registry,
plugin_manager.loader, plugin_manager.migration_manager,
plugin_manager._registered_prefixes)
try:
app = create_app('testing')
with app.app_context():
db.create_all()
# Leave THIS plugin's tables in place - that is the whole point -
# and drop the other post-cutover ones so their own (longer) chains
# run the way the sibling test already covers.
insp0 = inspect(db.engine)
for other in POST_CUTOVER_PLUGINS:
if other == plugin:
continue
for tablename in PLUGIN_TABLE_OWNERS[other]:
if insp0.has_table(tablename):
db.session.execute(text(f'DROP TABLE {tablename}'))
db.session.commit()
for tablename in PLUGIN_TABLE_OWNERS[plugin]:
assert inspect(db.engine).has_table(tablename), (
f'{tablename} should have been built by create_all; '
f'this test is meaningless without it')
first = app.extensions['plugin_manager'].upgrade_all_plugins()
assert first[plugin] == 'ok', first[plugin]
second = app.extensions['plugin_manager'].upgrade_all_plugins()
assert second[plugin] == 'ok', second[plugin]
insp = inspect(db.engine)
row = db.session.execute(
text(f'SELECT version_num FROM alembic_version_{plugin}')
).fetchone()
assert row and row[0] == EXPECTED_HEAD_REVISION[plugin]
# No second index over the same columns: create_all names indexes by
# SQLAlchemy's convention and the baseline by this repo's, so a
# name-keyed guard would have added a duplicate rather than skipped.
for tablename in PLUGIN_TABLE_OWNERS[plugin]:
seen = [tuple(ix.get('column_names') or ())
for ix in insp.get_indexes(tablename)]
assert len(seen) == len(set(seen)), (
f'{tablename} has two indexes over the same columns: {seen}')
finally:
(plugin_manager._app, plugin_manager._db, plugin_manager.registry,
plugin_manager.loader, plugin_manager.migration_manager,
plugin_manager._registered_prefixes) = saved