Make a plugin toggle take effect without a manual restart
Enabling a plugin could not make its routes live. Flask refuses
register_blueprint once the app has served a request, and each gunicorn worker
holds its own app object, so registering in the worker that served the toggle
would leave the others answering 404 for the same URL.
Worse, the window before the restart was a window in which workers disagreed
about reality. Four defects, all of which answered 200 while going wrong:
- Enabling a not-yet-installed plugin ALWAYS 500'd on a running app:
install_plugin ran the migrations, updated the registry, and only then called
register_blueprint and hit Flask's assertion. The toggle half-applied and
reported failure.
- Enabling one plugin could hang the entire site. @jwt_required does SELECT
users, so the request's transaction sits open holding a shared metadata lock;
the plugin's migration runs on its own Alembic connection and needs an
EXCLUSIVE lock on users for a foreign key. Each waits on the other, and MySQL
grants metadata locks in request order, so every later login queued behind the
pending exclusive. Observed twice as a total outage from one plugin toggle.
- Each worker rewrote the WHOLE registry file from a snapshot taken at boot, so
concurrent enables erased each other: seven enables persisted three, with
HTTP 200 for all seven.
- Dependency checks read that same stale snapshot, refusing a plugin whose
dependency another worker had just enabled.
Routes still require the process to restart; what changes is who initiates it.
Each worker fingerprints instance/plugins.json at boot and exits gracefully when
it changes, and the supervisor replaces it. That is the primitive gunicorn's own
--reload uses. Recycles are staggered per worker so the pool rolls rather than
dropping at once, and the exit is scoped to gunicorn: on Windows sys.exit() in a
daemon thread ends the thread rather than the process, so IIS keeps the manual
restart until that path is proven on the Windows VM.
The registry now merges under an exclusive lock and replaces atomically, saves
only the entry being changed, and re-reads before any decision is made from it.
ADR-020 records the decision; ADR-013 is amended where it said the UI must tell
the operator to restart. The restart requirement and the honesty requirement
both stand - only the manual step is removed, and only where it is proven.
(cherry picked from commit 9d6342b3ab)
This commit is contained in:
@@ -44,6 +44,10 @@ import shopdb.utils.mysql_charset # noqa: F401,E402
|
||||
|
||||
logger = logging.getLogger('alembic.env.plugin')
|
||||
|
||||
# Seconds a migration will wait for a metadata lock before failing. Long enough
|
||||
# to ride out a normal request, far short of MySQL's one-year default.
|
||||
MIGRATION_LOCK_WAIT_SECONDS = 20
|
||||
|
||||
# Explicit table-ownership map. Adding tables to a plugin requires updating
|
||||
# this dict so the per-plugin migration knows which tables to include.
|
||||
PLUGIN_TABLE_OWNERS: dict[str, Iterable[str]] = {
|
||||
@@ -250,7 +254,25 @@ def run_migrations():
|
||||
# config.get_section returns an empty dict and engine_from_config would
|
||||
# find no sqlalchemy.url. db_url is already resolved above.
|
||||
from sqlalchemy import create_engine
|
||||
connectable = create_engine(db_url, poolclass=pool.NullPool)
|
||||
|
||||
# Bound how long a migration waits for a metadata lock.
|
||||
#
|
||||
# THIS connection is the one that blocks. A plugin table with a foreign
|
||||
# key to a core table (users, assets) needs an EXCLUSIVE metadata lock
|
||||
# on it, and MySQL's lock_wait_timeout defaults to a YEAR, so the DDL
|
||||
# waits behind any open transaction effectively forever. Worse, MySQL
|
||||
# grants metadata locks in request order: every later reader queues
|
||||
# behind the pending exclusive, so one stuck migration stops the whole
|
||||
# site answering. Failing after a bounded wait turns a site-wide outage
|
||||
# into one failed enable the operator can retry.
|
||||
#
|
||||
# MySQL only - SQLite has no such statement and would reject it.
|
||||
connect_args = {}
|
||||
if db_url.startswith('mysql'):
|
||||
connect_args['init_command'] = (
|
||||
f'SET SESSION lock_wait_timeout = {MIGRATION_LOCK_WAIT_SECONDS}')
|
||||
connectable = create_engine(db_url, poolclass=pool.NullPool,
|
||||
connect_args=connect_args)
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection,
|
||||
|
||||
Reference in New Issue
Block a user