Files

cproudlock 9b7da9f2d2 Make a plugin toggle take effect without a manual restart

Enabling a plugin could not make its routes live. Flask refuses
register_blueprint once the app has served a request, and each gunicorn worker
holds its own app object, so registering in the worker that served the toggle
would leave the others answering 404 for the same URL.

Worse, the window before the restart was a window in which workers disagreed
about reality. Four defects, all of which answered 200 while going wrong:

- Enabling a not-yet-installed plugin ALWAYS 500'd on a running app:
  install_plugin ran the migrations, updated the registry, and only then called
  register_blueprint and hit Flask's assertion. The toggle half-applied and
  reported failure.
- Enabling one plugin could hang the entire site. @jwt_required does SELECT
  users, so the request's transaction sits open holding a shared metadata lock;
  the plugin's migration runs on its own Alembic connection and needs an
  EXCLUSIVE lock on users for a foreign key. Each waits on the other, and MySQL
  grants metadata locks in request order, so every later login queued behind the
  pending exclusive. Observed twice as a total outage from one plugin toggle.
- Each worker rewrote the WHOLE registry file from a snapshot taken at boot, so
  concurrent enables erased each other: seven enables persisted three, with
  HTTP 200 for all seven.
- Dependency checks read that same stale snapshot, refusing a plugin whose
  dependency another worker had just enabled.

Routes still require the process to restart; what changes is who initiates it.
Each worker fingerprints instance/plugins.json at boot and exits gracefully when
it changes, and the supervisor replaces it. That is the primitive gunicorn's own
--reload uses. Recycles are staggered per worker so the pool rolls rather than
dropping at once, and the exit is scoped to gunicorn: on Windows sys.exit() in a
daemon thread ends the thread rather than the process, so IIS keeps the manual
restart until that path is proven on the Windows VM.

The registry now merges under an exclusive lock and replaces atomically, saves
only the entry being changed, and re-reads before any decision is made from it.

ADR-020 records the decision; ADR-013 is amended where it said the UI must tell
the operator to restart. The restart requirement and the honesty requirement
both stand - only the manual step is removed, and only where it is proven.

(cherry picked from commit 9d6342b3ab)

2026-09-20 09:07:54 -04:00

4.6 KiB

Raw Blame History

ADR-020: Worker self-recycle when the enabled-plugin set changes

Status: Accepted Date: 2026-09-20 Amends: ADR-013 (see "Relationship to ADR-013")

Context

Enabling a plugin could not make its routes live in a running process. Flask refuses register_blueprint once the app has served its first request, and even without that rule each gunicorn worker holds its own app object, so registering in the worker that served the toggle would leave the other workers answering 404 for the same URL. The operator therefore had to restart the app by hand.

That restart was not merely inconvenient. Because every worker also holds its own PluginRegistry, loaded once at boot, the window between a toggle and a restart was a window in which workers disagreed about reality:

a dependency check refused a plugin whose dependency another worker had just enabled
a whole-file registry write from a stale snapshot erased other workers' changes, and the API answered 200 while doing it
a health check that compared the registry against loaded plugins reported "no restart needed" while the routes 404'd

Decision

Flask's constraint stands. A plugin toggle still requires each serving process to restart. What changes is who initiates it.

Each worker fingerprints instance/plugins.json (content hash) at boot.
After serving a response, if the fingerprint has changed, the worker exits gracefully and its supervisor starts a replacement.
instance/plugins.json is the single cross-worker source of truth for the enabled set. It is written only through the locked merge-write in PluginRegistry._save. Per-worker registries are caches and must be refreshed from disk before any decision is made from them.
Recycles are staggered per worker, so the pool rolls rather than dropping at once, and are bounded by the supervisor's graceful timeout.
This applies only where a supervisor will return the process AND the exit path has been exercised. Today that is gunicorn. It never applies under the development server.
/setup/health describes the process that answered it. Callers must not infer fleet state from a single reply; confirm across several distinct workers.

This is a convergence mechanism, not a security control

Disabling a plugin has never stopped its code being imported. GET /api/plugins instantiates every plugin class on disk, core request handlers import plugin models gated on ImportError rather than on enabled state, and ADR-013:23 says plainly that "Disabled is not absent". Containment comes from the lean build (ADR-013) and from signature enforcement (PLUGIN-SIGNING.md).

A stale worker briefly serving a disabled plugin's routes is therefore a user experience artifact, not a breach, and nothing here should be cited as though it were a boundary.

Relationship to ADR-013

ADR-013 states that an enable "takes full effect only on restart, and the UI says so. There is no 'install button' that pretends otherwise", and that "the marketplace UX is honest about it rather than working around it".

That principle is kept, not reversed. The restart still happens; the process still reloads through the ordinary boot path, so verification and the import guard apply exactly as on a manual restart. What is removed is the manual step, on the one runtime where the mechanism is proven.

The honesty requirement is now explicit rather than implicit:

Where a recycle will happen, the UI holds the plugin at "starting" until several distinct workers confirm it is live, and only then says "ready".
Where a recycle will NOT happen, the UI keeps the old message and the operator restarts. Saying "restarting" on a runtime that does not is worse than the manual step it replaced.

Consequences

A toggle takes effect in seconds without operator action, on gunicorn.
Worker disagreement about the enabled set becomes transient rather than standing, which removes a class of lost-update and stale-read bugs.
Windows/IIS (waitress behind HttpPlatformHandler) is single-process and unproven here, so it keeps the manual restart. Adding it requires testing on the Windows VM and handling HttpPlatformHandler's rapidFailsPerMinute, which stops relaunching after repeated exits.
A worker recycling is indistinguishable from a worker dying, so the supervisor must be configured to replace workers. All shipped compose files and the Kubernetes manifest run gunicorn and satisfy this.
Migrations run on their own connection with a bounded lock_wait_timeout, so a DDL that cannot take its metadata lock fails that one enable instead of queueing every later reader behind it.

4.6 KiB Raw Blame History