shopdb-flask/docs/adr/ADR-020-worker-self-recycle-on-plugin-change.md

ADR-020: Worker self-recycle when the enabled-plugin set changes

Status: Accepted Date: 2026-09-20 Amends: ADR-013 (see "Relationship to ADR-013")

Context

Enabling a plugin could not make its routes live in a running process. Flask refuses register_blueprint once the app has served its first request, and even without that rule each gunicorn worker holds its own app object, so registering in the worker that served the toggle would leave the other workers answering 404 for the same URL. The operator therefore had to restart the app by hand.

That restart was not merely inconvenient. Because every worker also holds its own PluginRegistry, loaded once at boot, the window between a toggle and a restart was a window in which workers disagreed about reality:

• a dependency check refused a plugin whose dependency another worker had just enabled
• a whole-file registry write from a stale snapshot erased other workers' changes, and the API answered 200 while doing it
• a health check that compared the registry against loaded plugins reported "no restart needed" while the routes 404'd

Decision

Flask's constraint stands. A plugin toggle still requires each serving process to restart. What changes is who initiates it.

1. Each worker fingerprints instance/plugins.json (content hash) at boot.
2. After serving a response, if the fingerprint has changed, the worker exits gracefully and its supervisor starts a replacement.
3. instance/plugins.json is the single cross-worker source of truth for the enabled set. It is written only through the locked merge-write in PluginRegistry._save. Per-worker registries are caches and must be refreshed from disk before any decision is made from them.
4. Recycles are staggered per worker, so the pool rolls rather than dropping at once, and are bounded by the supervisor's graceful timeout.
5. This applies only where a supervisor will return the process AND the exit path has been exercised. Today that is gunicorn. It never applies under the development server.
6. /setup/health describes the process that answered it. Callers must not infer fleet state from a single reply; confirm across several distinct workers.

This is a convergence mechanism, not a security control

Disabling a plugin has never stopped its code being imported. GET /api/plugins instantiates every plugin class on disk, core request handlers import plugin models gated on ImportError rather than on enabled state, and ADR-013:23 says plainly that "Disabled is not absent". Containment comes from the lean build (ADR-013) and from signature enforcement (PLUGIN-SIGNING.md).

A stale worker briefly serving a disabled plugin's routes is therefore a user experience artifact, not a breach, and nothing here should be cited as though it were a boundary.

Relationship to ADR-013

ADR-013 states that an enable "takes full effect only on restart, and the UI says so. There is no 'install button' that pretends otherwise", and that "the marketplace UX is honest about it rather than working around it".

That principle is kept, not reversed. The restart still happens; the process still reloads through the ordinary boot path, so verification and the import guard apply exactly as on a manual restart. What is removed is the manual step, on the one runtime where the mechanism is proven.

The honesty requirement is now explicit rather than implicit:

• Where a recycle will happen, the UI holds the plugin at "starting" until several distinct workers confirm it is live, and only then says "ready".
• Where a recycle will NOT happen, the UI keeps the old message and the operator restarts. Saying "restarting" on a runtime that does not is worse than the manual step it replaced.

Consequences

• A toggle takes effect in seconds without operator action, on gunicorn.
• Worker disagreement about the enabled set becomes transient rather than standing, which removes a class of lost-update and stale-read bugs.
• Windows/IIS (waitress behind HttpPlatformHandler) is single-process and unproven here, so it keeps the manual restart. Adding it requires testing on the Windows VM and handling HttpPlatformHandler's rapidFailsPerMinute, which stops relaunching after repeated exits.
• A worker recycling is indistinguishable from a worker dying, so the supervisor must be configured to replace workers. All shipped compose files and the Kubernetes manifest run gunicorn and satisfy this.
• Migrations run on their own connection with a bounded lock_wait_timeout, so a DDL that cannot take its metadata lock fails that one enable instead of queueing every later reader behind it.