ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make a plugin toggle take effect without a manual restart

Browse Source 
Enabling a plugin could not make its routes live. Flask refuses
register_blueprint once the app has served a request, and each gunicorn worker
holds its own app object, so registering in the worker that served the toggle
would leave the others answering 404 for the same URL.

Worse, the window before the restart was a window in which workers disagreed
about reality. Four defects, all of which answered 200 while going wrong:

- Enabling a not-yet-installed plugin ALWAYS 500'd on a running app:
  install_plugin ran the migrations, updated the registry, and only then called
  register_blueprint and hit Flask's assertion. The toggle half-applied and
  reported failure.
- Enabling one plugin could hang the entire site. @jwt_required does SELECT
  users, so the request's transaction sits open holding a shared metadata lock;
  the plugin's migration runs on its own Alembic connection and needs an
  EXCLUSIVE lock on users for a foreign key. Each waits on the other, and MySQL
  grants metadata locks in request order, so every later login queued behind the
  pending exclusive. Observed twice as a total outage from one plugin toggle.
- Each worker rewrote the WHOLE registry file from a snapshot taken at boot, so
  concurrent enables erased each other: seven enables persisted three, with
  HTTP 200 for all seven.
- Dependency checks read that same stale snapshot, refusing a plugin whose
  dependency another worker had just enabled.

Routes still require the process to restart; what changes is who initiates it.
Each worker fingerprints instance/plugins.json at boot and exits gracefully when
it changes, and the supervisor replaces it. That is the primitive gunicorn's own
--reload uses. Recycles are staggered per worker so the pool rolls rather than
dropping at once, and the exit is scoped to gunicorn: on Windows sys.exit() in a
daemon thread ends the thread rather than the process, so IIS keeps the manual
restart until that path is proven on the Windows VM.

The registry now merges under an exclusive lock and replaces atomically, saves
only the entry being changed, and re-reads before any decision is made from it.

ADR-020 records the decision; ADR-013 is amended where it said the UI must tell
the operator to restart. The restart requirement and the honesty requirement
both stand - only the manual step is removed, and only where it is proven.

(cherry picked from commit 9d6342b3ab)
This commit is contained in:
cproudlock
2026-09-20 09:06:03 -04:00
parent 0eed80f3b9
commit 9b7da9f2d2
10 changed files with 604 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/adr/ADR-013-plugin-catalog-and-lean-builds.md
View File

@@ -171,6 +171,11 @@ before signing (section 4).
Flask cannot register blueprints after the first request, any adopt or
enable takes full effect only on restart, and the UI says so. There is no
"install button" that pretends otherwise.
AMENDED by ADR-020: the restart requirement is unchanged, but on gunicorn the
worker now initiates it rather than the operator. The honesty requirement is
kept and made explicit - the UI holds a plugin at "starting" until several
distinct workers confirm it is live, and on runtimes where no self-recycle
happens (Windows/IIS) it still tells the operator to restart.
### 4. Trust model: verify at adopt AND at every load and migrate
@@ -334,6 +339,9 @@ against the real code, not the idealized layout:
until the deferred baseline re-org ADR.
- Restarts remain required after adopt/enable (Flask blueprint constraint);
the marketplace UX is honest about it rather than working around it.
See ADR-020: the restart is now self-initiated per worker on gunicorn. The
constraint and the honesty both stand; only the manual step is removed, and
only where the mechanism is proven.
- Boot adds a signature + tree-hash check per enabled plugin (milliseconds,
but nonzero).

92
docs/adr/ADR-020-worker-self-recycle-on-plugin-change.md Normal file
View File

@@ -0,0 +1,92 @@
# ADR-020: Worker self-recycle when the enabled-plugin set changes
Status: Accepted
Date: 2026-09-20
Amends: ADR-013 (see "Relationship to ADR-013")
## Context
Enabling a plugin could not make its routes live in a running process. Flask
refuses `register_blueprint` once the app has served its first request, and even
without that rule each gunicorn worker holds its own app object, so registering
in the worker that served the toggle would leave the other workers answering 404
for the same URL. The operator therefore had to restart the app by hand.
That restart was not merely inconvenient. Because every worker also holds its own
`PluginRegistry`, loaded once at boot, the window between a toggle and a restart
was a window in which workers disagreed about reality:
- a dependency check refused a plugin whose dependency another worker had just
enabled
- a whole-file registry write from a stale snapshot erased other workers'
changes, and the API answered 200 while doing it
- a health check that compared the registry against loaded plugins reported "no
restart needed" while the routes 404'd
## Decision
Flask's constraint stands. A plugin toggle still requires each serving process to
restart. What changes is who initiates it.
1. Each worker fingerprints `instance/plugins.json` (content hash) at boot.
2. After serving a response, if the fingerprint has changed, the worker exits
gracefully and its supervisor starts a replacement.
3. `instance/plugins.json` is the single cross-worker source of truth for the
enabled set. It is written only through the locked merge-write in
`PluginRegistry._save`. Per-worker registries are caches and must be
refreshed from disk before any decision is made from them.
4. Recycles are staggered per worker, so the pool rolls rather than dropping at
once, and are bounded by the supervisor's graceful timeout.
5. This applies only where a supervisor will return the process AND the exit
path has been exercised. Today that is gunicorn. It never applies under the
development server.
6. `/setup/health` describes the process that answered it. Callers must not
infer fleet state from a single reply; confirm across several distinct
workers.
## This is a convergence mechanism, not a security control
Disabling a plugin has never stopped its code being imported. `GET /api/plugins`
instantiates every plugin class on disk, core request handlers import plugin
models gated on `ImportError` rather than on enabled state, and ADR-013:23 says
plainly that "Disabled is not absent". Containment comes from the lean build
(ADR-013) and from signature enforcement (PLUGIN-SIGNING.md).
A stale worker briefly serving a disabled plugin's routes is therefore a user
experience artifact, not a breach, and nothing here should be cited as though it
were a boundary.
## Relationship to ADR-013
ADR-013 states that an enable "takes full effect only on restart, and the UI says
so. There is no 'install button' that pretends otherwise", and that "the
marketplace UX is honest about it rather than working around it".
That principle is kept, not reversed. The restart still happens; the process
still reloads through the ordinary boot path, so verification and the import
guard apply exactly as on a manual restart. What is removed is the manual step,
on the one runtime where the mechanism is proven.
The honesty requirement is now explicit rather than implicit:
- Where a recycle will happen, the UI holds the plugin at "starting" until
several distinct workers confirm it is live, and only then says "ready".
- Where a recycle will NOT happen, the UI keeps the old message and the operator
restarts. Saying "restarting" on a runtime that does not is worse than the
manual step it replaced.
## Consequences
- A toggle takes effect in seconds without operator action, on gunicorn.
- Worker disagreement about the enabled set becomes transient rather than
standing, which removes a class of lost-update and stale-read bugs.
- Windows/IIS (waitress behind HttpPlatformHandler) is single-process and
unproven here, so it keeps the manual restart. Adding it requires testing on
the Windows VM and handling HttpPlatformHandler's `rapidFailsPerMinute`, which
stops relaunching after repeated exits.
- A worker recycling is indistinguishable from a worker dying, so the supervisor
must be configured to replace workers. All shipped compose files and the
Kubernetes manifest run gunicorn and satisfy this.
- Migrations run on their own connection with a bounded `lock_wait_timeout`, so
a DDL that cannot take its metadata lock fails that one enable instead of
queueing every later reader behind it.

7
docs/adr/README.md
View File

@@ -31,14 +31,15 @@ anything else.
| [009](ADR-009-frontend-plugin-gating.md) | What happens when someone opens a URL belonging to a disabled plugin? | In force |
| [010](ADR-010-frontend-plugin-hooks.md) | How does a plugin add UI without editing core Vue files? | In force |
| [011](ADR-011-machines-rename.md) | Why is the code called `machines` and what happened to `machinetypes`? | In force |
| [012](ADR-012-geenforce-manifest-ownership.md) | Does ShopDB own GE-Enforce manifests, and does it own the engine? | In force |
| [012](ADR-012-geenforce-manifest-ownership.md) | Does ShopDB own GE-Enforce manifests, and does it own the engine? | **Record only on this line.** The plugin it decided is not bundled here; ADR-013 and the API-token resource scopes trace back to it |
| [013](ADR-013-plugin-catalog-and-lean-builds.md) | How does a site ship only the plugins it chose, and how is plugin code trusted? | In force |
| [014](ADR-014-schema-lean-per-site.md) | Does a lean site still create tables for plugins it never installed? | In force |
| [015](ADR-015-site-specific-configuration.md) | Where may one site's hostnames, shares and defaults live? | In force |
| [016](ADR-016-credential-delivery.md) | How does a bay get a controller credential without a cleartext file on a share? | **Superseded by [019](ADR-019-per-pc-endpoint-credentials.md), and never built.** Do not build against it |
| [017](ADR-017-buildings-and-levels.md) | How does the floor map hold more than one floor and more than one building? | In force |
| [018](ADR-018-metrology-instruments-are-machines.md) | Where do CMMs and vision systems live now that `measuringtools` is retired? | In force |
| [019](ADR-019-per-pc-endpoint-credentials.md) | How does the server know WHICH PC is calling, and what replaced the fleet-wide key? | In force. Built, not yet merged into this tree |
| [018](ADR-018-metrology-instruments-are-machines.md) | Where do CMMs and vision systems live now that `measuringtools` is retired? | In force, and this line is the one it describes: there is no measuring-tool asset class |
| [019](ADR-019-per-pc-endpoint-credentials.md) | How does the server know WHICH PC is calling, and what replaced the fleet-wide key? | In force |
| [020](ADR-020-worker-self-recycle-on-plugin-change.md) | How does enabling a plugin take effect without the operator restarting the app? | A worker fingerprints plugins.json at boot and exits when it changes; the supervisor replaces it. Amends [013](ADR-013-plugin-catalog-and-lean-builds.md) |
Related decisions that are not ADRs, kept as records of why something was built:
[docs/proposals](../proposals/README.md).