From 3ac5ed258017a637c8c63f07722e14524b48058e Mon Sep 17 00:00:00 2001 From: cproudlock Date: Sat, 18 Jul 2026 17:40:29 -0400 Subject: [PATCH] Add ADR-013: plugin catalog, curated shelf, lean per-site builds Design record for distributing optional plugins across GE sites: a small mandatory core plus a catalog of optional plugins, packaged as signed versioned artifacts, served from a transport-agnostic read-only shelf (a SharePoint-synced or sneakernet folder - untrusted either way because every decision-bearing byte is signed), verified at adopt AND at every load and migrate. Lean per-site builds stage only chosen plugins into the backend image and SPA bundle. Status PROPOSED. Grounds the design in the real loader/contract/migration/ frontend code and records defects to fix along the way (upgrade-all migrating unadopted folders, enable-without-install, reverse-dep checks blind to unloaded plugins, missing cycle detection and dependency closure, hardcoded plugin imports in the SPA router). Honest on scope: the frontend re-org is the long pole (one core-router change plus per-plugin relocation), not a mechanical move. Phased 0-5 with schema-lean and runtime-JS delivery explicitly deferred. --- CLAUDE.md | 1 + .../ADR-013-plugin-catalog-and-lean-builds.md | 399 ++++++++++++++++++ docs/adr/README.md | 1 + 3 files changed, 401 insertions(+) create mode 100644 docs/adr/ADR-013-plugin-catalog-and-lean-builds.md diff --git a/CLAUDE.md b/CLAUDE.md index c709fde..76e22ef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,7 @@ Architecture decisions live in `docs/adr/`. Read those before making schema or c - ADR-010: Frontend plugin hook contract - ACCEPTED - ADR-011: Machines rename + modeltypes retyping - ACCEPTED - ADR-012: GE-Enforce manifest ownership in shopdb - ACCEPTED +- ADR-013: Plugin catalog, curated shelf, and lean per-site builds - PROPOSED ## Coding convention diff --git a/docs/adr/ADR-013-plugin-catalog-and-lean-builds.md b/docs/adr/ADR-013-plugin-catalog-and-lean-builds.md new file mode 100644 index 0000000..06b007d --- /dev/null +++ b/docs/adr/ADR-013-plugin-catalog-and-lean-builds.md @@ -0,0 +1,399 @@ +# ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds + +- Status: PROPOSED +- Date: 2026-07-18 +- Deciders: cproudlock +- Relates to: ADR-002 (contract versioning), ADR-003 (plugin distribution), ADR-004 (per-site instances), ADR-008 (per-plugin migrations), ADR-009 (frontend plugin gating), ADR-010 (frontend hook contract) + +## Context + +Every site today ships identical code. The backend image contains all 13 plugin +directories (Dockerfile COPY at line 55; the header comment listing "eleven core +plugins" is stale), and the SPA compiles every plugin's routes and views via a +static glob (frontend/src/router/index.js:11) plus hardcoded imports. A site's +"chosen set" exists only as runtime enable flags in instance/plugins.json. +Disabled is not absent: a site that never wants printedparts/usb/network still +ships, and can execute, that code. + +Distribution per ADR-003 is "drop a directory into plugins/ by hand". There is +no artifact format, no signing, no catalog, no validate gate, and the loader +trusts whatever it finds on disk (loader.py:61-73 discovers any folder with a +plugin.py; loader.py:185-224 loads it; migrations.py:29-62 runs its DDL with +full DB privileges). Plugins run in-process with the full shopdb.api surface +including db, so the only tenable security model on air-gapped GE networks is +curation plus cryptographic provenance, enforced everywhere code can execute, +not sandboxing. + +Known defects this ADR also resolves: + +- upgrade_all_plugins checks hasattr(registry, 'list_installed') which never + exists (registry has only get_all/get_enabled_plugins), so it always falls + back to migrating every folder on disk, adopted or not (__init__.py:94). +- PILOT-DEPLOY.md enables plugins that were never installed; enable refuses. + There is no declarative "apply this chosen set" operation. +- Reverse-dependency checks on uninstall/disable read only LOADED plugin + instances, so an installed-but-unloaded dependent is invisible. +- Dependency install/enable is check-only; nothing computes a closure, and + install-a-dependent can fail at its own load step because the dependency was + installed disabled (default_enabled=false on employees). +- The dependency sort has no cycle detection. +- Soft couplings (geenforce -> computers, notifications -> employees) are + invisible to the manifest graph. + +Frontend reality check (this drove the design below): plugin UI is NOT one +folder per plugin. Routes live in routes/.js (slides has none), inside +the shared core.js (computers report, printers toner report, employees detail, +slides settings), and as six hardcoded top-level imports in index.js itself +(/parts-kiosk, /tv, /print/printer-qr x2, /print/usb-labels, +/print/printedparts-labels). View dirs mismatch plugin names (computers -> +views/pcs). Plugin settings cards sit in shared views/settings/ +(DellWarrantySettings, ZabbixSettings, SlideManager, EmployeeDirectory, +MeasuringToolTypesList, PrintedPartsSettings), plugin print views in shared +views/print/, and some views span plugins (AssetLabel.vue serves five asset +types; PCDetail imports WarrantyPanel). Any lean-frontend design that only +moves routes/*.js and views// fails the build the moment a plugin is +pruned. This ADR scopes that work honestly instead of calling it mechanical. + +## Decision + +### 1. Tiering: mandatory core is the core package; all plugins are catalog-optional + +- The mandatory core is the non-plugin shopdb/core/ package (auth, users, + assets, locations, vendors, models, settings, audit, dashboard, search, + reports, plugin management). It already survives every plugin being absent + via hasattr/lazy-import guards. No plugin is promoted into it. +- New optional manifest field `tier: "core" | "optional"`, default "optional". + All 13 existing manifests are unchanged and unchanged in meaning. The + lifecycle gains a guard: uninstall_plugin and disable_plugin refuse a + tier:core plugin (alongside the reverse-dependency checks at + __init__.py:269-278 and :351-360). No plugin ships tier:core initially; the + field and guard exist so a future curation decision is a manifest edit, not a + framework change. +- Per-site mandates live in the site profile (section 5): a `locked` list the + profile applier refuses to remove. This preserves ADR-004 site autonomy: a + wing site can mandate usb without the framework mandating it fleet-wide. +- New manifest field `optional_dependencies: []` (names only, loader-ignored). + Declared for the verified soft couplings: geenforce lists computers + (service.py:32-43 loses app-detection gates without it), notifications lists + employees (routes.py:151-171 loses name/photo enrichment). Catalog listing + and adopt WARN on unmet optional deps; nothing blocks. +- Hard `dependencies` gains optional PEP440 ranges ("employees>=1.1"). + validate/adopt honor ranges; the runtime loader keeps name-only semantics + (specifier stripped) so no loader behavior changes. The single existing hard + edge printedparts -> employees stays as-is; whether it can relax to optional + (badges.py has an external HR fallback) is a follow-up product question, not + blocked on this ADR. +- Dependency plumbing fixes: _sort_by_dependencies gains cycle detection + (raise PluginDependencyError on a back edge); reverse-dependency checks read + manifests of ALL installed plugins from disk, not loaded instances. + +### 2. Packaging: signed, versioned artifacts + +Artifact: `-.shopdbplugin` (a zip of the plugin directory: +manifest.json, plugin.py, api/, models/, migrations/, and frontend/ once +section 6 lands) plus two members generated at pack time: + +- `PROVENANCE.json`: plugin name, version, publisher id, build timestamp, and + a sorted map of every packaged file path to its SHA-256. PROVENANCE.json is + not listed in its own map, so there is no circular-hash problem and no zip + canonicalization needed; determinism comes from sorted per-file hashes. +- `PROVENANCE.sig`: detached ed25519 signature over the exact PROVENANCE.json + bytes. + +New CLI: + +- `flask plugin pack --key ` (producer side): runs validate on the + directory, then emits the artifact. +- `flask plugin validate ` (the missing pre-publish gate), + fail-closed pipeline: signature (artifact mode) -> per-file hashes -> manifest + against a new docs/plugin-manifest.schema.json -> name == directory -> + core_version parses as a specifier and admits the target contract version -> + static import-surface scan reusing tests/test_plugin_contract.py logic -> + alembic versions parse. The schema types the known fields (name, version, + description, dependencies, optional_dependencies, tier, core_version, + api_prefix, display_name, default_enabled, provides, settings) and PERMITS + additional properties, so all 13 existing manifests pass unmodified. + +The import-surface scan is documented as a lint, not a security control; it is +trivially bypassed by dynamic import. The security control is human review +before signing (section 4). + +### 3. The shelf: a read-only folder, transport-agnostic by design + +- One config knob: `PLUGIN_SHELF_DIR`. The app only ever reads this folder. It + never speaks SharePoint, OneDrive, or any network protocol. +- Transport is explicitly out of scope and explicitly untrusted. On networks + that can reach corporate M365, a SharePoint document library sync populates + the folder. On strictly air-gapped floors where no sync agent can run, the + folder is populated by robocopy/USB. Both are equally supported and equally + untrusted, because every decision-bearing byte is signed: swapping transport + changes nothing about the trust model. +- Layout: `//-.shopdbplugin` plus + `shelf-index.json` and `shelf-index.sig`. +- The index is SIGNED with the same publisher key and carries a monotonically + increasing `serial` plus a `revoked` list of name-version pairs. Each site + records the last-seen serial in instance state and refuses an index with a + lower serial (anti-rollback of the catalog itself). The index also carries + per-entry version/tier/core_version so `flask plugin shelf list` can display + compatibility without unpacking, but the index is a BROWSE layer only: + adopt reads dependencies, tier, and core_version from the signed manifest + inside the verified artifact, never from the index. +- Trusted keys: `PLUGIN_TRUSTED_KEYS` is a list of pinned public keys delivered + out-of-band in the site's deployed config/image. Keys are NEVER read from the + shelf; a folder that can be written by an attacker must not also carry the + keys that authenticate it. Multiple pinned keys allow overlap rotation. + Revocation of an artifact rides the signed index `revoked` list; a + `flask plugin audit` command warns when an installed version appears there. +- Partial-sync robustness: adopt copies the artifact to a temp location, + verifies signature and every file hash there, then unpacks to + plugins/.staging/ and renames into place atomically. OneDrive + placeholder stubs, zero-byte files, or an index referencing not-yet-synced + artifacts all fail closed with a clear "artifact not fully synced/verified" + error. +- `flask plugin adopt [==version]`: resolve version from the shelf, + verify, compute the hard-dependency closure from signed manifests, then for + each closure member in topological order: unpack, INSTALL, and ENABLE (not + install-only; the load gate at loader.py:201-206 checks is_enabled, so an + install-only closure with default_enabled=false deps would fail its own + load). Migrations run via the unchanged per-plugin chain (ADR-008). Refuses + to adopt a version lower than the installed one unless + `--force-downgrade` is given interactively. Prints the restart notice. +- Adopt/install/uninstall remain CLI-only. The admin HTTP surface stays a + read-only catalog view plus the existing enable/disable toggle; because + Flask cannot register blueprints after the first request, any adopt or + enable takes full effect only on restart, and the UI says so. There is no + "install button" that pretends otherwise. + +### 4. Trust model: verify at adopt AND at every load and migrate + +Signing that gates only adoption is bypassable through every other write path +into plugins/ (git clone, symlink, USB drop) and defeated by post-adoption +tampering. Therefore verification is enforced where code executes: + +- Adoption leaves PROVENANCE.json and PROVENANCE.sig inside plugins// + and records publisher + artifact hash in the registry entry. +- load_plugin verifies the signature against PLUGIN_TRUSTED_KEYS and re-hashes + the plugin tree against the provenance file map BEFORE importing plugin.py + (new step ahead of loader.py:185). Missing or invalid provenance is a + fail-closed refusal in production. +- run_plugin_migrations performs the same verification before executing any + revision, so a routine `flask plugin upgrade-all` can never run DDL from an + unverified folder. +- upgrade_all_plugins iterates registry.get_all() (fixing the phantom + list_installed fallback at __init__.py:94), so unadopted on-disk folders are + never migrated as a side effect of deploys. +- Development and the ADR-003 external-repo/symlink workflow (including + scripts/test-external-plugin.sh) are preserved via `PLUGIN_DEV_TRUST_DIRS`, + honored ONLY when DEBUG or TESTING is set. Production ignores it. +- Cost: hashing 13 small plugin trees at boot is milliseconds; accepted. + +What signing does NOT claim: a valid signature proves the artifact is exactly +what a curator reviewed and signed, nothing more. Plugins remain in-process +Python with full DB access. The actual safety control is the human review +before signing; the signature makes that review's verdict tamper-evident all +the way to execution. + +### 5. Declarative site profiles and lean backend builds + +- `site-profile.json` per site (kept in the site's deploy config): site name, + list of chosen plugins, optional `locked` list. `flask plugin apply-profile + ` resolves the closure, installs AND enables in dependency order, runs + migrations, reports which changes need a restart. This replaces the + imperative CLI sequences in DEPLOY.md/PILOT-DEPLOY.md and fixes the + enable-without-install bug. +- Lean backend image: `scripts/build-site.sh` reads the profile and stages + only core + chosen plugin directories into the Docker build context + (correcting the Dockerfile COPY and its stale header comment). Discovery + needs no change; it already scans whatever exists. +- Prerequisite the naive version misses: core hardcodes plugin imports. + shopdb/core/api/search.py (~15 sites), reports.py, assets.py, collector.py, + applications.py, auditlogs.py, and shopdb/cli/__init__.py import + plugins..* lazily. Some already guard ImportError; ALL must, with + graceful degradation, before any site prunes a folder. This is audited and + enforced by a new CI job that deletes one plugin directory and runs the full + test suite (repeated per plugin). Longer term these aggregators should move + to registry-driven contract hooks (get_search_providers/get_report_sources) + so a new catalog plugin can join search/reports without core edits; that is + scoped as follow-up work, not a blocker for lean builds. +- Schema-lean is DEFERRED to its own ADR. The core baseline 68b3947ae14f + unconditionally creates the 10 pre-cutover plugins' tables, and lifting them + into plugin baselines collides with cross-plugin foreign keys (the + computers-owned installedapps table FKs machines.machineid while computers + declares no dependency on machines). Reversing the cutover would either + introduce undeclared hard deps or drop FKs; neither is decided here. A lean + site therefore carries a handful of empty pre-cutover tables. Accepted. + +### 6. Frontend delivery: Path C for rich UIs, Path A for simple ones, Path B rejected + +Path B (runtime-loaded JS / module federation) is REJECTED: it moves executable +UI delivery from a signed, statically auditable build artifact to runtime +fetching, which is exactly the wrong direction for an air-gapped, +review-then-sign posture, for zero benefit given restarts are already required. + +Path A (declarative JSON UI over generic renderers) is COMMITTED and scheduled +EARLY: the three unwired ADR-010 endpoints (pluginui.py asset-panels:62, +map-overlays:88, asset-presentation:100) get generic core renderers, joining +the already-consumed settings-cards. After this, a simple plugin ships JSON-only +UI with zero frontend build involvement. Sequencing this before the relocation +gives every plugin an escape hatch during the migration instead of after it. + +Path C (self-contained plugin frontend) is the primary mechanism, scoped +against the real code, not the idealized layout: + +- Canonical home: plugins//frontend/ containing routes.js (the plugin's + complete route array, INCLUDING routes currently embedded in index.js and + core.js), views/, and settings views. +- A pre-Vite staging step (scripts/stage-frontend.mjs, run by build-site.sh + and the dev script) copies the CHOSEN plugins' frontend/ into + frontend/src/.plugins-staged// (gitignored) and generates two files + inside the Vite root: routes.gen.js (aggregated plugin routes) and + meta.gen.js (plugin-supplied icon names, title spellings, settings-standalone + flags, replacing the hardcoded iconMap/TITLE_SPELLINGS/SETTINGS_STANDALONE in + AppLayout.vue, settingsCatalog.js, and index.js). This exists because + import.meta.glob requires a static literal inside the project root and + cannot select a per-site subset by itself. +- ONE-TIME core-router surgery, done first and called what it is: the six + hardcoded plugin-view imports in index.js (PartsKiosk, TVDashboard, + PrinterQRBatch/Single, USBLabelBatch, PrintedPartsLabels) and the plugin + routes embedded in core.js move into their owning plugins' routes.js. Without + this, pruning slides/printers/usb/printedparts fails the Vite build on + unresolvable imports; no amount of glob work fixes it. +- Per-plugin relocation PRs (13), each REAL WORK, not a file move: carve routes + out of shared files, move views (handling name mismatches like computers -> + views/pcs), move the plugin's settings views out of shared views/settings/, + and rewrite relative ../../ imports of core shared code to the @/ alias + (relative paths break at the staged depth). A lint rule enforces alias-only + core imports in plugin frontend code from then on. +- Shared plugin-aware code STAYS CORE and ships to every site: AssetLabel.vue + (spans five asset types), views/print helpers (assetLabel.js, qrLogo.js), + MachineBadge.vue, and cross-plugin panels like WarrantyPanel used by + PCDetail. These already null-guard or gate via isPluginEnabled and must keep + degrading when a peer plugin is absent; over time they migrate to ADR-010 + asset-panels so the data becomes plugin-supplied. Lean v1 therefore prunes + plugin-EXCLUSIVE code; a small plugin-aware core remainder is accepted and + shrinks as Path A absorbs it. +- Dual-location transition: the staging step unions legacy locations + (routes/*.js glob, views//) with plugins//frontend/ until each + plugin has moved. The SPA builds green at every commit; each plugin's move is + independently revertable until the legacy glob is removed at the end. +- Nav and settings cards are already server-driven (dashboardApi.navigation, + settings-cards); the remaining hardcoded plugin entries in settingsNav.js + (/settings/zabbix, /settings/dellwarranty) move to those plugins' + get_settings_cards so pruning leaves no dead links. + +### 7. Effect on the 13 existing plugins + +- Backend: ZERO code changes required. tier/optional_dependencies/provenance + are additive; pack zips the directory as-is; all plugins keep passing + tests/test_plugin_contract.py. Bundled plugins in a site's image get + provenance stamped at build time by pack, so verify-at-load applies to them + identically. +- Frontend: one relocation PR each, of the honest scope above. Until a + plugin's PR lands it keeps working from its legacy location. +- Operationally nothing changes for a site that does nothing: default builds + remain all-plugins, apply-profile is opt-in, and enable/disable semantics + (including the restart requirement) are unchanged. + +## Consequences + +### Positive + +- A real catalog: sites declare their set in site-profile.json and apply it in + one idempotent command; the chosen set drives backend image, SPA bundle, and + runtime state from one source of truth. +- Curated marketplace with end-to-end provenance: review -> sign -> any + transport -> verify at adopt, at load, and at migrate. Transport (SharePoint + sync or sneakernet) is untrusted and interchangeable, which is exactly right + for air-gapped sites. +- Lean per-site builds: unchosen plugins exist in neither the image nor the + bundle, shrinking attack surface and download size. +- Fixes shipped along the way: upgrade-all migrating unadopted folders, + PILOT-DEPLOY install/enable ordering, reverse-dep checks blind to unloaded + plugins, missing cycle detection, missing dependency closure, hardcoded + frontend plugin metadata. +- Path A completion makes simple plugins UI-capable with no build glue, which + is the cheapest possible marketplace onboarding. + +### Negative + +- Key management is a per-site operational burden: pinned keys delivered + out-of-band, rotation is a config change everywhere. Accepted as the price of + not trusting the distribution folder. +- The frontend re-org is the long pole: one core-router surgery plus 13 + non-trivial PRs. It is sequenced to be always-green and per-plugin + revertable, but it is weeks of work, not a rename. +- Schema is not lean: pre-cutover plugin tables still appear at every site + until the deferred baseline re-org ADR. +- Restarts remain required after adopt/enable (Flask blueprint constraint); + the marketplace UX is honest about it rather than working around it. +- Boot adds a signature + tree-hash check per enabled plugin (milliseconds, + but nonzero). + +### Risks + +- Key compromise or curation failure: a signature proves provenance, not + safety; a compromised pinned key or a rubber-stamp review signs malware that + every gate will happily pass. Mitigations: multi-key pinning with overlap + rotation, signed revocation list with monotonic index serial, and keeping the + signing key offline with the curator. The static import scan is a lint and + must never be presented as a boundary. +- Rollback/downgrade: mitigated three ways: index serial monotonicity, adopt + refusing version downgrades without interactive --force-downgrade, and the + signed revoked list. Residual risk: a site that never syncs a newer index + cannot learn of revocations; `flask plugin audit` at deploy time narrows the + window. +- Version skew across ADR-004 sites: one shelf serves sites at different + contract versions. Adopt checks core_version from the signed manifest against + the site's own __contract_version__ (authoritative); shelf list shows an + advisory compatibility column from the index. Incompatible artifacts are + listable but not adoptable. +- Partial/placeholder sync files: fail closed on hash verification; the error + message distinguishes "not fully synced" from "tampered" only by wording, + intentionally, since the app cannot tell. +- Dev-trust misuse: PLUGIN_DEV_TRUST_DIRS silently ignored outside + DEBUG/TESTING; a prod config carrying it gets a startup warning. +- Frontend closure drift: plugin views importing cross-plugin components is a + graph the manifest does not model. The lint rule (plugin frontend may import + core @/ paths and its own tree only, never another plugin's) prevents new + edges; existing shared plugin-aware code is explicitly core-owned. + +## Implementation phases + +- Phase 0, groundwork (small, days): upgrade_all_plugins uses + registry.get_all(); reverse-dep checks read installed manifests from disk; + cycle detection in _sort_by_dependencies; docs/plugin-manifest.schema.json + + `flask plugin validate` (directory mode); `flask plugin apply-profile` with + install+enable closure ordering; fix Dockerfile stale comment. All additive, + zero risk to running sites. +- Phase 1, packaging and signing (medium, about a week): PROVENANCE format, + `flask plugin pack`, validate artifact mode, PLUGIN_TRUSTED_KEYS config, + ed25519 signing tooling and curator docs. No runtime behavior change yet. +- Phase 2, shelf and enforcement (medium-large, one to two weeks): + PLUGIN_SHELF_DIR, signed shelf-index with serial + revoked list, + `flask plugin shelf list` / `adopt` / `audit` with atomic verified unpack; + verify-at-load in load_plugin and verify-at-migrate in + run_plugin_migrations, fail-closed in prod; PLUGIN_DEV_TRUST_DIRS for + dev/test and the external-repo harness; tier:core lifecycle guard; + provenance stamping of bundled plugins at build. This phase completes the + security model; everything after it is delivery optimization. +- Phase 3, Path A completion (medium, one to two weeks): generic renderers for + asset-panels, map-overlays, asset-presentation; migrate settingsNav.js + hardcoded plugin cards to get_settings_cards. Done BEFORE relocation so + JSON-only UI is available during the migration. +- Phase 4, frontend re-org (large, the long pole, several weeks elapsed): + stage-frontend.mjs staging + routes.gen.js/meta.gen.js codegen; ONE core PR + moving the six index.js hardcoded plugin imports and the core.js-embedded + plugin routes into plugin route files; then 13 per-plugin relocation PRs + (views, settings views, name-mismatch dirs, @/ alias rewrite) under the + dual-location union; lint rule for plugin frontend imports. Always-green, + per-plugin revertable. +- Phase 5, lean builds end to end (medium, about a week after Phase 4): + build-site.sh staging backend dirs + frontend staging from site-profile.json; + core lazy-import guard audit finished, enforced by the delete-a-plugin CI + matrix; remove the legacy glob; pilot one real lean site (a location without + printedparts/usb/network) and diff its image and bundle against a full build. + +Deferred, each to its own future decision: schema-lean core-baseline re-org +(blocked on the installedapps -> machines FK question), pip/entry-point +distribution (ADR-003 v2), hook-based search/report aggregation contract, and +any revisit of Path B. diff --git a/docs/adr/README.md b/docs/adr/README.md index 453eac8..7c27c19 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -25,6 +25,7 @@ Each ADR captures a single architectural decision: the context, the decision its | [010](ADR-010-frontend-plugin-hooks.md) | Frontend plugin hook contract | ACCEPTED | | [011](ADR-011-machines-rename.md) | Machines rename + modeltypes retyping | ACCEPTED | | [012](ADR-012-geenforce-manifest-ownership.md) | GE-Enforce manifest ownership in shopdb | ACCEPTED | +| [013](ADR-013-plugin-catalog-and-lean-builds.md) | Plugin catalog, curated shelf, and lean per-site builds | PROPOSED | ## Authoring