shopdb-flask/docs/adr/ADR-013-plugin-catalog-and-lean-builds.md

ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds

• Status: PROPOSED
• Date: 2026-07-18
• Deciders: ShopDB maintainers
• Relates to: ADR-002 (contract versioning), ADR-003 (plugin distribution), ADR-004 (per-site instances), ADR-008 (per-plugin migrations), ADR-009 (frontend plugin gating), ADR-010 (frontend hook contract)

Context

Every site today ships identical code. The backend image contains all 13 plugin directories (Dockerfile COPY at line 55; the header comment listing "eleven core plugins" is stale), and the SPA compiles every plugin's routes and views via a static glob (frontend/src/router/index.js:11) plus hardcoded imports. A site's "chosen set" exists only as runtime enable flags in instance/plugins.json. Disabled is not absent: a site that never wants printedparts/usb/network still ships, and can execute, that code.

Distribution per ADR-003 is "drop a directory into plugins/ by hand". There is no artifact format, no signing, no catalog, no validate gate, and the loader trusts whatever it finds on disk (loader.py:61-73 discovers any folder with a plugin.py; loader.py:185-224 loads it; migrations.py:29-62 runs its DDL with full DB privileges). Plugins run in-process with the full shopdb.api surface including db, so the only tenable security model on air-gapped GE networks is curation plus cryptographic provenance, enforced everywhere code can execute, not sandboxing.

Known defects this ADR also resolves:

• upgrade_all_plugins checks hasattr(registry, 'list_installed') which never exists (registry has only get_all/get_enabled_plugins), so it always falls back to migrating every folder on disk, adopted or not (init.py:94).
• PILOT-DEPLOY.md enables plugins that were never installed; enable refuses. There is no declarative "apply this chosen set" operation.
• Reverse-dependency checks on uninstall/disable read only LOADED plugin instances, so an installed-but-unloaded dependent is invisible.
• Dependency install/enable is check-only; nothing computes a closure, and install-a-dependent can fail at its own load step because the dependency was installed disabled (default_enabled=false on employees).
• The dependency sort has no cycle detection.
• Soft couplings (geenforce -> computers, notifications -> employees) are invisible to the manifest graph.

Frontend reality check (this drove the design below): plugin UI is NOT one folder per plugin. Routes live in routes/.js (slides has none), inside the shared core.js (computers report, printers toner report, employees detail, slides settings), and as six hardcoded top-level imports in index.js itself (/parts-kiosk, /tv, /print/printer-qr x2, /print/usb-labels, /print/printedparts-labels). View dirs mismatch plugin names (computers -> views/pcs). Plugin settings cards sit in shared views/settings/ (DellWarrantySettings, ZabbixSettings, SlideManager, EmployeeDirectory, MeasuringToolTypesList, PrintedPartsSettings), plugin print views in shared views/print/, and some views span plugins (AssetLabel.vue serves five asset types; PCDetail imports WarrantyPanel). Any lean-frontend design that only moves routes/*.js and views// fails the build the moment a plugin is pruned. This ADR scopes that work honestly instead of calling it mechanical.

Decision

1. Tiering: mandatory core is the core package; all plugins are catalog-optional

• The mandatory core is the non-plugin shopdb/core/ package (auth, users, assets, locations, vendors, models, settings, audit, dashboard, search, reports, plugin management). It already survives every plugin being absent via hasattr/lazy-import guards. No plugin is promoted into it.
• New optional manifest field tier: "core" | "optional", default "optional". All 13 existing manifests are unchanged and unchanged in meaning. The lifecycle gains a guard: uninstall_plugin and disable_plugin refuse a tier:core plugin (alongside the reverse-dependency checks at init.py:269-278 and :351-360). No plugin ships tier:core initially; the field and guard exist so a future curation decision is a manifest edit, not a framework change.
• Per-site mandates live in the site profile (section 5): a locked list the profile applier refuses to remove. This preserves ADR-004 site autonomy: a wing site can mandate usb without the framework mandating it fleet-wide.
• New manifest field optional_dependencies: [] (names only, loader-ignored). Declared for the verified soft couplings: geenforce lists computers (service.py:32-43 loses app-detection gates without it), notifications lists employees (routes.py:151-171 loses name/photo enrichment). Catalog listing and adopt WARN on unmet optional deps; nothing blocks.
• Hard dependencies gains optional PEP440 ranges ("employees>=1.1"). validate/adopt honor ranges; the runtime loader keeps name-only semantics (specifier stripped) so no loader behavior changes. The single existing hard edge printedparts -> employees stays as-is; whether it can relax to optional (badges.py has an external HR fallback) is a follow-up product question, not blocked on this ADR.
• Dependency plumbing fixes: _sort_by_dependencies gains cycle detection (raise PluginDependencyError on a back edge); reverse-dependency checks read manifests of ALL installed plugins from disk, not loaded instances.

2. Packaging: signed, versioned artifacts

Artifact: <name>-<version>.shopdbplugin (a zip of the plugin directory: manifest.json, plugin.py, api/, models/, migrations/, and frontend/ once section 6 lands) plus two members generated at pack time:

• PROVENANCE.json: plugin name, version, publisher id, build timestamp, and a sorted map of every packaged file path to its SHA-256. PROVENANCE.json is not listed in its own map, so there is no circular-hash problem and no zip canonicalization needed; determinism comes from sorted per-file hashes.
• PROVENANCE.sig: detached ed25519 signature over the exact PROVENANCE.json bytes.

New CLI:

• flask plugin pack <name> --key <path> (producer side): runs validate on the directory, then emits the artifact.
• flask plugin validate <dir|artifact> (the missing pre-publish gate), fail-closed pipeline: signature (artifact mode) -> per-file hashes -> manifest against a new docs/plugin-manifest.schema.json -> name == directory -> core_version parses as a specifier and admits the target contract version -> static import-surface scan reusing tests/test_plugin_contract.py logic -> alembic versions parse. The schema types the known fields (name, version, description, dependencies, optional_dependencies, tier, core_version, api_prefix, display_name, default_enabled, provides, settings) and PERMITS additional properties, so all 13 existing manifests pass unmodified.

The import-surface scan is documented as a lint, not a security control; it is trivially bypassed by dynamic import. The security control is human review before signing (section 4).

3. The shelf: a read-only folder, transport-agnostic by design

• One config knob: PLUGIN_SHELF_DIR. The app only ever reads this folder. It never speaks SharePoint, OneDrive, or any network protocol.
• Transport is explicitly out of scope and explicitly untrusted. On networks that can reach corporate M365, a SharePoint document library sync populates the folder. On strictly air-gapped floors where no sync agent can run, the folder is populated by robocopy/USB. Both are equally supported and equally untrusted, because every decision-bearing byte is signed: swapping transport changes nothing about the trust model.
• Layout: <shelf>/<name>/<name>-<version>.shopdbplugin plus shelf-index.json and shelf-index.sig.
• The index is SIGNED with the same publisher key and carries a monotonically increasing serial plus a revoked list of name-version pairs. Each site records the last-seen serial in instance state and refuses an index with a lower serial (anti-rollback of the catalog itself). The index also carries per-entry version/tier/core_version so flask plugin shelf-list can display compatibility without unpacking, but the index is a BROWSE layer only: adopt reads dependencies, tier, and core_version from the signed manifest inside the verified artifact, never from the index.
• Trusted keys: PLUGIN_TRUSTED_KEYS is a list of pinned public keys delivered out-of-band in the site's deployed config/image. Keys are NEVER read from the shelf; a folder that can be written by an attacker must not also carry the keys that authenticate it. Multiple pinned keys allow overlap rotation. Revocation of an artifact rides the signed index revoked list; a flask plugin audit command warns when an installed version appears there.
• Partial-sync robustness: adopt copies the artifact to a temp location, verifies signature and every file hash there, then unpacks to plugins/.staging/ and renames into place atomically. OneDrive placeholder stubs, zero-byte files, or an index referencing not-yet-synced artifacts all fail closed with a clear "artifact not fully synced/verified" error.
• flask plugin adopt <name>[==version]: resolve version from the shelf, verify, compute the hard-dependency closure from signed manifests, then for each closure member in topological order: unpack, INSTALL, and ENABLE (not install-only; the load gate at loader.py:201-206 checks is_enabled, so an install-only closure with default_enabled=false deps would fail its own load). Migrations run via the unchanged per-plugin chain (ADR-008). Refuses to adopt a version lower than the installed one unless --force-downgrade is given interactively. Prints the restart notice.
• Adopt/install/uninstall remain CLI-only. The admin HTTP surface stays a read-only catalog view plus the existing enable/disable toggle; because Flask cannot register blueprints after the first request, any adopt or enable takes full effect only on restart, and the UI says so. There is no "install button" that pretends otherwise.

4. Trust model: verify at adopt AND at every load and migrate

Signing that gates only adoption is bypassable through every other write path into plugins/ (git clone, symlink, USB drop) and defeated by post-adoption tampering. Therefore verification is enforced where code executes:

• Adoption leaves PROVENANCE.json and PROVENANCE.sig inside plugins// and records publisher + artifact hash in the registry entry.
• load_plugin verifies the signature against PLUGIN_TRUSTED_KEYS and re-hashes the plugin tree against the provenance file map BEFORE importing plugin.py (new step ahead of loader.py:185). Missing or invalid provenance is a fail-closed refusal in production.
• run_plugin_migrations performs the same verification before executing any revision, so a routine flask plugin upgrade-all can never run DDL from an unverified folder.
• upgrade_all_plugins iterates registry.get_all() (fixing the phantom list_installed fallback at init.py:94), so unadopted on-disk folders are never migrated as a side effect of deploys.
• Development and the ADR-003 external-repo/symlink workflow (including scripts/test-external-plugin.sh) are preserved via PLUGIN_DEV_TRUST_DIRS, honored ONLY when DEBUG or TESTING is set. Production ignores it.
• Cost: hashing 13 small plugin trees at boot is milliseconds; accepted.

What signing does NOT claim: a valid signature proves the artifact is exactly what a curator reviewed and signed, nothing more. Plugins remain in-process Python with full DB access. The actual safety control is the human review before signing; the signature makes that review's verdict tamper-evident all the way to execution.

5. Declarative site profiles and lean backend builds

• site-profile.json per site (kept in the site's deploy config): site name, list of chosen plugins, optional locked list. flask plugin apply-profile <file> resolves the closure, installs AND enables in dependency order, runs migrations, reports which changes need a restart. This replaces the imperative CLI sequences in DEPLOY.md/PILOT-DEPLOY.md and fixes the enable-without-install bug.
• Lean backend image: scripts/build-site.sh reads the profile and stages only core + chosen plugin directories into the Docker build context (correcting the Dockerfile COPY and its stale header comment). Discovery needs no change; it already scans whatever exists.
• Prerequisite the naive version misses: core hardcodes plugin imports. shopdb/core/api/search.py (~15 sites), reports.py, assets.py, collector.py, applications.py, auditlogs.py, and shopdb/cli/init.py import plugins..* lazily. Some already guard ImportError; ALL must, with graceful degradation, before any site prunes a folder. This is audited and enforced by a new CI job that deletes one plugin directory and runs the full test suite (repeated per plugin). Longer term these aggregators should move to registry-driven contract hooks (get_search_providers/get_report_sources) so a new catalog plugin can join search/reports without core edits; that is scoped as follow-up work, not a blocker for lean builds.
• Schema-lean is DEFERRED to its own ADR. The core baseline 68b3947ae14f unconditionally creates the 10 pre-cutover plugins' tables, and lifting them into plugin baselines collides with cross-plugin foreign keys (the computers-owned installedapps table FKs machines.machineid while computers declares no dependency on machines). Reversing the cutover would either introduce undeclared hard deps or drop FKs; neither is decided here. A lean site therefore carries a handful of empty pre-cutover tables. Accepted.

6. Frontend delivery: Path C for rich UIs, Path A for simple ones, Path B rejected

Path B (runtime-loaded JS / module federation) is REJECTED: it moves executable UI delivery from a signed, statically auditable build artifact to runtime fetching, which is exactly the wrong direction for an air-gapped, review-then-sign posture, for zero benefit given restarts are already required.

Path A (declarative JSON UI over generic renderers) is COMMITTED and scheduled EARLY: the three unwired ADR-010 endpoints (pluginui.py asset-panels:62, map-overlays:88, asset-presentation:100) get generic core renderers, joining the already-consumed settings-cards. After this, a simple plugin ships JSON-only UI with zero frontend build involvement. Sequencing this before the relocation gives every plugin an escape hatch during the migration instead of after it.

Path C (self-contained plugin frontend) is the primary mechanism, scoped against the real code, not the idealized layout:

• Canonical home: plugins//frontend/ containing routes.js (the plugin's complete route array, INCLUDING routes currently embedded in index.js and core.js), views/, and settings views.
• A pre-Vite staging step (scripts/stage-frontend.mjs, run by build-site.sh and the dev script) copies the CHOSEN plugins' frontend/ into frontend/src/.plugins-staged// (gitignored) and generates two files inside the Vite root: routes.gen.js (aggregated plugin routes) and meta.gen.js (plugin-supplied icon names, title spellings, settings-standalone flags, replacing the hardcoded iconMap/TITLE_SPELLINGS/SETTINGS_STANDALONE in AppLayout.vue, settingsCatalog.js, and index.js). This exists because import.meta.glob requires a static literal inside the project root and cannot select a per-site subset by itself.
• ONE-TIME core-router surgery, done first and called what it is: the six hardcoded plugin-view imports in index.js (PartsKiosk, TVDashboard, PrinterQRBatch/Single, USBLabelBatch, PrintedPartsLabels) and the plugin routes embedded in core.js move into their owning plugins' routes.js. Without this, pruning slides/printers/usb/printedparts fails the Vite build on unresolvable imports; no amount of glob work fixes it.
• Per-plugin relocation PRs (13), each REAL WORK, not a file move: carve routes out of shared files, move views (handling name mismatches like computers -> views/pcs), move the plugin's settings views out of shared views/settings/, and rewrite relative ../../ imports of core shared code to the @/ alias (relative paths break at the staged depth). A lint rule enforces alias-only core imports in plugin frontend code from then on.
• Shared plugin-aware code STAYS CORE and ships to every site: AssetLabel.vue (spans five asset types), views/print helpers (assetLabel.js, qrLogo.js), MachineBadge.vue, and cross-plugin panels like WarrantyPanel used by PCDetail. These already null-guard or gate via isPluginEnabled and must keep degrading when a peer plugin is absent; over time they migrate to ADR-010 asset-panels so the data becomes plugin-supplied. Lean v1 therefore prunes plugin-EXCLUSIVE code; a small plugin-aware core remainder is accepted and shrinks as Path A absorbs it.
• Dual-location transition: the staging step unions legacy locations (routes/*.js glob, views//) with plugins//frontend/ until each plugin has moved. The SPA builds green at every commit; each plugin's move is independently revertable until the legacy glob is removed at the end.
• Nav and settings cards are already server-driven (dashboardApi.navigation, settings-cards); the remaining hardcoded plugin entries in settingsNav.js (/settings/zabbix, /settings/dellwarranty) move to those plugins' get_settings_cards so pruning leaves no dead links.

7. Effect on the 13 existing plugins

• Backend: ZERO code changes required. tier/optional_dependencies/provenance are additive; pack zips the directory as-is; all plugins keep passing tests/test_plugin_contract.py. Bundled plugins in a site's image get provenance stamped at build time by pack, so verify-at-load applies to them identically.
• Frontend: one relocation PR each, of the honest scope above. Until a plugin's PR lands it keeps working from its legacy location.
• Operationally nothing changes for a site that does nothing: default builds remain all-plugins, apply-profile is opt-in, and enable/disable semantics (including the restart requirement) are unchanged.

Consequences

Positive

• A real catalog: sites declare their set in site-profile.json and apply it in one idempotent command; the chosen set drives backend image, SPA bundle, and runtime state from one source of truth.
• Curated marketplace with end-to-end provenance: review -> sign -> any transport -> verify at adopt, at load, and at migrate. Transport (SharePoint sync or sneakernet) is untrusted and interchangeable, which is exactly right for air-gapped sites.
• Lean per-site builds: unchosen plugins exist in neither the image nor the bundle, shrinking attack surface and download size.
• Fixes shipped along the way: upgrade-all migrating unadopted folders, PILOT-DEPLOY install/enable ordering, reverse-dep checks blind to unloaded plugins, missing cycle detection, missing dependency closure, hardcoded frontend plugin metadata.
• Path A completion makes simple plugins UI-capable with no build glue, which is the cheapest possible marketplace onboarding.

Negative

• Key management is a per-site operational burden: pinned keys delivered out-of-band, rotation is a config change everywhere. Accepted as the price of not trusting the distribution folder.
• The frontend re-org is the long pole: one core-router surgery plus 13 non-trivial PRs. It is sequenced to be always-green and per-plugin revertable, but it is weeks of work, not a rename.
• Schema is not lean: pre-cutover plugin tables still appear at every site until the deferred baseline re-org ADR.
• Restarts remain required after adopt/enable (Flask blueprint constraint); the marketplace UX is honest about it rather than working around it.
• Boot adds a signature + tree-hash check per enabled plugin (milliseconds, but nonzero).

Risks

• Key compromise or curation failure: a signature proves provenance, not safety; a compromised pinned key or a rubber-stamp review signs malware that every gate will happily pass. Mitigations: multi-key pinning with overlap rotation, signed revocation list with monotonic index serial, and keeping the signing key offline with the curator. The static import scan is a lint and must never be presented as a boundary.
• Rollback/downgrade: mitigated three ways: index serial monotonicity, adopt refusing version downgrades without interactive --force-downgrade, and the signed revoked list. Residual risk: a site that never syncs a newer index cannot learn of revocations; flask plugin audit at deploy time narrows the window.
• Version skew across ADR-004 sites: one shelf serves sites at different contract versions. Adopt checks core_version from the signed manifest against the site's own contract_version (authoritative); shelf-list shows an advisory compatibility column from the index. Incompatible artifacts are listable but not adoptable.
• Partial/placeholder sync files: fail closed on hash verification; the error message distinguishes "not fully synced" from "tampered" only by wording, intentionally, since the app cannot tell.
• Dev-trust misuse: PLUGIN_DEV_TRUST_DIRS silently ignored outside DEBUG/TESTING; a prod config carrying it gets a startup warning.
• Frontend closure drift: plugin views importing cross-plugin components is a graph the manifest does not model. The lint rule (plugin frontend may import core @/ paths and its own tree only, never another plugin's) prevents new edges; existing shared plugin-aware code is explicitly core-owned.

Implementation phases

• Phase 0, groundwork (small, days): upgrade_all_plugins uses registry.get_all(); reverse-dep checks read installed manifests from disk; cycle detection in _sort_by_dependencies; shopdb/plugins/manifest_schema.json + flask plugin validate (directory mode); flask plugin apply-profile with install+enable closure ordering; fix Dockerfile stale comment. All additive, zero risk to running sites.
• Phase 1, packaging and signing (medium, about a week): PROVENANCE format, flask plugin pack, validate artifact mode, PLUGIN_TRUSTED_KEYS config, ed25519 signing tooling and curator docs. No runtime behavior change yet.
• Phase 2, shelf and enforcement (medium-large, one to two weeks): PLUGIN_SHELF_DIR, signed shelf-index with serial + revoked list, flask plugin shelf-list / adopt / audit with atomic verified unpack; verify-at-load in load_plugin and verify-at-migrate in run_plugin_migrations, fail-closed in prod; PLUGIN_DEV_TRUST_DIRS for dev/test and the external-repo harness; tier:core lifecycle guard; provenance stamping of bundled plugins at build. This phase completes the security model; everything after it is delivery optimization.
• Phase 3, Path A completion (medium, one to two weeks): generic renderers for asset-panels, map-overlays, asset-presentation; migrate settingsNav.js hardcoded plugin cards to get_settings_cards. Done BEFORE relocation so JSON-only UI is available during the migration.
• Phase 4, frontend re-org (large, the long pole, several weeks elapsed): stage-frontend.mjs staging + routes.gen.js/meta.gen.js codegen; ONE core PR moving the six index.js hardcoded plugin imports and the core.js-embedded plugin routes into plugin route files; then 13 per-plugin relocation PRs (views, settings views, name-mismatch dirs, @/ alias rewrite) under the dual-location union; lint rule for plugin frontend imports. Always-green, per-plugin revertable.
• Phase 5, lean builds end to end (medium, about a week after Phase 4): build-site.sh staging backend dirs + frontend staging from site-profile.json; core lazy-import guard audit finished, enforced by the delete-a-plugin CI matrix; remove the legacy glob; pilot one real lean site (a location without printedparts/usb/network) and diff its image and bundle against a full build.

Deferred, each to its own future decision: schema-lean core-baseline re-org (blocked on the installedapps -> machines FK question), pip/entry-point distribution (ADR-003 v2), hook-based search/report aggregation contract, and any revisit of Path B.