shopdb-flask/docs/adr/ADR-013-plugin-catalog-and-lean-builds.md

# ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds

- Status: PROPOSED
- Date: 2026-07-18
- Deciders: cproudlock
- Relates to: ADR-002 (contract versioning), ADR-003 (plugin distribution), ADR-004 (per-site instances), ADR-008 (per-plugin migrations), ADR-009 (frontend plugin gating), ADR-010 (frontend hook contract)

## Context

Every site today ships identical code. The backend image contains all 13 plugin
directories (Dockerfile COPY at line 55; the header comment listing "eleven core
plugins" is stale), and the SPA compiles every plugin's routes and views via a
static glob (frontend/src/router/index.js:11) plus hardcoded imports. A site's
"chosen set" exists only as runtime enable flags in instance/plugins.json.
Disabled is not absent: a site that never wants printedparts/usb/network still
ships, and can execute, that code.

Distribution per ADR-003 is "drop a directory into plugins/ by hand". There is
no artifact format, no signing, no catalog, no validate gate, and the loader
trusts whatever it finds on disk (loader.py:61-73 discovers any folder with a
plugin.py; loader.py:185-224 loads it; migrations.py:29-62 runs its DDL with
full DB privileges). Plugins run in-process with the full shopdb.api surface
including db, so the only tenable security model on air-gapped GE networks is
curation plus cryptographic provenance, enforced everywhere code can execute,
not sandboxing.

Known defects this ADR also resolves:

- upgrade_all_plugins checks hasattr(registry, 'list_installed') which never
  exists (registry has only get_all/get_enabled_plugins), so it always falls
  back to migrating every folder on disk, adopted or not (__init__.py:94).
- PILOT-DEPLOY.md enables plugins that were never installed; enable refuses.
  There is no declarative "apply this chosen set" operation.
- Reverse-dependency checks on uninstall/disable read only LOADED plugin
  instances, so an installed-but-unloaded dependent is invisible.
- Dependency install/enable is check-only; nothing computes a closure, and
  install-a-dependent can fail at its own load step because the dependency was
  installed disabled (default_enabled=false on employees).
- The dependency sort has no cycle detection.
- Soft couplings (geenforce -> computers, notifications -> employees) are
  invisible to the manifest graph.

Frontend reality check (this drove the design below): plugin UI is NOT one
folder per plugin. Routes live in routes/<plugin>.js (slides has none), inside
the shared core.js (computers report, printers toner report, employees detail,
slides settings), and as six hardcoded top-level imports in index.js itself
(/parts-kiosk, /tv, /print/printer-qr x2, /print/usb-labels,
/print/printedparts-labels). View dirs mismatch plugin names (computers ->
views/pcs). Plugin settings cards sit in shared views/settings/
(DellWarrantySettings, ZabbixSettings, SlideManager, EmployeeDirectory,
MeasuringToolTypesList, PrintedPartsSettings), plugin print views in shared
views/print/, and some views span plugins (AssetLabel.vue serves five asset
types; PCDetail imports WarrantyPanel). Any lean-frontend design that only
moves routes/*.js and views/<plugin>/ fails the build the moment a plugin is
pruned. This ADR scopes that work honestly instead of calling it mechanical.

## Decision

### 1. Tiering: mandatory core is the core package; all plugins are catalog-optional

- The mandatory core is the non-plugin shopdb/core/ package (auth, users,
  assets, locations, vendors, models, settings, audit, dashboard, search,
  reports, plugin management). It already survives every plugin being absent
  via hasattr/lazy-import guards. No plugin is promoted into it.
- New optional manifest field `tier: "core" | "optional"`, default "optional".
  All 13 existing manifests are unchanged and unchanged in meaning. The
  lifecycle gains a guard: uninstall_plugin and disable_plugin refuse a
  tier:core plugin (alongside the reverse-dependency checks at
  __init__.py:269-278 and :351-360). No plugin ships tier:core initially; the
  field and guard exist so a future curation decision is a manifest edit, not a
  framework change.
- Per-site mandates live in the site profile (section 5): a `locked` list the
  profile applier refuses to remove. This preserves ADR-004 site autonomy: a
  wing site can mandate usb without the framework mandating it fleet-wide.
- New manifest field `optional_dependencies: []` (names only, loader-ignored).
  Declared for the verified soft couplings: geenforce lists computers
  (service.py:32-43 loses app-detection gates without it), notifications lists
  employees (routes.py:151-171 loses name/photo enrichment). Catalog listing
  and adopt WARN on unmet optional deps; nothing blocks.
- Hard `dependencies` gains optional PEP440 ranges ("employees>=1.1").
  validate/adopt honor ranges; the runtime loader keeps name-only semantics
  (specifier stripped) so no loader behavior changes. The single existing hard
  edge printedparts -> employees stays as-is; whether it can relax to optional
  (badges.py has an external HR fallback) is a follow-up product question, not
  blocked on this ADR.
- Dependency plumbing fixes: _sort_by_dependencies gains cycle detection
  (raise PluginDependencyError on a back edge); reverse-dependency checks read
  manifests of ALL installed plugins from disk, not loaded instances.

### 2. Packaging: signed, versioned artifacts

Artifact: `<name>-<version>.shopdbplugin` (a zip of the plugin directory:
manifest.json, plugin.py, api/, models/, migrations/, and frontend/ once
section 6 lands) plus two members generated at pack time:

- `PROVENANCE.json`: plugin name, version, publisher id, build timestamp, and
  a sorted map of every packaged file path to its SHA-256. PROVENANCE.json is
  not listed in its own map, so there is no circular-hash problem and no zip
  canonicalization needed; determinism comes from sorted per-file hashes.
- `PROVENANCE.sig`: detached ed25519 signature over the exact PROVENANCE.json
  bytes.

New CLI:

- `flask plugin pack <name> --key <path>` (producer side): runs validate on the
  directory, then emits the artifact.
- `flask plugin validate <dir|artifact>` (the missing pre-publish gate),
  fail-closed pipeline: signature (artifact mode) -> per-file hashes -> manifest
  against a new docs/plugin-manifest.schema.json -> name == directory ->
  core_version parses as a specifier and admits the target contract version ->
  static import-surface scan reusing tests/test_plugin_contract.py logic ->
  alembic versions parse. The schema types the known fields (name, version,
  description, dependencies, optional_dependencies, tier, core_version,
  api_prefix, display_name, default_enabled, provides, settings) and PERMITS
  additional properties, so all 13 existing manifests pass unmodified.

The import-surface scan is documented as a lint, not a security control; it is
trivially bypassed by dynamic import. The security control is human review
before signing (section 4).

### 3. The shelf: a read-only folder, transport-agnostic by design

- One config knob: `PLUGIN_SHELF_DIR`. The app only ever reads this folder. It
  never speaks SharePoint, OneDrive, or any network protocol.
- Transport is explicitly out of scope and explicitly untrusted. On networks
  that can reach corporate M365, a SharePoint document library sync populates
  the folder. On strictly air-gapped floors where no sync agent can run, the
  folder is populated by robocopy/USB. Both are equally supported and equally
  untrusted, because every decision-bearing byte is signed: swapping transport
  changes nothing about the trust model.
- Layout: `<shelf>/<name>/<name>-<version>.shopdbplugin` plus
  `shelf-index.json` and `shelf-index.sig`.
- The index is SIGNED with the same publisher key and carries a monotonically
  increasing `serial` plus a `revoked` list of name-version pairs. Each site
  records the last-seen serial in instance state and refuses an index with a
  lower serial (anti-rollback of the catalog itself). The index also carries
  per-entry version/tier/core_version so `flask plugin shelf-list` can display
  compatibility without unpacking, but the index is a BROWSE layer only:
  adopt reads dependencies, tier, and core_version from the signed manifest
  inside the verified artifact, never from the index.
- Trusted keys: `PLUGIN_TRUSTED_KEYS` is a list of pinned public keys delivered
  out-of-band in the site's deployed config/image. Keys are NEVER read from the
  shelf; a folder that can be written by an attacker must not also carry the
  keys that authenticate it. Multiple pinned keys allow overlap rotation.
  Revocation of an artifact rides the signed index `revoked` list; a
  `flask plugin audit` command warns when an installed version appears there.
- Partial-sync robustness: adopt copies the artifact to a temp location,
  verifies signature and every file hash there, then unpacks to
  plugins/.staging/<name> and renames into place atomically. OneDrive
  placeholder stubs, zero-byte files, or an index referencing not-yet-synced
  artifacts all fail closed with a clear "artifact not fully synced/verified"
  error.
- `flask plugin adopt <name>[==version]`: resolve version from the shelf,
  verify, compute the hard-dependency closure from signed manifests, then for
  each closure member in topological order: unpack, INSTALL, and ENABLE (not
  install-only; the load gate at loader.py:201-206 checks is_enabled, so an
  install-only closure with default_enabled=false deps would fail its own
  load). Migrations run via the unchanged per-plugin chain (ADR-008). Refuses
  to adopt a version lower than the installed one unless
  `--force-downgrade` is given interactively. Prints the restart notice.
- Adopt/install/uninstall remain CLI-only. The admin HTTP surface stays a
  read-only catalog view plus the existing enable/disable toggle; because
  Flask cannot register blueprints after the first request, any adopt or
  enable takes full effect only on restart, and the UI says so. There is no
  "install button" that pretends otherwise.

### 4. Trust model: verify at adopt AND at every load and migrate

Signing that gates only adoption is bypassable through every other write path
into plugins/ (git clone, symlink, USB drop) and defeated by post-adoption
tampering. Therefore verification is enforced where code executes:

- Adoption leaves PROVENANCE.json and PROVENANCE.sig inside plugins/<name>/
  and records publisher + artifact hash in the registry entry.
- load_plugin verifies the signature against PLUGIN_TRUSTED_KEYS and re-hashes
  the plugin tree against the provenance file map BEFORE importing plugin.py
  (new step ahead of loader.py:185). Missing or invalid provenance is a
  fail-closed refusal in production.
- run_plugin_migrations performs the same verification before executing any
  revision, so a routine `flask plugin upgrade-all` can never run DDL from an
  unverified folder.
- upgrade_all_plugins iterates registry.get_all() (fixing the phantom
  list_installed fallback at __init__.py:94), so unadopted on-disk folders are
  never migrated as a side effect of deploys.
- Development and the ADR-003 external-repo/symlink workflow (including
  scripts/test-external-plugin.sh) are preserved via `PLUGIN_DEV_TRUST_DIRS`,
  honored ONLY when DEBUG or TESTING is set. Production ignores it.
- Cost: hashing 13 small plugin trees at boot is milliseconds; accepted.

What signing does NOT claim: a valid signature proves the artifact is exactly
what a curator reviewed and signed, nothing more. Plugins remain in-process
Python with full DB access. The actual safety control is the human review
before signing; the signature makes that review's verdict tamper-evident all
the way to execution.

### 5. Declarative site profiles and lean backend builds

- `site-profile.json` per site (kept in the site's deploy config): site name,
  list of chosen plugins, optional `locked` list. `flask plugin apply-profile
  <file>` resolves the closure, installs AND enables in dependency order, runs
  migrations, reports which changes need a restart. This replaces the
  imperative CLI sequences in DEPLOY.md/PILOT-DEPLOY.md and fixes the
  enable-without-install bug.
- Lean backend image: `scripts/build-site.sh` reads the profile and stages
  only core + chosen plugin directories into the Docker build context
  (correcting the Dockerfile COPY and its stale header comment). Discovery
  needs no change; it already scans whatever exists.
- Prerequisite the naive version misses: core hardcodes plugin imports.
  shopdb/core/api/search.py (~15 sites), reports.py, assets.py, collector.py,
  applications.py, auditlogs.py, and shopdb/cli/__init__.py import
  plugins.<name>.* lazily. Some already guard ImportError; ALL must, with
  graceful degradation, before any site prunes a folder. This is audited and
  enforced by a new CI job that deletes one plugin directory and runs the full
  test suite (repeated per plugin). Longer term these aggregators should move
  to registry-driven contract hooks (get_search_providers/get_report_sources)
  so a new catalog plugin can join search/reports without core edits; that is
  scoped as follow-up work, not a blocker for lean builds.
- Schema-lean is DEFERRED to its own ADR. The core baseline 68b3947ae14f
  unconditionally creates the 10 pre-cutover plugins' tables, and lifting them
  into plugin baselines collides with cross-plugin foreign keys (the
  computers-owned installedapps table FKs machines.machineid while computers
  declares no dependency on machines). Reversing the cutover would either
  introduce undeclared hard deps or drop FKs; neither is decided here. A lean
  site therefore carries a handful of empty pre-cutover tables. Accepted.

### 6. Frontend delivery: Path C for rich UIs, Path A for simple ones, Path B rejected

Path B (runtime-loaded JS / module federation) is REJECTED: it moves executable
UI delivery from a signed, statically auditable build artifact to runtime
fetching, which is exactly the wrong direction for an air-gapped,
review-then-sign posture, for zero benefit given restarts are already required.

Path A (declarative JSON UI over generic renderers) is COMMITTED and scheduled
EARLY: the three unwired ADR-010 endpoints (pluginui.py asset-panels:62,
map-overlays:88, asset-presentation:100) get generic core renderers, joining
the already-consumed settings-cards. After this, a simple plugin ships JSON-only
UI with zero frontend build involvement. Sequencing this before the relocation
gives every plugin an escape hatch during the migration instead of after it.

Path C (self-contained plugin frontend) is the primary mechanism, scoped
against the real code, not the idealized layout:

- Canonical home: plugins/<name>/frontend/ containing routes.js (the plugin's
  complete route array, INCLUDING routes currently embedded in index.js and
  core.js), views/, and settings views.
- A pre-Vite staging step (scripts/stage-frontend.mjs, run by build-site.sh
  and the dev script) copies the CHOSEN plugins' frontend/ into
  frontend/src/.plugins-staged/<name>/ (gitignored) and generates two files
  inside the Vite root: routes.gen.js (aggregated plugin routes) and
  meta.gen.js (plugin-supplied icon names, title spellings, settings-standalone
  flags, replacing the hardcoded iconMap/TITLE_SPELLINGS/SETTINGS_STANDALONE in
  AppLayout.vue, settingsCatalog.js, and index.js). This exists because
  import.meta.glob requires a static literal inside the project root and
  cannot select a per-site subset by itself.
- ONE-TIME core-router surgery, done first and called what it is: the six
  hardcoded plugin-view imports in index.js (PartsKiosk, TVDashboard,
  PrinterQRBatch/Single, USBLabelBatch, PrintedPartsLabels) and the plugin
  routes embedded in core.js move into their owning plugins' routes.js. Without
  this, pruning slides/printers/usb/printedparts fails the Vite build on
  unresolvable imports; no amount of glob work fixes it.
- Per-plugin relocation PRs (13), each REAL WORK, not a file move: carve routes
  out of shared files, move views (handling name mismatches like computers ->
  views/pcs), move the plugin's settings views out of shared views/settings/,
  and rewrite relative ../../ imports of core shared code to the @/ alias
  (relative paths break at the staged depth). A lint rule enforces alias-only
  core imports in plugin frontend code from then on.
- Shared plugin-aware code STAYS CORE and ships to every site: AssetLabel.vue
  (spans five asset types), views/print helpers (assetLabel.js, qrLogo.js),
  MachineBadge.vue, and cross-plugin panels like WarrantyPanel used by
  PCDetail. These already null-guard or gate via isPluginEnabled and must keep
  degrading when a peer plugin is absent; over time they migrate to ADR-010
  asset-panels so the data becomes plugin-supplied. Lean v1 therefore prunes
  plugin-EXCLUSIVE code; a small plugin-aware core remainder is accepted and
  shrinks as Path A absorbs it.
- Dual-location transition: the staging step unions legacy locations
  (routes/*.js glob, views/<plugin>/) with plugins/<name>/frontend/ until each
  plugin has moved. The SPA builds green at every commit; each plugin's move is
  independently revertable until the legacy glob is removed at the end.
- Nav and settings cards are already server-driven (dashboardApi.navigation,
  settings-cards); the remaining hardcoded plugin entries in settingsNav.js
  (/settings/zabbix, /settings/dellwarranty) move to those plugins'
  get_settings_cards so pruning leaves no dead links.

### 7. Effect on the 13 existing plugins

- Backend: ZERO code changes required. tier/optional_dependencies/provenance
  are additive; pack zips the directory as-is; all plugins keep passing
  tests/test_plugin_contract.py. Bundled plugins in a site's image get
  provenance stamped at build time by pack, so verify-at-load applies to them
  identically.
- Frontend: one relocation PR each, of the honest scope above. Until a
  plugin's PR lands it keeps working from its legacy location.
- Operationally nothing changes for a site that does nothing: default builds
  remain all-plugins, apply-profile is opt-in, and enable/disable semantics
  (including the restart requirement) are unchanged.

## Consequences

### Positive

- A real catalog: sites declare their set in site-profile.json and apply it in
  one idempotent command; the chosen set drives backend image, SPA bundle, and
  runtime state from one source of truth.
- Curated marketplace with end-to-end provenance: review -> sign -> any
  transport -> verify at adopt, at load, and at migrate. Transport (SharePoint
  sync or sneakernet) is untrusted and interchangeable, which is exactly right
  for air-gapped sites.
- Lean per-site builds: unchosen plugins exist in neither the image nor the
  bundle, shrinking attack surface and download size.
- Fixes shipped along the way: upgrade-all migrating unadopted folders,
  PILOT-DEPLOY install/enable ordering, reverse-dep checks blind to unloaded
  plugins, missing cycle detection, missing dependency closure, hardcoded
  frontend plugin metadata.
- Path A completion makes simple plugins UI-capable with no build glue, which
  is the cheapest possible marketplace onboarding.

### Negative

- Key management is a per-site operational burden: pinned keys delivered
  out-of-band, rotation is a config change everywhere. Accepted as the price of
  not trusting the distribution folder.
- The frontend re-org is the long pole: one core-router surgery plus 13
  non-trivial PRs. It is sequenced to be always-green and per-plugin
  revertable, but it is weeks of work, not a rename.
- Schema is not lean: pre-cutover plugin tables still appear at every site
  until the deferred baseline re-org ADR.
- Restarts remain required after adopt/enable (Flask blueprint constraint);
  the marketplace UX is honest about it rather than working around it.
- Boot adds a signature + tree-hash check per enabled plugin (milliseconds,
  but nonzero).

### Risks

- Key compromise or curation failure: a signature proves provenance, not
  safety; a compromised pinned key or a rubber-stamp review signs malware that
  every gate will happily pass. Mitigations: multi-key pinning with overlap
  rotation, signed revocation list with monotonic index serial, and keeping the
  signing key offline with the curator. The static import scan is a lint and
  must never be presented as a boundary.
- Rollback/downgrade: mitigated three ways: index serial monotonicity, adopt
  refusing version downgrades without interactive --force-downgrade, and the
  signed revoked list. Residual risk: a site that never syncs a newer index
  cannot learn of revocations; `flask plugin audit` at deploy time narrows the
  window.
- Version skew across ADR-004 sites: one shelf serves sites at different
  contract versions. Adopt checks core_version from the signed manifest against
  the site's own __contract_version__ (authoritative); shelf-list shows an
  advisory compatibility column from the index. Incompatible artifacts are
  listable but not adoptable.
- Partial/placeholder sync files: fail closed on hash verification; the error
  message distinguishes "not fully synced" from "tampered" only by wording,
  intentionally, since the app cannot tell.
- Dev-trust misuse: PLUGIN_DEV_TRUST_DIRS silently ignored outside
  DEBUG/TESTING; a prod config carrying it gets a startup warning.
- Frontend closure drift: plugin views importing cross-plugin components is a
  graph the manifest does not model. The lint rule (plugin frontend may import
  core @/ paths and its own tree only, never another plugin's) prevents new
  edges; existing shared plugin-aware code is explicitly core-owned.

## Implementation phases

- Phase 0, groundwork (small, days): upgrade_all_plugins uses
  registry.get_all(); reverse-dep checks read installed manifests from disk;
  cycle detection in _sort_by_dependencies; shopdb/plugins/manifest_schema.json +
  `flask plugin validate` (directory mode); `flask plugin apply-profile` with
  install+enable closure ordering; fix Dockerfile stale comment. All additive,
  zero risk to running sites.
- Phase 1, packaging and signing (medium, about a week): PROVENANCE format,
  `flask plugin pack`, validate artifact mode, PLUGIN_TRUSTED_KEYS config,
  ed25519 signing tooling and curator docs. No runtime behavior change yet.
- Phase 2, shelf and enforcement (medium-large, one to two weeks):
  PLUGIN_SHELF_DIR, signed shelf-index with serial + revoked list,
  `flask plugin shelf-list` / `adopt` / `audit` with atomic verified unpack;
  verify-at-load in load_plugin and verify-at-migrate in
  run_plugin_migrations, fail-closed in prod; PLUGIN_DEV_TRUST_DIRS for
  dev/test and the external-repo harness; tier:core lifecycle guard;
  provenance stamping of bundled plugins at build. This phase completes the
  security model; everything after it is delivery optimization.
- Phase 3, Path A completion (medium, one to two weeks): generic renderers for
  asset-panels, map-overlays, asset-presentation; migrate settingsNav.js
  hardcoded plugin cards to get_settings_cards. Done BEFORE relocation so
  JSON-only UI is available during the migration.
- Phase 4, frontend re-org (large, the long pole, several weeks elapsed):
  stage-frontend.mjs staging + routes.gen.js/meta.gen.js codegen; ONE core PR
  moving the six index.js hardcoded plugin imports and the core.js-embedded
  plugin routes into plugin route files; then 13 per-plugin relocation PRs
  (views, settings views, name-mismatch dirs, @/ alias rewrite) under the
  dual-location union; lint rule for plugin frontend imports. Always-green,
  per-plugin revertable.
- Phase 5, lean builds end to end (medium, about a week after Phase 4):
  build-site.sh staging backend dirs + frontend staging from site-profile.json;
  core lazy-import guard audit finished, enforced by the delete-a-plugin CI
  matrix; remove the legacy glob; pilot one real lean site (a location without
  printedparts/usb/network) and diff its image and bundle against a full build.

Deferred, each to its own future decision: schema-lean core-baseline re-org
(blocked on the installedapps -> machines FK question), pip/entry-point
distribution (ADR-003 v2), hook-based search/report aggregation contract, and
any revisit of Path B.