ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ADR-013 Phase 0: plugin lifecycle groundwork

Browse Source 
Additive, zero-risk-to-running-sites prep for the plugin catalog. No
distribution or lean-build behavior yet; fixes latent bugs and adds the
declarative + validate tooling later phases build on.

Fixes:
- upgrade_all_plugins iterates registry.get_all(); only adopted plugins are
  migrated. Removes the phantom hasattr(registry, 'list_installed') probe
  that always fell through to migrating every folder on disk (unadopted DDL
  ran with full DB rights on every deploy).
- Reverse-dependency checks on uninstall/disable read dependencies from the
  manifest on disk via _installed_dependents, so an installed-but-unloaded or
  disabled dependent is counted. Uninstall blocks on any installed dependent;
  disable blocks on an enabled dependent.
- _sort_by_dependencies detects a dependency cycle (back edge in the DFS) and
  raises PluginDependencyError instead of looping or dropping a plugin.

New:
- flask plugin validate <name>: manifest loads + name match, manifest-schema
  check, core_version admits the framework contract, declared dependencies
  exist on disk. No new dependency (lightweight checker); schema ships in the
  package at shopdb/plugins/manifest_schema.json (docs/ is stripped on
  publish). The check caught that provides is an object, not an array.
- flask plugin apply-profile <file>: declarative install AND enable of a
  chosen plugin set plus its hard-dependency closure, in dependency order,
  idempotent. Replaces the hand-ordered runbook sequences that could enable a
  plugin that was never installed. deploy/site-profile.example.json template.
- Dockerfile header corrected (all 13 catalog plugins, not "eleven core").

10 new lifecycle tests (reverse-deps from disk, cycle detection, upgrade-all
scope, profile closure, schema, all 13 manifests match schema). 1018 pass,
naming green.
This commit is contained in:
cproudlock
2026-07-18 18:08:34 -04:00
parent 3ac5ed2580
commit d178726687
7 changed files with 533 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
Dockerfile
View File

@@ -2,10 +2,12 @@
#
# One image, one site. Per ADR-004, each adopting facility runs its own
# stack with its own DB, secrets, and enabled-plugin list. This image
# bundles all eleven core plugins (computers, employees, knowledgebase,
# machines, measuringtools, network, notifications, printers, slides, usb,
# warranty);
# install them at runtime with `flask plugin install <name>`.
# bundles all 13 catalog plugins (computers, employees, geenforce,
# knowledgebase, machines, measuringtools, network, notifications,
# printedparts, printers, slides, usb, warranty); a site installs + enables
# the ones it wants with `flask plugin install <name>` (or, declaratively,
# `flask plugin apply-profile <profile.json>`). Per ADR-013 a future lean
# build stages only the chosen plugin directories into this image.
#
# The frontend is built in a first stage and its dist output is copied into
# the final image so Flask can serve the SPA (register_frontend_routes in

11
deploy/site-profile.example.json Normal file
View File

@@ -0,0 +1,11 @@
{
"site": "example-site",
"plugins": [
"machines",
"computers",
"printers",
"network"
],
"locked": [],
"_comment": "Declarative plugin selection for a site (ADR-013). Apply with: flask plugin apply-profile deploy/site-profile.example.json. 'plugins' is the set this site wants; their hard dependencies are pulled in automatically and everything installs + enables in dependency order (idempotent). Naming a plugin not on disk fails loudly. 'locked' is reserved for a future guard against removing a site-mandated plugin. apply-profile never removes plugins absent from the list - removal stays an explicit flask plugin uninstall. Run flask plugin upgrade-all after applying, then restart."
}

143
shopdb/plugins/__init__.py
View File

@@ -9,6 +9,7 @@ from .base import BasePlugin, PluginMeta
from .registry import PluginRegistry, PluginState
from .loader import PluginLoader
from .migrations import PluginMigrationManager
from ..exceptions import PluginError
logger = logging.getLogger(__name__)
@@ -91,11 +92,13 @@ class PluginManager:
results: Dict[str, str] = {}
if not self.migration_manager:
return results
plugin_names = list(self.registry.list_installed().keys()) \
if hasattr(self.registry, 'list_installed') else []
if not plugin_names:
# Fall back to whatever the loader found on disk.
plugin_names = self.loader.discover_plugins()
# Only ADOPTED plugins (those in the registry) get migrated. A plugin
# folder that merely sits on disk unadopted must not have its DDL run as
# a side effect of a deploy - migrations execute with full DB rights, so
# "on disk" is not "trusted to run". (The old code probed a
# list_installed() method the registry never had, so it always fell
# through to discover_plugins() and migrated every folder.)
plugin_names = list(self.registry.get_all().keys())
for name in plugin_names:
if not self.migration_manager.has_pending_migrations(name):
results[name] = 'no-migrations'
@@ -254,6 +257,30 @@ class PluginManager:
"Seeded %d permission(s) for plugin %s",
created, plugin.meta.name)
def _installed_dependents(self, name: str,
enabled_only: bool = False) -> List[str]:
"""Installed plugins that declare `name` as a hard dependency.
Reads each dependency list from the manifest on DISK, not from a loaded
plugin instance, so a dependent that is installed-but-not-loaded (e.g.
disabled, or failed to load) is still counted. enabled_only restricts to
currently-enabled dependents (what matters when disabling, since a
disabled dependent is not running).
"""
dependents = []
for other_name, state in self.registry.get_all().items():
if other_name == name:
continue
if enabled_only and not state.enabled:
continue
try:
manifest = self.loader.load_manifest(other_name)
except PluginError:
continue
if name in manifest.get('dependencies', []):
dependents.append(other_name)
return dependents
def uninstall_plugin(self, name: str, remove_data: bool = False) -> bool:
"""
Uninstall a plugin.
@@ -266,16 +293,13 @@ class PluginManager:
logger.warning(f"Plugin {name} is not installed")
return False
# Check if other plugins depend on this one
for other_name in self.registry.get_enabled_plugins():
if other_name == name:
continue
other_plugin = self.loader.get_loaded_plugin(other_name)
if other_plugin and name in other_plugin.meta.dependencies:
logger.error(
f"Cannot uninstall {name}: {other_name} depends on it"
)
return False
# Refuse if any INSTALLED plugin (enabled or not) depends on this one.
dependents = self._installed_dependents(name)
if dependents:
logger.error(
f"Cannot uninstall {name}: {', '.join(dependents)} depend(s) on it"
)
return False
# Get plugin instance
plugin = self.loader.get_loaded_plugin(name)
@@ -348,16 +372,14 @@ class PluginManager:
logger.info(f"Plugin {name} is already disabled")
return True
# Check if other plugins depend on this one
for other_name in self.registry.get_enabled_plugins():
if other_name == name:
continue
other_plugin = self.loader.get_loaded_plugin(other_name)
if other_plugin and name in other_plugin.meta.dependencies:
logger.error(
f"Cannot disable {name}: {other_name} depends on it"
)
return False
# Refuse if any ENABLED plugin depends on this one (a disabled dependent
# is not running, so it does not block). Manifests read from disk.
dependents = self._installed_dependents(name, enabled_only=True)
if dependents:
logger.error(
f"Cannot disable {name}: {', '.join(dependents)} depend(s) on it"
)
return False
plugin = self.loader.get_loaded_plugin(name)
if plugin:
@@ -367,6 +389,77 @@ class PluginManager:
logger.info(f"Disabled plugin: {name}")
return True
def _dependency_closure(self, names: List[str]) -> List[str]:
"""Chosen plugins plus their hard-dependency closure, dependencies first.
Post-order DFS over manifest `dependencies`, so the returned order is
safe to install/enable top to bottom (a dependency always precedes the
plugin that needs it). Deduplicated.
"""
closure: List[str] = []
seen = set()
def add(name):
if name in seen:
return
seen.add(name)
try:
deps = self.loader.load_manifest(name).get('dependencies', [])
except PluginError:
deps = []
for dep in deps:
add(dep)
closure.append(name)
for name in names:
add(name)
return closure
def apply_profile(self, plugins: List[str],
locked: Optional[List[str]] = None) -> Dict[str, List[str]]:
"""Install AND enable exactly the chosen plugins plus their dependency
closure, in dependency order.
Declarative replacement for the hand-ordered install/enable sequences in
the deploy runbooks (which could enable a plugin that was never
installed). Idempotent: already-installed/enabled plugins are left as-is.
This does NOT remove anything absent from the list - removal stays an
explicit, separate operation. `locked` is accepted for forward
compatibility (a future guard against removing a site-mandated plugin)
and is currently informational.
Raises PluginError if the profile names a plugin (or pulls in a
dependency) that does not exist on disk. Returns
{installed: [...], enabled: [...], already: [...]}.
"""
available = set(self.loader.discover_plugins())
unknown = [p for p in plugins if p not in available]
if unknown:
raise PluginError(
f"Profile names unknown plugins: {', '.join(sorted(unknown))}")
order = self._dependency_closure(plugins)
missing_deps = [p for p in order if p not in available]
if missing_deps:
raise PluginError(
f"Missing dependency plugins: {', '.join(sorted(missing_deps))}")
result: Dict[str, List[str]] = {
'installed': [], 'enabled': [], 'already': []}
for name in order:
changed = False
if not self.registry.is_installed(name):
if self.install_plugin(name):
result['installed'].append(name)
changed = True
if not self.registry.is_enabled(name):
if self.enable_plugin(name):
result['enabled'].append(name)
changed = True
if not changed:
result['already'].append(name)
return result
def get_plugin(self, name: str) -> Optional[BasePlugin]:
"""Get a loaded plugin instance."""
return self.loader.get_loaded_plugin(name)

174
shopdb/plugins/cli.py
View File

@@ -1,10 +1,60 @@
"""Flask CLI commands for plugin management."""
import json
from pathlib import Path
import click
from flask import current_app
from flask.cli import with_appcontext
# JSON-Schema primitive name -> Python type(s) for the no-dependency validator.
_SCHEMA_TYPES = {
'string': str,
'boolean': bool,
'array': list,
'object': dict,
'integer': int,
'number': (int, float),
}
def _load_manifest_schema() -> dict:
"""Load the packaged manifest schema (ships with the app, unlike docs/)."""
schema_path = Path(__file__).with_name('manifest_schema.json')
with open(schema_path) as f:
return json.load(f)
def _check_against_schema(manifest: dict, schema: dict) -> list:
"""Lightweight schema check without a jsonschema dependency.
Verifies required fields are present and that known typed fields hold the
right JSON type; unknown fields are allowed (additionalProperties). Returns
a list of human-readable error strings (empty = valid).
"""
errors = []
for field in schema.get('required', []):
if field not in manifest:
errors.append(f"missing required field '{field}'")
props = schema.get('properties', {})
for key, value in manifest.items():
spec = props.get(key)
if not spec:
continue # additionalProperties permitted
expected = spec.get('type')
pytype = _SCHEMA_TYPES.get(expected)
# bool is a subclass of int; guard so a boolean does not pass 'integer'
if pytype and (not isinstance(value, pytype)
or (expected in ('integer', 'number')
and isinstance(value, bool))):
errors.append(f"field '{key}' should be {expected}")
if spec.get('enum') and value not in spec['enum']:
errors.append(
f"field '{key}' must be one of {spec['enum']}, got '{value}'")
return errors
@click.group('plugin')
def plugin_cli():
"""Plugin management commands."""
@@ -230,6 +280,130 @@ def new_plugin(name: str, description: str, overwrite: bool):
click.echo(f' 6. Run: pytest plugins/{name}/tests/')
@plugin_cli.command('validate')
@click.argument('name')
@with_appcontext
def validate_plugin(name: str):
"""Validate a plugin directory against the manifest schema + contract.
Pre-publish gate (directory mode). Checks: manifest loads and its name
matches the directory, required/typed fields per the manifest schema, the
core_version range admits this framework's contract version, and every
declared hard dependency exists on disk. Exits non-zero on any failure.
Usage: flask plugin validate printers
"""
from shopdb import __contract_version__
from ..exceptions import PluginError
pm = current_app.extensions.get('plugin_manager')
if not pm:
click.echo(click.style("Plugin manager not initialized", fg='red'))
raise SystemExit(1)
failures = []
# 1. Manifest loads (raises on missing/unparseable/name-mismatch).
try:
manifest = pm.loader.load_manifest(name)
except PluginError as e:
click.echo(click.style(f" manifest: {e}", fg='red'))
raise SystemExit(1)
click.echo(click.style(" manifest loads + name matches directory", fg='green'))
# 2. Schema.
schema_errors = _check_against_schema(manifest, _load_manifest_schema())
if schema_errors:
for err in schema_errors:
failures.append(f"schema: {err}")
click.echo(click.style(f" schema: {err}", fg='red'))
else:
click.echo(click.style(" schema OK", fg='green'))
# 3. Contract version range admits this framework.
try:
pm.loader.check_contract_version(name, __contract_version__)
click.echo(click.style(
f" core_version admits contract {__contract_version__}", fg='green'))
except PluginError as e:
failures.append(str(e))
click.echo(click.style(f" core_version: {e}", fg='red'))
# 4. Declared hard dependencies exist on disk (name-only; ranges stripped
# later when the loader gains range semantics).
available = set(pm.loader.discover_plugins())
for dep in manifest.get('dependencies', []):
depname = dep.split('>')[0].split('<')[0].split('=')[0].split('!')[0].split('~')[0].strip()
if depname not in available:
failures.append(f"dependency '{depname}' not found on disk")
click.echo(click.style(
f" dependency '{depname}' not found on disk", fg='red'))
if not failures and manifest.get('dependencies'):
click.echo(click.style(" dependencies present on disk", fg='green'))
click.echo("")
if failures:
click.echo(click.style(
f"{name}: INVALID ({len(failures)} problem(s))", fg='red'))
raise SystemExit(1)
click.echo(click.style(f"{name}: valid", fg='green'))
@plugin_cli.command('apply-profile')
@click.argument('profile', type=click.Path(exists=True, dir_okay=False))
@with_appcontext
def apply_profile(profile: str):
"""Install AND enable exactly the plugins named in a site profile.
Declarative site setup: replaces the hand-ordered install/enable sequences
in the deploy runbooks. Resolves the hard-dependency closure and applies it
in dependency order; idempotent. Does NOT remove anything absent from the
list.
Profile JSON shape:
{ "site": "west-jefferson",
"plugins": ["machines", "printers", "computers"],
"locked": ["computers"] }
Usage: flask plugin apply-profile site-profile.json
"""
from ..exceptions import PluginError
pm = current_app.extensions.get('plugin_manager')
if not pm:
click.echo(click.style("Plugin manager not initialized", fg='red'))
raise SystemExit(1)
with open(profile) as f:
data = json.load(f)
names = data.get('plugins', [])
if not isinstance(names, list) or not names:
click.echo(click.style(
"Profile has no 'plugins' list to apply", fg='red'))
raise SystemExit(1)
click.echo(f"Applying profile: {data.get('site', profile)}")
try:
result = pm.apply_profile(names, locked=data.get('locked'))
except PluginError as e:
click.echo(click.style(f"Profile failed: {e}", fg='red'))
raise SystemExit(1)
if result['installed']:
click.echo(click.style(
f" installed: {', '.join(result['installed'])}", fg='green'))
if result['enabled']:
click.echo(click.style(
f" enabled: {', '.join(result['enabled'])}", fg='green'))
if result['already']:
click.echo(click.style(
f" unchanged: {', '.join(result['already'])}", fg='white'))
click.echo("")
click.echo(click.style(
"Run 'flask plugin upgrade-all' to apply plugin migrations, then "
"restart so new blueprints/routes register.", fg='yellow'))
@plugin_cli.command('migrate')
@click.argument('name')
@click.option('--revision', default='head', help='Target revision')

31
shopdb/plugins/loader.py
View File

@@ -242,24 +242,35 @@ class PluginLoader:
"""Sort plugins so dependencies come first.
Reads dependencies from manifest.json directly; does not
instantiate plugin classes during sort.
instantiate plugin classes during sort. Detects a dependency cycle
(a back edge in the DFS) and raises PluginDependencyError rather than
looping or silently dropping a plugin.
"""
sorted_list = []
visited = set()
visited = set() # fully processed (post-order emitted)
visiting = set() # on the current DFS stack; a revisit here is a cycle
def visit(name):
if name in visited:
return
visited.add(name)
if name in visiting:
raise PluginDependencyError(
f'Circular plugin dependency involving "{name}"',
plugin_name=name,
)
visiting.add(name)
# Only the manifest read is tolerant; a cycle raised by a nested
# visit must propagate, so the dep recursion sits outside the guard
# (PluginDependencyError is itself a PluginError).
try:
manifest = self.load_manifest(name)
for dep in manifest.get('dependencies', []):
if dep in plugin_names:
visit(dep)
deps = self.load_manifest(name).get('dependencies', [])
except PluginError:
pass
deps = []
for dep in deps:
if dep in plugin_names:
visit(dep)
visiting.discard(name)
visited.add(name)
sorted_list.append(name)
for name in plugin_names:

66
shopdb/plugins/manifest_schema.json Normal file
View File

@@ -0,0 +1,66 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://shopdb-flask/plugin-manifest.schema.json",
"title": "shopdb-flask plugin manifest",
"description": "Shape of a plugin's manifest.json (ADR-002, ADR-013). Known fields are typed; additionalProperties is permitted so existing manifests and future additive fields validate unchanged.",
"type": "object",
"additionalProperties": true,
"required": ["name", "version", "description"],
"properties": {
"name": {
"type": "string",
"description": "Machine name. Must equal the plugin directory name. Lowercase, concatenated.",
"pattern": "^[a-z][a-z0-9]*$"
},
"version": {
"type": "string",
"description": "Plugin version (semver-ish string)."
},
"description": {
"type": "string",
"description": "One-line human description."
},
"display_name": {
"type": "string",
"description": "Human label for UIs. Falls back to a title-cased name."
},
"author": {
"type": "string"
},
"dependencies": {
"type": "array",
"description": "Hard dependencies: plugin names that must be installed and enabled first. A name may carry an optional PEP440 range (e.g. 'employees>=1.1'); the runtime loader uses name-only semantics.",
"items": { "type": "string" }
},
"optional_dependencies": {
"type": "array",
"description": "Soft couplings: plugins that unlock extra behavior when present, but do not block install/enable. Loader-ignored; adopt/list only WARN when unmet.",
"items": { "type": "string" }
},
"tier": {
"type": "string",
"description": "core = mandatory, lifecycle refuses to uninstall/disable it; optional = catalog plugin (default).",
"enum": ["core", "optional"]
},
"core_version": {
"type": "string",
"description": "PEP440 specifier set the framework contract version must satisfy, e.g. '>=0.12.0,<1.0.0'."
},
"api_prefix": {
"type": "string",
"description": "URL prefix for the plugin blueprint, e.g. '/api/printedparts'. Must be unique across enabled plugins."
},
"default_enabled": {
"type": "boolean",
"description": "Whether install leaves the plugin enabled. Plugins that provision extra tables ship false so a site opts in."
},
"provides": {
"type": "object",
"description": "Capabilities this plugin advertises (e.g. asset_type, a features list). Free-form object read by catalog/UI, not the loader."
},
"settings": {
"type": "object",
"description": "Plugin-scoped default settings metadata."
}
}
}

137
tests/test_plugin_lifecycle.py
View File

@@ -149,6 +149,143 @@ def test_enable_seeds_plugin_permissions(manager, app, db):
assert Permission.query.filter_by(name='alpha.view').count() == 1
# --- ADR-013 Phase 0: reverse-deps from disk, cycles, upgrade-all, profiles --
def test_disable_blocked_by_unloaded_dependent(manager):
"""alpha cannot be disabled while an enabled dependent exists, even when
that dependent was never loaded as an instance (manifest read from disk)."""
# Mark both enabled directly in the registry WITHOUT loading beta.
manager.registry.enable('alpha')
manager.registry.enable('beta')
assert manager.loader.get_loaded_plugin('beta') is None
# The old loaded-instance-only check would wrongly allow this.
assert manager.disable_plugin('alpha') is False
assert manager.registry.is_enabled('alpha')
def test_uninstall_blocked_by_disabled_dependent(manager):
"""An installed-but-DISABLED dependent still blocks uninstalling its
dependency (uninstall reads all installed manifests, not enabled instances)."""
assert manager.registry.is_installed('beta')
assert not manager.registry.is_enabled('beta')
assert manager.uninstall_plugin('alpha') is False
assert manager.registry.is_installed('alpha')
def test_dependency_cycle_detected(tmp_path):
"""A -> B -> A raises instead of looping or silently dropping a plugin."""
from shopdb.exceptions import PluginDependencyError
plugins_dir = tmp_path / 'plugins'
plugins_dir.mkdir()
_write_plugin(plugins_dir, 'aa', deps=['bb'])
_write_plugin(plugins_dir, 'bb', deps=['aa'])
loader = PluginLoader(plugins_dir, PluginRegistry(tmp_path / 'plugins.json'))
with pytest.raises(PluginDependencyError):
loader._sort_by_dependencies(['aa', 'bb'])
def test_upgrade_all_only_touches_registered(manager, tmp_path):
"""upgrade-all migrates only ADOPTED plugins; an unregistered folder on
disk is never migrated as a side effect."""
_write_plugin(tmp_path / 'plugins', 'gamma') # on disk, not registered
class StubMigrationManager:
def has_pending_migrations(self, name):
return False
manager.migration_manager = StubMigrationManager()
results = manager.upgrade_all_plugins()
assert set(results) == {'alpha', 'beta'}
assert 'gamma' not in results
def test_apply_profile_enables_dependency_closure(manager):
"""Naming only the dependent enables it AND its dependency, in order."""
result = manager.apply_profile(['beta'])
assert manager.registry.is_enabled('alpha')
assert manager.registry.is_enabled('beta')
assert 'alpha' in result['enabled']
assert 'beta' in result['enabled']
def test_apply_profile_installs_and_enables_fresh_plugin(manager, tmp_path):
"""A plugin on disk but not yet registered is installed AND enabled."""
_write_plugin(tmp_path / 'plugins', 'gamma')
class StubMigrationManager:
def has_pending_migrations(self, name):
return False
def run_plugin_migrations(self, name, revision='head'):
return True
def has_unapplied_migrations(self, name):
return False
manager.migration_manager = StubMigrationManager()
result = manager.apply_profile(['gamma'])
assert manager.registry.is_installed('gamma')
assert manager.registry.is_enabled('gamma')
assert 'gamma' in result['installed']
def test_apply_profile_rejects_unknown_plugin(manager):
"""A profile naming a plugin that is not on disk fails loudly."""
from shopdb.exceptions import PluginError
with pytest.raises(PluginError):
manager.apply_profile(['doesnotexist'])
def test_manifest_schema_check():
"""The no-dependency manifest schema check flags missing/mistyped fields
and accepts a valid manifest (with unknown extra fields)."""
from shopdb.plugins.cli import _check_against_schema, _load_manifest_schema
schema = _load_manifest_schema()
good = {'name': 'x', 'version': '1.0.0', 'description': 'd',
'dependencies': [], 'tier': 'optional', 'somefuture': 42}
assert _check_against_schema(good, schema) == []
bad = {'name': 'x', 'version': '1.0.0', 'description': 'd',
'dependencies': 'notalist', 'tier': 'bogus',
'default_enabled': 'yes'}
errors = _check_against_schema(bad, schema)
assert any('dependencies' in e for e in errors)
assert any('tier' in e for e in errors)
assert any('default_enabled' in e for e in errors)
assert any('version' in e for e in _check_against_schema({'name': 'x'}, schema))
def test_all_bundled_manifests_match_schema(app):
"""Every bundled plugin's manifest passes the schema - the schema must
describe reality, not an idealized shape."""
from shopdb.plugins.cli import _check_against_schema, _load_manifest_schema
schema = _load_manifest_schema()
pm = app.extensions['plugin_manager']
for name in pm.loader.discover_plugins():
manifest = pm.loader.load_manifest(name)
assert _check_against_schema(manifest, schema) == [], name
def test_validate_bundled_plugin_passes(app):
"""`flask plugin validate <bundled>` exits clean on a real plugin."""
runner = app.test_cli_runner()
result = runner.invoke(args=['plugin', 'validate', 'printers'])
assert result.exit_code in (0, None), result.output
assert 'valid' in result.output
# --- Seed idempotency -------------------------------------------------------
def test_seed_settings_is_idempotent(app, db):