ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/wj' into wilm

Browse Source
This commit is contained in:
benjamin.baker
2026-09-19 14:40:46 -04:00
parent 32f4ccdd48 bd6e736798
commit 8d6e5db347
58 changed files with 37493 additions and 6385 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
.gitattributes vendored
View File

@@ -10,3 +10,10 @@ scripts/check-naming-and-style.sh text eol=lf
# nowhere on the error path - which is the path that matters in a promote
# script.
*.cmd text eol=crlf
# Vendored API-docs renderers (Redoc, Swagger UI). Minified upstream builds of
# code nobody here wrote: one line, over a megabyte each. -diff keeps a textual
# diff of that line out of every review, and linguist-vendored keeps them out of
# the language breakdown, which they would otherwise dominate.
shopdb/core/api/staticdocs/*.js linguist-vendored -diff
shopdb/core/api/staticdocs/*.css linguist-vendored -diff

36
README.md
View File

@@ -2,6 +2,10 @@
A modern rewrite of the classic ASP/VBScript ShopDB application using Flask (Python) and Vue 3. This application manages shop floor machines, PCs, printers, applications, and related infrastructure for manufacturing environments.
**A site runs ShopDB as containers.** `docker compose up -d --build`, then the
first-run wizard at `/setup`. The Windows/IIS installer is the legacy route,
supported for the sites already on it and not the path for a new one.
## Overview
ShopDB tracks and manages:
@@ -108,12 +112,12 @@ SQLite). Do not run dev or production against SQLite.
### Distribution
Source lives here. The Windows installer ships as a release asset rather than a
file in the repository - it is around 240 MB, well past what a repository takes
and well inside what a release asset does. There is no package or image
registry; a site installs from that `.exe` or builds from source.
Source lives here, and a site runs it as containers. The Windows installer is
the LEGACY route: it ships as a release asset rather than a file in the
repository (around 240 MB), and it exists for the sites already running that
way. New sites do not use it.
### Fast path (Docker)
### Standing up a site (Docker)
The Docker image builds the Vue frontend and serves it from the API container,
so a container deploy needs no separate Node build step.
@@ -140,7 +144,9 @@ admin headlessly instead of using the wizard:
docker compose exec api flask seed admin --username admin --email admin@facility.example.com
```
### Manual path (venv + Node)
### Manual path (venv + Node) - development only
For working on ShopDB itself. A site runs the containers above.
```bash
# Backend
@@ -179,18 +185,24 @@ example of the whole sequence.
### Which deployment route
**Docker is the supported route for a new site.** Windows/IIS is legacy: it is
what several sites run today, it stays supported, and it will be deprecated.
Nothing new is built for it.
| Target | Use |
|---|---|
| **Windows Server + IIS** (how sister sites run) | **the INSTALL-WINDOWS page in the project wiki** - one installer `.exe`, offline, no manual IIS work. Day 2: the OPERATE-WINDOWS page in the project wiki |
| Linux / Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki |
| Linux / Docker, connected | the DEPLOY page in the project wiki |
| **Docker, connected** | **the DEPLOY page in the project wiki** |
| **Docker, air-gapped** | **the DEPLOY-AIRGAP page in the project wiki** |
| Windows Server + IIS (existing sites) | the INSTALL-WINDOWS page in the project wiki - one installer `.exe`, offline. Day 2: OPERATE-WINDOWS |
| Moving off IIS onto Docker | the MIGRATE-TOPOLOGY page in the project wiki |
For every environment variable and Setting key see the CONFIG page in the project wiki.
### What the Windows installer requires
### What the Windows installer requires (legacy route)
The installer checks all of this before it changes anything and refuses rather
than half-installing.
For sites already on Windows/IIS. A new site should be reading the DEPLOY page
instead. The installer checks all of this before it changes anything and refuses
rather than half-installing.
| | Minimum | Notes |
|---|---|---|

200
docs/adr/ADR-001-asset-as-platform-contract.md Normal file
View File

@@ -0,0 +1,200 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why Asset, not the legacy Machine model, is the surface plugin authors may depend on.
loadwith: none
-->
# ADR-001: Asset model is the platform contract
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
shopdb-flask is being shaped as a framework that sister GE Aerospace facilities can adopt. The framework defines a stable core; sites install plugins for the asset classes they care about (equipment, computers, printers, measuring tools, network gear, etc.).
The codebase ran two parallel object models:
1. **Legacy `Machine` model** in `shopdb/core/models/machine.py`. Original schema inherited from the classic-ASP shopdb. Tables: `machines`, `pctypes`, `machinetypes`. Plugins like printers stored extension data via `PrinterData` keyed by `machineid`.
2. **New `Asset` model** in `shopdb/core/models/asset.py`. Generic asset abstraction with `AssetType`, `AssetStatus`, `AssetRelationship`. Plugins also exposed asset-based tables keyed by `assetid`.
For the framework to be adoptable, the platform contract has to be one model, documented, versioned, and stable.
## Decision
**`Asset` is the platform contract.** Plugin authors target the asset-based API. The `Machine` model and its dependents (`PCType`, `MachineType`, `PrinterData` keyed by `machineid`) are legacy and will be retired through a tracked migration.
### Platform contract surface
The following are the public, versioned surface. Plugin authors may depend on them. Breaking changes require a major version bump per ADR-002.
#### Models
- `Asset` (core entity)
- `AssetType` (asset classification, registered by plugins)
- `AssetStatus` (active / inactive / decommissioned / retired)
- `AssetRelationship` + `RelationshipType` (cross-plugin links)
- `Vendor`, `Location`, `LocationType`, `BusinessUnit`, `Model`, `OperatingSystem` (shared reference data)
#### Helpers
- `AuditLog` API: `audit_log(action, entitytype, entityid, ...)` for plugins to record audit entries with consistent schema
- `Setting` API: `plugin.get_setting(key)` and `plugin.set_setting(key, value)` for plugin-scoped config persisted via the core `Setting` model
- `resolve_asset_position(asset)` for the documented position-resolution algorithm
#### Import surface (`shopdb.api`) - expanded in __contract_version__ 0.3.0
`shopdb.api` is the ONLY core module plugins may import (plus `shopdb.plugins.base`
for the ABC). Deep imports of `shopdb.core.*`, `shopdb.extensions`, or
`shopdb.utils.*` are contract violations enforced by the
`test_plugins_only_import_contract_surface` test. The surface re-exports:
- Infrastructure: `db`, `cache`
- Model bases: `BaseModel`, `AuditMixin`
- Core models: `Asset`, `AssetType`, `AssetStatus`, `Vendor`, `Model`,
`Communication`, `CommunicationType`, `Location`, `Setting`, `AuditLog`,
`Application`, `AppVersion`, `OperatingSystem`
- Responses: `success_response`, `error_response`, `paginated_response`, `ErrorCodes`
- Pagination: `get_pagination_params`, `paginate_query`
- Helpers: `audit_log`, `resolve_asset_position`; legacy `employee_connection`
Adding a name here is a minor (additive) contract change; removing one is major.
#### Plugin contract
- `BasePlugin` ABC and its hooks (navigation, dashboard widgets, collector schema + `apply_collector_payload`). Note: `get_searchable_fields` was removed in contract 0.4.0 - global search is a core concern over the asset model, not a per-plugin hook.
#### Excluded from the contract for v1
- Event bus (`get_event_handlers` removed from `BasePlugin`). Add later if a real use case appears.
### Relationship types (seeded core values)
`RelationshipType` is seeded with three rows. Plugin authors do not add new types in v1.
| Type | Meaning | Boundary rule |
|------|---------|---------------|
| `partof` | Composition, siblings, sub-assemblies | Parent dies without it |
| `controls` | One asset has operational authority over another | PC commands a machine, measuring tool, etc. |
| `connectedto` | Network or data link without operational authority | Cable, switch port, NAS mount |
### AssetRelationship columns
```
AssetRelationship
- relationshipid PK
- sourceassetid FK to assets
- targetassetid FK to assets
- relationshiptypeid FK to relationshiptypes (one of partof, controls, connectedto)
- label text, free description (e.g., "ethernet PoE", "DNC feed")
- inheritsposition bool, true means resolved-position walk follows this edge
- propagatesthroughid FK to relationshiptypes, nullable, see propagation below
- notes text
- isactive bool
- createddate, modifieddate
```
Free-text `label` carries the domain nuance (`controls` with label "DNC feed" vs `controls` with label "operator workstation"). Avoids inflating the type list.
### Sibling propagation
When a relationship is created or deleted, the framework checks `RelationshipType.propagatesthroughid`. If set, it finds all assets related to the source via the propagation type and applies the same change.
Default seeding:
| RelationshipType | propagatesthroughid |
|------------------|---------------------|
| `partof` | null (the propagation rail itself) |
| `controls` | `partof` (controls relationships propagate across siblings) |
| `connectedto` | null (network paths don't propagate) |
Cycle protection: max walk depth 3, visited-set during traversal.
### Position resolution
Resolved map position for any asset follows this priority chain:
1. Asset-specific override (`assets.mapx`, `assets.mapy` if non-null)
2. Walk relationships where `inheritsposition=true`, ordered by relationship type priority (`partof` first, then `controls`), recursively resolve the related asset's position with cycle detection
3. Fall back to the asset's location coords (`locations.mapx`, `locations.mapy` via `assets.locationid`)
4. If none of the above, asset is "unplaced" and rendered in a tray, not on the map
API exposes resolved position with a source indicator: `{"mapx": 234, "mapy": 567, "positionsource": "self|related|location|none"}`.
### Hierarchical locations
`locations` is extended with `parentlocationid` (self-FK, nullable). Sites define their own location tree (cells, sub-cells, network closets, meeting rooms, labs).
`locationtypes` lookup is added with seeded values: `section`, `cell`, `subcell`, `meetingroom`, `lab`, `office`, `storage`, `hallway`, `networkcloset`, `building`. Sites can extend.
Asset-to-location is the existing `assets.locationid` FK only (single primary location for v1). Multi-location and transient placement are deferred.
## Migration scope (narrowed)
Only one class of legacy data migrates: physical manufacturing equipment with a `machinenumber` (5-axis mills, lathes, broachers, heat treatment, CMMs, etc.).
Migration filter:
```sql
SELECT * FROM legacy.machines
WHERE category = 'Equipment'
AND machinenumber IS NOT NULL
```
Rows without `machinenumber`: **skipped.** Add manually post-migration if any matter.
Migrated rows become `assets` with `assettype='Equipment'` plus a row in the equipment plugin's `equipment` table. Legacy `machinetypeid` is preserved on the equipment row to enable later reclassification (some "Equipment" rows are actually measuring tools, see ADR-005).
Skipped from migration:
- PCs (rebuilt via PXE collector pipeline, see ADR-006)
- Printers, USB devices, network gear, notifications, KB articles (operational or re-collectable, decision per class deferred until needed)
## Consequences
### Positive
- Plugin authors at sister sites have a single, documented contract to target
- Cross-plugin features (relationships, search, map) work uniformly
- Three relationship types are easy to learn; free-text label captures nuance
- Sibling propagation handles dual-path machines without code changes per use case
- Position resolution gives users one map without per-plugin map code
### Negative
- Migration of equipment data must run cleanly; `migrating-asset-schema` skill owns the procedure
- Frontend `views/machines/` will be repointed or replaced; effort tracked in Phase 5
- Legacy `core/api/machines.py` (641 LOC) becomes deprecated; deletion or shim deferred until equipment is migrated
### Neutral
- `Machine`, `PCType`, `MachineType`, `PrinterData` remain in the schema until the migration completes, behind a deprecation notice. New code does not touch them.
## Alternatives considered
1. **Keep both models indefinitely.** Doubles test surface, confuses contract docs, breaks cross-plugin features. Rejected.
2. **Make `Machine` the contract; retire `Asset`.** `Machine` has shop-floor-specific assumptions that don't generalize. Rejected.
3. **Define a higher abstraction above both.** Yet another layer. `Asset` is already the abstraction. Rejected.
4. **More relationship types** (`controls`, `operates`, `monitors`, `dncfeeds`, `mountedon`, `connectedvia`, `inspects`, `siblingbay`). Eight types proved unwieldy. Collapsed to three with free-text labels and per-row inheritance/propagation flags.
## Open questions deferred
- Lifecycle relationship type (`replaces`, `replacedby`). Add when first needed.
- Multi-location asset placement, transient placement (calibration trip, off-site repair). Add when first needed.
- Map editing UI (drag locations onto floor plan). Phase 6 polish.
- Plugin-extensible relationship types beyond the three seeded. Add when a real cross-plugin case can't be expressed via labels.
## References
- `shopdb/core/models/asset.py`
- (core `Machine` model retired per this decision; machine data now owned by the machines plugin at `plugins/machines/models/machine.py`)
- `shopdb/plugins/base.py`
- ADR-002 (versioning of the surface)
- ADR-003 (plugin distribution)
- ADR-004 (deployment topology)
- ADR-005 (equipment vs measuring tools split)
- ADR-006 (collector contract pattern)

68
docs/adr/ADR-002-plugin-versioning.md Normal file
View File

@@ -0,0 +1,68 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: How the plugin contract is versioned and what core_version a plugin must declare.
loadwith: none
-->
# ADR-002: Plugin contract versioning
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
Once sister sites start writing their own plugins (or pulling community plugins), the framework's plugin contract becomes a public API. Without a versioning story, any change to `BasePlugin` or the core platform models can silently break installed plugins at remote sites.
The existing `BasePlugin` and `PluginMeta` already declare a `core_version` field (default `">=1.0.0"`), but it is not enforced anywhere. The plugin loader does not check it before instantiation.
## Decision
The framework adopts **semantic versioning for the plugin contract**, declared in two places:
1. **Framework version** (`shopdb/__init__.py`): a single `__contract_version__` constant. This is the version of the platform contract as defined in ADR-001. Bumped according to semver:
- **Major**: breaking change to `BasePlugin` ABC, `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`).
- **Minor**: additive change (new optional hook, new field on a contract model with default).
- **Patch**: bug fix, no contract surface change.
2. **Plugin requirement** (`plugins/<name>/manifest.json`): the existing `core_version` field, expressed as a semver range (e.g., `">=1.0.0,<2.0.0"`).
The plugin loader (`shopdb/plugins/loader.py`) checks `core_version` against `__contract_version__` at load time. Mismatch in dev = re-raise (fail loud). Mismatch in prod = log error, mark plugin as incompatible, exclude from registration.
The `__contract_version__` starts at **`1.0.0`** when ADR-001 is accepted and the `Machine` retirement migration is complete (whichever comes later). Until then, the framework is pre-1.0; plugins should declare `core_version: ">=0.1.0,<1.0.0"`.
## Consequences
### Positive
- Sister sites can pin a known-good framework version. They will not be silently broken when the framework is upgraded.
- Plugin authors know what counts as a breaking change because the contract surface is enumerated in ADR-001.
- The loader fails predictably: a mismatched plugin is reported, not silently disabled.
### Negative / cost
- Discipline required: every change to the contract surface must be classified (major / minor / patch). Adding a `version-bump` skill (or a check in code review) reduces the chance of mis-classification.
- `__contract_version__` becomes a coupling point. Forgetting to bump it after a breaking change means downstream plugins crash silently at runtime instead of failing at install.
### Neutral
- Existing plugins (`plugins/printers/`, etc.) ship as part of the framework, so their `core_version` is always the current `__contract_version__`. The discipline matters mostly for external / sister-site plugins.
## Alternatives considered
1. **No versioning, just trust.** Works for an in-tree-only world. Fails the moment a sister site ships its own plugin. Rejected.
2. **Calendar versioning** (e.g., `2026.05.0`). Easier to bump, harder to communicate breaking changes. Rejected; semver is the industry standard for library-like contracts.
3. **Per-hook versioning.** Each hook has its own version. Too granular; plugins still couple to multiple hooks. Rejected.
## Open questions
- When does the framework declare `1.0.0`? Tied to ADR-001 (Asset retirement of Machine) and the framework being deemed "ready for sister sites". Best-effort target: end of Phase 5 in the refactor plan.
- Should `core_version` accept commercial-grade ranges (`^1.0.0`) or stick to PEP 440 / npm-style ranges? Recommend pip-style (`>=,<`) to match Python ecosystem.
## References
- `shopdb/plugins/base.py` (PluginMeta declaration)
- `shopdb/plugins/loader.py` (where the version check belongs)
- ADR-001 (defines what is in the contract)

76
docs/adr/ADR-003-plugin-distribution.md Normal file
View File

@@ -0,0 +1,76 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How a plugin reaches a site: bundled in-tree, or dropped into plugins/ from its own repo.
loadwith: none
-->
# ADR-003: Plugin distribution model
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
## Context
Sister sites adopting shopdb-flask need a way to:
1. Install the framework
2. Pick which plugins they want
3. Build their own plugins for site-specific equipment
4. Receive updates to both framework and plugins
Today, every plugin lives in the `plugins/` directory of the framework repo. There is no separation between framework code and plugin code, no install / uninstall, and no way for a site to develop a plugin without forking the whole repo.
Three viable distribution models:
| Model | How a site installs a plugin |
|---|---|
| **In-tree only** | Fork the framework repo, add plugin under `plugins/`, run their own deploy. No separation. |
| **Pip-installable plugins** | Each plugin published as a Python package. Site does `pip install shopdb-printers shopdb-network` etc. Discovery via Python entry points. Framework loads any installed plugin that registers itself. |
| **Git-based plugins** | Each plugin lives in its own git repo. Site clones / submodules into `plugins/<name>/`. Loader picks them up from the directory. |
## Decision
Use a **hybrid model** with two clearly-labeled paths.
1. **Bundled plugins**: a small set of plugins ships with the framework, in-tree at `plugins/`. These are the reference implementations and the default install (printers, computers, network, equipment, usb, notifications). A site that wants only what's bundled needs no extra work.
2. **External plugins**: sister sites or third parties build plugins in their own git repos. The site running the framework drops the plugin into `plugins/<name>/` (clone, submodule, or symlink) and runs `flask plugin install <name>`. No pip packaging required for v1.
Pip-installable plugins (Python entry-point discovery) are deferred to v2. The complexity is not justified until at least two sites are running their own plugins.
## Consequences
### Positive
- v1 is simple: filesystem-based discovery (already implemented in `shopdb/plugins/loader.py`), works for both bundled and external plugins.
- Sites can develop plugins without changing the framework repo.
- The `plugins/` directory is already the canonical location, so no architectural change is needed.
### Negative / cost
- No automatic update path for external plugins. Sites must `git pull` in each plugin directory manually. Acceptable for v1; revisit when plugin count grows.
- Multiple plugin authors writing in parallel can collide on namespace (e.g., two plugins both registering an `AssetType` named "Equipment"). Need a naming policy: plugin names and asset-type names should be prefixed with the site or org if not in the bundled set.
### Neutral
- The existing in-tree pattern keeps working. This decision just formalizes it and clarifies the path for outside-the-tree plugins.
## Alternatives considered
1. **Pip-installable from day one.** Cleaner for the long term but adds packaging, entry-point registration, and CI steps. Premature for current scale (one site running, no sister-site plugins yet).
2. **In-tree only forever.** Forces every site to fork. Doesn't scale beyond two or three sites.
3. **Submodules only.** Forces git-submodule discipline on every adopting site. Submodules are notoriously fiddly. Rejected.
## Open questions
- For external plugins, should there be a manifest field (`source_url`) declaring where the plugin can be cloned from, so `flask plugin install` could pull it for the site? Defer; manual clone is fine for v1.
- Naming convention for non-bundled plugin directory names: prefix with site? (`gea-wjsf-shipping`)? Adopt if and when we hit a name collision.
## References
- `shopdb/plugins/loader.py` (filesystem discovery)
- `shopdb/plugins/cli.py` (plugin install / uninstall command)
- ADR-001 (defines what plugins target)
- ADR-002 (defines plugin version compatibility)

100
docs/adr/ADR-004-deployment-topology.md Normal file
View File

@@ -0,0 +1,100 @@
<!-- docmeta
purpose: explanation
audience: adopter
status: current
summary: Why every site runs its own instance instead of one multi-tenant deployment.
loadwith: none
-->
# ADR-004: Deployment topology (per-site instances)
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
## Context
shopdb-flask manages shop-floor inventory. If multiple GE Aerospace sites adopt it, the deployment can take one of two shapes:
| Model | How it works |
|---|---|
| **Per-site instances** | Each site runs its own Flask + MySQL + Vue stack. Each site has its own DB, its own users, its own enabled-plugin list, its own deploy. Sites are isolated. |
| **Multi-tenant single instance** | One central Flask + MySQL + Vue stack serves all sites. A `siteid` foreign key on every asset partitions data. Auth distinguishes which site a user belongs to. |
The codebase today is single-tenant per deployment. There is no `siteid` column, no tenant filter, no cross-site auth model. Plugins can be enabled / disabled but only globally for the running instance.
## Decision
**Per-site instances.** Each adopting site runs its own dedicated stack. The framework does not support multi-tenancy.
Each site:
- Owns its database (own credentials, own backup policy, own retention). The
database charset is part of the contract: it must be **utf8mb4**
(`utf8mb4_unicode_ci`). The migration chain creates every table utf8mb4, and
the connection pins `?charset=utf8mb4`. A site that creates the database with
a different default charset (older MySQL defaults to latin1) gets a schema
that silently diverges from every other site. See `docs/DEPLOY.md`.
- Picks its own enabled plugins
- Configures its own JWT secret, CORS allowlist, Zabbix integration, Active Directory binding
- Deploys at its own cadence
The framework provides:
- A `Dockerfile` and `docker-compose.yml` template suitable for a single-site deploy
- A `.env.example` listing all required environment variables
- A `docs/DEPLOY.md` walking through a fresh-site install
## Consequences
### Positive
- Simpler code: no tenant filter on every query, no cross-tenant auth, no shared-state partitioning bugs.
- Sites are independent. A schema change at one site does not affect another. A plugin crash at one site does not blast radius to other sites.
- Clear ownership: each site's IT team owns their own stack and data. Compliance and audit boundaries match operational boundaries.
- Aligns with how GE Aerospace sites already operate (independent IT, independent shop floors).
### Negative / cost
- No cross-site reporting out of the box. If GE corporate ever wants a fleet-wide view, it has to be built on top (e.g., a roll-up dashboard that queries each site's API). That layer is out of scope for the framework.
- Each site administers its own stack. Higher operational overhead than a single central instance, but each site already runs its own infrastructure.
- Updates require visiting each site's deploy. Fine for the current adoption model; revisit if dozens of sites adopt.
### Neutral
- No `siteid` column needed. The existence of one DB per site is the partition.
## Alternatives considered
1. **Multi-tenant single instance.** Lower operational overhead at scale, easier cross-site reporting, but adds significant code complexity and risk: every query needs a tenant filter, auth gets complex, schema migrations affect every site at once, and a bug at one site can leak data across sites. Rejected for v1; revisit if and only if more than five sites adopt and operational overhead becomes painful.
2. **Hybrid: per-site DB but central app server.** Adds the operational complexity of multi-tenancy without isolating the failure domain (one app crash = all sites down). Rejected.
## Migration strategy (resolved)
Deploys run two commands: `flask db upgrade` then `flask plugin upgrade-all`
(lean/ADR-014 sites add an optional `flask plugin prune-schema` at initial
provisioning). The core Alembic chain applied by `flask db upgrade` creates the
full core AND bundled-plugin schema through the chain head (this includes
migration `7c04_fold_plugin_schema`). But every bundled plugin still carries its
own Alembic chain per ADR-008: `flask plugin upgrade-all` stamps each plugin's
own chain (the `alembic_version_<plugin>` tables) and applies any plugin-specific
migrations added after the ownership cutover. The earlier Phase 7B state that
folded everything into core with no per-plugin chains was superseded by ADR-008's
per-plugin ownership. A fresh `flask db upgrade` reproduces the live core schema
exactly (verified on a scratch DB).
External (out-of-tree) plugins per ADR-003 ship their own migrations too; the
framework runs the same per-plugin chain mechanism (ADR-008) for both bundled and
external plugins.
## Open questions
- Should the framework provide an optional **read-only fleet roll-up** mode where a "central" instance can pull aggregate metrics from each site's API? Defer. Out of scope for v1.
- Backup strategy per site: framework recommendation, or each site decides? Framework should publish a recommended backup runbook (mysqldump + offsite copy) but not enforce.
- Auth federation: each site has its own user table, or sites can share an LDAP / SSO? Recommend documenting the LDAP config knob in `.env.example` so sites can plug in their own auth without code change.
## References
- `shopdb/config.py` (currently single-tenant, no `siteid`)
- ADR-001 (asset model is per-site, not cross-site)
- ADR-003 (plugin distribution per site)

163
docs/adr/ADR-005-equipment-vs-measuringtools.md Normal file
View File

@@ -0,0 +1,163 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026 split of machinery from metrology tools into two plugins. Superseded by ADR-018.
loadwith: none
-->
# ADR-005: Equipment plugin scope vs measuringtools plugin
- **Status:** SUPERSEDED by [ADR-018](ADR-018-metrology-instruments-are-machines.md)
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
> **Do not build against this.** ADR-018 (2026-09-04) retired the
> `measuringtools` plugin and made metrology instruments machines. The split
> below was sound for the population it described, and that population never
> arrived: the plugin held zero rows. Kept as the record of why the split was
> made and why it was undone.
## Context
ADR-001 narrowed the migration to physical manufacturing equipment with a `machinenumber`. In practice, the legacy `category='Equipment'` rows contain two distinct asset classes:
1. **Manufacturing machinery** (5-axis mills, lathes, broachers, heat treatment ovens). These produce parts.
2. **Metrology and inspection instruments** (CMMs, Keyence vision systems, wax-and-trace surface profilometers, GenSpec instruments). These measure parts.
Both share `Asset` properties (vendor, model, location, controller). They differ in domain fields (axes vs measurement accuracy, cycle time vs calibration interval, controller protocol vs measurement software).
Mixing them under one plugin pollutes the schema and confuses cross-plugin queries ("show me all measuring tools" requires an enumeration of measuring-instrument equipmenttype values, which scales badly).
## Decision
Two plugins, separate concerns, shared platform contract.
### `equipment` plugin
Tracks manufacturing machinery. Bundled, in-tree.
Schema (per ADR-001 contract):
```
equipment
- assetid FK to assets, PK
- equipmenttypeid FK to equipmenttypes (5-axis mill, lathe, broacher, heat treat, ...)
- vendorid FK to vendors (platform)
- modelid FK to models (platform)
- controllertypeid FK to controllertypes (equipment plugin)
- controllerosid FK to controlleros (equipment plugin)
- (other shared fields: spindle count, axes, max workpiece size, ...)
equipmenttypes (lookup, equipment plugin)
- equipmenttypeid, name (5-axis mill, lathe, broacher, heat treat, ...)
controllertypes (lookup, equipment plugin)
- controllertypeid, name (Fanuc 31i, Siemens 840D, Mitsubishi M70, Heidenhain TNC640, ...)
- vendorid (FK to vendors)
controlleros (lookup, equipment plugin - separate from PC OS)
- controllerosid, name (FAPT, VxWorks, embedded Windows, Linux RT, ...)
equipmentfocas (subtype, optional, present only when FOCAS-equipped)
- assetid PK, FK to equipment
- focasipaddress text
- focasport integer
- focasversion text
- focasmachinenumber text
equipmentclm (subtype, optional, present only when CLM-equipped)
- assetid PK, FK to equipment
- (CLM-specific: address, port, station ID - finalize when plugin is built)
equipmentmtconnect (subtype, optional, present only when MTConnect-equipped)
- assetid PK, FK to equipment
- mtconnectagenturl text
- mtconnectdevicename text
```
The `equipment.protocol` enum field is deliberately **not** included. Presence or absence of a subtype row indicates which protocol applies. Avoids a denormalized field that can drift out of sync.
### `measuringtools` plugin
Tracks metrology and inspection instruments. Bundled, in-tree (built in Phase 3-4 of the refactor as the first new plugin built using the framework scaffold).
Schema (initial draft, refined when plugin is built):
```
measuringtools
- assetid FK to assets, PK
- measuringtooltypeid FK to measuringtooltypes (CMM, vision system, profilometer, surface tester, ...)
- vendorid FK to vendors (platform)
- modelid FK to models (platform)
- measurementaxes integer (e.g., 3 for a 3-axis CMM)
- accuracyspec text (e.g., "+/-0.5um")
- calibrationintervaldays integer
- lastcalibrationdate date
- nextcalibrationdate date (computed)
- (other domain fields as needed)
measuringtooltypes (lookup, measuringtools plugin)
- measuringtooltypeid, name (CMM, vision system, surface profilometer, gage block, ...)
```
Future extension: subtype tables for measurement-software integrations (PC-DMIS, Keyence, GenSpec). Same pattern as equipment subtype tables.
### Subtype-table pattern (general)
Both plugins use the same pattern for protocol- or software-specific fields:
- Core plugin table carries shared, common fields
- Optional subtype tables (one per protocol or software) hold extension fields
- Each subtype table is keyed by `assetid` (PK), one-to-one with the parent
- Subtype row exists if and only if the asset uses that protocol or software
- Sister sites add new subtype tables for their own integrations without touching core
## Reclassification of legacy data
ADR-001's migration moves all legacy `category='Equipment' AND machinenumber IS NOT NULL` rows to `assets` with `assettype='Equipment'` and into the equipment plugin's `equipment` table. This includes both manufacturing machinery and measuring tools.
After the equipment migration, when the measuringtools plugin is built:
1. Build a mapping table: legacy `machinetypeid` values that are measuring tools (CMM type, Keyence type, etc.)
2. Run a reclassification script:
- For each `assets` row where the original `machinetypeid` is in the measuring-tool mapping
- Change `assets.assettype` to `'MeasuringTool'`
- Move the row from `equipment` to `measuringtools`
- Map domain fields where they differ (e.g., legacy `axes` field maps to `measurementaxes`)
3. Verify counts pre- and post-reclassification
4. Audit log entry per reclassified row
Reclassification is one-shot, run once, archived. Like the original migration script.
## Consequences
### Positive
- Manufacturing machinery and measuring tools are first-class plugins, each with appropriate domain fields
- Sister sites can install one or both depending on what they track
- Subtype-table pattern is the canonical example for protocol-specific data and extends naturally to other plugins
- Building `measuringtools` mid-refactor validates the plugin scaffold tooling against a real new plugin
### Negative
- Reclassification is a second migration step. Lower risk than the initial migration because it is data-only (no schema change beyond moving rows between two tables that share the same `assetid` link).
- Sites that adopt the framework before `measuringtools` ships need to either keep measuring tools in `equipment` (workable but suboptimal) or wait for the plugin
### Neutral
- Legacy `machinetypeid` is preserved on the equipment row during migration to enable reclassification
## Alternatives considered
1. **Single equipment plugin with sub-typed assets.** Use `equipment.equipmenttypeid` to discriminate manufacturing vs metrology. Rejected: domain fields differ enough that a single table is wide and full of NULLs.
2. **Migrate split (build mapping before initial migration).** Cleaner end state but requires the `measuringtools` plugin to exist before the migration runs, which delays Phase 5. Rejected.
3. **JSON blob for protocol data instead of subtype tables.** Considered for both plugins. Rejected: weak typing, awkward queries, no schema validation.
## References
- ADR-001 (Asset is platform contract)
- ADR-002 (versioning of the surface)
- `plugins/machines/` (equipment plugin, renamed per ADR-011)
- `plugins/computers/` (existing example of plugin pattern)

156
docs/adr/ADR-006-collector-contract.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer, integrator
status: current
summary: How a plugin declares a collector schema so the framework wires the ingest endpoint and auth.
loadwith: none
-->
# ADR-006: Plugin collector contract pattern
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
PC inventory data was collected by PowerShell scripts pushing to `/api/collector/pc` (`shopdb/core/api/collector.py`, ~374 LOC). The endpoint is hardcoded for PCs: it accepts a fixed schema and writes to the legacy `Machine` model.
Per ADR-001, `Machine` is being retired in favor of `Asset`. Per the project shift to PXE-driven imaging, PC inventory is moving to a new collection pipeline (PXE / GE-Enforce / manifest engine produces JSON about each PC). Other asset classes may want similar collector pipelines (printers via Zabbix, network gear via SNMP scan).
This calls for a generalizable contract: any plugin that wants to accept external collector input declares a JSON schema, and the framework wires the endpoint, auth, and idempotency.
## Decision
`BasePlugin` gets two new hooks (added in __contract_version__ 0.2.x -> the surface is carried at 0.3.0):
```python
def get_collector_schema(self) -> Optional[dict]:
"""Return JSON Schema describing the collector payload for this plugin.
Return None if the plugin does not accept collector input.
The schema must include:
- 'identityfield': name of the field that uniquely identifies an asset
across submissions (e.g., 'hostname' for PCs, 'macaddress' for network
devices). Used for idempotent upsert.
- 'fields': JSON Schema definitions for the rest of the payload.
"""
return None
def apply_collector_payload(self, payload: dict) -> dict:
"""Idempotently upsert an asset from a validated collector payload.
Called by /api/collector/<pluginname> after identity validation.
CONDITIONAL hook: required only when get_collector_schema returns
non-None. Default raises NotImplementedError (the dispatcher returns
500) so a schema-without-upsert fails loud. Returns a dict with at
least 'action' ('created'|'updated'|'noop'), 'assetid', 'warnings'.
"""
raise NotImplementedError
```
The pairing (schema present => apply implemented) is enforced by the
`test_schema_declaring_plugins_implement_apply` contract test.
A single dynamic dispatch route `/api/collector/<pluginname>` serves every
plugin that returns a schema (rather than registering a blueprint per plugin),
because Flask forbids `register_blueprint` after the first request and plugins
can be enabled at runtime. Auth is API-key, separate from JWT. Per-plugin keys via env vars:
- `COLLECTOR_API_KEY_<PLUGINNAME>` (preferred, plugin-specific)
- `COLLECTOR_API_KEY` (fallback, shared)
### Idempotent upsert
The endpoint uses the `identityfield` to find an existing `Asset` for the same identity. Found = update. Not found = insert. Existing relationships are preserved on update.
### Response contract
```json
{
"status": "ok",
"action": "created" | "updated" | "noop",
"assetid": 12345,
"identityvalue": "PC-1234",
"warnings": []
}
```
### Audit logging
Every collector submission produces an audit log entry: `{action, plugin, identityvalue, before/after diff}`. Audit retention per site policy.
### Schema discovery
The framework exposes the registered schemas at `/api/collector/_schemas` (read-only, JWT-protected) so external collector authors can introspect what payloads are accepted by which plugins.
## Concrete first user: computers plugin
The `computers` plugin is the first to implement `get_collector_schema`. The PXE pipeline conforms.
Initial computers collector schema (sketch, finalized when plugin is built):
```json
{
"identityfield": "hostname",
"fields": {
"hostname": "string, required",
"macaddress": "string, optional, secondary identity",
"osname": "string",
"osversion": "string",
"lastboottime": "datetime",
"currentuser": "string",
"ipaddress": "string",
"memorygb": "number",
"cputype": "string",
"imagename": "string (PXE image deployed)",
"imageappliedat": "datetime",
"installedsoftware": "array of {name, version}"
}
}
```
The PC re-image case is handled by the identity field: a freshly imaged PC keeps its hostname, so the existing `Asset` row is updated rather than duplicated. Existing `AssetRelationship` rows pointing at that PC (e.g., `controls` to a machine) are preserved across re-images.
## Migration of the existing endpoint
`shopdb/core/api/collector.py` (`/api/collector/pc`) is **deprecated** in v1 and **removed** before v1.0.
Migration path:
1. Implement `get_collector_schema` on the `computers` plugin. New endpoint `/api/collector/computers` is auto-registered.
2. Run both endpoints in parallel for one cycle of PXE imaging across the floor. PXE pipeline switches to `/api/collector/computers`.
3. Remove `shopdb/core/api/collector.py` and the legacy blueprint registration.
## Consequences
### Positive
- Generalizable across plugins. Sister sites adopting `printers`, `network`, etc. can wire their own collectors with no core change.
- Identity-based idempotency makes PC re-imaging safe by default.
- Audit logging is uniform across plugins.
- Schema discovery enables external tools to validate before submission.
### Negative
- Plugin authors must write a JSON schema. Slight learning curve, but JSON Schema is widely understood and the framework can ship a few examples.
- The `/api/collector/_schemas` endpoint plus per-plugin endpoints expand the public API surface; minor maintenance cost.
### Neutral
- API-key auth pattern stays as it is today (separate from JWT). Sites manage their own collector keys per plugin via env vars.
## Alternatives considered
1. **Keep `/api/collector/pc` and add new plugin-specific endpoints alongside.** Two ways to send PC data, plugin authors confused. Rejected.
2. **Use JWT for collectors instead of API key.** Collectors are headless processes (PXE pipeline, scripts), not interactive users. JWT lifecycle (refresh tokens, expiry) is the wrong tool. API key is simpler. Rejected.
3. **Plugins write directly to the database, no collector endpoint.** Skips audit logging and schema validation. Rejected.
## References
- `shopdb/core/api/collector.py` (legacy endpoint to be removed)
- `shopdb/plugins/base.py` (`get_collector_schema` + `apply_collector_payload` hooks)
- ADR-001 (asset model the collectors target)
- ADR-002 (collector schema is part of plugin contract; changes to the hook signature are major bumps)
- The PXE project (the PXE imaging project) which feeds the computers collector

123
docs/adr/ADR-007-product-versioning-and-releases.md Normal file
View File

@@ -0,0 +1,123 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How the product is versioned and released, separately from the plugin contract version.
loadwith: none
-->
# ADR-007: Product versioning and releases
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
ADR-002 established semantic versioning for the plugin contract via
`__contract_version__` in `shopdb/__init__.py`. That number answers one
question only: is a given plugin compatible with this platform build. It
says nothing about the state of the product as a whole. A sister site
standing up its own instance needs a different answer: which release am I
running, and what changed since the last one.
Until now the product had no version, no tags, no changelog, and no CI.
ADR-002's pinning model implicitly assumes that a downstream site can
pin a known-good build, but there were no tags to pin to. Adopters had no
release record to read before upgrading, and no automated gate confirming
that a given commit builds and passes tests.
## Decision
The product carries its own release version, separate from the plugin
contract version.
1. **Product version** (`shopdb/__init__.py`): a single `__version__`
constant. This is the version of the shopdb-flask product as a whole.
It follows semantic versioning of the product's user-visible and
operator-visible behavior. It is deliberately NOT part of the
`shopdb.api` contract surface, so it is never re-exported through
`shopdb.api`; exporting it would itself be a contract change under
ADR-002.
2. **Plugin contract version** (`__contract_version__`): unchanged from
ADR-002. It moves only when the plugin contract surface changes, per
the major/minor/patch rules in ADR-002.
The two are distinct series with independent bump rules. They happen to
coincide at `0.5.0` for this release; that is a coincidence of timing,
not a coupling. A product release that changes no contract surface bumps
`__version__` while leaving `__contract_version__` fixed, and vice versa.
3. **Release record** (`CHANGELOG.md`, repo root): the canonical,
human-readable record of what changed in each release, in
Keep-a-Changelog format. Every release has an entry; work in flight
accumulates under `## [Unreleased]`.
4. **Git tags**: each product release is tagged `vX.Y.Z`, where `X.Y.Z`
matches `__version__` at the tagged commit. Tags are what ADR-002's
downstream-pinning model pins to.
5. **Frontend version** (`frontend/package.json`): kept in lock-step with
`__version__` so the shipped single-page app reports the same product
version as the backend that serves it.
### Release procedure
To cut release `X.Y.Z`:
1. Bump `__version__` in `shopdb/__init__.py` to `X.Y.Z`.
2. Bump `version` in `frontend/package.json` to the same `X.Y.Z`.
3. Move the accumulated `## [Unreleased]` notes in `CHANGELOG.md` into a
new `## [X.Y.Z] - YYYY-MM-DD` section, leaving a fresh empty
`## [Unreleased]` above it.
4. If the plugin contract surface changed this release, bump
`__contract_version__` per ADR-002 (independently of `__version__`).
5. Commit, then tag: `git tag -a vX.Y.Z -m "shopdb-flask X.Y.Z"`.
6. Push the commit and the tag.
## Consequences
### Positive
- Adopters have a real release to name in bug reports and a changelog to
read before upgrading.
- ADR-002's pinning story finally has tags to pin to.
- Product and contract can evolve at their own pace without one dragging
the other into a misleading bump.
### Negative / cost
- Two version numbers to keep straight. The comment in
`shopdb/__init__.py` and this ADR exist to keep the distinction clear.
- Release discipline: the changelog and both version constants must be
updated together, or the tagged build misreports itself.
### Neutral
- The internal CI workflow runs the backend tests, the naming/style
gate, and the frontend build on push and PR. It is best-effort: the internal CI
Actions availability on the host is unverified, so the workflow is
config-only until a runner is confirmed.
## Alternatives considered
1. **Reuse `__contract_version__` as the product version.** Conflates two
independent concerns; a docs-only or UI-only release would either
falsely bump the contract or leave the product looking unchanged.
Rejected.
2. **Calendar versioning for the product** (e.g. `2026.07.0`). Easy to
bump but poor at signaling breaking operator-facing changes. Rejected
for the same reasons ADR-002 rejected it for the contract.
3. **No product version, rely on git SHAs.** Opaque to adopters and
unpinnable in any human-meaningful way. Rejected.
## References
- ADR-001 (defines the contract surface)
- ADR-002 (plugin contract versioning; `__contract_version__` bump rules)
- `shopdb/__init__.py` (`__version__`, `__contract_version__`)
- `CHANGELOG.md` (release record)
- `frontend/package.json` (frontend version, kept in lock-step)
- the internal CI workflow (CI gate)

131
docs/adr/ADR-008-plugin-migration-ownership.md Normal file
View File

@@ -0,0 +1,131 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why each plugin owns its own Alembic chain instead of adding tables to the core chain.
loadwith: none
-->
# ADR-008: Plugin migration ownership (per-plugin chains from the cutover)
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** the "Migration strategy (resolved)" section of ADR-004
## Context
ADR-004 resolved a Phase 7B footgun by folding every bundled plugin's tables
into the single core Alembic chain (migration `7c04_fold_plugin_schema`), and
later core migrations (`7d04`..`7d16`) kept adding plugin schema directly to the
core chain. At the time this was the safe choice: bundled-plugin baselines and
the core baseline had both been creating the same tables, so
`flask plugin upgrade-all` would collide with `flask db upgrade`.
That resolution left the per-plugin Alembic engine
(`shopdb/plugins/migrations.py`, `shopdb/plugins/alembic_template.py`, the
`alembic_version_<plugin>` version tables, and `flask plugin upgrade-all`) fully
built but unused for the bundled plugins. The framework is the product (per the
project's charter); a plugin that cannot own its own schema is not really a
plugin. Sister sites that adopt or fork a plugin need its schema history to
travel with the plugin, not be entangled in the host's core chain. Keeping every
future plugin table change in the core chain also means the core chain grows
without bound and a plugin can never be cleanly removed.
The blocker ADR-004 worried about (double table creation) only exists while a
plugin's own migration tries to CREATE tables the core chain already created.
That is avoidable: history is immutable, so the tables already built by the core
chain stay owned by the core chain; only NEW schema needs a new home.
## Decision
Ownership splits at a fixed cutover, the current core-chain head.
1. **Core chain owns history through its head.** The core Alembic chain
(baseline `68b3947ae14f` .. head `7d16_directoryemployees`) remains
authoritative for every table that exists at the cutover, including the
bundled-plugin tables it created. Those migrations are immutable and are not
rewritten. `flask db upgrade` continues to reproduce the full schema.
2. **Plugin chains own plugin schema going forward.** From the cutover forward,
any change to a plugin's schema lands as
`plugins/<name>/migrations/versions/000N_*.py` in that plugin's own chain,
never in the core chain. The core chain is reserved for core tables.
3. **Every table-owning bundled plugin gets a `0001` anchor.** Each such plugin
carries a migration chain whose first revision is a stamp-only no-op:
`upgrade()` does nothing because the core chain already created the tables.
The anchor exists so the plugin chain has a base that
`flask plugin upgrade-all` can stamp into the per-plugin version table
`alembic_version_<plugin>`. Blueprint-only plugins that own no tables get no
chain.
4. **Deploy and upgrade sequence.** A deploy runs `flask db upgrade`
(core chain, creates everything through the head) then
`flask plugin upgrade-all` (stamps every plugin anchor and applies any later
per-plugin migrations). The same two commands upgrade an existing install;
both are idempotent. The registry (`instance/plugins.json`) tracks each
plugin's applied revisions in `migrations_applied`.
5. **Table-ownership registry.** `PLUGIN_TABLE_OWNERS` in
`shopdb/plugins/alembic_template.py` is the explicit map of which tables each
plugin owns; it is kept in sync with the plugins' `__tablename__` declarations
and pinned by `tests/test_plugin_migrations.py`.
External (out-of-tree) plugins per ADR-003 already shipped their own chains;
this ADR brings the bundled plugins onto the same model, so there is one rule
for all plugins.
## Consequences
### Positive
- A plugin's schema history travels with the plugin. Adopting or forking sites
get the plugin's migrations, not a slice of someone else's core chain.
- The core chain stops accreting plugin schema; it stays about core tables.
- A plugin can be evolved (or, with its own downgrade, removed) independently.
- The long-built per-plugin Alembic engine is finally exercised on every deploy,
so it cannot silently rot.
### Negative / cost
- Two migrate commands per deploy instead of one. Documented in `docs/DEPLOY.md`
and `docs/UPGRADE.md`; both are idempotent so the cost is one extra safe call.
- A plugin author must now put new tables in the plugin chain and register them
in `PLUGIN_TABLE_OWNERS`, rather than autogenerating into the core chain. The
`flask plugin new` guidance and this ADR spell that out.
- The cutover is a discontinuity: tables created before it are core-owned,
tables created after it are plugin-owned. The line is the core-chain head at
this ADR's date, recorded here so it is unambiguous.
### Neutral
- No schema changes and no data migration: the anchors are no-ops. A fresh
install and an existing install converge to the same state.
## Alternatives considered
1. **Keep everything in the core chain (status quo per ADR-004).** Simplest
operationally but defeats the plugin-as-product goal: plugin schema cannot
travel, the core chain grows without bound, and plugins can never be cleanly
removed. Rejected.
2. **Rewrite history so plugin tables move out of the core chain into plugin
`0001` CREATE migrations.** Would make each plugin chain self-contained from
empty, but breaks the immutability rule, forces every existing site to
re-run a rewritten chain, and re-introduces the exact double-creation footgun
ADR-004 fixed. Rejected.
3. **Anchor that CREATEs tables with `IF NOT EXISTS` guards.** Lets a from-empty
install build plugin tables from the plugin chain, but then two chains both
claim the same tables and drift can diverge silently. The no-op anchor keeps
a single authoritative creator (the core chain) for cutover-era tables.
Rejected.
## References
- ADR-003 (plugin distribution; external plugins already ship chains)
- ADR-004 (deployment topology; this ADR supersedes its migration-strategy note)
- `shopdb/plugins/alembic_template.py` (`PLUGIN_TABLE_OWNERS`, shared env runner)
- `shopdb/plugins/migrations.py`, `shopdb/plugins/cli.py` (`upgrade-all`)
- `plugins/<name>/migrations/` (per-plugin chains and `0001` anchors)
- `tests/test_plugin_migrations.py` (ownership + chain + idempotency guards)
- `docs/DEPLOY.md`, `docs/UPGRADE.md`, `docs/PLUGINS.md` (deploy sequence)

156
docs/adr/ADR-009-frontend-plugin-gating.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why the SPA gates plugin routes, so a disabled plugin URL redirects instead of breaking.
loadwith: none
-->
# ADR-009: Frontend plugin gating
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
The backend plugin system (ADR-002, ADR-003) lets an operator disable a
plugin. Disabling it unregisters the plugin's API blueprint, drops its
rows from global search (see `test_search_disabled`), and removes its
entry from `get_navigation_items()`, so the disabled feature's sidebar
link disappears.
The frontend told a different story. Every plugin's Vue routes and views
ship inside the core bundle. `frontend/src/router/index.js` auto-discovers
them with `import.meta.glob('./routes/*.js')`, so a route like `/usb` or
`/printers/3` is registered regardless of whether the owning backend
plugin is enabled. A user who typed the URL, followed a stale bookmark,
or clicked a cross-link reached a page whose API calls all 404, landing
on a broken shell instead of a clean redirect. The navigation was already
data-driven; direct-URL reachability was the gap.
There is no frontend plugin system yet. The product is packaged as one
core Flask app plus backend-only plugins; the Vue app is monolithic and
build-time static. Any gating has to live in core frontend code because
that is the only place the plugin routes exist.
## Decision
### Step 1 (this ADR, implemented): route-level gating
Plugin-owned frontend routes are gated against the backend's enabled-plugin
list. This is the whole of what ships now.
1. **Enabled-list endpoint.** A new `GET /api/plugins/enabled`
(`jwt_required(optional=True)`) returns a flat JSON array of enabled
plugin name strings and nothing else. It is a cheap registry read
(`registry.get_enabled_plugins()`), no database access. Exposing it to
anonymous callers is safe because `GET /api/dashboard/navigation`
already leaks the same enabled/disabled signal, and unauthenticated
kiosk routes (`/tv`) need the answer too. It carries no metadata, so it
reveals strictly less than the equally-anonymous but metadata-carrying
`GET /api/plugins` (also `jwt_required(optional=True)`; only the
`PUT /api/plugins/<name>` toggle is admin-gated).
2. **Route tagging.** Every plugin-owned route carries `meta.plugin =
'<pluginname>'`. This covers the per-plugin route modules
(`routes/computers.js`, `routes/equipment.js`, ...) and the
plugin-owned routes that physically live in core files: the
PC-relationships and toner reports, the slide manager and `/tv`
dashboard (slides), the printer-QR and USB-label print pages, and the
employee-detail page. Genuinely core routes (dashboard, search, map,
applications, reference-data settings) stay untagged and are never
gated.
3. **Cached fetch, fail-open.** A composable
(`composables/enabledPlugins.js`) fetches the list exactly once behind
a cached promise. If the fetch fails or returns a non-array, the code
fails **open**: every plugin is treated as enabled. A transient API
error must never brick navigation. The cost is that a disabled
plugin's page is briefly reachable during an outage, which is
acceptable because its API calls would fail anyway and the next
successful fetch closes the gap.
4. **Router guard.** `router.beforeEach` awaits the cached fetch when the
target route has `meta.plugin`. If that plugin is not enabled it
redirects to `/` and raises an info toast. The endpoint is
jwt-optional, so the guard works for both authenticated pages and the
unauthenticated `/tv` kiosk route.
This is intentionally a thin layer. It does not change the plugin
contract surface, so `__contract_version__` does not move: adding a core
HTTP endpoint and tagging core-shipped routes are not plugin-contract
changes. Plugins still ship no frontend code of their own.
### Non-goals for step 1
- No build-time or runtime loading of plugin-authored Vue code.
- No per-permission or per-role route gating (that stays with the
existing `requiresAuth` / `requiresAdmin` meta flags).
- No removal of disabled routes from the route table; they remain
registered and are intercepted by the guard. Keeping them registered
avoids a rebuild when a plugin is toggled and keeps the redirect path
simple.
## Future direction (PROPOSED, not implemented)
Step 1 gates routes that core already owns. The longer-term goal is a
real frontend-plugin contract where a plugin ships its own frontend and
core discovers it, mirroring the backend model. Sketch:
1. **Plugin-owned frontend tree.** Each plugin gains a
`plugins/<name>/frontend/` directory holding its route module, views,
and any plugin-specific components. Core stops carrying `views/usb`,
`views/printers`, and so on.
2. **Build-time discovery.** The Vite build discovers plugin frontends
with a glob over `plugins/*/frontend/routes.js` (analogous to today's
`import.meta.glob('./routes/*.js')`), so a plugin's presence in the
tree is what puts its routes in the bundle. Combined with the step-1
enabled-list gate, a plugin that is absent from the build ships no
code and a plugin that is present-but-disabled is route-gated at
runtime.
3. **Shared component + registration contract.** Plugins register into
named extension points instead of editing core files: an `iconMap`
registration for nav/asset icons, asset-detail panels, map-marker
renderers, and search-result renderers (the "Frontend hook contract"
already listed as deferred in docs/ROADMAP.md). Core exposes a
stable set of shared components (form controls, detail-page shells,
table primitives) as the plugin frontend's only allowed core imports,
the frontend analogue of the `shopdb.api` namespace.
4. **Versioned frontend contract.** The shared-component and
registration surface would be versioned the same way the backend
contract is (ADR-002), so a plugin frontend can declare the core
frontend range it needs.
### Tradeoffs of the future direction
- **Pro:** true plugin self-containment; a site can drop in or remove a
plugin (frontend and backend together) without patching core; smaller
core; clearer ownership.
- **Con:** significant build-system work (per-plugin Vite entry
discovery, code-splitting, dev-server HMR across the plugin tree); a
new versioned frontend contract to maintain and document; a migration
that moves ten plugins' worth of views out of core; risk of a leaky
shared-component surface becoming an accidental contract. The payoff
only matters once external (out-of-tree) plugins with their own
frontends are a real requirement. Until then, step 1's route gating
delivers the user-visible correctness (no reachable dead pages) at a
fraction of the cost.
## Consequences
- Disabling a backend plugin now makes its frontend routes redirect to
the dashboard instead of loading a broken shell. Behavior matches the
already-dynamic navigation.
- One extra lightweight request at app boot (`GET /api/plugins/enabled`),
cached for the session.
- Fail-open means gating is a UX guardrail, not a security control. It is
not a substitute for backend authorization: the API still enforces auth
and the disabled plugin's endpoints are simply unregistered. Never rely
on route gating to protect data.
- The future frontend-plugin contract remains open work; this ADR records
the direction and its cost so a later decision can pick it up.

326
docs/adr/ADR-010-frontend-plugin-hooks.md Normal file
View File

@@ -0,0 +1,326 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: The hook contract a plugin uses to contribute UI without hand-editing core Vue files.
loadwith: none
-->
# ADR-010: Frontend plugin hook contract
- **Status:** ACCEPTED
- **Date:** 2026-07-11
- **Accepted:** 2026-07-11
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
The backend plugin contract is settled (ADR-002 surface, ADR-006 collector,
ADR-008 migrations, ADR-009 route gating). The one undefined piece before 1.0 is
the frontend: how a plugin contributes UI without hand-editing core Vue files.
CONTRACT-STABILITY.md names this the single biggest churn item ("no server-side
hook for asset-detail panels, map markers, or search-result rendering; a plugin
that needs custom UI still hand-edits the Vue frontend").
The `measuringtools` plugin (bundled 2026-07-11, whose construction is narrated
in docs/PLUGIN-GUIDE.md) is the fresh evidence. Integrating it needed exactly
**two hand edits to core frontend files**:
1. `frontend/src/api/index.js` - an api-client block appended after
`warrantyApi` (PLUGIN-GUIDE.md section 9). The scaffolder now emits a
paste-in snippet, but it is still a hand edit to a shared, churn-heavy file.
2. `frontend/src/views/settings/settingsNav.js` - a card entry in the hardcoded
`settingsGroups` catalog (the "Measuring Tools" group, ~line 51), consumed by
`SettingsLayout` (left rail) and `SettingsIndex` (landing overview).
Everything else integrated with **zero** core edits, because a mechanism already
existed for it:
| Capability | Existing mechanism |
|---|---|
| Views / routes | Auto-discovery: `frontend/src/router/index.js` globs `router/routes/*.js` via `import.meta.glob`; a route starting `settings/` auto-nests under the settings shell. |
| Disabled-plugin gating | `meta: { plugin: '<name>' }` per ADR-009; router guard redirects when the backend plugin is off. |
| Sidebar / dashboard / reports | Declarative backend hooks rendered by core: `get_navigation_items`, `get_dashboard_widgets`, `get_reports` on `BasePlugin` (shopdb/plugins/base.py), merged by `shopdb/core/api/dashboard.py` and `shopdb/core/api/reports.py`. |
| Map placement | Data-driven off asset types + resolved positions (ADR-001); a typed, positioned asset appears on the map with its type color and no plugin-side map code. |
Capabilities a plugin **cannot have at all today** - no hook exists, and adding
one would mean forking a core view:
3. **Asset-detail extension panels.** A warranty-coverage section on a PC or
printer detail page, a calibration-status card on a measuring tool. Today
`WarrantyPanel.vue` is composed in by hand-editing each detail view.
4. **Map marker / overlay contributions.** A calibration-due badge on the
shop-floor map (`frontend/src/views/MapView.vue`). The map draws type
colors but has no plugin decoration path.
5. **Search-result rendering / routing for plugin asset types.** Global search
returns assets, but core hardcodes how each type renders and where its detail
link points; a plugin asset type has no way to declare its icon or route.
There is also an unsolved distribution wrinkle from ADR-003: external plugins
symlink into `plugins/<name>/` backend-only, but any frontend file they carry
must be physically copied into `frontend/src/`, because Vite compiles the tree
at build time and cannot reach outside it.
## Options considered
### A. Runtime dynamic component registration
Plugins ship real Vue components that core loads and mounts at runtime into
named extension points (an `iconMap` registration, a panel registry, a marker
renderer registry). This is the richest model and the one ADR-009's "Future
direction" sketched.
- Pro: a plugin can render anything; no core generic-renderer ceiling.
- Con: requires runtime module loading of plugin-authored code (dynamic import
of built chunks), a versioned shared-component surface that becomes an
accidental contract the moment it leaks, and it does nothing about the
build-time-only reach of Vite for external plugins. It is the most code for
the least near-term payoff. ADR-009 already priced this and deferred it.
### B. Declarative data-only hooks rendered by core generic components
Plugins return plain dicts from new `BasePlugin` hooks; a generic core component
renders them. This is the exact precedent already proven three times:
`get_navigation_items`, `get_dashboard_widgets`, and `get_reports` (0.6.0) all
return dicts that a core consumer endpoint merges and a core Vue component
renders. No plugin ships frontend code.
- Pro: additive, minor-bump changes under ADR-002; identical access pattern to
the existing consumers (skip disabled, fail-loud in dev, isolate in prod);
works untouched for external symlink-only plugins, because the data crosses
the wire and core owns the renderer.
- Con: bounded to what a generic renderer can draw. A panel that needs a bespoke
chart or a custom-interaction map overlay does not fit.
### C. Build-time file-convention discovery
Extend the `import.meta.glob` precedent from `./routes/*.js` to a plugin-owned
frontend tree (`plugins/*/frontend/`), so a plugin's real components (full
list/detail/form views, its api-client module) live with the plugin and the
build picks them up by convention. This is ADR-009 "Future direction" steps 1-2.
- Pro: true self-containment for genuine components; the natural home for the
api-client block (friction 1) and full views.
- Con: significant build-system work (per-plugin Vite entry discovery,
code-splitting, dev-server HMR across the tree) and it does not by itself
solve the external-plugin wrinkle - a symlinked out-of-tree `frontend/` is
still outside Vite's compiled root. Only pays off once out-of-tree plugins
with frontends are a real requirement.
### Hybrid
The evidence splits cleanly. The five friction points fall into two buckets:
surfaces where a generic renderer fed by plugin data is sufficient (2, 3, 4, 5),
and surfaces where a real component is genuinely unavoidable (full views, and
the api-client module in friction 1). Option B fits the first bucket exactly and
is cheap and additive. Option C is the right long-term answer for the second but
is expensive and, per ADR-009, gated on external-plugin demand. Option A buys
nothing B does not, at the highest cost. The decision is B now, C deferred, A
not pursued.
## Decision
**DECISION:** adopt a hybrid. Add **data-only declarative hooks** (Option B) for
the four presentation surfaces a generic core renderer can serve, and keep
**file-convention glob discovery** (Option C) as the deferred mechanism for the
residual cases where a real component is unavoidable. Do not pursue runtime
component registration (Option A).
### New hooks (Option B, data-only, additive)
Each is a new optional `BasePlugin` method returning a list of dicts, merged by
a core consumer endpoint using the same pattern as `get_reports` (inject
`plugin` name, skip disabled, re-raise in dev/test, log-and-isolate in prod),
and rendered by a generic core component. Icon values are string keys mapped to
Lucide components core-side, exactly like `get_navigation_items`.
**`get_settings_cards`** - resolves friction point 2 (settingsNav.js hand edit).
```
{
'group': 'Measuring Tools', # rail group title (created if new)
'to': '/settings/measuringtooltypes',
'icon': 'ruler', # string key, mapped core-side
'title': 'Measuring Tool Types',
'description': 'Manage measuring-tool subtypes + map colors',
'position': 22, # order within the group
}
```
Consumer: a new `GET /api/settings/cards` merges enabled plugins' cards into the
core `settingsGroups` catalog; `SettingsLayout` and `SettingsIndex` read the
merged catalog instead of the hardcoded JS array. The catalog's own core groups
stay in `settingsNav.js`; plugin groups are appended.
**`get_asset_panels`** - resolves friction point 3 (asset-detail panels).
```
{
'id': 'calibration',
'title': 'Calibration',
'assettypes': ['measuring_tool'], # detail pages it appears on; ['*'] = all
'endpoint': '/api/measuringtools/{assetid}/calibration-panel',
'render': 'keyvalue', # 'keyvalue' | 'table' | 'badge'
'position': 20,
}
```
Consumer: `GET /api/assets/{assetid}/panels` returns the panels whose
`assettypes` match that asset's type; a generic `AssetPanel` component on the
detail page fetches each `endpoint` and renders it in the declared style. This
covers the warranty-coverage and calibration-status cases. A panel that needs
bespoke UI (a chart) is out of scope for the data-only hook and falls to the
deferred component mechanism below - stated honestly, not hidden.
**`get_map_overlays`** - resolves friction point 4 (map marker/overlay badges).
```
{
'id': 'calibration-due',
'label': 'Calibration due', # legend label
'endpoint': '/api/measuringtools/map-overlay', # -> [{assetid, color, label}]
'style': 'badge', # 'badge' | 'ring'
'legend': True,
}
```
Consumer: `MapView.vue` fetches enabled plugins' overlay endpoints and decorates
the already-placed markers; legend entries append to the existing legend. The
map stays data-driven; plugins add decoration data, not map code.
**`get_asset_presentation`** - resolves friction point 5 (search rendering /
routing) and, as a bonus, removes the `AppLayout.vue` iconMap hand edit called
out in PLUGIN-GUIDE.md section 9.
```
{
'assettype': 'measuring_tool', # AssetType.assettype key the plugin owns
'icon': 'ruler',
'label': 'Measuring Tool',
'route': '/measuringtools/{assetid}', # detail-route pattern
}
```
Consumer: `GET /api/assets/presentation` returns the type-to-presentation map;
global-search result rows and any asset cross-link use it to pick the icon and
build the detail link, so core never hardcodes a plugin's route or icon.
### Friction map
| Friction point | Mechanism | Hook / change |
|---|---|---|
| 1. api-client block in api/index.js | C (deferred) | plugin-owned `frontend/` tree, glob-discovered; scaffolder snippet is the near-term mitigation |
| 2. settingsNav.js card entry | B (now) | `get_settings_cards` |
| 3. asset-detail panels | B (now) | `get_asset_panels` (bespoke panels -> C, deferred) |
| 4. map markers / overlays | B (now) | `get_map_overlays` (bespoke overlays -> C, deferred) |
| 5. search-result rendering / routing | B (now) | `get_asset_presentation` |
| views (full list/detail/form) | C (deferred) | already glob-discovered under core `routes/*.js`; long-term move to `plugins/*/frontend/` per ADR-009 |
### Deferred: file-convention frontend tree (Option C)
Extending `import.meta.glob` from `./routes/*.js` to `plugins/*/frontend/` (so
views and the api-client module live with the plugin) is ADR-009 "Future
direction" steps 1-2. It is deferred for the same reasons ADR-009 gave: it is
build-system-heavy and the payoff only lands once out-of-tree plugins with their
own frontends are a real requirement. Until then, friction 1 stays mitigated by
the scaffolder snippet, and full views keep shipping in the core bundle and
gated by ADR-009 `meta.plugin`.
This is also the honest limit on the external-plugin wrinkle. The four data-only
hooks need **zero** frontend files from a plugin, so an external symlink-only
(backend) plugin gets settings cards, detail panels, map overlays, and search
presentation with no copy-into-`frontend/src/` step at all. The wrinkle survives
only for the residual component-backed cases, which is exactly the deferred
Option C work; pushing external-repo frontend distribution to that later ADR
matches ADR-003's posture of deferring out-of-tree packaging until two sites run
their own plugins.
### Contract-version impact (ADR-002)
Each of the four hooks is a new optional `BasePlugin` method - an additive
change, so a **minor** bump per ADR-002, the same classification `get_reports`
took at 0.6.0. Landing all four together is a single minor bump (proposed
**0.7.0**); landing them incrementally is one minor bump each. Adding the core
consumer endpoints and the generic renderer components is core-internal and does
not itself move `__contract_version__`. The deferred Option C introduces a
separate, versioned **frontend** contract (ADR-009 step 4), tracked apart from
the backend `__contract_version__`; it is not part of this proposal's bumps.
## Consequences
### Positive
- Closes the CONTRACT-STABILITY.md "single biggest gap" for the four surfaces a
generic renderer can serve, using the already-proven declarative pattern - low
risk, low cost, additive-only.
- External symlink-only plugins get four presentation surfaces with no
copy-into-core step, shrinking (not yet eliminating) the ADR-003 wrinkle.
- Removes three of the two-plus hand edits the exemplar needed (settingsNav
card, plus the AppLayout iconMap edit from section 9), moving them to data.
- Docs-drift guard forces documentation: `tests/test_docs_contract.py` fails if
any new public `BasePlugin` hook is missing from `docs/PLUGIN-HOOKS.md`, so the
hooks cannot ship undocumented.
### Negative
- Four more hooks and four more core consumer endpoints to maintain, each with
the skip-disabled / fail-loud-in-dev / isolate-in-prod discipline.
- The generic renderers set a ceiling: bespoke panels and overlays still have no
home until Option C lands, so the contract is honest-but-partial, not total.
- One extra request per surface on the pages that use it (a detail page fetches
its panels; the map fetches overlays), on top of ADR-009's enabled-list fetch.
### Neutral
- Bundled plugins keep their current in-core views; migrating them to the hooks
is opt-in and incremental (see Adoption plan), not a flag-day rewrite.
- `meta.plugin` route gating (ADR-009) is unchanged and still gates the views;
these hooks add presentation, not routing.
- Contract tests (`tests/test_plugin_contract.py`) already assert every public
hook is exercised; the new hooks slot into that harness.
## Adoption plan
Prove the contract on bundled plugins before declaring it settled for sister
sites.
1. **`get_asset_panels` first, via `warranty`.** Warranty is asset-general and
already composes `WarrantyPanel.vue` onto multiple detail pages by hand, with
a clean per-asset endpoint behind it. Migrating it to a declarative
`get_asset_panels` entry rendered by the generic `AssetPanel` is the
lowest-risk proof and immediately removes hand edits from every detail view
that shows warranty. This is the recommended first migration.
2. **`get_settings_cards`, `get_map_overlays`, `get_asset_presentation` via
`measuringtools`.** The guide exemplar already needs a settings card, a
calibration-due map badge, and search routing for its `measuring_tool` type,
so it exercises all three at once and its PLUGIN-GUIDE.md walkthrough becomes
the reference for the new hooks.
3. Only after both plugins run on the hooks in a real build: bump
`__contract_version__` to 0.7.0, document the hooks in
`docs/PLUGIN-HOOKS.md`, and mark this ADR ACCEPTED.
## Open questions
- Should `get_asset_panels` `render` styles stay a small closed set
(`keyvalue` / `table` / `badge`), or grow? A closed set keeps the renderer
generic; growth pressure is the signal that a case actually needs Option C.
- Should the four consumer endpoints collapse into one bundled
`GET /api/plugins/frontend-contributions` call to save round-trips, or stay
separate per surface for cache locality? Defer until the per-surface request
cost is measured.
- When Option C lands, does the api-client module move under
`plugins/*/frontend/` or get replaced entirely by a generated client from the
backend blueprint? Out of scope here; belongs to the ADR-009 frontend-contract
follow-up.
## References
- ADR-001 (asset model as the map/search data source)
- ADR-002 (bump classification for the new hooks)
- ADR-003 (external-plugin distribution posture)
- ADR-009 (route gating; "Future direction" is the deferred Option C)
- docs/PLUGIN-GUIDE.md sections 9-10 (the measuringtools frontend hand edits)
- docs/CONTRACT-STABILITY.md (the expected-churn line this ADR answers)
- shopdb/plugins/base.py (existing declarative hooks this pattern extends)
- shopdb/core/api/dashboard.py, shopdb/core/api/reports.py (consumer precedent)

75
docs/adr/ADR-011-machines-rename.md Normal file
View File

@@ -0,0 +1,75 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why the equipment domain was renamed to machines and machinetypes became modeltypes.
loadwith: none
-->
# ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes
- **Status:** ACCEPTED
- **Date:** 2026-07-11
- **Deciders:** ShopDB maintainers
- **Relates to:** ADR-005 (equipment vs measuringtools scope), ADR-008 (plugin migration ownership)
## Context
ADR-005 split the old "equipment" concept: gage-lab instruments moved to the
measuringtools plugin, leaving the equipment plugin holding exactly one thing -
shop-floor machines (lathes, mills, CMMs, grinders, furnaces). The UI, the
routes (`/machines`), the frontend view folder (`views/machines/`), and even
the data vocabulary (asset numbers headed "MACHINE #") already said machines.
Only the code identity still said equipment: the plugin name, the API prefix,
the `equipment`/`equipmenttypes` tables, and the `assettype='equipment'` data
value. Every future contributor would pay a small permanent translation tax.
A display-label-only swap was shipped first (2026-07-11) and immediately showed
the dissonance: UI Machines over code equipment forever.
Separately, the legacy `machinetypes` table survived the Machine-model
retirement for one real reason: it types the vendor **models catalog**
(`models.machinetypeid`, 95 of 127 models) with fine-grained, cross-category
types (Lathe, Switch, Laser Printer). Its name was a misnomer - it never typed
machines specifically - and it squatted on the name the machines plugin's
subtype table wants.
The economics favor doing this now and never again: the plugin contract is
pre-1.0 (breaking changes are cheap per ADR-002), no sister site has adopted
yet, there is no production instance, and the test suite plus CI give the
strongest safety net the project has had.
## Decision
Rename the domain in one coordinated pass:
| Current | New | Rationale |
|---|---|---|
| `plugins/equipment/`, `/api/equipment`, `equipment.*` perms | `plugins/machines/`, `/api/machines`, `machines.*` | plugin identity matches the domain |
| `equipment` table, `equipmentid` | `machines`, `machineid` | main entity |
| `equipmenttypes`, `equipmenttypeid`, `equipmenttype` | `machinetypes`, `machinetypeid`, `machinetype` | subtypes take the freed name |
| legacy `machinetypes`, `models.machinetypeid` | `modeltypes`, `models.modeltypeid` | role-accurate: it types the models catalog, losslessly (no collapse to the 4 asset types) |
| `assettypes.assettype = 'equipment'` (data) | `'machine'` | the string compare sites flip with it |
Migration placement follows ADR-008: the modeltypes rename and the data flips
land in the core chain (core tables, core data); the plugin table renames land
in the machines plugin's own chain, idempotently guarded so fresh installs
(whose core chain still creates the old names at `7c04`) rename on first
`flask plugin upgrade-all`. The `alembic_version_equipment` version table is
renamed to `alembic_version_machines` by the core migration so the plugin chain
resumes seamlessly on existing installs. The plugin registry auto-migrates an
`equipment` key to `machines` on load.
Historical migrations and accepted ADRs are not rewritten; old ids and ADR-005
wording remain as history.
## Consequences
- Upgrading instances run `flask db upgrade` then `flask plugin upgrade-all`,
same as every deploy; both migrations are idempotent.
- `/api/equipment` and `equipmentApi` cease to exist - a breaking change made
while there is nobody to break. After contract 1.0 this class of rename
would be a major-version event; this is the last cheap moment.
- The "Machine Types" settings card becomes "Model Types" and finally says
what it does; the machines plugin's subtype card owns the machine-types name.
- Contributors stop translating between UI vocabulary and code vocabulary.

131
docs/adr/ADR-012-geenforce-manifest-ownership.md Normal file
View File

@@ -0,0 +1,131 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Why ShopDB owns GE-Enforce manifests as data while the enforcement engine stays external.
loadwith: none
-->
# ADR-012: GE-Enforce manifest ownership in shopdb
- **Status:** ACCEPTED
- **Date:** 2026-07-13
- **Deciders:** ShopDB maintainers
- **Relates to:** ADR-002 (plugin contract versioning), ADR-004 (per-site
deployment), ADR-006 (collector contract), ADR-008 (per-plugin Alembic chains)
## Context
GE-Enforce is a desired-state enforcement system for shopfloor PCs: a PowerShell
engine (`Install-FromManifest.ps1`) reads per-PC-type `manifest.json` files off
an SMB share every logon and installs / self-heals what they declare. Authoring
those manifests today means hand-editing JSON on a file share, and there is no
central view of what each PC actually did.
We want shopdb to own the manifests as data (author, version, publish, roll
back) and to observe fleet compliance, while NOT taking on the GE-Enforce engine
itself (which is the GE-Enforce framework's, maintained separately) and NOT
dictating any site's imaging path (per ADR-004, each site is single-tenant with
its own provisioning - PXE at the reference site, OOBE provisioning packages at
others).
The manifests are an enforcement PROGRAM, not an application inventory: entry
`Type` is not an app/config discriminator and entry `Name` is a manifest label,
not a Windows ARP DisplayName. Any design that treats them as an app catalog is
wrong.
## Decision
Build a bundled `geenforce` plugin that owns the manifest as shopdb data, with a
client kit and a deployment bootstrap. Specifically:
1. **Data model.** One wide `manifestentries` table with an `entrytype`
discriminator and nullable per-type columns (not SQLAlchemy STI, not a JSON
blob - the fleet is ~64 entries, so sparse columns are free and stay
queryable). Scopes are `manifestscopes`, unique on `(scopename, phase)`;
runtime is per-pctype scopes, preinstall is one flat scope. Multi-value gates
(PCTypes / hostnames / machine numbers) and the nested InUseCheck are child
tables. `sortorder` is the execution-order contract. RegValue is stored as a
raw JSON literal so DWord-vs-string typing survives. Per-plugin Alembic chain
(ADR-008).
2. **Published snapshots.** Editing touches a DRAFT only. Publish freezes the
rendered JSON document into an immutable `manifestpublishedversions` row; the
client is ALWAYS served the current published snapshot, never the draft;
rollback flips `iscurrent` to an older version. Freezing the document (not
row-mirroring) makes immutability structural.
3. **Behavioral-parity gate, not byte-identity.** A DB-free harness
(`parity.py`) imports each real manifest and renders it back, then proves
BEHAVIORAL equivalence (same ordered entries with identical detection /
targeting, and the same entries fire across machine-profile fixtures) - never
byte equality, which re-serialization would never satisfy. This gates any
build that touches the model.
4. **Filter mirror; engine is the single source of truth.** `filters.py`
mirrors the engine's four gate functions and alias graph for the "what would
this PC get" simulator and parity. The engine lib stays authoritative;
shopdb mirrors it (never the reverse). PCTypesStrict is honored only for the
preinstall phase, matching the runners.
5. **Payload integrity is separate from detection.** For `http`/`inline`
payloads a dedicated `payloadsha256` is verified before running - independent
of `DetectionMethod` (DetectionValue is a hash only for `Hash` detection).
`smb` payloads keep the share ACL as their trust boundary. Large binaries
stay on SMB; small config/scripts may move to http/inline later.
6. **Observed-state reporting.** Each PC POSTs its enforcement result;
`manifestenforcementreports` (+ results) records the applied version
(received-latest) and per-entry self-heal / failure. Status derives from
explicit self-heal flags only, never the raw installed count (Always/no-
detection scripts install every cycle without being drift corrections).
7. **Service-token auth.** Client endpoints authorize via managed service
tokens scoped `geenforce.fetch` / `geenforce.report`, through a new
`service_token_authorized(scope)` on the `shopdb.api` contract surface
(contract 0.11.0). Admin CRUD uses `geenforce.manage` / `geenforce.publish`.
8. **Client + deployment, engine referenced not vendored.** shopdb ships the
fetch/report kit (`plugins/geenforce/client/`) and a site-neutral bootstrap
(`Install-GEEnforce.ps1`) that provisions a PC's identity
(`C:\Enrollment\pc-type.txt` etc. - what determines the PC type; there is no
auto-detection, the provisioner supplies it), the shopdb registry config, and
the scheduled task. The GE-Enforce ENGINE is referenced (`-EngineSource`),
not carried by shopdb. Deployment is provisioning-path independent (PXE step,
OOBE ppkg, Intune, manual); the runtime task is fail-safe.
9. **Milestone 1 = export to share; staged cutover.** Until a site cuts its
client over to shopdb-sourced manifests, the plugin publishes and EXPORTS the
manifest to the share (with a `_meta/history` backup, atomic write); the
unchanged engine picks it up. Cutover is staged: shadow mode (fetch from
shopdb AND read the share, log diffs, install from share) then read cutover.
10. **No application auto-seeding.** The core Applications catalog already
tracks these apps (from the classic-shopdb migration) with version
histories; auto-creating Applications from manifest labels produced
duplicates and misclassified config drops. Application linkage, if wanted, is
a curated manifest-entry -> existing-Application link, not label scraping.
## Consequences
- **Positive.** Manifests become validated, versioned, publishable data with
one-click rollback and a fleet-compliance view; desired-state and observed-
state live in one system. The parity gate + published snapshots + separate
payload hash make a fleet-wide-SYSTEM system safe to author. The plugin is
provisioning-agnostic, so any GE Aerospace site can adopt it regardless of
imaging path. Validated end to end: parity green against the real manifests,
and the client kit + installer proven on a Windows VM (PS 5.1) and Linux
pwsh 7.
- **Boundaries / risks.** The engine remains the GE-Enforce framework's, so
shopdb's parity mirror must be kept in sync with the lib (guarded by the parity
fixtures; the plugin pins lib >= 2.6 for `_CmmVersion`). Provisioning writes
the PC identity - shopdb cannot set a PC's type at imaging (a PC is unknown
until it enrolls and reports). Manifest-label vs ARP-name mismatch means the
catalog link, when built, needs a curated alias layer.
- **Deferred.** Desired-vs-observed per-entry compliance (needs a collector
installedVersions field); curated manifest-entry -> Application linking; the
live client cutover (a site operational decision); inline payload upload.
See `docs/proposals/ge-enforce-plugin.md` (design + cutover), `docs/GE-ENFORCE.md`
(concepts + imaging timeline), `docs/GE-ENFORCE-CLIENT.md` (fetch/report
contract), and `docs/GE-ENFORCE-DEPLOY.md` (agent deployment).

421
docs/adr/ADR-013-plugin-catalog-and-lean-builds.md Normal file
View File

@@ -0,0 +1,421 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How a site gets only the plugins it chose, and why curation plus signing is the security model.
loadwith: none
-->
# ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds
- Status: ACCEPTED
- Date: 2026-07-18
- Deciders: ShopDB maintainers
- Relates to: ADR-002 (contract versioning), ADR-003 (plugin distribution), ADR-004 (per-site instances), ADR-008 (per-plugin migrations), ADR-009 (frontend plugin gating), ADR-010 (frontend hook contract)
## Context
Every site today ships identical code. The backend image contains all 13 plugin
directories (Dockerfile COPY at line 55; the header comment listing "eleven core
plugins" is stale), and the SPA compiles every plugin's routes and views via a
static glob (frontend/src/router/index.js:11) plus hardcoded imports. A site's
"chosen set" exists only as runtime enable flags in instance/plugins.json.
Disabled is not absent: a site that never wants printedparts/usb/network still
ships, and can execute, that code.
Distribution per ADR-003 is "drop a directory into plugins/ by hand". There is
no artifact format, no signing, no catalog, no validate gate, and the loader
trusts whatever it finds on disk (loader.py:61-73 discovers any folder with a
plugin.py; loader.py:185-224 loads it; migrations.py:29-62 runs its DDL with
full DB privileges). Plugins run in-process with the full shopdb.api surface
including db, so the only tenable security model on air-gapped GE networks is
curation plus cryptographic provenance, enforced everywhere code can execute,
not sandboxing.
Known defects this ADR also resolves:
- upgrade_all_plugins checks hasattr(registry, 'list_installed') which never
exists (registry has only get_all/get_enabled_plugins), so it always falls
back to migrating every folder on disk, adopted or not (__init__.py:94).
- PILOT-DEPLOY.md enables plugins that were never installed; enable refuses.
There is no declarative "apply this chosen set" operation.
- Reverse-dependency checks on uninstall/disable read only LOADED plugin
instances, so an installed-but-unloaded dependent is invisible.
- Dependency install/enable is check-only; nothing computes a closure, and
install-a-dependent can fail at its own load step because the dependency was
installed disabled (default_enabled=false on employees).
- The dependency sort has no cycle detection.
- Soft couplings (geenforce -> computers, notifications -> employees) are
invisible to the manifest graph.
Frontend reality check (this drove the design below): plugin UI is NOT one
folder per plugin. Routes live in routes/<plugin>.js (slides has none), inside
the shared core.js (computers report, printers toner report, employees detail,
slides settings), and as six hardcoded top-level imports in index.js itself
(/parts-kiosk, /tv, /print/printer-qr x2, /print/usb-labels,
/print/printedparts-labels). View dirs mismatch plugin names (computers ->
views/pcs). Plugin settings cards sit in shared views/settings/
(DellWarrantySettings, ZabbixSettings, SlideManager, EmployeeDirectory,
MeasuringToolTypesList, PrintedPartsSettings), plugin print views in shared
views/print/, and some views span plugins (AssetLabel.vue serves five asset
types; PCDetail imports WarrantyPanel). Any lean-frontend design that only
moves routes/*.js and views/<plugin>/ fails the build the moment a plugin is
pruned. This ADR scopes that work honestly instead of calling it mechanical.
## Decision
### 1. Tiering: mandatory core is the core package; all plugins are catalog-optional
- The mandatory core is the non-plugin shopdb/core/ package (auth, users,
assets, locations, vendors, models, settings, audit, dashboard, search,
reports, plugin management). It already survives every plugin being absent
via hasattr/lazy-import guards. No plugin is promoted into it.
- New optional manifest field `tier: "core" | "optional"`, default "optional".
All 13 existing manifests are unchanged and unchanged in meaning. The
lifecycle gains a guard: uninstall_plugin and disable_plugin refuse a
tier:core plugin (alongside the reverse-dependency checks at
__init__.py:269-278 and :351-360). No plugin ships tier:core initially; the
field and guard exist so a future curation decision is a manifest edit, not a
framework change.
- Per-site mandates live in the site profile (section 5): a `locked` list the
profile applier refuses to remove. This preserves ADR-004 site autonomy: a
wing site can mandate usb without the framework mandating it fleet-wide.
- New manifest field `optional_dependencies: []` (names only, loader-ignored).
Declared for the verified soft couplings: geenforce lists computers
(service.py:32-43 loses app-detection gates without it), notifications lists
employees (routes.py:151-171 loses name/photo enrichment). Catalog listing
and adopt WARN on unmet optional deps; nothing blocks.
- Hard `dependencies` gains optional PEP440 ranges ("employees>=1.1").
validate/adopt honor ranges; the runtime loader keeps name-only semantics
(specifier stripped) so no loader behavior changes. The single existing hard
edge printedparts -> employees stays as-is; whether it can relax to optional
(badges.py has an external HR fallback) is a follow-up product question, not
blocked on this ADR.
- Dependency plumbing fixes: _sort_by_dependencies gains cycle detection
(raise PluginDependencyError on a back edge); reverse-dependency checks read
manifests of ALL installed plugins from disk, not loaded instances.
### 2. Packaging: signed, versioned artifacts
Artifact: `<name>-<version>.shopdbplugin` (a zip of the plugin directory:
manifest.json, plugin.py, api/, models/, migrations/, and frontend/ once
section 6 lands) plus two members generated at pack time:
- `PROVENANCE.json`: plugin name, version, publisher id, build timestamp, and
a sorted map of every packaged file path to its SHA-256. PROVENANCE.json is
not listed in its own map, so there is no circular-hash problem and no zip
canonicalization needed; determinism comes from sorted per-file hashes.
- `PROVENANCE.sig`: detached ed25519 signature over the exact PROVENANCE.json
bytes.
New CLI:
- `flask plugin pack <name> --key <path>` (producer side): runs validate on the
directory, then emits the artifact.
- `flask plugin validate <dir|artifact>` (the missing pre-publish gate),
fail-closed pipeline: signature (artifact mode) -> per-file hashes -> manifest
against a new docs/plugin-manifest.schema.json -> name == directory ->
core_version parses as a specifier and admits the target contract version ->
static import-surface scan reusing tests/test_plugin_contract.py logic ->
alembic versions parse. The schema types the known fields (name, version,
description, dependencies, optional_dependencies, tier, core_version,
api_prefix, display_name, default_enabled, provides, settings) and PERMITS
additional properties, so all 13 existing manifests pass unmodified.
The import-surface scan is documented as a lint, not a security control; it is
trivially bypassed by dynamic import. The security control is human review
before signing (section 4).
### 3. The shelf: a read-only folder, transport-agnostic by design
- One config knob: `PLUGIN_SHELF_DIR`. The app only ever reads this folder. It
never speaks SharePoint, OneDrive, or any network protocol.
- Transport is explicitly out of scope and explicitly untrusted. On networks
that can reach corporate M365, a SharePoint document library sync populates
the folder. On strictly air-gapped floors where no sync agent can run, the
folder is populated by robocopy/USB. Both are equally supported and equally
untrusted, because every decision-bearing byte is signed: swapping transport
changes nothing about the trust model.
- Layout: `<shelf>/<name>/<name>-<version>.shopdbplugin` plus
`shelf-index.json` and `shelf-index.sig`.
- The index is SIGNED with the same publisher key and carries a monotonically
increasing `serial` plus a `revoked` list of name-version pairs. Each site
records the last-seen serial in instance state and refuses an index with a
lower serial (anti-rollback of the catalog itself). The index also carries
per-entry version/tier/core_version so `flask plugin shelf-list` can display
compatibility without unpacking, but the index is a BROWSE layer only:
adopt reads dependencies, tier, and core_version from the signed manifest
inside the verified artifact, never from the index.
- Trusted keys: `PLUGIN_TRUSTED_KEYS` is a list of pinned public keys delivered
out-of-band in the site's deployed config/image. Keys are NEVER read from the
shelf; a folder that can be written by an attacker must not also carry the
keys that authenticate it. Multiple pinned keys allow overlap rotation.
Revocation of an artifact rides the signed index `revoked` list; a
`flask plugin audit` command warns when an installed version appears there.
- Partial-sync robustness: adopt copies the artifact to a temp location,
verifies signature and every file hash there, then unpacks to
plugins/.staging/<name> and renames into place atomically. OneDrive
placeholder stubs, zero-byte files, or an index referencing not-yet-synced
artifacts all fail closed with a clear "artifact not fully synced/verified"
error.
- `flask plugin adopt <name>[==version]`: resolve version from the shelf,
verify, compute the hard-dependency closure from signed manifests, then for
each closure member in topological order: unpack, INSTALL, and ENABLE (not
install-only; the load gate at loader.py:201-206 checks is_enabled, so an
install-only closure with default_enabled=false deps would fail its own
load). Migrations run via the unchanged per-plugin chain (ADR-008). Refuses
to adopt a version lower than the installed one unless
`--force-downgrade` is given interactively. Prints the restart notice.
- Adopt/install/uninstall remain CLI-only. The admin HTTP surface stays a
read-only catalog view plus the existing enable/disable toggle; because
Flask cannot register blueprints after the first request, any adopt or
enable takes full effect only on restart, and the UI says so. There is no
"install button" that pretends otherwise.
### 4. Trust model: verify at adopt AND at every load and migrate
Signing that gates only adoption is bypassable through every other write path
into plugins/ (git clone, symlink, USB drop) and defeated by post-adoption
tampering. Therefore verification is enforced where code executes:
- Adoption leaves PROVENANCE.json and PROVENANCE.sig inside plugins/<name>/
and records publisher + artifact hash in the registry entry.
- load_plugin verifies the signature against PLUGIN_TRUSTED_KEYS and re-hashes
the plugin tree against the provenance file map BEFORE importing plugin.py
(new step ahead of loader.py:185). Missing or invalid provenance is a
fail-closed refusal in production.
- run_plugin_migrations performs the same verification before executing any
revision, so a routine `flask plugin upgrade-all` can never run DDL from an
unverified folder.
- upgrade_all_plugins iterates registry.get_all() (fixing the phantom
list_installed fallback at __init__.py:94), so unadopted on-disk folders are
never migrated as a side effect of deploys.
- Development and the ADR-003 external-repo/symlink workflow (including
scripts/test-external-plugin.sh) are preserved via `PLUGIN_DEV_TRUST_DIRS`,
honored ONLY when DEBUG or TESTING is set. Production ignores it.
- Cost: hashing 13 small plugin trees at boot is milliseconds; accepted.
What signing does NOT claim: a valid signature proves the artifact is exactly
what a curator reviewed and signed, nothing more. Plugins remain in-process
Python with full DB access. The actual safety control is the human review
before signing; the signature makes that review's verdict tamper-evident all
the way to execution.
### 5. Declarative site profiles and lean backend builds
- `site-profile.json` per site (kept in the site's deploy config): site name,
list of chosen plugins, optional `locked` list. `flask plugin apply-profile
<file>` resolves the closure, installs AND enables in dependency order, runs
migrations, reports which changes need a restart. This replaces the
imperative CLI sequences in DEPLOY.md/PILOT-DEPLOY.md and fixes the
enable-without-install bug.
- Lean backend image: `scripts/build-site.sh` reads the profile and stages
only core + chosen plugin directories into the Docker build context
(correcting the Dockerfile COPY and its stale header comment). Discovery
needs no change; it already scans whatever exists.
- Prerequisite the naive version misses: core hardcodes plugin imports.
shopdb/core/api/search.py (~15 sites), reports.py, assets.py, collector.py,
applications.py, auditlogs.py, and shopdb/cli/__init__.py import
plugins.<name>.* lazily. Some already guard ImportError; ALL must, with
graceful degradation, before any site prunes a folder. This is audited and
enforced by a new CI job that deletes one plugin directory and runs the full
test suite (repeated per plugin). Longer term these aggregators should move
to registry-driven contract hooks (get_search_providers/get_report_sources)
so a new catalog plugin can join search/reports without core edits; that is
scoped as follow-up work, not a blocker for lean builds.
- Schema-lean is DEFERRED to its own ADR. The core baseline 68b3947ae14f
unconditionally creates the 10 pre-cutover plugins' tables, and lifting them
into plugin baselines collides with cross-plugin foreign keys (the
computers-owned installedapps table FKs machines.machineid while computers
declares no dependency on machines). Reversing the cutover would either
introduce undeclared hard deps or drop FKs; neither is decided here. A lean
site therefore carries a handful of empty pre-cutover tables. Accepted.
### 6. Frontend delivery: Path C for rich UIs, Path A for simple ones, Path B rejected
Path B (runtime-loaded JS / module federation) is REJECTED: it moves executable
UI delivery from a signed, statically auditable build artifact to runtime
fetching, which is exactly the wrong direction for an air-gapped,
review-then-sign posture, for zero benefit given restarts are already required.
Path A (declarative JSON UI over generic renderers) is COMMITTED and scheduled
EARLY: the three unwired ADR-010 endpoints (pluginui.py asset-panels:62,
map-overlays:88, asset-presentation:100) get generic core renderers, joining
the already-consumed settings-cards. After this, a simple plugin ships JSON-only
UI with zero frontend build involvement. Sequencing this before the relocation
gives every plugin an escape hatch during the migration instead of after it.
Path C (self-contained plugin frontend) is the primary mechanism, scoped
against the real code, not the idealized layout:
- Canonical home: plugins/<name>/frontend/ containing routes.js (the plugin's
complete route array, INCLUDING routes currently embedded in index.js and
core.js), views/, and settings views.
- A pre-Vite staging step (scripts/stage-frontend.mjs, run by build-site.sh
and the dev script) copies the CHOSEN plugins' frontend/ into
frontend/src/.plugins-staged/<name>/ (gitignored) and generates two files
inside the Vite root: routes.gen.js (aggregated plugin routes) and
meta.gen.js (plugin-supplied icon names, title spellings, settings-standalone
flags, replacing the hardcoded iconMap/TITLE_SPELLINGS/SETTINGS_STANDALONE in
AppLayout.vue, settingsCatalog.js, and index.js). This exists because
import.meta.glob requires a static literal inside the project root and
cannot select a per-site subset by itself.
- ONE-TIME core-router surgery, done first and called what it is: the six
hardcoded plugin-view imports in index.js (PartsKiosk, TVDashboard,
PrinterQRBatch/Single, USBLabelBatch, PrintedPartsLabels) and the plugin
routes embedded in core.js move into their owning plugins' routes.js. Without
this, pruning slides/printers/usb/printedparts fails the Vite build on
unresolvable imports; no amount of glob work fixes it.
- Per-plugin relocation PRs (13), each REAL WORK, not a file move: carve routes
out of shared files, move views (handling name mismatches like computers ->
views/pcs), move the plugin's settings views out of shared views/settings/,
and rewrite relative ../../ imports of core shared code to the @/ alias
(relative paths break at the staged depth). A lint rule enforces alias-only
core imports in plugin frontend code from then on.
- Shared plugin-aware code STAYS CORE and ships to every site: AssetLabel.vue
(spans five asset types), views/print helpers (assetLabel.js, qrLogo.js),
MachineBadge.vue, and cross-plugin panels like WarrantyPanel used by
PCDetail. These already null-guard or gate via isPluginEnabled and must keep
degrading when a peer plugin is absent; over time they migrate to ADR-010
asset-panels so the data becomes plugin-supplied. Lean v1 therefore prunes
plugin-EXCLUSIVE code; a small plugin-aware core remainder is accepted and
shrinks as Path A absorbs it.
- Dual-location transition: the staging step unions legacy locations
(routes/*.js glob, views/<plugin>/) with plugins/<name>/frontend/ until each
plugin has moved. The SPA builds green at every commit; each plugin's move is
independently revertable until the legacy glob is removed at the end.
- Nav and settings cards are already server-driven (dashboardApi.navigation,
settings-cards); the remaining hardcoded plugin entries in settingsNav.js
(/settings/zabbix, /settings/dellwarranty) move to those plugins'
get_settings_cards so pruning leaves no dead links.
### 7. Effect on the 13 existing plugins
- Backend: ZERO code changes required. tier/optional_dependencies/provenance
are additive; pack zips the directory as-is; all plugins keep passing
tests/test_plugin_contract.py. Bundled plugins in a site's image get
provenance stamped at build time by pack, so verify-at-load applies to them
identically.
- Frontend: one relocation PR each, of the honest scope above. Until a
plugin's PR lands it keeps working from its legacy location.
- Operationally nothing changes for a site that does nothing: default builds
remain all-plugins, apply-profile is opt-in, and enable/disable semantics
(including the restart requirement) are unchanged.
## Consequences
### Positive
- A real catalog: sites declare their set in site-profile.json and apply it in
one idempotent command; the chosen set drives backend image, SPA bundle, and
runtime state from one source of truth.
- Curated marketplace with end-to-end provenance: review -> sign -> any
transport -> verify at adopt, at load, and at migrate. Transport (SharePoint
sync or sneakernet) is untrusted and interchangeable, which is exactly right
for air-gapped sites.
- Lean per-site builds: unchosen plugins exist in neither the image nor the
bundle, shrinking attack surface and download size.
- Fixes shipped along the way: upgrade-all migrating unadopted folders,
PILOT-DEPLOY install/enable ordering, reverse-dep checks blind to unloaded
plugins, missing cycle detection, missing dependency closure, hardcoded
frontend plugin metadata.
- Path A completion makes simple plugins UI-capable with no build glue, which
is the cheapest possible marketplace onboarding.
### Negative
- Key management is a per-site operational burden: pinned keys delivered
out-of-band, rotation is a config change everywhere. Accepted as the price of
not trusting the distribution folder.
- The frontend re-org is the long pole: one core-router surgery plus 13
non-trivial PRs. It is sequenced to be always-green and per-plugin
revertable, but it is weeks of work, not a rename.
- Schema is not lean: pre-cutover plugin tables still appear at every site
until the deferred baseline re-org ADR.
- Restarts remain required after adopt/enable (Flask blueprint constraint);
the marketplace UX is honest about it rather than working around it.
- Boot adds a signature + tree-hash check per enabled plugin (milliseconds,
but nonzero).
### Risks
- Key compromise or curation failure: a signature proves provenance, not
safety; a compromised pinned key or a rubber-stamp review signs malware that
every gate will happily pass. Mitigations: multi-key pinning with overlap
rotation, signed revocation list with monotonic index serial, and keeping the
signing key offline with the curator. The static import scan is a lint and
must never be presented as a boundary.
- Rollback/downgrade: mitigated three ways: index serial monotonicity, adopt
refusing version downgrades without interactive --force-downgrade, and the
signed revoked list. Residual risk: a site that never syncs a newer index
cannot learn of revocations; `flask plugin audit` at deploy time narrows the
window.
- Version skew across ADR-004 sites: one shelf serves sites at different
contract versions. Adopt checks core_version from the signed manifest against
the site's own __contract_version__ (authoritative); shelf-list shows an
advisory compatibility column from the index. Incompatible artifacts are
listable but not adoptable.
- Partial/placeholder sync files: fail closed on hash verification; the error
message distinguishes "not fully synced" from "tampered" only by wording,
intentionally, since the app cannot tell.
- Dev-trust misuse: PLUGIN_DEV_TRUST_DIRS silently ignored outside
DEBUG/TESTING; a prod config carrying it gets a startup warning.
- Frontend closure drift: plugin views importing cross-plugin components is a
graph the manifest does not model. The lint rule (plugin frontend may import
core @/ paths and its own tree only, never another plugin's) prevents new
edges; existing shared plugin-aware code is explicitly core-owned.
## Implementation phases
- Phase 0, groundwork (small, days): upgrade_all_plugins uses
registry.get_all(); reverse-dep checks read installed manifests from disk;
cycle detection in _sort_by_dependencies; shopdb/plugins/manifest_schema.json +
`flask plugin validate` (directory mode); `flask plugin apply-profile` with
install+enable closure ordering; fix Dockerfile stale comment. All additive,
zero risk to running sites.
- Phase 1, packaging and signing (medium, about a week): PROVENANCE format,
`flask plugin pack`, validate artifact mode, PLUGIN_TRUSTED_KEYS config,
ed25519 signing tooling and curator docs. No runtime behavior change yet.
- Phase 2, shelf and enforcement (medium-large, one to two weeks):
PLUGIN_SHELF_DIR, signed shelf-index with serial + revoked list,
`flask plugin shelf-list` / `adopt` / `audit` with atomic verified unpack;
verify-at-load in load_plugin and verify-at-migrate in
run_plugin_migrations, fail-closed in prod; PLUGIN_DEV_TRUST_DIRS for
dev/test and the external-repo harness; tier:core lifecycle guard;
provenance stamping of bundled plugins at build. This phase completes the
security model; everything after it is delivery optimization.
- Phase 3, Path A completion (medium, one to two weeks): generic renderers for
asset-panels, map-overlays, asset-presentation; migrate settingsNav.js
hardcoded plugin cards to get_settings_cards. Done BEFORE relocation so
JSON-only UI is available during the migration.
- Phase 4, frontend re-org (large, the long pole, several weeks elapsed):
stage-frontend.mjs staging + routes.gen.js/meta.gen.js codegen; ONE core PR
moving the six index.js hardcoded plugin imports and the core.js-embedded
plugin routes into plugin route files; then 13 per-plugin relocation PRs
(views, settings views, name-mismatch dirs, @/ alias rewrite) under the
dual-location union; lint rule for plugin frontend imports. Always-green,
per-plugin revertable.
- Phase 5, lean builds end to end (medium, about a week after Phase 4):
build-site.sh staging backend dirs + frontend staging from site-profile.json;
core lazy-import guard audit finished, enforced by the delete-a-plugin CI
matrix; remove the legacy glob; pilot one real lean site (a location without
printedparts/usb/network) and diff its image and bundle against a full build.
Deferred, each to its own future decision: schema-lean core-baseline re-org
(blocked on the installedapps -> machines FK question), pip/entry-point
distribution (ADR-003 v2), hook-based search/report aggregation contract, and
any revisit of Path B.
## Amendment, 2026-08-14
Status corrected from PROPOSED to ACCEPTED. The lean-build half of this decision
shipped some time ago - `scripts/build-site.sh`, `SITE_PLUGINS` staging,
`flask plugin prune-schema` and `default_enabled: false` are all live, and
ADR-014 was accepted on top of it - while the record still said the decision was
under consideration. A decision that has been implemented and depended upon is
not proposed, whatever the header says, and leaving it that way makes every
other status in the index worth less.
The catalog and signed-artifact parts of the decision remain unbuilt. That is
the ordinary state of an accepted decision: accepted means settled, not
delivered.

150
docs/adr/ADR-014-schema-lean-per-site.md Normal file
View File

@@ -0,0 +1,150 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Why a lean site stops creating database tables for plugins it never installed.
loadwith: none
-->
# ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables)
- Status: ACCEPTED
- Date: 2026-07-19
- Deciders: ShopDB maintainers
- Relates to: ADR-008 (per-plugin migration ownership), ADR-013 (plugin catalog + lean per-site builds), ADR-001 (asset model)
## Context
ADR-013 delivered lean per-site builds for plugin CODE (backend tree + frontend
bundle carry only chosen plugins). One residual was left, explicitly deferred:
the DATABASE. The core Alembic baseline (68b3947ae14f) creates EVERY table,
including ~30 plugin-owned tables (PLUGIN_TABLE_OWNERS). Each plugin's own
baseline is a stamp-only no-op (the core chain already made its tables, per
ADR-008). So a lean site that omits a plugin still creates that plugin's tables,
empty and unused.
The deferral cited a blocker: "the computers-owned installedapps table FKs
machines.machineid while computers declares no dependency on machines; reversing
the cutover would introduce undeclared hard deps or drop FKs; neither is
decided." Investigation refined this:
The cross-boundary foreign keys into the machines plugin table are ALL held by
DEAD legacy tables/columns that predate the asset model (ADR-001) and the
per-plugin cutover (ADR-008), and are queried nowhere in the codebase:
- `machinerelationships` (child/parentmachineid -> machines) - superseded by
`assetrelationships`. No model, no query.
- `printerdata` (machineid -> machines) - the pre-cutover printers table,
superseded by the printers plugin. No model, no query.
- `installedapps` (machineid -> machines) - a standalone machine-app link table;
the live relationship is `computerinstalledapps` (FK to computers only). The
standalone table has no model, no query.
- `communications.machineid` (-> machines) - a legacy column on the core
communications table (which is now assetid-based). Not read anywhere.
No LIVE plugin table hard-FKs another plugin's table. computerinstalledapps FKs
only computers.computerid (intra-plugin). So the blocker is dead cruft, not
live design.
## Decision
Two phases, both leaving existing databases correct.
### Phase 1: retire the dead cross-boundary cruft - ALREADY DONE
Investigation found this is already accomplished by existing migrations:
`7a01_adr001_position_contract` and `7c01_drop_legacy_machine` drop
`machinerelationships`, `printerdata`, `installedapps`, and
`communications.machineid` (with its FK). The current schema (verified on the
dev database) has none of them. So the cross-plugin FK blocker ADR-013 cited no
longer exists in the live schema - only in the baseline's transient
create-then-later-drop. No new migration is needed for Phase 1.
Precedent: ADR-001 dropped a cross-plugin FK the same way
(usbcheckouts.machineid -> machines became a soft sentinel).
### Enabling change (executed now): idempotent create_plugin_tables
`shopdb/plugins/alembic_template.py:create_plugin_tables` now skips any table
that already exists (inspects the bind first) instead of raising. This is the
mechanism Phase 2 needs: a plugin anchor can create its tables on a fresh lean
install AND be a safe no-op on an existing database that already has them from
the pre-cutover core baseline. Correct and inert regardless of Phase 2 (no
current caller creates against a populated schema). Verified against the
plugin-migration suite.
### Phase 2 (executed): prune not-installed plugin tables after upgrade
Two mechanisms were weighed to make a lean site's database carry only
core + chosen-plugin tables:
- **Relocate** (rejected): pull every plugin-table create/alter out of the core
chain into the plugin baselines, so the core chain never creates a
not-installed plugin's table. Measurement killed this: plugin tables are
created and altered across ~15 released core migrations (baseline plus 7c04,
7d05, 7d08, 7d13, 7d15, 7d16, 7d17, ...), not just the baseline. Because the
whole core chain runs before any plugin chain, removing a table's create from
core while a later core migration still alters it breaks FULL installs too, so
relocation means surgically rewriting ~15 released migrations - the highest
blast radius in the project - for a purely cosmetic gain (the omitted tables
are empty and the lean CODE build already never loads the plugin).
- **Prune-after-upgrade** (chosen): leave the entire core chain untouched. Add
`flask plugin prune-schema`, which drops the tables of every plugin in
PLUGIN_TABLE_OWNERS that is not installed on this site. Run once at deploy,
after `flask db upgrade` and `flask plugin upgrade-all`. Same end state
(core + chosen tables) with near-zero blast radius: no released migration is
edited, and an existing full site is unaffected because it never runs the
command.
`prune-schema` drops by table name (no plugin-code import), so it works on a
lean image where the omitted plugin's directory is absent. It is a dry-run by
default and refuses to drop a table that holds rows unless `--force`, so a
misfire on a populated site cannot silently delete data. Because the core chain
seeds a few plugin reference tables (e.g. 7d05 inserts default access
protocols), initial lean provisioning uses `--force` - at that point the tables
hold only migration-seeded defaults, before any site data exists.
The idempotent `create_plugin_tables` (enabling change above) is what lets a
lean site later ADD an omitted plugin: its anchor recreates the pruned tables.
Verified end to end on MySQL: fresh full install (86 tables) then prune is a
no-op; fresh lean install (machines + printers) then prune drops the other 19
plugin tables, leaving core + chosen; second prune is a no-op; the non-empty
guard refuses without `--force`. Four SQLite regression tests pin the behavior
(tests/test_plugin_prune_schema.py), running in the backend CI job via the real
CLI runner: drop-only-not-installed, full-site no-op, refuse-non-empty, and
force-drops-non-empty.
## Consequences
### Positive
- A lean site's database contains only core + chosen-plugin tables, with no edit
to any released migration (near-zero blast radius).
- The cross-plugin FK blocker ADR-013 cited is gone (dead cruft, dropped by
existing migrations), so plugin schemas are already FK-independent.
- Adding an omitted plugin to a lean site later just works: the idempotent
anchor recreates its tables.
### Negative / risk
- prune-schema is destructive by nature; the row-count guard + dry-run default +
required `--force` for non-empty tables contain that. It is a deploy-time
provisioning step, not something to run casually on a live populated site.
- A lean fresh install still transiently creates then drops the omitted plugins'
tables (the core chain builds them, prune removes them). Harmless and one-time
at provisioning; the trade for not touching the released baseline.
## Implementation
- Phase 1: nothing to do - the dead cross-boundary FK objects were already
dropped by existing migrations `7a01_adr001_position_contract` and
`7c01_drop_legacy_machine`; verified absent on a fresh full MySQL upgrade.
- Enabling change: `create_plugin_tables` made idempotent
(`shopdb/plugins/alembic_template.py`).
- Phase 2: `flask plugin prune-schema` (`shopdb/plugins/cli.py`), dry-run by
default, `--yes` to execute, `--force` for non-empty tables. Deploy order:
`flask db upgrade` -> `flask plugin upgrade-all` -> `flask plugin prune-schema
--yes --force`. Regression tests in `tests/test_plugin_prune_schema.py` (run in
the backend CI job).

156
docs/adr/ADR-015-site-specific-configuration.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Where a site own hostnames, paths and defaults are allowed to live, and where they are not.
loadwith: none
-->
# ADR-015: Where a site's own data is allowed to live
- Status: ACCEPTED
- Date: 2026-08-11
- Deciders: ShopDB maintainers
- Relates to: ADR-004 (per-site instances, not multi-tenant), ADR-013 (plugin catalog + lean per-site builds), ADR-014 (schema-lean per-site builds)
## Context
ShopDB is built as a product several GE Aerospace sites can adopt (ADR-004:
each site runs its own instance). It is also developed at one site, against
that site's live fleet. Every feature therefore arrives
carrying that site's hostnames, networks, share paths and vocabulary, and the
fastest way to ship any of them is to write the value where the code needs it.
That is not a hypothetical drift. A scan of `plugins/` and `shopdb/` for a site
host, a site FQDN or a site network returns 19 hits, of which roughly 8 are a
shipped default or a hardcode rather than a comment:
- `plugins/geenforce/seed_display_scope.py` baked one site's own ShopDB URL and
`device.geaerospace.net` into an inline PowerShell payload. A sister site
could not override either without editing the source. FIXED 2026-08-14: the
kiosk base URL now comes from the `site_base_url` setting at seed time and
from HKLM at run time, and the dispatcher refuses rather than guessing when
neither is set.
- `plugins/backups/services/registry.py` defaults the backup share root to one site's own
UNC path.
- `shopdb/core/api/settings.py` ships `device.geaerospace.net` and
`Printer-{ip}.printer.geaerospace.net` as SETTING DEFAULTS, so another site
inherits that site's domain silently until someone notices.
- `shopdb/core/models/dashboarddefault.py` holds the same domain a second time
as a module constant, and `plugins/computers/api/routes.py` a third time as a
literal fallback.
The problem is not that site-specific values exist - they must. It is that the
repository already has three good mechanisms for them and no rule about which to
use, so each new value lands wherever was convenient. The result reads as drift
because it is unmanaged, not because any single choice was wrong.
Nothing here is a security matter: these are internal hostnames on an air-gapped
network, and `tests/test_docs_publishable.py` already gates what reaches the
public wiki. This is about whether a sister site can adopt the product without
reading our source.
## Decision
A site's own data lives in one of three places, in this order of preference.
**1. A setting, with a NEUTRAL default.** Anything an operator could plausibly
change: hostnames, domains, share paths, networks, base URLs, label prefixes.
The default that ships must be blank or generic, never one site's value. Blank
must be a working state - code reads the setting and degrades sensibly, the way
`display_fqdn_domain` already treats blank as "use the hostname as-is". A
site-specific default is the failure mode this ADR exists to stop: it is
invisible, it looks deliberate, and it is wrong everywhere except here.
**2. A site-namespaced directory.** Anything that is genuinely one site's and
will never be another's, such as the reference site's import loader at
`scripts/site_imports/wjf/`. Never reached by another site's build.
**3. Seed data, not code.** Vocabularies and catalogs, loaded by `flask seed`
and editable per site afterwards.
Fleet-wide convention is NOT site-specific. The `gea-shopfloor-*` imaging types
are a GE fleet taxonomy, not one site's data, and they already carry a per-site
override through the `pctypemap_*` settings. They are out of scope here.
Comments and documentation may name a site where it explains WHY - "part markers
share a machine number at the reference site" is a fact a maintainer needs. Prefer
phrasing the rule generally and the example specifically.
## Enforcement
`scripts/check-naming-and-style.sh`, which already runs in CI, greps `plugins/`
and `shopdb/` for a site host, a site FQDN or a site network, excluding tests
and `site_imports/`.
It shipped in REPORT-ONLY mode, because turning a 19-item backlog into a hard
gate on the day it lands blocks every unrelated commit until someone works
through it. See the amendment below: it now fails the build.
## Consequences
Positive:
- A sister site can adopt without reading our source for values it must change.
- The backlog is visible and counted rather than felt.
- New site-specifics are caught at commit time, in the same place the naming
rules are already enforced, so there is nothing new to remember.
Negative:
- Report-only means the rule is advisory until someone flips it, and an
advisory rule can be ignored.
- A neutral default is more code than a working one: every read needs a blank
case, which is a real cost paid at every call site.
- The pattern list is a denylist and will miss a site value it has not been
taught. It catches drift, not everything.
## Backlog
CLEARED - see the amendment below. Kept as the record of what the decision was
answering:
- `plugins/geenforce/seed_display_scope.py` - kiosk base URL, FQDN domain, and
the legacy-shortcut sweep pattern
- `plugins/backups/services/registry.py` - `DEFAULTSHAREROOT`
- `shopdb/core/api/settings.py` - `display_fqdn_domain` and the printer
hostname pattern defaults
- `shopdb/core/models/dashboarddefault.py` - `DEFAULT_DISPLAY_FQDN_DOMAIN`
- `plugins/computers/api/routes.py` - literal fallback duplicating that default
The domain appears three times in three files, which is its own defect: one
setting, one default, one read.
Cosmetic mentions in comments and UI placeholder text are acceptable under this
ADR and are not part of the backlog, with one exception worth fixing when
convenient: the GE-Enforce settings page shows one site's real CIDRs as its
placeholder, which every site sees.
## Amendment, 2026-08-14
The backlog above is cleared and the scanner now ENFORCES: it fails the build,
and `SITE_LITERALS_ENFORCE=0` is the escape hatch rather than the default. The
report-only period ended for the reason the Consequences section predicted - the
hit count did not move for weeks, because a rule that only prints is read as no
rule.
Three things changed beyond clearing the list:
- The scan covers PowerShell, the installer, the seeds, generated JSON and the
frontend, case-insensitively, across `plugins/`, `shopdb/`, `scripts/`,
`deploy/`, `frontend/src/` and `tools/`. Its original scope - three file types
under two directories - missed every literal that actually reached a second
site, because those were all in the client scripts and the installer.
- A line may declare itself deliberate with a trailing `ADR-015-OK: <reason>`
marker. That is for an organisation-wide default which is right at every site
and configurable anyway, such as the corporate device domain. It is not a
way to defer work: the marker puts the claim in the diff where a reviewer
sees it.
- Both publication gates - the export scrub and the docs publishability test -
carry the site patterns, which neither did. The ADR said this was not a
security matter because the values were internal to an air-gapped network;
that was true of the fleet and false of the public mirror, which had been
carrying a production hostname and a share path.
This amendment is appended rather than edited into the text above, because what
the decision looked like when it was taken is the part worth keeping.

204
docs/adr/ADR-016-credential-delivery.md Normal file
View File

@@ -0,0 +1,204 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 plan for ShopDB-resolved credential targeting. Never built; superseded by ADR-019.
loadwith: none
-->
# ADR-016: Credential delivery to the fleet
- Status: SUPERSEDED by [ADR-019](ADR-019-per-pc-endpoint-credentials.md)
- Date: 2026-08-11
- Deciders: ShopDB maintainers
- Relates to: ADR-006 (collector contract), ADR-012 (GE-Enforce manifest ownership), ADR-015 (site-specific configuration)
> **Do not build against this. Nothing in it was built.** The model below has a
> PC ask ShopDB for credentials chosen by server-side targeting rules. The
> `GET /api/collector/credentials` endpoint, the `credentials.fetch` and
> `credentials.manage` permissions and the rule store do not exist in the code,
> and a reader searching for them will not find them.
>
> What shipped instead is the opposite shape: the endpoint holds ONE secret of
> its own, obtained by enrolling, and there are no targeting rules to resolve.
> [ADR-019](ADR-019-per-pc-endpoint-credentials.md) records that decision.
>
> The PROBLEM stated below is still the real problem, and ADR-019 cites it.
> Kept as the record of a design that was decided and then not taken.
## Context
Shopfloor bays need credentials they cannot prompt for. A collections bay loses
the Windows Credential Manager entry for its controller subnet periodically -
suspected Defender or Intune scrub - and without it, controller-side
connections need an interactive prompt, so an unattended bay simply stops
working. A machine whose NTLARS config points at a program share needs a
credential to reach that share for the same reason.
Today this is solved by `Set-ControllerCredential.ps1` on the SFLD share:
```powershell
$Target = '192.168.1.1'
$Username = 'CHANGEME'
$Password = 'CHANGEME'
```
A person edits the values into the file, GE-Enforce runs it every cycle, and
`cmdkey /add` re-applies it. It works, and it has three problems.
The password is **cleartext in a file on a share**, readable by anything that
can read the share - a wider exposure than the machine-local secret it becomes.
Rotating it means editing that file and hoping: there is no way to ask which
bays have picked up the new value. And it is **one value for everything**,
while the requirement is per-bay variation: sites will have machines needing
different credentials from their neighbours.
Per-bay variation is what rules out the obvious alternative. GE-Enforce scopes
are per PC TYPE, so expressing per-machine credentials as manifest scopes means
a scope per bay, which is not a model. The manifest can deliver a value
(`Registry` entries with inline payloads already do this for the display
scope), but it cannot decide WHICH value a given PC should get.
## Decision
ShopDB owns credential definitions and their targeting. A PC asks for its own
credentials and receives a decided answer; it never receives rules to evaluate.
### Targeting
A credential rule carries a targeting expression using the SAME axes
GE-Enforce already filters on, plus those the asset model knows:
| Axis | Source | Notes |
|---|---|---|
| hostname | reported | exact or `-like` glob (`WJRP*`) |
| machine | PC -> machine link | the machine the PC drives |
| model | asset model | e.g. every Okuma of one model |
| controller OS | asset / reported | version-valued, like the existing CmmVersion gate |
| PC type | ComputerType | with GE-Enforce ALIAS EXPANSION |
| GE-Enforce profile | manifest scope | the imaging profile |
| DNC/NTLARS setting | stored backup projection | e.g. any machine whose `DataHost` is a given UNC |
| site default | setting | the fallback |
The DNC axis is not a special case bolted on: the NTLARS projection ShopDB
already stores contains `DataHost`, `DataPath` and `MarkMasterPath`, so "every
machine that talks to \\ntshare" is a selector over data already held. It is
also the axis that answers "which machines even need a credential", which
nothing answers today.
### Precedence
Overlapping matches are guaranteed, not exceptional - a hostname glob and a
per-model rule will both hit the same bay. Precedence is therefore EXPLICIT,
never emergent:
1. Each rule carries an integer `priority`. Higher wins. This is the only
thing an operator needs to reason about.
2. Ties break on scope specificity, in the order of the table above (hostname
most specific, site default least).
3. Remaining ties break on the lowest rule id, so the result is stable rather
than dependent on row order.
A resolve endpoint must be able to EXPLAIN itself - "this PC got rule 12,
matched on model, beating rule 4 on priority" - or nobody will trust it. The
GE-Enforce simulator (`applicable_entry_names`) already set this precedent.
### Reuse the existing matcher
`plugins/geenforce/filters.py` already mirrors the engine's targeting: alias
groups, hostname globs, the rule that a machine-number filter with no machine
number EXCLUDES rather than defaults. The resolver reuses it. It must not
become a third implementation of the same matching logic - there are already
two (the PowerShell engine and its Python mirror), kept honest by a parity
harness, and a third would drift silently.
Alias expansion in particular is not optional: if a rule says `PCTypes:
Standard`, it must mean what it means in GE-Enforce, or the two systems will
disagree about who a rule covers.
### Targeting is not detection
Who gets a credential and whether it is already applied are separate
questions, exactly as `PCTypes` and `DetectionMethod` are separate in the
manifest. The client decides the second locally - is this `cmdkey` entry
already present and correct - and rewrites only on drift. Conflating them
rewrites the credential every cycle instead of healing it, and turns the audit
trail into noise.
### Storage
- Encrypted at rest with Fernet. `cryptography` is already a dependency.
- The key lives OUTSIDE the database - instance file or environment, like
`SECRET_KEY`. A key in the database it protects is not encryption.
- Write-only through the API. Masked on read, as `smtp_password` already is. A
fleet PC receives a value; a human never reads one back.
- A dedicated `credentials.manage` permission for authoring.
### Delivery
```
GET /api/collector/credentials X-API-Key: <credentials.fetch token>
```
- A DEDICATED `credentials.fetch` scope, not `collector.ingest`. A leaked
collector key must not yield controller passwords.
- The caller is resolved by hostname; it receives only what it is entitled to,
and never the rule set.
- Every fetch is audited: which PC, which credential, when. The audit log
exists; this belongs in it.
- The client half is a shared helper alongside `ShopdbBackupClient.psm1`:
fetch, compare, `cmdkey /add` on drift, log once per state.
## Consequences
Positive:
- The cleartext credential leaves the share.
- Rotation becomes a settings change plus a report of who has picked it up,
instead of a file edit and a hope.
- Per-bay, per-model and per-share credentials become expressible, which is the
actual requirement.
- "Which machines need a credential" becomes answerable from the DNC data
already stored.
Negative, and these are real:
- **ShopDB becomes a credential store.** That is a different security posture
from an asset database, and it earns the obligations above rather than
choosing them.
- **The key is a new single point of loss.** Restore the database without it
and every stored credential is unrecoverable. `docs/BACKUP-RESTORE.md` must
say so IN THE SAME CHANGE, not later.
- A PC that cannot reach ShopDB gets no credential. The client must keep the
last known good value rather than clearing a working entry on a failed fetch.
- Targeting complexity is real complexity. Eight axes with priorities will
produce a rule set someone has to debug, which is why the explain endpoint is
a requirement and not a nicety.
## Alternatives rejected
**Keep editing the script on the share.** Cannot express per-bay values, keeps
the cleartext, no rotation story.
**A manifest scope per bay.** Per-machine credentials as GE-Enforce scopes is a
spreadsheet, not a model, and the manifest cannot resolve WHICH value applies.
**Encrypted payload in the manifest.** DPAPI is per-machine, and one manifest
serves many machines, so the payload cannot be encrypted to its readers.
**Registry-only provisioning (no ShopDB).** A worthwhile FIRST STEP - it moves
the secret off the share immediately and needs no new endpoint - but it leaves
provisioning per-bay by hand and offers no rotation. Recommended as tier one
regardless, since the client helper is the same either way.
## Implementation status
Never implemented, and now superseded. The fetch endpoint and the
`credentials.*` permissions described above do not exist in the code.
The note that stood here until 2026-09-18 said the decision still stood and a
plugin author designing against it was designing correctly. That is no longer
true: design against
[ADR-019](ADR-019-per-pc-endpoint-credentials.md). What credentials the fleet
uses today, and where they live, is in
[FLEET-ARCHITECTURE](../FLEET-ARCHITECTURE.md).

151
docs/adr/ADR-017-buildings-and-levels.md Normal file
View File

@@ -0,0 +1,151 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Why the floor map became buildings and levels instead of one image with stacked floors.
loadwith: none
-->
# ADR-017: Buildings and levels as the map model
- Status: ACCEPTED
- Date: 2026-08-17
- Deciders: ShopDB maintainers
- Relates to: ADR-001 (asset as the platform contract), ADR-004 (per-site
instances), ADR-010 (frontend plugin hooks), ADR-015 (site-specific
configuration)
- Supersedes: the site-wide `map_blueprint_light` / `map_blueprint_dark` /
`map_width` / `map_height` settings
## Context
A site's floor map is one image. Four settings describe it - two blueprints
(light and dark) and the native pixel width and height - and `assets.mapx` /
`assets.mapy` are absolute pixel coordinates in that image's space.
That model has one floor in one building, which was true when it was written and
is no longer. The reference site is adding a second level, and a second building
is likely within a year or two. The immediate trigger is a new blueprint at
3308x4000 where the old one was 3300x2550.
The obvious cheap answer is to stack levels on one canvas: draw the second level
below the first and keep one image. It works, and it was seriously considered.
It was rejected because it makes the level an inference rather than a fact:
- "Which level is this asset on" becomes `mapy > 2550`. Every per-level filter,
count, report and export has to know that constant, and re-exporting the
drawing at a different height silently changes the answer everywhere. Not
broken - wrong, which is worse.
- Distance between two markers becomes computable and meaningless, so anything
doing nearest-asset or clustering quietly answers nonsense across levels.
- One canvas forces one scale. A mezzanine drawn at a different scale than the
floor below cannot be expressed at all.
- 3308x4000 is portrait for a building whose floor is landscape. At a zoom where
a marker is clickable, roughly half a level is visible, and "fit to level" is
not expressible.
- A third level makes each of these worse, and adds a second threshold.
Timing decided it. Because the level layout changed and machines moved, every
marker is going to be repositioned anyway. Introducing levels now costs one pass
over the positions; stacking now and splitting later costs two, and in between
every position placed encodes the threshold into real data - which a later
migration would then have to un-guess by comparing Y against it.
## Decision
Two tables, and assets reference the level.
```
buildings buildingid, buildingname, sortorder, isactive
maplevels levelid, buildingid, levelname, sortorder,
blueprintlight, blueprintdark, mapwidth, mapheight,
isdefault, isactive
assets levelid (nullable; the default level for existing rows)
```
**1. An asset references the level, never the building.** The building derives
from the level, so the two cannot disagree. Storing both would be a fact
recorded twice with no difference in granularity to justify it.
**2. Blueprints and native dimensions belong to the level.** This is what the
site-wide settings could not express: two levels in one building may be drawn at
different sizes and scales, and two buildings certainly are.
**3. Name and order are separate.** `levelname` is text and `sortorder` is an
integer. Levels are not reliably numbered - basement, ground, mezzanine, roof,
tunnel - and sort order gives adjacency and up/down navigation without
pretending the names are ordinal. It also lets a mezzanine be inserted between
two existing levels without renumbering anything.
**4. `mapx` / `mapy` keep their meaning, scoped to the level.** They stay
absolute pixels in the native coordinate space of the level's blueprint. No
normalisation to fractions: pixels are what the drawing tools produce, what an
operator can read off an image, and what the existing data already holds.
**5. A level is required to render a position, and absence is not a default.**
Where a payload carries a position without a level, the UI renders "level
unknown" rather than falling back to the default level. Falling back draws one
building's ground floor with a marker positioned for another building's
mezzanine: it renders perfectly and points at the wrong place, and nothing about
the result looks wrong. A visible gap is worth more than a confident wrong
answer.
**6. The level list is readable without authentication.** The printer installer
map runs optional-auth, before anyone logs in, and it needs a blueprint. This
follows the precedent already set for printer install-list and the slide feed.
Blueprint paths and level names are not secrets; the positions of assets on them
already render on public kiosk pages.
**7. Levels are not Locations.** Assets carry `locationid` already, and reusing
it is tempting. A Location answers "which operation owns this"; a level answers
"which drawing renders it, at what native size". Overloading Location with
blueprint images and pixel dimensions makes both concepts worse. They coexist:
an asset on level 2 in operation 0613.
## Migration
The existing four settings become one building and one level, marked default,
and every asset with a position points at it. Nothing renders differently on the
day it lands. The settings keys are then retired rather than left as a second
source of truth that can disagree with the rows.
## Consequences
Positive:
- A second building costs a row and a blueprint. So does a third level.
- Per-level export, per-level zoom, and per-level calibration all become
expressible. Calibrating one level against its own landmarks is more correct
than transforming a whole site at once.
- Level becomes queryable - counts and filters per level or per building are
ordinary queries rather than coordinate arithmetic.
Negative:
- Every surface that draws a marker learns there is more than one drawing. That
is two components, seven views, four asset position pickers, the printer
installer map, the PDF export, and eight API payloads that must emit `levelid`
beside `mapx`. A payload that forgets it produces the silent-wrong-map failure
described above, which is why rule 5 exists and why a build gate checks that
the two fields travel together.
- `get_map_overlays` gains level context, so this is a plugin contract change
(0.20.0) and plugins rendering overlays need to declare which level they are
for.
- One more admin surface: levels and buildings have to be managed somewhere, and
the existing floor-map settings page becomes that.
## Alternatives considered
**Stack levels on one canvas.** Rejected above. Zero cost today, and it makes
the level an inference over a magic number.
**Normalise coordinates to fractions of the image.** Would make a rescaled
blueprint self-correcting, which is genuinely attractive. Rejected for now
because it converts every existing integer position through a lossy division,
and because it does not help the actual problem: the levels changed and machines
moved, so the positions need human review regardless. Worth revisiting
independently.
**A `level` string on the asset, with one blueprint per level in settings.**
Rejected: settings keyed by level name is a table with extra steps, and it gives
no place for per-level dimensions or ordering.

132
docs/adr/ADR-018-metrology-instruments-are-machines.md Normal file
View File

@@ -0,0 +1,132 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why metrology instruments are machines and the measuringtools plugin is retired.
loadwith: none
-->
# ADR-018: Metrology instruments are machines
- Status: ACCEPTED
- Date: 2026-09-04
- Deciders: ShopDB maintainers
- Relates to: ADR-001 (asset as the platform contract), ADR-006 (collector
contract), ADR-008 (a plugin owns its tables and its migration chain)
- Supersedes: ADR-005 (equipment plugin scope vs measuringtools plugin)
## Context
ADR-005 split manufacturing machinery from metrology instruments into two
plugins on the grounds that they are two asset classes: machines make parts,
measuring tools measure them. It named the metrology population as "CMMs,
Keyence vision systems, wax-and-trace surface profilometers, GenSpec
instruments", and the `measuringtools` plugin was built against a docstring
naming a different population again: "calipers, micrometers, thread gages,
bore gages, height gages".
Those two lists are not the same thing, and the difference is the whole
decision. One is bay-mounted capital equipment with a controlling PC. The other
is hand tools in a drawer. The plugin was built for the second and populated
with neither.
Sixteen months on, the evidence is in:
- `measuringtools` holds **zero rows**, against 204 machines, 305 computers,
73 network devices and 53 printers.
- The metrology inventory actually waiting to be entered is 134 instruments:
35 CMM, 32 form tracer, 23 vision, 18 wax trace, 14 measuring, 8 Genspect,
2 eddy current, 1 spline. Every one is bay-mounted with a controlling PC.
- Not one caliper, micrometer, thread gage, bore gage or height gage. The
population the plugin's own docstring justifies it with does not exist at
the reference site and is not planned.
- `CMM` already exists as a `machinetype`, and `Part Marker` is already
classed as a machine while `Form Tracer` is classed as a measuring tool.
Two bay instruments with a controlling PC, split across two plugins, with
the line drawn somewhere no operator can predict.
The split has therefore been paying an ongoing cost - a duplicate type page, a
settings group, a nav section, a migration chain, a searcher, dashboard cards,
map colours, label printing and four collector `pctypemap` entries - to model a
distinction that the inventory does not make.
## Decision
Metrology and inspection instruments are machines. The `measuringtools` plugin
is retired and the `measuring_tool` asset type is removed.
### Calibration moves to machines, and is not lost
`machines` gains the calibration lifecycle as nullable columns:
```
machines
- calibrationintervaldays integer, null
- lastcalibrationdate date, null
- nextcalibrationdate date, null
- calibrationprovider varchar(150), null
```
Calibration status stays **derived at read time** from `nextcalibrationdate`
(`overdue` / `duesoon` / `current` / `unknown`, 30-day window), exactly as
`measuringtools` derived it and for the same reason: a stored status is stale
the day after it is written. The helper moves across unchanged.
This is a net gain rather than a port. A machine could not record calibration
at all before, and a part marker with a certified fixture, or a machine with a
calibrated probe, had nowhere to put it.
### `machinetypes.ismetrology` answers ADR-005's objection
ADR-005 made one concrete technical argument, and it deserves a concrete
answer rather than silence:
> "show me all measuring tools" requires an enumeration of measuring-instrument
> equipmenttype values, which scales badly.
That is correct about enumeration and it is why `machinetypes` gains a boolean
`ismetrology`. The metrology view stays a first-class single-column filter -
`WHERE mt.ismetrology = 1` - not a hardcoded list of type names that drifts the
first time someone adds an instrument type. The calibration report, the gage
lab view and any compliance export all hang off that flag.
### What does not change
- `assets.gaugelabreference` stays on the core asset. The gage tag was never a
`measuringtools` column, so it survives untouched and stays searchable for
every asset type.
- The collector's device identity file and the `measuringtoolid` payload alias
stay accepted. Bays staged with `measuringtool-id.txt` keep reporting; only
the asset type they resolve to changes.
- Instrument types keep their names. A Form Tracer is still a Form Tracer, now
as a `machinetype` with `ismetrology = 1`.
## Consequences
**The migration is free today and expensive later.** `measuringtools` has zero
rows, so retirement is a `DROP TABLE`, not a data migration. Once the 134
staged instruments are entered they carry relationships, audit history, map
positions and collector links, and every one of those would have to be
rewritten. This ADR is being taken now specifically because the window is open.
**A site that did populate it needs a real migration.** The reference site did
not and the sister site shows no sign of having done so - its branch contains
no deliberate work on the plugin in its entire history - but that is inference,
not a query against their database. Confirm before the removal ships to another
site. If rows exist, the retirement migration must copy them into `machines`
with `ismetrology` types rather than drop them.
**Four collector `pctypemap` entries change asset type.** `gea-shopfloor-cmm`,
`-keyence`, `-genspect` and `-waxtrace` resolve to `machine` instead of
`measuring_tool`. The `collector:measuringtool` relationship label is kept as a
recognised origin marker so links a collector already created are still
archived correctly rather than orphaned.
**One fewer of everything.** A duplicate type page, a settings group, a plugin
migration chain, a gated searcher, a set of dashboard cards, a map colour entry
and a label-printing branch all go. That is the point.
**ADR-005 is superseded, not reversed in principle.** Its reasoning was sound
for the population it described. The population never arrived. If a site ever
does track hand gages in quantity, the answer is a `handgages` plugin against
that real inventory - not the reinstatement of this one.

217
docs/adr/ADR-019-per-pc-endpoint-credentials.md Normal file
View File

@@ -0,0 +1,217 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why every endpoint gets its own enrolled secret instead of one fleet-wide shared key.
loadwith: none
-->
# ADR-019: Per-PC endpoint credentials
- Status: ACCEPTED
- Date: 2026-09-18
- Deciders: ShopDB maintainers
- Relates to: ADR-006 (collector contract), ADR-012 (GE-Enforce manifest
ownership), ADR-015 (site-specific configuration)
- Supersedes: ADR-016 (credential delivery to the fleet)
## Context
The endpoints plugin authenticates every agent-facing call with ONE key. The
same `X-API-Key` value is on every PC in the fleet, and the server cannot tell
one caller from another: a request names its own hostname and is believed.
That key is not only an inventory-ingest key. It is also the key on the command
channel (`AGENT-CONTRACT.md` section 8), where the server hands an agent running
as LocalSystem an allowlisted action to execute - including `run_script`, which
fetches a reviewed PowerShell body and runs it. Anyone holding the key can claim
commands as any PC and post results as any PC.
And the key is readable. The agent installer writes it to
`HKLM\SOFTWARE\GE\ShopDB` `CollectorKey`, and the DACL that subkey inherits by
default grants `BUILTIN\Users` read, so any logged-on user on any bay could read
the fleet key (the reason is recorded in `plugins/endpoints/api/siteagent.py`
beside the code that now locks the key down). The same value also lived in
`appsettings.json` in the agent directory, in a text file on the GE-Enforce
share, and in the `Args` of the manifest entry that wrote it.
One fleet-wide secret, readable by a standard user on any bay, authenticating a
SYSTEM command channel. Rotating it meant touching every one of those places at
once, and nothing could say which PCs had picked the new value up.
ADR-016 answered a different question about credentials - which controller
password a given bay should receive, decided by server-side targeting rules - and
was never built. The question underneath it is "which PC is this", and it needs
no targeting at all.
## Decision
Each endpoint holds its OWN secret, obtained by enrolling, and the server
derives the calling asset from the secret.
### The credential
- `X-Endpoint-Credential: sdbe_` + 43 characters of base64url (32 random bytes).
- The server stores only the SHA-256 of the secret plus its first 12 characters
as a display prefix (`EndpointCredential` in `plugins/endpoints/models/credential.py`).
The full value exists once, in the enroll response.
- One row per asset, so an asset never carries two credentials. Re-issuing after
a revoke updates the same row.
- A request whose credential resolves to one asset while naming another is
refused `403 ASSET_MISMATCH`. A PC can no longer report as another PC.
### Enrollment
`POST /api/endpoints/enroll` never refuses a call for its authentication. The
shared key, if present and still allowed, only decides whether the credential is
issued automatically. Auto-issue requires all of: the shared key authenticated
the call, `plugin.endpoints.autoenroll` is on, the hostname matches exactly one
computer record, the serial on record matches, and the asset has no active
credential. Anything else records a PENDING request an admin approves in the UI.
The agent generates an enrollment nonce once and persists it. The nonce is
recorded on first sight and an approval is bound to it, so a second machine
claiming the same hostname and serial cannot collect the approved credential. A
rejected request binds to the identity instead, so a reinstall cannot argue its
way back into the queue.
### Storage on the endpoint
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
`LocalMachine`, in a directory ACLed to SYSTEM and Administrators only. The tray
holds no secret at all: it runs as the interactive operator and asks the SYSTEM
service over a local named pipe. `HKLM\SOFTWARE\GE\ShopDB` is locked to SYSTEM
and Administrators in both registry views.
### Server pinning
The agent pins the server's public key: the lowercase hex SHA-256 of the leaf
certificate's SubjectPublicKeyInfo, in `ServerPins` (REG_MULTI_SZ, both views),
authoritative whenever the value exists even when empty. Locking the registry
key is what makes the registry safe to treat as authoritative, since a standard
user can no longer write a pin.
### The shared key stays, deliberately
`plugin.endpoints.sharedkeyallowed` keeps the shared key working for PCs that
have NOT enrolled. It is always refused for a PC that holds an active credential
(`401 SHARED_KEY_REFUSED`). A PC whose credential was REVOKED falls back to the
shared key, which is what makes a rollback to an older agent possible;
`plugin.endpoints.revokeddeniessharedkey` turns that off and is off by default,
because it is only safe once the fleet is off the shared key.
## Alternatives rejected
**Client certificates, 2026-09-14.** This was not a thought experiment: it was
built and then removed three days later. `client_cert.py`, certificate renewal,
expiry monitoring, an unauthenticated `/test/client-cert` route, nginx
client-certificate forwarding and an agent certificate script all shipped on a
branch and were all deleted.
Three reasons, and the first is fatal on its own:
1. **The header was forgeable.** Behind IIS the client certificate reaches the
app as a request header, and nothing stopped a caller from simply sending
that header itself.
2. **The private key was readable by users** on the endpoint, so the certificate
was no better protected than the key it replaced.
3. **The shared key remained underneath it** as a fallback, so the weakest path
was still open and the certificate bought nothing.
It also cost more than it looked. The attempt put five underscore-spelled
certificate columns on the CORE `Asset` model for a plugin's feature and
retracted them again, with no contract bump in either direction - an ADR-001 and
ADR-002 breach inside an unreleased branch. Its migrations
(`endpoints0021clientcerts`, `0022`) survive as retired no-ops so a database
that ran them still upgrades; `0023` drops the columns where a database has
them.
**ShopDB-resolved credential targeting (ADR-016).** Decided 2026-08-11 and never
implemented. It gave a PC a credential chosen by rules over eight targeting axes
with an explicit priority order and an explain endpoint to debug the result. The
targeting was the whole design, and the exposure that needed closing was not a
targeting problem: the fleet needed identity, not a rule engine. ADR-016 is
superseded rather than deleted because the problem it opens with - a cleartext
controller password on a share, with no rotation story - is still unsolved. This
ADR does not solve it.
**Keep one key and just lock down where it lives.** The registry ACL is part of
this decision anyway, and it is worth having on its own. It does not help: one
secret shared by every PC still means the server cannot identify a caller, a
revoked bay cannot be evicted without rotating the whole fleet, and any
administrator or any process running as SYSTEM on any bay still reads the key
that drives every other bay's command channel.
## Consequences
**The shared key remains until every consumer is migrated.** That is the main
cost of this decision and it is not short. The key lives in a GE-Enforce
manifest entry's `Args`, in `CollectorKey` in both registry views, in a text file
in the share's `common/configs`, in `appsettings.json` in the agent directory, in
every generated `install.ps1` that was kept, and on display kiosks. Each one has
to be cleared in a specific order - retire the entry that rewrites the value
before deleting the value - and the agent never deletes the shared key itself,
because other SYSTEM scripts may still read it. The runbook
(`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`) carries the table.
**Turning the shared key off is wider than the agent.** It is also refused on
`/api/collector/computers` and `/api/collector/printers`, so a PC with reporter
scripts but no enrolled agent goes silent at cutover.
**Enrollment is a new admin queue.** Anything that cannot be auto-issued waits
for a person: unknown or duplicate hostname, missing or mismatched serial, a
one-off PC nobody wrote a manifest entry for. A dashboard row counts PCs waiting
and PCs blocked from re-enrolling.
**Pins have to arrive before the agent does.** An agent that pins with no pin set
falls back to plain Windows chain validation, which a self-signed or internal-CA
server may fail, and a pin set after the fact is a per-bay debugging job.
Rotating the server key becomes a two-phase fleet operation: publish old and new
pins everywhere, switch the certificate, then retire the old pin. Switching
first cuts off every pinned PC, including its route to the corrected pin list.
**GE-Enforce can undo an upgrade.** Agents installed from the share are held to a
file hash by a manifest entry. A store upgrade that lands a newer agent while the
entry still carries the old hash is reverted on the next cycle, so the entries
have to be re-stamped or removed before the waves reach those PCs.
**Revoke means two different things depending on one setting.** With
`revokeddeniessharedkey` off, revoking drops a PC back to the shared key, which
is how rollback and credential rotation work. With it on, revoking is a real
eviction and the PC reports nothing until it enrolls again.
**A PC that cannot reach ShopDB keeps its credential.** Unlike ADR-016's fetch
model, there is nothing to re-fetch per cycle: the secret is local and long-lived
until an admin revokes it.
## Implementation status
**Built, and in this tree.** `plugins/endpoints/api/agentauth.py`,
`api/enrollment.py` and `models/credential.py` are all present,
`shopdb/core/api/collector.py` reads `X-Endpoint-Credential` on the generic
collector route, `AGENT-CONTRACT.md` carries section 9, and
`tests/test_plugins/test_endpoints_credentials.py` covers the auth decisions.
The C# agent enrolls and sends the header, and both shipped PowerShell reporters
send it when a credential is present and fall back to the shared key when it is
not.
Proven on a Windows test PC against a test server: self-enrollment, the shared
key refused for an enrolled PC, a request naming another asset refused, pins
enforced, the tray pipe working for a non-admin user, both SYSTEM reporters
sending with the per-PC credential, revoke followed by re-enrollment, rollback
to the previous agent, and enrolled PCs still working with the shared key
switched off.
What is NOT finished is the fleet migration, which is the consequence this ADR
opens with: every PC that has not enrolled is still on the shared key by design,
and the key still lives in the places listed above. The runbook is
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
## References
- `plugins/endpoints/api/agentauth.py` - credential and shared-key authentication
- `plugins/endpoints/api/enrollment.py` - enroll, approve, revoke, migration waves
- `plugins/endpoints/models/credential.py` - credential, enroll request, wave
- `plugins/endpoints/AGENT-CONTRACT.md` section 9 - the wire contract
- `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md` - the rollout runbook
- ADR-016 - the superseded targeting model

69
docs/adr/README.md Normal file
View File

@@ -0,0 +1,69 @@
<!-- docmeta
purpose: reference
audience: developer, adopter
status: current
summary: Which architectural decision governs your question, and whether it still holds.
loadwith: none
-->
# Architecture Decision Records
An ADR captures one decision: the context, the decision, the consequences and
the alternatives. ADRs are immutable once accepted. A superseded ADR stays in
this folder, marked at the top, pointing forward.
**Read the index, not the folder.** Find the one line that matches your
question, open that ADR, and skip the rest. Nothing here is a prerequisite for
anything else.
## Which decision governs your question
| ADR | The question it answers | Holds? |
|---|---|---|
| [001](ADR-001-asset-as-platform-contract.md) | What may a plugin depend on? What is the versioned platform surface? | In force |
| [002](ADR-002-plugin-versioning.md) | How is that surface versioned, and what does a plugin's `core_version` mean? | In force |
| [003](ADR-003-plugin-distribution.md) | How does a plugin reach a site - bundled, or from its own repo? | In force. ADR-013 added the catalog, artifact and signing on top |
| [004](ADR-004-deployment-topology.md) | One instance per site, or one multi-tenant instance for all sites? | In force, except its "Migration strategy" section, replaced by ADR-008 |
| [005](ADR-005-equipment-vs-measuringtools.md) | Are metrology instruments a separate asset class from machinery? | **Superseded by [018](ADR-018-metrology-instruments-are-machines.md).** Answer reversed |
| [006](ADR-006-collector-contract.md) | How does a plugin accept data pushed in from outside? | In force |
| [007](ADR-007-product-versioning-and-releases.md) | What version is the PRODUCT, as distinct from the plugin contract? | In force |
| [008](ADR-008-plugin-migration-ownership.md) | Whose Alembic chain owns a plugin's tables? | In force |
| [009](ADR-009-frontend-plugin-gating.md) | What happens when someone opens a URL belonging to a disabled plugin? | In force |
| [010](ADR-010-frontend-plugin-hooks.md) | How does a plugin add UI without editing core Vue files? | In force |
| [011](ADR-011-machines-rename.md) | Why is the code called `machines` and what happened to `machinetypes`? | In force |
| [012](ADR-012-geenforce-manifest-ownership.md) | Does ShopDB own GE-Enforce manifests, and does it own the engine? | In force |
| [013](ADR-013-plugin-catalog-and-lean-builds.md) | How does a site ship only the plugins it chose, and how is plugin code trusted? | In force |
| [014](ADR-014-schema-lean-per-site.md) | Does a lean site still create tables for plugins it never installed? | In force |
| [015](ADR-015-site-specific-configuration.md) | Where may one site's hostnames, shares and defaults live? | In force |
| [016](ADR-016-credential-delivery.md) | How does a bay get a controller credential without a cleartext file on a share? | **Superseded by [019](ADR-019-per-pc-endpoint-credentials.md), and never built.** Do not build against it |
| [017](ADR-017-buildings-and-levels.md) | How does the floor map hold more than one floor and more than one building? | In force |
| [018](ADR-018-metrology-instruments-are-machines.md) | Where do CMMs and vision systems live now that `measuringtools` is retired? | In force |
| [019](ADR-019-per-pc-endpoint-credentials.md) | How does the server know WHICH PC is calling, and what replaced the fleet-wide key? | In force. Built, not yet merged into this tree |
Related decisions that are not ADRs, kept as records of why something was built:
[docs/proposals](../proposals/README.md).
## Status vocabulary
The `Status:` line at the top of each ADR uses these, and nothing else:
- **PROPOSED** - drafted, awaiting decision.
- **ACCEPTED** - in effect.
- **SUPERSEDED by ADR-NNN** - replaced. The link forward is part of the status,
and the superseded ADR carries a note at the top saying not to build against
it and what changed.
- **DEPRECATED** - no longer in effect, with no replacement.
## Authoring
1. Copy the most recent ADR as a template and increment the number.
2. Start the file with a `docmeta` comment, before the H1. ADRs are
`purpose: explanation`.
3. Add a row to the table above: the QUESTION it answers, not its title.
4. Add it to [docs/INDEX.md](../INDEX.md) too - under "Decisions" while it is
current, under "Archive" once it is superseded. It has to be a table ROW
with its real line count, not a mention in a paragraph:
`tests/test_docs_index.py` fails on an ADR no row routes to, and on a row
whose line count is stale.
5. Never edit an accepted ADR's decision. Supersede it with a new one, and mark
the old one at the top - ADR-005 and ADR-016 are the worked examples.

20267
docs/api-inventory.json
View File

File diff suppressed because it is too large Load Diff

221
docs/deployment/DEPLOY-TO-VM.md
View File

@@ -1,177 +1,136 @@
# Deploy ShopDB to Hyper-V VM
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Put the Docker stack on a throwaway Windows VM to try it or rehearse a deploy, and what the helper script gets wrong.
loadwith: none
-->
# Deploying the stack to a Windows VM
Quick guide to deploy the Docker stack to a Windows 11 Hyper-V VM.
> **Where this sits:** a rehearsal, not a site. For a real site follow
> [DEPLOY](../DEPLOY.md). To only LOOK at a build with no source at all,
> [DEV-BUILD-TEST](../DEV-BUILD-TEST.md) pulls a published image and is one
> script.
## Prerequisites on VM
A Windows VM with Docker Desktop is the cheapest way to prove a deploy before
doing it for real: the stack is identical to a site's, the VM can be reverted,
and nothing on it matters. This page is the VM-specific part - getting the code
there, the firewall, reaching it from the host. Everything else is
[DEPLOY](../DEPLOY.md).
1. **Docker Desktop** installed and running
2. **Git Bash** or **WSL** installed (for bash commands)
3. **PowerShell** (already on Windows 11)
## On the VM
## Step 1: Copy Files to VM
1. **Docker Desktop**, installed and running.
2. **Git Bash or WSL**, for the bash commands.
3. Enough RAM given to Docker. The stack is MySQL plus a Python container plus
nginx; 8GB to Docker is comfortable.
## Step 1: Get the code onto the VM
**Method A: Shared Folder**
```powershell
# On host PC, share the repo
# Then on VM, copy from network share
Copy-Item -Recurse \\host-pc\shopdb-flask C:\shopdb
```
# From a network share
Copy-Item -Recurse \\<host>\shopdb-flask C:\shopdb
**Method B: Git Clone**
```bash
# On VM
cd C:\
git clone git@github.com:ge-aero/shopdb-flask.git
cd shopdb-flask
git checkout wj
```
# Or clone it
git clone <your-remote>/shopdb-flask.git C:\shopdb
**Method C: Export/Import**
```powershell
# On host PC
Compress-Archive -Path C:\Users\270015376\Documents\github\flask-shopdb -DestinationPath C:\shopdb-deploy.zip
# Copy shopdb-deploy.zip to VM, then on VM:
# Or hand it over as a zip, which is what an isolated VM usually wants
Expand-Archive -Path C:\shopdb-deploy.zip -DestinationPath C:\shopdb
```
## Step 2: Run Deployment Script
## Step 2: Bring it up
The pre-built path is the one to use on a VM: npm inside the image is slow and
is the step most likely to fail behind a proxy. Build the frontend once on the
host or on the VM, then build the image from it -
[DOCKER-PREBUILT-APPROACH](DOCKER-PREBUILT-APPROACH.md).
```powershell
# On VM, open PowerShell as Administrator
cd C:\shopdb
copy .env.example .env
# Edit .env: SECRET_KEY, JWT_SECRET_KEY, MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD,
# CORS_ORIGINS. Generate a secret with:
# python -c "import secrets; print(secrets.token_urlsafe(64))"
# Option 1: Quick deploy with defaults
.\deploy-to-vm.ps1
cd frontend; npm ci; npm run build; cd ..
# Option 2: Specify site name and password
.\deploy-to-vm.ps1 -SiteName "west-jefferson" -AdminPassword "YourSecurePassword123!"
# Option 3: See all options
.\deploy-to-vm.ps1 -Help
```
## What the Script Does
1. [OK] Checks Docker Desktop is running
2. [OK] Creates .env from template
3. [OK] Builds frontend (if needed)
4. [OK] Builds Docker images (2-3 minutes, offline packages)
5. [OK] Starts all services (db + api + backup)
6. [OK] Runs database migrations
7. [OK] Sets admin password
## Manual Deployment (Alternative)
If you prefer to run commands manually:
```bash
# 1. Create environment
cp .env.docker-dev .env
# 2. Build frontend (if not already built)
cd frontend
npm ci
npm run build
cd ..
# 3. Build and start Docker stack
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
# 4. Initialize database
docker exec flask-shopdb-api-1 flask db upgrade
# 5. Reset admin password (optional)
docker exec flask-shopdb-db-1 mysql -uroot -proot-dev-password-not-for-production shopdb_flask -e \
"UPDATE users SET passwordhash='scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547', failedlogins=0, lockeduntil=NULL WHERE username='admin';"
# (Password is: admin)
```
## Access
The `migrate` service runs the schema, the plugin bootstrap and the seeders
before `api` starts, so there is no separate migration step. Watch it with
`docker compose logs migrate`.
**URL:** https://localhost:5001 (or https://vm-ip-address:5001 from host)
**Login:**
- Username: `admin`
- Password: Whatever you set in deploy script (default: `ChangeMe123!`)
## Verify Deployment
Create the first admin at `https://localhost:5001/setup`, or headlessly:
```powershell
# Check all services are running
docker ps
# Should see:
# - flask-shopdb-db-1
# - flask-shopdb-api-1
# - flask-shopdb-backup-store-1
# - flask-shopdb-backup-watcher-1
# View logs
docker logs -f flask-shopdb-api-1
# Test backup
# Go to https://localhost:5001/settings/shopdbbackups
# Click "Back Up Now"
# Wait 30 seconds, should show as "Checked"
docker compose exec api flask seed admin --username admin --email admin@example.com
```
## Troubleshooting
The password is generated and printed once.
**Docker not running:**
```powershell
# Start Docker Desktop manually
& "C:\Program Files\Docker\Docker\Docker Desktop.exe"
Start-Sleep -Seconds 30
```
## Step 3: Reach it from the host
**Port 5001 already in use:**
```powershell
# Edit .env and change API_PORT
# API_PORT=5002
```
**Can't access from host PC:**
```powershell
# On VM, check Windows Firewall
New-NetFirewallRule -DisplayName "ShopDB" -Direction Inbound -LocalPort 5001 -Protocol TCP -Action Allow
```
**SSL certificate warnings:**
- Normal for self-signed certs
- Click "Advanced" -> "Proceed to localhost (unsafe)"
Then `https://<vm-ip>:5001` from the host. The certificate is self-signed until
you replace the pair in `certs/`, so the browser warns; that is expected on a
VM.
## Stop/Start
`CORS_ORIGINS` is an exact origin match, so it has to name the address the
browser really uses. Reaching the VM by IP while `CORS_ORIGINS` names
`localhost` is the usual cause of "the page loads and every request fails".
## About `deploy-to-vm.ps1`
The repository root carries `deploy-to-vm.ps1`, which wraps the above into eight
steps with `-SiteName` and `-AdminPassword` parameters. **Read it before you run
it.** As it stands it does not work on a clean clone, and neither problem is
subtle:
- It copies `.env.docker-dev` to create `.env`. That file is not in the
repository, so step 3 fails on a fresh checkout. Create `.env` from
`.env.example` yourself first and the script skips that branch.
- It addresses containers by the fixed names `flask-shopdb-api-1` and
`flask-shopdb-db-1`. Compose names containers after the project, which is the
directory name, so on a clone called `shopdb` those `docker exec` calls
miss. Use `docker compose exec api` instead, or `docker compose ps` to see
the real names.
It also runs only `flask db upgrade` for the database, which predates the
`migrate` service and is now redundant rather than wrong.
Its default `-AdminPassword` is a literal in the file and therefore public.
Pass your own, and never leave the default on anything reachable.
## Stop, start, update
```bash
# Stop
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml down
# Start (after first deployment)
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml --profile backup up -d
```
## Update After Code Changes
`down` keeps the volumes. `down -v` deletes them, which is the database and
`instance/` together, with no prompt.
To take new code:
```bash
# 1. Pull latest code
git pull origin wj
# 2. Rebuild frontend (if changed)
cd frontend && npm run build && cd ..
# 3. Rebuild and restart
git pull
cd frontend; npm run build; cd ..
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml build
docker compose -f docker-compose.prebuilt.yml -f docker-compose.backup.yml up -d
```
## Offline Deployment Notes
Rebuilding the frontend is not optional on the pre-built path: the image copies
whatever is in `frontend/dist/`, so a skipped `npm run build` ships the old UI
with no warning.
**This deployment works completely offline** because:
- [OK] Frontend pre-built and checked into repo
- [OK] Restic binary pre-downloaded (8.6MB in repo)
- [OK] No npm/pip downloads during Docker build
- [OK] All dependencies baked into images
## Why this works with no network
Perfect for air-gapped or restricted network environments!
With `frontend/dist/` already built and the Python wheels vendored, the image
build downloads nothing, which is what makes the same procedure the rehearsal
for [DEPLOY-AIRGAP](../DEPLOY-AIRGAP.md).

137
docs/deployment/DOCKER-PREBUILT-APPROACH.md
View File

@@ -1,96 +1,87 @@
# Docker Pre-Built Frontend Approach
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build the Docker image where npm cannot reach the registry, by compiling the frontend on the host first.
loadwith: none
-->
# Building the image with a pre-built frontend
**Problem:** Zscaler corporate proxy causes npm to fail inside Docker builds with "Exit handler never called" error.
> **Where this sits:** a variant of Step 2 of [DEPLOY](../DEPLOY.md), not a
> separate deployment. Everything else on that page still applies.
**Solution:** Build frontend on host (where npm works), copy pre-built dist/ into Docker image.
The shipped `Dockerfile` is a multi-stage build: stage 1 runs `npm ci` and
`npm run build` under Node, stage 2 copies the compiled SPA into the Python
image. Behind a proxy that re-signs HTTPS, stage 1 is where it dies - npm inside
the container does not trust the corporate root CA, and the failure reads
`npm ci failed` or the unhelpful `Exit handler never called`.
## Quick Start
The fix is to build the frontend where npm already works - the host, which has
the corporate CA in its trust store - and give the image the finished
`frontend/dist/`.
It is also much faster: two to three minutes instead of fifteen. A site with no
proxy problem at all may want it for that reason alone.
## Do it
```bash
# 1. Build frontend on host
# 1. Build the frontend on the host
cd frontend
npm ci
npm run build
# 2. Build Docker image (uses pre-built dist/)
npm run build # writes frontend/dist/
cd ..
# 2. Build and start the stack from the pre-built dist
docker compose -f docker-compose.prebuilt.yml build
# 3. Start services
docker compose -f docker-compose.prebuilt.yml up -d
# 4. Initialize (first time only)
docker compose -f docker-compose.prebuilt.yml exec api flask db upgrade
docker compose -f docker-compose.prebuilt.yml exec api bash /app/scripts/bootstrap-plugins.sh
# 5. Access
https://localhost:5001
```
## Files
The rest is [DEPLOY](../DEPLOY.md) unchanged: the `migrate` service runs the
schema, the plugin bootstrap and the seeders before `api` starts, nginx
terminates TLS on `API_PORT`, and the first admin comes from `/setup` or
`flask seed admin`.
- `Dockerfile.prebuilt` - Simplified Dockerfile without npm build
- `docker-compose.prebuilt.yml` - Uses Dockerfile.prebuilt
- `.dockerignore.prebuilt` - Allows frontend/dist to be copied
## The files this uses
## Why This Works
| File | What it is |
|---|---|
| `Dockerfile.prebuilt` | single-stage: Python dependencies plus a copy of `frontend/dist/`. No Node, no npm |
| `docker-compose.prebuilt.yml` | the same stack, building from `Dockerfile.prebuilt` |
| `.dockerignore.prebuilt` | permits `frontend/dist/` to be copied, which the normal `.dockerignore` excludes |
**Normal Dockerfile:**
- Multi-stage build
- Stage 1: Node 20 runs `npm ci` and `npm run build`
- Stage 2: Copies built frontend from stage 1
- **PROBLEM:** npm fails inside Docker with Zscaler proxy
## What it costs
**Pre-Built Approach:**
- Build frontend on Windows host (npm works here)
- Single-stage Dockerfile copies pre-built `frontend/dist/`
- Only installs Python dependencies
- **FASTER:** 2-3 minutes vs 15+ minutes
- **npm has to exist on the build host.** That is the whole trade: the trust
problem moves from the container to a machine that has already solved it.
- **It is two steps, and forgetting the first one ships a stale UI.** The image
copies whatever is in `frontend/dist/` at build time. A frontend change that
was not rebuilt is simply not in the image, and nothing warns you. If the UI
does not show your change, rebuild `dist/` before debugging anything else.
- **`frontend/dist/` has to reach the build.** Either it is committed, or the
pipeline builds it immediately before the image.
## Trade-offs
## The version of this that scales
**Pros:**
- Avoids corporate proxy issues
- Much faster builds (no npm in container)
- Frontend build errors visible immediately
- Can iterate on frontend without rebuilding container
**Cons:**
- Requires npm on host machine
- Two-step build process
- frontend/dist must be in version control OR built before deployment
## For Production (CI/CD)
The pre-built approach is a workaround for building at the deployment site. The
end state is not building there at all:
```yaml
# GitHub Actions / Jenkins example
- name: Build frontend
run: |
cd frontend
npm ci
npm run build
- name: Build Docker image
run: |
docker build -f Dockerfile.prebuilt -t shopdb:${{ github.sha }} .
- name: Push to registry
run: |
docker push shopdb:${{ github.sha }}
# CI: build the frontend, build the image, push it
- run: cd frontend && npm ci && npm run build
- run: docker build -f Dockerfile.prebuilt -t shopdb:$GIT_SHA .
- run: docker push <registry>/shopdb:$GIT_SHA
```
## Comparison
A site then pulls a tagged image and runs it - seconds, no build step, no npm
anywhere near the facility. `docker-compose.pull.yml` is the compose file for
that shape today, and `docker-compose.airgap.yml` is the same idea for a site
that loads an image tarball off removable media
([DEPLOY-AIRGAP](../DEPLOY-AIRGAP.md)). Finishing that path is the distributed
release item on [ROADMAP](../ROADMAP.md).
| Approach | Build Time | Corporate Proxy | Complexity | Production Ready |
|----------|-----------|----------------|------------|------------------|
| Multi-stage Dockerfile | 15+ min | [FAIL] Fails | High | [FAIL] No |
| Pre-built frontend | 2-3 min | [OK] Works | Medium | [WARN] Yes (with CI/CD) |
| Pre-built images from registry | 30 sec | [OK] Works | Low | [OK] Yes |
## Next Steps
For true production deployment:
1. Set up CI/CD to build frontend + Docker image
2. Push to container registry
3. Sites pull pre-built images
4. No build step at deployment site
| Approach | Build time | Behind a re-signing proxy | Where it belongs |
|---|---|---|---|
| Multi-stage `Dockerfile` | 15+ min | fails unless the CA is staged | a developer box with a clean network |
| Pre-built frontend (this page) | 2-3 min | works | a site that must build locally |
| Pull a published image | seconds | works | where every site should end up |

398
docs/deployment/DOCKER-SETUP.md
View File

@@ -1,398 +0,0 @@
# ShopDB Docker Setup
Complete Docker deployment for development and testing.
## Prerequisites
- Docker Desktop installed and running
- At least 8GB RAM allocated to Docker
- Zscaler corporate certificate extracted (if behind corporate proxy)
## Quick Start
```bash
# 1. Clone repository
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Extract Zscaler certificate (if needed)
powershell.exe -File C:/temp/extract-ca.ps1
# This creates corp-root-ca.pem in repo root
# 3. Build images
docker compose build
# 4. Start stack
docker compose up -d
# 5. Initialize database (first time only)
docker compose exec api flask db upgrade
# 6. Install plugins (first time only)
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
# 7. Access ShopDB
https://localhost:5001
```
Login: `admin` / (reset password - see below)
## Architecture
The Docker stack includes:
1. **MySQL Database** (`db` service)
- Port: 3306 (internal), mapped to host based on `.env`
- Volume: `db_data` for persistence
- Health checks enabled
2. **Flask API** (`api` service)
- Port: 5001 (HTTPS with self-signed cert)
- Built frontend served from `/app/frontend/dist`
- Gunicorn with gevent workers (4 workers, 2000 connections each)
- Volumes:
- `instance_data` - Plugin state, uploads, branding
- `./plugins` - Plugin code (read-only)
- `./scripts` - Deployment scripts (read-only)
- `./certs` - TLS certificates (read-only)
3. **Backup Service** (optional, via `docker-compose.backup.yml`)
- One-shot container for backups
- Triggered manually or via scheduler
## First-Time Setup
### 1. Extract Zscaler Certificate (Corporate Proxy)
Save this as `C:/temp/extract-ca.ps1`:
```powershell
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
$certPath = 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem'
Set-Content -Path $certPath -Value $builder.ToString() -Encoding ASCII
Write-Host "Staged corp-root-ca.pem"
```
Run: `powershell.exe -ExecutionPolicy Bypass -File C:/temp/extract-ca.ps1`
### 2. Generate TLS Certificates
```bash
cd certs
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout server.key -out server.crt -days 365 \
-subj "//CN=localhost"
```
Or use the provided script:
```bash
./scripts/generate-dev-certs.sh
```
### 3. Configure Environment
Copy and edit `.env`:
```bash
cp .env.docker-dev .env
# Edit if needed - defaults work for local dev
```
### 4. Build Images
```bash
docker compose build
```
**Note:** First build takes 10-15 minutes due to:
- npm install (~5 min with Zscaler proxy)
- Frontend Vite build (~2 min)
- Python dependencies (~3 min)
Subsequent builds use Docker layer cache and are much faster.
### 5. Start Services
```bash
docker compose up -d
```
Check status:
```bash
docker compose ps
docker compose logs -f api
```
### 6. Initialize Database
```bash
# Run migrations
docker compose exec api flask db upgrade
# Install and enable all plugins
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
```
The `apply-profile` command:
- Installs all 17 plugins
- Registers them in `instance/plugins.json`
- Leaves them disabled by default
- Enables them via Settings UI as needed
### 7. Reset Admin Password
```bash
# Generate hash for "admin" password
docker compose exec api python -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Update database
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
Or use quick method (sets password to `admin`):
```bash
docker compose exec db mysql -u root -p shopdb_flask -e "
UPDATE users SET
passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';"
```
## Import Production Database
```bash
# Copy SQL dump to container
docker cp latest.sql shopdb-api-1:/tmp/
# Import
docker compose exec db mysql -u root -p shopdb_flask < /tmp/latest.sql
# Run migrations to bring schema up to date
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```
## Common Commands
### View Logs
```bash
docker compose logs -f api
docker compose logs -f db
```
### Restart Services
```bash
docker compose restart api
docker compose down && docker compose up -d
```
### Rebuild After Code Changes
```bash
docker compose build api
docker compose up -d api
```
### Access Container Shell
```bash
docker compose exec api bash
docker compose exec db bash
```
### Run Flask Commands
```bash
docker compose exec api flask plugin list
docker compose exec api flask db current
docker compose exec api flask routes
```
### Backup & Restore
```bash
# Trigger backup
docker compose -f docker-compose.backup.yml up backup
# View backups
docker compose exec api flask plugin list
# (Accessible via Settings > Backups in UI)
# Restore (via UI at Settings > Backups > Select backup > Restore)
```
## Troubleshooting
### "npm ci failed" during build
**Cause:** Zscaler proxy blocking npm registry.
**Fix:**
1. Extract Zscaler certificate (see above)
2. Ensure `corp-root-ca.pem` exists in repo root
3. Rebuild: `docker compose build --no-cache`
### "database is unhealthy"
**Cause:** MySQL container not ready or port conflict.
**Fix:**
```bash
# Check logs
docker compose logs db
# Check if port 3306 is already in use
netstat -an | grep 3306
# If conflict, edit .env and set MYSQL_PORT=3307
```
### "Access denied for user 'shopdb'"
**Cause:** Database credentials mismatch.
**Fix:** Check `.env` file - `MYSQL_PASSWORD` must match `DATABASE_URL`.
### "Could not connect to database"
**Cause:** Database service not started or wrong hostname.
**Fix:**
```bash
# Check db is running
docker compose ps
# API must use hostname 'db' not 'localhost'
# In docker-compose.yml: DATABASE_URL=...@db:3306/...
```
### "ERR_CERT_AUTHORITY_INVALID" in browser
**Expected:** Self-signed certificate warning.
**Fix:** Click "Advanced" > "Proceed to localhost (unsafe)" - this is normal for dev.
Or generate proper cert:
```bash
# Add your machine's hostname to cert
openssl req -x509 -newkey rsa:2048 -nodes \
-keyout certs/server.key -out certs/server.crt -days 365 \
-subj "//CN=$(hostname)" \
-addext "subjectAltName=DNS:localhost,DNS:$(hostname),IP:127.0.0.1"
```
### "Plugin X is already installed" but not visible
**Cause:** Plugin installed but not enabled.
**Fix:**
```bash
docker compose exec api flask plugin enable X
docker compose restart api
```
### Build is very slow
**Expected:** First build with Zscaler proxy takes 10-15 minutes.
**Speed up:**
- Subsequent builds use cache (much faster)
- If rebuilding often, comment out `--no-cache` flag
- Build frontend separately on host, copy dist/ into image
## Production Deployment Notes
For production:
1. **Change all secrets** in `.env`:
- `SECRET_KEY`
- `JWT_SECRET_KEY`
- `MYSQL_ROOT_PASSWORD`
- `MYSQL_PASSWORD`
2. **Use proper TLS certificates** (not self-signed):
- Place in `./certs/server.crt` and `./certs/server.key`
- Or use reverse proxy (nginx/IIS) for TLS termination
3. **Set `FLASK_DEBUG=0`** in production
4. **Configure backups**:
- Set up cron job: `0 2 * * * cd /path/to/shopdb && docker compose -f docker-compose.backup.yml up backup`
- Configure SMB or S3 destination in Settings > Backups
5. **Set `SHOPDB_ENV_LABEL`** to site name (e.g., "WEST-JEFFERSON")
6. **Review `docker-compose.yml`**:
- Adjust worker count based on load
- Configure proper `restart` policy
- Set resource limits if needed
## Architecture Notes
### Why HTTPS in Docker?
Secure context APIs (clipboard, geolocation) require HTTPS. Running over HTTP disables these browser features.
### Why Gevent Workers?
Endpoint agents long-poll `/api/endpoints/commands/next`. With threading, 200 agents = 200 threads = resource exhaustion. Gevent uses greenlets (cheap coroutines) so 2000+ connections per worker is feasible.
### Plugin Bootstrap
The entrypoint (`scripts/docker-entrypoint.sh`) automatically:
1. Runs database migrations
2. Checks if `instance/plugins.json` exists
3. If not, installs all available plugins (disabled)
4. Starts gunicorn
This ensures imported databases with plugin tables get matching plugin registry.
### Instance Data Persistence
The `instance_data` volume persists:
- `plugins.json` - Which plugins are enabled
- Uploaded files (floor plans, branding, photos, warranties, slides)
- Dell OAuth tokens
- Any plugin-specific state
**Important:** `docker compose down -v` deletes volumes - use plain `docker compose down` to preserve data.
## File Structure
```
flask-shopdb/
|-- docker-compose.yml # Main stack definition
|-- docker-compose.backup.yml # Backup service
|-- Dockerfile # Multi-stage build
|-- .env # Environment config (gitignored)
|-- corp-root-ca.pem # Zscaler cert (gitignored)
|-- certs/ # TLS certificates (gitignored)
| |-- server.crt
| +-- server.key
|-- scripts/
| |-- docker-entrypoint.sh # Container startup
| +-- bootstrap-plugins.sh # Plugin auto-install
+-- deploy/
+-- site-profile-universal.json # Full plugin list
```
## Next Steps
After setup:
1. Browse to https://localhost:5001
2. Login as `admin`
3. Go to Settings > Plugins
4. Enable plugins your site needs
5. Configure settings (CORS, backup destination, etc.)
6. Import data or start fresh
For local development with hot-reload, see `LOCAL-DEV-SETUP.md`.

372
docs/deployment/LOCAL-DEV-SETUP.md
View File

@@ -1,372 +0,0 @@
# ShopDB Local Development Setup (Windows)
Complete guide for setting up ShopDB Flask development environment on Windows.
## Prerequisites
- Python 3.13+
- Docker Desktop (for MySQL and testing)
- Git
- Node.js 20+ (for frontend development)
## Quick Start
```bash
# 1. Clone and enter repo
cd C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb
# 2. Create Python virtual environment
python -m venv venv
./venv/Scripts/activate
pip install -r requirements.txt
# 3. Start MySQL in Docker
docker compose up -d db
# 4. Configure environment
cp .env.docker .env
# Edit .env and set DATABASE_URL (see below)
# 5. Install and enable plugins
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe plugin install backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty
./venv/Scripts/flask.exe plugin enable <each-plugin-name>
# 6. Start Flask
./venv/Scripts/python.exe run_dev.py
```
Visit http://localhost:5001 and login with `admin` / `admin`.
## Detailed Setup
### 1. Database Setup
MySQL runs in Docker on **port 3307** (not 3306) to avoid conflicts with any local MySQL instance.
```bash
# Start MySQL container
docker compose up -d db
# Verify it's running
docker ps --filter name=shopdb-dev-db
# Should show: 127.0.0.1:3307->3306/tcp
# Test connection
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' -e "SELECT 1;"
```
### 2. Environment Configuration
Create `.env` file with:
```bash
# Database (port 3307 because Docker maps it there)
DATABASE_URL=mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev
# Flask
FLASK_ENV=development
FLASK_DEBUG=1
# Secrets (dev only)
SECRET_KEY=docker-dev-secret-key-change-for-production
JWT_SECRET_KEY=docker-dev-jwt-secret-change-for-production
# CORS
CORS_ORIGINS=http://localhost:5173,http://localhost:5001
# API Port
API_PORT=5001
# ShopDB Label
SHOPDB_ENV_LABEL=LOCAL-DEV
# Log Level
LOG_LEVEL=DEBUG
# Backup Storage (for local dev)
SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups
```
**Important:** Change `YOUR_USERNAME` to your actual Windows username.
### 3. Import Database (Optional)
If you have a production database dump:
```bash
# Import latest.sql
docker exec -i shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev < latest.sql
# Run migrations
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### 4. Plugin Installation
All 17 plugins must be installed before they can be enabled:
```bash
# Set DATABASE_URL for CLI commands
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
# Install all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin install $plugin
done
# Enable all plugins
for plugin in backups computers employees endpoints geenforce guides knowledgebase machines measuringtools network notifications printedparts printers slides tools usb warranty; do
./venv/Scripts/flask.exe plugin enable $plugin
done
```
**Why this is necessary:**
- Plugin state is tracked in `instance/plugins.json`, not the database
- When you import a prod database, it has plugin tables but no plugin registry
- Installing registers plugins in the JSON file
- Enabling makes them active (but requires app restart)
### 5. Reset Admin Password
If you imported a prod database and don't know the password:
```bash
# Generate new password hash
./venv/Scripts/python.exe -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
# Reset via MySQL
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'PASTE_HASH_HERE',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Or use the simpler method:
```bash
docker exec shopdb-dev-db mysql -u root -p'root-dev-password-not-for-production' shopdb_flask_dev -e "
UPDATE users
SET passwordhash = 'scrypt:32768:8:1\$B4LhpPrM6qxijInK\$b99b47a0bb5fd45761a1823f401f70a3ceccf71a5d38dc433378aa07426ef8eaf7e679ff3dd7b0ce9772600f83368ac642e9419b9beaa49f87483318bae34547',
failedlogins = 0,
lockeduntil = NULL
WHERE username = 'admin';
"
```
Password is now: `admin` / `admin`
### 6. Starting Flask
**Always use `run_dev.py`** (not `flask run` directly):
```bash
./venv/Scripts/python.exe run_dev.py
```
This script:
- Loads `.env` file with DATABASE_URL
- Sets all environment variables
- Starts Flask without auto-reloader (prevents env var loss)
- Binds to 0.0.0.0:5001
**Do NOT use:**
- `flask run` - Won't load `.env` properly
- `flask run --reload` - Loses environment variables
### 7. Frontend Development (Optional)
To develop the Vue frontend:
```bash
cd frontend
npm ci
npm run dev
```
Frontend dev server runs on http://localhost:5173 and proxies API requests to Flask on port 5001.
To point the proxy at a backend on another machine, set `VITE_DEV_API_TARGET`
in your shell instead of editing `vite.config.js`:
```bash
VITE_DEV_API_TARGET=https://10.0.0.5:5001 npm run dev
```
Certificate verification is off for an https target because those boxes
normally use a self-signed cert; set `VITE_DEV_SECURE=true` to enforce it.
For production build (served by Flask):
```bash
cd frontend
npm run build
# Outputs to frontend/dist/
```
## Troubleshooting
### "Can't connect to MySQL on 127.0.0.1:3306"
**Cause:** Flask is trying port 3306 instead of 3307.
**Fix:**
1. Make sure `.env` has `DATABASE_URL=...@localhost:3307/...`
2. Use `run_dev.py` to start Flask (NOT `flask run`)
3. If problem persists, restart Flask completely
### "Cannot import name '_backups_configured'"
**Cause:** Merge conflict between wj and wilm backup implementations.
**Fix:** This was fixed in commit `8a1fe0ada8` - pull latest wj branch.
### "Plugin X is not installed"
**Cause:** Plugin registry (`instance/plugins.json`) doesn't exist or is incomplete.
**Fix:** Run plugin installation commands (section 4 above).
### "Access is denied: '/backups'"
**Cause:** Backup system trying to use Linux path on Windows.
**Fix:**
1. Add to `.env`: `SHOPDB_BACKUPS_ROOT=C:/Users/YOUR_USERNAME/Documents/github/flask-shopdb/backups`
2. Create directory: `mkdir backups`
3. Restart Flask with `run_dev.py`
### Docker npm install fails (Zscaler proxy)
**Cause:** Corporate proxy intercepts HTTPS, npm doesn't trust certificate.
**Fix:** Extract Zscaler root CA certificate:
```powershell
# Run this PowerShell script
$request = [System.Net.HttpWebRequest]::Create('https://registry.npmjs.org/')
$request.Method = 'HEAD'
$request.Timeout = 10000
$response = $request.GetResponse()
$root = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($request.ServicePoint.Certificate)
$builder = [System.Text.StringBuilder]::new()
[void]$builder.AppendLine('-----BEGIN CERTIFICATE-----')
[void]$builder.AppendLine([System.Convert]::ToBase64String($root.RawData, 'InsertLineBreaks'))
[void]$builder.AppendLine('-----END CERTIFICATE-----')
Set-Content -Path 'C:\Users\270015376\Documents\github\flask-shopdb\corp-root-ca.pem' -Value $builder.ToString() -Encoding ASCII
```
Then rebuild Docker: `docker compose build --no-cache`
## Common Tasks
### Run Tests
```bash
pytest
pytest tests/test_specific.py
pytest -v -s # Verbose with print output
```
### Run Migrations
```bash
export DATABASE_URL="mysql+pymysql://root:root-dev-password-not-for-production@localhost:3307/shopdb_flask_dev"
./venv/Scripts/flask.exe db upgrade
```
### Create Migration
```bash
./venv/Scripts/flask.exe db migrate -m "Description of changes"
```
### Check Plugin Status
```bash
./venv/Scripts/flask.exe plugin list
```
### Enable/Disable Plugin
```bash
./venv/Scripts/flask.exe plugin enable slides
./venv/Scripts/flask.exe plugin disable slides
```
### Trigger Backup
```bash
# Via API (requires admin JWT token)
curl -X POST http://localhost:5001/api/shopdbbackups \
-H "Authorization: Bearer YOUR_TOKEN"
# Or via Docker
docker compose -f docker-compose.backup.yml up backup
```
## Architecture Notes
### Why Port 3307?
Docker MySQL maps internal port 3306 to host port 3307 to avoid conflicts with any local MySQL installation. This is configured in `docker-compose.yml`:
```yaml
ports:
- "127.0.0.1:3307:3306"
```
### Why run_dev.py?
Flask's `flask run` command uses an auto-reloader that spawns a subprocess. This subprocess loses environment variables loaded from `.env`. The `run_dev.py` wrapper:
1. Loads `.env` via python-dotenv BEFORE importing Flask
2. Uses `--no-reload` flag to prevent subprocess spawning
3. Ensures DATABASE_URL and other vars persist
### Plugin Architecture
- Plugins are Python packages in `plugins/` directory
- Plugin state tracked in `instance/plugins.json` (not database)
- Plugin tables tracked via Alembic migration chains
- Each plugin has own migrations in `plugins/<name>/migrations/`
### Backup System
- Designed for Docker compose deployment
- API queues backups via sentinel file (`.backup-requested`)
- Actual backup execution requires Docker sidecar container
- On Windows dev, API can queue but won't execute without Docker
## File Structure
```
flask-shopdb/
|-- .env # Local environment config (gitignored)
|-- run_dev.py # Local development launcher
|-- shopdb/ # Flask application
| |-- __init__.py # App factory
| |-- config.py # Configuration classes
| |-- core/ # Core models and APIs
| +-- plugins/ # Plugin manager
|-- plugins/ # Plugin packages
| |-- backups/
| |-- computers/
| +-- ...
|-- frontend/ # Vue 3 SPA
| |-- src/
| +-- dist/ # Production build (gitignored)
|-- instance/ # Runtime state (gitignored)
| |-- plugins.json # Plugin registry
| +-- ...
|-- migrations/ # Core Alembic migrations
|-- scripts/ # Deployment and backup scripts
|-- backups/ # Local backup storage (gitignored)
+-- tests/ # Test suite
```
## Git Workflow
- `main` branch - Published/public version (GitHub)
- `wj` branch - Production deployment (West Jefferson site)
- `wilm` branch - Development branch (upstream from benjamin.baker)
- `dev` branch - Ops deployment branch
**Never commit to `main` directly** - it's published to public wiki.
## Additional Resources
- [Backup Setup](scripts/backup/README.md)
- [Plugin Development](docs/PLUGIN-HOOKS.md)
- [ADRs](wiki/ADRs/)
- [API Docs](http://localhost:5001/api/docs) (when Flask is running)

47
docs/deployment/PRODUCTION-READINESS.md
View File

@@ -1,7 +1,29 @@
# ShopDB Production Readiness Assessment
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: historical
summary: The 2026-09-07 assessment of why deployment was hard. Its gaps became the roadmap; its recommendation was reversed.
loadwith: none
-->
# Production readiness assessment, 2026-09-07
**Last Updated:** 2026-09-07
**Branch:** wj (post-wilm merge)
> **This is a record, not guidance.** Read it for WHY the deployment story
> needed work, not for what to do. Two things have moved since:
>
> - **Its recommendation was reversed.** It recommends Windows/IIS for the
> reference site and Docker only for new ones. The decision since is the
> other way round: Docker is the path for every site and Windows/IIS is the
> legacy topology that will be deprecated.
> - **Its "critical path" became the roadmap.** Pre-built images in a registry,
> one-command setup, automatic plugin registration and a simpler backup mode
> are now the distributed-release item on [ROADMAP](../ROADMAP.md). Several
> of its gaps are already closed: the `migrate` service runs the schema, the
> plugin bootstrap and the seeders with no manual step, and
> `docker-compose.prebuilt.yml` removes npm from the image build.
>
> Current pages: [DEPLOY](../DEPLOY.md) to stand a site up,
> [DOCKER-PREBUILT-APPROACH](DOCKER-PREBUILT-APPROACH.md) to build it behind a
> proxy, [ROADMAP](../ROADMAP.md) for what is left.
## Executive Summary
@@ -196,15 +218,17 @@ The Docker deployment requires extensive manual setup, multiple services, and as
## Honest Recommendations
### For West Jefferson (Your Site)
### For the reference site
**Recommended:** Option 1 (IIS + Windows Services)
**Why:**
- You already have Windows infrastructure
*Reversed since. See the note at the top of this page.*
**Why (as argued at the time):**
- The site already had Windows infrastructure
- Simpler to operate
- Faster to deploy
- Your team knows IIS
- The team knew IIS
**Steps:**
1. Build Windows installer
@@ -262,6 +286,9 @@ https://your-server:5001
- `PRODUCTION-READINESS.md` - This file (meta/assessment)
- `scripts/backup/README.md` - Backup system (3 containers, restic)
*Closed since: the first two were folded into `DEVELOPMENT-SETUP` and `DEPLOY`,
and `INDEX.md` is now the single entry point this section asks for.*
**Problem:** Which one do I follow?
**Solution:** Create decision tree:
@@ -279,7 +306,7 @@ https://your-server:5001
## What We Did Today (Honest Assessment)
[OK] **Merged wilm->wj** - 98 commits, wilm features now in wj
[OK] **Merged the upstream development branch down** - 98 commits
[OK] **Fixed local dev** - `run_dev.py` works for Windows development
[OK] **Documented Docker** - Comprehensive but too complex
[OK] **Added backup docs** - System works but needs expertise
@@ -296,7 +323,7 @@ https://your-server:5001
- Gets people to working system fast
2. **Test Windows Installer**
- Build installer for wj branch
- Build the installer for the deployment branch
- Deploy to test Windows Server
- Document any issues
@@ -308,7 +335,7 @@ https://your-server:5001
4. **Set up CI/CD for image builds**
- GitHub Actions or Jenkins
- Build on every push to wj
- Build on every push to the deployment branch
- Push to internal registry
5. **Create `deploy.sh` script**

29
docs/deployment/README.md
View File

@@ -1,29 +0,0 @@
# ShopDB Deployment Documentation
Complete deployment guides for development, Docker, and production environments.
## Quick Start
**Local development**: `LOCAL-DEV-SETUP.md`
**Docker deployment**: `DOCKER-SETUP.md`
**VM deployment**: `DEPLOY-TO-VM.md`
## Docker Deployment
- `DOCKER-SETUP.md` - Complete Docker setup and configuration
- `DOCKER-PREBUILT-APPROACH.md` - Workaround for corporate proxy blocking npm in Docker
- `PRODUCTION-READINESS.md` - Production deployment checklist
## Development
- `LOCAL-DEV-SETUP.md` - Setting up local development environment (Flask + Vite)
## VM Deployment
- `DEPLOY-TO-VM.md` - Deploying the C# endpoint agent to test VMs
## Related Documentation
- `.claude/setup/` - MCP servers, VM configuration, and development tools
- `.claude/maintenance/` - Operational checklists and maintenance procedures
- `../shopdb-flask.wiki/ADR-004-deployment-topology.md` - Architecture decision record for deployment topology

136
docs/llms.txt Normal file
View File

@@ -0,0 +1,136 @@
# ShopDB - guide for an AI agent
ShopDB is a plugin-based asset-management system (PCs, printers, machines,
network devices, measuring tools, applications, knowledge base, USB,
warranties, backups and more) for manufacturing sites. Every site runs its
own instance.
This file is the machine entry point, served at `/api/docs/llms.txt`. It covers
the API, which is what most callers need, and routes everything else.
## Routing - read this before opening any document
`docs/INDEX.md` is the routing table. Rows are TASKS, each row gives the
smallest set of pages that answers it, and each row says how many lines that
costs. Find the row, read those pages, skip the rest. The corpus is far larger
than any one context window and almost none of it applies to a given question.
Every page opens with an HTML comment naming `purpose`
(tutorial/howto/reference/explanation), `audience`, `status` and a one-sentence
`summary`, plus `loadwith` for the pages it is not complete without. Filter on
that metadata before opening anything: `status: legacy` means true but
superseded (do not follow it for new work) and `status: historical` means a
finished migration or a proposal kept as a record, so neither is an answer to
"how do I do this today".
`docs/START-HERE.md` is the same routing for a human, and shorter.
Not an API question? Still start from the index rather than guessing at a
filename: its rows cover installing and running a site, loading its data, the
shop-floor PCs, and building on the platform. For "why is it built this way",
`docs/adr/README.md` says which decision governs the question and whether it
still holds. For writing a plugin, `docs/PLUGIN-HOOKS.md` is the contract and
everything else defers to it.
## Answering "how do I install or fix this server"
The people running these servers are often not Windows, IIS or Python
specialists. `docs/INSTALL-WINDOWS.md` (a new site, one offline installer
`.exe`) and `docs/OPERATE-WINDOWS.md` (restart, logs, backups, health) are
authoritative, and both ship in `docs/` inside the install directory on every
installed server.
Do NOT walk anyone through `docs/INSTALL-WINDOWS-IIS.md` for a new site. It is
the manual procedure kept for servers that predate the installer, and following
it produces a server the installer then refuses to upgrade.
Before diagnosing anything, ask for the output of `shopdb-admin.ps1 check -Json`
from the install directory (default `C:\shopdb-flask`). It reports version,
publishing method, IIS and pool state, HTTP reachability, database host and
reachability, Python version, installed plugins and errors, and it contains no
secrets. The other day-2 verbs are `status`, `start`, `stop`, `restart`, `logs`,
`open`, `backup`, `repair`, `catalog`, `sessions`, `plugins`, `add-plugin`,
`verify`, `uninstall`. `verify -Path <name>` answers "does this server carry
component X" from the on-box CycloneDX SBOM.
Python is 3.14 and the wheelhouse is locked to it; an upgrade against a venv
built by a different minor version is refused by design.
## Bulk-loading a site's data
Two routes, and the right one depends on what the site has:
- SPREADSHEET, no developer (the common case): `flask csv templates --out <dir>`
generates templates FROM THE LIVE SCHEMA, then `flask csv import --dir <dir>`
checks and `--commit` applies. Do NOT hand-write the templates - generate
them. `docs/CSV-IMPORT.md`.
- A SOURCE DATABASE to script against: the HTTP import API,
`docs/IMPORT-API.md`, with `docs/IMPORT-ADOPTION.md` for the loader around it.
User accounts are deliberately not CSV-importable.
## Base URL
Each site runs its own instance (ADR-004), so there is no single production URL:
it is whatever that site serves, e.g. `https://shopdb.example.net/shopdb`.
All API paths are under `/api` (e.g. `<base>/api/assets`). Dev: `http://localhost:5001`.
## Auth
Three schemes:
- **Bearer JWT** - most endpoints. Get one by logging in, or use a managed
Personal Access Token (PAT). Send `Authorization: Bearer <token>`.
- Login: `POST /api/auth/login` `{ "username": "...", "password": "..." }`
-> `data.access_token`. Refresh: `POST /api/auth/refresh`.
- PATs are minted in the UI (Settings > API Tokens); a *scoped* PAT is limited
to named permissions and suspends the admin bypass.
- **X-API-Key** - unattended/service endpoints (collector ingest, GE-Enforce
fetch). Send `X-API-Key: <managed-token>`. How a shop-floor PC gets its own
credential is `docs/COLLECTOR-AUTH.md`.
- **Public** - some read endpoints (e.g. printer install-list, employee search,
dashboards) need no auth.
Auth level per endpoint is in the OpenAPI `security` field: `bearerAuth`,
`apiKeyAuth`, or none. Admin-only and permission-gated routes both use bearer.
## Response envelope
JSON endpoints return `{ "status": "success", "data": <payload>, "meta": {...} }`.
Errors nest under `data`, NOT at the top level:
`{ "status": "error", "data": { "error": { "code": "...", "message": "...", "details": {...} } } }`
with an HTTP 4xx/5xx. Reading `message` or `code` off the root gives `undefined`.
Lists carry `meta.pagination.{page, perpage, total, pages}` - again nested, not
`meta.total`. A few feed endpoints (screensaver, some installer text formats)
return raw text/JSON without the envelope - noted per route.
## Common recipes
- Search everything: `GET /api/search?q=<term>` (multi-word = AND across words).
- List assets on the map: `GET /api/assets/map`.
- List a type: `GET /api/printers`, `/api/computers`, `/api/machines`,
`/api/network`, `/api/measuringtools` (paginated: `?page=&perpage=`). A
plugin that is disabled at that site serves none of its routes.
- Get one: `GET /api/printers/<id>` etc.
- Create (bearer): `POST /api/printers` `{assetnumber, windowsname, vendorid, ...}`.
- Reports: `GET /api/reports` (list), `GET /api/reports/pc-relationships` (PC<->machine).
- Printer installer data: `GET /api/printers/install-list` (public; add
`?format=text` for a pipe-delimited variant); `GET /api/printers/pc-default?machine=<n>`.
- Collector ingest (X-API-Key): `POST /api/collector/computers`.
- GE-Enforce: `GET /api/geenforce/manifest?pctype=<scope>`,
`GET /api/geenforce/payload/<sha256>`, `POST /api/geenforce/report`.
- Import (admin PAT, preserves timestamps with `X-Import-Mode`): `docs/IMPORT-API.md`.
## Conventions
- DB-mirrored params/fields use lowercase concatenated names (no underscores):
`locationid`, `vendorid`, `windowsname` - match them exactly.
- IDs in paths are integers.
- Plugin endpoints live under the plugin's prefix (`/api/<plugin>/...`).
## Full reference
- Machine spec: `GET /api/docs/openapi.json` (OpenAPI 3.1). Operation and
plugin counts live in `docs/PROJECT-MAP.md`, which is generated.
- Interactive: `GET /api/docs` (Redoc).
- Human reference: the same spec at `GET /api/docs`, and `docs/INDEX.md` for everything else.

7
docs/notes/endpoint-telemetry-and-ui-phases.md
View File

@@ -1,3 +1,10 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Working notes for the phased endpoint telemetry and PC Details UI effort. Not a contract.
loadwith: none
-->
# Endpoint telemetry and PC Details UI - phased working notes
Scratch/reference notes for a multi-phase effort. Not a contract document. When

14255
docs/openapi.json
View File

File diff suppressed because it is too large Load Diff

29
docs/proposals/README.md Normal file
View File

@@ -0,0 +1,29 @@
<!-- docmeta
purpose: reference
audience: developer
status: current
summary: What each proposal designed, whether it was built, and which ADR decided it.
loadwith: none
-->
# Proposals
A proposal is a design written before the work, kept afterwards as the record of
why the thing looks the way it does. A proposal is NOT a decision record: where
a proposal and an ADR disagree, the ADR wins.
**None of these is current guidance.** Read one to understand why something
exists, not to learn how it works today - for that, read the page named in the
right-hand column.
| Proposal | What it designed | Built? | Read instead |
|---|---|---|---|
| [dashboard-live-fleet](dashboard-live-fleet.md) | A dashboard that shows what needs doing instead of row counts | Yes | [ADR-010](../adr/ADR-010-frontend-plugin-hooks.md) for the contract it introduced |
| [device-identity](device-identity.md) | One durable identity for the device a PC drives, ending in the registry | Phase 1 built (the collector `deviceid` field). Phases 2 and 3 not started | The proposal itself, for the remaining phases |
| [ge-enforce-plugin](ge-enforce-plugin.md) | GE-Enforce manifests as ShopDB data, served over HTTP | Yes | [ADR-012](../adr/ADR-012-geenforce-manifest-ownership.md), then [GE-ENFORCE](../GE-ENFORCE.md) |
| [printedparts-plugin](printedparts-plugin.md) | A 3D-printed-parts storefront, bin labels and a kiosk | Yes, as `plugins/printedparts/` | [PLUGIN-LAB-PRINTEDPARTS](../PLUGIN-LAB-PRINTEDPARTS.md) |
| [printer-assignment](printer-assignment.md) | Printers as a property of the machine, installed by the PC | Yes, v0.11.3 and v0.12.0 | [PRINTER-INSTALLER](../PRINTER-INSTALLER.md), [COLLECTOR-INTEGRATION](../COLLECTOR-INTEGRATION.md) |
A proposal that is adopted says so in its own status line, and names what
adopted it. A proposal nobody built stays here anyway: the alternatives it
rejected are the part worth keeping.

179
docs/proposals/dashboard-live-fleet.md Normal file
View File

@@ -0,0 +1,179 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 case for a worst-first dashboard queue. Built; kept as the record of why.
loadwith: none
-->
# Proposal: a dashboard that shows the fleet, not the row count
- Status: IMPLEMENTED (a proposal, not a decision record - the
contract it introduced is ADR-010 and contract 0.19.0)
- Date: 2026-08-11
- Author: ShopDB maintainers
- Relates to: ADR-010 (frontend plugin hooks), ADR-013 / ADR-014 (lean per-site builds), ADR-006 (collector contract), ADR-012 (GE-Enforce manifest ownership)
## The problem
The dashboard reports totals: assets, machines, PCs, network devices, printers,
measuring tools. Those numbers are true every day and actionable on none of
them. Nobody opens the page to learn there are 417 assets.
Meanwhile the system already knows things worth acting on and shows them
nowhere:
- a bay returned 500 to every collector report for a day and a half, and the
only evidence was a log file on the PC
- eleven part markers' backups were overwriting each other for weeks
- a PC can stop reporting entirely and nothing says so
- GE-Enforce records enforcement failures, stale manifests and repeated
self-heals, and none of it surfaces outside a report modal
Every one of those was found by someone going to look. The dashboard should be
where they find you.
## What already exists
Two discoveries shaped this proposal.
**The plugin mechanism is built and dead.** `BasePlugin.get_dashboard_widgets`
exists, `GET /api/dashboard/widgets` aggregates across enabled plugins, skips
disabled ones, isolates a broken plugin in production and sorts by position.
Five plugins - notifications, network, machines, computers, printers - already
declare widgets.
**Nothing consumes it.** The frontend never calls the endpoint, and not one of
the declared components (`PrinterStatusWidget`, `ComputerStatusWidget`,
`MachineStatusWidget`, `NotificationsWidget`, `NetworkStatusWidget`) exists.
The declarations point at components that were never written.
So this is less "build a dashboard" than "finish one", plus a contract fix.
**The data is already collected.** Nothing here needs a new collector field:
| Signal | Source |
|---|---|
| PC reporting / silent | `computers.lastreporteddate` |
| Machine has a controlling PC | `controls` links, `collector:machine` |
| Backed up, stale, never | `backuprevisions` per asset and kind |
| Enforcement failed | `manifestenforcementreports.status`, `failedcount` |
| Which entry failed, and why | `manifestenforcementresults.action`, `exitcode`, `message` |
| Stale manifest | report `appliedversion` versus the scope's published version |
| Drift that will not stick | repeated `selfhealed` results for one entry |
| Enforcer version spread | `manifestenforcementreports.enforcerversion` |
| Installed software and versions | `computerinstalledapps` |
| Toner critical / low | printer supplies, Zabbix-fed |
| Warranty expiring | warranty plugin |
| Mis-numbered bays | the `check-shared-machines` query |
## Proposed change to the widget contract
The existing contract names a **component per widget**. That does not scale and
does not survive a lean build: every widget needs a bespoke Vue component, and a
plugin-provided component has to be staged into the frontend bundle to exist at
all.
ADR-010 already solved this for asset panels: a plugin declares `render: 'list'`
or `'keyvalue'` or `'tabs'` plus a `map` of fields, and a GENERIC core component
renders it. Do the same here. A plugin declares data and shape; core owns the
rendering.
```python
{
'id': 'geenforce-failures',
'title': 'Enforcement failures',
'endpoint': '/api/geenforce/dashboard/failures',
'render': 'exceptions', # generic renderer
'severity': 'critical', # critical | warning | info
'permission': 'geenforce.view',
'position': 10,
'empty': 'hide', # hide | line
'map': {
'title': 'hostname',
'detail': 'entryname',
'meta': [{'key': 'message'}, {'key': 'exitcode'}],
'link': '/pcs/{computerid}',
},
}
```
Three renderers cover everything listed above:
- `exceptions` - a list of things that are wrong, each linking to the thing
- `metric` - a single number with a trend or threshold, for the cases where the
count IS the story (toner critical: 3)
- `list` - recent items, for notifications and KB adds
## Principles
**An exception board, not a stat board.** The default view is what needs a
person today. A count appears only where the count is the story.
**Empty cards must shrink.** A card that says "nothing wrong" every day trains
people to stop reading the page. That is exactly how `ntlars-backup.log` reached
3,234 lines with 17 that mattered. `empty: 'hide'` is the default; `'line'` for
the few where absence is itself news.
**Every row links to the thing.** A dashboard that says three printers are low
without linking to them is a worse version of a report.
**Cards respect RBAC.** A card declares a `permission`; a user without it never
sees the card or its endpoint. Toner levels are not sensitive, but "which PCs
expose VNC" is, and the dashboard must not become a way around role gating.
**Thresholds are settings with neutral defaults** (ADR-015): quiet window, toner
percentage, warranty horizon. Every site will disagree with the numbers.
**One endpoint per card, lazily loaded, failing independently.** A hung Zabbix
call must not blank the page.
## Wave one
Cards whose data is already reliable, ordered by how actionable they are:
1. **Enforcement failures** - current reports with `status = failed`, showing
the failing entry and exit code.
2. **PCs not reporting** - `lastreporteddate` beyond the quiet window, and PCs
with no GE-Enforce report at all, which is a different and worse case.
3. **Backups missing or stale** - machines with no revision of a kind their
PC type should produce, and revisions older than the interval.
4. **Toner critical, then low** - links to the printer, then to its admin page.
5. **Warranties expiring** - within the configured horizon.
6. **Mis-numbered bays** - `check-shared-machines`, promoted from a CLI command
nobody will remember to run.
Then, lower and quieter: stale manifest versions, repeated self-heals, enforcer
version spread, active notifications, recent application and KB additions.
## Wave two: desired versus observed
The highest-value card and the most work. The manifest declares what a PC type
should have; `computerinstalledapps` records what it does; `filters.py` already
decides which entries apply to a given PC. Diffing those answers "which bays are
missing something they should have" - which nothing in the system answers today,
and which is the thing GE-Enforce exists to guarantee.
Deferred to wave two because it needs the resolver wired into a query path, and
because wave one needs nothing new.
## Risks
**Aggregate queries on the most-visited page.** These are fleet-wide scans on
every load. Cache 30-60 seconds, and log slow cards - a lazily-loaded card can
be slow for months before anyone mentions it.
**Cards become the new noise.** Six cards that are always empty are as useless
as one number that never changes. If a card is empty for a month, delete it.
**Plugin gating must be real.** A lean site without `printers` must render no
toner card. The existing endpoint already skips disabled plugins; the frontend
must not hardcode a card list alongside it.
## Non-goals
- Per-user dashboard customisation. Later, if asked for.
- Historical charting. This is a "what needs doing now" board; trends belong in
reports.
- Replacing the shopfloor TV dashboard, which is a different audience with
different needs.

235
docs/proposals/device-identity.md Normal file
View File

@@ -0,0 +1,235 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Plan for one durable identity for the device a PC drives. Phase 1 is built here; phases 2 and 3 are not started.
loadwith: none
-->
# Proposal: one identity for the device a PC drives, ending in the registry
Status: PHASE 1 BUILT AND IN THIS TREE - `plugins/computers/plugin.py` declares
`deviceid` on the computers collector schema and prefers it over the older
per-family fallbacks. Phases 2 and 3 not started.
Author: planning session 2026-08-20.
## 1. What this is
A shop-floor PC often drives something: a CMM, a Keyence, a Genspect, a
wax-trace unit, a part marker. ShopDB needs to know WHICH one, and the answer
has to survive the PC being replaced.
Today that answer is assembled from whatever each bay family happens to have,
and two of the three sources are derived from the PC itself, so they do not
survive a swap. This proposal converges them on one identity, moves it from
files to the registry, and does it in three phases so no bay is stranded.
## 2. Why PC-derived identity fails
Two mechanisms stand in for a real identity today:
- **Reuse**: look for a prior collector link from THIS PC asset.
- **Mint**: build `<PC assetnumber>-<SUFFIX>`, and a PC's asset number is its
hostname.
Both survive a re-image. Neither survives a PC SWAP. A new hostname is a new PC
asset with no prior link and a predicted number that has never existed, so the
same physical device gets a SECOND record while the first keeps its config and
its backup history under a dead PC's name.
That is not hypothetical: minting from the hostname left 43 legacy `MT-####`
tools shadowed by minted twins, three records deep in places.
## 3. The rail that must not be crossed
`machinenumber` answers "WHICH BAY is this" and is what GE-Enforce
`TargetMachineNumbers` gates on. The device identity answers "WHICH DEVICE is
this". They are different facts and must stay in different fields.
Putting a device id into the machine-number channel would silently stop every
bay-gated manifest entry from matching - a failure that looks like nothing
happening, on every cycle, with no error anywhere.
## 4. Where identity comes from today
| Bay family | Identity source | Survives a PC swap |
|---|---|---|
| DNC / collections | `MachineNo` in the eDNC/NTLARS registry | yes |
| CMM | `C:\Enrollment\cmm\cmmid.txt` | yes, but see below |
| Keyence, Genspect, wax-trace | nothing, before phase 1 | no |
| Part markers | nothing, before phase 1 | no |
**`cmmid.txt` is not only an identity file.** It is a PATH KEY, and that is why
it cannot simply be retired:
| Consumer | Use |
|---|---|
| `resolve-cmm-bay-config.ps1` | writes it |
| `Verify-And-Heal-Staging.ps1` | stages `installers-post\cmm\backups\<cmmid>` |
| `09-Setup-CMM.ps1` | locates the bay's staged backup set |
| `Restore-CMM.ps1` | restores from `<BackupRoot>\<cmmid>\`, same-bay by design |
| collector | reads it as `machineNo` |
Only the last row is an identity use. The rest address directories by it.
It is also doing the job badly: for a CMM bay it reports the INSTRUMENT's id in
the `machinenumber` channel, and the server then resolves it back to a measuring
tool at step 4 of its own fallback chain. That round trip is what phases 2 and 3
remove.
## Phase 1 - one field, one file (DONE)
The collector accepts a single `deviceid`, read from `C:\Enrollment\asset-id.txt`
and resolved BEFORE anything PC-derived, for every device family.
- One file for every bay with no `MachineNo` registry to identify it.
- It names the asset and NOT its type: the pc-type already says what kind of
device it is, through `SUBORDINATE_DEVICE_MAP`, so a device type declared
later through `subordinatedevice_<pctype>` needs no new file and no client
change.
- Wire field is `deviceid`, not `assetid`, because `assetid` is already this
contract's RESPONSE field for a PC's integer primary key.
- `measuringtoolid` (0.12.0) is accepted as an alias and the older file is still
read, so a bay staged in that window keeps reporting.
- Guards: an unresolvable value warns rather than minting a phantom, and a value
resolving to the WRONG KIND of asset is refused with the asset named.
Nothing was removed. Every previous fallback still runs, one place lower.
## Phase 2 - imaging writes it, and `cmmid.txt` stops being an identity
Repository: the PXE/imaging repo, not this one. **Needs deploying to the live
server, not only committing** - the live boot image is authoritative.
1. `resolve-cmm-bay-config.ps1` writes `C:\Enrollment\asset-id.txt` ALONGSIDE
`cmmid.txt`. Two files, two questions: the identity, and the backup-set key.
`cmmid.txt` keeps every path use it has today.
2. Every other device pc-type writes `asset-id.txt` at imaging from its bay
config. Bays with no device write nothing, and the file's absence stays
meaningful.
3. The collector demotes `cmmid.txt` to last place in the machine-number chain
and stops being the intended path for CMM identity. It is still read, so a
grandfathered bay that never gets re-imaged keeps reporting.
4. The value written is the device's `assetnumber` as ShopDB holds it. Where a
CMM's asset number is already its bay id (`CMM3`), phase 2 is a no-op in
content and a change only in which file carries it.
Exit criterion: a report from a re-imaged bay of each device family carries
`deviceid`, and no bay of those families relies on the `cmmid.txt` fallback.
The collector response warnings are the check: an adopted device is silent, a
minted one is not.
## Phase 3 - identity moves to the registry
`HKLM:\SOFTWARE\GE\ShopDB\DeviceId`.
This is the destination because the registry survives the profile cleanup and
disk hygiene that eats `C:\Enrollment`, is readable as SYSTEM with no file ACL
to get wrong, and is already how a DNC bay is identified. It needs no new key
and no new read path: the collector already opens
`HKLM:\SOFTWARE\GE\ShopDB` (and the `WOW6432Node` variant) for `BaseUrl` and
`ApiToken`, and `Install-GEEnforce.ps1` already writes that key.
**One value, every family.** `DeviceId` is a single registry value, not a key
per device type. The client sends one field and the pc-type decides which sync
consumes it, exactly as `asset-id.txt` works in phase 1, so:
| Bay | `DeviceId` holds | Replaces |
|---|---|---|
| CMM | the instrument's `assetnumber` | `cmmid.txt` as IDENTITY only |
| Keyence | the instrument's `assetnumber` | nothing; had none |
| Genspect | the instrument's `assetnumber` | nothing; had none |
| wax-trace | the instrument's `assetnumber` | nothing; had none |
| part marker | the marker's `assetnumber` | minting from the hostname |
A device family added later through `subordinatedevice_<pctype>` is covered on
the day it is declared, with no client change and no new registry value.
Resolution order after phase 3:
1. `HKLM:\SOFTWARE\GE\ShopDB\DeviceId`
2. `C:\Enrollment\asset-id.txt`
3. `C:\Enrollment\measuringtool-id.txt` (0.12.0 window)
4. `C:\Enrollment\cmm\cmmid.txt` as `machineNo` (grandfathered CMM bays)
Then, and only then, the file branches can be deleted one at a time, newest
first, each when no bay has reported through it for a full inventory cycle.
**`cmmid.txt` survives phase 3 regardless**, as the CMM backup-set path key. It
is not on the deletion list; only its identity use is.
### PXE and GE-Enforce prefer it too, not just the collector
Phase 3 is not only "the collector reads a registry value". All three writers
and readers move together:
| Component | Change |
|---|---|
| PXE / imaging | writes `DeviceId` at enrollment instead of (or as well as) the file |
| GE-Enforce | resolves device identity registry-first, mirroring how it already resolves the machine number |
| collector | reads the registry ahead of the file |
**GE-Enforce already made this exact decision for the machine number, and wrote
down why.** `Install-FromManifest.ps1` resolves `TargetMachineNumbers` gating
from the eDNC registry BEFORE `C:\Enrollment\machine-number.txt`, because:
> the imaging-time `machine-number.txt` is written ONCE by `startnet.cmd` at the
> PXE menu and is NOT updated on reassignment, so it goes stale.
**`asset-id.txt` inherits that defect exactly.** It is written once at imaging
and nothing updates it when a device is swapped, moved between bays, or
replaced. The bay then reports an identity that was true on imaging day, and the
collector - which now trusts that value FIRST, ahead of every other source -
adopts the wrong device with no warning, because the value still resolves.
That makes phase 3 a correctness fix rather than a tidy-up. A file that only
imaging writes cannot be the top of a resolution order for something that
changes during a PC's life.
It also implies a reassignment path, the way `Set-MachineNumber` /
`Update-MachineNumber` exist for the bay number: something that rewrites
`DeviceId` when a device is swapped, and migrates any per-device state with it.
Without that, phase 3 moves the staleness from a file to a registry value rather
than curing it.
### Doing phase 3 early is cheaper than doing it late
If the registry read lands before imaging is changed, phase 2 can write the
REGISTRY for new bays and skip the file era entirely for them. The file then
exists only for bays staged in the window between phase 1 and phase 2, and the
eventual cleanup is deleting a code branch rather than sweeping the fleet.
The cost of adding the read now is one branch in the client and one row in the
docs. The cost of adding it after every bay has a file is a migration.
## 5. What this does not change
- `machinenumber` and its GE-Enforce gating. Untouched in all three phases.
- A part marker still files `partof` the operation from `machinenumber`. The
device identity says WHICH marker the PC controls, not which operation that
marker belongs to.
- Minting. It stays as the last resort for a bay that names nothing, because a
device nobody declared is still better recorded than not recorded. Anything
minted should be treated as a placeholder to be reconciled.
## 6. Before phase 2: find what is already duplicated
Phase 1 stops NEW duplicates. It does not merge the ones a PC swap has already
created. Before imaging starts writing files, run a read-only query for device
assets whose controlling PC no longer exists, or whose asset number carries a
hostname that is not a current PC. That list is the reconciliation backlog, and
it is much easier to read before the fleet starts reporting new identities over
the top of it.
## 7. Rollback
Each phase is independently reversible because nothing is removed until the
phase after it proves the replacement:
- Phase 1: stop sending `deviceid` and every previous fallback still runs.
- Phase 2: stop writing `asset-id.txt` and bays fall back to what they used
before.
- Phase 3: stop writing the registry value and bays fall back to the file.
The only irreversible step is deleting a fallback branch, which is deliberately
outside all three phases.

656
docs/proposals/ge-enforce-plugin.md Normal file
View File

@@ -0,0 +1,656 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-07 design for GE-Enforce manifests as ShopDB data. Adopted by ADR-012 and built.
loadwith: none
-->
# Proposal: GE-Enforce as a shopdb plugin
Status: ACCEPTED / built - see ADR-012 and plugins/geenforce/.
Author: planning session 2026-07-12.
## 1. What this is
Today GE-Enforce is a PowerShell manifest engine that reads per-PC-type
`manifest.json` files off an SMB share (`\\shopdb.example.net\
shared\dt\shopfloor\`). Each logon, a scheduled task running as SYSTEM mounts
the share, reads the manifest for the machine's PC type, and installs or
self-heals apps, files, drivers, registry values, and scripts. A parallel
`preinstall.json` runs the same schema once at imaging.
This proposal turns the *manifest* into shopdb data: the authoritative manifest
lives in the shopdb database, is edited through the shopdb UI (an expansion of
`/settings/pctypemapping`), and is served to clients over HTTP as JSON. The
*payloads* (MSI/EXE/PS1/config bytes) stay on SMB, on HTTP, or both, referenced
by URL/path from the manifest rows. GE-Enforce.ps1 changes from "read a file on
W:" to "GET a manifest from shopdb, then fetch each payload from wherever the
row says."
The result: managing imaging PC types, their apps, scripts, files, registry
rules, and version gates becomes a first-class shopdb feature instead of hand-
edited JSON on a file share.
## 2. Why it fits shopdb
- shopdb already models the fleet (the collector ingests every PC's hostname,
pctype, installed software, versions). Making shopdb *also* own what SHOULD be
installed closes the loop: desired-state (manifest) and observed-state
(collector) live in one system and can be diffed.
- `/settings/pctypemapping` already maps `gea-shopfloor-*` PC types to
`ComputerType`. That page becomes the entry point for full imaging-PC-type
management.
- The plugin contract (per-plugin models, migrations, API prefix, settings
cards, collector hooks) is exactly the shape this needs.
- ADR-004 (per-site instances) matches: each site's shopdb owns each site's
manifest. No multi-tenant complication.
## 3. Grounding: the real manifest schema
Source of truth for these field names (do not invent others):
- Schema: `<imaging-share>/SHOPDBHOST-v2/shared/dt/shopfloor/_meta/manifest-schema.json`
- Engine: `<imaging-share>/common/lib/Install-FromManifest.ps1`
- Dispatcher: `.../shopfloor/common/GE-Enforce.ps1`
- Architecture: `pxe/docs/ge-enforce-v2-architecture.md`
A manifest is `{ "Version": str, "_comment": str, "Applications": [entry, ...] }`.
Only `Name` and `Type` are required per entry.
### Per-entry fields (complete set)
Identity / action:
- `Name` (required, unique, also the status-key `<scope>/<Name>`)
- `Type` (required): one of `MSI EXE CMD BAT PS1 INF File Registry`
- `_comment` (documentation, heavily used in practice)
Type-specific payload references (sparse; depends on Type):
- MSI/EXE/CMD/BAT/INF: `Installer` (relative path) + `InstallArgs`
- PS1: `Script` (relative path, falls back to `Installer`) + `Args`
- File: `Source` (relative) + `Destination` (absolute on-PC path)
- Registry: `RegPath` + `RegName` + `RegValue` + `RegType`
(`RegType` in `String DWord QWord MultiString ExpandString Binary`)
- Optional `LogFile`, `WaitTimeoutSec` (EXE hang kill), `InUseCheck`
Detection (decides whether the action fires / self-heals):
- `DetectionMethod`: one of
`Registry File FileVersion Hash MarkerFile ValueMatches pnputil Always`
- `DetectionPath`, `DetectionName`, `DetectionValue`, `DetectionPattern`
- Note: `DetectionValue` is method-dependent - SHA256 for Hash, a 4-part
version for FileVersion, a registry value for Registry, ignored for
Always/File. Same column, different meaning per method.
- No `DetectionMethod` = always installs.
Targeting filters (all ANDed; each is multi-value):
- `PCTypes` (array; `"*"` = all; alias graph expands old<->new names)
- `PCSubTypes` / subtype via `<pctype>-<subtype>` values
- `TargetHostnames` (array; exact + `-like WJS-*` wildcards)
- `TargetMachineNumbers` (array; per-bay)
- `_CmmVersion` (scalar; per-entry PC-DMIS version gate, needs lib >= 2.6)
Nested:
- `InUseCheck`: `{ Behavior, Processes: [{Name, ExePath, GracefulCloseTimeoutSec}] }`
Behavior in `Defer CloseAndReopen ForceClose ScheduleForReboot`
Parsed-but-inert today (model them, mark inert):
- `ApplyMode` (`Nightly Immediate ImmediateReboot`), `UpdateWindow` (`HH:MM-HH:MM`)
Preinstall-only extras (phase discriminator):
- `PreEnrollment`, `KillAfterDetection`, `PCTypesStrict`, `_pcTypesNote`
### Load-bearing behaviors the model must preserve
1. **Array order IS execution order.** Config-restore entries are deliberately
placed AFTER their vendor installer so a mid-cycle overwrite heals the same
cycle (eMxInfo.txt after eDNC; udc_webserver_settings after UDC). We MUST
store an explicit per-scope `sortorder`, not a set.
2. **PCTypes alias graph** is many-to-many old<->new names resolved by set
intersection, with a `PCTypesStrict` escape hatch. Not a simple FK.
3. **Polymorphic entry by Type** - sparse column set per type. DECISION: one
wide `manifestentries` table with an `entrytype` discriminator column and
nullable per-type columns. NOT SQLAlchemy STI subclasses, NOT a JSON blob.
Justification (section 4): the whole fleet is ~64 entries, so sparse columns
cost nothing; real columns get validated, indexed, field-diffed, joined
against collector data, and read in plain SQL by an IT tech - a JSON blob
hides all of that, and class-per-type STI is expert ceremony for no gain. A
`validate()` that switches on `entrytype` (mirroring the engine's own
`switch ($App.Type)`) is ~40 obvious lines.
4. **Two manifest phases** - runtime (self-heal, per logon) and preinstall
(once at imaging) share the schema. One table with a `phase` discriminator.
## 4. Data model (new `geenforce` plugin)
Per-plugin Alembic chain (ADR-008). Tables (lowercase concatenated per naming
convention). Sizing that shapes every decision here: the real fleet is 10
runtime scopes = 43 entries, plus 1 preinstall manifest = 21 entries, so ~64
rows total. That smallness is why this stays deliberately low-tech (one wide
table, JSON-document snapshots, no row-mirroring) - the design target is an
average site IT tech maintaining it, not a specialist.
- `manifestscopes` - one row per imaging PC type / scope.
- `scopeid` PK
- `scopename` (e.g. `gea-shopfloor-cmm`)
- `phase` enum (`runtime` | `preinstall`)
- UNIQUE (`scopename`, `phase`), NOT `scopename` alone: `common` exists in
runtime, and a scope name can appear in both phases. Note the phases are
shaped differently - runtime is many per-pctype scopes (one manifest file
each), preinstall is ONE flat manifest gated internally by `PCTypes`, so
preinstall is modeled as a single `phase=preinstall` scope, not per-pctype
scopes.
- `computertypeid` FK -> `computertypes` (this REPLACES the thin
`pctypemap_<pxetype>` setting; the mapping becomes a column here).
Runtime-scope only; null for the preinstall scope.
- `measuringtooltypeid` FK -> `measuringtooltypes`, nullable (metrology
scopes: what device this scope implies; keeps imaging + collector agreed,
see section 11).
- `manifestversion` (string, mirrors manifest `Version`)
- `description`, `isactive`
- `iscommon` bool (the `common/` fleet-wide scope)
- `manifestentries` - one row per Applications[] entry (the working/draft copy).
- `entryid` PK, `scopeid` FK
- `sortorder` int (preserves array order; the ordering contract)
- `name`, `entrytype` (MSI/EXE/.../Registry), `comment`
- payload columns (nullable, per type): `installer`, `installargs`,
`scriptpath`, `scriptargs`, `sourcepath`, `destination`,
`regpath`, `regname`, `regvalue`, `regtype`
- `payloadsource` enum (`smb` | `http` | `inline`) + `payloadref`
(see section 5)
- `payloadsha256` - integrity hash of the payload bytes, INDEPENDENT of the
detection method. Mandatory for `http`/`inline` payloads; optional for
`smb`. Do NOT reuse `detectionvalue` for this - `detectionvalue` is a
SHA256 only when `detectionmethod = Hash`; an MSI with `Registry`/
`FileVersion` detection has no payload hash, so an HTTP fetch would
otherwise run unverified bytes (see section 5).
- `regvalue` stores the RAW JSON literal (`1` vs `"1"`) and is emitted
verbatim on export. `RegValue` is untyped in the manifest schema and real
entries carry numbers; the engine string-coerces for `ValueMatches` but
`Set-ItemProperty -Type DWord` cares, so preserve the literal.
- detection columns: `detectionmethod`, `detectionpath`, `detectionname`,
`detectionvalue`, `detectionpattern`
- gates: `cmmversion`, plus child tables for the multi-value filters
- control: `logfile`, `waittimeoutsec`, `applymode`, `updatewindow`
(`applymode`/`updatewindow` are parsed-but-INERT in the engine today; the
UI must label them "not yet enforced" so a tech does not trust a dead gate)
- preinstall flags: `preenrollment`, `killafterdetection`, `pctypesstrict`
- `isactive`
- `manifestpublishedversions` - immutable published snapshots, SIMPLIFIED to
freeze the rendered JSON DOCUMENT in a single `manifestjson` column (drop the
row-mirrored `manifestpublishedentries` family the earlier draft proposed).
The only consumer of a snapshot is the client, and it consumes exactly that
document, so freezing the text makes immutability structural (no UPDATE path),
rollback a one-flag `iscurrent` flip, serving a single-row read, and version
diffing a plain text diff - all things average IT can debug; row-mirroring
would add ~6 shadow tables and a copy routine that can drift. Columns:
`publishedversionid`, `scopeid`, `versionnumber` (1,2,3 per scope),
`manifestjson` (MEDIUMTEXT, verbatim), `publishedat`, `publishedby`,
`iscurrent`, `notes`. Editing `manifestentries` never affects the fleet;
"publish" freezes a new snapshot; the client is ALWAYS served the current
snapshot, never the live draft. Rollback = flip `iscurrent` to an older
version (the post-cutover safety net once the on-share JSON is retired).
Mirrors today's `_meta/history/<date>-<scope>.json` backups, but authoritative.
Revision history: every publish is a permanent, immutable revision kept
indefinitely (snapshots are small JSON text, ~10 scopes - storage is a
non-issue). An OPTIONAL retention policy (keep last M per scope, or prune
older than N months) can be added later; default is keep-everything, off.
- Draft-edit audit trail (field-level history BETWEEN publishes): drafts
(`manifestentries`) are not versioned - editing overwrites the working copy.
To answer "who changed this entry and when" in the window between two
published revisions, log every draft mutation through the EXISTING core audit
system (no new table): on create/update/delete of a scope, entry, or child
row, write an audit record with the actor, timestamp, entry name, and the
changed field(s). This gives per-edit provenance for free and shows up in the
same Audit Logs UI IT already uses; the published snapshots remain the
coarse-grained "what the fleet actually got" record.
- `manifestentrypctypes`, `manifestentryhostnames`, `manifestentrymachinenumbers`
- child rows for the ANDed multi-value filters (one value + a `sortorder` per
row, wildcards stored verbatim as patterns)
- `manifestinusechecks` + `manifestinusecheckprocesses`
- the nested InUseCheck object and its Processes[] child list (leave
`gracefulclosetimeoutsec` nullable; do not bake the engine's default of 10
into the row, emit it only when set)
- `manifestpayloads` - inline payload bytes for `payloadsource = inline`
(`entryid`, `filename`, `contenttype`, `payloadbytes` LONGBLOB, `payloadsha256`,
`uploadedat`). App-enforced size cap ~1 MB; the upload UI rejects larger with
"use SMB for this" so nobody pastes an MSI into the database. Can ship empty
and unused until P6.
- `pctypealiases` - a MIRROR of the old<->new name alias graph from
`Install-FromManifest.ps1:463-475`, for server-side resolve/validate only.
The engine lib stays the single source of truth (see section 10); shopdb
never becomes the authority the client depends on for aliases.
The JSON the client receives is REBUILT from a published snapshot in exact
array order. Parity with the current engine is proven by BEHAVIORAL equivalence,
not byte-identity (see section 9): re-serialized JSON will differ in key order
and whitespace, so the test is that both manifests parse to the same ordered
entry set with the same detection/targeting/action semantics.
## 5. Payloads: SMB and/or HTTP (both supported)
The user asked whether payloads can be SMB and/or HTTP. Yes - per entry:
- `payloadsource = smb`: `payloadref` is the current relative path
(`apps/eDNC_6-4-5.msi`); the client still mounts W: and resolves it against
the scope root exactly as today. The engine is unchanged for these rows (the
mount + scope-root resolution still happen; an HTTP-only site skips the mount
because it has no `smb` rows). This is the default and the migration target
for large binaries (MSIs are hundreds of MB; SMB streaming beats HTTP).
- `payloadsource = http`: `payloadref` is a URL (absolute, or relative to a
configured payload base). The client downloads to a local temp dir, verifies
the Hash/FileVersion detection value, then runs it. Good for small
config/script payloads and for sites with no SMB share.
- `payloadsource = inline`: for small text payloads (a `.ps1`, a config file, a
registry value), the bytes live in shopdb itself and are served in-band. No
external store at all. Best for scripts and File-type config drops.
Manifest generation emits, per entry, whatever the client needs to fetch the
bytes. The engine's existing "stage network EXE to local temp first" logic
(SYSTEM access-denied workaround) generalizes cleanly to HTTP download.
Payload integrity uses the dedicated `payloadsha256` column, NOT `DetectionValue`.
This is the correction to a subtle trap: `DetectionValue` is a SHA256 only when
`DetectionMethod = Hash`. Most binaries detect by `Registry` or `FileVersion`
and carry no payload hash at all, so relying on `DetectionValue` would let an
HTTP/inline-fetched MSI run unverified. Instead, publishing an `http`/`inline`
payload computes and stores `payloadsha256`, and the client verifies the fetched
bytes against it BEFORE running, independent of how the entry detects install
state. `smb` payloads may set it too (defense in depth) but the share ACL is
their primary trust boundary. Detection stays a separate concern: it decides
whether to act; the payload hash decides whether the bytes are trustworthy.
Transport security: the client fetches as SYSTEM, so the shopdb TLS cert must be
trusted machine-wide. Sites with a self-signed or air-gapped shopdb need the CA
in the machine trust store (provisioned by the same Azure DSC step that writes
the token). Plain HTTP is acceptable only inside a trusted segment, and even
then the `payloadsha256` check is what actually guarantees payload integrity.
## 6. API surface (`/api/geenforce/...`)
Two permissions via the plugin's `get_permissions()` hook (split so day-to-day
techs can edit but only a lead ships to the fleet):
- `geenforce.manage` - create/edit/reorder scopes, entries, drafts, payloads.
- `geenforce.publish` - publish, rollback, export-to-share (the fleet-affecting
actions).
Draft editing (`geenforce.manage`):
- `GET/POST /scopes`, `GET/PUT/DELETE /scopes/<id>` - imaging PC types
- `GET/POST /scopes/<id>/entries`, `PUT/DELETE /entries/<id>` - manifest entries
- `PUT /scopes/<id>/entries/reorder` - the ordering contract; Move Up/Down in the
UI (plain buttons + visible `sortorder`), not a drag-and-drop dependency
- `POST /entries/<id>/payload` - upload an inline/http payload (multipart),
compute + store its `payloadsha256` (the integrity hash; NOT `detectionvalue`)
- `GET /scopes/<id>/preview` - the draft JSON a client WOULD receive on next
publish; `GET /scopes/<id>/published` shows the currently-served snapshot
- `GET /scopes/<id>/simulate?pctype=&subtype=&hostname=&machinenumber=&cmmversion=`
- the "what would this PC get" simulator: runs the entry list through the same
filter logic the engine uses and returns which entries apply and why the rest
are filtered out. Reuses the P1 parity harness's filter engine, so it is
nearly free, and it is the single most IT-empowering endpoint - it answers
"why did/didn't app X install on PC Y" without reading a PowerShell log.
Publishing (`geenforce.publish`):
- `POST /scopes/<id>/publish` - freeze the current draft into a new immutable
`manifestpublishedversions` snapshot (this is what the fleet gets)
- `POST /scopes/<id>/rollback/<version>` - mark an older snapshot current
- `POST /scopes/<id>/export-share` (or a `flask geenforce export-share` CLI) -
write the current published JSON to `<shareroot>/<scope>/manifest.json` after
copying the existing file to `_meta/history/<date>-<scope>.json`. This is a
first-class feature, not a footnote: it is the Milestone 1 product (author in
shopdb, engine untouched) and the permanent break-glass path.
Client-facing (gated by a collector-style service token, `geenforce.fetch`
scope, reusing the PAT + `X-API-Key` machinery already built for the collector):
- `GET /manifest?pctype=<scope>&subtype=<s>&hostname=<h>&machinenumber=<n>`
Returns the latest PUBLISHED snapshot for that scope (never the live draft).
The server can pre-apply the PCTypes/hostname/machinenumber/cmmversion filters
(thin client) OR return the full scope and let the engine filter (fat client,
matches today). Start fat: return the scope manifest unchanged so the engine
logic is untouched. Include the snapshot version + an ETag so the client can
cache and no-op when unchanged.
- Payload fetch for `http`/`inline` rows: `GET /payload/<entryid>` streaming the
bytes; the client verifies them against `payloadsha256` from the manifest.
## 7. Frontend: expand `/settings/pctypemapping`
The current page (`PCTypeMappingSettings.vue`, "Collector PC Types") is a read-
only-ish table of `pxetype -> ComputerType` dropdowns. It grows into the imaging-
PC-type manager:
- **Scopes list**: add/rename/delete imaging PC types; each still carries its
`ComputerType` mapping (that column moves from a setting into `manifestscopes`).
A `phase` toggle (runtime vs preinstall). Common scope flagged.
- **Scope detail / manifest editor**: an ordered list of entries with Move
Up/Down buttons and a visible `sortorder` (the ordering contract made visible;
NOT drag-and-drop - a drag library is the kind of dependency that breaks
silently and average IT cannot fix; add drag later if wanted). Each entry is a
typed form - the visible fields switch on `entrytype` (MSI shows
Installer+InstallArgs; PS1 shows Script+Args; File shows Source+Destination;
Registry shows the Reg* quartet), one line of help per detection method.
Filter chips for PCTypes/hostnames/machine numbers. InUseCheck sub-editor.
Payload source selector (smb/http/inline) with upload for the latter two.
`applymode`/`updatewindow` sit behind an "Advanced (not yet enforced by the
engine)" disclosure. Ship the editor in three usable-alone increments: (a)
scope list + entry table, (b) the typed entry form, (c) publish + diff. That
keeps the biggest chunk of the build from ballooning.
- **Simulator ("what would this PC get")**: a small form (pctype, subtype,
hostname, machine number, CMM version) that calls `GET /scopes/<id>/simulate`
and lists which entries apply and why the rest are filtered. The single most
IT-empowering piece of the UI.
- **Draft, preview, publish**: editing changes only the draft; "publish" freezes
an immutable snapshot (see section 4) and is what the fleet then gets. Show the
draft-vs-published diff before publishing. Rollback republishes a prior
snapshot.
- **Desired vs observed (BUILT: observed-state reporting)**: rather than extend
the collector, the plugin has its own reporting path. Each enforcement cycle a
PC POSTs `POST /api/geenforce/report` (geenforce.report service token) with the
published version it applied, the installed/skipped/failed/filtered counts, and
per-entry outcomes. Stored in `manifestenforcementreports` (latest-per-host +
history) and `manifestenforcementresults` (per-entry). Two payoffs fall out:
RECEIVED - `receivedlatest` compares the applied version to the scope's current
published version, so the fleet view shows which PCs picked up an update; and
SELF-HEAL - each entry's action (installed = drift corrected, skipped = already
good, failed) with any warning/error message. Admin reads: `GET /reports`
(fleet compliance) and `GET /reports/<id>` (per-entry detail). This is the
observed half that makes the manifest a closed desired-vs-observed loop.
This is an ADR-010 settings card contributed by the geenforce plugin, so it only
appears when the plugin is enabled.
## 8. Client change (minimal, staged)
`GE-Enforce.ps1` today: mount W:, read `<scope>\manifest.json`, hand to
`Install-FromManifest`. New path: GET the manifest from shopdb, write it to the
same local location the engine reads, then run the engine unchanged. That is the
smallest possible client delta - the engine, detection logic, self-heal, and
SMB payload resolution all stay identical. Only the *source of the JSON* moves
from file to HTTP.
Payloads: `smb` rows need no client change. `http`/`inline` rows need a small
fetch-and-verify helper (download to temp, check SHA256, then the existing
installer action runs against the local copy). The engine already stages network
EXEs to temp, so this is an extension, not a rewrite.
Auth: the client already has SFLD credentials in
`HKLM:\SOFTWARE\GE\SFLD\Credentials`. Add a shopdb service token (a
`geenforce.fetch` PAT) provisioned the same way (Azure DSC writes it to
registry), sent as `X-API-Key`. If shopdb is unreachable, the client falls back
to the last-known-good manifest cached locally (fail-safe: never leave a PC
unmanaged because the web app is down). This mirrors today's "creds missing =
exit 0, retry next cycle" resilience.
## 9. Cutover strategy
The manifest is desired-state that runs as SYSTEM and installs software fleet-
wide. A bad cutover = a fleet-wide mis-install. Stage it:
1. **Import + parity.** Write a one-shot importer that reads the current
on-share manifests (common + every `gea-shopfloor-*` + preinstall.json;
skip `.bak` / `.pre-mtconnect.bak` variants) into the new tables. Then
generate JSON back out and prove BEHAVIORAL equivalence for every scope - do
NOT chase byte-identity. Re-serialized JSON will differ in key order,
whitespace, and `_comment` formatting, so a raw `diff` would never converge.
The correct test: parse both the original and the regenerated manifest,
normalize, and assert the same ordered entry list with identical
detection/targeting/action fields per entry (ideally a small harness that
mimics the engine's filter+detect decisions and confirms the same entries
would fire in the same order on representative machine profiles). That, not
byte equality, is what proves the model is lossless. (Same discipline as the
ADR-001 data migration.)
2. **Shadow mode.** shopdb serves the manifest at a new endpoint; a canary PC
fetches from shopdb but ALSO reads the share, and logs any diff. No install
behavior changes. Run across one of each PC type for a few cycles.
3. **Read cutover, payloads still SMB.** Flip GE-Enforce to source the JSON from
shopdb (payloads stay `smb`). The blast radius is only "where the JSON comes
from"; the bytes and engine are unchanged. Keep the share manifests as the
rollback (revert the dispatcher one-liner).
4. **Payload migration (optional, per entry).** Move small scripts/configs to
`inline`/`http` opportunistically. Leave big MSIs on SMB indefinitely - SMB
is the right transport for them.
5. **Author in shopdb.** Once read-cutover is stable, new manifest edits happen
in the shopdb UI and the on-share JSON is retired (or auto-exported as a
backup for break-glass).
Rollback during cutover (stages 2-4) is a one-line dispatcher revert, because
the engine and payload layout never stop working from the share. AFTER the share
JSON is retired (stage 5), that escape hatch is gone - post-cutover rollback is
republishing a prior `manifestpublishedversions` snapshot (section 4). Both
mechanisms must exist before stage 5, not just the dispatcher revert.
## 10. Risks / open questions
- **The engine is the contract.** Any drift between shopdb's generated JSON and
what `Install-FromManifest.ps1` expects is a fleet-wide install bug. The
byte-identical round-trip test (step 1) is non-negotiable, and the plugin must
pin which engine lib version it targets (>= 2.6 for `_CmmVersion`).
- **PCTypes alias graph** must be kept in sync with
`Install-FromManifest.ps1:463-475`. The engine lib stays the single source of
truth; shopdb only MIRRORS the map for server-side validation. Do NOT invert
this to have the engine fetch aliases from shopdb - that would add exactly the
availability coupling the next bullet warns against. When the lib's alias map
changes, update shopdb's mirror as part of shipping that lib version.
- **Availability coupling.** GE-Enforce currently depends only on SMB. Adding an
HTTP dependency on shopdb means shopdb downtime could stall enforcement -
hence the last-known-good local cache in section 8. Must be built in from day
one, not bolted on. This is also why alias resolution and payloads stay
independent of a live shopdb wherever possible.
- **Transport trust.** The client runs as SYSTEM, so shopdb's TLS cert must be
in the machine trust store (self-signed/air-gapped sites need the CA
provisioned via the same DSC step as the token). `payloadsha256` verification
is the real integrity guarantee and holds even over plain HTTP inside a
trusted segment (section 5).
- **Secrets in payloads.** Some config drops (site-config, credentials) may
contain secrets. `inline` payloads live in the shopdb DB - those must respect
the existing "secrets stay in .env, not the settings table" rule. Likely keep
any secret-bearing payload on SMB with ACLs, never inline.
- **Preinstall runner** is a separate consumer (`00-PreInstall-*` at imaging,
before enrollment). It may not have a shopdb token yet at that point in the
imaging sequence. Preinstall may need to stay share-sourced longer than
runtime, or fetch a bootstrap manifest anonymously over HTTP.
- **This is a big build.** Realistically phased: (P1) model + importer +
behavioral-parity test; (P2) admin API + CRUD + publish/snapshot/rollback;
(P3) frontend editor on /settings/pctypemapping; (P4) client fetch + shadow
mode; (P5) read cutover; (P6) payload migration. P1 is the gating de-risk - if
behavioral parity does not hold, stop. Snapshots (P2) must land before any
client points at shopdb (P4), since serving the live draft to the fleet is
unacceptable.
## 11. Relationship to existing work
- Replaces `plugins/computers/pctypemap.py` (the thin `pctypemap_<pxetype>`
settings) - the pctype -> ComputerType mapping becomes the `computertypeid`
column on `manifestscopes`. Two-source transition window: `pctype_mapping()`
must keep reading the settings until the geenforce plugin is enabled, then
fall back geenforce-table-first / settings-second, and only retire
`seed_pctype_settings` + the settings at Milestone 1 close. Also reconcile the
scope inventory: `pctypemap.py` lists `gea-shopfloor-display` but the share has
no such manifest dir, and the share has a `main/` legacy dir the model ignores
- the importer creates scopes only from what it finds (plus empty scopes for
mapped-but-absent pctypes), and the P1 gate review reconciles the list with
the floor team.
- Also folds in the metrology mapping now living in `pctypemap.py`
(`METROLOGY_TOOL_MAP`). The collector already auto-creates a MeasuringTool
asset and a directional PC->tool `controls` relationship when it sees a
metrology pctype (CMM / Keyence / Genspect / wax-and-trace); the PC stays a
shopfloor PC. A metrology scope in the manifest model should carry the
attached-measuring-tool type alongside its ComputerType so imaging and
collector agree on what device the scope implies.
- Reuses the collector's token machinery (PAT + `X-API-Key` + scopes) for the
client-facing endpoints.
- Reuses `get_permissions()` (contract 0.10.0) for `geenforce.manage` (edit
drafts) / `geenforce.publish` (publish, rollback, export) / `geenforce.fetch`
(the client service token).
- Pairs with the collector: desired-state (this plugin) + observed-state
(collector) enable a fleet compliance view.
## 12. Recommendation
Feasible and a strong architectural fit, but it is a multi-phase build with a
fleet-wide blast radius. The single most important gate is P1: import the real
manifests and prove BEHAVIORAL parity (same entries fire in the same order with
the same detection/targeting), not byte-identity. Do not build the UI or touch a
client until that parity holds. Three things separate a safe build from a
dangerous one and must not be cut: behavioral-parity import (P1), immutable
published snapshots with rollback before any client points at shopdb (P2/P4),
and a dedicated `payloadsha256` for every HTTP/inline payload (section 5). If and
when we proceed, this warrants a new ADR (ADR-012: GE-Enforce manifest
ownership) capturing the desired-state model, the published-snapshot contract,
the SMB/HTTP/inline payload + integrity model, and the fail-safe cache.
## 13. Execution plan (build order, gates, milestones)
Governing constraint: every step must be runnable and maintainable by average
site IT, not just the original developer. Where an earlier draft implied expert
machinery, this section simplifies it (and the model above already reflects
those simplifications: one wide table, JSON-document snapshots, no row-mirroring).
### Phases and gates
- **P0 - Scaffold (S, ~0.5-1 day).** `flask plugin new geenforce`, structure
copied from `plugins/measuringtools/`. Unlike bundled plugins' no-op migration
anchors, this NEW plugin's `0001_geenforce_baseline` actually creates the
tables and registers them in `PLUGIN_TABLE_OWNERS` (ADR-008). Deploy stays the
standard `flask db upgrade` + `flask plugin upgrade-all`. Manifest:
`api_prefix: /api/geenforce`, `default_enabled: false`, tight `core_version`.
- **P1 - Model + importer + parity harness (M, ~1-1.5 wk). THE GATE.** Order
inside: tables -> `flask geenforce import-share` (reads common + every
`gea-shopfloor-*` + preinstall.json, skips `.bak`, idempotent) -> exporter
(rebuilds each scope's JSON from rows in `sortorder`) -> the parity harness
(below). **GATE A:** `flask geenforce parity` prints PASS for all scopes. If
it cannot pass, STOP the project. No API/UI/client work before Gate A.
- **P2 - Publish/snapshot/rollback + admin API + export-to-share (M, ~1.5-2 wk).**
Publish freezes rendered JSON into `manifestpublishedversions`. CRUD per
section 6. Plus `flask geenforce export-share` + an "Export to share" button
that writes each scope's published JSON to the share after backing up the old
file to `_meta/history/`. Engine, dispatcher, share layout, payloads, PCs all
untouched. **GATE B = Milestone 1** (below).
- **P3 - Frontend editor (L, ~2-3 wk; parallel with P4 after P2 API freezes).**
Expand `PCTypeMappingSettings.vue` per section 7, in three shippable
increments; Move Up/Down not drag; the simulator.
- **P4 - Client fetch + shadow mode (M effort + soak time; needs P2, not P3).**
Week-1 spike: a ~20-line PS1 on ONE canary PC proves SYSTEM-context HTTP auth +
TLS trust before any real client change. Then `GE-Enforce.ps1` fetches JSON to
a local cache and hands the file to `Install-FromManifest.ps1` unchanged;
shadow mode installs from the share but logs any diff vs shopdb; ETag +
last-known-good cache from day one. **GATE C:** zero shadow diffs across one PC
of every pctype for >= 20 cycles.
- **P5 - Read cutover (S effort, M calendar).** Per-scope flip, canary first via
`TargetHostnames`. Payloads stay `smb`. Rollback = dispatcher revert; share
export continues as break-glass. **GATE D:** all scopes cut over.
- **P6 - Payload migration (S per entry, optional forever).** Small configs to
`inline` (verified by `payloadsha256`); MSIs stay on SMB. Each entry
independently revertible (flip `payloadsource`).
Hard ordering: P0 -> P1 -> P2 -> rest. **Snapshots (P2) MUST precede any client
pointing at shopdb (P4).** P3 and P4 parallelize. Preinstall stays share-sourced
through at least Milestone 1 (no token pre-enrollment; export writes
`preinstall.json` too, so it is authored-in-shopdb for free with no client risk).
### The P1 parity harness (concrete, IT-re-runnable)
`plugins/geenforce/parity.py` + a CLI, also wrapped as a CI test. Two checks per
scope, output one readable line per scope (`entries N/N identical profiles M/M
same-fire PASS`), exit 0/1, prints the first differing entry/field on fail:
1. **Lossless field check (order-preserving).** Canonicalize each entry to
exactly the fields the engine reads (Name, Type, the payload fields, all
Detection*, the filter arrays, `_CmmVersion`, InUseCheck, preinstall flags);
exclude `_comment` and key order (documentation, not behavior). Compare the
ordered lists position by position.
2. **Same-entries-fire-in-same-order.** Re-implement in ~120 lines of Python the
engine's four filter functions exactly as written in `Install-FromManifest.ps1`
(`Test-PCTypeMatches` incl. the alias groups at lines 463-475, `"*"`, and
`<Type>-<SubType>`; `Test-HostnameMatches` exact + `-like`;
`Test-MachineNumberMatches`; `Test-CmmVersionMatches`). For each machine-
profile fixture, run BOTH manifests through it and assert the identical
ordered list of entry names that pass all filters. Detection itself is not
executed - check 1 already proved detection fields identical, so identical
inputs to detection are guaranteed. This pair proves losslessness without
byte-diffing.
Fixtures (`plugins/geenforce/parityfixtures.json`, ~16-18 profiles): one per
pctype; CMM version variants `2016/2019/2026`/empty; collections machine-number
variants (a credentialed bay, an MTConnect bay, neither); legacy-alias profiles
(`Standard`+`Machine`, `CMM`) to exercise the alias graph both ways; a `WJS-*`
hostname-wildcard profile; preinstall profiles including one that hits
`PCTypesStrict`. Watch-items the harness must handle: empty `Applications: []`
scopes (4 exist), entries with NO `DetectionMethod` (fire every run), and the
`regvalue` literal typing.
### First slice: one vertical through `gea-shopfloor-cmm`
Only 4 entries but hits every hard part - MSI type, Registry detection with and
without a pinned value, nested InUseCheck with Processes[], and the `_CmmVersion`
gate. Tables: scopes, entries, entrypctypes, inusechecks + processes,
publishedversions, pctypealiases. `flask geenforce import-share --scope
gea-shopfloor-cmm`; `flask geenforce publish gea-shopfloor-cmm`; one endpoint
`GET /api/geenforce/manifest?pctype=gea-shopfloor-cmm` serving the published
snapshot (fat-client, ETag, collector-style `X-API-Key`/PAT auth reusing
`shopdb/core/api/collector.py`). **Done =** parity PASS for cmm; the endpoint's
JSON fed to `Install-FromManifest.ps1` on a bench CMM PC logs `4 skipped`
identically to the share manifest; editing a draft does NOT change the served
bytes but publishing does, and rollback restores the prior published bytes;
unauth = 401, wrong-scope = 401.
### Milestone 1 (the recommended first stop)
End of P2 plus the publish/scope-list slice of P3: **manifests are authored and
published in shopdb, exported to the share by a button, and the engine,
dispatcher, share layout, payloads, and every PC are completely unchanged.**
That delivers the real pain relief - validated editing instead of hand-edited
JSON, version history, one-click rollback (republish + re-export), desired-state
data sitting next to collector data - at ZERO client risk, with a rollback any
IT tech already knows (restore the `_meta/history` backup file). Natural point to
write ADR-012 with real experience behind it. P4/P5 (HTTP fetch, cutover) are a
separately green-lit second milestone.
### Ranked risks / fail-fast
1. **Generated-JSON vs engine drift (fleet-wide mis-install).** Parity harness
first; CI re-proves parity against checked-in real manifests on every
exporter change; pin lib >= 2.6.
2. **Serving a half-finished draft.** Structural: client reads only
`iscurrent` snapshots; test asserts a draft edit leaves served bytes
unchanged. Must exist before P4.
3. **Availability coupling.** Last-known-good local cache in the first client
prototype; shadow test blocks shopdb and confirms enforce-from-cache + WARN.
4. **SYSTEM HTTP auth + TLS trust.** The ~20-line canary spike in P4 week 1,
before the real client change. Hours of cost; if it fails, Milestone 1 still
delivers full value.
5. **Alias-graph drift.** Seed pins a lib version; harness legacy-name profiles
fail loudly on divergence; new-lib runbook includes "update the alias seed".
6. **Preinstall has no pre-enrollment token.** Keep share-sourced through
Milestone 1/2; decide later.
7. **Editor scope creep.** Three shippable increments; buttons over drag; reuse
JSON preview.
### IT operability (day-to-day runbook, proving the design is manageable)
All in Settings > Imaging PC Types. No PowerShell, no SQL, no share edits.
- **Add an app to a PC type:** open the PC type, Add Entry, pick Type (fields
adapt), fill installer + detection + targeting, Move Up/Down to order, Preview
(+ simulator), Publish with a note. PCs pick it up next 5-min cycle.
- **Bump a version:** drop the new MSI in the scope's `apps/` on the share,
update the entry's Installer + Detection value, Preview, Publish.
- **Roll back a bad publish:** History -> pick last-good version -> Roll Back
(during Milestone 1 also click Export to Share).
- **Canary a risky change:** add the one test PC under Target Hostnames, Publish;
when happy, remove the filter and Publish again.
- **Check "did PC Y get app X":** the simulator with that PC's type/machine
number/CMM version shows exactly which entries apply and why others are filtered.
- **See revision history / who changed what:** the PC type's History tab lists
every published version (date, author, note) with a Roll Back on each; the
Audit Logs page shows the finer-grained draft edits (who touched which entry
field, when) between publishes.

219
docs/proposals/printedparts-plugin.md Normal file
View File

@@ -0,0 +1,219 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The design behind the printedparts plugin and kiosk. Built; kept as the record of why.
loadwith: none
-->
# Proposal: printedparts plugin (3D-printed parts storefront + kiosk)
Status: BUILT. The plugin ships as `plugins/printedparts/`, kiosk route
included. This page is the design behind it, kept as the record of why, and it
is also the reference design for the plugin-development lab in
`docs/PLUGIN-LAB-PRINTEDPARTS.md`. It is NOT current guidance: where it and the
shipped plugin disagree, the code wins.
## 1. Problem
The 3D-printer engineers stock bins of printed parts (fixtures, clips, covers,
spacers). Anyone on the floor can take parts, so stock silently runs out and
nobody knows who took what or how fast items burn down. They need:
- a catalog ("storefront") of printable items: photo, description, quantity on
hand;
- a barcode label per item (1in x 0.5in) stuck on each bin;
- a touch-screen kiosk: scan the bin barcode, scan your badge, enter how many
you took, submit;
- restock and correction flows for the engineers;
- stock monitoring plus consumption metrics.
## 2. Shape: standalone model plugin, NOT an asset type
These are quantity-based consumables: one row represents a *kind* of part with
a count, not an individually tracked machine. ADR-001 assets are one-row-per-
physical-thing (a PC, a printer). So printedparts follows the
knowledgebase/usb shape - own tables, own blueprint, no AssetType row - and
does NOT join the asset-label TYPE_CONFIG; it ships its own print view the way
USB labels do.
Item identity: `itemcode`, generated `3DP-<zero-padded id>` (prefix
configurable via setting `printedparts_code_prefix`). Short, CODE128-friendly,
human-readable. This is what the bin label encodes.
## 3. Data model (2 tables, LOCKED naming, per-plugin Alembic)
### printeditems
| column | type | notes |
|---|---|---|
| printeditemid | int PK autoincrement | |
| itemcode | varchar(20) unique, indexed | generated on create |
| itemname | varchar(120) NOT NULL | |
| itemdescription | varchar(500) | brief description |
| imageurl | varchar(255) | served upload, models.py pattern |
| quantityonhand | int NOT NULL default 0 | cached; ledger is truth |
| lowstockthreshold | int NOT NULL default 5 | per-item, seeds from setting |
| binlocation | varchar(100) | where the bin lives |
| printnotes | mediumtext | material, print time, slicer file path |
| isactive | tinyint(1) | soft retire |
| createddate / modifieddate | datetime | AuditMixin/BaseModel |
### printeditemtransactions (the ledger - source of truth)
| column | type | notes |
|---|---|---|
| transactionid | int PK | |
| printeditemid | int FK -> printeditems CASCADE, indexed | |
| transactiontype | varchar(10) NOT NULL | take / restock / adjust |
| quantitychange | int NOT NULL | negative for take, signed for adjust |
| employeesso | varchar(20) NOT NULL, indexed | who (badge-resolved) |
| employeename | varchar(120) | resolved at write time (USB pattern) |
| reason | varchar(255) | required for adjust |
| transactiondate | datetime NOT NULL default naive-UTC, indexed | |
Invariants: `quantityonhand` = sum of `quantitychange` (enforced by writing
both in one session/commit; an `adjust` can never drive it below 0 - reject).
Every write records WHO via badge scan; there is no anonymous mutation.
Both tables registered in `PLUGIN_TABLE_OWNERS`
(`shopdb/plugins/alembic_template.py`); migration 0001 is a REAL baseline
(measuringtools pattern - hand-written `op.create_table`, no cross-schema FK
so `create_plugin_tables` would also work, but write the ops explicitly for
the exercise).
## 4. Badge resolution (reuse the USB contract exactly)
Same input shapes as `plugins/usb/api/routes.py`:
- all digits -> SSO;
- `0<digits>BZ` (case-insensitive) -> physical badge wrapping a PayNo;
- resolution to a display name via the employees plugin directory
(`DirectoryEmployee`, selfhosted mode) with graceful "" fallback.
Extract-or-copy decision for the lab: copy the small `_PAYNO_BADGE` regex +
lookup into the plugin (contract-pure, no cross-plugin import of usb).
Resolution happens SERVER-side on the kiosk endpoint - the kiosk client never
supplies a name, only the raw badge string.
Manifest `dependencies: ["employees"]` (name lookup). Badge that resolves to
no employee: configurable policy setting `printedparts_unknown_badge`
(`allow` = record SSO with empty name, `deny` = 422). Default deny.
## 5. API surface (blueprint at /api/printedparts)
Authenticated management (JWT + permission):
| route | method | permission |
|---|---|---|
| `/items` | GET list (search, paginate, lowstock filter) | open read (jwt optional) |
| `/items/<id>` | GET detail + recent transactions | open read |
| `/items` | POST create (mints itemcode) | printedparts.create |
| `/items/<id>` | PUT update | printedparts.edit |
| `/items/<id>` | DELETE soft-retire | printedparts.delete |
| `/items/<id>/image` | POST/DELETE upload/remove | printedparts.edit |
| `/image/<filename>` | GET serve | public (models.py pattern) |
| `/items/<id>/restock` | POST {quantity, badge} | printedparts.restock |
| `/items/<id>/adjust` | POST {quantitychange, reason, badge} | printedparts.restock |
| `/items/<id>/transactions` | GET history, ?format=csv | open read |
Kiosk (unauthenticated, notifications/employees open-endpoint precedent):
| route | method | body |
|---|---|---|
| `/kiosk/item/<itemcode>` | GET | item summary by scanned code |
| `/kiosk/take` | POST | {itemcode, badge, quantity} |
`/kiosk/take` validation: item exists + active; quantity 1..quantityonhand
(clamp/reject configurable? no - reject with clear message, kiosk shows it);
badge resolves per policy. Writes ledger row (negative) + decrements cached
quantity in one commit. Rate of abuse is low (plant floor), but the endpoint
only ever DECREMENTS stock with a recorded badge - it cannot edit the catalog.
Permissions declared via `get_permissions()`: printedparts.view/create/edit/
delete/restock (category `printedparts`), seeded on install/enable.
## 6. Frontend
Management pages (scaffold output, standard layout, master templates
PrintersList/PrinterDetail):
- `PrintedItemsList.vue` - table: image thumb, code, name, qty (red badge when
<= threshold), bin; filters: search, low-stock-only; row click -> detail.
- `PrintedItemDetail.vue` - hero image + fields, transaction history table,
restock/adjust buttons (modal w/ quantity + badge + reason).
- `PrintedItemForm.vue` - create/edit incl. image upload, threshold, bin.
- Router file `router/routes/printedparts.js`, list/detail plugin-gated only,
new/edit + requiresAuth (ADR-009, usb.js precedent).
- Nav via `get_navigation_items()` -> "3D Parts".
Kiosk (net-new, top-level route `/parts-kiosk`, NO requiresAuth, outside
AppLayout - shopfloor precedent):
- Full-screen, 3-step flow: (1) SCAN ITEM - a focused invisible input catches
the keyboard-wedge scan of the bin barcode, shows item card w/ photo + qty;
(2) SCAN BADGE - same wedge input pattern for the badge; (3) QUANTITY - big
touch keypad (0-9, clear, backspace - net-new component
`TouchKeypad.vue`) + TAKE button. Success screen w/ remaining count, auto
reset after a few seconds. All state client-side; one POST at the end.
- Scanner UX rule: keyboard-wedge scanners type the code + Enter. A hidden
always-focused input with @keydown.enter handles both scans; on-screen
prompt tells the user what to scan. Touch fallback: item search + manual
badge entry (small link, for damaged labels).
Labels (own print view, USBLabelBatch precedent):
- `/print/printedparts-labels` public print route.
- NEW physical size: 1in x 0.5in stock -> `@page { size: 1in 0.5in; margin: 0 }`
one label per page (label printers feed roll stock; per-page = per-label).
Layout: CODE128 barcode (JsBarcode, ~0.9in x 0.28in, displayValue false) +
itemcode text under it (~7pt) + optional item name truncated. QR variant
offered but barcode is default at this size (a 0.4in QR is at the edge of
scanner tolerance; CODE128 of `3DP-0042` is comfortable).
- Batch mode: pick items -> one label per page sequence for roll printers;
also a ULINE mini-grid fallback for sheet printers (reuse mini72 pattern).
## 7. Metrics / reports (get_reports hook)
- `printedparts-stock` - current stock levels w/ threshold flags (CSV).
- `printedparts-consumption` - takes per item over a date range (CSV).
- `printedparts-by-person` - takes grouped by employee (CSV).
- Dashboard widget via `get_dashboard_widgets()`: low-stock item count.
- Nice-to-have later: burn-rate (avg takes/week per item + weeks-to-empty
projection) - plain SQL over the ledger, add once basics work.
## 8. Settings (get_settings_cards, category printedparts)
| key | default | purpose |
|---|---|---|
| printedparts_code_prefix | 3DP | itemcode prefix |
| printedparts_default_threshold | 5 | seed for new items |
| printedparts_unknown_badge | deny | kiosk policy for unresolvable badges |
## 9. Manifest
name printedparts, version 0.1.0, api_prefix /api/printedparts,
core_version ">=0.12.0,<1.0.0", dependencies ["employees"],
default_enabled false (site opts in - USB precedent).
## 10. Explicitly out of scope (v1)
- Reservations/approvals, per-item cost, print-queue integration, multi-bin
per item, email low-stock alerts (the reports + dashboard widget cover
monitoring; alerting can ride the existing report-email endpoint later).
## 11. Risks / decisions taken
- Cached quantity vs ledger drift: single-commit writes + a reconcile query in
the stock report (flags items where cache != ledger sum).
### Decision: the kiosk take endpoint is an unauthenticated WRITE
This is the first open mutation in the product - every existing kiosk
endpoint (notifications, employees, shopfloor) is a read, and the closest
write (USB checkout) is JWT + permission gated. Accepted deliberately, on
these grounds, and any future open-write endpoint must meet the same bar:
1. Decrement-only: it can reduce stock of an active item, nothing else - no
catalog edits, no restocks, no reads it does not already expose.
2. Fully attributed: it refuses to act without a badge that resolves per the
site policy; every action lands in the ledger with SSO + name + time.
3. Bounded blast radius: worst case is stock counts driven low, which the
ledger makes visible and reversible (adjust with reason).
4. Physically rate-limited: it exists for a touch screen on the shop floor;
there is nothing to enumerate and nothing returned worth scraping.
- 1x0.5in QR marginal: default to CODE128 barcode.
- Not an Asset: no floor-map plotting or warranty for items. If a site later
wants bins on the floor map, revisit via get_map_overlays (ADR-010).

346
docs/proposals/printer-assignment.md Normal file
View File

@@ -0,0 +1,346 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 design making printers a property of the machine. Shipped in v0.11.3 and v0.12.0.
loadwith: none
-->
# Proposal: assign printers to a machine in ShopDB, let the PC install them
Status: SHIPPED AND RELEASED. Built out 2026-08-18/19; the client script is
built too. The feature itself released in v0.11.3, the observation half and
`seed-from-observed` adoption in v0.12.0.
Author: planning session 2026-08-18.
One thing this proposal did NOT anticipate, resolved 2026-08-24 in `adc7ef4`:
the computers collector carried a `_sync_printer_links` that wrote
`defaultprinter` / `connectedto` rows straight from a PC report, which is the
ASSIGNED-vs-OBSERVED split this proposal rests on, broken from the other end.
It is retired; `defaultprinter` and `printers` are now ignored (not rejected) on
the collector payload. See [COLLECTOR-INTEGRATION.md](../COLLECTOR-INTEGRATION.md).
## 1. What this is
Today a printer reaches a shop-floor PC because a person walks up to it, opens
the printer installer, finds the printer on a floor plan and clicks it. That is
fine for someone choosing a printer, and wrong for a bay whose printers are a
property of the bay.
This proposal makes the assignment data. The printers belong to the MACHINE, not
to the box currently driving it: tick the printers that belong on the machine,
mark one default, and the assignment reaches whichever PC controls that machine.
The PC converges on its next GE-Enforce cycle - installing what is missing and
setting the default - and keeps converging, so a reimaged bay comes back with
its printers and a bay that drifts is corrected.
The point of putting the assignment on the machine is that a reimaged PC needs
no backup and no restore step. The asset register is the source of truth, and a
replacement PC that inherits the `controls` edge inherits the printers with it.
The map installer stays, for the case it is actually good at: a person at an
unmanaged or office PC picking a printer that nobody assigned.
## 2. Why it is worth doing
- **The assignment becomes a record.** "Which printers does that bay have" is a
question ShopDB can answer, and today it cannot.
- **A reimage stops costing a visit.** The bay reinstalls its own printers, from
the machine's record, with nothing saved off the old PC.
- **Swapping the PC keeps the printers.** They were never the PC's.
- **Drift is corrected, not just detected.** A queue deleted by a user comes
back.
- **It removes the walk-up from the common case.** The installer's map remains
for the uncommon one.
## 3. What already exists
Most of the model is in place, which is why this is a small feature rather than
a project.
| piece | state |
|---|---|
| PC to printer link | `defaultprinter` asset relationship, seeded by `flask seed reference-data` |
| Propagation mechanism | `RelationshipTypePropagation` (ADR-001): "type X propagates through connections of type Y" |
| A working precedent | `resolve_asset_position` walks `partof` then `controls` to give a PC the machine's map position |
| Default lookup | `GET /api/printers/pc-default?machine=NNNN` |
| Host lookup | `GET /api/computers/by-hostname/<hostname>` |
| Printer model | `Printer.modelnumberid` - populated for 44 of 44 printers at the reference site |
| Driver record | `PrinterDriver` (name, `location` as SMB path or URL, optional `modelnumberid`) |
| Per-printer install path | `Printer.installpath` |
| Batch install | `GET /api/printers/install-batch?printerids=1,2,3` |
| Client transport | GE-Enforce manifest entries, `Type=PS1`, running as SYSTEM every cycle |
| Silent driver staging | Proven in `PrinterInstaller.iss`: trust the catalog's signing cert, then `pnputil /add-driver` |
## 4. The data model
### 4.1 One new relationship type
`usesprinter`, directional, source -> printer, meaning "this printer is
installed here". It is seeded next to `defaultprinter`, which already exists and
means "which of them is the default". Both are seed data, not a migration, which
is how every other relationship type shipped.
### 4.2 Two propagation rails, consumed at READ time
`usesprinter` propagates through `controls`, and so does `defaultprinter`. Both
are rows in `relationshiptypepropagations`, the same mechanism map positions
use. Inventing a second mechanism for this was the alternative, and it was
rejected.
The rails are inert at write time on purpose. The create-time fan-out
(`propagate_relationship`) skips directional through-types, and `controls` is
directional, so assigning a printer to a machine does not copy rows onto its PC.
The walk happens when something asks, which is what makes the next rule possible.
### 4.3 Resolution order for a PC
1. The PC's OWN active `usesprinter` / `defaultprinter` rows, if it has any.
2. Otherwise, one hop out along its `controls` edges to the machines it drives,
and those machines' rows instead, tagged as inherited.
Own beats inherited, whole set at a time: a PC with its own assignment is
overriding the bay, not adding to it. An office PC controls no machine and still
works, because step 1 is the normal case for it.
The override is a real trap and the UI has to say so. A tech who "fixes" a bay
by editing the PC has shadowed the machine's record, and the machine will keep
disagreeing until someone clears the PC's own rows.
### 4.4 One default, optional, and never dangling
The unique constraint is `(source, target, type)`, which happily accepts two
different defaults. So the rule is enforced in the API on write:
- Exactly one `defaultprinter` per asset. Setting a default replaces the
existing one.
- A default is OPTIONAL. A bay with three printers and no default is valid.
- The default must be one of the assigned printers. Unassigning the printer that
is currently default clears the default rather than leaving it dangling.
### 4.5 One column on `printerdrivers`
`drivername` - the driver's exact name as the INF declares it, e.g.
`HP Universal Printing PCL 6`. `Add-PrinterDriver` matches on that string, not
on `name`, which is ours to choose, and a mismatch is the usual failure.
Deriving it by parsing the INF on hundreds of bays is fragile; a human
confirming it once in ShopDB is not.
It is a plugin-chain migration (`printers0003drivername`), nullable, guarded so a
re-run is a no-op. `printerdrivers` was created by a core migration but its DDL
moved to the printers chain at the ADR-008 cutover.
The `installmethod` column (`pnputil` or `dpinst`) proposed earlier is NOT being
built. See section 8: if production confirms no Brother printers, everything is
`pnputil` and the column has no second value to hold.
## 5. What has to be built
### 5.1 A resolution helper in core
The read-time walk of section 4.3, beside `resolve_asset_position` and exported
on the `shopdb.api` contract surface (an additive minor bump). It has to live in
core because `RelationshipTypePropagation` is not on the contract surface, and a
plugin may not reach past it (ADR-002).
The through-type comes from the seeded rails, not from a hardcoded `'controls'`,
so a site that adds a rail gets the behaviour without a code change.
### 5.2 Two endpoints
```
GET /api/printers/for-host/<hostname>
```
The desired printer set for one PC, resolved per section 4.3, each entry with
what a client needs to install it: queue name, host or IP, port, driver name,
driver location, and which one is default.
Resolved by hostname, not machine number: the collector already upserts PCs by
hostname, and an office PC has no machine number. Matched case-insensitively -
`COMPUTERNAME` is uppercase and MySQL forgives that where SQLite does not.
An unknown host, a site without the computers plugin, or nothing assigned all
return an empty set. That is the client's designed no-op and it must stay
indistinguishable from "assigned nothing".
```
PUT /api/printers/assignments/for-asset/<asset_id>
```
The whole assignment for one asset - machine or PC - in one call:
`{printerassetids: [...], defaultprinterassetid: N|null}`. It reconciles rather
than inserting: rows that went away are soft-deleted, rows that come back
REACTIVATE the soft-deleted row (the unique constraint spans inactive rows, so a
blind insert is an integrity error on assign, unassign, re-assign), new rows are
created, and the single default is replaced.
The rules in 4.4 hold here or nowhere. Row-at-a-time writes through the generic
relationships path leave two-default windows and know nothing of the subset rule.
**Removing an assignment NEVER uninstalls anything.** Server-side the row simply
goes: no cascade, no side effects, nothing queued for the client to undo.
**Per-PC assignments must NOT go in the manifest.** Manifests are keyed by scope
and PC type and sync broadly; putting per-PC rows there would leak every bay's
configuration to every bay and grow without limit. One manifest entry runs one
script that asks the API what THIS host gets - the mirror image of
`Report-AssetToShopDB.ps1`.
### 5.3 UI on the PC form
A printer multi-select plus a default dropdown whose options are only the
currently selected printers, clearing itself when its printer is deselected.
Saved through the reconcile endpoint against the PC's asset.
The computers plugin does not depend on the printers plugin and must not start:
the section hides itself when the printers API is not there.
The machine-side picker is out of scope for now, which means the machine's
assignment is editable only through the generic relationships card. That is the
side the design says is primary, so it is the obvious next piece of UI.
### 5.4 One client script, in two contexts
`Set-ShopdbPrinters.ps1`, shipped in `plugins/printers/client/` beside the
contract it consumes, and run as a manifest entry with `DetectionMethod=Always`.
*As SYSTEM, every cycle:*
1. `GET /api/printers/for-host/$env:COMPUTERNAME`
2. For each assigned printer with no queue: trust the driver catalog's cert,
`pnputil /add-driver`, create the port, create the queue
3. Write the desired default to `HKLM:\SOFTWARE\GE\ShopDB DefaultPrinter`
4. Ensure the per-user task exists
*In the user's context, at logon and on a repeat:*
5. Read that value, compare with the current default, set it if it differs, and
clear "Let Windows manage my default printer" - otherwise Windows silently
overrides the choice the next time someone prints elsewhere
The default printer is per-user state, which is the only reason this needs two
contexts. Everything else is machine state and belongs to the cycle.
Converge, do not reinstall: when the state matches, the script does nothing.
Nothing here needs the manifest to know when a printer changes, because the
desired state is fetched, not declared.
## 6. Desired state and observed state are not the same thing
Everything above is DESIRED state: what SHOULD be installed on a PC. Nothing in
this feature knows what IS installed on it. The client reads the desired state,
converges toward it, and reports nothing back.
Reporting the observed state is the obvious next feature and is deliberately not
this one. If the collector sent the installed queues per host - name, port,
driver, which is default - then comparing that against the resolved assignment
gives drift detection for free: "this bay is missing the label printer", "this
PC has three queues nobody assigned", "the default is not the assigned one".
Keeping them apart is a rule, not a preference:
- **Observed data never writes `usesprinter` rows.** A register that learns from
what it finds mirrors the drift instead of correcting it, and the fault
becomes the desired state.
- **Observed data belongs on the computer record, timestamped**, like the rest
of the collector payload. It is an observation with an age, not a decision.
- **An empty answer from the API means "nothing assigned", not "nothing
installed"**, which is exactly why section 5.2 refuses to make removal
uninstall anything.
- The two can disagree indefinitely and that is a report to read, not an error
to resolve automatically.
## 7. Decisions to take before writing the client
1. **Never remove a queue by default.** A transient API failure would otherwise
strip printers fleet-wide. Deletion is an explicit opt-in, per PC.
2. **Enforced or set-once for the default?** Re-applying every cycle overrides a
user who chose their own default - correct for a locked bay, irritating on an
office PC. Set-once is Active Setup or RunOnce. Make it a per-PC-type flag
rather than one global answer.
3. **Failure is silent and safe**: unreachable API means change nothing, log,
exit 0 - the convention `Report-AssetToShopDB.ps1` already follows.
Open on the server side, and each one changes the response contract:
4. **How a universal driver resolves.** `PrinterDriver` links to a printer by
exact `modelnumberid`, and the target state is roughly four rows dominated by
HP UPD and Xerox GPD, which match no single model. Either the driver row
gains a vendor, or a `modelnumberid IS NULL` row matches on the printer's
resolved vendor name. Until this is settled, `for-host` returns no driver for
41 of 44 printers.
5. **What `port` means when it is null.** RAW 9100 is the obvious default; whose
job it is to apply it - server or script - has to be written down once.
6. **Who may read `for-host`.** `pc-default` and `install-list` are anonymous;
the collector and the GE-Enforce fetch use scoped service tokens. This one
discloses per-PC configuration keyed by hostname.
7. **Two inherited defaults.** A PC can legitimately control both bays of a
dual-bay machine, or several machines. The union of assigned printers is
easy; the default needs a deterministic rule, or none when it is ambiguous.
8. **Legacy `defaultprinter` rows have no `usesprinter` row**, because they
predate the type. Either an active default implies assignment on read
(zero-touch, preferred) or a one-time backfill writes the missing rows.
Otherwise existing defaults vanish from `for-host` while still showing in
`pc-default`.
9. **Deletions through the generic relationships card bypass the reconcile
endpoint** and can strand an active default pointing at an unassigned
printer. Either the resolver drops dangling defaults or the core delete path
learns the rule.
## 8. What the fleet data says, and the one prerequisite
The reference site's 44 printers are HP 26, Xerox 15, Zebra 1, HID 1, Epson 1.
- **HP and Xerox are 41 of 44, and both have true universal drivers** (HP UPD,
Xerox Global Print Driver). One driver record each serves every queue of that
make.
- **There are no Brother printers at all**, yet the installer carries 208 files
of per-model Brother MFC-J inkjet drivers. Those are host-based GDI devices
with no Printer-class INF, which is the only reason a second staging method
(DPInst) exists. If production confirms no Brother, that payload and that code
path can both go - and with them the `installmethod` column.
- **Zebra, HID and Epson are one printer each**, and the HP DesignJet plotter is
a fourth special case - a PostScript device the UPD does not cover.
**Prerequisite: populate `printerdrivers`.** It currently holds ONE row, and it
points at a per-model folder (`HP LaserJet Pro M607 Driver`) rather than the
universal driver - the opposite of how a UPD should be used. The table needs
roughly four rows: HP UPD, Xerox GPD, one per oddity, and DesignJet when its
payload is restored. Each needs `drivername` copied verbatim from its INF.
Nothing in this proposal works until a printer can resolve to a driver.
## 9. Deployment constraint that shapes the design
**The SFLD share is mounted only during GE-Enforce's cycle.** Any work touching
a share path must run as a manifest entry inside that cycle, never as its own
scheduled task. The failure is silent - the task reports 0 processed, 0
installed, 0 failed - and it has cost a session before.
This is why driver staging belongs in the cycle even though the per-user default
does not, and why "the assignment script schedules a task that installs drivers"
is the wrong shape.
## 10. Upgrading an existing site
Three steps, and the third is the one that gets forgotten:
1. `flask db upgrade` - no core migration in this feature, but a deploy runs it.
2. `flask plugin upgrade-all` - applies `drivername`. Skipping it is the classic
1054 unknown-column error.
3. `flask seed reference-data` - REQUIRED. Without it the `usesprinter` type and
both propagation rails do not exist, and `for-host` resolves nothing, quietly,
because empty is also the healthy answer.
Pair the upgrade with a smoke check against a known bay. A site with reversed
legacy `controls` rows (machine -> PC) should run
`flask relationships fix-controls-direction` first, or inheritance resolves for
none of those PCs.
## 11. What this does not change
- The printer installer keeps working, for walk-up and self-service.
- Nothing about how printers are modelled, mapped or reported.
- The collector contract. Section 6 would change it; this feature does not.
- Sites not running GE-Enforce: the same endpoint suits an Intune remediation or
a DSC `Script` resource, since it is a plain HTTP GET and a PowerShell script.

19
docs/wiki-drafts/ADR-019-Plugin-Dynamic-Settings-Routes.md
View File

@@ -1,9 +1,24 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Draft decision record for the plugin settings-routes hook, checked against the code on 2026-09-18.
loadwith: none
-->
# ADR-019: Plugin Dynamic Settings Routes
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Status:** Accepted (2026-09-11)
**Context:** Plugin UI extensions, lean builds
**Supersedes:** Hardcoded plugin routes in core router
**Last checked against code:** 2026-09-18, branch gh-wj at 35d4fcd41
**Last checked against code:** 2026-09-18, the deployment branch at 35d4fcd41
This draft lives only in `docs/wiki-drafts/`. There is no `docs/adr/` directory
on this branch, so this is the sole ADR-shaped record here and nothing enforces
@@ -68,7 +83,7 @@ class ComputersPlugin(BasePlugin):
### Backend: API Endpoint
In `shopdb/core/api/plugins.py:46-95`, as it stands on gh-wj at 35d4fcd41:
In `shopdb/core/api/plugins.py:46-95`, as it stands on the deployment branch at 35d4fcd41:
```python
@plugins_bp.route('/settings-routes', methods=['GET'])

54
docs/wiki-drafts/ADR-Conformance-2026-09-18.md
View File

@@ -1,17 +1,32 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: ADR-by-ADR conformance review of one 20-commit range, as it stood on 2026-09-18.
loadwith: none
-->
# ADR Conformance Review - September 18, 2026
**Branch:** gh-wj at 35d4fcd41
**Range reviewed:** 2ac2e6d33..gh-wj, 20 commits, 149 files, +22055 lines
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Branch:** the deployment branch at 35d4fcd41
**Range reviewed:** 2ac2e6d33..HEAD, 20 commits, 149 files, +22055 lines
**Method:** read-only. Every statement below cites a file and line that was read.
No test suite was run except `tests/test_plugins/test_kiosk_network_allowlist.py`
(20 passed) and `bash scripts/check-naming-and-style.sh` (exit 0).
## Read this first: the ADRs are not in this repository
`ls docs/adr` fails on gh-wj. There is no ADR directory on this branch. The
`ls docs/adr` fails on the deployment branch. There is no ADR directory on this branch. The
whole set, ADR-001 through ADR-018 plus a README, lives on the other lineage at
`docs/adr/ in the product repo` and was read there for this review.
`docs/` on gh-wj holds only `ShopDB-Planner-Tasks-v2.csv`, `deployment/`,
`docs/` on the deployment branch holds only `ShopDB-Planner-Tasks-v2.csv`, `deployment/`,
`notes/`, `openapi.json` and `wiki-drafts/`.
Three consequences, and they are the reason this page exists:
@@ -36,7 +51,7 @@ Three consequences, and they are the reason this page exists:
implemented". Both corrections belong in files that are not in this
repository.
The one ADR-shaped file on gh-wj is the ADR-019 draft in this folder, and it is
The one ADR-shaped file on the deployment branch is the ADR-019 draft in this folder, and it is
a draft.
## Result by ADR
@@ -78,9 +93,10 @@ This covers the names added and kept. It does NOT cover a name added to the
Asset model and then withdrawn again without a bump, which happened in this
range; see finding 2.
**ADR-002, the bumps.** `__contract_version__` moved 0.32.0 to 0.35.0 with a
per-bump rationale naming what was added and why it is additive
(`shopdb/__init__.py:124-152`). Every name those comments claim is genuinely
**ADR-002, the bumps.** The plugin contract moved three minors over this range,
each with a rationale naming what was added and why it is additive
(`shopdb/__init__.py:124-152`; the current value is in
[PROJECT-MAP](../PROJECT-MAP.md), which is generated). Every name those comments claim is genuinely
exported. Every new `BasePlugin` hook is optional with a safe default
(`shopdb/plugins/base.py:313-332`), so minor is the right classification. The
loader still checks `core_version` against `__contract_version__`
@@ -106,12 +122,12 @@ only for its own asset (`plugins/computers/plugin.py:729-733`,
and the dispatcher maps the `PermissionError` to 403 ASSET_MISMATCH
(`collector.py:294-299`).
**ADR-007, the version lock-step.** `__version__ = '0.12.0'`
(`shopdb/__init__.py:158`) matches `frontend/package.json:3`, and `__version__`
is still absent from `shopdb.api`.
**ADR-007, the version lock-step.** The product version in
`shopdb/__init__.py:158` matches `frontend/package.json:3`, and the version
literal is still absent from `shopdb.api`.
**ADR-008 in the forward direction.**
`git diff --stat 2ac2e6d33..gh-wj -- migrations/` is empty. Not one line of
`git diff --stat 2ac2e6d33..HEAD -- migrations/` is empty. Not one line of
plugin schema went into the core chain. The endpoints chain is 24 revisions with
exactly one root and one head, no dangling `down_revision`, longest revision id
29 characters against the 32-character `alembic_version` cap.
@@ -167,7 +183,7 @@ shipped code.
`CHANGELOG.md:11`
`git diff --stat 2ac2e6d33..gh-wj -- CHANGELOG.md` returns nothing at all. The
`git diff --stat 2ac2e6d33..HEAD -- CHANGELOG.md` returns nothing at all. The
file is byte-identical across all 20 commits, which include per-PC agent
credentials, enrollment, bootstrap tokens, server pinning, kiosk network
admission and three contract bumps. Grepping the whole file for "per-PC",
@@ -215,7 +231,7 @@ it stands now. `git show 18e5fdc00:shopdb/core/models/asset.py` puts
`client_cert_fingerprint` (declared `index=True`), `cert_issued_at`,
`cert_expires_at`, `cert_revoked_at` and `cert_revocation_reason` on the CORE
`Asset` model. 18e5fdc00 (2026-09-14, "Fix critical security vulnerabilities in
client certificate authentication") is an ancestor of `gh-wj`; 7480e4fe3
client certificate authentication") is an ancestor of the deployment branch; 7480e4fe3
(2026-09-15) deleted all 28 lines again. The grep comes back empty because it
runs after the retraction, not because the contract was never touched.
@@ -323,7 +339,7 @@ re-litigated.
### 3. ADR-016: four files here describe a decision ADR-016 never took
`docs/wiki-drafts/README.md:316-323` (corrected 2026-09-18), `CLAUDE.md:119`,
`docs/wiki-drafts/README.md:316-323` (corrected 2026-09-18), the assistant-instructions file, line 119,
`the assistant quick-reference file, lines 135 and 14`,
`docs/wiki-drafts/Security-Audit-2026-09-11.md:392`
@@ -398,7 +414,7 @@ Unlike ADR-016, ADR-018 carries no implementation-status paragraph, so its text
reads as done. Low severity: the table holds zero rows, so nothing is broken
today, and ADR-018's own argument is that the window is time-boxed - the
migration is free while the 134 staged metrology instruments have not been
entered. gh-wj is the branch that deploys to the reference site, so entering them here is what
entered. The deployment branch is the branch that deploys to the reference site, so entering them here is what
closes the window.
### 6. ADR-001 / ADR-002: raw SQL over core tables from a plugin
@@ -503,7 +519,7 @@ Recorded because it is real and someone will look for it here.
vendored Redoc at `/api/docs`, specifically for the air-gapped production box.
The tracked spec holds 377 paths and `info.version` is still `0.12.0`. It
contains zero paths matching `enroll` or `bootstrap`, while
`git diff 2ac2e6d33..gh-wj -- 'plugins/*/api/*.py' 'shopdb/core/api/*.py'` adds
`git diff 2ac2e6d33..HEAD -- 'plugins/*/api/*.py' 'shopdb/core/api/*.py'` adds
24 route decorators, including the whole enrollment surface
(`plugins/endpoints/api/enrollment.py:289`, `:580`, `:657`, `:752`, `:773`,
`:827`, `:853`, `:1095`, `:1135`, `:1233`, `:1253`) and the bootstrap-token
@@ -520,14 +536,14 @@ this branch: `docs/api-inventory.json` is not tracked here, so
`tests/test_api_inventory_parity.py` skips and `scripts/gen_openapi.py` has no
input.
**`CLAUDE.md` in this repo contradicts the repo.** Line 3 and line 9 describe
**The assistant-instructions file in this repo contradicts the repo.** Line 3 and line 9 describe
the database as "PostgreSQL 15+", while `docker-compose.yml:20` runs
`mysql:8.4` and `.env.example:29` is a `mysql+pymysql://` URL. Line 31 states
"**Columns**: snake_case (e.g., `serial_number`, `last_seen`)", which is the
opposite of `CONTRIBUTING.md:17` ("Column names: lowercase, concatenated,
singular. No dashes, no underscores") and of what the gate enforces at
`scripts/check-naming-and-style.sh:236`. Line 119 carries the ADR-016 device
token misattribution described above. `CLAUDE.md` is outside this folder and was
token misattribution described above. That file is outside this folder and was
not edited.
## Scope note

19
docs/wiki-drafts/Architectural-Findings-2026-09-11.md
View File

@@ -1,8 +1,23 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-09-11 architecture findings with a 2026-09-18 verdict on each; most were refuted.
loadwith: none
-->
# Architectural Findings - September 11, 2026
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Scope:** Full codebase analysis via Gortex semantic code intelligence
**Original severity claim:** 22 findings (6 critical, 6 high, 10 medium)
**Status:** RE-CHECKED against branch gh-wj at 35d4fcd41 on 2026-09-18. Most of
**Status:** RE-CHECKED against the deployment branch at 35d4fcd41 on 2026-09-18. Most of
the original findings do not survive a read of the code. This page keeps them,
with the verdict and the evidence, so nobody acts on a claim that was never
true.
@@ -380,7 +395,7 @@ Re-checked on 2026-09-18:
## Testing Performed (2026-09-18 re-check)
```bash
# Every claim above was checked with a grep or a read on gh-wj at 35d4fcd41.
# Every claim above was checked with a grep or a read on the deployment branch at 35d4fcd41.
grep -rn "^from plugins\." plugins/ --include=*.py # 0 hits
grep -rn "from shopdb\.\(core\|utils\|extensions\|services\)" \
plugins/ --include=*.py # 0 hits

21
docs/wiki-drafts/Maintenance-Footprint-Baseline.md
View File

@@ -1,9 +1,24 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Point-in-time measurement of what maintaining this codebase costs, annotated where the tree moved on.
loadwith: none
-->
# Maintenance Footprint Baseline - 2026-09-11
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Total annual burden:** 936 hours/year (estimate, see the note below)
**Risk level:** originally stated as 8 critical + 10 high operational risks
**Optimization potential:** 147-205 hours/year savings with automation
**Last checked against code:** 2026-09-18, branch gh-wj at 35d4fcd41
**Last checked against code:** 2026-09-18, the deployment branch at 35d4fcd41
## About the numbers (added 2026-09-18)
@@ -140,7 +155,7 @@ Backup credential redaction landed in the same commit as audit retention
**Time breakdown per deploy:**
- Bundle creation: 15 minutes (build frontend, tar source, verify integrity)
- Transfer dev->work: 10 minutes (USB copy + verification)
- GitHub push: 5 minutes (pull wilm, merge to wj, resolve conflicts, push)
- GitHub push: 5 minutes (pull the upstream branch, merge down, resolve conflicts, push)
- Transfer work->VM: 10 minutes (USB or shared folder)
- VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test)
- Rollback (if needed): +60 minutes (restore snapshot, rollback migrations)
@@ -438,7 +453,7 @@ rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/
`:218`), filtering out `migrations/versions/`, `frontend/dist/`, `staticdocs/`
and `.plugins-staged/`. Setting `USE_GIT_LS_FILES=0` falls back to recursive
grep with `--exclude-dir` for non-git checkouts. The whole script exits 0 on
gh-wj at 35d4fcd41.
the deployment branch at 35d4fcd41.
**The 5x / 10s-to-2s claim is unsourced** and no timing was recorded for this
page. The savings line below it follows from it and is equally unsourced.

39
docs/wiki-drafts/README.md
View File

@@ -1,9 +1,24 @@
<!-- docmeta
purpose: reference
audience: developer
status: historical
summary: Index of the 2026-09-11 audit drafts, what each one got wrong, and the 2026-09-18 corrections.
loadwith: none
-->
# Wiki Page Drafts - 2026-09-11, revised 2026-09-18
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
This directory contains drafted wiki pages based on work completed Q3 2026. Copy
these to the GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki.
**Revised 2026-09-18, branch gh-wj at 35d4fcd41.** A full review re-checked every
**Revised 2026-09-18, the deployment branch at 35d4fcd41.** A full review re-checked every
page here against the code. `Architectural-Findings-2026-09-11.md` was
substantially wrong and now carries a verdict and evidence per finding.
`Maintenance-Footprint-Baseline.md` is annotated where the tree has moved on.
@@ -116,7 +131,7 @@ than "device token auth". Docker log rotation is still genuinely missing:
**Remediation timeline:** Q4 2026 - Q4 2027
**2026-09-18: do not publish this page as it stood.** Re-checking it against
gh-wj refuted most of it. There is no module-level cross-plugin import anywhere
the deployment branch refuted most of it. There is no module-level cross-plugin import anywhere
(`grep -rn "^from plugins\." plugins/ --include=*.py` = 0 hits) and `machines`
does not import `computers` at all, so finding 1's cycle does not exist. No
plugin deep-imports `shopdb.core` (0 hits), so finding 2 is refuted outright. No
@@ -131,12 +146,12 @@ form.
### 5. ADR Conformance - September 18, 2026
**File:** `ADR-Conformance-2026-09-18.md`
**Wiki path:** `ADRs/ADR-Conformance-2026-09-18`
**Summary:** ADR-by-ADR result of the 2026-09-18 review of gh-wj at 35d4fcd41.
**Summary:** ADR-by-ADR result of the 2026-09-18 review of the deployment branch at 35d4fcd41.
Fourteen ADRs hold with cited evidence; eight items are recorded as violations
or gaps, each with a file and line.
**Reads first, before the table:** the ADR set does not exist on this branch.
`ls docs/adr` fails on gh-wj; ADR-001 through ADR-018 live only on the other
`ls docs/adr` fails on the deployment branch; ADR-001 through ADR-018 live only on the other
lineage. Nothing here states the rules and nothing here enforces them except
`scripts/check-naming-and-style.sh`, which names exactly two ADRs (ADR-017 at
line 343, ADR-015 at line 395). A handful of tests cover ADR ground without
@@ -263,7 +278,7 @@ Dev PC (the internal git server) <--USB--> Work PC (GitHub) <--USB--> Production
```bash
# On work PC
cd C:\Users\270015376\Documents\github\flask-shopdb
cd C:\Users\<you>\Documents\github\flask-shopdb
./pull-shopdb-bundle.cmd # Creates tarball for transfer
```
@@ -337,9 +352,9 @@ Gate tests validate generated documentation and API consistency:
git add docs/openapi.json
```
**Caveat for the wj branch (2026-09-18):** the parity gate cannot run here.
**Caveat for the deployment branch (2026-09-18):** the parity gate cannot run here.
`tests/test_api_inventory_parity.py` skips itself when `docs/api-inventory.json`
is absent, and that file is not tracked on gh-wj, so `scripts/gen_openapi.py`
is absent, and that file is not tracked on the deployment branch, so `scripts/gen_openapi.py`
has no input either. `docs/openapi.json` on this branch is 377 paths at
`info.version` 0.12.0 and documents none of the 24 routes added since
2ac2e6d33. Regenerating it needs the inventory file from the other lineage.
@@ -408,7 +423,7 @@ reads the collector key from `HKLM:\SOFTWARE\GE\ShopDB`, and
pins to the same key by design. Publishing the old text would have told the next
person to delete values the fleet depends on.
**What is actually built, as of gh-wj 35d4fcd41:**
**What is actually built, as of the deployment branch at 35d4fcd41:**
- **Per-PC endpoint credentials.** An enrolled PC authenticates with
`X-Endpoint-Credential` (`plugins/endpoints/api/agentauth.py`). The server
@@ -563,7 +578,7 @@ yet.
## Next Steps
1. Read the 2026-09-18 notes inside each page before pasting anything. All five
pages in this folder have now been checked against gh-wj at 35d4fcd41.
pages in this folder have now been checked against the deployment branch at 35d4fcd41.
2. Copy to GitHub wiki at https://github.com/ge-aero/shopdb-flask/wiki
3. Update wiki navigation/sidebar to include the new pages
4. Link related pages. In particular, `Security-Audit-2026-09-11.md` finding 31
@@ -574,12 +589,12 @@ yet.
**Three corrections that live outside this folder** and were therefore NOT made:
- `CLAUDE.md:3` and `:9` say PostgreSQL 15+; `docker-compose.yml:20` runs
- the assistant-instructions file, lines 3 and 9, say PostgreSQL 15+; `docker-compose.yml:20` runs
`mysql:8.4` and `.env.example:29` is a `mysql+pymysql://` URL.
- `CLAUDE.md:31` says columns are snake_case; `CONTRIBUTING.md:17` says
- the assistant-instructions file, line 31 says columns are snake_case; `CONTRIBUTING.md:17` says
lowercase concatenated with no underscores, and that is what
`scripts/check-naming-and-style.sh:236` enforces.
- `CLAUDE.md:119` and the assistant quick-reference file, lines 14 and 135 repeat the
- the assistant-instructions file, line 119 and the assistant quick-reference file, lines 14 and 135 repeat the
ADR-016 device-token misattribution.
---

27
docs/wiki-drafts/Security-Audit-2026-09-11.md
View File

@@ -1,7 +1,22 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-09-11 security audit with a 2026-09-18 verdict per finding and thirteen added ones.
loadwith: none
-->
# Security Audit - September 11, 2026, refreshed September 18, 2026
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Scope:** Endpoints plugin, C# agent, system tray application, and (new on 2026-09-18) the USB and printed-parts kiosks
**Branch reviewed:** `gh-wj` at `35d4fcd41` (2026-09-16). Range re-read for this refresh: `2ac2e6d33..gh-wj`, 38 commits, 2026-09-11 to 2026-09-16
**Branch reviewed:** the deployment branch at `35d4fcd41` (2026-09-16). Range re-read for this refresh: `2ac2e6d33..HEAD`, 38 commits, 2026-09-11 to 2026-09-16
**Severity, 2026-09-11 set:** 4 critical, 6 high, 8 medium
**Severity, added 2026-09-18:** 1 high, 4 medium, 9 low
**Status:** see the remediation summary. Every 2026-09-11 finding below now carries a verdict read off the code on this branch, with a file and line.
@@ -138,7 +153,7 @@ Partial compensating control added since: the app now sets `X-Content-Type-Optio
### 14-18. Additional Medium Findings [PARTLY ADDRESSED, SOURCE NOW MISSING]
**2026-09-18 verdict:** the file this section pointed at is gone. the assistant memory directory does not exist on this branch; commit `8eee6f19d` moved the 2026-09-12 audit reports and scan output out of the product tree on purpose. The detail behind findings 14 to 18 is therefore not readable from here.
**2026-09-18 verdict:** the file this section pointed at is gone. That assistant memory directory does not exist on this branch; commit `8eee6f19d` moved the 2026-09-12 audit reports and scan output out of the product tree on purpose. The detail behind findings 14 to 18 is therefore not readable from here.
Of the five items summarised in the original bullet list, two can be answered from the code:
@@ -261,13 +276,13 @@ Three operator-facing places enumerate what depends on the `X-Forwarded-For` rew
### 31. This page, and three other documents, attributed a credential policy to ADR-016 that ADR-016 does not contain
The References section of the 2026-09-11 version of this page described ADR-016 as "Credential Delivery (device token authentication)". That is wrong, and the same misattribution is repeated at `docs/wiki-drafts/README.md:316-323`, `CLAUDE.md:119` and the assistant quick-reference file line 135. Grepping the ADR set for device tokens returns nothing in any ADR.
The References section of the 2026-09-11 version of this page described ADR-016 as "Credential Delivery (device token authentication)". That is wrong, and the same misattribution is repeated at `docs/wiki-drafts/README.md:316-323`, the assistant-instructions file, line 119 and the assistant quick-reference file line 135. Grepping the ADR set for device tokens returns nothing in any ADR.
The worst form is in the SECURITY.md page that `docs/wiki-drafts/README.md:429` queues for publication. Under "ADR-016: Credential Delivery" it lists device tokens, personal access tokens and environment variables as the delivery mechanisms and then bans, at `docs/wiki-drafts/README.md:323`, "Hardcoded in source, registry keys, git history". ADR-016 decides none of that. Its Decision is a fleet credential store with Fernet at rest, a targeting and precedence resolver reusing `plugins/geenforce/filters.py`, an explain endpoint and `GET /api/collector/credentials` behind a dedicated `credentials.fetch` scope. Its own alternatives section recommends registry-only provisioning as a worthwhile first step.
The blanket ban on registry keys also contradicts shipped code on this branch: `plugins/computers/client/Report-AssetToShopDB.ps1:367-374` reads the collector key from `HKLM:\SOFTWARE\GE\ShopDB`, and `plugins/endpoints/client/Set-ShopDBServerPins.ps1:4-20` writes the server trust pins there by design.
**Status:** the References section of this page is corrected below. The other three are not, and the SECURITY.md draft should not be published as written. Note that `gh-wj` carries no `docs/adr` directory at all, so there is no ADR text on this branch to correct a reader who follows the wiki.
**Status:** the References section of this page is corrected below. The other three are not, and the SECURITY.md draft should not be published as written. Note that the deployment branch carries no `docs/adr` directory at all, so there is no ADR text on this branch to correct a reader who follows the wiki.
**Related, and unverifiable from this branch:** ADR-016's own mechanism is still unbuilt. Grepping `shopdb/` and `plugins/` for `credentials.fetch`, `collector/credentials` and `Fernet` returns nothing, and `plugins/endpoints/plugin.py:163-253` declares only `endpoints.*` permissions. The per-PC credential programme in this range is a different subject under the same word: the agent's own identity secret, not a store that delivers third-party credentials to a PC. The design is sound and consistent with ADR-016's intent, but nothing on this branch records it as a decision. That record belongs on the lineage that holds `docs/adr`.
@@ -402,5 +417,5 @@ git ls-files | xargs grep -l "shopdb_pat_" | wc -l
- ADR-006: Collector API, the single dynamic dispatch route and its response contract
- ADR-015: Site-specific configuration, and the 2026-08-14 amendment that set the scan's scope
- ADR-016: Credential Delivery. A fleet credential store with Fernet at rest and a `credentials.fetch` scope. It is **not** about device tokens, and it does not ban registry storage. Still unbuilt on this branch. See finding 31.
- The ADR set is not in this repository. It lives on the product lineage; `gh-wj` has no `docs/adr` directory.
- The 2026-09-12 detailed audit reports were removed from the product tree by `8eee6f19d` and are kept with the site's working records. the assistant memory directory does not exist here, so the 2026-09-11 page's pointers into it are dead.
- The ADR set is not in this repository. It lives on the product lineage; the deployment branch has no `docs/adr` directory.
- The 2026-09-12 detailed audit reports were removed from the product tree by `8eee6f19d` and are kept with the site's working records. That directory does not exist here, so the 2026-09-11 page's pointers into it are dead.

980
scripts/gen_openapi.py
View File

File diff suppressed because it is too large Load Diff

17
scripts/gen_project_map.py
View File

@@ -29,7 +29,22 @@ from pathlib import Path
REPO = Path(__file__).resolve().parents[1]
OUTPUT = REPO / 'docs' / 'PROJECT-MAP.md'
HEADER = """<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
# The docmeta block is part of the generated output, not something a human
# re-adds afterwards. This script rewrites the page whole, so anything hand-added
# above the GENERATED comment survives exactly until the next regeneration - and
# tests/test_docs_index.py requires the block to be the FIRST thing in the file,
# so losing it turns a routine regeneration into a red docs gate. Emitting it
# here is the only arrangement where "regenerate" and "the gate is green" are
# the same action. Keep the summary under the 120-character limit that test
# enforces, and keep the field vocabulary in step with it.
HEADER = """<!-- docmeta
purpose: reference
audience: developer, adopter
status: current
summary: Generated inventory - versions, plugins, Alembic heads, ADR index and codebase size.
loadwith: none
-->
<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
Regenerate with: venv/bin/python scripts/gen_project_map.py -->
# Project map

32
shopdb/config.py
View File

@@ -185,6 +185,22 @@ class Config:
AUTH_RATELIMIT_WINDOW_SECONDS = int(
os.environ.get('AUTH_RATELIMIT_WINDOW_SECONDS', 300))
# Swagger UI "try it" page at /api/docs/try, beside the read-only Redoc
# page at /api/docs. One spec, two renderers.
#
# Off unless a site turns it on, and that is the point rather than caution:
# Swagger UI's Execute button fires a REAL request at THIS server from the
# operator's browser, carrying whatever session that browser already holds.
# On a production box that is a one-click way to POST against live fleet
# data from a page anyone can open. Redoc, which cannot send anything, stays
# the default and is always on.
#
# When off the routes are never bound at all, so /api/docs/try is Flask's
# own 404 and the Redoc page does not mention it. See
# shopdb/core/api/docs.py.
API_DOCS_TRYIT_ENABLED = os.environ.get(
'API_DOCS_TRYIT_ENABLED', 'false').lower() == 'true'
DEFAULT_PAGE_SIZE = 20
MAX_PAGE_SIZE = 100
@@ -214,6 +230,12 @@ class DevelopmentConfig(Config):
DEBUG = True
SQLALCHEMY_ECHO = True
# On in development: firing the request is the whole reason a developer
# opens the API docs. Still read from the environment, so a shared dev
# instance can turn it back off without editing this file.
API_DOCS_TRYIT_ENABLED = os.environ.get(
'API_DOCS_TRYIT_ENABLED', 'true').lower() == 'true'
SQLALCHEMY_DATABASE_URI = os.environ.get(
'DATABASE_URL',
'mysql+pymysql://root:changeme@127.0.0.1:3306/shopdb_flask',
@@ -234,6 +256,16 @@ class TestingConfig(Config):
# xdist distribution, so the failure moves around. The tests that are about
# the limiter turn it on themselves.
ENDPOINTS_ENROLL_RATELIMIT_ENABLED = False
# Off, and stated here rather than inherited, because a test suite that
# silently ran with it on would be testing a configuration no site ships.
# The session-wide app in tests/conftest.py is what
# tests/test_core/test_public_read_surface.py sweeps for ungated GET
# routes, and the try-it page plus its two vendored assets are
# deliberately unauthenticated. Keeping the default here means that sweep
# keeps describing what production actually exposes; the tests that ARE
# about the page build their own app with the flag on
# (tests/test_core/test_api_docs_tryit.py).
API_DOCS_TRYIT_ENABLED = False
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
SQLALCHEMY_ENGINE_OPTIONS = {
'connect_args': {'check_same_thread': False},

245
shopdb/core/api/docs.py
View File

@@ -4,11 +4,21 @@ Served at /api/docs (relative to the mount). The Redoc bundle is vendored in
staticdocs/ so this works fully offline on the air-gapped prod box - no CDN.
The spec is docs/openapi.json in the repo (regenerate with
scripts/gen_openapi.py after API changes).
Two renderers over ONE spec. Redoc is read-only by design and is the default,
because what is published is a contract. Swagger UI at /api/docs/try adds the
one thing Redoc will not do: fire a real request from the page, against this
server, with whatever session the browser is already carrying. That is its
whole value and the whole reason it is off unless a site turns it on - see
API_DOCS_TRYIT_ENABLED in shopdb/config.py.
"""
import json
import os
from flask import Blueprint, Response, send_file, url_for
from markupsafe import escape
from flask import Blueprint, Response, current_app, request, send_file, url_for
docs_bp = Blueprint('docs', __name__)
@@ -19,32 +29,168 @@ _SPEC = os.path.join(_REPO_ROOT, 'docs', 'openapi.json')
_LLMS = os.path.join(_REPO_ROOT, 'docs', 'llms.txt')
_REDOC = os.path.join(_HERE, 'staticdocs', 'redoc.standalone.js')
# Vendored from swagger-ui-dist 5.33.0. TWO files out of the ~30 in that dist,
# because those are the two the page below actually loads:
# swagger-ui.css - the stylesheet. Its images are inline data: URIs,
# so it pulls nothing else. The trailing
# sourceMappingURL comment was stripped: the .map
# is not vendored, and left in place it turns every
# devtools open into a 404 on a box with no
# internet to fall back on.
# swagger-ui-bundle.min.js - the renderer plus the core presets, self-
# contained. NOT vendored beside it:
# swagger-ui-standalone-preset.js (the topbar with
# the "load any spec URL" box - this page serves
# one spec on purpose), the ES bundles (module
# builds of the same code), the .map files (~10 MB
# of source maps that only devtools reads),
# oauth2-redirect.html (no OAuth flow here), and
# the favicons.
# The upstream name is swagger-ui-bundle.js; the vendored copy carries .min.js,
# which is what it is - a minified single-line build. That suffix is also what
# scripts/check-naming-and-style.sh already filters vendored bundles by, so the
# style gate does not report thousands of hits in code nobody here wrote.
# swagger-ui.LICENSE.txt is the Apache-2.0 text that ships with the dist. It is
# never served; it is here because redistributing the bundle requires it.
_SWAGGER_CSS = os.path.join(_HERE, 'staticdocs', 'swagger-ui.css')
_SWAGGER_JS = os.path.join(_HERE, 'staticdocs', 'swagger-ui-bundle.min.js')
_TRYIT_ENDPOINT = 'docs.swagger_tryit'
def _tryit_is_mounted():
"""True when the try-it page exists on THIS app.
Reads the url map rather than the config flag on purpose: the routes are
bound once, at blueprint registration, so a config value changed after boot
would have the Redoc page advertise a link that answers 404.
"""
return _TRYIT_ENDPOINT in current_app.view_functions
@docs_bp.route('/', strict_slashes=False)
def docs_index():
"""Redoc page. url_for keeps the asset URLs correct under any mount."""
spec_url = url_for('docs.openapi_spec')
redoc_url = url_for('docs.redoc_js')
# Mentioned only when it is actually there. An unconditional note would be
# an advert for a 404 on every box that ships the shipped default.
tryit_note = ''
if _tryit_is_mounted():
tryit_note = (
'<p class="docs-note">'
'This page is read-only. To send a real request against this '
'server, use <a href="%s">the try-it view</a>.</p>'
% url_for(_TRYIT_ENDPOINT)
)
# DARK MODE. Redoc does not read prefers-color-scheme on its own, and
# the <redoc spec-url> element takes no theme - so the page goes through
# Redoc.init instead, which does. The app has a dark theme and this was
# the one page that stayed white when the rest of the site went dark.
#
# The page RE-INITIALISES when the OS scheme flips rather than restyling:
# Redoc bakes its colours into styled-components at init, so a CSS
# override reaches the chrome and misses the content.
page = (
'<!doctype html><html><head>'
'<title>ShopDB API</title>'
'<meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
'<style>body{margin:0;padding:0}</style></head><body>'
'<redoc spec-url="%s"></redoc>'
'<style>'
':root{color-scheme:light dark}'
'body{margin:0;padding:0;background:#fff}'
'.docs-note{font:14px/1.4 system-ui,sans-serif;margin:0;'
'padding:10px 16px;background:#eef2f7;color:#0f172a;'
'border-bottom:1px solid #cbd5e1}'
'.docs-note a{color:#1d4ed8}'
'@media (prefers-color-scheme: dark){'
'body{background:#0f172a}'
'.docs-note{background:#1e293b;color:#e2e8f0;'
'border-bottom-color:#334155}'
'.docs-note a{color:#7dd3fc}'
'}'
'</style></head><body>'
'%s'
'<div id="redoc"></div>'
'<script src="%s"></script>'
'</body></html>' % (spec_url, redoc_url)
'<script>'
'var SPEC = %s;'
# Redoc computes text and the right-hand sample panel from the
# background and gets both wrong on a dark one, so they are named
# rather than left to it.
'var DARK={colors:{primary:{main:"#7dd3fc"},'
'text:{primary:"#e2e8f0",secondary:"#94a3b8"},'
'border:{dark:"#334155",light:"#1e293b"}},'
'sidebar:{backgroundColor:"#111827",textColor:"#e2e8f0",'
'activeTextColor:"#7dd3fc"},'
'rightPanel:{backgroundColor:"#020617",textColor:"#e2e8f0"},'
'codeBlock:{backgroundColor:"#0b1220"},'
'schema:{nestedBackground:"#0b1220"},'
'typography:{fontFamily:"system-ui,sans-serif",'
'code:{color:"#e2e8f0",backgroundColor:"#0b1220"}}};'
'var LIGHT={colors:{primary:{main:"#1d4ed8"}},'
'typography:{fontFamily:"system-ui,sans-serif"}};'
'var q=window.matchMedia("(prefers-color-scheme: dark)");'
'function draw(){'
'document.getElementById("redoc").innerHTML="";'
'Redoc.init(SPEC,{theme:q.matches?DARK:LIGHT},'
'document.getElementById("redoc"));'
'}'
'draw();'
# addEventListener is missing from MediaQueryList in older WebViews,
# and a shop-floor kiosk browser is exactly where this gets read.
'if(q.addEventListener){q.addEventListener("change",draw);}'
'else if(q.addListener){q.addListener(draw);}'
'</script>'
'</body></html>' % (tryit_note, redoc_url, json.dumps(spec_url))
)
return Response(page, mimetype='text/html')
_SPEC_CACHE = {}
def _spec_with_server():
"""The generated spec, with `servers` set to where this instance is mounted.
THE MOUNT IS A DEPLOYMENT FACT, NOT A BUILD FACT. gen_openapi.py writes
servers: [{"url": "/"}] because it cannot know where the app will be hung,
and wsgi.py moves MOUNT_PATH from PATH_INFO into SCRIPT_NAME at runtime. So
an instance under /shopdb served a spec whose paths start at /api, and both
renderers dutifully built https://host/api/apitokens - a URL that 404s.
Anyone who copied it out of the docs page got a broken call, and Swagger's
try-it fired at the wrong place.
request.script_root is the runtime answer: '' for a root install, '/shopdb'
under the subpath method, and correct behind a proxy that sets the prefix.
The file is ~1 MB, so it is parsed once and cached by mtime; each request
gets a shallow copy with only `servers` replaced, which keeps two requests
on different mounts from racing on one dict.
"""
stamp = os.path.getmtime(_SPEC)
if _SPEC_CACHE.get('stamp') != stamp:
with open(_SPEC, 'r', encoding='utf-8') as handle:
_SPEC_CACHE['spec'] = json.load(handle)
_SPEC_CACHE['stamp'] = stamp
root = request.script_root or ''
spec = dict(_SPEC_CACHE['spec'])
spec['servers'] = [{
'url': root or '/',
'description': 'this instance',
}]
return spec
@docs_bp.route('/openapi.json')
def openapi_spec():
"""The generated OpenAPI 3.1 spec (machine + LLM readable)."""
if not os.path.isfile(_SPEC):
return Response('{"error":"openapi.json not generated"}',
status=404, mimetype='application/json')
return send_file(_SPEC, mimetype='application/json')
return Response(json.dumps(_spec_with_server()),
mimetype='application/json')
@docs_bp.route('/redoc.standalone.js')
@@ -59,3 +205,92 @@ def llms_txt():
if not os.path.isfile(_LLMS):
return Response('llms.txt not found', status=404, mimetype='text/plain')
return send_file(_LLMS, mimetype='text/plain')
def swagger_tryit():
"""Swagger UI over the SAME spec the Redoc page renders.
Every URL on the page comes from url_for, so the spec and both assets stay
same-origin and correct under a mount path (prod runs under /shopdb).
"""
spec_url = url_for('docs.openapi_spec')
css_url = url_for('docs.swagger_css')
js_url = url_for('docs.swagger_js')
redoc_page_url = url_for('docs.docs_index')
page = (
'<!doctype html><html><head>'
'<title>ShopDB API - try it</title>'
'<meta charset="utf-8">'
'<meta name="viewport" content="width=device-width, initial-scale=1">'
'<link rel="stylesheet" href="%(css)s">'
'<style>body{margin:0;padding:0}'
'.tryit-warn{font:14px/1.4 system-ui,sans-serif;background:#7f1d1d;'
'color:#fff;margin:0;padding:10px 16px}'
'.tryit-warn a{color:#fff}</style>'
'</head><body>'
'<p class="tryit-warn"><b>Execute sends a real request to this '
'server</b>, as whoever is signed in on this browser. A write '
'operation changes live data. The read-only contract is at '
'<a href="%(redoc)s">/api/docs</a>.</p>'
'<div id="swagger-ui"></div>'
'<script src="%(js)s"></script>'
'<script>\n'
'window.ui = SwaggerUIBundle({\n'
# json.dumps, NOT a bare "%s". SCRIPT_NAME is proxy-controlled and
# ProxyFix honours X-Forwarded-Prefix, so a mount path carrying a quote
# closed this string literal and executed what followed: SCRIPT_NAME of
# /a"+alert(1)+" rendered url: "/a"+alert(1)+"/api/docs/openapi.json".
# Escaping it as JSON is what makes the value data rather than code.
' url: %(spec)s,\n'
' dom_id: "#swagger-ui",\n'
' presets: [SwaggerUIBundle.presets.apis],\n'
' layout: "BaseLayout",\n'
' deepLinking: true,\n'
' filter: true,\n'
# 573 operations. Expanded by default, the page spends many seconds
# laying out before it answers a scroll.
' docExpansion: "none",\n'
' withCredentials: true,\n'
# THE air-gap setting. Left unset, Swagger UI renders a "valid" badge
# by handing this spec to the public swagger validator - an outbound
# request that cannot succeed here, and that would ship the internal
# API surface off the box if it ever could.
' validatorUrl: null\n'
'});\n'
'</script>'
'</body></html>'
% {'css': escape(css_url), 'js': escape(js_url),
'spec': json.dumps(spec_url),
'redoc': escape(redoc_page_url)}
)
return Response(page, mimetype='text/html')
def swagger_css():
"""Vendored Swagger UI stylesheet (offline)."""
return send_file(_SWAGGER_CSS, mimetype='text/css')
def swagger_js():
"""Vendored Swagger UI bundle (offline)."""
return send_file(_SWAGGER_JS, mimetype='application/javascript')
@docs_bp.record_once
def _mount_tryit(state):
"""Bind the try-it routes only when the flag is on for this app.
Binding at registration rather than checking the flag inside the view, so
that when it is off the routes DO NOT EXIST: the URL gets Flask's own 404
and confirms nothing, and the unauthenticated-GET sweep in
tests/test_core/test_public_read_surface.py never has to make an exception
for three routes that are only meant to be reachable on a dev box.
"""
if not state.app.config.get('API_DOCS_TRYIT_ENABLED'):
return
state.add_url_rule('/try', endpoint='swagger_tryit',
view_func=swagger_tryit, strict_slashes=False)
state.add_url_rule('/swagger-ui.css', endpoint='swagger_css',
view_func=swagger_css)
state.add_url_rule('/swagger-ui-bundle.min.js', endpoint='swagger_js',
view_func=swagger_js)

2
shopdb/core/api/staticdocs/redoc.standalone.js vendored
View File

File diff suppressed because one or more lines are too long

2
shopdb/core/api/staticdocs/swagger-ui-bundle.min.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

202
shopdb/core/api/staticdocs/swagger-ui.LICENSE.txt Normal file
View File

@@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

2
shopdb/core/api/staticdocs/swagger-ui.css vendored Normal file
View File

File diff suppressed because one or more lines are too long

60
tests/test_api_inventory_parity.py
View File

@@ -183,3 +183,63 @@ def test_documented_auth_is_what_the_code_enforces():
'inventory is derived, so fix the code or teach the classifier in '
'scripts/gen_openapi.py, then re-run it with --refresh:\n %s'
% (len(disagreements), '\n '.join(disagreements[:15])))
# ---------------------------------------------------------------------------
# The third derived field: query parameters
# ---------------------------------------------------------------------------
#
# A Flask rule carries its path arguments and nothing else, so a route whose
# real interface is '?code=' or '?search=' documented no parameters at all -
# /api/usb/kiosk/setup among them, where '?code=' is the ONLY way to call it
# successfully. gen_openapi.py now reads them out of each view's AST and writes
# them into the inventory as `queryparams`.
#
# That field is DERIVED, like auth, and gated the same way and for the same
# reason: a refresh only rewrites, so without a comparison a hand-edit or a
# stale file publishes a query contract the handler does not implement. A
# parameter that is not really there is the worst kind of documentation defect,
# because a reader believes it, sends it, and is silently ignored.
def derived_queryparams():
"""{(METHOD, path): [parameter, ...]} as the sweep reads the code now."""
generator = _generator()
app = create_app('testing')
derived = {}
for rule, methods, endpoint, view in generator._served_routes(app):
parameters = generator._query_parameters(view)
for method in methods:
derived[(method, normalise(rule))] = parameters
return derived
def documented_queryparams():
"""{(METHOD, path): [parameter, ...]} as the inventory has it committed."""
documented = {}
for surface in json.loads(INVENTORY.read_text()):
for entry in surface.get('endpoints', []):
path = normalise(entry.get('path') or '')
for method in re.split(r'[/,|]', entry.get('method') or 'GET'):
method = method.strip().upper()
if method:
documented[(method, path)] = entry.get('queryparams') or []
return documented
def test_documented_query_parameters_are_what_the_handler_reads():
derived = derived_queryparams()
documented = documented_queryparams()
disagreements = []
for key in sorted(set(derived) & set(documented)):
if derived[key] != documented[key]:
disagreements.append(
'%s %s\n documented: %s\n read: %s'
% (key[0], key[1],
[item['name'] for item in documented[key]],
[item['name'] for item in derived[key]]))
assert not disagreements, (
'%d route(s) document query parameters that are not what the handler '
'reads. The field is derived, so regenerate with '
'scripts/gen_openapi.py --refresh:\n %s'
% (len(disagreements), '\n '.join(disagreements[:15])))

283
tests/test_core/test_api_docs_tryit.py Normal file
View File

@@ -0,0 +1,283 @@
"""The Swagger UI "try it" page at /api/docs/try.
Two things are being proved, and they are the two that would hurt on the
production box:
1. It is OFF unless a site turns it on, and off means the URL does not exist.
The page fires real requests at this server with the browser's own session,
so a disabled deployment must not serve it, advertise it, or confirm it is
a thing by answering anything other than 404.
2. Nothing it loads comes from the internet. Production is air-gapped: a CDN
reference, a webfont, or a source map pointing at a public host is not a
slow page, it is a page that never renders and a request that hangs.
These build their own Flask app around the docs blueprint rather than using the
session app from conftest. The blueprint touches no database, and a second
create_app() would rebind the SQLAlchemy extension and break whatever module
runs next (see the note on the gated_app fixture in test_public_read_surface).
It also lets one test see the flag on and the next see it off, which is the
whole behaviour under test.
"""
import os
import re
import pytest
from flask import Flask
from shopdb.core.api.docs import docs_bp
# Everything the try-it page is allowed to load, and the files behind them.
_CSS_PATH = '/api/docs/swagger-ui.css'
_JS_PATH = '/api/docs/swagger-ui-bundle.min.js'
_STATICDOCS = os.path.join(
os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
'shopdb', 'core', 'api', 'staticdocs')
_VENDORED = ('swagger-ui.css', 'swagger-ui-bundle.min.js')
def _docs_app(tryit_enabled):
"""A bare app carrying only the docs blueprint, mounted where the real app
mounts it (register_blueprints uses /api/<name>)."""
application = Flask(__name__)
application.config['API_DOCS_TRYIT_ENABLED'] = tryit_enabled
application.register_blueprint(docs_bp, url_prefix='/api/docs')
return application
@pytest.fixture
def enabled_client():
return _docs_app(True).test_client()
@pytest.fixture
def disabled_client():
return _docs_app(False).test_client()
def test_enabled_page_renders_swagger_over_the_published_spec(enabled_client):
response = enabled_client.get('/api/docs/try')
assert response.status_code == 200
page = response.get_data(as_text=True)
assert 'SwaggerUIBundle(' in page
assert 'id="swagger-ui"' in page
# One spec, two renderers: the try-it page must read the same document the
# Redoc page serves, not a second copy that could drift from it.
assert '/api/docs/openapi.json' in page
def test_the_page_references_no_other_origin_at_all(enabled_client):
"""Absolute AND protocol-relative.
The first version of this scanned for https?://, so
<script src="//cdn.jsdelivr.net/npm/x.js"> - the form a copy-paste from any
CDN's own docs produces - matched nothing and passed.
"""
page = enabled_client.get('/api/docs/try').get_data(as_text=True)
offenders = re.findall(r'(?:src|href)\s*=\s*[\'"]((?:https?:)?//[^\'"]+)', page)
assert not offenders, 'the try-it page loads from another origin: %s' % offenders
def test_the_redoc_page_survives_try_it_being_unmounted(app):
"""The banner linking to the try-it page is built with url_for, and url_for
raises BuildError when those routes are not registered - which is every
production box. An unconditional link therefore 500s /api/docs/ in exactly
the deployment nobody tests locally.
"""
# The testing config is the unmounted case, and it is the one available
# here: ProductionConfig refuses to build without a real SECRET_KEY, which
# is correct of it and makes it useless as a fixture.
page = app.test_client().get('/api/docs/')
assert page.status_code == 200, page.get_data(as_text=True)[:200]
body = page.get_data(as_text=True)
assert 'redoc' in body.lower()
assert '/api/docs/try' not in body
def test_enabled_page_loads_only_the_vendored_assets(enabled_client):
page = enabled_client.get('/api/docs/try').get_data(as_text=True)
assert 'href="%s"' % _CSS_PATH in page
assert 'src="%s"' % _JS_PATH in page
for path, expected_type in ((_CSS_PATH, 'text/css'),
(_JS_PATH, 'application/javascript')):
asset = enabled_client.get(path)
assert asset.status_code == 200, path
assert expected_type in asset.headers['Content-Type']
assert asset.headers.get('Content-Length', '0') != '0'
def test_the_page_names_no_external_host(enabled_client):
"""Every URL on the page has to be same-origin. On an air-gapped box an
absolute http(s) URL is a request that hangs until it times out."""
page = enabled_client.get('/api/docs/try').get_data(as_text=True)
external = re.findall(r'https?://[^\s"\'<>)]+', page)
assert not external, 'try-it page references external hosts: %s' % external
def test_the_spec_validator_is_turned_off(enabled_client):
"""Swagger UI's one outbound default: unset, it posts the spec to a public
validator to draw a badge. That is a hung request here, and it would be an
upload of the internal API surface anywhere it worked."""
page = enabled_client.get('/api/docs/try').get_data(as_text=True)
assert re.search(r'validatorUrl:\s*null', page), (
'validatorUrl is not disabled; Swagger UI will call the public '
'validator')
def test_disabled_deployment_has_no_such_route(disabled_client):
"""404, not a rendered page explaining that it is disabled. The route is
never bound, so the URL confirms nothing."""
for path in ('/api/docs/try', '/api/docs/try/', _CSS_PATH, _JS_PATH):
assert disabled_client.get(path).status_code == 404, path
def test_the_redoc_page_advertises_try_it_only_when_it_exists(
enabled_client, disabled_client):
enabled_page = enabled_client.get('/api/docs/').get_data(as_text=True)
assert '/api/docs/try' in enabled_page
disabled_page = disabled_client.get('/api/docs/').get_data(as_text=True)
assert '/api/docs/try' not in disabled_page
# The word too, not just the href: a note about a view that 404s is worse
# than no note.
assert 'try-it' not in disabled_page
def test_every_url_follows_the_mount_path(enabled_client):
"""Prod runs under /shopdb. url_for has to carry the prefix onto the spec
and both assets, or the page loads nothing and can validate nothing."""
page = enabled_client.get(
'/api/docs/try',
environ_overrides={'SCRIPT_NAME': '/shopdb'}).get_data(as_text=True)
assert '/shopdb/api/docs/openapi.json' in page
assert '/shopdb%s' % _CSS_PATH in page
assert '/shopdb%s' % _JS_PATH in page
# No unprefixed twin left behind by a hardcoded path.
assert 'href="%s"' % _CSS_PATH not in page
assert 'src="%s"' % _JS_PATH not in page
redoc_page = enabled_client.get(
'/api/docs/',
environ_overrides={'SCRIPT_NAME': '/shopdb'}).get_data(as_text=True)
assert '/shopdb/api/docs/try' in redoc_page
@pytest.mark.parametrize('filename', _VENDORED)
def test_a_vendored_asset_fetches_nothing_from_the_internet(filename):
"""The assets themselves, not just the page.
Scoped to what a BROWSER would actually request. The swagger-ui bundle
carries around a hundred http(s) strings that are prose - RFC and
json-schema citations, licence URLs, the example.com host in the spec's own
documentation - and none of them is fetched, so a blanket "no http in this
file" assertion would be a lie that fails on the next upstream bump. What
would really produce a request is a CDN or webfont reference, a stylesheet
url() pointing off-box, or a source map: hence these.
"""
body = open(os.path.join(_STATICDOCS, filename),
encoding='utf-8', errors='replace').read()
# NOT a host allowlist. The first version of this named five CDNs, so an
# upstream bump to esm.sh or cdn.skypack.dev walked straight through it, and
# the one real leak in this directory was cdn.redoc.ly - which was on no
# list. Match the SHAPES a browser fetches instead, whatever the host:
# a stylesheet import, a css url(), a script/link/img source.
fetching = [
(r'@import\s+(?:url\()?[\'"]?(?:https?:)?//', 'an @import from another origin'),
(r'url\(\s*[\'"]?(?:https?:)?//', 'a css url() on another origin'),
(r'(?:src|href)\s*=\s*[\'"](?:https?:)?//', 'an element loaded from another origin'),
(r'@font-face', 'a webfont, which is a fetch unless it is a data: URI'),
]
for pattern, what in fetching:
for match in re.finditer(pattern, body):
window = body[max(0, match.start() - 60):match.start() + 120]
# data: URIs are inline, not fetched, and this CSS is full of them.
if 'data:' in window[:len(window) // 2 + 40]:
continue
raise AssertionError(
'%s carries %s: ...%s...' % (filename, what, window.strip()[:140]))
# A map reference is a request too. The upstream .map files are not
# vendored (they are tens of megabytes and only devtools reads them), so
# any surviving pointer is a guaranteed 404 on a box that cannot reach the
# internet to try elsewhere.
assert 'sourceMappingURL' not in body, (
'%s still points at a source map that is not vendored' % filename)
# Stylesheet resources: swagger-ui inlines its images as data: URIs, and
# anything else would be an off-box fetch on first paint. CSS only - in the
# minified bundle "url(" is a function call, not a resource.
if filename.endswith('.css'):
for match in re.findall(r'url\(\s*["\']?([^"\')]+)', body):
assert match.startswith('data:'), (
'%s loads %s rather than an inline data: URI'
% (filename, match))
# ---------------------------------------------------------------------------
# The spec has to say where this instance actually lives
# ---------------------------------------------------------------------------
def test_the_spec_declares_the_mount_it_is_served_from(enabled_client):
"""Production runs under /shopdb, and the generated file cannot know that.
gen_openapi.py writes servers: [{"url": "/"}] - correct for a root install
and wrong for every subpath one - and wsgi.py only moves MOUNT_PATH into
SCRIPT_NAME at runtime. Served as written, both renderers built
https://host/api/... for an app mounted at /shopdb: a URL an operator copied
out of the docs page and got a 404 from, and the address Swagger's try-it
fired at.
"""
import json as _json
at_root = _json.loads(
enabled_client.get('/api/docs/openapi.json').get_data(as_text=True))
assert at_root['servers'] == [{'url': '/', 'description': 'this instance'}]
mounted = _json.loads(enabled_client.get(
'/api/docs/openapi.json',
environ_overrides={'SCRIPT_NAME': '/shopdb'}).get_data(as_text=True))
assert mounted['servers'] == [
{'url': '/shopdb', 'description': 'this instance'}]
# The paths themselves stay mount-free: the server prefix is what varies,
# and duplicating it into every path would double it under a subpath.
assert '/api/apitokens' in mounted['paths']
assert '/shopdb/api/apitokens' not in mounted['paths']
# ---------------------------------------------------------------------------
# The read-only page follows the OS theme
# ---------------------------------------------------------------------------
def test_the_redoc_page_follows_the_system_theme(client):
"""The app has a dark theme; /api/docs was the one page that stayed white.
Redoc ignores prefers-color-scheme and the <redoc spec-url> element takes no
theme, so the page has to go through Redoc.init - which is also why this
test pins the mechanism and not just the CSS. A future edit that puts the
element back would lose dark mode without changing a single colour.
"""
page = client.get('/api/docs/').get_data(as_text=True)
assert 'Redoc.init' in page
assert '<redoc spec-url' not in page, 'the element takes no theme'
assert '@media (prefers-color-scheme: dark)' in page
assert 'var DARK=' in page and 'var LIGHT=' in page
# Re-render on a live scheme change, with the fallback for the older
# MediaQueryList that a kiosk browser may still have.
assert 'addEventListener("change",draw)' in page
assert 'q.addListener(draw)' in page
def test_the_docs_page_still_loads_nothing_from_another_origin(client):
"""Theming is the kind of change that quietly introduces a font or a CDN."""
import re as _re
page = client.get('/api/docs/').get_data(as_text=True)
assert not _re.findall(r'(?:src|href)\s*=\s*[\'"](?:https?:)?//', page)

393
tests/test_docs_index.py Normal file
View File

@@ -0,0 +1,393 @@
"""docs/INDEX.md is the router, and a router nobody checks is a router that lies.
The corpus is larger than any reader - person or agent - can hold, so the only
thing that makes it usable is being able to find the minimum correct set of
pages for a task and skip the rest. That needs two things to stay true, and
neither survives on good intentions:
1. Every page declares what it is for and whether it is still current, in a
`docmeta` block a machine can filter on without opening the page.
2. docs/INDEX.md routes every current page, and quarantines the ones that are
superseded or are only a record.
The failure this gate exists to stop is the quiet one: a page is added, nothing
routes to it, and it rots unread - or worse, a page goes `historical` and keeps
being followed because it is still sitting in a task row.
A second pass over the whole corpus added three more, because the first three
turned out to be satisfiable by an index that was still useless:
3. Being NAMED somewhere in INDEX.md is not the same as being routed. A
current page that appears only in a paragraph, or only in the Archive
section's prose, is a page no task row sends anyone to. It must be in a
row.
4. The line counts have to be right, not roughly right. The cost column is
the reason to trust the index over opening pages at random, and a corpus
that is being cut shrinks pages constantly - so a stale number is the
normal failure, not an exotic one.
5. The two routers - INDEX.md and START-HERE.md - must not point at a file
that is not there. Pages get merged and renamed; a dead link in the router
is worse than a missing row, because the reader trusts it.
Line-count tolerance: INDEX.md prints what a row costs to read, so a reader can
see the price before paying it. That number is checked against the real files
within 3 percent or 10 lines, whichever is larger. That absorbs a reflowed
paragraph and nothing else: the tolerance used to be 10 percent or 25 lines,
which let a 52-line row claim 77 and let whole sections drift after an edit
pass. If a row is out by more than a reflow, the number is simply wrong and
recomputing it is one command.
"""
import re
from pathlib import Path
import pytest
REPO = Path(__file__).resolve().parents[1]
DOCS = REPO / 'docs'
INDEX = DOCS / 'INDEX.md'
START_HERE = DOCS / 'START-HERE.md'
# The two routers. Everything else in docs/ is a destination; these two are how
# a reader arrives, so they are exempt from "must sit in a task row" (a row
# pointing at the index the row is in would be noise) and their links are held
# to a stricter standard than anyone else's.
ROUTERS = (INDEX, START_HERE)
# Same skip rule as tests/test_docs_publishable.py: docs/ prose is stripped from
# the published mirror and lives in the wiki, so a checkout with no docs/*.md
# has nothing to route and this module is inapplicable. docs/openapi.json is
# tracked, so keying off the DIRECTORY would make this run on the mirror.
pytestmark = pytest.mark.skipif(
not any(DOCS.glob('*.md')),
reason='no docs/ prose in this checkout - it is published to the wiki instead')
PURPOSES = {'tutorial', 'howto', 'reference', 'explanation'}
AUDIENCES = {'operator', 'adopter', 'developer', 'integrator'}
STATUSES = {'current', 'legacy', 'historical'}
FIELDS = ('purpose', 'audience', 'status', 'summary', 'loadwith')
ARCHIVED = {'legacy', 'historical'}
SUMMARY_LIMIT = 120
# The block must be the FIRST thing in the file, before the H1. A GitHub wiki
# renders YAML front matter as visible junk, which is why this is an HTML
# comment, and a comment further down the page is one a reader trips over.
DOCMETA = re.compile(r'\A<!-- docmeta\n(.*?)\n-->\n', re.S)
LINK = re.compile(r'\[[^\]]*\]\(([^)#]+)(?:#[^)]*)?\)')
ARCHIVE_HEADING = re.compile(r'^##\s+Archive\b', re.M)
ARCHIVE_TOTAL = re.compile(r'they are ([\d,]+) lines')
ALL_LINKS = re.compile(r'\[[^\]]*\]\(([^)]+)\)')
def pages():
"""Every authored page in docs/, including adr/ and proposals/."""
return sorted(p for p in DOCS.rglob('*.md'))
def rel(path):
return path.relative_to(DOCS).as_posix()
def line_count(path):
return len(path.read_text(errors='replace').splitlines())
def parse_docmeta(path):
"""Return the docmeta fields, or None when the block is missing."""
match = DOCMETA.match(path.read_text(errors='replace'))
if not match:
return None
fields = {}
for line in match.group(1).splitlines():
key, sep, value = line.partition(':')
if not sep:
return None
fields[key.strip()] = value.strip()
return fields
def value(fields, key):
"""A field with the template's trailing '# comment' hint removed."""
return re.sub(r'\s+#.*$', '', fields.get(key, '')).strip()
def status_of(path):
fields = parse_docmeta(path)
return value(fields, 'status') if fields else None
def links_in(text):
"""Every in-repo .md target a page links to, as written."""
return [t for t in LINK.findall(text) if t.endswith('.md') and '://' not in t]
def all_targets_in(text):
"""Every relative link target on a page, whatever its extension.
links_in() only sees .md, which is right for routing questions and wrong
for "does this file exist": the index links llms.txt, and a router that
points at a file nobody kept is the failure this catches.
"""
out = []
for target in ALL_LINKS.findall(text):
target = target.split()[0] if target.split() else target
if '://' in target or target.startswith(('mailto:', '#')):
continue
out.append(target.split('#')[0])
return [t for t in out if t]
def resolve(page, target):
"""A link target on `page`, as a path under docs/, or None if it escapes."""
candidate = (page.parent / target).resolve()
try:
candidate.relative_to(DOCS.resolve())
except ValueError:
return None
return candidate
def index_sections():
"""INDEX.md split into (task side, archive side) at the Archive heading."""
text = INDEX.read_text()
match = ARCHIVE_HEADING.search(text)
assert match, 'docs/INDEX.md must carry a "## Archive" section heading'
return text[:match.start()], text[match.start():]
def table_rows(text):
"""(link targets, stated line total) for every table row that states one."""
rows = []
for line in text.splitlines():
line = line.strip()
if not line.startswith('|') or set(line) <= set('|- '):
continue
cells = [c.strip() for c in line.strip('|').split('|')]
if len(cells) < 2 or not cells[-1].isdigit():
continue
targets = links_in(line)
if targets:
rows.append((targets, int(cells[-1])))
return rows
def tolerance(real):
"""How far a stated line count may sit from the real one. See the docstring."""
return max(10, real * 0.03)
def test_there_is_something_to_check():
"""A path change that matched nothing would make this whole file vacuous."""
assert len(pages()) > 20, 'docs/ prose disappeared, or the glob stopped matching'
assert INDEX.is_file(), 'docs/INDEX.md is missing - there is no router'
assert len(table_rows(index_sections()[0])) > 20, 'docs/INDEX.md routes almost nothing'
@pytest.mark.parametrize('page', pages(), ids=rel)
def test_every_page_carries_a_docmeta_block(page):
fields = parse_docmeta(page)
assert fields is not None, (
'%s has no docmeta block. It must be the first thing in the file, '
'before the H1: an HTML comment opening "<!-- docmeta" and closing '
'with "-->", one "key: value" per line.' % rel(page))
missing = [f for f in FIELDS if f not in fields]
assert not missing, '%s docmeta is missing: %s' % (rel(page), ', '.join(missing))
unknown = [f for f in fields if f not in FIELDS]
assert not unknown, '%s docmeta has unknown fields: %s' % (rel(page), ', '.join(unknown))
@pytest.mark.parametrize('page', pages(), ids=rel)
def test_docmeta_uses_the_allowed_vocabularies(page):
fields = parse_docmeta(page)
if fields is None:
pytest.skip('no docmeta - reported by test_every_page_carries_a_docmeta_block')
purpose = value(fields, 'purpose')
assert purpose in PURPOSES, (
'%s: purpose "%s" is not one of %s. A page that is two of these is a '
'page that should be split.' % (rel(page), purpose, sorted(PURPOSES)))
audience = [a.strip() for a in value(fields, 'audience').split(',') if a.strip()]
assert audience, '%s: audience is empty' % rel(page)
bad = [a for a in audience if a not in AUDIENCES]
assert not bad, '%s: audience %s not in %s' % (rel(page), bad, sorted(AUDIENCES))
status = value(fields, 'status')
assert status in STATUSES, (
'%s: status "%s" is not one of %s' % (rel(page), status, sorted(STATUSES)))
@pytest.mark.parametrize('page', pages(), ids=rel)
def test_summary_answers_is_this_the_page_i_need(page):
fields = parse_docmeta(page)
if fields is None:
pytest.skip('no docmeta - reported by test_every_page_carries_a_docmeta_block')
summary = fields.get('summary', '').strip()
assert summary, '%s: summary is empty. It is what a router shows INSTEAD of the page.' % rel(page)
assert len(summary) <= SUMMARY_LIMIT, (
'%s: summary is %d chars, limit is %d. Say what question the page '
'answers, not what it contains.' % (rel(page), len(summary), SUMMARY_LIMIT))
@pytest.mark.parametrize('page', pages(), ids=rel)
def test_loadwith_targets_exist(page):
"""loadwith names the pages this one is not complete without."""
fields = parse_docmeta(page)
if fields is None:
pytest.skip('no docmeta - reported by test_every_page_carries_a_docmeta_block')
raw = value(fields, 'loadwith')
if not raw or raw.lower() == 'none':
return
for name in [n.strip() for n in raw.split(',') if n.strip()]:
target = resolve(page, name) or (DOCS / name)
if not target.is_file():
target = DOCS / name
assert target.is_file(), (
'%s: loadwith names "%s", which does not exist' % (rel(page), name))
@pytest.mark.parametrize(
'page', [p for p in pages() if p != INDEX], ids=lambda p: rel(p))
def test_every_page_is_routed_or_archived(page):
"""A page nothing routes to is a page nobody reads and nobody maintains."""
text = INDEX.read_text()
targets = {resolve(INDEX, t) for t in links_in(text)}
assert page in targets, (
'%s is in docs/ but docs/INDEX.md does not name it. Add a task row for '
'it, or - if it is superseded or only a record - mark it legacy or '
'historical in its docmeta and list it in the Archive section.' % rel(page))
@pytest.mark.parametrize('router', ROUTERS, ids=lambda p: rel(p))
def test_a_router_never_points_at_a_file_that_is_not_there(router):
"""INDEX.md and START-HERE.md are the two pages everyone is sent to first.
A dead link here is worse than a missing row: a missing row makes a reader
look further, a dead link makes them think the answer is gone. Pages get
merged, split and renamed, and both routers name files by hand, so this is
the check that catches it the same day.
Every relative target counts, not just .md - the index links llms.txt too -
and a target outside docs/ is dead by definition once these pages are
published to the wiki, which carries docs/ and nothing above it.
"""
broken = []
for target in sorted(set(all_targets_in(router.read_text()))):
page = resolve(router, target)
if page is None:
broken.append('%s (outside docs/)' % target)
elif not page.is_file():
broken.append(target)
assert not broken, (
'docs/%s links to files that do not exist: %s'
% (rel(router), ', '.join(broken)))
def test_every_current_page_is_reachable_from_a_task_row():
"""Named in the index is not the same as routed by it.
Nothing stops a page being mentioned in a paragraph, or slipped into the
Archive section's prose, and passing "INDEX.md names it". A reader works
the rows: they find the row that matches the job and read what it lists. A
current page that is in no row is a page that gets written, gets stale and
never gets opened - and the cost of keeping it is paid by everyone who
greps the corpus.
The routers themselves are exempt: they are where a reader starts.
"""
tasks, _ = index_sections()
routed = set()
for targets, _lines in table_rows(tasks):
for target in targets:
page = resolve(INDEX, target)
if page:
routed.add(page)
stranded = sorted(
rel(p) for p in pages()
if status_of(p) == 'current' and p not in routed and p not in ROUTERS)
assert not stranded, (
'these pages are current guidance but no task row in docs/INDEX.md '
'sends anyone to them: %s. Give each one a row saying what task it '
'answers, or - if it is superseded or only a record - mark it legacy '
'or historical and move it to the Archive section.' % ', '.join(stranded))
def test_task_rows_never_route_to_an_archived_page():
"""Historical and legacy pages belong in the Archive section, nowhere else."""
tasks, _ = index_sections()
offenders = []
for target in sorted(set(links_in(tasks))):
page = resolve(INDEX, target)
if page and page.is_file() and status_of(page) in ARCHIVED:
offenders.append('%s (%s)' % (target, status_of(page)))
assert not offenders, (
'docs/INDEX.md routes a task to a page that is not current guidance: %s'
% ', '.join(offenders))
def test_the_archive_section_holds_nothing_current():
_, archive = index_sections()
offenders = []
for target in sorted(set(links_in(archive))):
page = resolve(INDEX, target)
if page and page.is_file() and status_of(page) == 'current':
offenders.append(target)
assert not offenders, (
'docs/INDEX.md files these current pages under Archive, where readers '
'are told not to follow them: %s' % ', '.join(offenders))
def test_no_current_page_hides_behind_an_archived_one():
"""Walk out from INDEX.md without passing through legacy or historical pages.
A current page that can only be reached through a superseded one is a page
a careful reader will never open, because they were told to skip the page
that links it.
"""
seen = set()
queue = [INDEX]
while queue:
page = queue.pop()
if page in seen:
continue
seen.add(page)
if status_of(page) in ARCHIVED:
continue # a dead end on purpose: nothing may depend on it
for target in links_in(page.read_text(errors='replace')):
nxt = resolve(page, target)
if nxt and nxt.is_file():
queue.append(nxt)
stranded = sorted(
rel(p) for p in pages() if status_of(p) == 'current' and p not in seen)
assert not stranded, (
'these current pages are only reachable through an archived page: %s'
% ', '.join(stranded))
def test_index_line_totals_are_honest():
"""The cost column is the point of the index. A wrong one is worse than none."""
tasks, archive = index_sections()
offenders = []
for targets, stated in table_rows(tasks) + table_rows(archive):
real = 0
for target in targets:
page = resolve(INDEX, target)
if page and page.is_file():
real += line_count(page)
if abs(real - stated) > tolerance(real):
offenders.append('%s: says %d lines, is %d' % ('+'.join(targets), stated, real))
assert not offenders, (
'docs/INDEX.md misstates what a row costs to read:\n %s' % '\n '.join(offenders))
def test_the_archive_total_is_honest():
_, archive = index_sections()
match = ARCHIVE_TOTAL.search(archive)
assert match, 'the Archive section should say how many lines it adds up to'
stated = int(match.group(1).replace(',', ''))
real = sum(count for _, count in table_rows(archive))
assert abs(real - stated) <= tolerance(real), (
'the Archive section claims %d lines; its rows add up to %d' % (stated, real))

8
tests/test_docs_publishable.py
View File

@@ -34,14 +34,6 @@ DOCS = REPO / 'docs'
# published mirror - keying off the directory made this module run there against
# a prose-less tree and fail. A published checkout is one with no authored
# prose, matching the canonical definition in scripts/check-gates-ran.sh.
# KNOWN GAP, left deliberately for a decision rather than closed here: this
# keys on docs/*.md, so prose that lives only in docs/wiki-drafts/ does NOT
# turn the module on, and on gh-wj that is every page there is. Pointed at the
# drafts on 2026-09-18 the scrub fails on 27 lines - site name, internal git
# server, developer home paths - which is a question about what those drafts
# are for (internal wiki, or publication) and not a bug to fix by rewriting
# them. scripts/check-gates-ran.sh matches this definition exactly; change both
# together or the gate runner reports a skip it was told to expect.
pytestmark = pytest.mark.skipif(
not any(DOCS.glob('*.md')),
reason='no docs/ prose in this checkout - it is excluded from publication and lives in the wiki')

637
tests/test_openapi_examples.py Normal file
View File

@@ -0,0 +1,637 @@
"""The two derived halves of the published spec: query parameters and examples.
docs/openapi.json is served at /api/docs on production AND published to the
wiki, and it is what the MCP server builds its tools from. Two things were
missing from every operation in it, and both are now derived rather than
written:
QUERY PARAMETERS. A Flask rule carries its path arguments and nothing else,
so 0 of 573 operations published a query parameter - including
/api/usb/kiosk/setup, where '?code=' is the only way to call it successfully.
scripts/gen_openapi.py now reads them out of each view's AST.
EXAMPLES. 0 of 573 carried a machine-readable example. They are harvested
from the test suite, which already sends real bodies and asserts real
responses.
Harvesting is where this module earns its place. A harvested body is whatever a
test happened to build, and tests are written with real-looking material -
passwords, tokens, badge numbers, hostnames, addresses. None of that matters in
a test file and all of it matters on a public page, so redaction runs before
anything reaches the spec. `redact` is tested here directly, and the generated
spec is swept for the same material, because a scrubber nobody checks is a
scrubber that stops working quietly.
The other property tested here is that NOTHING IS INVENTED. A query argument
whose name the generator cannot resolve to a literal is not published at all:
an argument that is not really there is worse than a missing one, because a
reader believes it, sends it and is silently ignored.
"""
import importlib.util
import json
import textwrap
from pathlib import Path
import pytest
import re
REPO = Path(__file__).resolve().parents[1]
SPEC = REPO / 'docs' / 'openapi.json'
def _generator():
"""scripts/gen_openapi.py, loaded by path.
It is a script, not a package module. Loading it runs its imports and
nothing else - it builds no app and reads no database until it is asked to.
"""
path = REPO / 'scripts' / 'gen_openapi.py'
spec = importlib.util.spec_from_file_location('gen_openapi_examples', path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
@pytest.fixture(scope='module')
def generator():
return _generator()
@pytest.fixture(scope='module')
def spec():
if not SPEC.is_file():
pytest.skip('no generated spec in this checkout')
return json.loads(SPEC.read_text())
def operations(spec):
for path, item in spec['paths'].items():
for verb, operation in item.items():
yield path, verb, operation
def examples(spec):
"""Every published example, as (where, value)."""
for path, verb, operation in operations(spec):
body = operation.get('requestBody') or {}
media = (body.get('content') or {}).get('application/json') or {}
if 'example' in media:
yield '%s %s request' % (verb.upper(), path), media['example']
success = ((operation.get('responses') or {}).get('200') or {})
media = (success.get('content') or {}).get('application/json') or {}
if 'example' in media:
yield '%s %s response' % (verb.upper(), path), media['example']
# ---------------------------------------------------------------------------
# redact
# ---------------------------------------------------------------------------
def test_a_credential_is_redacted_by_its_key(generator):
"""The key is the reliable signal, whatever the value happens to be today.
A fixture that leaves passwordhash empty must not teach the scrubber that
passwordhash is safe.
"""
scrubbed = generator.redact({
'password': 'hunter2',
'passwordhash': '',
'apikey': 'abc',
'refresh_token': 'abc',
'X-API-Key': 'abc',
'signature': 'abc',
})
assert set(scrubbed.values()) == {'<redacted:secret>'}
def test_a_person_and_a_machine_are_redacted_by_their_key(generator):
"""Not secrets, and not ours to publish either. ADR-015 for the machine
half: one plant's server name is meaningless to every other site reading
these pages."""
scrubbed = generator.redact({
'username': 'jsmith',
'badge': '204411',
'checkedoutby': 'J Smith',
'hostname': 'WJSF0123',
'fqdn': 'wjsf0123.example.net',
'ipaddress': '10.4.2.19',
'servicetag': 'H3K9QP2',
})
assert scrubbed['username'] == '<redacted:person>'
assert scrubbed['badge'] == '<redacted:person>'
assert scrubbed['checkedoutby'] == '<redacted:person>'
assert scrubbed['hostname'] == '<redacted:host>'
assert scrubbed['fqdn'] == '<redacted:host>'
assert scrubbed['ipaddress'] == '<redacted:host>'
assert scrubbed['servicetag'] == '<redacted:host>'
def test_a_credential_with_no_telling_key_is_redacted_by_its_value(generator):
"""The half the key rules cannot reach: material in a free-text field, or
in a list, where nothing names it."""
scrubbed = generator.redact({
'note': 'mail bob.smith@example.com when it lands',
'detail': 'bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.aaaa.bbbb',
'path': '\\\\fileserver\\deploy\\agent.msi',
'where': 'reported from 10.11.12.13',
'digest': 'd41d8cd98f00b204e9800998ecf8427e',
'bag': ['carol@example.com'],
})
assert scrubbed['note'] == '<redacted:email>'
assert scrubbed['detail'] == '<redacted:token>'
assert scrubbed['path'] == '<redacted:sharepath>'
assert scrubbed['where'] == '<redacted:address>'
assert scrubbed['digest'] == '<redacted:digest>'
assert scrubbed['bag'] == ['<redacted:email>']
def test_ordinary_material_survives(generator):
"""Over-redaction has a cost too: an example scrubbed to nothing shows a
reader nothing. What is safe stays."""
payload = {
'assetnumber': 'A-10024',
'model': 'Dell Latitude 5420',
'isactive': True,
'quantity': 12,
'status': 'In Service',
'url': 'http://localhost:5001/api/assets',
}
assert generator.redact(payload) == payload
def test_a_numeric_id_is_not_a_credential(generator):
"""'tokenid': 3 is the row number, not the token.
Redacting it cost every token route the one field a caller needs to address
the row. The exemption is numeric-only, so a string session id is still
scrubbed.
"""
scrubbed = generator.redact({'tokenid': 3, 'apikeyid': 7,
'sessionid': 'a-long-opaque-string-value'})
assert scrubbed['tokenid'] == 3
assert scrubbed['apikeyid'] == 7
assert scrubbed['sessionid'] == '<redacted:secret>'
def test_redaction_reaches_every_level(generator):
"""A token two objects down is as published as one at the top."""
scrubbed = generator.redact(
{'data': {'items': [{'inner': {'password': 'hunter2'}}]}})
assert scrubbed['data']['items'][0]['inner']['password'] == '<redacted:secret>'
def test_redaction_is_idempotent(generator):
"""A harvest runs again over a spec that already carries markers, and the
markers must not be re-scrubbed into something else."""
once = generator.redact({'password': 'hunter2', 'model': 'Dell 5420'})
assert generator.redact(once) == once
def test_a_long_list_is_trimmed_and_says_so(generator):
"""Three rows show the shape, which is the whole job of an example - but a
reader must not conclude the endpoint returns three."""
scrubbed = generator.redact({'data': list(range(50))})
assert scrubbed['data'][:3] == [0, 1, 2]
assert 'more' in scrubbed['data'][-1]
def test_a_cyclic_depth_is_bounded(generator):
"""Harvested bodies come from a running app, not from this file. A deeply
nested one must not take the generator down with it."""
deep = current = {}
for _ in range(50):
current['child'] = {}
current = current['child']
generator.redact(deep)
# ---------------------------------------------------------------------------
# The published spec, swept for the same material
# ---------------------------------------------------------------------------
# Patterns held HERE, deliberately not imported from the generator. A gate that
# checks the scrubber with the scrubber's own patterns can only ever assert that
# redact() ran: it was green while the spec published a live bootstrap token and
# a live kiosk setup code, because both hid in a URL whose key ('scripturl',
# 'url') names no secret and whose value the opaque rule excludes by design.
# These must keep catching that class even if someone edits the generator.
INDEPENDENT_URL_CREDENTIAL = re.compile(
r'(?i)[?&](t|token|code|key|apikey|api_key|secret|sig|signature|password|'
r'auth|access_token|refresh_token)=([^&\s"\'<]{8,})')
# Product token prefixes. If a new one is minted, add it here the same day.
INDEPENDENT_TOKEN_PREFIX = re.compile(r'\b(sdbb_|sdbe_|shopdb_pat_)[A-Za-z0-9_-]{8,}')
def _looks_high_entropy(text):
"""A long mixed-case, digit-bearing run with no word structure.
Deliberately crude: it is a backstop for a secret shape nobody has thought
of yet, not a classifier. Anything it flags is either a secret or an example
that would read better with a shorter value.
"""
for chunk in re.split(r'[\s"\'<>]+', text):
if len(chunk) < 28 or '.' in chunk or '/' in chunk:
continue
has = lambda pattern: re.search(pattern, chunk)
if has(r'[a-z]') and has(r'[A-Z]') and has(r'\d'):
return chunk
return None
def test_no_published_example_hides_a_credential_in_a_url(spec):
"""The class the scrubber's own patterns cannot see.
Two real credentials reached docs/openapi.json this way on 2026-09-18, in a
file served at /api/docs on production and copied to a published wiki.
"""
offenders = []
for where, example in examples(spec):
blob = json.dumps(example)
found = INDEPENDENT_URL_CREDENTIAL.search(blob)
if found and not found.group(2).startswith('<redacted:'):
offenders.append('%s: %s=%s' % (where, found.group(1), found.group(2)[:16]))
prefixed = INDEPENDENT_TOKEN_PREFIX.search(blob)
if prefixed:
offenders.append('%s: token prefix %s' % (where, prefixed.group(0)[:20]))
entropy = _looks_high_entropy(blob)
if entropy:
offenders.append('%s: high-entropy %r' % (where, entropy[:20]))
assert not offenders, (
'Published examples carry what looks like a credential:\n '
+ '\n '.join(sorted(set(offenders))))
def test_no_published_example_carries_a_secret_a_person_or_a_machine(spec, generator):
"""The gate on the whole harvest, from the side a reader sees.
Checked against the scrubber's own patterns rather than a second list, so
the gate and the scrubber cannot drift apart: this fails if redaction stops
being applied, if it is applied to a stale copy, or if a hand-edit puts
real material back.
"""
offenders = []
for where, example in examples(spec):
for key, value in _walk(example):
if not isinstance(value, str) or value.startswith('<redacted:'):
continue
name = (key or '').lower()
# Only strings reach here, and the numeric-id exemption in redact()
# is numeric-only, so no key is exempt on this side.
if name and generator.SECRET_KEY_PATTERN.search(name):
offenders.append('%s: key %r' % (where, key))
elif name and generator.PERSON_KEY_PATTERN.search(name):
offenders.append('%s: key %r' % (where, key))
elif name and generator.HOST_KEY_PATTERN.search(name):
offenders.append('%s: key %r' % (where, key))
elif generator._redacted_string(value):
offenders.append('%s: value under %r' % (where, key))
assert not offenders, (
'%d published example value(s) look like a secret, a person or a '
'machine. docs/openapi.json is served on production and published to '
'the wiki; regenerate it with scripts/gen_openapi.py:\n %s'
% (len(offenders), '\n '.join(sorted(set(offenders))[:15])))
def _walk(value, key=''):
"""(key, value) for every leaf in a harvested example."""
if isinstance(value, dict):
for itemkey, itemvalue in value.items():
for found in _walk(itemvalue, str(itemkey)):
yield found
elif isinstance(value, list):
for item in value:
for found in _walk(item, key):
yield found
else:
yield key, value
def test_a_published_example_is_bounded(spec, generator):
"""Everybody downloads this document and every generated client parses it.
One 400-row list response costs all of them."""
offenders = [where for where, example in examples(spec)
if len(json.dumps(example)) > generator.EXAMPLE_MAX_CHARS]
assert not offenders, '\n '.join(offenders[:15])
def test_a_response_example_never_contradicts_the_schema_beside_it(spec):
"""An example and a `$ref` that disagree are worse than no example at all.
Nineteen routes here do not return the standard envelope and every one was
documented as if it did. A harvested transcript is evidence about which,
so the generator drops the envelope `$ref` on any operation whose real
response is a different shape. This is the gate on that: an operation
published with the envelope `$ref` and an example that is not an envelope
is the exact contradiction the harvest exists to remove.
"""
offenders = []
for path, verb, operation in operations(spec):
media = (((operation.get('responses') or {}).get('200') or {})
.get('content') or {}).get('application/json') or {}
if 'example' not in media:
continue
claims = 'SuccessEnvelope' in json.dumps(media.get('schema') or {})
isenvelope = (isinstance(media['example'], dict)
and media['example'].get('status') == 'success')
if claims != isenvelope:
offenders.append('%s %s: schema says envelope=%s, example is=%s'
% (verb.upper(), path, claims, isenvelope))
assert not offenders, '\n '.join(offenders[:15])
# ---------------------------------------------------------------------------
# Derived query parameters: what is emitted, and what is deliberately not
# ---------------------------------------------------------------------------
def _derive(tmp_path, source):
"""Run the generator's query sweep over one written-out module."""
module = tmp_path / 'sample_views.py'
module.write_text(textwrap.dedent(source))
return _generator()._query_index(str(module))
def test_a_literal_query_read_is_emitted(tmp_path):
index = _derive(tmp_path, '''
def view():
search = request.args.get('search')
active = request.args.get('active', 'true')
days = request.args.get('days', 30, type=int)
tags = request.args.getlist('tag')
page = request.args['page']
''')
found = {item['name']: item for item in index['view']}
assert sorted(found) == ['active', 'days', 'page', 'search', 'tag']
assert found['search'] == {'name': 'search', 'type': 'string', 'required': False}
assert found['active']['default'] == 'true'
assert found['days'] == {'name': 'days', 'type': 'integer',
'required': False, 'default': 30}
assert found['tag']['type'] == 'array'
# request.args['x'] raises without the argument. It is the one read in the
# language that proves the caller must send it.
assert found['page']['required'] is True
def test_a_name_that_is_not_a_literal_is_not_published(tmp_path):
"""The honesty rule, and the reason this is safe to derive at all.
A key built at runtime cannot be resolved, and a parameter that is not
really there is worse than a missing one: a reader believes it, sends it,
and is silently ignored. A module-level constant DOES resolve, because it
is a literal with a name.
"""
index = _derive(tmp_path, '''
TOKEN_ARG = 'bootstraptoken'
def view(field):
built = request.args.get('filter_' + field)
indirect = request.args.get(field)
formatted = request.args.get(f'{field}id')
named = request.args.get(TOKEN_ARG)
''')
assert [item['name'] for item in index['view']] == ['bootstraptoken']
def test_the_repos_own_argument_helpers_are_read(tmp_path):
"""get_pagination_params and intarg/intargs read arguments on a handler's
behalf, so a sweep that only looks for request.args misses the paging
contract of every list route in the product."""
index = _derive(tmp_path, '''
def view():
page, perpage = get_pagination_params(request)
typeid = intarg('typeid')
since = intarg('days', 30)
rest = intargs('locationid', 'businessunitid')
''')
found = {item['name']: item for item in index['view']}
assert sorted(found) == ['businessunitid', 'days', 'locationid', 'page',
'per_page', 'perpage', 'typeid']
assert found['page'] == {'name': 'page', 'type': 'integer',
'required': False, 'default': 1}
assert found['days']['default'] == 30
assert found['typeid']['type'] == 'integer'
def test_a_view_that_reads_nothing_publishes_nothing(tmp_path):
index = _derive(tmp_path, '''
def view(asset_id):
return success_response({'assetid': asset_id})
''')
assert index['view'] == []
# ---------------------------------------------------------------------------
# The generator without any of this
# ---------------------------------------------------------------------------
def test_a_spec_generates_with_no_harvest_and_no_query_parameters(generator):
"""Both halves are OPTIONAL. A published checkout has no tests to harvest
from, and an inventory written before this existed has no `queryparams`. A
generator that needs either is a generator that cannot be run where it
matters."""
built = generator.build([{
'surface': 'core-assets',
'endpoints': [{
'method': 'GET|POST',
'path': '/api/assets',
'auth': 'permission:assets.view',
'params': 'not documented',
'purpose': 'List assets.',
'example': 'curl http://localhost:5001/api/assets',
}],
}])
operation = built['paths']['/api/assets']['get']
assert 'parameters' not in operation
media = built['paths']['/api/assets']['post']['requestBody']['content']
assert 'example' not in media['application/json']
success = operation['responses']['200']['content']['application/json']
assert 'example' not in success
assert success['schema'] == {'$ref': '#/components/schemas/SuccessEnvelope'}
def test_a_query_parameter_survives_into_the_spec(generator):
"""The other direction: an inventory entry that has them publishes them."""
built = generator.build([{
'surface': 'plugin-usb',
'endpoints': [{
'method': 'GET',
'path': '/api/usb/kiosk/setup',
'auth': 'setup-code (single-use kiosk setup code in ?code=)',
'params': 'not documented',
'purpose': 'Trade a setup code for this browser kiosk cookie.',
'example': 'curl http://localhost:5001/api/usb/kiosk/setup',
'queryparams': [{'name': 'code', 'type': 'string', 'required': False}],
'responseexample': {'status': 'success', 'data': None},
}],
}])
operation = built['paths']['/api/usb/kiosk/setup']['get']
assert operation['parameters'] == [{
'name': 'code', 'in': 'query', 'required': False,
'schema': {'type': 'string'},
}]
success = operation['responses']['200']['content']['application/json']
assert success['example'] == {'status': 'success', 'data': None}
# A query filter that matches nothing is an empty list and a 200. Only a
# path parameter can be a record that is not there.
assert '404' not in operation['responses']
def test_a_per_request_value_is_pinned_but_keeps_its_shape(generator):
"""A transcript carries the instant it was taken and the id of the request
that produced it, and both differ on every run.
Published verbatim, re-harvesting rewrote 313 of 320 examples with no change
of meaning - an unreviewable diff over a committed, published file. An
example shows a reader the FORMAT of a field, not the moment one call
happened, so the format is kept exactly and the value is pinned.
"""
scrubbed = generator.redact({
'meta': {'timestamp': '2026-09-18T18:30:48.498252Z',
'requestid': '859aa86b'},
'createddate': '2026-09-18 08:15:00',
'expiresat': '2026-12-31T23:59:59+00:00',
'installedon': '2026-09-18',
})
# Microseconds, the space separator and the offset all survive: those are
# three different contracts and a reader is reading the example for which.
assert scrubbed['meta']['timestamp'] == '2026-01-01T00:00:00.000000Z'
assert scrubbed['meta']['requestid'] == '00000000'
assert scrubbed['createddate'] == '2026-01-01 00:00:00'
assert scrubbed['expiresat'] == '2026-01-01T00:00:00+00:00'
assert scrubbed['installedon'] == '2026-01-01'
def test_pinning_does_not_reach_ordinary_text(generator):
"""Only a value that IS a timestamp. A model number that happens to contain
digits is not one."""
payload = {'model': '5420-2026', 'note': 'due 2026-09-18 at the latest',
'version': '1.2.3'}
assert generator.redact(payload) == payload
def test_a_local_path_is_not_published(generator):
"""A path off the machine the harvest ran on says nothing to a reader and
is different on every run. A developer's home directory is also the exact
material tests/test_docs_publishable.py exists to keep off a public page."""
scrubbed = generator.redact({
'shareroot': '/tmp/pytest-of-someone/pytest-240/test_roundtrip0',
'checkout': '/home/someone/projects/shopdb-flask',
'relative': 'deploy/windows/installer',
'route': '/api/assets',
})
assert scrubbed['shareroot'] == '<redacted:path>'
assert scrubbed['checkout'] == '<redacted:path>'
# A path that names no machine is the useful half, and it stays.
assert scrubbed['relative'] == 'deploy/windows/installer'
assert scrubbed['route'] == '/api/assets'
def test_a_harvest_adds_and_never_clears(generator, tmp_path):
"""A targeted harvest has to be targeted.
Clearing on absence meant `--harvest tests/test_core` stripped the examples
off every route those tests do not touch, which is most of them - the
narrow command would have done the most damage. It also made the committed
file churn for nothing: run under xdist against a shared instance
directory, two full runs of the same tree reached 311 and 327 operations.
"""
inventory = tmp_path / 'api-inventory.json'
inventory.write_text(json.dumps([{
'surface': 'core-assets',
'endpoints': [
{'method': 'GET', 'path': '/api/assets',
'responseexample': {'status': 'success', 'data': ['kept']}},
{'method': 'POST', 'path': '/api/assets'},
],
}]))
original = generator.INVENTORY
generator.INVENTORY = str(inventory)
try:
generator.fold_examples({
('POST', '/api/assets'): {
'request': [{'assetnumber': 'A-1'}],
'response': [{'status': 'success', 'data': {'assetid': 1}}],
},
})
finally:
generator.INVENTORY = original
entries = json.loads(inventory.read_text())[0]['endpoints']
# Untouched by this harvest, and still carrying what it had.
assert entries[0]['responseexample'] == {'status': 'success', 'data': ['kept']}
assert entries[1]['requestexample'] == {'assetnumber': 'A-1'}
assert entries[1]['responseexample'] == {'status': 'success',
'data': {'assetid': 1}}
def test_picking_one_example_out_of_many_is_deterministic(generator):
"""The suite runs in parallel, so the order samples arrive in is not stable.
Largest-that-fits with a text tiebreak means two runs over the same samples
produce the same spec, which is what makes a regenerated docs/openapi.json
reviewable as a diff instead of a rewrite.
"""
samples = [
{'status': 'success', 'data': {'assetid': 1}},
{'status': 'success', 'data': {'assetid': 2, 'assetnumber': 'A-2'}},
{'status': 'success', 'data': None},
]
picked = generator._best_example(samples)
assert picked == generator._best_example(list(reversed(samples)))
assert picked['data']['assetnumber'] == 'A-2'
def test_an_example_that_cannot_be_made_to_fit_is_dropped(generator):
"""Silence beats a truncated JSON document: half an object is not an
example of anything, and it is not what the route returned."""
# A wide object, not a long string: strings and lists are already capped
# inside redact, so an object with hundreds of keys is the shape that can
# still come out over the limit.
huge = {'status': 'success',
'data': {'field%03d' % index: 'value' for index in range(500)}}
assert generator._best_example([huge]) is None
# And a smaller sample of the same operation still wins, rather than the
# whole operation losing its example to one oversized transcript.
small = {'status': 'success', 'data': {'assetid': 1}}
assert generator._best_example([huge, small]) == small
def test_a_delete_that_reads_a_body_says_so(generator):
"""DELETE /api/applications/{id} refuses without {"confirm": "OK"}.
The generator declared a body for POST, PUT and PATCH only, so that route
published a call that cannot succeed and a client built from the spec fails
on the day.
Evidence is the HANDLER reading a body (`readsbody`, derived from its AST),
not a harvested transcript. A transcript proves the test CLIENT sent
something, and the suite sends {} to routes that ignore it: on 2026-09-18
transcript-only evidence published "this route reads a JSON body" on DELETE
/api/endpoints/agentkey, whose handler never calls get_json. Both halves are
pinned below.
"""
entries = [
{'method': 'DELETE', 'path': '/api/applications/<int:app_id>',
'auth': 'permission:applications.delete', 'purpose': 'Delete it.',
'readsbody': True, 'requestexample': {'confirm': 'OK'}},
{'method': 'DELETE', 'path': '/api/assets/<int:assetid>',
'auth': 'permission:assets.delete', 'purpose': 'Delete it.'},
# A transcript with no handler evidence: the suite sent a body, the
# route ignores it, so the contract must stay silent.
{'method': 'DELETE', 'path': '/api/endpoints/agentkey',
'auth': 'permission:endpoints.manage', 'purpose': 'Revoke it.',
'requestexample': {}},
]
built = generator.build([{'surface': 'core-applications', 'endpoints': entries}])
withbody = built['paths']['/api/applications/{app_id}']['delete']
media = withbody['requestBody']['content']['application/json']
assert withbody['requestBody']['required'] is False
assert media['example'] == {'confirm': 'OK'}
# Transcript without handler evidence declares nothing.
ignored = built['paths']['/api/endpoints/agentkey']['delete']
assert 'requestBody' not in ignored
assert 'requestBody' not in built['paths']['/api/assets/{assetid}']['delete']

55
tests/test_openapi_spec.py
View File

@@ -189,3 +189,58 @@ def test_summaries_are_not_cut_mid_word(spec):
if body and not body[-1].isalnum() and body[-1] not in ')]"\'':
offenders.append('%s %s: %r' % (verb.upper(), path, summary[-40:]))
assert not offenders, '\n '.join(offenders[:15])
def test_every_parameter_is_well_formed(spec):
"""A parameter with no name, no location or no schema is not a parameter.
Query parameters are derived from each handler's AST now rather than left
out, which is 87 operations that could not be filtered, searched or paged
from this document before. Derived means a code change can produce a
malformed one, so the shape is checked here rather than assumed.
"""
offenders = []
for path, verb, operation in operations(spec):
for parameter in operation.get('parameters', []):
where = '%s %s: %r' % (verb.upper(), path, parameter.get('name'))
if not parameter.get('name'):
offenders.append('%s %s: a parameter with no name' % (verb.upper(), path))
if parameter.get('in') not in ('path', 'query', 'header', 'cookie'):
offenders.append(where + ' has no valid `in`')
if not isinstance(parameter.get('required'), bool):
offenders.append(where + ' does not say whether it is required')
if not parameter.get('schema'):
offenders.append(where + ' has no schema')
if parameter.get('in') == 'path' and not parameter.get('required'):
offenders.append(where + ' is in the path and optional, which '
'OpenAPI does not allow')
assert not offenders, '\n '.join(offenders[:15])
def test_no_parameter_is_declared_twice(spec):
"""(name, in) is the identity of a parameter. Two entries for one name is
ambiguous to a generated client, which picks one and drops the other."""
offenders = []
for path, verb, operation in operations(spec):
seen = set()
for parameter in operation.get('parameters', []):
key = (parameter.get('name'), parameter.get('in'))
if key in seen:
offenders.append('%s %s: %s in %s'
% (verb.upper(), path, key[0], key[1]))
seen.add(key)
assert not offenders, '\n '.join(offenders[:15])
def test_a_query_parameter_is_not_the_path_argument_under_another_name(spec):
"""`/api/assets/{assetid}?assetid=` is a contradiction: the same value in
two places, and a generated client has to guess which one the route reads."""
offenders = []
for path, verb, operation in operations(spec):
intemplate = set(re.findall(r'\{([^}]+)\}', path))
query = {p['name'] for p in operation.get('parameters', [])
if p.get('in') == 'query'}
clash = sorted(intemplate & query)
if clash:
offenders.append('%s %s: %s' % (verb.upper(), path, clash))
assert not offenders, '\n '.join(offenders[:15])