Rebuild wiki: full docs mirror with working links
Replaces the previous push, whose pages were empty files with mangled names. One page per docs/ file (20 guides, 12 ADRs, 1 proposal) plus a grouped Home and sidebar. Doc-to-doc links rewritten to wiki page names; links to repo files rewritten to the repo browser. docs/ in the repo stays canonical.
192
ADR-001-asset-as-platform-contract.md
Normal file
192
ADR-001-asset-as-platform-contract.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# ADR-001: Asset model is the platform contract
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
shopdb-flask is being shaped as a framework that sister GE Aerospace facilities can adopt. The framework defines a stable core; sites install plugins for the asset classes they care about (equipment, computers, printers, measuring tools, network gear, etc.).
|
||||
|
||||
The codebase ran two parallel object models:
|
||||
|
||||
1. **Legacy `Machine` model** in `shopdb/core/models/machine.py`. Original schema inherited from the classic-ASP shopdb. Tables: `machines`, `pctypes`, `machinetypes`. Plugins like printers stored extension data via `PrinterData` keyed by `machineid`.
|
||||
2. **New `Asset` model** in `shopdb/core/models/asset.py`. Generic asset abstraction with `AssetType`, `AssetStatus`, `AssetRelationship`. Plugins also exposed asset-based tables keyed by `assetid`.
|
||||
|
||||
For the framework to be adoptable, the platform contract has to be one model, documented, versioned, and stable.
|
||||
|
||||
## Decision
|
||||
|
||||
**`Asset` is the platform contract.** Plugin authors target the asset-based API. The `Machine` model and its dependents (`PCType`, `MachineType`, `PrinterData` keyed by `machineid`) are legacy and will be retired through a tracked migration.
|
||||
|
||||
### Platform contract surface
|
||||
|
||||
The following are the public, versioned surface. Plugin authors may depend on them. Breaking changes require a major version bump per ADR-002.
|
||||
|
||||
#### Models
|
||||
|
||||
- `Asset` (core entity)
|
||||
- `AssetType` (asset classification, registered by plugins)
|
||||
- `AssetStatus` (active / inactive / decommissioned / retired)
|
||||
- `AssetRelationship` + `RelationshipType` (cross-plugin links)
|
||||
- `Vendor`, `Location`, `LocationType`, `BusinessUnit`, `Model`, `OperatingSystem` (shared reference data)
|
||||
|
||||
#### Helpers
|
||||
|
||||
- `AuditLog` API: `audit_log(action, entitytype, entityid, ...)` for plugins to record audit entries with consistent schema
|
||||
- `Setting` API: `plugin.get_setting(key)` and `plugin.set_setting(key, value)` for plugin-scoped config persisted via the core `Setting` model
|
||||
- `resolve_asset_position(asset)` for the documented position-resolution algorithm
|
||||
|
||||
#### Import surface (`shopdb.api`) - expanded in __contract_version__ 0.3.0
|
||||
|
||||
`shopdb.api` is the ONLY core module plugins may import (plus `shopdb.plugins.base`
|
||||
for the ABC). Deep imports of `shopdb.core.*`, `shopdb.extensions`, or
|
||||
`shopdb.utils.*` are contract violations enforced by the
|
||||
`test_plugins_only_import_contract_surface` test. The surface re-exports:
|
||||
|
||||
- Infrastructure: `db`, `cache`
|
||||
- Model bases: `BaseModel`, `AuditMixin`
|
||||
- Core models: `Asset`, `AssetType`, `AssetStatus`, `Vendor`, `Model`,
|
||||
`Communication`, `CommunicationType`, `Location`, `Setting`, `AuditLog`,
|
||||
`Application`, `AppVersion`, `OperatingSystem`
|
||||
- Responses: `success_response`, `error_response`, `paginated_response`, `ErrorCodes`
|
||||
- Pagination: `get_pagination_params`, `paginate_query`
|
||||
- Helpers: `audit_log`, `resolve_asset_position`; legacy `employee_connection`
|
||||
|
||||
Adding a name here is a minor (additive) contract change; removing one is major.
|
||||
|
||||
#### Plugin contract
|
||||
|
||||
- `BasePlugin` ABC and its hooks (navigation, dashboard widgets, collector schema + `apply_collector_payload`). Note: `get_searchable_fields` was removed in contract 0.4.0 - global search is a core concern over the asset model, not a per-plugin hook.
|
||||
|
||||
#### Excluded from the contract for v1
|
||||
|
||||
- Event bus (`get_event_handlers` removed from `BasePlugin`). Add later if a real use case appears.
|
||||
|
||||
### Relationship types (seeded core values)
|
||||
|
||||
`RelationshipType` is seeded with three rows. Plugin authors do not add new types in v1.
|
||||
|
||||
| Type | Meaning | Boundary rule |
|
||||
|------|---------|---------------|
|
||||
| `partof` | Composition, siblings, sub-assemblies | Parent dies without it |
|
||||
| `controls` | One asset has operational authority over another | PC commands a machine, measuring tool, etc. |
|
||||
| `connectedto` | Network or data link without operational authority | Cable, switch port, NAS mount |
|
||||
|
||||
### AssetRelationship columns
|
||||
|
||||
```
|
||||
AssetRelationship
|
||||
- relationshipid PK
|
||||
- sourceassetid FK to assets
|
||||
- targetassetid FK to assets
|
||||
- relationshiptypeid FK to relationshiptypes (one of partof, controls, connectedto)
|
||||
- label text, free description (e.g., "ethernet PoE", "DNC feed")
|
||||
- inheritsposition bool, true means resolved-position walk follows this edge
|
||||
- propagatesthroughid FK to relationshiptypes, nullable, see propagation below
|
||||
- notes text
|
||||
- isactive bool
|
||||
- createddate, modifieddate
|
||||
```
|
||||
|
||||
Free-text `label` carries the domain nuance (`controls` with label "DNC feed" vs `controls` with label "operator workstation"). Avoids inflating the type list.
|
||||
|
||||
### Sibling propagation
|
||||
|
||||
When a relationship is created or deleted, the framework checks `RelationshipType.propagatesthroughid`. If set, it finds all assets related to the source via the propagation type and applies the same change.
|
||||
|
||||
Default seeding:
|
||||
|
||||
| RelationshipType | propagatesthroughid |
|
||||
|------------------|---------------------|
|
||||
| `partof` | null (the propagation rail itself) |
|
||||
| `controls` | `partof` (controls relationships propagate across siblings) |
|
||||
| `connectedto` | null (network paths don't propagate) |
|
||||
|
||||
Cycle protection: max walk depth 3, visited-set during traversal.
|
||||
|
||||
### Position resolution
|
||||
|
||||
Resolved map position for any asset follows this priority chain:
|
||||
|
||||
1. Asset-specific override (`assets.mapx`, `assets.mapy` if non-null)
|
||||
2. Walk relationships where `inheritsposition=true`, ordered by relationship type priority (`partof` first, then `controls`), recursively resolve the related asset's position with cycle detection
|
||||
3. Fall back to the asset's location coords (`locations.mapx`, `locations.mapy` via `assets.locationid`)
|
||||
4. If none of the above, asset is "unplaced" and rendered in a tray, not on the map
|
||||
|
||||
API exposes resolved position with a source indicator: `{"mapx": 234, "mapy": 567, "positionsource": "self|related|location|none"}`.
|
||||
|
||||
### Hierarchical locations
|
||||
|
||||
`locations` is extended with `parentlocationid` (self-FK, nullable). Sites define their own location tree (cells, sub-cells, network closets, meeting rooms, labs).
|
||||
|
||||
`locationtypes` lookup is added with seeded values: `section`, `cell`, `subcell`, `meetingroom`, `lab`, `office`, `storage`, `hallway`, `networkcloset`, `building`. Sites can extend.
|
||||
|
||||
Asset-to-location is the existing `assets.locationid` FK only (single primary location for v1). Multi-location and transient placement are deferred.
|
||||
|
||||
## Migration scope (narrowed)
|
||||
|
||||
Only one class of legacy data migrates: physical manufacturing equipment with a `machinenumber` (5-axis mills, lathes, broachers, heat treatment, CMMs, etc.).
|
||||
|
||||
Migration filter:
|
||||
|
||||
```sql
|
||||
SELECT * FROM legacy.machines
|
||||
WHERE category = 'Equipment'
|
||||
AND machinenumber IS NOT NULL
|
||||
```
|
||||
|
||||
Rows without `machinenumber`: **skipped.** Add manually post-migration if any matter.
|
||||
|
||||
Migrated rows become `assets` with `assettype='Equipment'` plus a row in the equipment plugin's `equipment` table. Legacy `machinetypeid` is preserved on the equipment row to enable later reclassification (some "Equipment" rows are actually measuring tools, see ADR-005).
|
||||
|
||||
Skipped from migration:
|
||||
|
||||
- PCs (rebuilt via PXE collector pipeline, see ADR-006)
|
||||
- Printers, USB devices, network gear, notifications, KB articles (operational or re-collectable, decision per class deferred until needed)
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Plugin authors at sister sites have a single, documented contract to target
|
||||
- Cross-plugin features (relationships, search, map) work uniformly
|
||||
- Three relationship types are easy to learn; free-text label captures nuance
|
||||
- Sibling propagation handles dual-path machines without code changes per use case
|
||||
- Position resolution gives users one map without per-plugin map code
|
||||
|
||||
### Negative
|
||||
|
||||
- Migration of equipment data must run cleanly; `migrating-asset-schema` skill owns the procedure
|
||||
- Frontend `views/machines/` will be repointed or replaced; effort tracked in Phase 5
|
||||
- Legacy `core/api/machines.py` (641 LOC) becomes deprecated; deletion or shim deferred until equipment is migrated
|
||||
|
||||
### Neutral
|
||||
|
||||
- `Machine`, `PCType`, `MachineType`, `PrinterData` remain in the schema until the migration completes, behind a deprecation notice. New code does not touch them.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Keep both models indefinitely.** Doubles test surface, confuses contract docs, breaks cross-plugin features. Rejected.
|
||||
2. **Make `Machine` the contract; retire `Asset`.** `Machine` has shop-floor-specific assumptions that don't generalize. Rejected.
|
||||
3. **Define a higher abstraction above both.** Yet another layer. `Asset` is already the abstraction. Rejected.
|
||||
4. **More relationship types** (`controls`, `operates`, `monitors`, `dncfeeds`, `mountedon`, `connectedvia`, `inspects`, `siblingbay`). Eight types proved unwieldy. Collapsed to three with free-text labels and per-row inheritance/propagation flags.
|
||||
|
||||
## Open questions deferred
|
||||
|
||||
- Lifecycle relationship type (`replaces`, `replacedby`). Add when first needed.
|
||||
- Multi-location asset placement, transient placement (calibration trip, off-site repair). Add when first needed.
|
||||
- Map editing UI (drag locations onto floor plan). Phase 6 polish.
|
||||
- Plugin-extensible relationship types beyond the three seeded. Add when a real cross-plugin case can't be expressed via labels.
|
||||
|
||||
## References
|
||||
|
||||
- `shopdb/core/models/asset.py`
|
||||
- `shopdb/core/models/machine.py` (legacy, deprecated)
|
||||
- `shopdb/plugins/base.py`
|
||||
- ADR-002 (versioning of the surface)
|
||||
- ADR-003 (plugin distribution)
|
||||
- ADR-004 (deployment topology)
|
||||
- ADR-005 (equipment vs measuring tools split)
|
||||
- ADR-006 (collector contract pattern)
|
||||
60
ADR-002-plugin-versioning.md
Normal file
60
ADR-002-plugin-versioning.md
Normal file
@@ -0,0 +1,60 @@
|
||||
# ADR-002: Plugin contract versioning
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
Once sister sites start writing their own plugins (or pulling community plugins), the framework's plugin contract becomes a public API. Without a versioning story, any change to `BasePlugin` or the core platform models can silently break installed plugins at remote sites.
|
||||
|
||||
The existing `BasePlugin` and `PluginMeta` already declare a `core_version` field (default `">=1.0.0"`), but it is not enforced anywhere. The plugin loader does not check it before instantiation.
|
||||
|
||||
## Decision
|
||||
|
||||
The framework adopts **semantic versioning for the plugin contract**, declared in two places:
|
||||
|
||||
1. **Framework version** (`shopdb/__init__.py`): a single `__contract_version__` constant. This is the version of the platform contract as defined in ADR-001. Bumped according to semver:
|
||||
- **Major**: breaking change to `BasePlugin` ABC, `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`).
|
||||
- **Minor**: additive change (new optional hook, new field on a contract model with default).
|
||||
- **Patch**: bug fix, no contract surface change.
|
||||
2. **Plugin requirement** (`plugins/<name>/manifest.json`): the existing `core_version` field, expressed as a semver range (e.g., `">=1.0.0,<2.0.0"`).
|
||||
|
||||
The plugin loader (`shopdb/plugins/loader.py`) checks `core_version` against `__contract_version__` at load time. Mismatch in dev = re-raise (fail loud). Mismatch in prod = log error, mark plugin as incompatible, exclude from registration.
|
||||
|
||||
The `__contract_version__` starts at **`1.0.0`** when ADR-001 is accepted and the `Machine` retirement migration is complete (whichever comes later). Until then, the framework is pre-1.0; plugins should declare `core_version: ">=0.1.0,<1.0.0"`.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Sister sites can pin a known-good framework version. They will not be silently broken when the framework is upgraded.
|
||||
- Plugin authors know what counts as a breaking change because the contract surface is enumerated in ADR-001.
|
||||
- The loader fails predictably: a mismatched plugin is reported, not silently disabled.
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- Discipline required: every change to the contract surface must be classified (major / minor / patch). Adding a `version-bump` skill (or a check in code review) reduces the chance of mis-classification.
|
||||
- `__contract_version__` becomes a coupling point. Forgetting to bump it after a breaking change means downstream plugins crash silently at runtime instead of failing at install.
|
||||
|
||||
### Neutral
|
||||
|
||||
- Existing plugins (`plugins/printers/`, etc.) ship as part of the framework, so their `core_version` is always the current `__contract_version__`. The discipline matters mostly for external / sister-site plugins.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **No versioning, just trust.** Works for an in-tree-only world. Fails the moment a sister site ships its own plugin. Rejected.
|
||||
2. **Calendar versioning** (e.g., `2026.05.0`). Easier to bump, harder to communicate breaking changes. Rejected; semver is the industry standard for library-like contracts.
|
||||
3. **Per-hook versioning.** Each hook has its own version. Too granular; plugins still couple to multiple hooks. Rejected.
|
||||
|
||||
## Open questions
|
||||
|
||||
- When does the framework declare `1.0.0`? Tied to ADR-001 (Asset retirement of Machine) and the framework being deemed "ready for sister sites". Best-effort target: end of Phase 5 in the refactor plan.
|
||||
- Should `core_version` accept commercial-grade ranges (`^1.0.0`) or stick to PEP 440 / npm-style ranges? Recommend pip-style (`>=,<`) to match Python ecosystem.
|
||||
|
||||
## References
|
||||
|
||||
- `shopdb/plugins/base.py` (PluginMeta declaration)
|
||||
- `shopdb/plugins/loader.py` (where the version check belongs)
|
||||
- ADR-001 (defines what is in the contract)
|
||||
68
ADR-003-plugin-distribution.md
Normal file
68
ADR-003-plugin-distribution.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# ADR-003: Plugin distribution model
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
|
||||
## Context
|
||||
|
||||
Sister sites adopting shopdb-flask need a way to:
|
||||
|
||||
1. Install the framework
|
||||
2. Pick which plugins they want
|
||||
3. Build their own plugins for site-specific equipment
|
||||
4. Receive updates to both framework and plugins
|
||||
|
||||
Today, every plugin lives in the `plugins/` directory of the framework repo. There is no separation between framework code and plugin code, no install / uninstall, and no way for a site to develop a plugin without forking the whole repo.
|
||||
|
||||
Three viable distribution models:
|
||||
|
||||
| Model | How a site installs a plugin |
|
||||
|---|---|
|
||||
| **In-tree only** | Fork the framework repo, add plugin under `plugins/`, run their own deploy. No separation. |
|
||||
| **Pip-installable plugins** | Each plugin published as a Python package. Site does `pip install shopdb-printers shopdb-network` etc. Discovery via Python entry points. Framework loads any installed plugin that registers itself. |
|
||||
| **Git-based plugins** | Each plugin lives in its own git repo. Site clones / submodules into `plugins/<name>/`. Loader picks them up from the directory. |
|
||||
|
||||
## Decision
|
||||
|
||||
**PROPOSED:** Use a **hybrid model** with two clearly-labeled paths.
|
||||
|
||||
1. **Bundled plugins**: a small set of plugins ships with the framework, in-tree at `plugins/`. These are the reference implementations and the default install (printers, computers, network, equipment, usb, notifications). A site that wants only what's bundled needs no extra work.
|
||||
2. **External plugins**: sister sites or third parties build plugins in their own git repos. The site running the framework drops the plugin into `plugins/<name>/` (clone, submodule, or symlink) and runs `flask plugin install <name>`. No pip packaging required for v1.
|
||||
|
||||
Pip-installable plugins (Python entry-point discovery) are deferred to v2. The complexity is not justified until at least two sites are running their own plugins.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- v1 is simple: filesystem-based discovery (already implemented in `shopdb/plugins/loader.py`), works for both bundled and external plugins.
|
||||
- Sites can develop plugins without changing the framework repo.
|
||||
- The `plugins/` directory is already the canonical location, so no architectural change is needed.
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- No automatic update path for external plugins. Sites must `git pull` in each plugin directory manually. Acceptable for v1; revisit when plugin count grows.
|
||||
- Multiple plugin authors writing in parallel can collide on namespace (e.g., two plugins both registering an `AssetType` named "Equipment"). Need a naming policy: plugin names and asset-type names should be prefixed with the site or org if not in the bundled set.
|
||||
|
||||
### Neutral
|
||||
|
||||
- The existing in-tree pattern keeps working. This decision just formalizes it and clarifies the path for outside-the-tree plugins.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Pip-installable from day one.** Cleaner for the long term but adds packaging, entry-point registration, and CI steps. Premature for current scale (one site running, no sister-site plugins yet).
|
||||
2. **In-tree only forever.** Forces every site to fork. Doesn't scale beyond two or three sites.
|
||||
3. **Submodules only.** Forces git-submodule discipline on every adopting site. Submodules are notoriously fiddly. Rejected.
|
||||
|
||||
## Open questions
|
||||
|
||||
- For external plugins, should there be a manifest field (`source_url`) declaring where the plugin can be cloned from, so `flask plugin install` could pull it for the site? Defer; manual clone is fine for v1.
|
||||
- Naming convention for non-bundled plugin directory names: prefix with site? (`gea-wjsf-shipping`)? Adopt if and when we hit a name collision.
|
||||
|
||||
## References
|
||||
|
||||
- `shopdb/plugins/loader.py` (filesystem discovery)
|
||||
- `shopdb/plugins/cli.py` (plugin install / uninstall command)
|
||||
- ADR-001 (defines what plugins target)
|
||||
- ADR-002 (defines plugin version compatibility)
|
||||
87
ADR-004-deployment-topology.md
Normal file
87
ADR-004-deployment-topology.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# ADR-004: Deployment topology (per-site instances)
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
|
||||
## Context
|
||||
|
||||
shopdb-flask manages shop-floor inventory. If multiple GE Aerospace sites adopt it, the deployment can take one of two shapes:
|
||||
|
||||
| Model | How it works |
|
||||
|---|---|
|
||||
| **Per-site instances** | Each site runs its own Flask + MySQL + Vue stack. Each site has its own DB, its own users, its own enabled-plugin list, its own deploy. Sites are isolated. |
|
||||
| **Multi-tenant single instance** | One central Flask + MySQL + Vue stack serves all sites. A `siteid` foreign key on every asset partitions data. Auth distinguishes which site a user belongs to. |
|
||||
|
||||
The codebase today is single-tenant per deployment. There is no `siteid` column, no tenant filter, no cross-site auth model. Plugins can be enabled / disabled but only globally for the running instance.
|
||||
|
||||
## Decision
|
||||
|
||||
**PROPOSED:** **Per-site instances.** Each adopting site runs its own dedicated stack. The framework does not support multi-tenancy.
|
||||
|
||||
Each site:
|
||||
|
||||
- Owns its database (own credentials, own backup policy, own retention). The
|
||||
database charset is part of the contract: it must be **utf8mb4**
|
||||
(`utf8mb4_unicode_ci`). The migration chain creates every table utf8mb4, and
|
||||
the connection pins `?charset=utf8mb4`. A site that creates the database with
|
||||
a different default charset (older MySQL defaults to latin1) gets a schema
|
||||
that silently diverges from every other site. See `docs/DEPLOY.md`.
|
||||
- Picks its own enabled plugins
|
||||
- Configures its own JWT secret, CORS allowlist, Zabbix integration, Active Directory binding
|
||||
- Deploys at its own cadence
|
||||
|
||||
The framework provides:
|
||||
|
||||
- A `Dockerfile` and `docker-compose.yml` template suitable for a single-site deploy
|
||||
- A `.env.example` listing all required environment variables
|
||||
- A `docs/DEPLOY.md` walking through a fresh-site install
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Simpler code: no tenant filter on every query, no cross-tenant auth, no shared-state partitioning bugs.
|
||||
- Sites are independent. A schema change at one site does not affect another. A plugin crash at one site does not blast radius to other sites.
|
||||
- Clear ownership: each site's IT team owns their own stack and data. Compliance and audit boundaries match operational boundaries.
|
||||
- Aligns with how GE Aerospace sites already operate (independent IT, independent shop floors).
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- No cross-site reporting out of the box. If GE corporate ever wants a fleet-wide view, it has to be built on top (e.g., a roll-up dashboard that queries each site's API). That layer is out of scope for the framework.
|
||||
- Each site administers its own stack. Higher operational overhead than a single central instance, but each site already runs its own infrastructure.
|
||||
- Updates require visiting each site's deploy. Fine for the current adoption model; revisit if dozens of sites adopt.
|
||||
|
||||
### Neutral
|
||||
|
||||
- No `siteid` column needed. The existence of one DB per site is the partition.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Multi-tenant single instance.** Lower operational overhead at scale, easier cross-site reporting, but adds significant code complexity and risk: every query needs a tenant filter, auth gets complex, schema migrations affect every site at once, and a bug at one site can leak data across sites. Rejected for v1; revisit if and only if more than five sites adopt and operational overhead becomes painful.
|
||||
2. **Hybrid: per-site DB but central app server.** Adds the operational complexity of multi-tenancy without isolating the failure domain (one app crash = all sites down). Rejected.
|
||||
|
||||
## Migration strategy (resolved)
|
||||
|
||||
Deploys run a single core Alembic chain: `flask db upgrade`. Bundled plugins do
|
||||
NOT carry their own migration chains - their tables are folded into the core
|
||||
chain (migration `7c04_fold_plugin_schema`). This was a deliberate resolution of
|
||||
the Phase 7B footgun where bundled-plugin baselines and the core baseline both
|
||||
created the same tables, so `flask plugin upgrade-all` would conflict. A fresh
|
||||
`flask db upgrade` reproduces the live schema exactly (verified on a scratch DB).
|
||||
|
||||
External (out-of-tree) plugins per ADR-003 may still ship their own migrations;
|
||||
the framework supports per-plugin chains for them. Only the in-tree bundled
|
||||
plugins are consolidated into core.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should the framework provide an optional **read-only fleet roll-up** mode where a "central" instance can pull aggregate metrics from each site's API? Defer. Out of scope for v1.
|
||||
- Backup strategy per site: framework recommendation, or each site decides? Framework should publish a recommended backup runbook (mysqldump + offsite copy) but not enforce.
|
||||
- Auth federation: each site has its own user table, or sites can share an LDAP / SSO? Recommend documenting the LDAP config knob in `.env.example` so sites can plug in their own auth without code change.
|
||||
|
||||
## References
|
||||
|
||||
- `shopdb/config.py` (currently single-tenant, no `siteid`)
|
||||
- ADR-001 (asset model is per-site, not cross-site)
|
||||
- ADR-003 (plugin distribution per site)
|
||||
149
ADR-005-equipment-vs-measuringtools.md
Normal file
149
ADR-005-equipment-vs-measuringtools.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# ADR-005: Equipment plugin scope vs measuringtools plugin
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
ADR-001 narrowed the migration to physical manufacturing equipment with a `machinenumber`. In practice, the legacy `category='Equipment'` rows contain two distinct asset classes:
|
||||
|
||||
1. **Manufacturing machinery** (5-axis mills, lathes, broachers, heat treatment ovens). These produce parts.
|
||||
2. **Metrology and inspection instruments** (CMMs, Keyence vision systems, wax-and-trace surface profilometers, GenSpec instruments). These measure parts.
|
||||
|
||||
Both share `Asset` properties (vendor, model, location, controller). They differ in domain fields (axes vs measurement accuracy, cycle time vs calibration interval, controller protocol vs measurement software).
|
||||
|
||||
Mixing them under one plugin pollutes the schema and confuses cross-plugin queries ("show me all measuring tools" requires an enumeration of measuring-instrument equipmenttype values, which scales badly).
|
||||
|
||||
## Decision
|
||||
|
||||
Two plugins, separate concerns, shared platform contract.
|
||||
|
||||
### `equipment` plugin
|
||||
|
||||
Tracks manufacturing machinery. Bundled, in-tree.
|
||||
|
||||
Schema (per ADR-001 contract):
|
||||
|
||||
```
|
||||
equipment
|
||||
- assetid FK to assets, PK
|
||||
- equipmenttypeid FK to equipmenttypes (5-axis mill, lathe, broacher, heat treat, ...)
|
||||
- vendorid FK to vendors (platform)
|
||||
- modelid FK to models (platform)
|
||||
- controllertypeid FK to controllertypes (equipment plugin)
|
||||
- controllerosid FK to controlleros (equipment plugin)
|
||||
- (other shared fields: spindle count, axes, max workpiece size, ...)
|
||||
|
||||
equipmenttypes (lookup, equipment plugin)
|
||||
- equipmenttypeid, name (5-axis mill, lathe, broacher, heat treat, ...)
|
||||
|
||||
controllertypes (lookup, equipment plugin)
|
||||
- controllertypeid, name (Fanuc 31i, Siemens 840D, Mitsubishi M70, Heidenhain TNC640, ...)
|
||||
- vendorid (FK to vendors)
|
||||
|
||||
controlleros (lookup, equipment plugin - separate from PC OS)
|
||||
- controllerosid, name (FAPT, VxWorks, embedded Windows, Linux RT, ...)
|
||||
|
||||
equipmentfocas (subtype, optional, present only when FOCAS-equipped)
|
||||
- assetid PK, FK to equipment
|
||||
- focasipaddress text
|
||||
- focasport integer
|
||||
- focasversion text
|
||||
- focasmachinenumber text
|
||||
|
||||
equipmentclm (subtype, optional, present only when CLM-equipped)
|
||||
- assetid PK, FK to equipment
|
||||
- (CLM-specific: address, port, station ID - finalize when plugin is built)
|
||||
|
||||
equipmentmtconnect (subtype, optional, present only when MTConnect-equipped)
|
||||
- assetid PK, FK to equipment
|
||||
- mtconnectagenturl text
|
||||
- mtconnectdevicename text
|
||||
```
|
||||
|
||||
The `equipment.protocol` enum field is deliberately **not** included. Presence or absence of a subtype row indicates which protocol applies. Avoids a denormalized field that can drift out of sync.
|
||||
|
||||
### `measuringtools` plugin
|
||||
|
||||
Tracks metrology and inspection instruments. Bundled, in-tree (built in Phase 3-4 of the refactor as the first new plugin built using the framework scaffold).
|
||||
|
||||
Schema (initial draft, refined when plugin is built):
|
||||
|
||||
```
|
||||
measuringtools
|
||||
- assetid FK to assets, PK
|
||||
- measuringtooltypeid FK to measuringtooltypes (CMM, vision system, profilometer, surface tester, ...)
|
||||
- vendorid FK to vendors (platform)
|
||||
- modelid FK to models (platform)
|
||||
- measurementaxes integer (e.g., 3 for a 3-axis CMM)
|
||||
- accuracyspec text (e.g., "+/-0.5um")
|
||||
- calibrationintervaldays integer
|
||||
- lastcalibrationdate date
|
||||
- nextcalibrationdate date (computed)
|
||||
- (other domain fields as needed)
|
||||
|
||||
measuringtooltypes (lookup, measuringtools plugin)
|
||||
- measuringtooltypeid, name (CMM, vision system, surface profilometer, gage block, ...)
|
||||
```
|
||||
|
||||
Future extension: subtype tables for measurement-software integrations (PC-DMIS, Keyence, GenSpec). Same pattern as equipment subtype tables.
|
||||
|
||||
### Subtype-table pattern (general)
|
||||
|
||||
Both plugins use the same pattern for protocol- or software-specific fields:
|
||||
|
||||
- Core plugin table carries shared, common fields
|
||||
- Optional subtype tables (one per protocol or software) hold extension fields
|
||||
- Each subtype table is keyed by `assetid` (PK), one-to-one with the parent
|
||||
- Subtype row exists if and only if the asset uses that protocol or software
|
||||
- Sister sites add new subtype tables for their own integrations without touching core
|
||||
|
||||
## Reclassification of legacy data
|
||||
|
||||
ADR-001's migration moves all legacy `category='Equipment' AND machinenumber IS NOT NULL` rows to `assets` with `assettype='Equipment'` and into the equipment plugin's `equipment` table. This includes both manufacturing machinery and measuring tools.
|
||||
|
||||
After the equipment migration, when the measuringtools plugin is built:
|
||||
|
||||
1. Build a mapping table: legacy `machinetypeid` values that are measuring tools (CMM type, Keyence type, etc.)
|
||||
2. Run a reclassification script:
|
||||
- For each `assets` row where the original `machinetypeid` is in the measuring-tool mapping
|
||||
- Change `assets.assettype` to `'MeasuringTool'`
|
||||
- Move the row from `equipment` to `measuringtools`
|
||||
- Map domain fields where they differ (e.g., legacy `axes` field maps to `measurementaxes`)
|
||||
3. Verify counts pre- and post-reclassification
|
||||
4. Audit log entry per reclassified row
|
||||
|
||||
Reclassification is one-shot, run once, archived. Like the original migration script.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Manufacturing machinery and measuring tools are first-class plugins, each with appropriate domain fields
|
||||
- Sister sites can install one or both depending on what they track
|
||||
- Subtype-table pattern is the canonical example for protocol-specific data and extends naturally to other plugins
|
||||
- Building `measuringtools` mid-refactor validates the plugin scaffold tooling against a real new plugin
|
||||
|
||||
### Negative
|
||||
|
||||
- Reclassification is a second migration step. Lower risk than the initial migration because it is data-only (no schema change beyond moving rows between two tables that share the same `assetid` link).
|
||||
- Sites that adopt the framework before `measuringtools` ships need to either keep measuring tools in `equipment` (workable but suboptimal) or wait for the plugin
|
||||
|
||||
### Neutral
|
||||
|
||||
- Legacy `machinetypeid` is preserved on the equipment row during migration to enable reclassification
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Single equipment plugin with sub-typed assets.** Use `equipment.equipmenttypeid` to discriminate manufacturing vs metrology. Rejected: domain fields differ enough that a single table is wide and full of NULLs.
|
||||
2. **Migrate split (build mapping before initial migration).** Cleaner end state but requires the `measuringtools` plugin to exist before the migration runs, which delays Phase 5. Rejected.
|
||||
3. **JSON blob for protocol data instead of subtype tables.** Considered for both plugins. Rejected: weak typing, awkward queries, no schema validation.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-001 (Asset is platform contract)
|
||||
- ADR-002 (versioning of the surface)
|
||||
- `plugins/equipment/` (current placeholder)
|
||||
- `plugins/computers/` (existing example of plugin pattern)
|
||||
148
ADR-006-collector-contract.md
Normal file
148
ADR-006-collector-contract.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# ADR-006: Plugin collector contract pattern
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-05-08
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
PC inventory data was collected by PowerShell scripts pushing to `/api/collector/pc` (`shopdb/core/api/collector.py`, ~374 LOC). The endpoint is hardcoded for PCs: it accepts a fixed schema and writes to the legacy `Machine` model.
|
||||
|
||||
Per ADR-001, `Machine` is being retired in favor of `Asset`. Per the project shift to PXE-driven imaging, PC inventory is moving to a new collection pipeline (PXE / GE-Enforce / manifest engine produces JSON about each PC). Other asset classes may want similar collector pipelines (printers via Zabbix, network gear via SNMP scan).
|
||||
|
||||
This calls for a generalizable contract: any plugin that wants to accept external collector input declares a JSON schema, and the framework wires the endpoint, auth, and idempotency.
|
||||
|
||||
## Decision
|
||||
|
||||
`BasePlugin` gets two new hooks (added in __contract_version__ 0.2.x -> the surface is carried at 0.3.0):
|
||||
|
||||
```python
|
||||
def get_collector_schema(self) -> Optional[dict]:
|
||||
"""Return JSON Schema describing the collector payload for this plugin.
|
||||
Return None if the plugin does not accept collector input.
|
||||
|
||||
The schema must include:
|
||||
- 'identityfield': name of the field that uniquely identifies an asset
|
||||
across submissions (e.g., 'hostname' for PCs, 'macaddress' for network
|
||||
devices). Used for idempotent upsert.
|
||||
- 'fields': JSON Schema definitions for the rest of the payload.
|
||||
"""
|
||||
return None
|
||||
|
||||
def apply_collector_payload(self, payload: dict) -> dict:
|
||||
"""Idempotently upsert an asset from a validated collector payload.
|
||||
|
||||
Called by /api/collector/<pluginname> after identity validation.
|
||||
CONDITIONAL hook: required only when get_collector_schema returns
|
||||
non-None. Default raises NotImplementedError (the dispatcher returns
|
||||
500) so a schema-without-upsert fails loud. Returns a dict with at
|
||||
least 'action' ('created'|'updated'|'noop'), 'assetid', 'warnings'.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
```
|
||||
|
||||
The pairing (schema present => apply implemented) is enforced by the
|
||||
`test_schema_declaring_plugins_implement_apply` contract test.
|
||||
|
||||
A single dynamic dispatch route `/api/collector/<pluginname>` serves every
|
||||
plugin that returns a schema (rather than registering a blueprint per plugin),
|
||||
because Flask forbids `register_blueprint` after the first request and plugins
|
||||
can be enabled at runtime. Auth is API-key, separate from JWT. Per-plugin keys via env vars:
|
||||
|
||||
- `COLLECTOR_API_KEY_<PLUGINNAME>` (preferred, plugin-specific)
|
||||
- `COLLECTOR_API_KEY` (fallback, shared)
|
||||
|
||||
### Idempotent upsert
|
||||
|
||||
The endpoint uses the `identityfield` to find an existing `Asset` for the same identity. Found = update. Not found = insert. Existing relationships are preserved on update.
|
||||
|
||||
### Response contract
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"action": "created" | "updated" | "noop",
|
||||
"assetid": 12345,
|
||||
"identityvalue": "PC-1234",
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
### Audit logging
|
||||
|
||||
Every collector submission produces an audit log entry: `{action, plugin, identityvalue, before/after diff}`. Audit retention per site policy.
|
||||
|
||||
### Schema discovery
|
||||
|
||||
The framework exposes the registered schemas at `/api/collector/_schemas` (read-only, JWT-protected) so external collector authors can introspect what payloads are accepted by which plugins.
|
||||
|
||||
## Concrete first user: computers plugin
|
||||
|
||||
The `computers` plugin is the first to implement `get_collector_schema`. The PXE pipeline conforms.
|
||||
|
||||
Initial computers collector schema (sketch, finalized when plugin is built):
|
||||
|
||||
```json
|
||||
{
|
||||
"identityfield": "hostname",
|
||||
"fields": {
|
||||
"hostname": "string, required",
|
||||
"macaddress": "string, optional, secondary identity",
|
||||
"osname": "string",
|
||||
"osversion": "string",
|
||||
"lastboottime": "datetime",
|
||||
"currentuser": "string",
|
||||
"ipaddress": "string",
|
||||
"memorygb": "number",
|
||||
"cputype": "string",
|
||||
"imagename": "string (PXE image deployed)",
|
||||
"imageappliedat": "datetime",
|
||||
"installedsoftware": "array of {name, version}"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The PC re-image case is handled by the identity field: a freshly imaged PC keeps its hostname, so the existing `Asset` row is updated rather than duplicated. Existing `AssetRelationship` rows pointing at that PC (e.g., `controls` to a machine) are preserved across re-images.
|
||||
|
||||
## Migration of the existing endpoint
|
||||
|
||||
`shopdb/core/api/collector.py` (`/api/collector/pc`) is **deprecated** in v1 and **removed** before v1.0.
|
||||
|
||||
Migration path:
|
||||
|
||||
1. Implement `get_collector_schema` on the `computers` plugin. New endpoint `/api/collector/computers` is auto-registered.
|
||||
2. Run both endpoints in parallel for one cycle of PXE imaging across the floor. PXE pipeline switches to `/api/collector/computers`.
|
||||
3. Remove `shopdb/core/api/collector.py` and the legacy blueprint registration.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Generalizable across plugins. Sister sites adopting `printers`, `network`, etc. can wire their own collectors with no core change.
|
||||
- Identity-based idempotency makes PC re-imaging safe by default.
|
||||
- Audit logging is uniform across plugins.
|
||||
- Schema discovery enables external tools to validate before submission.
|
||||
|
||||
### Negative
|
||||
|
||||
- Plugin authors must write a JSON schema. Slight learning curve, but JSON Schema is widely understood and the framework can ship a few examples.
|
||||
- The `/api/collector/_schemas` endpoint plus per-plugin endpoints expand the public API surface; minor maintenance cost.
|
||||
|
||||
### Neutral
|
||||
|
||||
- API-key auth pattern stays as it is today (separate from JWT). Sites manage their own collector keys per plugin via env vars.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Keep `/api/collector/pc` and add new plugin-specific endpoints alongside.** Two ways to send PC data, plugin authors confused. Rejected.
|
||||
2. **Use JWT for collectors instead of API key.** Collectors are headless processes (PXE pipeline, scripts), not interactive users. JWT lifecycle (refresh tokens, expiry) is the wrong tool. API key is simpler. Rejected.
|
||||
3. **Plugins write directly to the database, no collector endpoint.** Skips audit logging and schema validation. Rejected.
|
||||
|
||||
## References
|
||||
|
||||
- `shopdb/core/api/collector.py` (legacy endpoint to be removed)
|
||||
- `shopdb/plugins/base.py` (`get_collector_schema` + `apply_collector_payload` hooks)
|
||||
- ADR-001 (asset model the collectors target)
|
||||
- ADR-002 (collector schema is part of plugin contract; changes to the hook signature are major bumps)
|
||||
- The PXE project (`/home/camp/projects/pxe/`) which feeds the computers collector
|
||||
115
ADR-007-product-versioning-and-releases.md
Normal file
115
ADR-007-product-versioning-and-releases.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# ADR-007: Product versioning and releases
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-10
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
ADR-002 established semantic versioning for the plugin contract via
|
||||
`__contract_version__` in `shopdb/__init__.py`. That number answers one
|
||||
question only: is a given plugin compatible with this platform build. It
|
||||
says nothing about the state of the product as a whole. A sister site
|
||||
standing up its own instance needs a different answer: which release am I
|
||||
running, and what changed since the last one.
|
||||
|
||||
Until now the product had no version, no tags, no changelog, and no CI.
|
||||
ADR-002's pinning model implicitly assumes that a downstream site can
|
||||
pin a known-good build, but there were no tags to pin to. Adopters had no
|
||||
release record to read before upgrading, and no automated gate confirming
|
||||
that a given commit builds and passes tests.
|
||||
|
||||
## Decision
|
||||
|
||||
The product carries its own release version, separate from the plugin
|
||||
contract version.
|
||||
|
||||
1. **Product version** (`shopdb/__init__.py`): a single `__version__`
|
||||
constant. This is the version of the shopdb-flask product as a whole.
|
||||
It follows semantic versioning of the product's user-visible and
|
||||
operator-visible behavior. It is deliberately NOT part of the
|
||||
`shopdb.api` contract surface, so it is never re-exported through
|
||||
`shopdb.api`; exporting it would itself be a contract change under
|
||||
ADR-002.
|
||||
|
||||
2. **Plugin contract version** (`__contract_version__`): unchanged from
|
||||
ADR-002. It moves only when the plugin contract surface changes, per
|
||||
the major/minor/patch rules in ADR-002.
|
||||
|
||||
The two are distinct series with independent bump rules. They happen to
|
||||
coincide at `0.5.0` for this release; that is a coincidence of timing,
|
||||
not a coupling. A product release that changes no contract surface bumps
|
||||
`__version__` while leaving `__contract_version__` fixed, and vice versa.
|
||||
|
||||
3. **Release record** (`CHANGELOG.md`, repo root): the canonical,
|
||||
human-readable record of what changed in each release, in
|
||||
Keep-a-Changelog format. Every release has an entry; work in flight
|
||||
accumulates under `## [Unreleased]`.
|
||||
|
||||
4. **Git tags**: each product release is tagged `vX.Y.Z`, where `X.Y.Z`
|
||||
matches `__version__` at the tagged commit. Tags are what ADR-002's
|
||||
downstream-pinning model pins to.
|
||||
|
||||
5. **Frontend version** (`frontend/package.json`): kept in lock-step with
|
||||
`__version__` so the shipped single-page app reports the same product
|
||||
version as the backend that serves it.
|
||||
|
||||
### Release procedure
|
||||
|
||||
To cut release `X.Y.Z`:
|
||||
|
||||
1. Bump `__version__` in `shopdb/__init__.py` to `X.Y.Z`.
|
||||
2. Bump `version` in `frontend/package.json` to the same `X.Y.Z`.
|
||||
3. Move the accumulated `## [Unreleased]` notes in `CHANGELOG.md` into a
|
||||
new `## [X.Y.Z] - YYYY-MM-DD` section, leaving a fresh empty
|
||||
`## [Unreleased]` above it.
|
||||
4. If the plugin contract surface changed this release, bump
|
||||
`__contract_version__` per ADR-002 (independently of `__version__`).
|
||||
5. Commit, then tag: `git tag -a vX.Y.Z -m "shopdb-flask X.Y.Z"`.
|
||||
6. Push the commit and the tag.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Adopters have a real release to name in bug reports and a changelog to
|
||||
read before upgrading.
|
||||
- ADR-002's pinning story finally has tags to pin to.
|
||||
- Product and contract can evolve at their own pace without one dragging
|
||||
the other into a misleading bump.
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- Two version numbers to keep straight. The comment in
|
||||
`shopdb/__init__.py` and this ADR exist to keep the distinction clear.
|
||||
- Release discipline: the changelog and both version constants must be
|
||||
updated together, or the tagged build misreports itself.
|
||||
|
||||
### Neutral
|
||||
|
||||
- CI (`.gitea/workflows/ci.yml`) runs the backend tests, the naming/style
|
||||
gate, and the frontend build on push and PR. It is best-effort: Gitea
|
||||
Actions availability on the host is unverified, so the workflow is
|
||||
config-only until a runner is confirmed.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Reuse `__contract_version__` as the product version.** Conflates two
|
||||
independent concerns; a docs-only or UI-only release would either
|
||||
falsely bump the contract or leave the product looking unchanged.
|
||||
Rejected.
|
||||
2. **Calendar versioning for the product** (e.g. `2026.07.0`). Easy to
|
||||
bump but poor at signaling breaking operator-facing changes. Rejected
|
||||
for the same reasons ADR-002 rejected it for the contract.
|
||||
3. **No product version, rely on git SHAs.** Opaque to adopters and
|
||||
unpinnable in any human-meaningful way. Rejected.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-001 (defines the contract surface)
|
||||
- ADR-002 (plugin contract versioning; `__contract_version__` bump rules)
|
||||
- `shopdb/__init__.py` (`__version__`, `__contract_version__`)
|
||||
- `CHANGELOG.md` (release record)
|
||||
- `frontend/package.json` (frontend version, kept in lock-step)
|
||||
- `.gitea/workflows/ci.yml` (CI gate)
|
||||
123
ADR-008-plugin-migration-ownership.md
Normal file
123
ADR-008-plugin-migration-ownership.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# ADR-008: Plugin migration ownership (per-plugin chains from the cutover)
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-10
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** the "Migration strategy (resolved)" section of ADR-004
|
||||
|
||||
## Context
|
||||
|
||||
ADR-004 resolved a Phase 7B footgun by folding every bundled plugin's tables
|
||||
into the single core Alembic chain (migration `7c04_fold_plugin_schema`), and
|
||||
later core migrations (`7d04`..`7d16`) kept adding plugin schema directly to the
|
||||
core chain. At the time this was the safe choice: bundled-plugin baselines and
|
||||
the core baseline had both been creating the same tables, so
|
||||
`flask plugin upgrade-all` would collide with `flask db upgrade`.
|
||||
|
||||
That resolution left the per-plugin Alembic engine
|
||||
(`shopdb/plugins/migrations.py`, `shopdb/plugins/alembic_template.py`, the
|
||||
`alembic_version_<plugin>` version tables, and `flask plugin upgrade-all`) fully
|
||||
built but unused for the bundled plugins. The framework is the product (per the
|
||||
project's charter); a plugin that cannot own its own schema is not really a
|
||||
plugin. Sister sites that adopt or fork a plugin need its schema history to
|
||||
travel with the plugin, not be entangled in the host's core chain. Keeping every
|
||||
future plugin table change in the core chain also means the core chain grows
|
||||
without bound and a plugin can never be cleanly removed.
|
||||
|
||||
The blocker ADR-004 worried about (double table creation) only exists while a
|
||||
plugin's own migration tries to CREATE tables the core chain already created.
|
||||
That is avoidable: history is immutable, so the tables already built by the core
|
||||
chain stay owned by the core chain; only NEW schema needs a new home.
|
||||
|
||||
## Decision
|
||||
|
||||
Ownership splits at a fixed cutover, the current core-chain head.
|
||||
|
||||
1. **Core chain owns history through its head.** The core Alembic chain
|
||||
(baseline `68b3947ae14f` .. head `7d16_directoryemployees`) remains
|
||||
authoritative for every table that exists at the cutover, including the
|
||||
bundled-plugin tables it created. Those migrations are immutable and are not
|
||||
rewritten. `flask db upgrade` continues to reproduce the full schema.
|
||||
|
||||
2. **Plugin chains own plugin schema going forward.** From the cutover forward,
|
||||
any change to a plugin's schema lands as
|
||||
`plugins/<name>/migrations/versions/000N_*.py` in that plugin's own chain,
|
||||
never in the core chain. The core chain is reserved for core tables.
|
||||
|
||||
3. **Every table-owning bundled plugin gets a `0001` anchor.** Each such plugin
|
||||
carries a migration chain whose first revision is a stamp-only no-op:
|
||||
`upgrade()` does nothing because the core chain already created the tables.
|
||||
The anchor exists so the plugin chain has a base that
|
||||
`flask plugin upgrade-all` can stamp into the per-plugin version table
|
||||
`alembic_version_<plugin>`. Blueprint-only plugins that own no tables get no
|
||||
chain.
|
||||
|
||||
4. **Deploy and upgrade sequence.** A deploy runs `flask db upgrade`
|
||||
(core chain, creates everything through the head) then
|
||||
`flask plugin upgrade-all` (stamps every plugin anchor and applies any later
|
||||
per-plugin migrations). The same two commands upgrade an existing install;
|
||||
both are idempotent. The registry (`instance/plugins.json`) tracks each
|
||||
plugin's applied revisions in `migrations_applied`.
|
||||
|
||||
5. **Table-ownership registry.** `PLUGIN_TABLE_OWNERS` in
|
||||
`shopdb/plugins/alembic_template.py` is the explicit map of which tables each
|
||||
plugin owns; it is kept in sync with the plugins' `__tablename__` declarations
|
||||
and pinned by `tests/test_plugin_migrations.py`.
|
||||
|
||||
External (out-of-tree) plugins per ADR-003 already shipped their own chains;
|
||||
this ADR brings the bundled plugins onto the same model, so there is one rule
|
||||
for all plugins.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- A plugin's schema history travels with the plugin. Adopting or forking sites
|
||||
get the plugin's migrations, not a slice of someone else's core chain.
|
||||
- The core chain stops accreting plugin schema; it stays about core tables.
|
||||
- A plugin can be evolved (or, with its own downgrade, removed) independently.
|
||||
- The long-built per-plugin Alembic engine is finally exercised on every deploy,
|
||||
so it cannot silently rot.
|
||||
|
||||
### Negative / cost
|
||||
|
||||
- Two migrate commands per deploy instead of one. Documented in `docs/DEPLOY.md`
|
||||
and `docs/UPGRADE.md`; both are idempotent so the cost is one extra safe call.
|
||||
- A plugin author must now put new tables in the plugin chain and register them
|
||||
in `PLUGIN_TABLE_OWNERS`, rather than autogenerating into the core chain. The
|
||||
`flask plugin new` guidance and this ADR spell that out.
|
||||
- The cutover is a discontinuity: tables created before it are core-owned,
|
||||
tables created after it are plugin-owned. The line is the core-chain head at
|
||||
this ADR's date, recorded here so it is unambiguous.
|
||||
|
||||
### Neutral
|
||||
|
||||
- No schema changes and no data migration: the anchors are no-ops. A fresh
|
||||
install and an existing install converge to the same state.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Keep everything in the core chain (status quo per ADR-004).** Simplest
|
||||
operationally but defeats the plugin-as-product goal: plugin schema cannot
|
||||
travel, the core chain grows without bound, and plugins can never be cleanly
|
||||
removed. Rejected.
|
||||
2. **Rewrite history so plugin tables move out of the core chain into plugin
|
||||
`0001` CREATE migrations.** Would make each plugin chain self-contained from
|
||||
empty, but breaks the immutability rule, forces every existing site to
|
||||
re-run a rewritten chain, and re-introduces the exact double-creation footgun
|
||||
ADR-004 fixed. Rejected.
|
||||
3. **Anchor that CREATEs tables with `IF NOT EXISTS` guards.** Lets a from-empty
|
||||
install build plugin tables from the plugin chain, but then two chains both
|
||||
claim the same tables and drift can diverge silently. The no-op anchor keeps
|
||||
a single authoritative creator (the core chain) for cutover-era tables.
|
||||
Rejected.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-003 (plugin distribution; external plugins already ship chains)
|
||||
- ADR-004 (deployment topology; this ADR supersedes its migration-strategy note)
|
||||
- `shopdb/plugins/alembic_template.py` (`PLUGIN_TABLE_OWNERS`, shared env runner)
|
||||
- `shopdb/plugins/migrations.py`, `shopdb/plugins/cli.py` (`upgrade-all`)
|
||||
- `plugins/<name>/migrations/` (per-plugin chains and `0001` anchors)
|
||||
- `tests/test_plugin_migrations.py` (ownership + chain + idempotency guards)
|
||||
- `docs/DEPLOY.md`, `docs/UPGRADE.md`, `docs/PLUGINS.md` (deploy sequence)
|
||||
146
ADR-009-frontend-plugin-gating.md
Normal file
146
ADR-009-frontend-plugin-gating.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# ADR-009: Frontend plugin gating
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-10
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
The backend plugin system (ADR-002, ADR-003) lets an operator disable a
|
||||
plugin. Disabling it unregisters the plugin's API blueprint, drops its
|
||||
rows from global search (see `test_search_disabled`), and removes its
|
||||
entry from `get_navigation_items()`, so the disabled feature's sidebar
|
||||
link disappears.
|
||||
|
||||
The frontend told a different story. Every plugin's Vue routes and views
|
||||
ship inside the core bundle. `frontend/src/router/index.js` auto-discovers
|
||||
them with `import.meta.glob('./routes/*.js')`, so a route like `/usb` or
|
||||
`/printers/3` is registered regardless of whether the owning backend
|
||||
plugin is enabled. A user who typed the URL, followed a stale bookmark,
|
||||
or clicked a cross-link reached a page whose API calls all 404, landing
|
||||
on a broken shell instead of a clean redirect. The navigation was already
|
||||
data-driven; direct-URL reachability was the gap.
|
||||
|
||||
There is no frontend plugin system yet. The product is packaged as one
|
||||
core Flask app plus backend-only plugins; the Vue app is monolithic and
|
||||
build-time static. Any gating has to live in core frontend code because
|
||||
that is the only place the plugin routes exist.
|
||||
|
||||
## Decision
|
||||
|
||||
### Step 1 (this ADR, implemented): route-level gating
|
||||
|
||||
Plugin-owned frontend routes are gated against the backend's enabled-plugin
|
||||
list. This is the whole of what ships now.
|
||||
|
||||
1. **Enabled-list endpoint.** A new `GET /api/plugins/enabled`
|
||||
(`jwt_required(optional=True)`) returns a flat JSON array of enabled
|
||||
plugin name strings and nothing else. It is a cheap registry read
|
||||
(`registry.get_enabled_plugins()`), no database access. Exposing it to
|
||||
anonymous callers is safe because `GET /api/dashboard/navigation`
|
||||
already leaks the same enabled/disabled signal, and unauthenticated
|
||||
kiosk routes (`/tv`) need the answer too. It carries no metadata, so it
|
||||
reveals strictly less than the admin-gated `GET /api/plugins`.
|
||||
|
||||
2. **Route tagging.** Every plugin-owned route carries `meta.plugin =
|
||||
'<pluginname>'`. This covers the per-plugin route modules
|
||||
(`routes/computers.js`, `routes/equipment.js`, ...) and the
|
||||
plugin-owned routes that physically live in core files: the
|
||||
PC-relationships and toner reports, the slide manager and `/tv`
|
||||
dashboard (slides), the printer-QR and USB-label print pages, and the
|
||||
employee-detail page. Genuinely core routes (dashboard, search, map,
|
||||
applications, reference-data settings) stay untagged and are never
|
||||
gated.
|
||||
|
||||
3. **Cached fetch, fail-open.** A composable
|
||||
(`composables/enabledPlugins.js`) fetches the list exactly once behind
|
||||
a cached promise. If the fetch fails or returns a non-array, the code
|
||||
fails **open**: every plugin is treated as enabled. A transient API
|
||||
error must never brick navigation. The cost is that a disabled
|
||||
plugin's page is briefly reachable during an outage, which is
|
||||
acceptable because its API calls would fail anyway and the next
|
||||
successful fetch closes the gap.
|
||||
|
||||
4. **Router guard.** `router.beforeEach` awaits the cached fetch when the
|
||||
target route has `meta.plugin`. If that plugin is not enabled it
|
||||
redirects to `/` and raises an info toast. The endpoint is
|
||||
jwt-optional, so the guard works for both authenticated pages and the
|
||||
unauthenticated `/tv` kiosk route.
|
||||
|
||||
This is intentionally a thin layer. It does not change the plugin
|
||||
contract surface, so `__contract_version__` does not move: adding a core
|
||||
HTTP endpoint and tagging core-shipped routes are not plugin-contract
|
||||
changes. Plugins still ship no frontend code of their own.
|
||||
|
||||
### Non-goals for step 1
|
||||
|
||||
- No build-time or runtime loading of plugin-authored Vue code.
|
||||
- No per-permission or per-role route gating (that stays with the
|
||||
existing `requiresAuth` / `requiresAdmin` meta flags).
|
||||
- No removal of disabled routes from the route table; they remain
|
||||
registered and are intercepted by the guard. Keeping them registered
|
||||
avoids a rebuild when a plugin is toggled and keeps the redirect path
|
||||
simple.
|
||||
|
||||
## Future direction (PROPOSED, not implemented)
|
||||
|
||||
Step 1 gates routes that core already owns. The longer-term goal is a
|
||||
real frontend-plugin contract where a plugin ships its own frontend and
|
||||
core discovers it, mirroring the backend model. Sketch:
|
||||
|
||||
1. **Plugin-owned frontend tree.** Each plugin gains a
|
||||
`plugins/<name>/frontend/` directory holding its route module, views,
|
||||
and any plugin-specific components. Core stops carrying `views/usb`,
|
||||
`views/printers`, and so on.
|
||||
|
||||
2. **Build-time discovery.** The Vite build discovers plugin frontends
|
||||
with a glob over `plugins/*/frontend/routes.js` (analogous to today's
|
||||
`import.meta.glob('./routes/*.js')`), so a plugin's presence in the
|
||||
tree is what puts its routes in the bundle. Combined with the step-1
|
||||
enabled-list gate, a plugin that is absent from the build ships no
|
||||
code and a plugin that is present-but-disabled is route-gated at
|
||||
runtime.
|
||||
|
||||
3. **Shared component + registration contract.** Plugins register into
|
||||
named extension points instead of editing core files: an `iconMap`
|
||||
registration for nav/asset icons, asset-detail panels, map-marker
|
||||
renderers, and search-result renderers (the "Frontend hook contract"
|
||||
already listed as deferred in docs/ROADMAP.md). Core exposes a
|
||||
stable set of shared components (form controls, detail-page shells,
|
||||
table primitives) as the plugin frontend's only allowed core imports,
|
||||
the frontend analogue of the `shopdb.api` namespace.
|
||||
|
||||
4. **Versioned frontend contract.** The shared-component and
|
||||
registration surface would be versioned the same way the backend
|
||||
contract is (ADR-002), so a plugin frontend can declare the core
|
||||
frontend range it needs.
|
||||
|
||||
### Tradeoffs of the future direction
|
||||
|
||||
- **Pro:** true plugin self-containment; a site can drop in or remove a
|
||||
plugin (frontend and backend together) without patching core; smaller
|
||||
core; clearer ownership.
|
||||
- **Con:** significant build-system work (per-plugin Vite entry
|
||||
discovery, code-splitting, dev-server HMR across the plugin tree); a
|
||||
new versioned frontend contract to maintain and document; a migration
|
||||
that moves ten plugins' worth of views out of core; risk of a leaky
|
||||
shared-component surface becoming an accidental contract. The payoff
|
||||
only matters once external (out-of-tree) plugins with their own
|
||||
frontends are a real requirement. Until then, step 1's route gating
|
||||
delivers the user-visible correctness (no reachable dead pages) at a
|
||||
fraction of the cost.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Disabling a backend plugin now makes its frontend routes redirect to
|
||||
the dashboard instead of loading a broken shell. Behavior matches the
|
||||
already-dynamic navigation.
|
||||
- One extra lightweight request at app boot (`GET /api/plugins/enabled`),
|
||||
cached for the session.
|
||||
- Fail-open means gating is a UX guardrail, not a security control. It is
|
||||
not a substitute for backend authorization: the API still enforces auth
|
||||
and the disabled plugin's endpoints are simply unregistered. Never rely
|
||||
on route gating to protect data.
|
||||
- The future frontend-plugin contract remains open work; this ADR records
|
||||
the direction and its cost so a later decision can pick it up.
|
||||
320
ADR-010-frontend-plugin-hooks.md
Normal file
320
ADR-010-frontend-plugin-hooks.md
Normal file
@@ -0,0 +1,320 @@
|
||||
# ADR-010: Frontend plugin hook contract
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-11
|
||||
- **Accepted:** 2026-07-11
|
||||
- **Deciders:** cproudlock
|
||||
- **Supersedes:** none
|
||||
|
||||
## Context
|
||||
|
||||
The backend plugin contract is settled (ADR-002 surface, ADR-006 collector,
|
||||
ADR-008 migrations, ADR-009 route gating). The one undefined piece before 1.0 is
|
||||
the frontend: how a plugin contributes UI without hand-editing core Vue files.
|
||||
CONTRACT-STABILITY.md names this the single biggest churn item ("no server-side
|
||||
hook for asset-detail panels, map markers, or search-result rendering; a plugin
|
||||
that needs custom UI still hand-edits the Vue frontend").
|
||||
|
||||
The `measuringtools` plugin (bundled 2026-07-11, whose construction is narrated
|
||||
in docs/PLUGIN-GUIDE.md) is the fresh evidence. Integrating it needed exactly
|
||||
**two hand edits to core frontend files**:
|
||||
|
||||
1. `frontend/src/api/index.js` - an api-client block appended after
|
||||
`warrantyApi` (PLUGIN-GUIDE.md section 9). The scaffolder now emits a
|
||||
paste-in snippet, but it is still a hand edit to a shared, churn-heavy file.
|
||||
2. `frontend/src/views/settings/settingsNav.js` - a card entry in the hardcoded
|
||||
`settingsGroups` catalog (the "Measuring Tools" group, ~line 51), consumed by
|
||||
`SettingsLayout` (left rail) and `SettingsIndex` (landing overview).
|
||||
|
||||
Everything else integrated with **zero** core edits, because a mechanism already
|
||||
existed for it:
|
||||
|
||||
| Capability | Existing mechanism |
|
||||
|---|---|
|
||||
| Views / routes | Auto-discovery: `frontend/src/router/index.js` globs `router/routes/*.js` via `import.meta.glob`; a route starting `settings/` auto-nests under the settings shell. |
|
||||
| Disabled-plugin gating | `meta: { plugin: '<name>' }` per ADR-009; router guard redirects when the backend plugin is off. |
|
||||
| Sidebar / dashboard / reports | Declarative backend hooks rendered by core: `get_navigation_items`, `get_dashboard_widgets`, `get_reports` on `BasePlugin` (shopdb/plugins/base.py), merged by `shopdb/core/api/dashboard.py` and `shopdb/core/api/reports.py`. |
|
||||
| Map placement | Data-driven off asset types + resolved positions (ADR-001); a typed, positioned asset appears on the map with its type color and no plugin-side map code. |
|
||||
|
||||
Capabilities a plugin **cannot have at all today** - no hook exists, and adding
|
||||
one would mean forking a core view:
|
||||
|
||||
3. **Asset-detail extension panels.** A warranty-coverage section on a PC or
|
||||
printer detail page, a calibration-status card on a measuring tool. Today
|
||||
`WarrantyPanel.vue` is composed in by hand-editing each detail view.
|
||||
4. **Map marker / overlay contributions.** A calibration-due badge on the
|
||||
shop-floor map (`frontend/src/views/map/MapView.vue`). The map draws type
|
||||
colors but has no plugin decoration path.
|
||||
5. **Search-result rendering / routing for plugin asset types.** Global search
|
||||
returns assets, but core hardcodes how each type renders and where its detail
|
||||
link points; a plugin asset type has no way to declare its icon or route.
|
||||
|
||||
There is also an unsolved distribution wrinkle from ADR-003: external plugins
|
||||
symlink into `plugins/<name>/` backend-only, but any frontend file they carry
|
||||
must be physically copied into `frontend/src/`, because Vite compiles the tree
|
||||
at build time and cannot reach outside it.
|
||||
|
||||
## Options considered
|
||||
|
||||
### A. Runtime dynamic component registration
|
||||
|
||||
Plugins ship real Vue components that core loads and mounts at runtime into
|
||||
named extension points (an `iconMap` registration, a panel registry, a marker
|
||||
renderer registry). This is the richest model and the one ADR-009's "Future
|
||||
direction" sketched.
|
||||
|
||||
- Pro: a plugin can render anything; no core generic-renderer ceiling.
|
||||
- Con: requires runtime module loading of plugin-authored code (dynamic import
|
||||
of built chunks), a versioned shared-component surface that becomes an
|
||||
accidental contract the moment it leaks, and it does nothing about the
|
||||
build-time-only reach of Vite for external plugins. It is the most code for
|
||||
the least near-term payoff. ADR-009 already priced this and deferred it.
|
||||
|
||||
### B. Declarative data-only hooks rendered by core generic components
|
||||
|
||||
Plugins return plain dicts from new `BasePlugin` hooks; a generic core component
|
||||
renders them. This is the exact precedent already proven three times:
|
||||
`get_navigation_items`, `get_dashboard_widgets`, and `get_reports` (0.6.0) all
|
||||
return dicts that a core consumer endpoint merges and a core Vue component
|
||||
renders. No plugin ships frontend code.
|
||||
|
||||
- Pro: additive, minor-bump changes under ADR-002; identical access pattern to
|
||||
the existing consumers (skip disabled, fail-loud in dev, isolate in prod);
|
||||
works untouched for external symlink-only plugins, because the data crosses
|
||||
the wire and core owns the renderer.
|
||||
- Con: bounded to what a generic renderer can draw. A panel that needs a bespoke
|
||||
chart or a custom-interaction map overlay does not fit.
|
||||
|
||||
### C. Build-time file-convention discovery
|
||||
|
||||
Extend the `import.meta.glob` precedent from `./routes/*.js` to a plugin-owned
|
||||
frontend tree (`plugins/*/frontend/`), so a plugin's real components (full
|
||||
list/detail/form views, its api-client module) live with the plugin and the
|
||||
build picks them up by convention. This is ADR-009 "Future direction" steps 1-2.
|
||||
|
||||
- Pro: true self-containment for genuine components; the natural home for the
|
||||
api-client block (friction 1) and full views.
|
||||
- Con: significant build-system work (per-plugin Vite entry discovery,
|
||||
code-splitting, dev-server HMR across the tree) and it does not by itself
|
||||
solve the external-plugin wrinkle - a symlinked out-of-tree `frontend/` is
|
||||
still outside Vite's compiled root. Only pays off once out-of-tree plugins
|
||||
with frontends are a real requirement.
|
||||
|
||||
### Hybrid
|
||||
|
||||
The evidence splits cleanly. The five friction points fall into two buckets:
|
||||
surfaces where a generic renderer fed by plugin data is sufficient (2, 3, 4, 5),
|
||||
and surfaces where a real component is genuinely unavoidable (full views, and
|
||||
the api-client module in friction 1). Option B fits the first bucket exactly and
|
||||
is cheap and additive. Option C is the right long-term answer for the second but
|
||||
is expensive and, per ADR-009, gated on external-plugin demand. Option A buys
|
||||
nothing B does not, at the highest cost. The decision is B now, C deferred, A
|
||||
not pursued.
|
||||
|
||||
## Decision
|
||||
|
||||
**DECISION:** adopt a hybrid. Add **data-only declarative hooks** (Option B) for
|
||||
the four presentation surfaces a generic core renderer can serve, and keep
|
||||
**file-convention glob discovery** (Option C) as the deferred mechanism for the
|
||||
residual cases where a real component is unavoidable. Do not pursue runtime
|
||||
component registration (Option A).
|
||||
|
||||
### New hooks (Option B, data-only, additive)
|
||||
|
||||
Each is a new optional `BasePlugin` method returning a list of dicts, merged by
|
||||
a core consumer endpoint using the same pattern as `get_reports` (inject
|
||||
`plugin` name, skip disabled, re-raise in dev/test, log-and-isolate in prod),
|
||||
and rendered by a generic core component. Icon values are string keys mapped to
|
||||
Lucide components core-side, exactly like `get_navigation_items`.
|
||||
|
||||
**`get_settings_cards`** - resolves friction point 2 (settingsNav.js hand edit).
|
||||
|
||||
```
|
||||
{
|
||||
'group': 'Measuring Tools', # rail group title (created if new)
|
||||
'to': '/settings/measuringtooltypes',
|
||||
'icon': 'ruler', # string key, mapped core-side
|
||||
'title': 'Measuring Tool Types',
|
||||
'description': 'Manage measuring-tool subtypes + map colors',
|
||||
'position': 22, # order within the group
|
||||
}
|
||||
```
|
||||
|
||||
Consumer: a new `GET /api/settings/cards` merges enabled plugins' cards into the
|
||||
core `settingsGroups` catalog; `SettingsLayout` and `SettingsIndex` read the
|
||||
merged catalog instead of the hardcoded JS array. The catalog's own core groups
|
||||
stay in `settingsNav.js`; plugin groups are appended.
|
||||
|
||||
**`get_asset_panels`** - resolves friction point 3 (asset-detail panels).
|
||||
|
||||
```
|
||||
{
|
||||
'id': 'calibration',
|
||||
'title': 'Calibration',
|
||||
'assettypes': ['measuring_tool'], # detail pages it appears on; ['*'] = all
|
||||
'endpoint': '/api/measuringtools/{assetid}/calibration-panel',
|
||||
'render': 'keyvalue', # 'keyvalue' | 'table' | 'badge'
|
||||
'position': 20,
|
||||
}
|
||||
```
|
||||
|
||||
Consumer: `GET /api/assets/{assetid}/panels` returns the panels whose
|
||||
`assettypes` match that asset's type; a generic `AssetPanel` component on the
|
||||
detail page fetches each `endpoint` and renders it in the declared style. This
|
||||
covers the warranty-coverage and calibration-status cases. A panel that needs
|
||||
bespoke UI (a chart) is out of scope for the data-only hook and falls to the
|
||||
deferred component mechanism below - stated honestly, not hidden.
|
||||
|
||||
**`get_map_overlays`** - resolves friction point 4 (map marker/overlay badges).
|
||||
|
||||
```
|
||||
{
|
||||
'id': 'calibration-due',
|
||||
'label': 'Calibration due', # legend label
|
||||
'endpoint': '/api/measuringtools/map-overlay', # -> [{assetid, color, label}]
|
||||
'style': 'badge', # 'badge' | 'ring'
|
||||
'legend': True,
|
||||
}
|
||||
```
|
||||
|
||||
Consumer: `MapView.vue` fetches enabled plugins' overlay endpoints and decorates
|
||||
the already-placed markers; legend entries append to the existing legend. The
|
||||
map stays data-driven; plugins add decoration data, not map code.
|
||||
|
||||
**`get_asset_presentation`** - resolves friction point 5 (search rendering /
|
||||
routing) and, as a bonus, removes the `AppLayout.vue` iconMap hand edit called
|
||||
out in PLUGIN-GUIDE.md section 9.
|
||||
|
||||
```
|
||||
{
|
||||
'assettype': 'measuring_tool', # AssetType.assettype key the plugin owns
|
||||
'icon': 'ruler',
|
||||
'label': 'Measuring Tool',
|
||||
'route': '/measuringtools/{assetid}', # detail-route pattern
|
||||
}
|
||||
```
|
||||
|
||||
Consumer: `GET /api/assets/presentation` returns the type-to-presentation map;
|
||||
global-search result rows and any asset cross-link use it to pick the icon and
|
||||
build the detail link, so core never hardcodes a plugin's route or icon.
|
||||
|
||||
### Friction map
|
||||
|
||||
| Friction point | Mechanism | Hook / change |
|
||||
|---|---|---|
|
||||
| 1. api-client block in api/index.js | C (deferred) | plugin-owned `frontend/` tree, glob-discovered; scaffolder snippet is the near-term mitigation |
|
||||
| 2. settingsNav.js card entry | B (now) | `get_settings_cards` |
|
||||
| 3. asset-detail panels | B (now) | `get_asset_panels` (bespoke panels -> C, deferred) |
|
||||
| 4. map markers / overlays | B (now) | `get_map_overlays` (bespoke overlays -> C, deferred) |
|
||||
| 5. search-result rendering / routing | B (now) | `get_asset_presentation` |
|
||||
| views (full list/detail/form) | C (deferred) | already glob-discovered under core `routes/*.js`; long-term move to `plugins/*/frontend/` per ADR-009 |
|
||||
|
||||
### Deferred: file-convention frontend tree (Option C)
|
||||
|
||||
Extending `import.meta.glob` from `./routes/*.js` to `plugins/*/frontend/` (so
|
||||
views and the api-client module live with the plugin) is ADR-009 "Future
|
||||
direction" steps 1-2. It is deferred for the same reasons ADR-009 gave: it is
|
||||
build-system-heavy and the payoff only lands once out-of-tree plugins with their
|
||||
own frontends are a real requirement. Until then, friction 1 stays mitigated by
|
||||
the scaffolder snippet, and full views keep shipping in the core bundle and
|
||||
gated by ADR-009 `meta.plugin`.
|
||||
|
||||
This is also the honest limit on the external-plugin wrinkle. The four data-only
|
||||
hooks need **zero** frontend files from a plugin, so an external symlink-only
|
||||
(backend) plugin gets settings cards, detail panels, map overlays, and search
|
||||
presentation with no copy-into-`frontend/src/` step at all. The wrinkle survives
|
||||
only for the residual component-backed cases, which is exactly the deferred
|
||||
Option C work; pushing external-repo frontend distribution to that later ADR
|
||||
matches ADR-003's posture of deferring out-of-tree packaging until two sites run
|
||||
their own plugins.
|
||||
|
||||
### Contract-version impact (ADR-002)
|
||||
|
||||
Each of the four hooks is a new optional `BasePlugin` method - an additive
|
||||
change, so a **minor** bump per ADR-002, the same classification `get_reports`
|
||||
took at 0.6.0. Landing all four together is a single minor bump (proposed
|
||||
**0.7.0**); landing them incrementally is one minor bump each. Adding the core
|
||||
consumer endpoints and the generic renderer components is core-internal and does
|
||||
not itself move `__contract_version__`. The deferred Option C introduces a
|
||||
separate, versioned **frontend** contract (ADR-009 step 4), tracked apart from
|
||||
the backend `__contract_version__`; it is not part of this proposal's bumps.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- Closes the CONTRACT-STABILITY.md "single biggest gap" for the four surfaces a
|
||||
generic renderer can serve, using the already-proven declarative pattern - low
|
||||
risk, low cost, additive-only.
|
||||
- External symlink-only plugins get four presentation surfaces with no
|
||||
copy-into-core step, shrinking (not yet eliminating) the ADR-003 wrinkle.
|
||||
- Removes three of the two-plus hand edits the exemplar needed (settingsNav
|
||||
card, plus the AppLayout iconMap edit from section 9), moving them to data.
|
||||
- Docs-drift guard forces documentation: `tests/test_docs_contract.py` fails if
|
||||
any new public `BasePlugin` hook is missing from `docs/PLUGIN-HOOKS.md`, so the
|
||||
hooks cannot ship undocumented.
|
||||
|
||||
### Negative
|
||||
|
||||
- Four more hooks and four more core consumer endpoints to maintain, each with
|
||||
the skip-disabled / fail-loud-in-dev / isolate-in-prod discipline.
|
||||
- The generic renderers set a ceiling: bespoke panels and overlays still have no
|
||||
home until Option C lands, so the contract is honest-but-partial, not total.
|
||||
- One extra request per surface on the pages that use it (a detail page fetches
|
||||
its panels; the map fetches overlays), on top of ADR-009's enabled-list fetch.
|
||||
|
||||
### Neutral
|
||||
|
||||
- Bundled plugins keep their current in-core views; migrating them to the hooks
|
||||
is opt-in and incremental (see Adoption plan), not a flag-day rewrite.
|
||||
- `meta.plugin` route gating (ADR-009) is unchanged and still gates the views;
|
||||
these hooks add presentation, not routing.
|
||||
- Contract tests (`tests/test_plugin_contract.py`) already assert every public
|
||||
hook is exercised; the new hooks slot into that harness.
|
||||
|
||||
## Adoption plan
|
||||
|
||||
Prove the contract on bundled plugins before declaring it settled for sister
|
||||
sites.
|
||||
|
||||
1. **`get_asset_panels` first, via `warranty`.** Warranty is asset-general and
|
||||
already composes `WarrantyPanel.vue` onto multiple detail pages by hand, with
|
||||
a clean per-asset endpoint behind it. Migrating it to a declarative
|
||||
`get_asset_panels` entry rendered by the generic `AssetPanel` is the
|
||||
lowest-risk proof and immediately removes hand edits from every detail view
|
||||
that shows warranty. This is the recommended first migration.
|
||||
2. **`get_settings_cards`, `get_map_overlays`, `get_asset_presentation` via
|
||||
`measuringtools`.** The guide exemplar already needs a settings card, a
|
||||
calibration-due map badge, and search routing for its `measuring_tool` type,
|
||||
so it exercises all three at once and its PLUGIN-GUIDE.md walkthrough becomes
|
||||
the reference for the new hooks.
|
||||
3. Only after both plugins run on the hooks in a real build: bump
|
||||
`__contract_version__` to 0.7.0, document the hooks in
|
||||
`docs/PLUGIN-HOOKS.md`, and mark this ADR ACCEPTED.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should `get_asset_panels` `render` styles stay a small closed set
|
||||
(`keyvalue` / `table` / `badge`), or grow? A closed set keeps the renderer
|
||||
generic; growth pressure is the signal that a case actually needs Option C.
|
||||
- Should the four consumer endpoints collapse into one bundled
|
||||
`GET /api/plugins/frontend-contributions` call to save round-trips, or stay
|
||||
separate per surface for cache locality? Defer until the per-surface request
|
||||
cost is measured.
|
||||
- When Option C lands, does the api-client module move under
|
||||
`plugins/*/frontend/` or get replaced entirely by a generated client from the
|
||||
backend blueprint? Out of scope here; belongs to the ADR-009 frontend-contract
|
||||
follow-up.
|
||||
|
||||
## References
|
||||
|
||||
- ADR-001 (asset model as the map/search data source)
|
||||
- ADR-002 (bump classification for the new hooks)
|
||||
- ADR-003 (external-plugin distribution posture)
|
||||
- ADR-009 (route gating; "Future direction" is the deferred Option C)
|
||||
- docs/PLUGIN-GUIDE.md sections 9-10 (the measuringtools frontend hand edits)
|
||||
- docs/CONTRACT-STABILITY.md (the expected-churn line this ADR answers)
|
||||
- shopdb/plugins/base.py (existing declarative hooks this pattern extends)
|
||||
- shopdb/core/api/dashboard.py, shopdb/core/api/reports.py (consumer precedent)
|
||||
</content>
|
||||
</invoke>
|
||||
67
ADR-011-machines-rename.md
Normal file
67
ADR-011-machines-rename.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-11
|
||||
- **Deciders:** cproudlock
|
||||
- **Relates to:** ADR-005 (equipment vs measuringtools scope), ADR-008 (plugin migration ownership)
|
||||
|
||||
## Context
|
||||
|
||||
ADR-005 split the old "equipment" concept: gage-lab instruments moved to the
|
||||
measuringtools plugin, leaving the equipment plugin holding exactly one thing -
|
||||
shop-floor machines (lathes, mills, CMMs, grinders, furnaces). The UI, the
|
||||
routes (`/machines`), the frontend view folder (`views/machines/`), and even
|
||||
the data vocabulary (asset numbers headed "MACHINE #") already said machines.
|
||||
Only the code identity still said equipment: the plugin name, the API prefix,
|
||||
the `equipment`/`equipmenttypes` tables, and the `assettype='equipment'` data
|
||||
value. Every future contributor would pay a small permanent translation tax.
|
||||
|
||||
A display-label-only swap was shipped first (2026-07-11) and immediately showed
|
||||
the dissonance: UI Machines over code equipment forever.
|
||||
|
||||
Separately, the legacy `machinetypes` table survived the Machine-model
|
||||
retirement for one real reason: it types the vendor **models catalog**
|
||||
(`models.machinetypeid`, 95 of 127 models) with fine-grained, cross-category
|
||||
types (Lathe, Switch, Laser Printer). Its name was a misnomer - it never typed
|
||||
machines specifically - and it squatted on the name the machines plugin's
|
||||
subtype table wants.
|
||||
|
||||
The economics favor doing this now and never again: the plugin contract is
|
||||
pre-1.0 (breaking changes are cheap per ADR-002), no sister site has adopted
|
||||
yet, there is no production instance, and the test suite plus CI give the
|
||||
strongest safety net the project has had.
|
||||
|
||||
## Decision
|
||||
|
||||
Rename the domain in one coordinated pass:
|
||||
|
||||
| Current | New | Rationale |
|
||||
|---|---|---|
|
||||
| `plugins/equipment/`, `/api/equipment`, `equipment.*` perms | `plugins/machines/`, `/api/machines`, `machines.*` | plugin identity matches the domain |
|
||||
| `equipment` table, `equipmentid` | `machines`, `machineid` | main entity |
|
||||
| `equipmenttypes`, `equipmenttypeid`, `equipmenttype` | `machinetypes`, `machinetypeid`, `machinetype` | subtypes take the freed name |
|
||||
| legacy `machinetypes`, `models.machinetypeid` | `modeltypes`, `models.modeltypeid` | role-accurate: it types the models catalog, losslessly (no collapse to the 4 asset types) |
|
||||
| `assettypes.assettype = 'equipment'` (data) | `'machine'` | the string compare sites flip with it |
|
||||
|
||||
Migration placement follows ADR-008: the modeltypes rename and the data flips
|
||||
land in the core chain (core tables, core data); the plugin table renames land
|
||||
in the machines plugin's own chain, idempotently guarded so fresh installs
|
||||
(whose core chain still creates the old names at `7c04`) rename on first
|
||||
`flask plugin upgrade-all`. The `alembic_version_equipment` version table is
|
||||
renamed to `alembic_version_machines` by the core migration so the plugin chain
|
||||
resumes seamlessly on existing installs. The plugin registry auto-migrates an
|
||||
`equipment` key to `machines` on load.
|
||||
|
||||
Historical migrations and accepted ADRs are not rewritten; old ids and ADR-005
|
||||
wording remain as history.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Upgrading instances run `flask db upgrade` then `flask plugin upgrade-all`,
|
||||
same as every deploy; both migrations are idempotent.
|
||||
- `/api/equipment` and `equipmentApi` cease to exist - a breaking change made
|
||||
while there is nobody to break. After contract 1.0 this class of rename
|
||||
would be a major-version event; this is the last cheap moment.
|
||||
- The "Machine Types" settings card becomes "Model Types" and finally says
|
||||
what it does; the machines plugin's subtype card owns the machine-types name.
|
||||
- Contributors stop translating between UI vocabulary and code vocabulary.
|
||||
123
ADR-012-geenforce-manifest-ownership.md
Normal file
123
ADR-012-geenforce-manifest-ownership.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# ADR-012: GE-Enforce manifest ownership in shopdb
|
||||
|
||||
- **Status:** ACCEPTED
|
||||
- **Date:** 2026-07-13
|
||||
- **Deciders:** cproudlock
|
||||
- **Relates to:** ADR-002 (plugin contract versioning), ADR-004 (per-site
|
||||
deployment), ADR-006 (collector contract), ADR-008 (per-plugin Alembic chains)
|
||||
|
||||
## Context
|
||||
|
||||
GE-Enforce is a desired-state enforcement system for shopfloor PCs: a PowerShell
|
||||
engine (`Install-FromManifest.ps1`) reads per-PC-type `manifest.json` files off
|
||||
an SMB share every logon and installs / self-heals what they declare. Authoring
|
||||
those manifests today means hand-editing JSON on a file share, and there is no
|
||||
central view of what each PC actually did.
|
||||
|
||||
We want shopdb to own the manifests as data (author, version, publish, roll
|
||||
back) and to observe fleet compliance, while NOT taking on the GE-Enforce engine
|
||||
itself (which is the GE-Enforce framework's, maintained separately) and NOT
|
||||
dictating any site's imaging path (per ADR-004, each site is single-tenant with
|
||||
its own provisioning - PXE at West Jefferson, OOBE provisioning packages at
|
||||
others).
|
||||
|
||||
The manifests are an enforcement PROGRAM, not an application inventory: entry
|
||||
`Type` is not an app/config discriminator and entry `Name` is a manifest label,
|
||||
not a Windows ARP DisplayName. Any design that treats them as an app catalog is
|
||||
wrong.
|
||||
|
||||
## Decision
|
||||
|
||||
Build a bundled `geenforce` plugin that owns the manifest as shopdb data, with a
|
||||
client kit and a deployment bootstrap. Specifically:
|
||||
|
||||
1. **Data model.** One wide `manifestentries` table with an `entrytype`
|
||||
discriminator and nullable per-type columns (not SQLAlchemy STI, not a JSON
|
||||
blob - the fleet is ~64 entries, so sparse columns are free and stay
|
||||
queryable). Scopes are `manifestscopes`, unique on `(scopename, phase)`;
|
||||
runtime is per-pctype scopes, preinstall is one flat scope. Multi-value gates
|
||||
(PCTypes / hostnames / machine numbers) and the nested InUseCheck are child
|
||||
tables. `sortorder` is the execution-order contract. RegValue is stored as a
|
||||
raw JSON literal so DWord-vs-string typing survives. Per-plugin Alembic chain
|
||||
(ADR-008).
|
||||
|
||||
2. **Published snapshots.** Editing touches a DRAFT only. Publish freezes the
|
||||
rendered JSON document into an immutable `manifestpublishedversions` row; the
|
||||
client is ALWAYS served the current published snapshot, never the draft;
|
||||
rollback flips `iscurrent` to an older version. Freezing the document (not
|
||||
row-mirroring) makes immutability structural.
|
||||
|
||||
3. **Behavioral-parity gate, not byte-identity.** A DB-free harness
|
||||
(`parity.py`) imports each real manifest and renders it back, then proves
|
||||
BEHAVIORAL equivalence (same ordered entries with identical detection /
|
||||
targeting, and the same entries fire across machine-profile fixtures) - never
|
||||
byte equality, which re-serialization would never satisfy. This gates any
|
||||
build that touches the model.
|
||||
|
||||
4. **Filter mirror; engine is the single source of truth.** `filters.py`
|
||||
mirrors the engine's four gate functions and alias graph for the "what would
|
||||
this PC get" simulator and parity. The engine lib stays authoritative;
|
||||
shopdb mirrors it (never the reverse). PCTypesStrict is honored only for the
|
||||
preinstall phase, matching the runners.
|
||||
|
||||
5. **Payload integrity is separate from detection.** For `http`/`inline`
|
||||
payloads a dedicated `payloadsha256` is verified before running - independent
|
||||
of `DetectionMethod` (DetectionValue is a hash only for `Hash` detection).
|
||||
`smb` payloads keep the share ACL as their trust boundary. Large binaries
|
||||
stay on SMB; small config/scripts may move to http/inline later.
|
||||
|
||||
6. **Observed-state reporting.** Each PC POSTs its enforcement result;
|
||||
`manifestenforcementreports` (+ results) records the applied version
|
||||
(received-latest) and per-entry self-heal / failure. Status derives from
|
||||
explicit self-heal flags only, never the raw installed count (Always/no-
|
||||
detection scripts install every cycle without being drift corrections).
|
||||
|
||||
7. **Service-token auth.** Client endpoints authorize via managed service
|
||||
tokens scoped `geenforce.fetch` / `geenforce.report`, through a new
|
||||
`service_token_authorized(scope)` on the `shopdb.api` contract surface
|
||||
(contract 0.11.0). Admin CRUD uses `geenforce.manage` / `geenforce.publish`.
|
||||
|
||||
8. **Client + deployment, engine referenced not vendored.** shopdb ships the
|
||||
fetch/report kit (`plugins/geenforce/client/`) and a site-neutral bootstrap
|
||||
(`Install-GEEnforce.ps1`) that provisions a PC's identity
|
||||
(`C:\Enrollment\pc-type.txt` etc. - what determines the PC type; there is no
|
||||
auto-detection, the provisioner supplies it), the shopdb registry config, and
|
||||
the scheduled task. The GE-Enforce ENGINE is referenced (`-EngineSource`),
|
||||
not carried by shopdb. Deployment is provisioning-path independent (PXE step,
|
||||
OOBE ppkg, Intune, manual); the runtime task is fail-safe.
|
||||
|
||||
9. **Milestone 1 = export to share; staged cutover.** Until a site cuts its
|
||||
client over to shopdb-sourced manifests, the plugin publishes and EXPORTS the
|
||||
manifest to the share (with a `_meta/history` backup, atomic write); the
|
||||
unchanged engine picks it up. Cutover is staged: shadow mode (fetch from
|
||||
shopdb AND read the share, log diffs, install from share) then read cutover.
|
||||
|
||||
10. **No application auto-seeding.** The core Applications catalog already
|
||||
tracks these apps (from the classic-shopdb migration) with version
|
||||
histories; auto-creating Applications from manifest labels produced
|
||||
duplicates and misclassified config drops. Application linkage, if wanted, is
|
||||
a curated manifest-entry -> existing-Application link, not label scraping.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive.** Manifests become validated, versioned, publishable data with
|
||||
one-click rollback and a fleet-compliance view; desired-state and observed-
|
||||
state live in one system. The parity gate + published snapshots + separate
|
||||
payload hash make a fleet-wide-SYSTEM system safe to author. The plugin is
|
||||
provisioning-agnostic, so any GE Aerospace site can adopt it regardless of
|
||||
imaging path. Validated end to end: parity green against the real manifests,
|
||||
and the client kit + installer proven on a Windows VM (PS 5.1) and Linux
|
||||
pwsh 7.
|
||||
- **Boundaries / risks.** The engine remains the GE-Enforce framework's, so
|
||||
shopdb's parity mirror must be kept in sync with the lib (guarded by the parity
|
||||
fixtures; the plugin pins lib >= 2.6 for `_CmmVersion`). Provisioning writes
|
||||
the PC identity - shopdb cannot set a PC's type at imaging (a PC is unknown
|
||||
until it enrolls and reports). Manifest-label vs ARP-name mismatch means the
|
||||
catalog link, when built, needs a curated alias layer.
|
||||
- **Deferred.** Desired-vs-observed per-entry compliance (needs a collector
|
||||
installedVersions field); curated manifest-entry -> Application linking; the
|
||||
live client cutover (a site operational decision); inline payload upload.
|
||||
|
||||
See `docs/proposals/ge-enforce-plugin.md` (design + cutover), `docs/GE-ENFORCE.md`
|
||||
(concepts + imaging timeline), `docs/GE-ENFORCE-CLIENT.md` (fetch/report
|
||||
contract), and `docs/GE-ENFORCE-DEPLOY.md` (agent deployment).
|
||||
125
BACKUP-RESTORE.md
Normal file
125
BACKUP-RESTORE.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Backup and Restore
|
||||
|
||||
Each site owns its own data (single-tenant, ADR-004), so backups are the site's
|
||||
responsibility. A complete backup is two parts:
|
||||
|
||||
1. **The MySQL database** - all asset, user, audit, and settings data.
|
||||
2. **The `instance/` directory** - uploaded floor plans, branding assets,
|
||||
`plugins.json` (the enabled-plugin list), and any tokens or files the app
|
||||
writes to disk. These are NOT in the database, so a DB-only backup loses
|
||||
them. Back up `instance/` alongside every database dump.
|
||||
|
||||
Restoring the database without the matching `instance/` directory leaves the
|
||||
app pointing at floor plans and logos that no longer exist.
|
||||
|
||||
## What to back up
|
||||
|
||||
| Item | Location | Why |
|
||||
|------|----------|-----|
|
||||
| Database | MySQL `shopdb_flask` | All application data. |
|
||||
| `instance/branding/` | repo `instance/` dir | Uploaded logos and favicon. |
|
||||
| `instance/modelimages/` | repo `instance/` dir | Uploaded vendor-model photos. |
|
||||
| `instance/employeephotos/` | repo `instance/` dir | Uploaded self-hosted employee photos (external mode serves photos from the HR database instead). |
|
||||
| `instance/` floor plans | repo `instance/` dir | Uploaded map blueprints. |
|
||||
| `instance/plugins.json` | repo `instance/` dir | Which plugins this site enabled. |
|
||||
| `.env` | repo root (offline, secured) | Secrets needed to bring the stack back up. Store separately from the data backup, in a secrets manager. |
|
||||
|
||||
## Backup
|
||||
|
||||
### Database (Docker)
|
||||
|
||||
```bash
|
||||
docker compose exec -T db mysqldump \
|
||||
-u root -p"${MYSQL_ROOT_PASSWORD}" \
|
||||
--single-transaction --routines --triggers \
|
||||
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
`--single-transaction` gives a consistent dump without locking the tables (InnoDB).
|
||||
|
||||
### Database (external MySQL, no container)
|
||||
|
||||
```bash
|
||||
mysqldump -h <host> -u <user> -p \
|
||||
--single-transaction --routines --triggers \
|
||||
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
### instance directory
|
||||
|
||||
```bash
|
||||
tar czf instance-$(date +%F).tar.gz instance/
|
||||
```
|
||||
|
||||
Recommended cadence: nightly database dump to offsite storage, 14-day
|
||||
retention; `instance/` captured on the same schedule (and always right before an
|
||||
upgrade). Verify a restore quarterly.
|
||||
|
||||
## Restore
|
||||
|
||||
Restoring replaces the current database contents. Do it into a known-empty or a
|
||||
throwaway target first if you are unsure.
|
||||
|
||||
### Step 1: Bring up the stack (or a fresh one)
|
||||
|
||||
```bash
|
||||
cp .env.example .env # or restore your saved .env
|
||||
# ensure MYSQL_* and DATABASE_URL match the dump's database name (shopdb_flask)
|
||||
docker compose up -d db
|
||||
```
|
||||
|
||||
Wait for the `db` container to report healthy (`docker compose ps`).
|
||||
|
||||
### Step 2: Load the database dump
|
||||
|
||||
```bash
|
||||
gunzip -c shopdb-2026-07-10.sql.gz | \
|
||||
docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask
|
||||
```
|
||||
|
||||
For an external MySQL:
|
||||
|
||||
```bash
|
||||
gunzip -c shopdb-2026-07-10.sql.gz | mysql -h <host> -u <user> -p shopdb_flask
|
||||
```
|
||||
|
||||
If the target database does not exist yet, create it as utf8mb4 first (matching
|
||||
the schema charset):
|
||||
|
||||
```sql
|
||||
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
```
|
||||
|
||||
### Step 3: Restore the instance directory
|
||||
|
||||
```bash
|
||||
tar xzf instance-2026-07-10.tar.gz # restores ./instance/
|
||||
```
|
||||
|
||||
The docker-compose api container reads `instance/` from the repo working
|
||||
directory; make sure it is present before starting `api`.
|
||||
|
||||
### Step 4: Bring up the API and reconcile migrations
|
||||
|
||||
```bash
|
||||
docker compose up -d api
|
||||
docker compose exec api flask db upgrade
|
||||
```
|
||||
|
||||
`flask db upgrade` is a safety net: if the dump predates the current code, this
|
||||
applies any newer migrations. If the dump is at the same version it is a no-op.
|
||||
|
||||
### Step 5: Verify
|
||||
|
||||
- Log in with a known account.
|
||||
- Confirm the floor map renders (branding and map blueprints resolve from
|
||||
`instance/`).
|
||||
- Spot-check a few asset records and the audit log.
|
||||
- `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .`
|
||||
should return a `VALIDATION_ERROR`, not a 500.
|
||||
|
||||
## See also
|
||||
|
||||
- [DEPLOY.md](DEPLOY) - first-time deploy
|
||||
- [UPGRADE.md](UPGRADE) - upgrade procedure (back up first)
|
||||
- [CONFIG.md](CONFIG) - environment variables and Setting keys
|
||||
662
COLLECTOR-INTEGRATION.md
Normal file
662
COLLECTOR-INTEGRATION.md
Normal file
@@ -0,0 +1,662 @@
|
||||
# Collector integration (PC auto-update)
|
||||
|
||||
How the shopfloor PC fleet pushes inventory into shopdb-flask, replacing the
|
||||
classic ASP `api.asp?action=updateCompleteAsset` path. The generic collector
|
||||
contract is defined in ADR-006; this doc is the operational reference for wiring
|
||||
a real caller (the GE-Enforce fleet agent) to it.
|
||||
|
||||
- Server code: `shopdb/core/api/collector.py`
|
||||
- Computers schema + upsert: `plugins/computers/plugin.py`
|
||||
(`get_collector_schema` / `apply_collector_payload`)
|
||||
- Contract rationale: `docs/adr/ADR-006-collector-contract.md`
|
||||
|
||||
---
|
||||
|
||||
## 1. How it works now
|
||||
|
||||
### Auth model (header-only, env keys, fail-closed)
|
||||
|
||||
- The API key travels in the `X-API-Key` request header. Nothing else is
|
||||
accepted. The old `?api_key=<key>` querystring fallback has been removed on
|
||||
every collector endpoint (see the breaking-change section below).
|
||||
- Keys come from environment variables (never the database, never a file the app
|
||||
serves):
|
||||
- `COLLECTOR_API_KEY_<PLUGINNAME>` - per-plugin override, uppercased plugin
|
||||
name (e.g. `COLLECTOR_API_KEY_COMPUTERS`).
|
||||
- `COLLECTOR_API_KEY` - shared fallback used when no per-plugin key is set.
|
||||
- Resolution order per request: per-plugin key first, then the shared key
|
||||
(`_plugin_api_key` in `collector.py`).
|
||||
- Fail-closed: if neither variable is set server-side, the endpoint returns
|
||||
HTTP 500 `Collector API key not configured` and rejects every request. An
|
||||
unconfigured server never silently accepts unauthenticated data.
|
||||
- A caller that sends the wrong key (or no key) gets HTTP 401 `Invalid API key`.
|
||||
- In addition to the env keys, a managed API token scoped to `collector.ingest`
|
||||
is accepted as a collector credential on every collector endpoint. See
|
||||
"Managed collector tokens" below; env keys remain the fallback.
|
||||
|
||||
### Managed collector tokens (recommended)
|
||||
|
||||
Alongside the env keys, every collector endpoint (`/api/collector/<plugin>`,
|
||||
`/pc`, `/apps`, `/heartbeat`, `/bulk`, `/status`) also accepts a **managed API
|
||||
token** (PAT) scoped to the `collector.ingest` permission. The env keys stay
|
||||
supported as a bootstrap/legacy fallback - nothing breaks - but a managed token
|
||||
is the preferred credential because it can be minted, rotated, and revoked from
|
||||
the UI (Settings > API Tokens) and its use shows up in `lastusedat` and the
|
||||
audit log.
|
||||
|
||||
What makes a token a collector service token: it is scoped to ONLY
|
||||
`collector.ingest`. That scope authorizes the collector ingest API and NOTHING
|
||||
else. The existing scoped-token machinery contains it automatically - a scoped
|
||||
token passes `require_permission` only for its listed permissions and is denied
|
||||
on every role-gated (`require_role`) endpoint and on import mode, and
|
||||
`collector.ingest` gates no normal route. So a collector token that leaks cannot
|
||||
be used to read or write anything through the regular API; it can only submit
|
||||
collector payloads.
|
||||
|
||||
Both wire transports are accepted (send whichever is convenient; GE-Enforce
|
||||
sends `X-API-Key` today, so that stays ergonomic):
|
||||
|
||||
```
|
||||
POST /api/collector/computers
|
||||
X-API-Key: shopdb_pat_<40 hex>
|
||||
```
|
||||
or
|
||||
```
|
||||
POST /api/collector/computers
|
||||
Authorization: Bearer shopdb_pat_<40 hex>
|
||||
```
|
||||
|
||||
An unscoped PAT, or a PAT scoped to some other permission, is NOT a collector
|
||||
token and is rejected (401) - only `collector.ingest` in the scope list counts.
|
||||
A revoked or expired token is rejected (401) on both transports.
|
||||
|
||||
#### How to mint one (admin flow)
|
||||
|
||||
The simplest contained flow: an **admin** mints the token, scoped to
|
||||
`collector.ingest`. Because the token is scoped, the admin-role bypass is
|
||||
suspended for it, so the token is contained to the collector API even though its
|
||||
owner is an admin - it cannot act with admin authority anywhere.
|
||||
|
||||
1. Settings > API Tokens > New Token.
|
||||
2. Click the **Collector service token** preset (pre-selects only
|
||||
`collector.ingest`), name it (e.g. `wj-fleet-collector`), optionally set an
|
||||
expiry, Create.
|
||||
3. Copy the `shopdb_pat_...` secret (shown once) and deploy it to the fleet the
|
||||
same way as the env key: the `collectorApiKey` field in per-site
|
||||
`site-config.json` (see "Delivering the API key to clients" below). The
|
||||
client sends it in `X-API-Key` exactly as it sends an env key today - no
|
||||
client code change.
|
||||
|
||||
Service identity (documented, not built): if you prefer a non-admin owner,
|
||||
create a dedicated low-privilege user (e.g. `svc-collector`) whose role holds
|
||||
only `collector.ingest`, plus `apitokens.create` if that user is to mint its own
|
||||
token. The scope ceiling then caps any token it mints at `collector.ingest`.
|
||||
The admin-minted route above is simpler and equally contained, so it is the
|
||||
recommended default.
|
||||
|
||||
#### Rotation
|
||||
|
||||
Managed tokens rotate without a fleet re-image:
|
||||
|
||||
1. Mint a new collector token (steps above).
|
||||
2. Deploy it via `site-config.json` (`collectorApiKey`) - update the one per-site
|
||||
value.
|
||||
3. Confirm the new token is in use: watch its `lastusedat` climb in Settings >
|
||||
API Tokens (and the old token's `lastusedat` go stale).
|
||||
4. Revoke the old token once traffic has moved. Revocation is immediate.
|
||||
|
||||
### Generic endpoint contract: `POST /api/collector/<plugin>`
|
||||
|
||||
One dynamic route serves every enabled plugin that returns a collector schema.
|
||||
For the PC fleet that is `POST /api/collector/computers`.
|
||||
|
||||
Request flow inside `generic_collect`:
|
||||
|
||||
1. Look up the plugin's schema. Unknown / disabled plugin -> HTTP 404
|
||||
`No collector registered for plugin <plugin>`.
|
||||
2. Resolve and check the API key (per-plugin then shared, as above).
|
||||
3. Parse the JSON body. Missing or non-object body -> HTTP 400 `No data provided`.
|
||||
4. Identity-field resolution: the schema names an `identityfield`
|
||||
(`hostname` for computers). If that field is missing or blank in the payload,
|
||||
HTTP 400 `<identityfield> is required`.
|
||||
5. Idempotent upsert: the plugin's `apply_collector_payload` finds the existing
|
||||
asset by identity and updates it, or inserts a new one. Same identity on a
|
||||
later submission updates the same row - re-imaging a PC does not duplicate it,
|
||||
and existing asset relationships are preserved.
|
||||
6. Audit logging: every accepted submission writes an `AuditLog` row
|
||||
(`action` = created/updated, entity `Collector`, `details = collector:<plugin>
|
||||
action=<action>`), then commits.
|
||||
|
||||
Response body (HTTP 200), wrapped in the standard envelope
|
||||
`{status, data, message, meta}`, with the collector result under `data`:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"status": "ok",
|
||||
"action": "created",
|
||||
"assetid": 12345,
|
||||
"identityvalue": "WJRP2335",
|
||||
"warnings": ["unknown operating system: Microsoft Windows 11 Enterprise 23H2 (build 22631)"]
|
||||
},
|
||||
"message": "computers collector created",
|
||||
"meta": { "timestamp": "...", "requestid": "..." }
|
||||
}
|
||||
```
|
||||
|
||||
`action` is `created`, `updated`, or `noop`. `warnings` is a list of soft
|
||||
problems (unmapped pc-type, unknown OS, unknown app, un-stored pcsubtype) that
|
||||
did NOT fail the request - the row was still written.
|
||||
|
||||
### Error responses
|
||||
|
||||
Errors use the same envelope with `status: "error"` and the detail under
|
||||
`data.error`:
|
||||
|
||||
```json
|
||||
{ "status": "error", "data": { "error": { "code": "UNAUTHORIZED", "message": "Invalid API key" } }, "meta": { } }
|
||||
```
|
||||
|
||||
The plugin can raise `ValueError` for controlled validation failures; its message
|
||||
is returned verbatim (HTTP 400). Any other exception is caught, logged
|
||||
server-side, the transaction is rolled back, and the caller gets a generic
|
||||
HTTP 500 `Internal error processing collector payload` with no internal detail
|
||||
(by design - check the server log to see what actually failed).
|
||||
|
||||
### Schema discovery: `GET /api/collector/_schemas`
|
||||
|
||||
Returns the collector schema for every enabled plugin. This route is
|
||||
JWT-protected (an interactive/admin token), NOT API-key protected - it is for
|
||||
humans and tooling introspecting what payloads are accepted, not for the
|
||||
headless collectors themselves.
|
||||
|
||||
### Legacy endpoints (computers-only, predate ADR-006)
|
||||
|
||||
These still exist for the older PowerShell callers and all require the same
|
||||
`X-API-Key` header:
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /api/collector/pc` | Update one PC matched by `hostname` (lastboot, user, serial). |
|
||||
| `POST /api/collector/apps` | Update installed apps for a PC (known apps only). |
|
||||
| `POST /api/collector/heartbeat` | Record check-in for one or many hostnames. |
|
||||
| `POST /api/collector/bulk` | Update many PCs in one call. |
|
||||
| `GET /api/collector/status` | Liveness + endpoint list. |
|
||||
|
||||
New integrations should target `POST /api/collector/computers`, not these. Per
|
||||
ADR-006 the legacy `/pc` path is deprecated and slated for removal before v1.0.
|
||||
|
||||
### Computers plugin field mapping
|
||||
|
||||
Payload naming follows the project convention (lowercase concatenated). The
|
||||
identity field is `hostname`. All other fields are optional; an omitted field
|
||||
leaves the existing value untouched (patch-style), so a bare report never blanks
|
||||
a column.
|
||||
|
||||
| Payload field | Type | Server behaviour (`apply_collector_payload`) |
|
||||
|---|---|---|
|
||||
| `hostname` (required) | string | Identity. Matches `Computer.hostname` (case-insensitive), then falls back to `Asset.assetnumber`. New asset created if no match. |
|
||||
| `machinenumber` | string | Business tag -> `Asset.assetnumber`. The placeholder `9999` and empty string are skipped; when skipped a new PC falls back to `assetnumber = hostname`. On an existing PC a real value updates `assetnumber`. |
|
||||
| `pctype` | string | `gea-shopfloor-*` imaging type -> `Computer.computertypeid` via the configurable `pctypemap` settings. Unmapped value -> warning, not error. |
|
||||
| `pcsubtype` | string | Accepted but not stored yet -> warning. |
|
||||
| `serialnumber` | string | `Asset.serialnumber`. |
|
||||
| `loggedinuser` | string | `Computer.loggedinuser`. (`currentuser` is also accepted as a legacy alias.) |
|
||||
| `lastboottime` | ISO-8601 datetime | `Computer.lastboottime`. Unparseable -> warning. |
|
||||
| `lastcheckin` | ISO-8601 datetime | Accepted (heartbeat semantics). `lastreporteddate` is set server-side on every call regardless. |
|
||||
| `ipaddress` | string | Primary `Communication` row (`isprimary=True`, type `IP`). Updated in place or created. |
|
||||
| `vendorname` | string | `Computer.vendorid`. Vendor row auto-created if missing (free vocab). |
|
||||
| `modelnumber` | string | `Computer.modelnumberid`, scoped to the vendor when known. Model row auto-created if missing. |
|
||||
| `osname` | string | `Computer.osid`. Controlled vocab: looked up in `operatingsystems`, NOT auto-created. Unknown value -> warning (row still written, `osid` left unset). |
|
||||
| `installedsoftware` | array of `{name, version}` | `ComputerInstalledApp` rows for applications shopdb already tracks. Unknown app name -> warning, skipped. |
|
||||
| `defaultprinter` | string | The default printer's identifier (windows name / share / hostname / port IP). Resolved to a printer asset and linked PC -> printer as a `defaultprinter` relationship. Unresolved -> warning. |
|
||||
| `printers` | array of strings | All installed network printer identifiers. Each resolves to a printer asset and is linked PC -> printer as a `connectedto` relationship (the default is skipped here since it already links as `defaultprinter`). Unresolved entries -> warning. |
|
||||
|
||||
Schema source of truth: `get_collector_schema` in `plugins/computers/plugin.py`.
|
||||
If you change the payload, change it there and re-check this table.
|
||||
|
||||
### PC -> printer relationship sync
|
||||
|
||||
When a payload carries `defaultprinter` and/or `printers`, the collector syncs
|
||||
`AssetRelationship` rows so a PC page shows its printers and a printer page shows
|
||||
the PCs that use it (both render in the shared Relationships card).
|
||||
|
||||
- Resolution: each identifier is matched, first hit wins, against the printer's
|
||||
`windowsname`, `hostname`, `sharename`, its asset number/name, then any active
|
||||
printer communications IP. Case-insensitive except the IP (exact). An
|
||||
identifier that resolves to nothing adds a warning and is skipped; it never
|
||||
fails the whole push.
|
||||
- Link types: the default printer links with `defaultprinter` (directional, PC
|
||||
is the source); every other reported printer links with `connectedto`
|
||||
(symmetric). A printer that is both default and in `printers` links only as
|
||||
the default.
|
||||
- Idempotent: re-reporting the same set creates no duplicate rows (an existing
|
||||
matching row is reactivated if it was archived, otherwise left as is).
|
||||
- Stale-link archive: on every push, collector-created links to printers no
|
||||
longer reported are set inactive. Collector-created rows are tagged in
|
||||
`assetrelationships.label = 'collector:printers'`; only tagged rows are ever
|
||||
archived, so links you create by hand in the UI are never touched. A payload
|
||||
that omits BOTH printer keys leaves all existing printer links untouched
|
||||
(report an empty `printers: []` to clear the auto links instead).
|
||||
- Response: the collector response carries `printerlinkcount` and a
|
||||
`printerlinks` list of `{assetid, relationshiptype}` for the links kept.
|
||||
|
||||
### pc-type mapping (configurable per site)
|
||||
|
||||
`pctype` (e.g. `gea-shopfloor-cmm`) maps to a shopdb Computer Type through
|
||||
`pctypemap_<pxetype>` settings (Settings > Collector PC Types).
|
||||
Defaults live in `plugins/computers/pctypemap.py` and are seeded on plugin
|
||||
install; edit per site in the UI. Unmapped pc-types produce a warning, not a
|
||||
failure.
|
||||
|
||||
### Classic api.asp field mapping (for porting the PowerShell reporter)
|
||||
|
||||
The fleet's classic-ASP reporter posts form fields to
|
||||
`api.asp?action=updateCompleteAsset`. Map them to the collector JSON as follows:
|
||||
|
||||
| Classic `updateCompleteAsset` form field | Collector field |
|
||||
|---|---|
|
||||
| `hostname` | `hostname` |
|
||||
| `machineNo` | `machinenumber` |
|
||||
| `pcType` | `pctype` |
|
||||
| `serialNumber` | `serialnumber` |
|
||||
| `loggedInUser` | `loggedinuser` |
|
||||
| `lastBootUpTime` / `lastBootTime` | `lastboottime` |
|
||||
| `manufacturer` | `vendorname` |
|
||||
| `model` | `modelnumber` |
|
||||
| `osVersion` | `osname` |
|
||||
| `installedApps` | `installedsoftware` |
|
||||
|
||||
Not carried over (no current home in the computers schema): warranty fields, DNC
|
||||
config, multi-NIC detail beyond the single primary IP, VNC/WinRM flags. The
|
||||
classic reporter posts a full `networkInterfaces` array; the collector accepts
|
||||
only one `ipaddress`, so pick the corp/routable NIC (see the corp-range gate in
|
||||
the PowerShell below).
|
||||
|
||||
---
|
||||
|
||||
## 2. Breaking change: querystring `api_key` removed
|
||||
|
||||
The API key must now be sent in the `X-API-Key` header. The `?api_key=<key>`
|
||||
querystring form has been removed on every collector endpoint
|
||||
(`/api/collector/<plugin>`, `/pc`, `/apps`, `/heartbeat`, `/bulk`, `/status`).
|
||||
Querystring keys leak into web-server access logs, proxy history, and browser
|
||||
history. Header-only keeps the secret out of those logs.
|
||||
|
||||
Before (no longer works - the key is ignored and the request is rejected 401):
|
||||
|
||||
```
|
||||
POST /api/collector/computers?api_key=SECRET
|
||||
Content-Type: application/json
|
||||
|
||||
{ "hostname": "WJRP2335", "machinenumber": "2335" }
|
||||
```
|
||||
|
||||
After (correct):
|
||||
|
||||
```
|
||||
POST /api/collector/computers
|
||||
X-API-Key: SECRET
|
||||
Content-Type: application/json
|
||||
|
||||
{ "hostname": "WJRP2335", "machinenumber": "2335" }
|
||||
```
|
||||
|
||||
PowerShell before/after:
|
||||
|
||||
```powershell
|
||||
# BEFORE (broken)
|
||||
Invoke-RestMethod -Uri "https://$SiteHost/api/collector/computers?api_key=$key" `
|
||||
-Method Post -Body $json -ContentType 'application/json'
|
||||
|
||||
# AFTER (correct)
|
||||
Invoke-RestMethod -Uri "https://$SiteHost/api/collector/computers" `
|
||||
-Method Post -Body $json -ContentType 'application/json' `
|
||||
-Headers @{ 'X-API-Key' = $key }
|
||||
```
|
||||
|
||||
Any caller still putting `api_key` in the URL must move it to the header.
|
||||
|
||||
---
|
||||
|
||||
## 3. GE-Enforce implementation guide
|
||||
|
||||
GE-Enforce is the SYSTEM-context fleet agent that runs every enforcement cycle on
|
||||
each shopfloor PC (scheduled task, at-logon + periodic). It lives OUTSIDE this
|
||||
repo. The facts below are drawn from the real scripts so the payload matches what
|
||||
the PC already knows:
|
||||
|
||||
- `playbook/shopfloor-setup/common/GE-Enforce.ps1` - the enforcement pass. At the
|
||||
end of each run it writes a status JSON to
|
||||
`<share>\_outputs\logs\<hostname>\status.json` (interim transport; the fleet
|
||||
dashboard reads those files). It logs via `Write-EnforceLog` to
|
||||
`C:\Logs\Shopfloor\enforce-YYYYMMDD.log`. Its machine-number resolution is:
|
||||
registry `HKLM:\SOFTWARE\WOW6432Node\GE Aircraft Engines\Dnc\General` then
|
||||
`HKLM:\SOFTWARE\GE Aircraft Engines\Dnc\General`, property `MachineNo`,
|
||||
skipping the `9999` placeholder, then `C:\Enrollment\machine-number.txt` as
|
||||
fallback. hostname is `[System.Environment]::MachineName` (live NetBIOS name,
|
||||
not `$env:COMPUTERNAME`, which goes stale after a post-image rename).
|
||||
- `playbook/shopfloor-setup/gea-shopfloor-collections/Report-AssetToShopDB.ps1` -
|
||||
the existing HTTP reporter that POSTs to the classic ASP
|
||||
`api.asp?action=updateCompleteAsset`. It runs every cycle as a `Type=PS1`
|
||||
manifest entry (DetectionMethod `Always`) under the SYSTEM task, logs to
|
||||
`C:\Logs\Shopfloor\report-asset-YYYYMMDD.log`, always exits 0, and reads the
|
||||
same machine number (reg, then `cmm\cmmid.txt`, then `machine-number.txt`) plus
|
||||
BIOS serial, OS caption, make/model, logged-in user, and corp-NIC IP from WMI.
|
||||
The function below is the direct-to-flask analogue of this reporter.
|
||||
- `playbook/shopfloor-setup/Shopfloor/lib/Update-MachineNumber.ps1` - keeps the
|
||||
reg `MachineNo` and `C:\Enrollment\machine-number.txt` in sync when a tech
|
||||
reassigns a bay, so both sources agree.
|
||||
|
||||
### Paste-ready reporter
|
||||
|
||||
Drop this in as a `Type=PS1` / DetectionMethod `Always` manifest entry (mirroring
|
||||
`Report-AssetToShopDB.ps1`), or dot-source `Send-ShopdbCollectorReport` from
|
||||
GE-Enforce.ps1 and call it at the end of the pass. It reads the machine number
|
||||
exactly the way GE-Enforce already does, builds a payload matching the computers
|
||||
collector schema, and POSTs with the `X-API-Key` header over TLS 1.2. Every
|
||||
field name below was checked against `get_collector_schema` in
|
||||
`plugins/computers/plugin.py`.
|
||||
|
||||
The `X-API-Key` value can be EITHER a `COLLECTOR_API_KEY[_COMPUTERS]` env key OR
|
||||
a managed token scoped to `collector.ingest` (a `shopdb_pat_...` secret; see
|
||||
"Managed collector tokens"). The script is identical for both - it just carries
|
||||
whatever `collectorApiKey` the site-config supplies - so switching a site from an
|
||||
env key to a managed token (and rotating it) is a config change, not a script
|
||||
change.
|
||||
|
||||
```powershell
|
||||
# Send-ShopdbCollectorReport.ps1
|
||||
# Reports this PC's identity to shopdb-flask via POST /api/collector/computers.
|
||||
# Runs as SYSTEM from the GE-Enforce cycle. Always exits without throwing;
|
||||
# failures are logged, never fatal.
|
||||
|
||||
function Get-ShopdbCollectorApiKey {
|
||||
# NEVER hardcode the key in this script (it lives on the share). Read it
|
||||
# from a per-site value the SYSTEM/machine account can already reach.
|
||||
# Preferred: a field in the site-config.json GE-Enforce already parses.
|
||||
# Fallback: a machine-local file staged at imaging, ACL'd to SYSTEM.
|
||||
$key = ''
|
||||
$siteCfg = 'C:\Enrollment\site-config.json'
|
||||
if (Test-Path -LiteralPath $siteCfg) {
|
||||
try {
|
||||
$cfg = Get-Content -LiteralPath $siteCfg -Raw | ConvertFrom-Json
|
||||
if ($cfg.collectorApiKey) { $key = "$($cfg.collectorApiKey)".Trim() }
|
||||
} catch {}
|
||||
}
|
||||
if (-not $key) {
|
||||
$keyFile = 'C:\Enrollment\collector-api-key.txt'
|
||||
if (Test-Path -LiteralPath $keyFile) {
|
||||
try { $key = (Get-Content -LiteralPath $keyFile -First 1 -ErrorAction Stop).Trim() } catch {}
|
||||
}
|
||||
}
|
||||
return $key
|
||||
}
|
||||
|
||||
function Get-ShopdbMachineNumber {
|
||||
# Same resolution GE-Enforce.ps1 / Report-AssetToShopDB.ps1 use:
|
||||
# eDNC registry (skip 9999 placeholder) then C:\Enrollment\machine-number.txt.
|
||||
$machineNumber = ''
|
||||
foreach ($rp in @(
|
||||
'HKLM:\SOFTWARE\WOW6432Node\GE Aircraft Engines\Dnc\General',
|
||||
'HKLM:\SOFTWARE\GE Aircraft Engines\Dnc\General'
|
||||
)) {
|
||||
if ($machineNumber) { break }
|
||||
if (Test-Path $rp) {
|
||||
try {
|
||||
$v = (Get-ItemProperty -Path $rp -Name MachineNo -ErrorAction Stop).MachineNo
|
||||
if ($v -and "$v".Trim() -ne '9999') { $machineNumber = "$v".Trim() }
|
||||
} catch {}
|
||||
}
|
||||
}
|
||||
if (-not $machineNumber -and (Test-Path 'C:\Enrollment\machine-number.txt')) {
|
||||
try {
|
||||
$v = (Get-Content 'C:\Enrollment\machine-number.txt' -First 1 -ErrorAction Stop).Trim()
|
||||
if ($v -and $v -ne '9999') { $machineNumber = $v }
|
||||
} catch {}
|
||||
}
|
||||
return $machineNumber
|
||||
}
|
||||
|
||||
function Get-ShopdbCorpIPv4 {
|
||||
# Pick the corp/AESFMA NIC IP. Same allowed-range gate as
|
||||
# Report-AssetToShopDB.ps1 - update the ranges if the site re-VLANs.
|
||||
$allowedRanges = @(
|
||||
@{ Network = '10.134.48.0'; PrefixLen = 23 },
|
||||
@{ Network = '10.48.249.0'; PrefixLen = 26 }
|
||||
)
|
||||
function ConvertTo-Uint32([string]$ip) {
|
||||
$bytes = ([System.Net.IPAddress]::Parse($ip)).GetAddressBytes()
|
||||
[Array]::Reverse($bytes)
|
||||
return [BitConverter]::ToUInt32($bytes, 0)
|
||||
}
|
||||
try {
|
||||
$ips = Get-NetIPAddress -AddressFamily IPv4 -ErrorAction Stop |
|
||||
Where-Object { $_.IPAddress -notmatch '^169\.254' -and $_.IPAddress -ne '127.0.0.1' }
|
||||
foreach ($ipo in $ips) {
|
||||
$ipInt = ConvertTo-Uint32 $ipo.IPAddress
|
||||
foreach ($r in $allowedRanges) {
|
||||
$netInt = ConvertTo-Uint32 $r.Network
|
||||
$mask = [uint32]([math]::Pow(2, 32) - [math]::Pow(2, 32 - $r.PrefixLen))
|
||||
if (($ipInt -band $mask) -eq ($netInt -band $mask)) { return $ipo.IPAddress }
|
||||
}
|
||||
}
|
||||
} catch {}
|
||||
return ''
|
||||
}
|
||||
|
||||
function Send-ShopdbCollectorReport {
|
||||
param(
|
||||
[string]$SiteHost = 'tsgwp00525.wjs.geaerospace.net',
|
||||
[string]$ApiKey = (Get-ShopdbCollectorApiKey),
|
||||
[int]$TimeoutSec = 30,
|
||||
[string]$LogFile = ('C:\Logs\Shopfloor\collector-{0}.log' -f (Get-Date -Format 'yyyyMMdd'))
|
||||
)
|
||||
|
||||
$logDir = Split-Path -Parent $LogFile
|
||||
if (-not (Test-Path $logDir)) {
|
||||
New-Item -ItemType Directory -Path $logDir -Force -ErrorAction SilentlyContinue | Out-Null
|
||||
}
|
||||
function Write-CollectorLog([string]$msg) {
|
||||
$ts = Get-Date -Format 'yyyy-MM-dd HH:mm:ss'
|
||||
"$ts $msg" | Tee-Object -FilePath $LogFile -Append | Out-Null
|
||||
}
|
||||
|
||||
Write-CollectorLog '=== Report to shopdb-flask collector ==='
|
||||
|
||||
if (-not $ApiKey) {
|
||||
Write-CollectorLog 'ERROR no collector API key (site-config.json / collector-api-key.txt) - skipping.'
|
||||
return
|
||||
}
|
||||
|
||||
# PowerShell 5.1 does not always negotiate TLS 1.2 by default.
|
||||
try { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 } catch {}
|
||||
|
||||
# --- Gather identity (matches the computers collector schema) ---
|
||||
$hostname = [System.Environment]::MachineName
|
||||
if (-not $hostname) { $hostname = $env:COMPUTERNAME }
|
||||
|
||||
$machineNumber = Get-ShopdbMachineNumber
|
||||
$ipAddress = Get-ShopdbCorpIPv4
|
||||
|
||||
$serialNumber = ''
|
||||
try {
|
||||
$serialNumber = "$((Get-CimInstance -ClassName Win32_BIOS -ErrorAction Stop).SerialNumber)".Trim()
|
||||
} catch { Write-CollectorLog "WARN BIOS serial read failed: $($_.Exception.Message)" }
|
||||
|
||||
$vendorName = ''; $modelNumber = ''; $loggedInUser = ''
|
||||
try {
|
||||
$cs = Get-CimInstance -ClassName Win32_ComputerSystem -ErrorAction Stop
|
||||
$vendorName = "$($cs.Manufacturer)".Trim()
|
||||
$modelNumber = "$($cs.Model)".Trim()
|
||||
# UserName is COMPUTERNAME\user or DOMAIN\user; keep the bare username.
|
||||
if ($cs.UserName) { $loggedInUser = ($cs.UserName -split '\\')[-1].Trim() }
|
||||
} catch { Write-CollectorLog "WARN ComputerSystem read failed: $($_.Exception.Message)" }
|
||||
|
||||
$osName = ''; $lastBootTime = ''
|
||||
try {
|
||||
$os = Get-CimInstance -ClassName Win32_OperatingSystem -ErrorAction Stop
|
||||
$osName = "$($os.Caption)".Trim()
|
||||
$dv = ''
|
||||
try { $dv = (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' -Name DisplayVersion -ErrorAction Stop).DisplayVersion } catch {}
|
||||
if ($dv) { $osName += " $dv" }
|
||||
if ($os.BuildNumber) { $osName += " (build $($os.BuildNumber))" }
|
||||
$osName = $osName.Trim()
|
||||
# ISO-8601 so the server's datetime parse accepts it.
|
||||
try { $lastBootTime = $os.LastBootUpTime.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ') } catch {}
|
||||
} catch { Write-CollectorLog "WARN OS read failed: $($_.Exception.Message)" }
|
||||
|
||||
$pcType = ''
|
||||
if (Test-Path -LiteralPath 'C:\Enrollment\pc-type.txt') {
|
||||
try { $pcType = (Get-Content -LiteralPath 'C:\Enrollment\pc-type.txt' -First 1 -ErrorAction Stop).Trim() } catch {}
|
||||
}
|
||||
$pcSubType = ''
|
||||
if (Test-Path -LiteralPath 'C:\Enrollment\pc-subtype.txt') {
|
||||
try { $pcSubType = (Get-Content -LiteralPath 'C:\Enrollment\pc-subtype.txt' -First 1 -ErrorAction Stop).Trim() } catch {}
|
||||
}
|
||||
|
||||
# --- Installed printers (Win32_Printer). The Default flag marks the one
|
||||
# default printer. We report each printer's port name (an IP or a queue
|
||||
# host for network printers) and fall back to the share/printer name, which
|
||||
# the collector resolves flexibly against printer windowsname/hostname/IP. ---
|
||||
$defaultPrinter = ''
|
||||
$printerIds = @()
|
||||
try {
|
||||
$printers = Get-CimInstance -ClassName Win32_Printer -ErrorAction Stop
|
||||
foreach ($p in $printers) {
|
||||
if ($p.Local) { continue } # skip local-only (XPS/PDF/OneNote)
|
||||
# Prefer the port name (IP or queue host); fall back to ShareName,
|
||||
# then the printer Name.
|
||||
$identity = $p.PortName
|
||||
if (-not $identity) { $identity = $p.ShareName }
|
||||
if (-not $identity) { $identity = $p.Name }
|
||||
if (-not $identity) { continue }
|
||||
$printerIds += $identity
|
||||
if ($p.Default) { $defaultPrinter = $identity }
|
||||
}
|
||||
$printerIds = @($printerIds | Select-Object -Unique)
|
||||
} catch { Write-CollectorLog "WARN printer read failed: $($_.Exception.Message)" }
|
||||
|
||||
# --- Build payload. Field names MUST match get_collector_schema exactly. ---
|
||||
$payload = @{ hostname = $hostname }
|
||||
if ($defaultPrinter) { $payload['defaultprinter'] = $defaultPrinter }
|
||||
if ($printerIds.Count) { $payload['printers'] = $printerIds }
|
||||
if ($machineNumber) { $payload['machinenumber'] = $machineNumber }
|
||||
if ($pcType) { $payload['pctype'] = $pcType }
|
||||
if ($pcSubType) { $payload['pcsubtype'] = $pcSubType }
|
||||
if ($serialNumber) { $payload['serialnumber'] = $serialNumber }
|
||||
if ($loggedInUser) { $payload['loggedinuser'] = $loggedInUser }
|
||||
if ($lastBootTime) { $payload['lastboottime'] = $lastBootTime }
|
||||
if ($ipAddress) { $payload['ipaddress'] = $ipAddress }
|
||||
if ($vendorName) { $payload['vendorname'] = $vendorName }
|
||||
if ($modelNumber) { $payload['modelnumber'] = $modelNumber }
|
||||
if ($osName) { $payload['osname'] = $osName }
|
||||
$payload['lastcheckin'] = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
|
||||
# installedsoftware is optional; add an array of @{ name=..; version=.. }
|
||||
# here if the manifest introspection is wired to real Application names.
|
||||
|
||||
$uri = "https://$SiteHost/api/collector/computers"
|
||||
$json = $payload | ConvertTo-Json -Depth 5
|
||||
$headers = @{ 'X-API-Key' = $ApiKey }
|
||||
|
||||
Write-CollectorLog ("POST {0} host={1} machineNo={2} pcType={3} serial={4} ip={5}" -f `
|
||||
$uri, $hostname, $machineNumber, $pcType, $serialNumber, $ipAddress)
|
||||
|
||||
try {
|
||||
$resp = Invoke-RestMethod -Uri $uri -Method Post -Body $json `
|
||||
-ContentType 'application/json' -Headers $headers `
|
||||
-TimeoutSec $TimeoutSec -ErrorAction Stop
|
||||
$d = $resp.data
|
||||
Write-CollectorLog ("OK action={0} assetid={1} identity={2} warnings={3}" -f `
|
||||
$d.action, $d.assetid, $d.identityvalue, ((@($d.warnings)) -join '; '))
|
||||
} catch {
|
||||
$code = $null
|
||||
try { $code = [int]$_.Exception.Response.StatusCode.value__ } catch {}
|
||||
Write-CollectorLog "ERROR POST failed (http=$code): $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
|
||||
# Entry point when run as a standalone manifest PS1 entry:
|
||||
Send-ShopdbCollectorReport
|
||||
```
|
||||
|
||||
### Where the call slots into the GE-Enforce run
|
||||
|
||||
Put the POST at the END of the enforcement pass, after the enforcement scopes are
|
||||
processed and the PC identity is settled. Two placements, pick one:
|
||||
|
||||
- Preferred - a sibling manifest entry, mirroring `Report-AssetToShopDB.ps1`:
|
||||
add `Send-ShopdbCollectorReport.ps1` as a `Type=PS1`, DetectionMethod `Always`
|
||||
manifest entry in `common\manifest.json`. It runs every cycle under the SYSTEM
|
||||
task, keeps GE-Enforce.ps1 untouched, and gets its own log file. This matches
|
||||
the established pattern the classic reporter already uses.
|
||||
- Alternative - inline: dot-source the function and call
|
||||
`Send-ShopdbCollectorReport` inside GE-Enforce.ps1's status-write-back `try`
|
||||
block (right after `status.json` is written, near the end of the pass). Use
|
||||
this only if you want the HTTP push to share GE-Enforce's own log and lifecycle.
|
||||
|
||||
Either way, run it ALONGSIDE the existing `status.json` share write (and
|
||||
alongside the classic `Report-AssetToShopDB.ps1`) during rollout - do not remove
|
||||
the share write until the HTTP path is proven.
|
||||
|
||||
### Delivering the API key to clients
|
||||
|
||||
Do NOT hardcode the key in the script - the script itself lives on the SFLD
|
||||
share, so a literal key there is effectively published to everyone with share
|
||||
read. Two viable options:
|
||||
|
||||
1. Per-site value the machine account already reads. GE-Enforce already parses
|
||||
`C:\Enrollment\site-config.json` (its local copy of the per-site config on the
|
||||
share). Add one field, `collectorApiKey`, populated per site. The script reads
|
||||
it with the machine/SYSTEM identity it already runs as. Rotation = update the
|
||||
one config value at the site, no re-image, no script redeploy.
|
||||
2. Baked at imaging into a machine-local file (`C:\Enrollment\collector-api-key.txt`)
|
||||
or an HKLM value, ACL'd to SYSTEM/Administrators, written by the enrollment
|
||||
pipeline. Tighter blast radius (the key never sits on the share at all), but
|
||||
rotation requires touching every PC or re-imaging.
|
||||
|
||||
Recommendation: use option 1 (`collectorApiKey` in the per-site config), matching
|
||||
how GE-Enforce already sources its share paths and mirroring how SFLD credentials
|
||||
are provisioned per site. It keeps the secret out of source control and off the
|
||||
script, sits at the same trust boundary as everything else the SYSTEM agent
|
||||
already reads, and supports rotation without a fleet touch. `Get-ShopdbCollectorApiKey`
|
||||
above already prefers this and falls back to the imaging-baked file, so a site
|
||||
can start on option 1 and tighten to option 2 later with no code change. The
|
||||
server side pairs this with `COLLECTOR_API_KEY_COMPUTERS` (per-plugin) so the PC
|
||||
fleet's key is scoped to the computers collector only.
|
||||
|
||||
### Staged rollout
|
||||
|
||||
1. Keep everything as-is: `status.json` share write + classic
|
||||
`Report-AssetToShopDB.ps1` continue running. Provision `collectorApiKey` per
|
||||
site and set `COLLECTOR_API_KEY_COMPUTERS` on the shopdb-flask server.
|
||||
2. Add `Send-ShopdbCollectorReport` as a new manifest entry on a handful of pilot
|
||||
bays (one per pc-type: collections, cmm, keyence, waxtrace, ...). It logs to
|
||||
`C:\Logs\Shopfloor\collector-YYYYMMDD.log` and cannot break enforcement (it
|
||||
never throws, exits cleanly).
|
||||
3. Verify on the server: pilot PCs appear/update in shopdb-flask, `action` is
|
||||
`created`/`updated`, warnings are understood (map any unmapped `pctype`, add
|
||||
any missing OS strings to the `operatingsystems` vocab). Cross-check the
|
||||
collector log's `OK` lines against the audit log.
|
||||
4. Roll the manifest entry out fleet-wide (it deploys from `common\`, so every
|
||||
pc-type gets it). Continue running both the share write and the HTTP POST.
|
||||
5. Once shopdb-flask is the system of record and stable across a full imaging
|
||||
cycle, retire the classic `Report-AssetToShopDB.ps1` and, if desired, the
|
||||
`status.json` share write.
|
||||
|
||||
---
|
||||
|
||||
## 4. Troubleshooting
|
||||
|
||||
| Symptom | HTTP | Cause / fix |
|
||||
|---|---|---|
|
||||
| `Invalid API key` | 401 | The `X-API-Key` header is missing or wrong. Confirm the client key matches `COLLECTOR_API_KEY_COMPUTERS` (or the shared `COLLECTOR_API_KEY`) on the server. Check the key is being read (`Get-ShopdbCollectorApiKey` returned non-empty). Remember the querystring form no longer works. |
|
||||
| `Collector API key not configured` | 500 | Fail-closed: neither `COLLECTOR_API_KEY_COMPUTERS` nor `COLLECTOR_API_KEY` is set in the server environment. Set one and restart the app. This is a SERVER config gap, not a client problem. |
|
||||
| `No collector registered for plugin computers` | 404 | The computers plugin is disabled or not loaded on that instance. Enable it. |
|
||||
| `hostname is required` / `No data provided` | 400 | Empty body, non-JSON body, or missing identity field. Check `-ContentType 'application/json'` and that `hostname` is set. |
|
||||
| `Internal error processing collector payload` | 500 | Generic by design - the server does not leak the cause to the caller. The real error (DB, unexpected exception) is in the server log (`Collector upsert failed for computers`). Check there. |
|
||||
| `warnings` present but `action` is created/updated | 200 | Soft issues only; the row WAS written. Common: unmapped `pctype` (fix the pctypemap setting), unknown `osname` (add it to the `operatingsystems` vocab), unknown app name, `pcsubtype` not stored. No action needed unless the warning matters to you. |
|
||||
|
||||
Client-side log for the fleet reporter: `C:\Logs\Shopfloor\collector-YYYYMMDD.log`.
|
||||
Server-side: the Flask app log (the collector logs upsert failures and per-plugin
|
||||
schema failures there).
|
||||
339
CONFIG.md
339
CONFIG.md
@@ -0,0 +1,339 @@
|
||||
# Configuration Reference
|
||||
|
||||
shopdb-flask reads configuration from two places, and the split is deliberate:
|
||||
|
||||
- **Environment variables** (`.env` / container env) hold **secrets and
|
||||
deploy-time wiring**: database credentials, signing keys, CORS origins, ports,
|
||||
API keys. These are read once at boot by `shopdb/config.py`. Never put a
|
||||
secret in the Settings table.
|
||||
- **The Settings table** (seeded by `flask seed settings`, edited in the UI
|
||||
under Settings or the setup wizard) holds **site preferences**: branding,
|
||||
ServiceNow links, floor-map images, search toggles, facility identity. These
|
||||
can change at runtime without a restart and are per-instance.
|
||||
|
||||
Rule of thumb: if leaking it would be a security incident, it is an environment
|
||||
variable. If it is a site preference an admin should be able to change in the
|
||||
UI, it is a Setting.
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Environment variables
|
||||
|
||||
Defined in `shopdb/config.py`. Copy `.env.example` to `.env` and fill in
|
||||
values. In `production` (`FLASK_ENV=production`), `ProductionConfig.validate()`
|
||||
refuses to boot if `SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, or
|
||||
`CORS_ORIGINS` are missing or set to the dev defaults.
|
||||
|
||||
### Flask core
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `FLASK_APP` | No | `wsgi.py` | Entry point for the `flask` CLI. |
|
||||
| `FLASK_ENV` | Yes | `development` | `production` for live sites (triggers `validate()`). Other values: `development`, `testing`. |
|
||||
| `SECRET_KEY` | Yes (prod) | dev default | Flask session/signing key. Generate: `python -c "import secrets; print(secrets.token_urlsafe(64))"`. |
|
||||
| `JWT_SECRET_KEY` | Yes (prod) | dev default | JWT signing key. Different value from `SECRET_KEY`. |
|
||||
| `JWT_ACCESS_TOKEN_EXPIRES` | No | `3600` | Access-token TTL in seconds. |
|
||||
| `JWT_REFRESH_TOKEN_EXPIRES` | No | `2592000` | Refresh-token TTL in seconds (30 days). |
|
||||
| `CORS_ORIGINS` | Yes (prod) | `http://localhost:5173` | Comma-separated explicit origins. Wildcard `*` is rejected in production. |
|
||||
| `LOG_LEVEL` | No | `INFO` | Logging verbosity. |
|
||||
|
||||
### Database
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `DATABASE_URL` | Yes (prod) | dev localhost URL | `mysql+pymysql://<user>:<pass>@<host>:<port>/<db>?charset=utf8mb4`. Keep `?charset=utf8mb4`. |
|
||||
|
||||
### Authentication rate limiting
|
||||
|
||||
IP-based fixed-window limit on the login endpoint, defense-in-depth atop the
|
||||
per-account lockout. Uses the existing cache extension (per-process, so the
|
||||
limit is approximate across multiple gunicorn workers).
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `AUTH_RATELIMIT_ENABLED` | No | `True` | Set `False` to disable (TestingConfig disables it). |
|
||||
| `AUTH_RATELIMIT_MAX` | No | `30` | Max login attempts per source IP per window before 429. |
|
||||
| `AUTH_RATELIMIT_WINDOW_SECONDS` | No | `300` | Window length in seconds. |
|
||||
|
||||
### Collector ingest (ADR-006)
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `COLLECTOR_API_KEY` | No | (empty) | Shared key for `/api/collector/*`. Endpoint fails closed (denies) when unset and no managed token is presented. Sent as the `X-API-Key` header. |
|
||||
| `COLLECTOR_API_KEY_<PLUGIN>` | No | (empty) | Per-plugin override, e.g. `COLLECTOR_API_KEY_COMPUTERS`. Checked before the shared key. |
|
||||
|
||||
The collector endpoints ALSO accept a managed API token (PAT) scoped to the
|
||||
`collector.ingest` permission, sent in `X-API-Key` or as an
|
||||
`Authorization: Bearer` token. Env keys stay supported as a bootstrap/legacy
|
||||
fallback; a managed token is preferred because it is minted, rotated, and
|
||||
revoked from Settings > API Tokens with `lastusedat` visibility. A
|
||||
collector-scoped token is contained to the collector API and nothing else. See
|
||||
`docs/COLLECTOR-INTEGRATION.md` (Managed collector tokens).
|
||||
|
||||
### Zabbix (printer supply monitoring)
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `ZABBIX_ENABLED` | No | `false` | Enable the Zabbix integration. |
|
||||
| `ZABBIX_URL` | No | (empty) | Zabbix API URL. |
|
||||
| `ZABBIX_TOKEN` | No | (empty) | Zabbix API bearer token. |
|
||||
|
||||
Note: Zabbix can also be configured via the Settings table (`zabbix_enabled`,
|
||||
`zabbix_url`, `zabbix_token`). The environment values are the boot-time wiring;
|
||||
prefer the Settings entries for runtime changes.
|
||||
|
||||
### Employee directory database (optional, read-only)
|
||||
|
||||
Separate HR/employee lookup DB consumed by the notifications plugin and the
|
||||
public kiosks. There is no safe default for the password; an unset password
|
||||
fails loud rather than trying a guessed credential.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `EMPLOYEE_DB_HOST` | No | `localhost` | HR DB host. |
|
||||
| `EMPLOYEE_DB_USER` | No | (empty) | HR DB user. |
|
||||
| `EMPLOYEE_DB_PASSWORD` | No | (empty) | HR DB password. No safe default. |
|
||||
| `EMPLOYEE_DB_NAME` | No | `wjf_employees` | HR DB name. |
|
||||
|
||||
Only used when `employee_directory_mode` (Setting) is `external`.
|
||||
|
||||
### CMMC USB database (optional, read-write)
|
||||
|
||||
Separate MySQL DB used by the USB plugin for check-in/out, lockers, and the log.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `CMMC_USB_DB_HOST` | No | `localhost` | USB DB host. |
|
||||
| `CMMC_USB_DB_USER` | No | (empty) | USB DB user. |
|
||||
| `CMMC_USB_DB_PASSWORD` | No | (empty) | USB DB password. No safe default. |
|
||||
| `CMMC_USB_DB_NAME` | No | `cmmc_usb` | USB DB name. |
|
||||
|
||||
Only used when `usb_directory_mode` (Setting) is `external`.
|
||||
|
||||
### docker-compose only
|
||||
|
||||
Read by `docker-compose.yml`, not by the Flask app directly.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `MYSQL_ROOT_PASSWORD` | Yes | (none) | Root password for the bundled MySQL container. |
|
||||
| `MYSQL_PASSWORD` | Yes | (none) | App-user password; must match the `DATABASE_URL` password. |
|
||||
| `MYSQL_PORT` | No | `3306` | Host port for MySQL. Bound to `127.0.0.1` only. |
|
||||
| `API_PORT` | No | `5001` | Host port for the API container. |
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Settings table keys
|
||||
|
||||
Seeded by `flask seed settings` (idempotent; re-running adds anything missing).
|
||||
Edited in the UI under Settings, or captured in the first-run setup wizard.
|
||||
Values are stored as strings and typed by `valuetype`. Secrets in this table
|
||||
(anything whose key contains `password`, `token`, or `secret`) are masked when
|
||||
read back through the API.
|
||||
|
||||
### site
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `setup_complete` | `false` | Set true once the first-run wizard finishes; gates the `/setup` route. |
|
||||
| `employee_directory_mode` | `selfhosted` | `selfhosted` (tables in this app) or `external` (a separate HR database, see `EMPLOYEE_DB_*`). |
|
||||
| `usb_directory_mode` | `selfhosted` | `selfhosted` or `external` (a separate `cmmc_usb` database, see `CMMC_USB_DB_*`). |
|
||||
| `site_base_url` | (empty) | Public base URL (scheme + host) for QR codes and absolute links. Blank = use the browsing origin. |
|
||||
| `facility_name` | (empty) | Facility name in the dashboard header. Blank = frontend falls back to `ShopDB`. |
|
||||
| `pc_access_domain` | `device.geaerospace.net` | Domain appended to a PC hostname for remote-access links. Blank = hostname as-is. |
|
||||
| `employeeid_pattern` | `^\d{9}$` | Regex a search term must match to be treated as an employee id. Invalid regex falls back to the default and never 500s. |
|
||||
| `printer_hostname_template` | `Printer-{ip}.printer.geaerospace.net` | Printer hostname template. `{ip}` is the dash-separated IP address. |
|
||||
| `contact_email_domain` | `geaerospace.com` | Email domain appended to a support contact's SSO to build email (`sso@domain`) and Teams-chat links. Blank hides the contact action buttons. |
|
||||
| `dualpath_single_machine` | `true` | Treat a Dualpath pair (a dual-bay machine with one controller) as a single machine in the machines list, dashboard/report counts, and the floor map (the secondary bay is hidden). The data model always keeps both bay records; detail pages stay per-bay with a sibling banner. `false` lists and counts both bays separately. |
|
||||
|
||||
### branding
|
||||
|
||||
Blank values fall back to the shipped GE default asset so an un-reconfigured
|
||||
install still renders. Upload replacements at Settings > Branding, which saves
|
||||
them under `instance/branding/`.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `site_logo` | `/ge-aerospace-logo.svg` | Header and login-page logo. |
|
||||
| `qr_logo` | `/ge-monogram.svg` | Logo composited in printer QR labels. Blank = no overlay. |
|
||||
| `badge_logo` | `/ge-aerospace-logo.svg` | Logo on the machine badge print page. |
|
||||
| `site_favicon` | (empty) | Browser tab favicon. Blank = shipped `/favicon.svg`. |
|
||||
| `brand_primary_color` | (empty) | Primary brand color as a CSS color value (maps to `--primary`). Blank = built-in theme color. |
|
||||
| `brand_primary_dark_color` | (empty) | Primary hover/active color (maps to `--primary-dark`). Blank = auto-derived by darkening the primary color ~15%. |
|
||||
| `brand_accent_color` | (empty) | Accent color for secondary buttons and badges (maps to `--secondary`). Blank = built-in theme color. |
|
||||
| `brand_sidebar_color` | (empty) | Sidebar background color (maps to `--sidebar-bg`). Blank = built-in theme color. |
|
||||
|
||||
### printing
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `qr_target_printer` | (empty) | Custom URL template for printer QR labels. Blank = link to the printer page on this instance. Placeholders: `{printerid}`, `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{ip}`, `{hostname}`. |
|
||||
| `qr_target_usb` | (empty) | Custom URL template for USB label QR codes. Blank = link to the USB device page. Placeholders: `{id}`, `{serialnumber}`, `{alias}`. |
|
||||
| `usb_label_style` | `barcode` | USB mini-label code style: `barcode` (CODE128 of the serial) or `qr` (QR code linking to the USB QR target). |
|
||||
| `qr_target_machine` | (empty) | Custom URL template for machine labels. Blank = link to the machine page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
|
||||
| `qr_target_computer` | (empty) | Custom URL template for computer labels. Blank = link to the computer page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
|
||||
| `qr_target_network_device` | (empty) | Custom URL template for network-device labels. Blank = link to the device page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
|
||||
| `qr_target_measuring_tool` | (empty) | Custom URL template for measuring-tool labels. Blank = link to the tool page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`, `{locationcode}`, `{locationname}`. |
|
||||
| `label_default_style` | `card` | Default asset-label layout used when a label first opens: `card` (badge with image and identity) or `plain` (just the code and a caption). |
|
||||
| `label_default_codetype` | `qr` | Default asset-label code type used when a label first opens: `qr` (QR code) or `barcode` (CODE128). |
|
||||
| `label_default_encodes_machine` | `assetnumber` | What a machine label encodes by default. |
|
||||
| `label_default_encodes_computer` | `assetpage` | What a computer label encodes by default. |
|
||||
| `label_default_encodes_printer` | `assetpage` | What a printer label encodes by default. |
|
||||
| `label_default_encodes_network_device` | `assetpage` | What a network-device label encodes by default. |
|
||||
| `label_default_encodes_measuring_tool` | `location` | What a measuring-tool label encodes by default. Values across these five: `assetpage`, `assetnumber`, `serialnumber`, `location` (measuring tools only), or `custom`. Overridable on the label page. |
|
||||
|
||||
The shared asset-label generator lives at `/print/asset-label/<assettype>/<id>` (public, like the other `/print/*` pages; `assettype` is one of `machine`, `computer`, `printer`, `network_device`, `measuring_tool`, and `id` is the asset's plugin id). It can encode the asset page link, the asset number, the serial number, a custom `qr_target_<type>` template, or - for measuring tools by default - the asset's inspection location code (the leading token of the location name, e.g. `0615`). A measuring tool with no location falls back to its asset page.
|
||||
|
||||
The batch generator at `/print/asset-label-batch/<assettype>` (reached from the "Print Labels" button on each asset list page) lays a multi-selection of one type onto ULINE label sheets: a 6-up 3 in x 3 in format or a dense 72-up mini-label format, with a start-cell offset for reusing partial sheets. It reuses the same code-type and `label_default_encodes_<type>` defaults as the single label.
|
||||
|
||||
### map
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `map_blueprint_light` | `/static/images/floorplan-placeholder.svg` | Floor-map blueprint (light theme). Re-upload your own in Settings > Floor Map. |
|
||||
| `map_blueprint_dark` | `/static/images/floorplan-placeholder.svg` | Floor-map blueprint (dark theme). |
|
||||
| `map_width` | `3300` | Blueprint native width in pixels. |
|
||||
| `map_height` | `2550` | Blueprint native height in pixels. |
|
||||
|
||||
### integrations
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `servicenow_enabled` | `true` | Enable ServiceNow ticket recognition and links. Disabled = tickets render as plain text. |
|
||||
| `servicenow_search_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. |
|
||||
| `servicenow_ticket_prefixes` | `GEINC,GECHG,GERIT,GESCT` | Comma-separated prefixes recognized as ServiceNow tickets. |
|
||||
| `servicenow_incident_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct incident URL if your instance has one. |
|
||||
| `servicenow_change_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct change URL if your instance has one. |
|
||||
| `zabbix_enabled` | `false` | Enable Zabbix for printer supply monitoring. |
|
||||
| `zabbix_url` | (empty) | Zabbix API URL. |
|
||||
| `zabbix_token` | (empty) | Zabbix API token (masked). |
|
||||
| `warranty_dell_enabled` | `false` | Enable Dell warranty (service-tag) lookups. |
|
||||
| `warranty_dell_clientid` | (empty) | Dell TechDirect API client id. |
|
||||
| `warranty_dell_clientsecret` | (empty) | Dell TechDirect API client secret (masked). |
|
||||
| `warranty_dell_tokenurl` | (empty) | Dell OAuth token URL. Blank = Dell default. |
|
||||
| `warranty_dell_apiurl` | (empty) | Dell warranty API URL. Blank = Dell default. |
|
||||
|
||||
### email
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `smtp_enabled` | `false` | Enable email notifications and alerts. |
|
||||
| `smtp_host` | (empty) | SMTP server hostname. |
|
||||
| `smtp_port` | `587` | SMTP port (587 TLS, 465 SSL, 25 plain). |
|
||||
| `smtp_username` | (empty) | SMTP auth username. |
|
||||
| `smtp_password` | (empty) | SMTP auth password (masked). |
|
||||
| `smtp_use_tls` | `true` | Use TLS for the SMTP connection. |
|
||||
| `smtp_from_address` | (empty) | From address for outgoing email. |
|
||||
| `smtp_from_name` | `ShopDB` | From name for outgoing email. |
|
||||
| `alert_recipients` | (empty) | Default alert/report recipients (comma-separated). |
|
||||
|
||||
#### Email flows and delivery model
|
||||
|
||||
The mail service (`shopdb/utils/mailer.py`, stdlib `smtplib`/`ssl`/`email`
|
||||
only) reads the keys above settings-first via the cached settings map, with an
|
||||
environment-variable fallback (`SMTP_HOST`, `SMTP_PORT`, `SMTP_USERNAME`,
|
||||
`SMTP_PASSWORD`, `SMTP_USE_TLS`, `SMTP_FROM_ADDRESS`, `SMTP_FROM_NAME`,
|
||||
`SMTP_ALERT_RECIPIENTS`, `SMTP_ENABLED`) applied only when any `SMTP_*` env var
|
||||
is present. When `smtp_enabled` is false or `smtp_host` is blank, every send is
|
||||
a graceful no-op that logs a warning and returns without error, so an
|
||||
unconfigured site never crashes. The SMTP password is never logged.
|
||||
|
||||
Three flows use it:
|
||||
|
||||
- Welcome email. When an admin creates a user (POST `/api/users`), the account
|
||||
is flagged `mustchangepassword` and a best-effort welcome email is sent with
|
||||
the facility name (`facility_name`), the username, the temporary password,
|
||||
and the sign-in link (`site_base_url` + `/login`). Mail is best-effort: the
|
||||
user is created even if the send fails (the response carries a `warning`). On
|
||||
first login the API returns `mustchangepassword: true`; the frontend forces
|
||||
the user through `/change-password` (POST `/api/auth/change-password`) before
|
||||
the app. Changing the password clears the flag and resets lockout counters.
|
||||
Set `sendwelcome: false` or `mustchangepassword: false` in the create body to
|
||||
opt out.
|
||||
|
||||
- Test email. POST `/api/settings/test-email` (settings.edit) sends a probe to
|
||||
the supplied `to` (or `alert_recipients`). The Email / SMTP settings page
|
||||
"Send Test Email" button calls it and shows the result; a real SMTP error is
|
||||
surfaced with the password scrubbed out.
|
||||
|
||||
- Alerts and report delivery (on-demand). POST `/api/reports/email`
|
||||
(reports.export) takes `{subject, columns, rows, intro?, to?}` and mails the
|
||||
rows as an HTML table. Recipients default to `alert_recipients` when `to` is
|
||||
omitted, so the same endpoint serves both report delivery and alerts. Report
|
||||
pages (Warranty, Toner) carry an "Email report" button that posts the rows
|
||||
they already loaded.
|
||||
|
||||
There is NO scheduler in this app: sending is on-demand. To automate a
|
||||
recurring send (e.g. a nightly warranty digest), point an external cron job
|
||||
at `/api/reports/email` using an API token (PAT) scoped to `reports.export`.
|
||||
See `docs/IMPORT-API.md` for the token model.
|
||||
|
||||
### audit
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `audit_retention_days` | `90` | Days to retain audit logs (0 = keep forever). |
|
||||
|
||||
### auth
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `saml_enabled` | `false` | Enable SAML SSO. |
|
||||
| `saml_idp_metadata_url` | (empty) | SAML IdP metadata URL. |
|
||||
| `saml_entity_id` | (empty) | SAML SP entity id. |
|
||||
| `saml_acs_url` | (empty) | SAML Assertion Consumer Service URL. |
|
||||
| `saml_allow_local_login` | `true` | Allow local username/password login when SAML is on. |
|
||||
| `saml_auto_create_users` | `true` | Auto-create users on first SAML login. |
|
||||
| `saml_admin_group` | (empty) | SAML group name that grants the admin role. |
|
||||
|
||||
**Personal API tokens.** Besides login JWTs and SAML, a user may create
|
||||
personal API tokens (PATs) for scripts and integrations, from Settings > API
|
||||
Tokens (or `POST /api/apitokens`). A PAT is sent like a JWT
|
||||
(`Authorization: Bearer shopdb_pat_...`), authenticates as its owning user
|
||||
across the whole API, and does not carry the hourly `JWT_ACCESS_TOKEN_EXPIRES`
|
||||
limit (it never expires unless an explicit expiry is set). Only the sha256 hash
|
||||
is stored; the secret is shown once at creation. This is the recommended
|
||||
credential for long-running imports (see `docs/IMPORT-API.md`). There is no env
|
||||
var to configure; PATs are managed entirely through the API/UI.
|
||||
|
||||
Creating or managing a PAT requires the `apitokens.create` permission (admins
|
||||
hold it by default; grant it to other roles from Settings > Users & Roles). By
|
||||
default a PAT is unscoped and acts with the full authority of its owner. A PAT
|
||||
may optionally carry a scopes list (a subset of the owner's permissions, capped
|
||||
at what the owner actually holds): a scoped token grants ONLY those permissions,
|
||||
intersected with the owner's live permissions at use time, and suspends the
|
||||
admin bypass, so it is denied on role-gated (admin-only) endpoints and on import
|
||||
mode. Use an unscoped token for admin-only work and imports.
|
||||
|
||||
### identifiers (dynamic)
|
||||
|
||||
One boolean key per asset identifier per asset type, keyed
|
||||
`identifier_<name>_<assettype>_enabled` (default `true`). Admins choose which
|
||||
optional identifiers show on which asset types. See ADR-001. The exact set is
|
||||
generated from `IDENTIFIER_LABELS` x `IDENTIFIER_ASSETTYPES` in
|
||||
`shopdb/core/api/settings.py`.
|
||||
|
||||
### search (dynamic)
|
||||
|
||||
One boolean key per search domain, keyed `search_<type>_enabled` (default
|
||||
`true`). Toggles whether a domain appears in global search results. The set is
|
||||
generated from `SEARCH_DOMAINS` in `shopdb/core/api/settings.py`.
|
||||
|
||||
## Custom fields
|
||||
|
||||
Site-defined extra attributes per asset type (Settings > Custom Fields, table
|
||||
`customfields`). Each field has a `searchable` flag (default off). When on, the
|
||||
field's stored values are matched by global search and a hit routes to the
|
||||
owning asset's detail page. The asset's `search_<type>_enabled` domain toggle
|
||||
still applies, so a custom-field hit on a computer only shows when the computer
|
||||
search domain is enabled. Inactive or non-searchable fields are never matched.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [DEPLOY.md](DEPLOY) - per-site deployment runbook
|
||||
- [UPGRADE.md](UPGRADE) - upgrading an existing site
|
||||
- [BACKUP-RESTORE.md](BACKUP-RESTORE) - backup and restore
|
||||
- `shopdb/config.py` - authoritative env-var definitions
|
||||
- `shopdb/core/api/settings.py` (`build_default_settings`) - authoritative Setting defaults
|
||||
|
||||
117
CONTRACT-STABILITY.md
Normal file
117
CONTRACT-STABILITY.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# Path to contract 1.0
|
||||
|
||||
This page is an honest read of how stable the plugin contract is today, for a
|
||||
sister site deciding how much to build on it. It is derived from the ADRs and
|
||||
the live code, not aspiration. The authoritative hook reference is
|
||||
[PLUGIN-HOOKS.md](PLUGIN-HOOKS); the versioning rules are in
|
||||
[ADR-002](ADR-002-plugin-versioning).
|
||||
|
||||
## Current version
|
||||
|
||||
The plugin contract is at **0.10.0**, declared in `shopdb/__init__.py` as
|
||||
`__contract_version__`. It is pre-1.0, which under semver means any 0.x minor
|
||||
bump is allowed to break the contract, and this project has used that latitude.
|
||||
|
||||
The product release version (`__version__`, currently 0.5.0) is a separate
|
||||
series with its own bump rules; see [ADR-007](ADR-007-product-versioning-and-releases).
|
||||
Do not pin against it for compatibility - pin against `__contract_version__`.
|
||||
|
||||
### 0.x history
|
||||
|
||||
Recorded in the comment block in `shopdb/__init__.py`:
|
||||
|
||||
| Version | Change | Kind |
|
||||
|---------|--------|------|
|
||||
| 0.3.0 | `shopdb.api` expanded to the full plugin import surface (db, cache, model bases, core models, response + pagination helpers, `employee_connection`) so plugins stop importing internal core paths | additive (minor) |
|
||||
| 0.4.0 | Removed the never-implemented `get_searchable_fields` hook (search is a core concern over the asset model) and wired `get_dashboard_widgets` to a real consumer (`/api/dashboard/widgets`) | pre-1.0 contract reduction |
|
||||
| 0.6.0 | Added the `get_reports` hook, consumed by `GET /api/reports` to merge plugin report cards into the Reports hub | additive optional hook (minor) |
|
||||
| 0.7.0 | Added the four ADR-010 frontend-contribution hooks (`get_settings_cards`, `get_asset_panels`, `get_map_overlays`, `get_asset_presentation`), consumed by the `GET /api/pluginui/*` endpoints | additive optional hooks (minor) |
|
||||
| 0.9.0 | Exposed the dualpath pair-resolution helpers on `shopdb.api` for the machines plugin | additive surface (minor) |
|
||||
| 0.10.0 | Added the `get_permissions` hook so plugins declare their own RBAC permissions; the catalog is resolved dynamically from core + enabled plugins | additive optional hook (minor) |
|
||||
|
||||
The source comment block documents 0.3.0, 0.4.0, 0.6.0, 0.7.0, 0.9.0, and 0.10.0. Earlier points
|
||||
(0.1.x / 0.2.x) predate that recorded rationale; `PluginMeta`'s fallback
|
||||
`core_version` default of `>=0.2.0,<1.0.0` is the only remaining trace of the
|
||||
0.2 baseline.
|
||||
|
||||
## Settled surface
|
||||
|
||||
These are unlikely to break before 1.0. A plugin can depend on them with
|
||||
reasonable confidence; a breaking change to any would be a major bump and would
|
||||
land with a new or amended ADR.
|
||||
|
||||
| Surface | What it is |
|
||||
|---------|-----------|
|
||||
| `meta` / manifest schema | `PluginMeta` fields and `manifest.json` (ADR-002 single source of truth) |
|
||||
| `get_blueprint` | Flask Blueprint registration at the manifest `api_prefix` |
|
||||
| `get_models` | SQLAlchemy model classes the plugin owns |
|
||||
| `init_app` | Custom init after blueprint + models are known |
|
||||
| `get_cli_commands` | Click commands added to the Flask CLI |
|
||||
| `get_services` | Named service classes, consumed by `plugin_manager.get_service` |
|
||||
| Lifecycle hooks | `on_install`, `on_uninstall`, `on_enable`, `on_disable` |
|
||||
| `get_navigation_items` | Sidebar menu entries |
|
||||
| `get_dashboard_widgets` | Dashboard widgets, consumed by `/api/dashboard/widgets` |
|
||||
| `get_reports` | Report cards, consumed by `/api/reports` (added 0.6.0) |
|
||||
| `get_permissions` | Plugin RBAC permissions, merged into the catalog for roles and token scopes (added 0.10.0) |
|
||||
| Frontend-contribution hooks | `get_settings_cards`, `get_asset_panels`, `get_map_overlays`, `get_asset_presentation`, consumed by `/api/pluginui/*` (added 0.7.0, [ADR-010](ADR-010-frontend-plugin-hooks)) |
|
||||
| Collector pair | `get_collector_schema` + `apply_collector_payload` per [ADR-006](ADR-006-collector-contract) |
|
||||
| Settings helpers | `get_setting` / `set_setting`, namespaced to the plugin |
|
||||
| `get_provisioning_note` | Setup-wizard transparency note for extra tables |
|
||||
| `get_config_schema` | Setup-wizard config field declarations |
|
||||
| `shopdb.api` import surface | The only core module plugins may import (plus `shopdb.plugins.base`); adding a name is minor, removing one is major |
|
||||
|
||||
The asset model itself - `Asset`, `AssetType`, `AssetStatus`,
|
||||
`AssetRelationship`, and the shared reference models - is the platform contract
|
||||
locked in [ADR-001](ADR-001-asset-as-platform-contract).
|
||||
|
||||
## Expected churn before 1.0
|
||||
|
||||
Known-unstable areas. Building on these means expecting rework.
|
||||
|
||||
| Area | Status | Reference |
|
||||
|------|--------|-----------|
|
||||
| Frontend renderers (residual) | The four data-only hooks and their `/api/pluginui/*` consumers are settled (0.7.0). The generic core renderers are landing incrementally: the settings-cards rail/landing renderer ships with 0.7.0; the asset-panel, map-overlay, and search-presentation renderers are wired opt-in per the ADR adoption plan. Real component-backed panels (bespoke charts, custom overlays) remain deferred to Option C (build-time glob discovery). | [ADR-010](ADR-010-frontend-plugin-hooks) (ACCEPTED) |
|
||||
| Per-plugin migrations | Brand new. The per-plugin Alembic engine exists and every bundled plugin now carries a chain, but the pattern has one release of production mileage, not years. | [ADR-008](ADR-008-plugin-migration-ownership) (2026-07-10) |
|
||||
| Pip distribution | Deferred to v2. External plugins install by clone / submodule / symlink; there is no entry-point discovery and no automatic update path yet. | [ADR-003](ADR-003-plugin-distribution) |
|
||||
|
||||
## Bump rules
|
||||
|
||||
From [ADR-002](ADR-002-plugin-versioning), applied to `__contract_version__`:
|
||||
|
||||
| Bump | Trigger |
|
||||
|------|---------|
|
||||
| major | Breaking change to the `BasePlugin` ABC, the `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`). Removing a name from `shopdb.api` is major. |
|
||||
| minor | Additive change: a new optional hook, a new field on a contract model with a default, a new name added to `shopdb.api`. |
|
||||
| patch | Bug fix with no change to the contract surface. |
|
||||
|
||||
### Deprecation policy
|
||||
|
||||
ADR-002 defines the bump classification above but is **silent** on any
|
||||
deprecation window or notice period for pre-1.0 removals. In practice a removal
|
||||
is simply a major (or, pre-1.0, a breaking minor) bump: the hook or name is
|
||||
gone, and the loader fails loud in dev or excludes the mismatched plugin in prod
|
||||
(see the load-time table in [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO)).
|
||||
Because the ADR says nothing about a grace period, this document proposes none;
|
||||
the honest guidance for sister sites is the mitigation that already exists - pin
|
||||
a tight `core_version` range and re-test before widening it.
|
||||
|
||||
## Criteria for declaring 1.0
|
||||
|
||||
Provisional. This is the maintainer's working list, not a committed checklist,
|
||||
and it will move. ADR-002's own open question ("When does the framework declare
|
||||
1.0.0?") ties 1.0 to the `Machine` retirement from ADR-001 and the framework
|
||||
being "ready for sister sites"; the items below expand that intent.
|
||||
|
||||
- Frontend hook contract defined via its own ADR (asset-detail panels, map
|
||||
markers, search results), closing the biggest churn item above.
|
||||
- `Machine` / legacy-model retirement complete per ADR-001, so the asset model
|
||||
is the only contract.
|
||||
- At least two external plugins running in production at a second site - the
|
||||
bar ADR-003 already sets before pip distribution and sister-site readiness
|
||||
are considered justified.
|
||||
- The per-plugin migration pattern (ADR-008) proven across real upgrades, not
|
||||
just fresh installs.
|
||||
- No contract bump needed for N consecutive product releases (N to be fixed
|
||||
when the list is firmed up), showing the surface has actually settled.
|
||||
|
||||
Until then: pre-1.0, pin tight, re-test on every framework bump.
|
||||
196
DEPLOY-WINDOWS-IIS.md
Normal file
196
DEPLOY-WINDOWS-IIS.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# Deploy shopdb-flask to Windows IIS (MySQL 5.6)
|
||||
|
||||
Runbook for standing up a single-site instance on the production Windows Server
|
||||
that already runs the classic ASP shopdb, using IIS + HttpPlatformHandler +
|
||||
waitress, against the existing MySQL 5.6. This is the test-instance path; keep
|
||||
developing on the Linux dev box and redeploy as needed.
|
||||
|
||||
The Docker path in `DEPLOY.md` does NOT apply on Windows (gunicorn is Linux
|
||||
only, and there is no MySQL container here). This file replaces it for IIS.
|
||||
|
||||
Notation: `APP_ROOT` = the deploy folder, e.g. `C:\shopdb-flask`. The IIS site
|
||||
physical path must be `APP_ROOT` (where `wsgi.py` lives).
|
||||
|
||||
## 0. Prerequisites on the box
|
||||
|
||||
- Python 3.12 (same minor as dev). `py -3.12 --version` to confirm.
|
||||
- IIS with the **HttpPlatformHandler** module:
|
||||
https://www.iis.net/downloads/microsoft/httpplatformhandler
|
||||
- **URL Rewrite** module (only for the optional real-client-IP rule).
|
||||
- Network access to the MySQL 5.6 server.
|
||||
- If the box is air-gapped, you cannot `pip install` live. On the dev box run
|
||||
`pip download -r requirements.txt waitress -d wheels\` (on a matching
|
||||
Windows/Python target, or use `--platform` wheels), copy `wheels\` over, and
|
||||
install with `pip install --no-index --find-links wheels\ ...`.
|
||||
|
||||
## 1. Copy the code
|
||||
|
||||
Copy the repo to `APP_ROOT`, INCLUDING `frontend/dist` (the built SPA the API
|
||||
serves). Build it on dev first if stale:
|
||||
|
||||
```bash
|
||||
# on the dev box
|
||||
cd frontend && npm run build # produces frontend/dist
|
||||
```
|
||||
|
||||
Ship `frontend/dist` with the code (Node is not needed on the prod box).
|
||||
|
||||
## 2. Python venv + dependencies
|
||||
|
||||
```powershell
|
||||
cd C:\shopdb-flask
|
||||
py -3.12 -m venv venv
|
||||
venv\Scripts\python -m pip install --upgrade pip
|
||||
venv\Scripts\pip install -r requirements.txt
|
||||
venv\Scripts\pip install waitress
|
||||
```
|
||||
|
||||
The DB driver is `pymysql` (pure Python) so no C compiler / MySQL client libs
|
||||
are needed. `waitress` is the WSGI server (installed separately, same as the
|
||||
Docker image installs gunicorn separately).
|
||||
|
||||
## 3. Prepare MySQL 5.6 (the utf8mb4 gotcha)
|
||||
|
||||
MySQL 5.6 defaults cannot index utf8mb4 VARCHAR(255) columns (767-byte prefix
|
||||
limit) and often defaults the server charset to latin1. The schema is utf8mb4,
|
||||
so the server needs Barracuda + large-prefix, made durable in `my.ini` under
|
||||
`[mysqld]`, then restart the MySQL service:
|
||||
|
||||
```ini
|
||||
[mysqld]
|
||||
innodb_file_per_table = 1
|
||||
innodb_file_format = Barracuda
|
||||
innodb_large_prefix = 1
|
||||
```
|
||||
|
||||
Then create the database as utf8mb4 and a least-privilege app user:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
CREATE USER 'shopdb'@'%' IDENTIFIED BY 'CHANGE_ME';
|
||||
GRANT ALL PRIVILEGES ON shopdb_flask.* TO 'shopdb'@'%';
|
||||
FLUSH PRIVILEGES;
|
||||
```
|
||||
|
||||
Without the `[mysqld]` flags, `flask db upgrade` fails with error 1071
|
||||
("Specified key was too long"). The migration chain emits `ROW_FORMAT=DYNAMIC`
|
||||
per table (see `migrations/env.py`), which fits the 3072-byte prefix those
|
||||
flags unlock.
|
||||
|
||||
## 4. Configure secrets and connection (.env)
|
||||
|
||||
Create `APP_ROOT\.env` (loaded by `wsgi.py` via `load_dotenv()`). Keep secrets
|
||||
here, not in `web.config`. Lock the file's ACLs to the IIS app-pool identity +
|
||||
administrators.
|
||||
|
||||
```
|
||||
FLASK_ENV=production
|
||||
SECRET_KEY=<64+ random chars>
|
||||
JWT_SECRET_KEY=<another 64+ random chars>
|
||||
DATABASE_URL=mysql+pymysql://shopdb:CHANGE_ME@<mysql-host>:3306/shopdb_flask?charset=utf8mb4
|
||||
CORS_ORIGINS=https://<the site's own hostname>
|
||||
```
|
||||
|
||||
`ProductionConfig.validate()` refuses to boot if any of `SECRET_KEY`,
|
||||
`JWT_SECRET_KEY`, `DATABASE_URL`, `CORS_ORIGINS` is missing or left at a dev
|
||||
default. `CORS_ORIGINS` is the browser origin users hit (the IIS binding).
|
||||
|
||||
Generate a key: `venv\Scripts\python -c "import secrets;print(secrets.token_urlsafe(64))"`.
|
||||
|
||||
## 5. Initialize schema, data, plugins, admin
|
||||
|
||||
Run from `APP_ROOT` with the venv active and `.env` present:
|
||||
|
||||
```powershell
|
||||
$env:FLASK_APP="shopdb"
|
||||
venv\Scripts\flask db upgrade
|
||||
venv\Scripts\flask seed reference-data
|
||||
|
||||
# Enable the plugins this site tracks (registry lives in the gitignored
|
||||
# instance/plugins.json, so a fresh box starts with none enabled):
|
||||
venv\Scripts\flask plugin list
|
||||
venv\Scripts\flask plugin install computers
|
||||
venv\Scripts\flask plugin install equipment
|
||||
venv\Scripts\flask plugin install network
|
||||
venv\Scripts\flask plugin install notifications
|
||||
venv\Scripts\flask plugin install printers
|
||||
venv\Scripts\flask plugin install usb
|
||||
venv\Scripts\flask plugin install knowledgebase
|
||||
venv\Scripts\flask plugin install slides
|
||||
venv\Scripts\flask plugin install employees
|
||||
|
||||
# First admin (password is generated and printed once):
|
||||
venv\Scripts\flask seed admin --username admin --email admin@yourfacility.example.com
|
||||
```
|
||||
|
||||
(Alternatively copy the dev box's `instance/plugins.json` to `APP_ROOT\instance\`
|
||||
to reproduce the exact enabled set, then just run `flask plugin upgrade-all`.)
|
||||
|
||||
## 6. Create the IIS site + web.config
|
||||
|
||||
This describes the own-site method (the app gets its own IIS site + port). To
|
||||
mount the app at a subpath under an existing site instead (e.g.
|
||||
`https://<host>/ops/` sharing the classic site's binding and cert), see
|
||||
**docs/INSTALL-WINDOWS-IIS.md section 7b**: same web.config, but the site is a
|
||||
`New-WebApplication` under the parent, `MOUNT_PATH=/ops` is set (web.config or
|
||||
`.env`), and the frontend is built with `VITE_BASE_PATH=/ops/`.
|
||||
|
||||
1. In IIS Manager, add a new **Site** (separate from the classic ASP site):
|
||||
- Physical path: `APP_ROOT`
|
||||
- Binding: a free port or a dedicated hostname (e.g. `https` 443 with the
|
||||
facility cert, or `http` on a test port like 8081 to start).
|
||||
- App pool: No Managed Code, and an identity that can read `APP_ROOT`.
|
||||
2. Copy `deploy\windows\web.config` to `APP_ROOT\web.config` and edit the paths
|
||||
(`C:\shopdb-flask` -> your `APP_ROOT`). It launches
|
||||
`waitress-serve --port=%HTTP_PLATFORM_PORT% wsgi:app` and sets
|
||||
`FLASK_ENV=production` + `PYTHONPATH`.
|
||||
3. Create `APP_ROOT\logs` for the HttpPlatform stdout log.
|
||||
4. **Unlock the handler sections** (locked server-wide by default; without this
|
||||
IIS returns **HTTP 500.19** "section cannot be used at this path"):
|
||||
```powershell
|
||||
%windir%\system32\inetsrv\appcmd unlock config /section:system.webServer/handlers
|
||||
%windir%\system32\inetsrv\appcmd unlock config /section:system.webServer/httpPlatform
|
||||
```
|
||||
5. Grant the app-pool identity read/execute on `APP_ROOT` and modify on
|
||||
`APP_ROOT\logs` (e.g. `icacls APP_ROOT /grant "IIS AppPool\<pool>:(OI)(CI)RX" /T`).
|
||||
6. Recycle the app pool / restart the site.
|
||||
|
||||
TLS terminates at the IIS binding. The `X-Forwarded-For` URL Rewrite rule in the
|
||||
web.config (real client IP for audit logs / kiosk visitor-location) is
|
||||
**commented out by default** because it needs the URL Rewrite module - with it
|
||||
active but URL Rewrite absent, IIS returns HTTP 500.19. Install URL Rewrite and
|
||||
uncomment the `<rewrite>` block to enable it.
|
||||
|
||||
## 7. Smoke test
|
||||
|
||||
```powershell
|
||||
# SPA loads:
|
||||
curl.exe -k https://<host>/ # returns index.html
|
||||
# API rejects an empty login with a validation error (health signal):
|
||||
curl.exe -k -X POST https://<host>/api/auth/login -H "Content-Type: application/json" -d "{}"
|
||||
# expect JSON containing VALIDATION_ERROR
|
||||
```
|
||||
|
||||
Then log in through the browser as the admin from step 5 and confirm the
|
||||
dashboard renders.
|
||||
|
||||
## 8. Redeploying as dev advances
|
||||
|
||||
Because this is a test instance you keep iterating on:
|
||||
|
||||
1. Pull/copy new code to `APP_ROOT` (rebuild `frontend/dist` on dev if the UI
|
||||
changed).
|
||||
2. `venv\Scripts\pip install -r requirements.txt` (if deps changed).
|
||||
3. `venv\Scripts\flask db upgrade` (if new migrations).
|
||||
4. Recycle the app pool.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| Site 502 / process won't start | Check `APP_ROOT\logs\httpplatform*`. Usually a bad `processPath`, missing waitress, or `wsgi:app` not importable (set `PYTHONPATH`). |
|
||||
| Boots but SQL echoes / debug on | `FLASK_ENV` not `production` (web.config env var or `.env`). |
|
||||
| `flask db upgrade` error 1071 | MySQL 5.6 `[mysqld]` flags in step 3 not applied / server not restarted. |
|
||||
| ConfigError on boot | A required var (SECRET_KEY / JWT_SECRET_KEY / DATABASE_URL / CORS_ORIGINS) missing or left at a dev default in `.env`. |
|
||||
| Login works, CORS errors in browser | `CORS_ORIGINS` does not match the exact origin (scheme + host + port) the browser used. |
|
||||
| Audit logs show 127.0.0.1 | Expected without the URL Rewrite X-Forwarded-For rule (step 6). |
|
||||
214
DEPLOY.md
214
DEPLOY.md
@@ -0,0 +1,214 @@
|
||||
# Per-Site Deployment Runbook
|
||||
|
||||
shopdb-flask is single-tenant per ADR-004. Each adopting facility runs its own stack: own DB, own users, own enabled plugins, own secrets. This document is the runbook for a fresh site deploy.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker 24+ and Docker Compose v2 (or equivalent container runtime)
|
||||
- A reverse proxy with TLS termination (nginx, traefik, Caddy, GE corporate LB) -- the framework does not terminate TLS itself
|
||||
- A MySQL backup destination (offsite recommended)
|
||||
- Access to the GE Aerospace Gitea or a clone of the repo
|
||||
|
||||
## Step 1: Clone and configure
|
||||
|
||||
```bash
|
||||
git clone https://gitea.proudtech.net/ge-aerospace/shopdb-flask.git
|
||||
cd shopdb-flask
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env`:
|
||||
|
||||
| Variable | Required | Notes |
|
||||
|----------|----------|-------|
|
||||
| `FLASK_ENV` | Yes | `production` for live sites |
|
||||
| `SECRET_KEY` | Yes | `python -c "import secrets; print(secrets.token_urlsafe(64))"` |
|
||||
| `JWT_SECRET_KEY` | Yes | Same generation, different value |
|
||||
| `DATABASE_URL` | Yes | `mysql+pymysql://shopdb:PASSWORD@db:3306/shopdb_flask` (matches docker-compose) |
|
||||
| `CORS_ORIGINS` | Yes | Comma-separated explicit origins. Wildcard rejected. |
|
||||
| `MYSQL_ROOT_PASSWORD` | Yes | Container only |
|
||||
| `MYSQL_PASSWORD` | Yes | Container only, must match `DATABASE_URL` password |
|
||||
| `MYSQL_PORT` | No | Default 3306 |
|
||||
| `API_PORT` | No | Default 5001 |
|
||||
| `LOG_LEVEL` | No | Default INFO |
|
||||
| `ZABBIX_URL`, `ZABBIX_TOKEN` | No | Only if printers plugin uses Zabbix |
|
||||
| `COLLECTOR_API_KEY` | No | Shared key for `/api/collector/*` ingest. Required only if unattended collectors push data. Endpoint fails closed (denies) when unset. |
|
||||
| `COLLECTOR_API_KEY_<PLUGIN>` | No | Per-plugin override (e.g. `COLLECTOR_API_KEY_COMPUTERS`), checked before the shared key (ADR-006) |
|
||||
| `EMPLOYEE_DB_HOST/USER/PASSWORD/NAME` | No | Read-only HR directory for notifications + kiosks. No safe default for the password. |
|
||||
|
||||
## Step 2: Bring up the stack
|
||||
|
||||
```bash
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
The Docker image builds the Vue frontend in a first stage and copies the
|
||||
compiled SPA into the API image, so `docker compose build` produces a
|
||||
self-contained image with the UI already built. No separate Node step is
|
||||
needed for a container deploy. (For a bare-metal/venv install instead, build
|
||||
the frontend by hand: `cd frontend && npm ci && npm run build`, which writes
|
||||
`frontend/dist/` for Flask to serve.)
|
||||
|
||||
The MySQL container initializes its volume on first run. The API container waits for `db` to be healthy via `healthcheck`. Check logs:
|
||||
|
||||
```bash
|
||||
docker compose logs -f api
|
||||
```
|
||||
|
||||
If `ProductionConfig.validate()` raises, the container exits with the offending env-var named in the log. Fix `.env` and `docker compose up -d` again.
|
||||
|
||||
## Step 3: Initialize the database schema
|
||||
|
||||
```bash
|
||||
docker compose exec api flask db upgrade
|
||||
docker compose exec api flask plugin upgrade-all
|
||||
```
|
||||
|
||||
`flask db upgrade` applies the core Alembic chain: the baseline migration plus
|
||||
every later migration, which together create all core AND bundled-plugin tables
|
||||
through the chain head. `flask plugin upgrade-all` then stamps each bundled
|
||||
plugin's own migration chain (the `alembic_version_<plugin>` tables) and applies
|
||||
any plugin-specific migrations added after the ownership cutover. Both commands
|
||||
are idempotent, so re-running them is safe. See ADR-008 for why plugin schema
|
||||
splits into per-plugin chains from the cutover forward.
|
||||
|
||||
**Charset:** the schema is utf8mb4 (`utf8mb4_unicode_ci`). The docker-compose `db` service sets `--character-set-server=utf8mb4`, so the auto-created `shopdb_flask` database is utf8mb4. If you point at an external MySQL instead of the bundled container, create the database as utf8mb4 first, or it inherits the server default (often latin1) and the schema silently drifts:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
```
|
||||
|
||||
`DATABASE_URL` must keep `?charset=utf8mb4` so the connection matches. On MySQL older than 5.7 also enable `innodb_large_prefix=ON` + `innodb_file_format=Barracuda`, or the utf8mb4 indexes exceed the 767-byte prefix limit (error 1071). MySQL 5.7+ and 8.0 need no extra config.
|
||||
|
||||
## Step 4: Seed permissions, settings, and reference data
|
||||
|
||||
Run all three seeders. They are idempotent, so re-running them on an existing
|
||||
database is safe (it adds anything missing and leaves existing rows alone).
|
||||
|
||||
```bash
|
||||
docker compose exec api flask seed permissions
|
||||
docker compose exec api flask seed settings
|
||||
docker compose exec api flask seed reference-data
|
||||
```
|
||||
|
||||
- `seed permissions` - creates the RBAC permission rows and the default roles
|
||||
the app checks with `@require_permission`. Run this before anyone logs in or
|
||||
permission checks have nothing to match.
|
||||
- `seed settings` - writes the default Setting rows (branding, ServiceNow
|
||||
integration, floor-map placeholders, search toggles, site identity). A site
|
||||
overrides these later in Settings or the setup wizard.
|
||||
- `seed reference-data` - creates default `Vendor`, `Location`, `BusinessUnit`,
|
||||
`OperatingSystem`, `AssetStatus`, `RelationshipType` rows seeded with the
|
||||
platform contract values (`partof`, `controls`, `connectedto`).
|
||||
|
||||
## Step 5: Pick plugins to enable
|
||||
|
||||
The image bundles eleven plugins (computers, employees, knowledgebase, machines, measuringtools, network, notifications, printers, slides, usb, warranty). Only enabled plugins are loaded.
|
||||
|
||||
```bash
|
||||
docker compose exec api flask plugin list
|
||||
docker compose exec api flask plugin install computers
|
||||
docker compose exec api flask plugin install machines
|
||||
# ... repeat for each plugin the site tracks
|
||||
```
|
||||
|
||||
To install a sister-site or third-party plugin (per ADR-003), drop its directory into `<repo>/plugins/<name>/` (the docker-compose mounts this read-only into the container) and run `flask plugin install <name>`.
|
||||
|
||||
## Step 6: Create the first admin (setup wizard)
|
||||
|
||||
The primary path is the first-run setup wizard. Once the stack is up and the DB
|
||||
is seeded, browse to the site (through the reverse proxy configured in Step 7,
|
||||
or directly at the API port during bring-up) and go to `/setup`. The wizard
|
||||
creates the first admin account and captures site identity (facility name,
|
||||
optional logo). It runs only while `setup_complete` is false; after it finishes
|
||||
the route redirects to the app.
|
||||
|
||||
Headless alternative (no browser, e.g. automated provisioning):
|
||||
|
||||
```bash
|
||||
docker compose exec api flask seed admin --username admin --email admin@facility.example.com
|
||||
# Password is generated and printed once. Store in your password manager.
|
||||
```
|
||||
|
||||
Subsequent users are managed through the UI.
|
||||
|
||||
## Step 7: Front the API with TLS
|
||||
|
||||
The Flask container listens on `5001/tcp` over plain HTTP. Production exposure must go through a reverse proxy that terminates TLS:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name shopdb.facility-a.example.com;
|
||||
|
||||
ssl_certificate /etc/ssl/certs/shopdb.crt;
|
||||
ssl_certificate_key /etc/ssl/private/shopdb.key;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:5001;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The framework reads `X-Forwarded-For` for audit logging.
|
||||
|
||||
## Step 8: Backups
|
||||
|
||||
Per-site MySQL backups are the site's responsibility. Recommended: nightly `mysqldump` to offsite storage with 14-day retention.
|
||||
|
||||
```bash
|
||||
docker compose exec -T db mysqldump -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask | gzip > backup-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
Verify a restore quarterly. Back up the `instance/` directory alongside the DB;
|
||||
it holds uploaded floor plans, branding, `plugins.json`, and tokens that are not
|
||||
in MySQL. See [docs/BACKUP-RESTORE.md](BACKUP-RESTORE) for the full backup and
|
||||
restore procedure.
|
||||
|
||||
## Step 9: Updates
|
||||
|
||||
```bash
|
||||
git pull origin main
|
||||
docker compose build api
|
||||
docker compose up -d api
|
||||
docker compose exec api flask db upgrade
|
||||
```
|
||||
|
||||
The framework's `__contract_version__` may have moved. Check `docs/adr/` for any new ADRs since the last update. If an ADR introduces a breaking change, the upgrade may require coordinated work; the ADR's "Consequences" section documents it. See [docs/UPGRADE.md](UPGRADE) for the full upgrade procedure, including re-seeding and the v0.5+ floor-plan note.
|
||||
|
||||
## Common issues
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `ConfigError: SECRET_KEY is required in production` | `.env` missing or blank | Set `SECRET_KEY` in `.env`, re-up |
|
||||
| `ConfigError: CORS_ORIGINS must be a comma-separated allowlist` | `.env` has `*` | Set explicit origins |
|
||||
| `PluginVersionError: requires core_version X but framework is Y` | Plugin pinned a too-narrow range | Update `manifest.json` `core_version` or pin framework version |
|
||||
| 500s after `flask db upgrade` | Migration ran but app cached old schema | `docker compose restart api` |
|
||||
| Cannot reach API after restart | Reverse proxy not pointing at the container's exposed port | Confirm `API_PORT` and proxy config |
|
||||
|
||||
## Health check
|
||||
|
||||
```bash
|
||||
curl -s -X POST -H "Content-Type: application/json" \
|
||||
-d '{}' http://localhost:5001/api/auth/login \
|
||||
| jq .
|
||||
# Expect: {"status": "error", "data": {"error": {"code": "VALIDATION_ERROR", ...}}}
|
||||
```
|
||||
|
||||
If this returns a 500 or no JSON, the container is unhealthy. Check `docker compose logs api`.
|
||||
|
||||
## References
|
||||
|
||||
- [docs/adr/ADR-004-deployment-topology.md](ADR-004-deployment-topology) - per-site instances rationale
|
||||
- [docs/adr/ADR-003-plugin-distribution.md](ADR-003-plugin-distribution) - bundled vs external plugins
|
||||
- [docs/adr/ADR-006-collector-contract.md](ADR-006-collector-contract) - per-plugin collector endpoints
|
||||
- [docs/PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART) - building a custom plugin for your site
|
||||
- [docs/CONFIG.md](CONFIG) - every environment variable and every Setting key
|
||||
- [docs/UPGRADE.md](UPGRADE) - upgrade procedure for an existing site
|
||||
- [docs/BACKUP-RESTORE.md](BACKUP-RESTORE) - backup and restore procedure
|
||||
- [shopdb/config.py](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/shopdb/config.py) - all the env-vars in one place
|
||||
|
||||
126
GE-ENFORCE-CLIENT.md
Normal file
126
GE-ENFORCE-CLIENT.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# GE-Enforce client integration (shopdb manifest source + reporting)
|
||||
|
||||
This is the client-side contract for the GE-Enforce manifest-store plugin: how a
|
||||
PC sources its install manifest from shopdb instead of a share file, and how it
|
||||
reports its enforcement result back. It pairs with the plugin proposal in
|
||||
`docs/proposals/ge-enforce-plugin.md`.
|
||||
|
||||
The reference kit lives in `plugins/geenforce/client/`:
|
||||
|
||||
- `ShopdbEnforceClient.psm1` - fetch (with ETag + last-known-good cache),
|
||||
shadow compare, and report helpers.
|
||||
- `Invoke-ShopdbEnforce.ps1` - a reference orchestrator that fetches a manifest,
|
||||
runs the UNCHANGED engine against it, and reports the result.
|
||||
|
||||
These are site-neutral references, not the live dispatcher. A site adapts them
|
||||
into its GE-Enforce.ps1 flow. The engine (`Install-FromManifest.ps1`),
|
||||
detection, self-heal, and SMB payload resolution are untouched - only the source
|
||||
of the manifest JSON moves, plus a result report.
|
||||
|
||||
## What does NOT change
|
||||
|
||||
- The engine and its four filters, all detection methods, self-heal, marker
|
||||
files, and SMB payload staging.
|
||||
- Payload transport for `smb` rows: the client still mounts the share and
|
||||
resolves `apps/...` paths exactly as today. Only the manifest JSON source moves.
|
||||
- The fail-safe posture: any error exits 0. A PC is never blocked or broken
|
||||
because shopdb is unreachable.
|
||||
|
||||
## Configuration
|
||||
|
||||
Registry (provisioned by Azure DSC, same channel as the SFLD credentials):
|
||||
|
||||
```
|
||||
HKLM:\SOFTWARE\GE\ShopDB
|
||||
BaseUrl https://shopdb.<site>.geaerospace.net
|
||||
ApiToken <a geenforce.fetch (+ geenforce.report) managed service token>
|
||||
```
|
||||
|
||||
Mint the token in shopdb: Settings > API Tokens, scopes `geenforce.fetch` and
|
||||
`geenforce.report`. It is a service token (owner must hold those permissions).
|
||||
|
||||
## Fetch contract
|
||||
|
||||
```
|
||||
GET /api/geenforce/manifest?pctype=<scope>[&phase=runtime]
|
||||
X-API-Key: <token>
|
||||
If-None-Match: <cached ETag> (optional)
|
||||
```
|
||||
|
||||
- `200` - body is the full published manifest JSON for the scope (fat client:
|
||||
the engine filters locally, exactly as today). Response headers carry `ETag`
|
||||
and `X-Manifest-Version`. Cache the body + ETag + version.
|
||||
- `304` - your cached copy is current; use it.
|
||||
- `404` - no such scope, or the scope has no published version yet.
|
||||
- Network failure - enforce from the last-known-good cached manifest (the kit
|
||||
does this automatically) and log a warning.
|
||||
|
||||
The served manifest is always the current PUBLISHED snapshot, never a live draft
|
||||
being edited in shopdb, so a half-finished edit can never reach a PC.
|
||||
|
||||
## Report contract
|
||||
|
||||
Each enforcement cycle, POST the result (best-effort; a failed report never
|
||||
fails the cycle):
|
||||
|
||||
```
|
||||
POST /api/geenforce/report
|
||||
X-API-Key: <token>
|
||||
Content-Type: application/json
|
||||
{
|
||||
"hostname": "WJCMM01",
|
||||
"scopename": "gea-shopfloor-cmm",
|
||||
"appliedversion": 3, // the published version you actually ran
|
||||
"enforcerversion": "2.6",
|
||||
"counts": { "installed": 1, "skipped": 3, "failed": 0, "filtered": 2 },
|
||||
"results": [
|
||||
{ "name": "PC-DMIS 2019 R2", "action": "installed", "selfhealed": true },
|
||||
{ "name": "Protect Viewer", "action": "skipped" },
|
||||
{ "name": "eDNC", "action": "failed", "exitcode": 1603,
|
||||
"message": "MSI 1603" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `appliedversion` lets shopdb show which PCs received the latest manifest
|
||||
(`receivedlatest` in the fleet view).
|
||||
- `action` per entry: `installed` (fired - a self-heal when it should already be
|
||||
present), `skipped` (detected present), `failed`, `filtered`. `selfhealed`
|
||||
marks a drift correction.
|
||||
- shopdb keeps the latest report per (hostname, scope, phase) plus history, and
|
||||
surfaces it under Settings > Enforcement Reports.
|
||||
|
||||
The engine already computes these counts (`installed/skipped/failed/pcFiltered`
|
||||
at the end of its main loop) and knows each entry's action; shape them into the
|
||||
`results` list at the call site (`New-ShopdbReport` in the kit takes a summary
|
||||
with `Installed/Skipped/Failed/Filtered` + a `Results` list).
|
||||
|
||||
## Cutover (safe, staged)
|
||||
|
||||
1. **Configure** the registry values on a canary PC; mint the token.
|
||||
2. **Shadow mode**: run `Invoke-ShopdbEnforce.ps1 -ShadowMode -ShareManifestPath
|
||||
<current share manifest>`. It installs from the SHARE (no behavior change),
|
||||
fetches the shopdb manifest, logs any diff, and reports. Watch for zero diffs
|
||||
across one PC of every pctype for ~20 cycles.
|
||||
3. **Read cutover**: drop `-ShadowMode`. The engine now runs against the
|
||||
shopdb-sourced manifest; payloads still come from the share. Rollback is a
|
||||
one-line revert to the share-sourced call. Keep exporting manifests from
|
||||
shopdb to the share (Settings > Imaging PC Types > Export to Share) so the
|
||||
share stays a break-glass copy.
|
||||
4. **Payload migration** (optional, later): move small scripts/configs to
|
||||
`http`/`inline` payloads, verified by `payloadsha256`. Big MSIs stay on SMB.
|
||||
|
||||
Do not cut a fleet over before the shadow diffs are clean. Preinstall
|
||||
(`phase=preinstall`) stays share-sourced until its own cutover is planned - it
|
||||
runs before enrollment provisions a token.
|
||||
|
||||
## Security notes
|
||||
|
||||
- The client runs as SYSTEM, so shopdb's TLS certificate must be in the machine
|
||||
trust store (air-gapped/self-signed sites provision the CA via the same DSC
|
||||
step as the token).
|
||||
- `http`/`inline` payloads are verified against `payloadsha256` before running,
|
||||
independent of how the entry detects install state. This is the real integrity
|
||||
guarantee and holds even over plain HTTP inside a trusted segment.
|
||||
- The token is a scoped service token: it can fetch manifests and report, and
|
||||
nothing else.
|
||||
135
GE-ENFORCE-DEPLOY.md
Normal file
135
GE-ENFORCE-DEPLOY.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Deploying the GE-Enforce agent on a PC
|
||||
|
||||
This is the deploy contract: what has to be laid down on a PC so GE-Enforce runs,
|
||||
and how to do it regardless of imaging path (PXE, OOBE provisioning package,
|
||||
Intune, or by hand). It complements `docs/GE-ENFORCE.md` (concepts) and
|
||||
`docs/GE-ENFORCE-CLIENT.md` (the fetch/report contract).
|
||||
|
||||
The reference installer is `plugins/geenforce/client/Install-GEEnforce.ps1`. It
|
||||
is site-neutral: you pass the PC's identity in, it writes the files/registry the
|
||||
engine reads and registers the enforcement task.
|
||||
|
||||
---
|
||||
|
||||
## 1. What "deploying GE-Enforce" means
|
||||
|
||||
A PC needs three things present before enforcement works. HOW they get there is
|
||||
up to your imaging path; WHAT they are is fixed:
|
||||
|
||||
1. **The GE-Enforce client** - the engine (`Install-FromManifest.ps1`), the
|
||||
shopdb client kit (`ShopdbEnforceClient.psm1`, `Invoke-ShopdbEnforce.ps1`),
|
||||
and a scheduled task (at logon + periodic) that runs as SYSTEM.
|
||||
2. **Identity** in `C:\Enrollment` - so the PC knows what it is (see section 2).
|
||||
3. **A credential** - the SFLD share credential (for a share-sourced manifest)
|
||||
and/or the shopdb service token (for the fetch/report client). This is what
|
||||
gates enforcement actually starting; until it exists the task exits 0 and
|
||||
retries.
|
||||
|
||||
`Install-GEEnforce.ps1` lays down 1 and 2, and can write the shopdb token for 3.
|
||||
The engine itself is the GE-Enforce framework's, not shopdb's - point the
|
||||
installer at your copy with `-EngineSource`, or place it under the install root
|
||||
first (see section 5).
|
||||
|
||||
---
|
||||
|
||||
## 2. Identity: how a PC determines its PC type (and bay)
|
||||
|
||||
There is NO auto-detection. The provisioner supplies the values; the engine only
|
||||
reads files. This is the core of "set the PC up to know its type."
|
||||
|
||||
| Value | Written to | Purpose | Required? |
|
||||
|---|---|---|---|
|
||||
| **PC type** | `C:\Enrollment\pc-type.txt` (first line) | picks the manifest scope (`gea-shopfloor-<type>`) | YES |
|
||||
| Machine (bay) number | `C:\Enrollment\machine-number.txt` (fallback; DNC registry `MachineNo` wins) | per-bay gates | only for bay-gated entries |
|
||||
| CMM version | `C:\Enrollment\cmm\version.txt` | `_CmmVersion` gating (CMM PCs) | CMM only |
|
||||
| CMM bay id | `C:\Enrollment\cmm\cmmid.txt` | CMM bay identity | CMM only |
|
||||
| Share root + site | `C:\Enrollment\site-config.json` | where manifests/payloads live | for share-sourced |
|
||||
| shopdb URL + token | `HKLM:\SOFTWARE\GE\ShopDB` (BaseUrl, ApiToken) | fetch/report client | for shopdb client |
|
||||
|
||||
Valid `pc-type` values are the manifest scope names
|
||||
(`gea-shopfloor-cmm`, `-collections`, `-nocollections`, `-common`, `-keyence`,
|
||||
`-genspect`, `-heattreat`, `-partmarker`, `-waxtrace`) or a legacy alias the
|
||||
engine maps (`Standard`, `CMM`, ...).
|
||||
|
||||
**shopdb cannot set the type at imaging** - a PC is not known to shopdb until it
|
||||
enrolls and reports. If you want the value to come from an asset system, pre-map
|
||||
asset-tag / hostname -> PC type in your provisioning and feed it to the
|
||||
installer.
|
||||
|
||||
---
|
||||
|
||||
## 3. Running it, per imaging path
|
||||
|
||||
`Install-GEEnforce.ps1` is the same in every case; only how you invoke it differs.
|
||||
|
||||
### PXE / imaging step (identity known at image time)
|
||||
Run it as an imaging step after the OS lays down, passing the type the operator
|
||||
selected:
|
||||
|
||||
```
|
||||
powershell -ExecutionPolicy Bypass -File Install-GEEnforce.ps1 `
|
||||
-PCType gea-shopfloor-cmm -MachineNumber 0615 -CmmVersion 2019 `
|
||||
-ShareRoot \\server\share\dt\shopfloor -Site "West Jefferson" `
|
||||
-ShopdbUrl https://shopdb.site.geaerospace.net -ShopdbToken shopdb_pat_xxx `
|
||||
-EngineSource \\server\share\dt\shopfloor\common
|
||||
```
|
||||
|
||||
### OOBE provisioning package (ppkg)
|
||||
Sites that apply a ppkg during OOBE (no PXE/WinPE step) embed the installer + the
|
||||
client kit in the ppkg and run it from a `CommandLine` / `ProvisioningCommands`
|
||||
action. Supply the PC type from a ppkg variable, a first-boot prompt, or an
|
||||
asset lookup:
|
||||
|
||||
```
|
||||
powershell -ExecutionPolicy Bypass -File Install-GEEnforce.ps1 -PCType %PCTYPE% ...
|
||||
```
|
||||
|
||||
Timing is forgiving: the scheduled task is fail-safe, so if OOBE finishes before
|
||||
Intune/DSC provisions the credential, enforcement simply waits and starts once
|
||||
the credential lands. There is no ordering trap.
|
||||
|
||||
### Intune / manual
|
||||
Same script as a Win32 app / remediation, or run by hand on an existing PC to
|
||||
retrofit it. `-NoTask` provisions identity + kit without registering the task.
|
||||
|
||||
---
|
||||
|
||||
## 4. What the installer does (idempotent)
|
||||
|
||||
1. Writes the `C:\Enrollment` identity files (section 2).
|
||||
2. Writes `HKLM:\SOFTWARE\GE\ShopDB` (BaseUrl + token) if provided.
|
||||
3. Copies the client kit (the two files shipped next to it) to `-InstallRoot`
|
||||
(default `C:\ProgramData\GE-Enforce`).
|
||||
4. If `-EngineSource` is given, copies `GE-Enforce.ps1` + `lib\Install-FromManifest.ps1`.
|
||||
5. Registers the scheduled task (SYSTEM, at logon + every `-IntervalMinutes`) to
|
||||
run `Invoke-ShopdbEnforce.ps1 -Scope <PCType> -EnginePath <engine>`.
|
||||
|
||||
Re-running it updates identity/config and re-registers the task in place.
|
||||
|
||||
---
|
||||
|
||||
## 5. The engine boundary
|
||||
|
||||
shopdb ships the **manifest store + client kit + this installer**, not the
|
||||
GE-Enforce **engine** (`Install-FromManifest.ps1`) or dispatcher - those live in
|
||||
the GE-Enforce framework. So one of:
|
||||
|
||||
- pass `-EngineSource <path>` pointing at a folder that has `GE-Enforce.ps1` and
|
||||
`lib\Install-FromManifest.ps1` (e.g. your share's `common` dir), or
|
||||
- place the engine under `<InstallRoot>\lib\Install-FromManifest.ps1` yourself
|
||||
before enforcement runs.
|
||||
|
||||
The installer warns if the engine is missing but still provisions identity so a
|
||||
PC is at least correctly labelled. Use engine lib >= 2.6 (required for the
|
||||
`_CmmVersion` gate).
|
||||
|
||||
---
|
||||
|
||||
## 6. Verify a provisioned PC
|
||||
|
||||
- `Get-Content C:\Enrollment\pc-type.txt` -> the expected scope.
|
||||
- `Get-ItemProperty HKLM:\SOFTWARE\GE\ShopDB` -> BaseUrl + ApiToken set.
|
||||
- `Get-ScheduledTask GE-Enforce` -> Ready.
|
||||
- Trigger it once and check the client log
|
||||
(`C:\Logs\Shopfloor\shopdb-enforce-*.log`), then confirm the PC appears under
|
||||
**GE-Enforce > Enforcement Reports** in shopdb with the right PC type.
|
||||
329
GE-ENFORCE.md
Normal file
329
GE-ENFORCE.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# GE-Enforce: concepts, the shopdb plugin, and imaging-time integration
|
||||
|
||||
This guide explains how GE-Enforce works, how the shopdb `geenforce` plugin
|
||||
manages it, and when GE-Enforce installs and takes over during the imaging
|
||||
process. It is written for site IT.
|
||||
|
||||
It pairs with two companion docs:
|
||||
- `docs/GE-ENFORCE-CLIENT.md` - the client fetch/report contract + the reference
|
||||
PowerShell kit (`plugins/geenforce/client/`).
|
||||
- `docs/proposals/ge-enforce-plugin.md` - the design/plan and the staged cutover.
|
||||
|
||||
The ground truth for behavior is the engine itself
|
||||
(`Install-FromManifest.ps1`) and the on-share manifests; this guide describes
|
||||
what they do, it does not replace them.
|
||||
|
||||
---
|
||||
|
||||
## 1. What GE-Enforce is
|
||||
|
||||
GE-Enforce is a **desired-state enforcement** system for shopfloor PCs. Instead
|
||||
of a one-time install during imaging, it continuously makes each PC match a
|
||||
declared list of what should be installed - and RE-installs anything that drifts
|
||||
(uninstalled, corrupted, or overwritten). It is the shopfloor equivalent of a
|
||||
lightweight, air-gapped-friendly configuration-management agent.
|
||||
|
||||
Two things make up the system:
|
||||
|
||||
1. **The engine + dispatcher on each PC** - PowerShell that reads a manifest and
|
||||
enforces it every logon and periodically.
|
||||
2. **The manifests** - JSON files that declare, per imaging PC type, what to
|
||||
install / copy / write and how to detect whether it is already correct.
|
||||
|
||||
The shopdb `geenforce` plugin adds a third piece: it lets you **author, publish,
|
||||
and version those manifests in shopdb** (instead of hand-editing JSON on a file
|
||||
share) and **see what every PC actually did** (fleet compliance reporting).
|
||||
|
||||
---
|
||||
|
||||
## 2. How GE-Enforce works (the framework)
|
||||
|
||||
### 2.1 Two phases
|
||||
|
||||
Every shopfloor PC is governed in two distinct phases:
|
||||
|
||||
| Phase | When | Runs what | Purpose |
|
||||
|---|---|---|---|
|
||||
| **Preinstall** | ONCE, at imaging | `preinstall.json` (via the imaging `00-PreInstall` step) | Day-zero foundation: PowerShell 7, the VC++ redistributable matrix, Oracle Client, Adobe Reader, HostExplorer, serial drivers, etc. "Install once at imaging, no drift correction." |
|
||||
| **Runtime** | EVERY logon + periodically | `common/manifest.json`, then `gea-shopfloor-<type>/manifest.json`, then an optional `<type>-<subtype>` manifest | Ongoing enforcement + self-heal: app versions, config-file drift, registry drift, per-cycle scripts (asset report, VNC firewall, EventSaver), version-gated installs. |
|
||||
|
||||
The two phases share the same entry SHAPE (field names) but are run by different
|
||||
runners with different capabilities. Preinstall is a one-shot at imaging that
|
||||
implements only `Type=MSI` and `Type=EXE`, with only `Registry` / `File`
|
||||
detection (other types/detections are skipped). Runtime is the continuous
|
||||
enforcement loop and implements the full Type + DetectionMethod matrix below.
|
||||
|
||||
### 2.2 The runtime loop, step by step
|
||||
|
||||
On each cycle (`GE-Enforce.ps1` on the PC):
|
||||
|
||||
1. Read the PC's identity from `C:\Enrollment\` (see 2.4).
|
||||
2. Look up the SFLD share credential in the registry and **mount the share**
|
||||
(SYSTEM cannot reach the share as its computer account, so it mounts as the
|
||||
provisioned SFLD user - `net use W: ...`). If no credential yet, exit 0 and
|
||||
retry next cycle (Azure DSC has not provisioned it).
|
||||
3. Run the engine (`Install-FromManifest.ps1`) against `common/manifest.json`,
|
||||
then `gea-shopfloor-<pctype>/manifest.json`, then a `<pctype>-<subtype>`
|
||||
manifest if one exists. Common runs first so shared prerequisites (e.g.
|
||||
Oracle Client) land before type-specific apps that depend on them.
|
||||
4. Write a status file back to the share (and, in the shopdb model, POST a
|
||||
report - see 4.3).
|
||||
|
||||
Every failure is non-fatal (exit 0) so a network blip or a not-yet-provisioned
|
||||
credential never blocks or breaks a PC.
|
||||
|
||||
### 2.3 The manifest: scopes and entries
|
||||
|
||||
A manifest is `{ "Version", "_comment", "Applications": [ entry, ... ] }`. Each
|
||||
imaging PC type is a **scope** with its own manifest, plus the fleet-wide
|
||||
`common` scope:
|
||||
|
||||
- `common` - runs on EVERY PC type; entries use a `PCTypes` filter to target
|
||||
subsets (e.g. "EventSaver on collections + heattreat, but not CMM").
|
||||
- `gea-shopfloor-collections`, `-nocollections`, `-cmm`, `-keyence`, `-common`
|
||||
(lab/timeclock), `-genspect`, `-heattreat`, `-partmarker`, `-waxtrace` - each
|
||||
runs only on PCs of that type, so its entries usually do NOT set `PCTypes`
|
||||
(the manifest already only runs there). Keyence is the exception: it uses
|
||||
`PCTypes` for hardware SUBTYPE targeting (`keyence-vr6000` vs `keyence-vr3000`).
|
||||
|
||||
Each **entry** declares one action. Its `Type` picks the action:
|
||||
|
||||
| Type | Action |
|
||||
|---|---|
|
||||
| MSI / EXE / CMD / BAT | run an installer with `InstallArgs` |
|
||||
| PS1 | run a script from the share |
|
||||
| INF | install a driver via `pnputil` |
|
||||
| File | copy `Source` -> `Destination` |
|
||||
| Registry | write a value |
|
||||
|
||||
### 2.4 Self-heal via detection
|
||||
|
||||
Every entry has a `DetectionMethod` that decides whether the action fires:
|
||||
|
||||
| Method | Means "already correct" when... |
|
||||
|---|---|
|
||||
| Registry | the key/value exists (optionally equals a value) |
|
||||
| File | the file exists |
|
||||
| FileVersion | the file's version string matches exactly (fleet convention is a 4-part string like 6.4.5.0; the engine does a raw string compare, it does not enforce 4 parts) |
|
||||
| Hash | the file's SHA256 matches (case-insensitive) |
|
||||
| MarkerFile | a marker file exists (the engine writes it after a clean install) |
|
||||
| ValueMatches | a registry value equals the entry's target |
|
||||
| pnputil | a driver matching a pattern is present |
|
||||
| Always / (none) | fires EVERY cycle (used for per-cycle scripts) |
|
||||
|
||||
If detection says "not correct," the action runs. That is the self-heal: delete
|
||||
`DncMain.exe` and next cycle re-installs eDNC; corrupt a config file whose Hash
|
||||
no longer matches and next cycle re-copies it. **Entry order is execution
|
||||
order** - config-restore entries sit AFTER their installer so a mid-cycle vendor
|
||||
overwrite is healed on the same cycle.
|
||||
|
||||
### 2.5 Targeting gates (all ANDed)
|
||||
|
||||
An entry can be narrowed by any combination of:
|
||||
|
||||
- `PCTypes` - which PC types (alias-aware: old names like `Standard` map to
|
||||
`collections`/`nocollections`/`common`). Fleet-wide `common` uses this heavily.
|
||||
- `TargetHostnames` - specific hostnames (supports `*` wildcards).
|
||||
- `TargetMachineNumbers` - specific bay machine numbers (e.g. Okuma bays).
|
||||
- `_CmmVersion` - CMM PCs only: a tagged entry applies when it equals the bay's
|
||||
resolved PC-DMIS version (`C:\Enrollment\cmm\version.txt`). IMPORTANT: if no
|
||||
version is resolved (file missing/empty - a pre-picker bay), ALL tagged
|
||||
entries apply (deliberate legacy "install-all" behavior), so such a bay gets
|
||||
every PC-DMIS version, not none. Requires engine lib >= 2.6.
|
||||
- `PCTypesStrict` - disables alias expansion (PREINSTALL runner only; the runtime
|
||||
engine ignores it).
|
||||
|
||||
Different PC types have different niche gates: CMM uses a version gate, Keyence a
|
||||
model subtype, Collections per-bay machine numbers. The shopdb editor shows only
|
||||
the gates a given scope actually uses (see 4.1).
|
||||
|
||||
### 2.6 What the PC needs to know about itself (enrollment)
|
||||
|
||||
The runtime engine reads the PC's identity from `C:\Enrollment\`:
|
||||
|
||||
- `pc-type.txt` - the imaging PC type (which scope to run). `pc-subtype.txt` is
|
||||
LEGACY (no longer written at imaging since the 2026-05-04 rename reorg; the
|
||||
dispatcher still honors it if present on older fleet PCs).
|
||||
- `machine-number.txt` - the bay number FALLBACK; the eDNC/DNC registry
|
||||
`MachineNo` value wins if present. `9999` is the imaging placeholder by
|
||||
convention - the enforcement engine does NOT special-case it; it is simply a
|
||||
value that won't match a real bay number in a `TargetMachineNumbers` gate.
|
||||
(The 9999-skip you may see is only in the status write-back, not enforcement.)
|
||||
- `cmm/version.txt` - CMM bays only: the resolved PC-DMIS version for `_CmmVersion`.
|
||||
- `site-config.json` - the share root and site settings.
|
||||
- SFLD credentials at `HKLM:\SOFTWARE\GE\SFLD\Credentials` - provisioned by
|
||||
Azure DSC after enrollment (this is what gates the runtime phase starting).
|
||||
|
||||
Note: all of the identity files above (pc-type, machine-number, cmm version,
|
||||
site-config) are written in WinPE at the PXE menu, BEFORE the image boots - the
|
||||
preinstall phase already reads them. What happens post-imaging is only Intune
|
||||
enrollment + the Azure DSC credential (see the timeline below).
|
||||
|
||||
---
|
||||
|
||||
## 3. When GE-Enforce installs / takes over (the imaging timeline)
|
||||
|
||||
This is the "when to implement it during imaging" question. The order is:
|
||||
|
||||
```
|
||||
[0] WinPE / PXE menu (BEFORE the image boots)
|
||||
- identity written to C:\Enrollment: pc-type.txt, machine-number.txt,
|
||||
cmm/version.txt, site-config.json (startnet.cmd). The PC already knows
|
||||
what it is before Windows starts.
|
||||
|
|
||||
v
|
||||
PXE image applied, Windows boots
|
||||
|
|
||||
v
|
||||
[1] PREINSTALL (00-PreInstall runner runs preinstall.json ONCE)
|
||||
- reads the step-0 identity files, then installs the foundation:
|
||||
PowerShell 7, VC++ redists, Oracle Client, Adobe Reader, HostExplorer,
|
||||
serial drivers, Display kiosk app, ... (things later runtime apps need)
|
||||
- preinstall implements MSI/EXE + Registry/File detection only
|
||||
|
|
||||
v
|
||||
[2] GE-ENFORCE ITSELF is laid down during imaging
|
||||
- the dispatcher (GE-Enforce.ps1), the engine lib (Install-FromManifest.ps1),
|
||||
and a scheduled task (at-logon + every ~5 min + shift windows) are
|
||||
registered as part of the image / shopfloor setup
|
||||
|
|
||||
v
|
||||
[3] ENROLLMENT (post-imaging)
|
||||
- Intune / GCCH enrollment, THEN Azure DSC provisions the SFLD share
|
||||
credential into HKLM:\SOFTWARE\GE\SFLD\Credentials
|
||||
- (the identity files already exist from step 0 - enrollment adds only the
|
||||
credential, which is what unblocks runtime)
|
||||
|
|
||||
v
|
||||
[4] FIRST LOGON -> RUNTIME ENFORCEMENT BEGINS
|
||||
- the scheduled task runs GE-Enforce.ps1: mount share, run common + the
|
||||
PC-type (+ subtype) manifests, install/self-heal, report
|
||||
- repeats every logon + periodically forever after
|
||||
```
|
||||
|
||||
Key points on timing:
|
||||
|
||||
- **Preinstall (step 1) is the imaging-time install.** Put anything that must
|
||||
exist before first logon, or that never needs drift correction, here (runtimes,
|
||||
redistributables, drivers). It runs once and is done.
|
||||
- **Runtime enforcement (step 4) does not start until enrollment (step 3)
|
||||
provisions the SFLD credential.** Before that, GE-Enforce exits 0 each cycle
|
||||
and waits. So a freshly imaged PC that is not yet enrolled is inert, by design.
|
||||
- **The engine lib version matters.** `_CmmVersion` gating needs lib >= 2.6 on
|
||||
the PC; deploy the lib before a manifest that uses it.
|
||||
- Some apps appear in BOTH phases: preinstalled at imaging for day-zero, then
|
||||
carried by a runtime entry so drift is corrected later (Oracle, UDC, Adobe,
|
||||
HostExplorer, Defect Tracker).
|
||||
|
||||
Rule of thumb: **imaging-time (preinstall) = foundation that must be there or
|
||||
never drifts; runtime = everything that needs to stay correct over the PC's
|
||||
life.**
|
||||
|
||||
---
|
||||
|
||||
## 4. How the shopdb plugin manages this
|
||||
|
||||
The `geenforce` plugin turns the manifest from hand-edited JSON on a share into
|
||||
shopdb data you author, version, publish, and monitor. It lives under the
|
||||
top-level **GE-Enforce** section (Manifests | Enforcement Reports), not Settings,
|
||||
because it is a full management surface.
|
||||
|
||||
### 4.1 Manifests - authoring (GE-Enforce > Manifests)
|
||||
|
||||
- **PC Types (scopes):** each imaging PC type is a row; add/edit/delete. (The
|
||||
scope carries an optional `computertypeid` reference field, but the collector's
|
||||
imaging-pc-type -> ComputerType mapping is configured separately at
|
||||
Settings > Collector PC Types.)
|
||||
- **Entries:** an ordered list (Up/Down = the execution-order contract). Add/Edit
|
||||
opens a typed form: the payload fields switch on `Type` (MSI shows Installer +
|
||||
InstallArgs, PS1 shows Script + Args, File shows Source + Destination, Registry
|
||||
shows the Reg* fields), a detection block, an InUseCheck editor, and a
|
||||
**Targeting** section that shows only the gates the scope uses (CMM shows the
|
||||
version gate; the common/preinstall scopes show PC types; a scope whose entries
|
||||
use machine numbers shows those) with a "Show all targeting options" escape
|
||||
hatch.
|
||||
- **Simulate ("what would a PC get?"):** enter a PC profile (type, subtype,
|
||||
hostname, machine number, CMM version) and see which entries apply and why the
|
||||
rest are filtered - without reading a PowerShell log.
|
||||
- **Publish / Versions / Roll Back:** editing changes a DRAFT only. Publish
|
||||
freezes an immutable version; PCs are only ever served the published version;
|
||||
Roll Back restores an earlier one. History (date, author, note) per version.
|
||||
|
||||
### 4.2 Milestone 1 - export to the share (engine unchanged)
|
||||
|
||||
Today the enforcement engine still reads manifests from the SFLD share. The
|
||||
plugin's **Export to Share** button writes the current published manifest to
|
||||
`<shareroot>/<scope>/manifest.json` (backing up the old file to `_meta/history`
|
||||
first). So the workflow is:
|
||||
|
||||
**author + publish in shopdb -> Export to Share -> the unchanged engine picks it
|
||||
up next cycle.**
|
||||
|
||||
Nothing about the engine, the share layout, or the PCs changes. Rollback is
|
||||
restoring the `_meta/history` backup (or re-publishing an older version and
|
||||
re-exporting). This is the safe first milestone: all the authoring benefit, zero
|
||||
client risk.
|
||||
|
||||
Configure the share root once at the top of the Manifests page.
|
||||
|
||||
### 4.3 Enforcement Reports - fleet compliance (GE-Enforce > Enforcement Reports)
|
||||
|
||||
Each PC reports its enforcement result back to shopdb (see the client kit). The
|
||||
Reports page shows, per PC:
|
||||
|
||||
- **Received** - did the PC apply the latest published version? (applied vs
|
||||
latest). "behind" means it has not picked up your newest publish yet.
|
||||
- **Status** - `ok` (nothing needed), `selfhealed` (drift corrected), `failed`.
|
||||
- **Counts** - installed / skipped / failed, plus per-entry detail (action,
|
||||
self-heal flag, exit code, message) in the row's Detail view.
|
||||
|
||||
This is the observed-state half of the loop: the manifest is what SHOULD be
|
||||
installed; the report is what each PC ACTUALLY did.
|
||||
|
||||
### 4.4 The client side (per PC)
|
||||
|
||||
The engine sources the manifest and reports results using the reference kit in
|
||||
`plugins/geenforce/client/` (`ShopdbEnforceClient.psm1` +
|
||||
`Invoke-ShopdbEnforce.ps1`), configured from `HKLM:\SOFTWARE\GE\ShopDB`
|
||||
(BaseUrl + a `geenforce.fetch`/`geenforce.report` service token). See
|
||||
`docs/GE-ENFORCE-CLIENT.md` for the fetch/report contract, the last-known-good
|
||||
cache, shadow mode, and the staged cutover from share-sourced to shopdb-sourced
|
||||
manifests. Until that cutover, the client only REPORTS; the manifest still comes
|
||||
from the share via Export to Share (4.2).
|
||||
|
||||
---
|
||||
|
||||
## 5. Day-to-day: common tasks
|
||||
|
||||
All in GE-Enforce > Manifests. No PowerShell, no editing JSON on the share.
|
||||
|
||||
- **Add an app to a PC type:** open the PC type, Add Entry, pick the Type (the
|
||||
form adapts), fill the installer + detection + any targeting, place it in order
|
||||
with Up/Down (config restores go BELOW their installer), Preview, Publish, then
|
||||
Export to Share.
|
||||
- **Bump an app version:** drop the new installer in the scope's `apps/` folder
|
||||
on the share, open the entry, update the Installer filename + the Detection
|
||||
value (the new version), Publish, Export to Share. PCs self-heal next cycle.
|
||||
- **Roll back a bad publish:** the PC type's Versions list -> Roll Back to the
|
||||
last good version -> Export to Share.
|
||||
- **Canary a risky change:** add the one test PC under Target hostnames (via
|
||||
"Show all targeting options"), Publish; when happy, remove the filter and
|
||||
Publish again.
|
||||
- **Check "did PC Y get app X":** use Simulate with that PC's type / machine
|
||||
number / CMM version; and check Enforcement Reports for what it actually did.
|
||||
|
||||
---
|
||||
|
||||
## 6. Reference
|
||||
|
||||
- Engine (behavior ground truth): `Install-FromManifest.ps1` (lib >= 2.6).
|
||||
- Dispatcher: `GE-Enforce.ps1` (mount + run common then type scope).
|
||||
- Preinstall runner: `00-PreInstall-*` over `preinstall.json` (imaging-time).
|
||||
- shopdb model + API: `plugins/geenforce/` (models, importer/serializer,
|
||||
filters mirror, service, routes).
|
||||
- Behavioral parity gate (proves the shopdb model round-trips the real
|
||||
manifests): `plugins/geenforce/parity.py` + `flask geenforce parity`.
|
||||
- Client kit + contract: `plugins/geenforce/client/`, `docs/GE-ENFORCE-CLIENT.md`.
|
||||
- Agent deployment (per PC, any imaging path): `docs/GE-ENFORCE-DEPLOY.md` +
|
||||
`plugins/geenforce/client/Install-GEEnforce.ps1`.
|
||||
- Design + cutover plan: `docs/proposals/ge-enforce-plugin.md`.
|
||||
59
Home.md
59
Home.md
@@ -0,0 +1,59 @@
|
||||
# ShopDB Flask wiki
|
||||
|
||||
Mirror of the `docs/` folder in the [shopdb-flask repo](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/docs). The repo copy is
|
||||
canonical; edit there, not here.
|
||||
|
||||
## Install and operate
|
||||
|
||||
- [[INSTALL-WINDOWS-IIS]]
|
||||
- [[DEPLOY]]
|
||||
- [[DEPLOY-WINDOWS-IIS]]
|
||||
- [[PILOT-DEPLOY]]
|
||||
- [[CONFIG]]
|
||||
- [[UPGRADE]]
|
||||
- [[BACKUP-RESTORE]]
|
||||
|
||||
## Data import
|
||||
|
||||
- [[IMPORT-API]]
|
||||
- [[IMPORT-ADOPTION]]
|
||||
|
||||
## Plugins
|
||||
|
||||
- [[PLUGINS]]
|
||||
- [[PLUGIN-QUICKSTART]]
|
||||
- [[PLUGIN-GUIDE]]
|
||||
- [[PLUGIN-HOOKS]]
|
||||
- [[PLUGIN-EXTERNAL-REPO]]
|
||||
- [[CONTRACT-STABILITY]]
|
||||
|
||||
## Integrations
|
||||
|
||||
- [[COLLECTOR-INTEGRATION]]
|
||||
- [[GE-ENFORCE]]
|
||||
- [[GE-ENFORCE-DEPLOY]]
|
||||
- [[GE-ENFORCE-CLIENT]]
|
||||
|
||||
## Project
|
||||
|
||||
- [[ROADMAP]]
|
||||
|
||||
## Architecture decisions
|
||||
|
||||
- [[ADR-001-asset-as-platform-contract]]
|
||||
- [[ADR-002-plugin-versioning]]
|
||||
- [[ADR-003-plugin-distribution]]
|
||||
- [[ADR-004-deployment-topology]]
|
||||
- [[ADR-005-equipment-vs-measuringtools]]
|
||||
- [[ADR-006-collector-contract]]
|
||||
- [[ADR-007-product-versioning-and-releases]]
|
||||
- [[ADR-008-plugin-migration-ownership]]
|
||||
- [[ADR-009-frontend-plugin-gating]]
|
||||
- [[ADR-010-frontend-plugin-hooks]]
|
||||
- [[ADR-011-machines-rename]]
|
||||
- [[ADR-012-geenforce-manifest-ownership]]
|
||||
- [[README]]
|
||||
|
||||
## Proposals
|
||||
|
||||
- [[PROPOSAL-ge-enforce-plugin]]
|
||||
|
||||
71
IMPORT-ADOPTION.md
Normal file
71
IMPORT-ADOPTION.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# Importing a site's legacy data
|
||||
|
||||
Every adopting site has its own source database - it will not match another
|
||||
site's schema. So the import is split in two layers:
|
||||
|
||||
1. **The import API is the stable contract** (`docs/IMPORT-API.md`). Whatever
|
||||
your source looks like, you create flask records through the same documented
|
||||
REST endpoints, authenticated with an admin PAT and the `X-Import-Mode`
|
||||
header (which preserves legacy timestamps). This layer is the product; it is
|
||||
schema-agnostic.
|
||||
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
|
||||
to those endpoints. Nobody runs another site's loader - you copy the pattern.
|
||||
|
||||
The West Jefferson loader in `scripts/site_imports/wjf/` is reference
|
||||
implementation #1. Read it alongside this guide.
|
||||
|
||||
## The shape of a loader
|
||||
|
||||
- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
|
||||
unscoped admin PAT in-process, and drives the real endpoints through the app
|
||||
test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
|
||||
exercises the same routes/authz/validation an HTTP client would, no running
|
||||
server needed. It also holds read-only access to the source DB and a JSON
|
||||
`IdMap` of legacy-id -> new-id crosswalks.
|
||||
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
|
||||
POSTs it, and records the crosswalk later stages resolve foreign keys against.
|
||||
|
||||
### Stage order matters
|
||||
|
||||
Reference/lookup tables first (so foreign keys resolve), then the entity hub,
|
||||
then dependents, then links:
|
||||
|
||||
```
|
||||
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
|
||||
-> dependents (installs, warranties, notifications, ...) -> relationships
|
||||
```
|
||||
|
||||
The **crosswalk is the keystone**: capture every legacy id -> new id as you
|
||||
create rows, and resolve foreign keys through it in later stages. New
|
||||
autoincrement ids will not match the source's.
|
||||
|
||||
## Producing the mapping
|
||||
|
||||
You do not have to hand-derive the source -> target mapping. Point the
|
||||
agent-assisted workflow at a source database plus this API contract and it emits
|
||||
a per-table mapping (source columns -> endpoint fields, transforms, what is
|
||||
importable vs out of scope) and a loader skeleton. That is the repeatable
|
||||
onboarding path.
|
||||
|
||||
## Running (against a THROWAWAY import database)
|
||||
|
||||
1. Build a fresh target: `flask db upgrade` + `flask plugin upgrade-all` +
|
||||
`flask seed permissions/settings/reference-data`. Enable every bundled plugin
|
||||
you need (some ship disabled; a plugin's routes only register when it is
|
||||
enabled at app start).
|
||||
2. Load your source dump into a scratch DB the loader can read.
|
||||
3. Run the loader stages in order, dry-running / spot-checking as you go.
|
||||
4. Verify: row-count + foreign-key-resolution audit against the source, then a
|
||||
UI spot-check (log in, eyeball the lists / map / a detail page).
|
||||
5. Only then point a real instance at the imported database.
|
||||
|
||||
## What the WJ loader demonstrates
|
||||
|
||||
- Fanning one legacy "machine" table out to the flask asset types
|
||||
(computer/machine/network/measuring-tool) by a routing rule, with the
|
||||
duplicate/placeholder/skip decisions applied.
|
||||
- Synthesizing a natural key when the source lacks one (printers -> `PRN-{id}`).
|
||||
- Folding a primary IP onto an asset, pairing a check-in/out event log into
|
||||
checkouts, deduping colliding names, reversing an inverse relationship type.
|
||||
- The handful of narrow gaps the API cannot cover (e.g. no bulk-communications
|
||||
endpoint) handled as documented direct-ORM writes.
|
||||
527
IMPORT-API.md
Normal file
527
IMPORT-API.md
Normal file
@@ -0,0 +1,527 @@
|
||||
# Import API: migrating the classic ASP shopdb through HTTP alone
|
||||
|
||||
This is the operator manual for importing the legacy Classic-ASP shopdb database
|
||||
(`prodscratch` on the dev MySQL container) into shopdb-flask using ONLY the HTTP
|
||||
API. No direct writes to the `shopdb_flask` database are needed or wanted: every
|
||||
row is created through a documented endpoint so authorization, validation,
|
||||
auditing, and plugin hooks all run exactly as they do for a human operator.
|
||||
|
||||
A plain Python script can run the whole migration from this document.
|
||||
|
||||
Contents:
|
||||
|
||||
1. [Prerequisites](#1-prerequisites)
|
||||
2. [Order of operations](#2-order-of-operations)
|
||||
3. [Full table-by-table mapping](#3-full-table-by-table-mapping)
|
||||
4. [Tables with no target yet](#4-tables-with-no-target-yet)
|
||||
5. [Idempotency recipe and a worked importer](#5-idempotency-recipe-and-a-worked-importer)
|
||||
6. [Verification: row-count parity](#6-verification-row-count-parity)
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
### Dev URLs
|
||||
|
||||
- Flask API: `http://localhost:5001`
|
||||
- All import calls target `/api/...` on that host.
|
||||
|
||||
### Admin token
|
||||
|
||||
Every write needs authentication, and import mode additionally needs an admin.
|
||||
|
||||
A large import can outlast a login JWT: `access_token` expires after one hour,
|
||||
so a long run dies mid-import with 401s. Use a **personal API token (PAT)**
|
||||
instead. A PAT never expires (unless you set an expiry), acts as the user that
|
||||
created it, and is sent exactly like a JWT. Create one as an admin (via the
|
||||
Settings > API Tokens page, or the API):
|
||||
|
||||
```bash
|
||||
# Bootstrap: a short login JWT is fine just to mint the long-lived PAT.
|
||||
JWT=$(curl -s http://localhost:5001/api/auth/login \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"username":"<admin>","password":"<password>"}' | jq -r '.data.access_token')
|
||||
|
||||
# The full secret (shopdb_pat_...) is returned ONCE. Save it now.
|
||||
curl -s http://localhost:5001/api/apitokens \
|
||||
-H "Authorization: Bearer $JWT" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"legacy import runner"}' | jq -r '.data.secret'
|
||||
```
|
||||
|
||||
Send the PAT on every request as `Authorization: Bearer shopdb_pat_...`. It
|
||||
authenticates the whole import surface (every create/update/delete plus import
|
||||
mode) as its owning admin, exactly as a login JWT would, but without the hourly
|
||||
expiry. Revoke it from the same Settings page (or `DELETE /api/apitokens/<id>`)
|
||||
when the import is done.
|
||||
|
||||
Use an **unscoped** token for imports. A token may optionally carry a scopes
|
||||
list that limits it to specific permissions; a scoped token suspends the admin
|
||||
bypass and is denied on role-gated endpoints AND on import mode, so it cannot
|
||||
run an import. Leave the "Restrict permissions" option off (the default) so the
|
||||
token acts with the full authority of its admin owner. Minting a token itself
|
||||
requires the `apitokens.create` permission (admins have it by default).
|
||||
|
||||
A short-lived login JWT still works for quick one-off calls if you prefer.
|
||||
|
||||
### Import mode: the `X-Import-Mode` header
|
||||
|
||||
By default the server stamps `createddate`/`modifieddate` to "now" on every
|
||||
create and update, which would erase a migrated row's real history. To preserve
|
||||
it, send the request header:
|
||||
|
||||
```
|
||||
X-Import-Mode: true
|
||||
```
|
||||
|
||||
When (and only when) the caller is an admin AND that header is present:
|
||||
|
||||
- create/update endpoints on timestamped entities accept optional
|
||||
`createddate` and `modifieddate` fields in the JSON body and store them
|
||||
verbatim (naive UTC). Both `2020-01-05T12:00:00` (ISO) and the legacy
|
||||
`2020-01-05 12:00:00` (MySQL) forms are parsed. A bare `2020-01-05` works too.
|
||||
- the selfhosted USB checkout/checkin endpoints accept optional `checkouttime`
|
||||
and `checkintime` overrides so historical events keep their real timestamps.
|
||||
|
||||
Without the header, or for a non-admin caller, those fields are silently ignored
|
||||
and the server behaves exactly as it does normally. This is enforced centrally
|
||||
by `shopdb/utils/import_mode.py` (`import_mode_active`, `apply_import_timestamps`,
|
||||
`parse_import_datetime`), exposed on the plugin contract surface `shopdb.api`.
|
||||
|
||||
Timestamped entities that honor `createddate`/`modifieddate`: assets (all five
|
||||
type plugins), vendors, models, modeltypes, businessunits, locations, operating
|
||||
systems, applications, knowledge base, USB devices, asset relationships.
|
||||
|
||||
Entities that carry history in domain fields instead (createddate passthrough is
|
||||
a no-op there, by design): notifications (`starttime`/`endtime`), warranties
|
||||
(`startdate`/`enddate`/`lastcheckeddate`). Set those fields directly in the
|
||||
payload; they are already accepted.
|
||||
|
||||
### Reference data seed
|
||||
|
||||
Before importing, seed the reference tables that have no CRUD endpoint of their
|
||||
own (communication types such as IP/Serial/USB, default statuses, canonical
|
||||
relationship types, permissions, settings):
|
||||
|
||||
```bash
|
||||
flask seed permissions
|
||||
flask seed settings
|
||||
flask seed reference-data
|
||||
```
|
||||
|
||||
`communicationtypes` (the target of legacy `comstypes`) is populated here, so
|
||||
the primary-IP mapping below can resolve `comtype='IP'`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Order of operations
|
||||
|
||||
Import in dependency order so foreign keys always resolve. Each step is a
|
||||
lookup-then-upsert loop (see section 5); rerunning any step is safe.
|
||||
|
||||
1. **Reference / lookup data first**
|
||||
1. Vendors (`vendors`)
|
||||
2. Model types (`modeltypes`) - from legacy `machinetypes`
|
||||
3. Models (`models`) - needs vendors + model types
|
||||
4. Business units (`businessunits`)
|
||||
5. Location types, then Locations (`locations/types`, `locations`) - the
|
||||
legacy LocationOnly machines land here, not as assets
|
||||
6. Operating systems (`operatingsystems`)
|
||||
7. Asset statuses (`assets/statuses`) - from legacy `machinestatus`
|
||||
8. Relationship types (`assets/relationshiptypes`) - from legacy
|
||||
`relationshiptypes`
|
||||
9. Notification types (`notifications/types`)
|
||||
10. Per-plugin subtypes: computer types (from `pctype`), machine types,
|
||||
printer types, network device types, measuring-tool types
|
||||
11. Support teams + support-team contacts (`supportteams`,
|
||||
`supportteams/{id}/contacts` - see section 3.3); import these BEFORE
|
||||
applications because `applications.supportteamid` points at them
|
||||
12. Applications (`applications`) and their versions; import legacy `topics`
|
||||
as applications too (KB links to applications, section 3)
|
||||
2. **Assets, per type** (each creates the core Asset row plus its extension):
|
||||
computers, machines, printers, network devices, measuring tools, and USB
|
||||
devices. Fan out the legacy `machines` table by category (section 3).
|
||||
3. **Communications**: the primary IP is set through the asset payload's
|
||||
`ipaddress` field during step 2. There is no bulk-communications endpoint;
|
||||
see the mapping note.
|
||||
4. **Relationships** (`assets/relationships`): needs both endpoint assets and
|
||||
the relationship types to already exist.
|
||||
5. **Installed applications**: attach apps to computers
|
||||
(`computers/{id}/apps`), needs computers + applications.
|
||||
6. **Knowledge base, notifications, warranties, USB checkouts** (including
|
||||
backdated history).
|
||||
7. **Custom fields**: for any legacy column with no home in the target schema,
|
||||
define a custom field for the asset type and store the value per asset.
|
||||
|
||||
---
|
||||
|
||||
## 3. Full table-by-table mapping
|
||||
|
||||
Legend: `->` maps to. Endpoints are relative to `http://localhost:5001`. "NK"
|
||||
is the natural key used for the idempotent lookup (section 5).
|
||||
|
||||
### 3.1 The `machines` hub fans out into the asset plugins
|
||||
|
||||
`machines` (885 rows) is the central legacy asset table. Two columns drive the
|
||||
fan-out: `machinetypeid` (what the asset physically is) and `pctypeid` (a
|
||||
computer's sub-type). Route each row by `machinetypeid`:
|
||||
|
||||
| legacy `machinetypeid` | `machinetypes.machinetype` | target plugin | subtype source |
|
||||
|---|---|---|---|
|
||||
| 1 | LocationOnly (also `islocationonly=1`) | core **Locations** (NOT an asset) | `locationtype` |
|
||||
| 33 | PC | **computers** | `computertype` <- `pctype.typename` via `machines.pctypeid` |
|
||||
| 20 | Server | **computers** | computer type "Server" |
|
||||
| 15 | Printer | **printers** | `printertype` |
|
||||
| 16 Access Point / 17 IDF / 18 Camera / 19 Switch / 46 Firewall | | **network** | `networkdevicetype` |
|
||||
| 44 | USB Device | **usb** | (usb device) |
|
||||
| 23 Measuring Machine / 3 CMM / 48 Spline Checker / 8 Eddy Current / 47 Inspection | | **measuringtools** (gage-lab judgment call; ADR-005) | `measuringtooltype` |
|
||||
| 2,4,5,6,7,9,10,11,12,13,14,21,22,24,25,45 (lathes, mills, welders, grinders, ...) | | **machines** | `machinetype` |
|
||||
|
||||
This mapping is a recommended default, not a hard rule; a site may re-route a
|
||||
`machinetypeid` (for example send CMM to `machines` rather than
|
||||
`measuringtools`). Decide the routing table once, up front.
|
||||
|
||||
Common `machines` columns -> core Asset fields (same for every target plugin):
|
||||
|
||||
| legacy column | target field | notes |
|
||||
|---|---|---|
|
||||
| `machinenumber` | `assetnumber` | the business identifier / NK |
|
||||
| `alias` or `hostname` | `name` | layperson label |
|
||||
| `serialnumber` | `serialnumber` | |
|
||||
| `machinestatusid` | `statusid` | remap via `machinestatus` -> asset statuses |
|
||||
| `businessunitid` | `businessunitid` | remap via imported business units |
|
||||
| `mapleft` | `mapx` | |
|
||||
| `maptop` | `mapy` | |
|
||||
| `machinenotes` | `notes` | |
|
||||
| `dateadded` | `createddate` | import mode only |
|
||||
| `lastupdated` | `modifieddate` | import mode only |
|
||||
|
||||
Per-plugin extension fields:
|
||||
|
||||
- **computers** (`POST /api/computers`): `hostname` <- `machines.hostname`,
|
||||
`osid` <- remapped `machines.osid`, `computertypeid` <- computer type from
|
||||
`pctype`, `loggedinuser`, `lastboottime`, `vendorid`, `modelnumberid`,
|
||||
`ipaddress` <- `machines.ipaddress1` (primary IP). NK: `assetnumber`.
|
||||
- **machines** (`POST /api/machines`): `machinetypeid`, `vendorid`,
|
||||
`modelnumberid`, `controllervendorid`/`controllermodelid` (from
|
||||
`controllertypes` remapped to vendors/models), `requiresmanualconfig` <-
|
||||
`requires_manual_machine_config`, `islocationonly`. NK: `assetnumber`.
|
||||
- **printers** (`POST /api/printers`): see 3.2 (authoritative source is the
|
||||
legacy `printers` table).
|
||||
- **network** (`POST /api/network`): `networkdevicetypeid`, `hostname`,
|
||||
`vendorid`, `ipaddress` <- `machines.ipaddress1`. NK: `assetnumber`.
|
||||
- **measuringtools** (`POST /api/measuringtools`): `measuringtooltypeid`,
|
||||
calibration fields where known. NK: `assetnumber`.
|
||||
|
||||
### 3.2 Reference and lookup tables
|
||||
|
||||
| legacy table | target endpoint | field mapping | NK |
|
||||
|---|---|---|---|
|
||||
| `vendors` | `POST /api/vendors` | `vendor` -> `vendor` | `vendor` |
|
||||
| `machinetypes` | `POST /api/modeltypes` | `machinetype` -> `modeltype`; set `category` (Equipment/Computer/...) | `modeltype` |
|
||||
| `models` | `POST /api/models` | `modelnumber`, `vendorid` (remapped), `machinetypeid` -> `modeltypeid`, `notes`, `image` -> `imageurl`, `documentationpath` -> `documentationurl` | `modelnumber` + `vendor` |
|
||||
|
||||
`imageurl` imports as a plain URL string (an external URL or a legacy
|
||||
`/images/models/*` path). Binary photos are not part of the import payload;
|
||||
upload them after import via `POST /api/models/<modelid>/image` (multipart
|
||||
`file`), which stores the file under `instance/modelimages/` and rewrites
|
||||
`imageurl` to the served URL.
|
||||
| `businessunits` | `POST /api/businessunits` | `businessunit` -> `businessunit` | `businessunit` |
|
||||
| `operatingsystems` | `POST /api/operatingsystems` | `operatingsystem` -> `osname` | `osname` (+`osversion`) |
|
||||
| `machinestatus` | `POST /api/assets/statuses` | `machinestatus` -> `status` | `status` |
|
||||
| `relationshiptypes` | `POST /api/assets/relationshiptypes` | `relationshiptype` -> `relationshiptype`, `description`, `isdirectional` (bool, default true; false = symmetric connection) | `relationshiptype` |
|
||||
| `notificationtypes` | `POST /api/notifications/types` | `typename`, `typedescription`, `typecolor` | `typename` |
|
||||
| `pctype` | `POST /api/computers/types` | `typename` -> `computertype`, `description` | `computertype` |
|
||||
| `subnettypes` | (see subnets) | used as `subnettype` string on subnets | - |
|
||||
| `subnets` | `POST /api/network/subnets` | `cidr`, `description` -> `name`/`description`, `vlan` -> create VLAN first (`POST /api/network/vlans`) then `vlanid`, `subnettypeid` -> `subnettype` name | `cidr` |
|
||||
| `dashboarddefaults` | `POST /api/dashboarddefaults` | `ipaddress` -> `ipaddress`, `businessunitid` (remapped), `description` | `ipaddress` |
|
||||
| `controllertypes` | remap into `vendors` + `models` | e.g. "Fanuc" -> a Vendor; the controller model -> a Model; then set `controllervendorid`/`controllermodelid` on the machine | - |
|
||||
| `comstypes` | `communicationtypes` (seeded, no API) | ensure `flask seed reference-data` created IP/Serial/USB/... before importing comms | - |
|
||||
|
||||
Note on communication types: the classic `comstypes.typename` values
|
||||
(IP, Serial, Network_Interface, USB, Parallel, VNC, FTP, DNC) correspond to the
|
||||
seeded `communicationtypes.comtype`. They are created by the reference-data seed,
|
||||
not imported per-row.
|
||||
|
||||
### 3.3 Support teams, applications, topics, installed apps
|
||||
|
||||
Support teams and their contacts import BEFORE applications, because
|
||||
`applications.supportteamid` references a team. The legacy `appowners` table
|
||||
is folded into contacts: each legacy `supportteams` row carries one
|
||||
`appownerid`, so import that owner as ONE contact on the team (legacy
|
||||
`appowner` -> `name`, `sso` -> `sso`).
|
||||
|
||||
| legacy table | target endpoint | field mapping | NK |
|
||||
|---|---|---|---|
|
||||
| `supportteams` | `POST /api/supportteams` | `teamname`, `teamurl` (ServiceNow group deep link) | `teamname` |
|
||||
| `appowners` (via each team's `appownerid`) | `POST /api/supportteams/{supportteamid}/contacts` | `appowner` -> `name`, `sso` -> `sso`, `sortorder` (default 0) | (supportteamid, name) |
|
||||
| `applications` | `POST /api/applications` | `appname`, `appdescription`, `supportteamid` (remap by team `teamname`, GET `/api/supportteams?teamname=...`), `isinstallable`, `applicationnotes`, `installpath`, `applicationlink`, `documentationpath`, `ishidden`, `isprinter`, `islicenced`, `image` | `appname` |
|
||||
| `appversions` | `POST /api/applications/{appid}/versions` | `version`, `releasedate`, `notes` | `version` (per app) |
|
||||
| `topics` | `POST /api/applications` | `topics` is a near-clone of `applications` and `knowledgebase.appid` points at it; import each distinct topic as an Application (`appname` = topic name), so KB links resolve against `applications` | `appname` |
|
||||
| `installedapps` | `POST /api/computers/{computerid}/apps` | body `{appid, appversionid}`; resolve `machineid` -> the imported computer, `appid`/`appversionid` -> imported app + version | (computerid, appid) |
|
||||
|
||||
`installedapps` only makes sense for computer-class assets; skip rows whose
|
||||
`machineid` did not map to a computer.
|
||||
|
||||
### 3.4 Communications
|
||||
|
||||
| legacy table | target | field mapping | notes |
|
||||
|---|---|---|---|
|
||||
| `communications` (comstypeid=1, isprimary) | asset `ipaddress` on create/update | `address` -> `ipaddress` | Sets the primary IP communication for the asset. |
|
||||
| `communications` (other comstypeids / secondary rows) | none yet | | No bulk-communication create endpoint exists. Import the primary IP only; capture extra interfaces as custom fields, or defer. |
|
||||
|
||||
### 3.5 Relationships
|
||||
|
||||
| legacy table | target endpoint | field mapping |
|
||||
|---|---|---|
|
||||
| `machinerelationships` | `POST /api/assets/relationships` | `machineid` -> `sourceassetid` (the imported asset id), `related_machineid` -> `targetassetid`, `relationshiptypeid` -> `relationshiptypeid` remapped by name, `relationship_notes` -> `notes` |
|
||||
|
||||
Suggested legacy-name -> target relationship-type mapping (create these types
|
||||
first, or map onto the canonical `partof`/`controls`/`connectedto`):
|
||||
|
||||
| legacy `relationshiptype` | recommended target |
|
||||
|---|---|
|
||||
| Controls | Controls |
|
||||
| Controlled By | Controls (reverse the source/target) |
|
||||
| Dualpath | Dualpath (or `connectedto` with label "dualpath") |
|
||||
| Cluster Member | partof |
|
||||
| Backup For | Backup For |
|
||||
| Master-Slave | Controls |
|
||||
| Contains | partof |
|
||||
| Stored At | Stored At |
|
||||
| Connected To | connectedto |
|
||||
|
||||
Resolve each machine id to the asset id you got back when you created that
|
||||
asset (keep a `legacy_machineid -> assetid` map as you import).
|
||||
|
||||
### 3.6 Knowledge base, notifications, warranties, USB
|
||||
|
||||
| legacy table | target endpoint | field mapping | NK |
|
||||
|---|---|---|---|
|
||||
| `knowledgebase` | `POST /api/knowledgebase` | `shortdescription`, `linkurl`, `keywords`, `appid` (remapped to the imported application/topic); `lastupdated` -> `modifieddate` in import mode | `linkurl` (fallback `shortdescription`) |
|
||||
| `notifications` | `POST /api/notifications` | `notification`, `notificationtypeid` (remapped), `businessunitid` (remapped), `starttime`, `endtime`, `ticketnumber`, `link`, `isshopfloor`, `employeesso`; note legacy `endtime` sentinel `2099-00-03 09:52:32` is invalid - drop or clamp it | `ticketnumber` when set, else append-only |
|
||||
| `warranties` | `POST /api/warranty` | `warrantyname`/`servicelevel` -> `servicelevel`, `enddate` -> `enddate`, link the covered asset via `assetids: [assetid]`; set `vendor` (required) from the source or "Dell"; `servicetag` if known | `servicetag` + `vendor` |
|
||||
| `usbcheckouts` | see below | historical checkout/checkin events | - |
|
||||
|
||||
USB devices and their history:
|
||||
|
||||
1. Create each USB device (legacy `machines` rows with `machinetypeid=44`, or a
|
||||
dedicated device list) via `POST /api/usb` in selfhosted mode with body
|
||||
`{device_id: <serial>, device_desc, locker_location}`. NK: `device_id`.
|
||||
2. Replay each `usbcheckouts` row as a checkout then (if returned) a checkin,
|
||||
with import-mode backdating:
|
||||
- `POST /api/usb/{device_id}/checkout` body
|
||||
`{badge: <sso>, reason: <checkout_reason>, checkouttime: <checkout_time>}`
|
||||
- if `checkin_time` is set:
|
||||
`POST /api/usb/{device_id}/checkin` body
|
||||
`{badge: <sso>, sanitized: <was_wiped>, notes: <checkin_notes>, checkintime: <checkin_time>}`
|
||||
|
||||
The `checkouttime`/`checkintime` overrides are honored only in import mode.
|
||||
|
||||
Employee directory (people): only self-hosted mode (`employee_directory_mode =
|
||||
selfhosted`) owns people in this app; import them via the directory bulk-upsert
|
||||
`POST /api/employees/directory/import` (CSV headers `SSO,First_Name,Last_Name,
|
||||
Team,Role,Picture`) or per-person `POST /api/employees/directory`. Photos:
|
||||
|
||||
- External mode: the photo is a URL/relative path supplied by the HR database
|
||||
(`Picture` column); it is a read-only pass-through and cannot be uploaded here.
|
||||
- Self-hosted mode: the `Picture` CSV field is a legacy text label and does not
|
||||
drive the displayed photo. Upload the real photo after import via
|
||||
`POST /api/employees/<sso>/photo` (multipart `file`, png/jpg/jpeg/gif/webp),
|
||||
which stores it under `instance/employeephotos/` and serves it publicly.
|
||||
|
||||
### 3.7 Anything unmappable -> custom fields
|
||||
|
||||
For a legacy column with no target field (for example `machines.logicmonitorurl`,
|
||||
`machines.fqdn`, `printers.printerpin`), define a custom field on the asset type
|
||||
and store the value per asset:
|
||||
|
||||
- `POST /api/customfields` body `{assettypeid, label, datatype}` (once per field)
|
||||
- `PUT /api/customfields/asset/{assetid}` body `{values: {<fieldid>: <value>}}`
|
||||
|
||||
Custom-field values are not timestamped, so they carry no history.
|
||||
|
||||
---
|
||||
|
||||
## 4. Tables with no target yet
|
||||
|
||||
These legacy tables have no import target in the current schema. The
|
||||
dispositions below are DECIDED, not open questions.
|
||||
|
||||
### DECIDED: not migrated
|
||||
|
||||
- **`dncconfig` and `commconfig`** - intentionally NOT migrated. DNC
|
||||
communication settings drift constantly, so a one-shot import of stale data
|
||||
has little value. The plan is a future DNC feature fed live by the GE-Enforce
|
||||
collector/reporting tool rather than a historical import. When that DNC
|
||||
support is eventually built, the expected ingestion pattern is one attribute
|
||||
at a time across the whole facility (for example, sweep every machine's baud
|
||||
rate in one pass, then ports, and so on) via the GE-Enforce collector, so the
|
||||
future design should favor per-field fleet-wide updates over per-machine
|
||||
full-record imports.
|
||||
|
||||
### DECIDED: skip (structure only or low value)
|
||||
|
||||
- **`compliance`, `compliancescans`** - 0 rows in `prodscratch`. No data to
|
||||
migrate; a future compliance plugin would own them. Skip.
|
||||
- **`ednc_installations`, `ednc_logs`** - 0 rows, and they belong to the eDNC
|
||||
tooling rather than the asset catalog. Skip.
|
||||
- **`distributiongroups`** (2 rows) - email distribution lists referenced by
|
||||
`businessunits.distributiongroupid`. No target; skip, or attach as a business
|
||||
unit custom field if a site needs it.
|
||||
- **`functionalaccounts`** (7 rows) - service-account concept referenced by
|
||||
`pctype`/`machinetypes`; no equivalent in the new schema. Skip, or capture as
|
||||
a computer-type custom field.
|
||||
- **`skilllevels`** (2 rows) - orphaned lookup (no FK from `machines`). Skip.
|
||||
|
||||
---
|
||||
|
||||
## 5. Idempotency recipe and a worked importer
|
||||
|
||||
The endpoints are NOT upserts. The idempotent unit is a two-step recipe that
|
||||
composes with import mode:
|
||||
|
||||
1. **Look up** the row by its natural key using the exact-match list filter.
|
||||
2. If found, **PUT** to update it; if not, **POST** to create it.
|
||||
|
||||
Each import-relevant list endpoint has an exact-match filter for its natural key
|
||||
(added for exactly this purpose):
|
||||
|
||||
| entity | lookup |
|
||||
|---|---|
|
||||
| assets (all 5 plugins) | `GET /api/{plugin}?assetnumber=<n>` |
|
||||
| vendors | `GET /api/vendors?vendor=<name>` |
|
||||
| models | `GET /api/models?modelnumber=<m>&vendor=<vendorid>` |
|
||||
| model types | `GET /api/modeltypes?modeltype=<name>` |
|
||||
| business units | `GET /api/businessunits?businessunit=<name>` |
|
||||
| locations | `GET /api/locations?locationname=<name>` |
|
||||
| operating systems | `GET /api/operatingsystems?osname=<name>` |
|
||||
| applications | `GET /api/applications?appname=<name>` |
|
||||
| knowledge base | `GET /api/knowledgebase?linkurl=<url>` |
|
||||
| warranties | `GET /api/warranty?servicetag=<tag>&vendor=<name>` |
|
||||
| notifications | `GET /api/notifications?ticketnumber=<t>` |
|
||||
| USB devices | `GET /api/usb/{device_id}` (exact by id) |
|
||||
|
||||
### Worked example
|
||||
|
||||
A small, dependency-free importer (`requests`) that authenticates with a PAT
|
||||
(so a multi-hour run cannot expire mid-import), does the lookup-then-upsert loop
|
||||
in import mode, supports a `--dry-run` flag, and reports errors without aborting
|
||||
the whole run:
|
||||
|
||||
```python
|
||||
import argparse
|
||||
import os
|
||||
import requests
|
||||
|
||||
BASE = "http://localhost:5001"
|
||||
|
||||
|
||||
class ImportClient:
|
||||
def __init__(self, token=None, dryrun=False):
|
||||
self.session = requests.Session()
|
||||
self.dryrun = dryrun
|
||||
# A personal API token (shopdb_pat_...) does not expire like a login
|
||||
# JWT, so it survives a long import. See section 1 to mint one.
|
||||
token = token or os.environ["SHOPDB_TOKEN"]
|
||||
# X-Import-Mode makes createddate/modifieddate passthrough take effect.
|
||||
self.session.headers.update({
|
||||
"Authorization": f"Bearer {token}",
|
||||
"X-Import-Mode": "true",
|
||||
})
|
||||
|
||||
def lookup(self, path, params):
|
||||
"""Return the first matching row, or None."""
|
||||
resp = self.session.get(f"{BASE}{path}", params=params)
|
||||
resp.raise_for_status()
|
||||
rows = resp.json().get("data") or []
|
||||
return rows[0] if rows else None
|
||||
|
||||
def upsert(self, path, idfield, lookupparams, payload):
|
||||
"""Lookup by natural key; PUT if found, else POST. Returns the row."""
|
||||
existing = self.lookup(path, lookupparams)
|
||||
if self.dryrun:
|
||||
verb = "PUT" if existing else "POST"
|
||||
print(f"[dry-run] {verb} {path} {lookupparams}")
|
||||
return existing or payload
|
||||
if existing:
|
||||
rowid = existing[idfield]
|
||||
resp = self.session.put(f"{BASE}{path}/{rowid}", json=payload)
|
||||
else:
|
||||
resp = self.session.post(f"{BASE}{path}", json=payload)
|
||||
if resp.status_code >= 400:
|
||||
# report and keep going; a single bad row must not abort the run
|
||||
print(f"ERROR {resp.status_code} {path}: {resp.text[:200]}")
|
||||
return None
|
||||
return resp.json()["data"]
|
||||
|
||||
|
||||
def import_vendors(client, legacyrows):
|
||||
for row in legacyrows:
|
||||
client.upsert(
|
||||
"/api/vendors",
|
||||
idfield="vendorid",
|
||||
lookupparams={"vendor": row["vendor"]},
|
||||
payload={
|
||||
"vendor": row["vendor"],
|
||||
# legacy history preserved because X-Import-Mode is set
|
||||
"createddate": row.get("dateadded"),
|
||||
"modifieddate": row.get("lastupdated"),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
# PAT from the SHOPDB_TOKEN env var, or pass --token explicitly.
|
||||
parser.add_argument("--token", default=None)
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
client = ImportClient(args.token, dryrun=args.dry_run)
|
||||
# read legacy rows from prodscratch (read-only) and call the import_* fns
|
||||
# in the order of section 2, keeping a legacy-id -> new-id map as you go.
|
||||
```
|
||||
|
||||
Keep a `legacy_id -> new_id` dictionary for every entity as you import it; you
|
||||
need it to remap foreign keys (a machine's `businessunitid`, a checkout's
|
||||
`machineid`, a relationship's `machineid`/`related_machineid`, and so on).
|
||||
|
||||
---
|
||||
|
||||
## 6. Verification: row-count parity
|
||||
|
||||
After each phase, compare counts. Legacy side (read-only), for example:
|
||||
|
||||
```bash
|
||||
docker exec dev-mysql mysql -uroot -prootpassword prodscratch \
|
||||
-e "SELECT COUNT(*) FROM vendors;"
|
||||
```
|
||||
|
||||
New side, via the API pagination metadata (`meta.pagination.total`):
|
||||
|
||||
```bash
|
||||
curl -s "http://localhost:5001/api/vendors?per_page=1" \
|
||||
-H "Authorization: Bearer <token>" | jq '.meta.pagination.total'
|
||||
```
|
||||
|
||||
Suggested parity checks:
|
||||
|
||||
| entity | legacy count | new count |
|
||||
|---|---|---|
|
||||
| vendors | `SELECT COUNT(*) FROM vendors` | `GET /api/vendors` total |
|
||||
| models | `SELECT COUNT(*) FROM models` | `GET /api/models` total |
|
||||
| business units | `SELECT COUNT(*) FROM businessunits` | `GET /api/businessunits` total |
|
||||
| applications | `SELECT COUNT(*) FROM applications` | `GET /api/applications?showhidden=true` total |
|
||||
| knowledge base | `SELECT COUNT(*) FROM knowledgebase WHERE isactive` | `GET /api/knowledgebase` total |
|
||||
| computers | `SELECT COUNT(*) FROM machines WHERE machinetypeid IN (33,20)` | `GET /api/computers` total |
|
||||
| machines | `SELECT COUNT(*) FROM machines WHERE machinetypeid IN (2,4,5,6,7,9,10,11,12,13,14,21,22,24,25,45)` | `GET /api/machines` total |
|
||||
| printers | `SELECT COUNT(*) FROM printers WHERE isactive` | `GET /api/printers` total |
|
||||
| network devices | `SELECT COUNT(*) FROM machines WHERE machinetypeid IN (16,17,18,19,46)` | `GET /api/network` total |
|
||||
| USB devices | `SELECT COUNT(*) FROM machines WHERE machinetypeid=44` | `GET /api/usb` total |
|
||||
| relationships | `SELECT COUNT(*) FROM machinerelationships WHERE isactive` | per-asset `GET /api/assets/{id}/relationships` |
|
||||
| USB checkouts | `SELECT COUNT(*) FROM usbcheckouts` | `GET /api/usb/checkouts` |
|
||||
|
||||
Exact counts will differ where the fan-out routing table (section 3.1) sends a
|
||||
`machinetypeid` to a different plugin than the example above; adjust the legacy
|
||||
`WHERE` clause to match the routing you chose. Investigate any gap beyond that.
|
||||
232
INSTALL-WINDOWS-IIS.md
Normal file
232
INSTALL-WINDOWS-IIS.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# ShopDB - Windows + IIS install runbook
|
||||
|
||||
A step-by-step, **tested** install for a new site on Windows Server / Windows 11
|
||||
with IIS in front of the Flask app (HttpPlatformHandler -> waitress), backed by
|
||||
MySQL. This runbook was validated end to end on a win11 + IIS + MySQL 5.6 box.
|
||||
|
||||
`APP_ROOT` below = the deploy folder, e.g. `C:\shopdb-flask` (where `wsgi.py`
|
||||
lives). Run PowerShell as Administrator.
|
||||
|
||||
---
|
||||
|
||||
## 0. Prerequisites
|
||||
|
||||
| Need | Notes |
|
||||
| --- | --- |
|
||||
| **Python 3.12** (64-bit) | `python --version` |
|
||||
| **IIS** with **HttpPlatformHandler** | https://www.iis.net/downloads/microsoft/httpplatformhandler (direct MSI: `download.microsoft.com/download/8/1/3/813AC4E6-9203-4F7A-8DD5-F3D54D10C5CD/httpPlatformHandler_amd64.msi`) |
|
||||
| **MySQL 5.7+/8.0** (or 5.6 with the flags in step 1) | reachable from the app host |
|
||||
| URL Rewrite (optional) | only for the real-client-IP rule; skip it and the app still runs |
|
||||
|
||||
The app itself pulls in `waitress` and `tzdata` from `requirements.txt` (step 4).
|
||||
|
||||
---
|
||||
|
||||
## 1. MySQL: flags (5.6 only) + database + user
|
||||
|
||||
On **MySQL 5.6 only**, add to `my.ini`/`my.cnf` under `[mysqld]` and restart MySQL
|
||||
(5.7+/8.0 need none of this):
|
||||
|
||||
```
|
||||
innodb_file_per_table = 1
|
||||
innodb_file_format = Barracuda
|
||||
innodb_large_prefix = 1
|
||||
```
|
||||
|
||||
Without them, `flask db upgrade` fails with **error 1071** ("key too long") - the
|
||||
migrations use `ROW_FORMAT=DYNAMIC`, which needs the 3072-byte prefix these unlock.
|
||||
|
||||
Then create the database (utf8mb4) and an app user:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
CREATE USER 'shopdb'@'%' IDENTIFIED BY 'CHANGE_ME';
|
||||
GRANT ALL PRIVILEGES ON shopdb_flask.* TO 'shopdb'@'%';
|
||||
FLUSH PRIVILEGES;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Deploy the app files
|
||||
|
||||
Copy the release (the repo minus `venv/`, `.git/`, `node_modules/`,
|
||||
`frontend/src/`) to `APP_ROOT`. It must contain `wsgi.py`, `shopdb/`, `plugins/`,
|
||||
`migrations/`, `requirements.txt`, and the pre-built `frontend/dist/`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Virtual env + dependencies
|
||||
|
||||
```powershell
|
||||
cd APP_ROOT
|
||||
python -m venv venv
|
||||
venv\Scripts\python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
This installs Flask, SQLAlchemy, PyMySQL, **waitress** (the WSGI server IIS
|
||||
launches) and **tzdata** (Windows has no IANA tz database; without it the
|
||||
notifications plugin fails with "No time zone found with key America/New_York").
|
||||
|
||||
---
|
||||
|
||||
## 4. Secrets + connection (.env)
|
||||
|
||||
Create `APP_ROOT\.env` (read by `wsgi.py` via `load_dotenv()`). Lock its ACLs to
|
||||
the app-pool identity + admins.
|
||||
|
||||
```
|
||||
FLASK_ENV=production
|
||||
SECRET_KEY=<64+ random chars>
|
||||
JWT_SECRET_KEY=<another 64+ random chars>
|
||||
DATABASE_URL=mysql+pymysql://shopdb:CHANGE_ME@<mysql-host>:3306/shopdb_flask?charset=utf8mb4
|
||||
CORS_ORIGINS=http://<the site's own hostname-or-ip:port>
|
||||
```
|
||||
|
||||
Generate a key: `venv\Scripts\python -c "import secrets;print(secrets.token_urlsafe(64))"`.
|
||||
Production **refuses to boot** if any of `SECRET_KEY`, `JWT_SECRET_KEY`,
|
||||
`DATABASE_URL`, `CORS_ORIGINS` is missing or a dev default.
|
||||
|
||||
---
|
||||
|
||||
## 5. Preflight (catch problems before installing)
|
||||
|
||||
```powershell
|
||||
$env:FLASK_APP="shopdb"
|
||||
venv\Scripts\flask db-utils preflight
|
||||
```
|
||||
|
||||
Checks Python, required env, DB connectivity, and the MySQL 5.6 index flags, and
|
||||
prints exactly what to fix. Fix any **FAIL** before continuing.
|
||||
|
||||
---
|
||||
|
||||
## 6. Schema + data + plugins + admin
|
||||
|
||||
```powershell
|
||||
$env:FLASK_APP="shopdb"
|
||||
|
||||
venv\Scripts\flask db upgrade # creates every table (to head)
|
||||
venv\Scripts\flask seed reference-data # statuses, machine/location/rel types
|
||||
venv\Scripts\flask seed permissions
|
||||
venv\Scripts\flask seed settings
|
||||
|
||||
# enable the plugins this site tracks (registry is empty on a fresh box).
|
||||
# usb + employees install DISABLED by default - enable them later in the wizard
|
||||
# if the site wants those (they create extra tables).
|
||||
foreach ($p in "computers","equipment","network","notifications","printers","knowledgebase","slides","warranty") {
|
||||
venv\Scripts\flask plugin install $p
|
||||
}
|
||||
|
||||
# first admin (password generated + printed once - store it):
|
||||
venv\Scripts\flask seed admin --username admin --email admin@yourfacility.example.com
|
||||
```
|
||||
|
||||
> Prefer no CLI? Skip `seed admin` (and even the seed steps): start the site, and
|
||||
> the login page offers to **create the first admin** on a fresh instance, then
|
||||
> the setup wizard can seed reference data. Either path works.
|
||||
|
||||
---
|
||||
|
||||
## 7. IIS site
|
||||
|
||||
Two supported deployment methods:
|
||||
|
||||
- **Method A - own site (recommended, default):** the app gets its own IIS
|
||||
site, port (or hostname), app pool, and venv. Steps 1-5 below.
|
||||
- **Method B - subpath under an existing site:** the app runs as an IIS
|
||||
**Application** (e.g. `/ops`) under a site you already have (such as the
|
||||
classic ASP site or Default Web Site), so it shares that site's binding and
|
||||
TLS cert: `https://<host>/ops/`. Do steps 1-4 below, then follow **7b**
|
||||
instead of step 5.
|
||||
|
||||
1. Copy `deploy\windows\web.config` to `APP_ROOT\web.config`. If `APP_ROOT` is not
|
||||
`C:\shopdb-flask`, fix the paths inside it. Create `APP_ROOT\logs`.
|
||||
2. Create an app pool with **No Managed Code**:
|
||||
```powershell
|
||||
Import-Module WebAdministration
|
||||
New-WebAppPool -Name shopdbflask
|
||||
Set-ItemProperty IIS:\AppPools\shopdbflask -Name managedRuntimeVersion -Value ""
|
||||
```
|
||||
3. Grant the app-pool identity access:
|
||||
```powershell
|
||||
icacls APP_ROOT /grant "IIS AppPool\shopdbflask:(OI)(CI)RX" /T
|
||||
icacls APP_ROOT\logs /grant "IIS AppPool\shopdbflask:(OI)(CI)M" /T
|
||||
```
|
||||
4. **Unlock the handler sections** (locked server-wide by default; without this
|
||||
IIS returns **HTTP 500.19**):
|
||||
```powershell
|
||||
%windir%\system32\inetsrv\appcmd unlock config /section:system.webServer/handlers
|
||||
%windir%\system32\inetsrv\appcmd unlock config /section:system.webServer/httpPlatform
|
||||
```
|
||||
5. Create the site (own port; the classic ASP site can keep 8080):
|
||||
```powershell
|
||||
New-Website -Name shopdb-flask -Port 8090 -PhysicalPath APP_ROOT -ApplicationPool shopdbflask
|
||||
New-NetFirewallRule -DisplayName "shopdb-flask 8090" -Direction Inbound -Protocol TCP -LocalPort 8090 -Action Allow
|
||||
Start-Website shopdb-flask
|
||||
```
|
||||
|
||||
IIS launches `waitress-serve --port=%HTTP_PLATFORM_PORT% wsgi:app` per the
|
||||
web.config and reverse-proxies the site port to it. First request takes ~15s
|
||||
(the app boots + connects to MySQL).
|
||||
|
||||
### 7b. Method B: subpath under an existing site
|
||||
|
||||
The mount path must match in **three places**: the IIS Application alias, the
|
||||
`MOUNT_PATH` the backend sees, and the `VITE_BASE_PATH` the frontend was built
|
||||
with. `/ops` is the example throughout; any alias works.
|
||||
|
||||
1. Rebuild the frontend for the subpath (on the dev box, then copy `dist`):
|
||||
```bash
|
||||
cd frontend && VITE_BASE_PATH=/ops/ npm run build # note the trailing slash
|
||||
```
|
||||
2. Create the Application under the existing site (instead of `New-Website`):
|
||||
```powershell
|
||||
New-WebApplication -Site "Default Web Site" -Name ops -PhysicalPath APP_ROOT -ApplicationPool shopdbflask
|
||||
```
|
||||
3. Tell the backend its mount path: in `APP_ROOT\web.config`, uncomment the
|
||||
`MOUNT_PATH` environment variable (value `/ops`), or set `MOUNT_PATH=/ops`
|
||||
in `APP_ROOT\.env`. `wsgi.py` then serves everything under the prefix
|
||||
(requests outside it get a plain 404 naming the mount).
|
||||
4. Recycle the app pool. The app is at `http(s)://<host>/ops/` and the API at
|
||||
`/ops/api/...`.
|
||||
|
||||
The handler mappings in the app's web.config apply only inside the
|
||||
Application, so the parent site's own handlers (classic ASP, static files)
|
||||
are untouched. `CORS_ORIGINS` in `.env` is origin-only (scheme + host + port,
|
||||
no path), so it is the same for both methods.
|
||||
|
||||
> The `X-Forwarded-For` URL Rewrite rule in web.config is **commented out by
|
||||
> default**. It needs the URL Rewrite module; with it active but the module
|
||||
> absent, IIS returns 500.19. Install URL Rewrite, then uncomment the
|
||||
> `<rewrite>` block, to record real client IPs in audit logs.
|
||||
|
||||
---
|
||||
|
||||
## 8. Smoke test + first run
|
||||
|
||||
```powershell
|
||||
(Invoke-WebRequest http://localhost:8090/ -UseBasicParsing).StatusCode # 200 (SPA)
|
||||
Invoke-WebRequest http://localhost:8090/api/auth/login -Method POST `
|
||||
-Body '{"username":"admin","password":"<the printed password>"}' `
|
||||
-ContentType application/json -UseBasicParsing # 200 + token
|
||||
```
|
||||
|
||||
Browse to `http://<host>:8090`, sign in as the admin, and the **setup wizard**
|
||||
walks through site name, features (per-plugin: create tables here vs connect a DB),
|
||||
floor-map upload, and starter data. Multiple Flask apps can share one IIS box -
|
||||
each gets its own site, app pool, port, and venv.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
| --- | --- |
|
||||
| `flask db upgrade` -> error **1071** | MySQL 5.6 without the step-1 flags (or server not restarted). |
|
||||
| IIS **500.19** | handler sections not unlocked (step 7.4), or the `<rewrite>` block active without URL Rewrite. |
|
||||
| **500** with an empty HttpPlatform log | app-pool identity can't read `APP_ROOT` / run the venv (step 7.3), or `.env` missing/invalid. |
|
||||
| "No time zone found with key America/New_York" | `tzdata` not installed (`pip install tzdata`). |
|
||||
| Nav missing Equipment/PCs/... | plugins not installed (step 6 `flask plugin install`), or site not recycled. |
|
||||
| Method B: blank page / assets 404 under `/ops` | frontend `dist` built without `VITE_BASE_PATH=/ops/` (step 7b.1). |
|
||||
| Method B: SPA loads but every API call 404s | `MOUNT_PATH` unset or not matching the Application alias (step 7b.3). |
|
||||
| ConfigError on boot | a required `.env` var missing or left at a dev default. |
|
||||
160
PILOT-DEPLOY.md
Normal file
160
PILOT-DEPLOY.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Production pilot runbook (West Jefferson)
|
||||
|
||||
Goal: stand up a real shopdb-flask instance loaded with WJ's classic-ASP data,
|
||||
run it **in parallel** with the classic app for a validation window, then cut
|
||||
over. This runbook adds the legacy-data import + verification + cutover on top of
|
||||
the generic stand-up in [`DEPLOY.md`](DEPLOY). Read that first; this only
|
||||
calls out the pilot-specific steps.
|
||||
|
||||
Related: [`IMPORT-ADOPTION.md`](IMPORT-ADOPTION) (import model),
|
||||
[`IMPORT-API.md`](IMPORT-API) (the contract), [`BACKUP-RESTORE.md`](BACKUP-RESTORE),
|
||||
`scripts/site_imports/wjf/` (the loader).
|
||||
|
||||
---
|
||||
|
||||
## 0. Pre-flight checklist
|
||||
|
||||
- [ ] Host provisioned (Docker + compose, or a VM with Python 3 + MySQL 8).
|
||||
- [ ] Three current classic dumps in hand: `shopdb` (main), `cmmc_usb`,
|
||||
`wjf_employees`. Take fresh dumps at import time - the classic app is live.
|
||||
- [ ] Target MySQL 8, utf8mb4 (charset is contract, ADR-004). Old MySQL <5.7
|
||||
needs `innodb_large_prefix=ON` + Barracuda.
|
||||
- [ ] Decide the pilot URL (e.g. `shopdb-pilot.wjs.geaerospace.net`) - separate
|
||||
from the classic app; do not reuse its hostname yet.
|
||||
- [ ] Confirm the import decisions still hold (see the loader README / the
|
||||
import plan): assetnumber fallback + skip-dups, metrology routing,
|
||||
cmmc-only USB, warranties = Dell, occurrences parked.
|
||||
|
||||
## 1. Stand up the pilot instance
|
||||
|
||||
Follow `DEPLOY.md` steps 1-6 against a NEW empty database (name it clearly, e.g.
|
||||
`shopdb_flask_pilot`):
|
||||
|
||||
```bash
|
||||
flask db upgrade
|
||||
flask plugin upgrade-all # applies every plugin's chain
|
||||
flask seed permissions
|
||||
flask seed settings
|
||||
flask seed reference-data # seeds communicationtypes (IP) + the rest
|
||||
```
|
||||
|
||||
**Enable every bundled plugin the site tracks - including usb**, which ships
|
||||
disabled. A plugin's routes only register when it is enabled at app start, and
|
||||
the importer needs them:
|
||||
|
||||
```bash
|
||||
for p in computers employees machines measuringtools network notifications \
|
||||
printers slides usb warranty knowledgebase geenforce; do
|
||||
flask plugin enable "$p"
|
||||
done
|
||||
```
|
||||
|
||||
Do **not** run the setup wizard yet - the import fills the data the wizard would
|
||||
otherwise ask you to seed.
|
||||
|
||||
## 2. Load the classic data
|
||||
|
||||
The loader (`scripts/site_imports/wjf/`) reads the classic dumps and drives the
|
||||
import API. It is site glue, not product code.
|
||||
|
||||
1. Load the three dumps into scratch source DBs the loader can read (strip the
|
||||
`CREATE DATABASE`/`USE` lines so they land under scratch names, no clobber):
|
||||
|
||||
```bash
|
||||
for pair in "shopdb_src:shopdb_dump.sql" "cmmc_usb_src:cmmc_usb_dump.sql" \
|
||||
"wjf_employees_src:wjf_employees_dump.sql"; do
|
||||
db="${pair%%:*}"; f="${pair##*:}"
|
||||
mysql -h HOST -u root -p -e "CREATE DATABASE $db CHARACTER SET utf8mb4;"
|
||||
sed -E '/^CREATE DATABASE/d; /^USE `/d' "$f" | mysql -h HOST -u root -p "$db"
|
||||
done
|
||||
```
|
||||
|
||||
2. Point the loader at the PILOT database and run all stages:
|
||||
|
||||
```bash
|
||||
DATABASE_URL='mysql+pymysql://USER:PW@HOST:3306/shopdb_flask_pilot?charset=utf8mb4' \
|
||||
venv/bin/python -m scripts.site_imports.wjf.run
|
||||
```
|
||||
|
||||
The 15 stages run in order (reference -> catalog -> assets hub -> locations ->
|
||||
printers -> dependents -> relationships -> subnets -> usb -> verify). It is
|
||||
idempotent - a crashed run resumes from `idmap.json`.
|
||||
|
||||
Expected magnitude (from the WJ dumps used in development - your fresh dumps will
|
||||
differ slightly):
|
||||
|
||||
| entity | count |
|
||||
|---|---|
|
||||
| assets | ~983 (computer ~663, machine ~76, network ~58, measuring-tool ~136, printer ~50) |
|
||||
| locations | ~24 |
|
||||
| employees | ~415 |
|
||||
| installs | ~850 |
|
||||
| primary IPs | ~461 |
|
||||
| warranties | ~464 |
|
||||
| notifications | ~261 |
|
||||
| knowledge base | ~341 |
|
||||
| relationships | ~93 |
|
||||
| subnets | ~37 |
|
||||
| USB devices / events | ~18 / ~232 |
|
||||
|
||||
The `verify` stage prints a source-vs-target row-count audit; the gaps are the
|
||||
documented skips (inactive rows, duplicate machinenumbers, LocationOnly, the
|
||||
9999 placeholder).
|
||||
|
||||
## 3. Verify the import
|
||||
|
||||
- [ ] Read the `verify` stage output - source vs target counts line up modulo
|
||||
the documented skips.
|
||||
- [ ] Create the admin: `flask seed admin --username ... --email ...` (password
|
||||
printed once). Mark setup done so the app does not force the wizard:
|
||||
set `setup_complete=true` in settings (or click through the wizard,
|
||||
skipping the seed steps).
|
||||
- [ ] UI spot-check (log in): Computers list paginates the full fleet; the Shop
|
||||
Floor Map plots assets, color-coded by type (positions came from
|
||||
mapleft/maptop); open a PC detail (installs), a printer (IP + share), an
|
||||
application (installed-on list), a KB article; check the employee
|
||||
directory; check a couple of asset relationships.
|
||||
- [ ] Branding: upload the site logo + floor-plan blueprint under Settings, set
|
||||
facility name (Settings drive these per `CONFIG.md`).
|
||||
- [ ] Photos are deferred - employees show initials until a photo batch is run.
|
||||
|
||||
## 4. Parallel-run window
|
||||
|
||||
- Keep the classic app authoritative during the window. The pilot is read-mostly
|
||||
for validation; do not dual-write.
|
||||
- Have a few real users (IT + a floor lead) work the pilot and log gaps.
|
||||
- Re-import is cheap: fix a loader mapping, drop + rebuild the pilot DB, re-run.
|
||||
Nothing you do to the pilot touches classic.
|
||||
- Point the **collector** (GE-Enforce fleet ingest) at the pilot in parallel to
|
||||
confirm live PC check-ins land (see `COLLECTOR-INTEGRATION.md`), using a
|
||||
scoped service token.
|
||||
|
||||
## 5. Cutover
|
||||
|
||||
When the window is clean:
|
||||
|
||||
1. Freeze classic writes (announce a short read-only window).
|
||||
2. Take final fresh dumps; re-run the loader into a clean pilot DB so the
|
||||
cutover data is current.
|
||||
3. Verify counts + a fast UI spot-check.
|
||||
4. Repoint the production hostname/DNS (or the reverse proxy) at the pilot.
|
||||
5. Retire the classic app to read-only standby (do not delete - keep it as the
|
||||
rollback for the agreed period).
|
||||
|
||||
## 6. Rollback
|
||||
|
||||
- Pre-cutover: trivially point back at classic (it never stopped being
|
||||
authoritative).
|
||||
- Post-cutover, within the standby window: repoint DNS/proxy back at classic;
|
||||
investigate; re-cut when fixed. Because the loader is deterministic and the
|
||||
classic DB is untouched, a re-run reproduces the flask DB exactly.
|
||||
|
||||
## 7. Post-cutover
|
||||
|
||||
- [ ] Backups on a schedule (`BACKUP-RESTORE.md`) - mysqldump + the `instance/`
|
||||
dir (uploaded logos, floor plans, tokens).
|
||||
- [ ] Run the employee-photo batch.
|
||||
- [ ] GE-Enforce: publish manifests + cut the fleet over to the flask endpoints
|
||||
when ready (`GE-ENFORCE-DEPLOY.md`) - independent of this pilot.
|
||||
- [ ] Schedule the deferred data (occurrences, full communications fidelity)
|
||||
only if a real need appears.
|
||||
341
PLUGIN-EXTERNAL-REPO.md
Normal file
341
PLUGIN-EXTERNAL-REPO.md
Normal file
@@ -0,0 +1,341 @@
|
||||
# Developing a plugin in its own repo
|
||||
|
||||
This guide is for a sister GE Aerospace site (or any third party) building a
|
||||
shopdb-flask plugin in a git repo it owns, outside the framework tree. It is
|
||||
the "external plugin" path in [ADR-003](ADR-003-plugin-distribution):
|
||||
the framework ships a bundled set, and you drop your own plugin into
|
||||
`<framework>/plugins/<name>/` by clone, submodule, or symlink. No pip packaging
|
||||
is required for v1 (pip distribution is deferred to v2 per ADR-003).
|
||||
|
||||
If you have not written a plugin before, start with
|
||||
[PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART) and the hook reference in
|
||||
[PLUGIN-HOOKS.md](PLUGIN-HOOKS). This document only covers the parts that are
|
||||
different when the plugin lives in its own repo.
|
||||
|
||||
## Recommended repo layout
|
||||
|
||||
Your repo root holds the plugin directory contents directly, so the whole repo
|
||||
can be dropped in at `<framework>/plugins/<name>/`. A plugin named `shipping`
|
||||
in a repo named `wjsf-shipping` looks like this:
|
||||
|
||||
```
|
||||
wjsf-shipping/ # your git repo root == the plugin directory
|
||||
manifest.json # required: name, version, description, core_version
|
||||
plugin.py # required: your BasePlugin subclass
|
||||
__init__.py
|
||||
models/
|
||||
__init__.py # exports every model
|
||||
shipping.py
|
||||
api/
|
||||
__init__.py
|
||||
routes.py # Flask Blueprint returned by get_blueprint()
|
||||
migrations/ # per-plugin Alembic chain (ADR-008), if you own tables
|
||||
env.py
|
||||
script.py.mako
|
||||
versions/
|
||||
0001_shipping_baseline.py
|
||||
tests/ # your own tests (the CI harness runs these)
|
||||
test_shipping.py
|
||||
README.md # what it tracks, who maintains it, where to file issues
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `manifest.json` is the single source of truth for metadata (ADR-002). The
|
||||
`name` field must match the directory name the site installs it under and
|
||||
follows the framework naming convention (lowercase concatenated, no
|
||||
underscores or dashes). Prefix site-specific names with the site code when a
|
||||
collision across sites is possible, e.g. `wjsf-shipping`
|
||||
(see [PLUGINS.md](PLUGINS) naming policy and ADR-003).
|
||||
- `migrations/` is only needed if your plugin owns tables. A plugin built
|
||||
outside the tree never had its tables created by the framework's core chain,
|
||||
so its `0001` is a REAL baseline that creates them, not a stamp-only anchor.
|
||||
This is the same "baseline vs anchor" distinction ADR-008 draws for
|
||||
post-cutover plugins; see [ADR-008](ADR-008-plugin-migration-ownership).
|
||||
- Import core code ONLY through `shopdb.api` (plus `shopdb.plugins.base` for the
|
||||
ABC). Deep imports of `shopdb.core.*`, `shopdb.extensions`, or `shopdb.utils.*`
|
||||
are contract violations. See PLUGIN-HOOKS.md for the exposed surface.
|
||||
|
||||
## Local dev workflow
|
||||
|
||||
Symlinking lets you edit in your own repo while the framework loads the plugin
|
||||
live. The loader discovers a symlinked directory the same as a real one.
|
||||
|
||||
```bash
|
||||
# 1. Clone the framework and your plugin repo side by side.
|
||||
git clone https://gitea.proudtech.net/ge-aerospace/shopdb-flask.git
|
||||
git clone https://gitea.proudtech.net/wjsf/wjsf-shipping.git
|
||||
|
||||
# 2. Symlink your repo into the framework's plugins/ directory.
|
||||
# The link name is the plugin name from your manifest.json.
|
||||
cd shopdb-flask
|
||||
ln -s ../../wjsf-shipping plugins/shipping
|
||||
# (use an absolute path if you prefer: ln -s "$(pwd)/../wjsf-shipping" plugins/shipping)
|
||||
|
||||
# 3. Set up the framework as usual.
|
||||
python3 -m venv venv
|
||||
venv/bin/pip install -r requirements.txt
|
||||
|
||||
# 4. Install (enable) your plugin.
|
||||
venv/bin/flask plugin install shipping
|
||||
|
||||
# 5. If your plugin owns tables, apply its migrations.
|
||||
venv/bin/flask db upgrade # core chain
|
||||
venv/bin/flask plugin upgrade-all # plugin chains, including yours
|
||||
|
||||
# 6. Run tests.
|
||||
venv/bin/python -m pytest tests/test_plugin_contract.py
|
||||
venv/bin/python -m pytest ../wjsf-shipping/tests
|
||||
```
|
||||
|
||||
Edits in `../wjsf-shipping` are picked up on the next framework restart, because
|
||||
the symlink points back at your working tree.
|
||||
|
||||
## core_version pinning for sister sites
|
||||
|
||||
The framework is pre-1.0. Under semver, any 0.x minor bump is allowed to break
|
||||
the contract, and this project uses that latitude (see the history in
|
||||
[CONTRACT-STABILITY.md](CONTRACT-STABILITY)). So pin a TIGHT range that
|
||||
admits only the contract minor you tested against, not the whole 0.x line.
|
||||
|
||||
The current contract version is declared in `shopdb/__init__.py`:
|
||||
|
||||
```python
|
||||
__contract_version__ = '0.6.0'
|
||||
```
|
||||
|
||||
Recommended pin in your `manifest.json`, per ADR-002 (pip-style `>=,<` ranges):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "shipping",
|
||||
"version": "1.0.0",
|
||||
"description": "Tracks shipping-station scanners and label printers",
|
||||
"core_version": ">=0.6.0,<0.7.0",
|
||||
"dependencies": []
|
||||
}
|
||||
```
|
||||
|
||||
Do NOT pin the loose `>=0.2.0,<1.0.0` default that `PluginMeta` falls back to.
|
||||
That default exists so bundled plugins keep loading across minor bumps; an
|
||||
external plugin should be deliberate and re-test before widening its range.
|
||||
|
||||
What happens at load time on a mismatch (ADR-002):
|
||||
|
||||
| Environment | Behavior on core_version mismatch |
|
||||
|-------------|-----------------------------------|
|
||||
| dev / test | The loader re-raises. Startup fails loud so you notice immediately. |
|
||||
| production | The loader logs an error, marks the plugin incompatible, and excludes it from registration. The rest of the app still starts. |
|
||||
|
||||
When you move a site to a newer framework, bump your `core_version` upper bound
|
||||
only after the harness below passes against the new ref.
|
||||
|
||||
## CI recipe
|
||||
|
||||
`scripts/test-external-plugin.sh` (in the framework repo) stands up a throwaway
|
||||
framework at a pinned ref, drops your plugin in as a symlink, and runs the
|
||||
framework contract tests plus your own `tests/`. It has two modes:
|
||||
|
||||
- **CI / remote** (default): clones the framework at `FRAMEWORK_REF` and builds
|
||||
a fresh venv. Needs network access to git and PyPI.
|
||||
- **Local / offline**: set `LOCAL_FRAMEWORK` to a framework checkout on disk.
|
||||
The script exports that checkout at HEAD with `git archive` and reuses its
|
||||
existing venv, so it runs with no internet. Useful for air-gapped verification.
|
||||
|
||||
```bash
|
||||
# CI: test the plugin in the current repo against a pinned tag
|
||||
PLUGIN_DIR=. FRAMEWORK_REF=v0.5.0 scripts/test-external-plugin.sh
|
||||
|
||||
# Offline: test against a framework checkout already on disk
|
||||
LOCAL_FRAMEWORK=/opt/shopdb-flask PLUGIN_DIR=. scripts/test-external-plugin.sh
|
||||
```
|
||||
|
||||
The full script:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# test-external-plugin.sh
|
||||
#
|
||||
# Verify an out-of-tree shopdb-flask plugin against a pinned framework build.
|
||||
# Stand up a throwaway copy of the framework, drop the plugin into
|
||||
# plugins/<name>/ as a symlink (the mechanism ADR-003 documents for sister
|
||||
# sites), and run the framework contract tests plus the plugin's own tests.
|
||||
# Nonzero exit means the plugin is not compatible with that framework ref.
|
||||
#
|
||||
# Two modes:
|
||||
#
|
||||
# CI / remote (default): clone the framework at FRAMEWORK_REF from
|
||||
# FRAMEWORK_URL, build a fresh venv, pip install requirements. Needs
|
||||
# network access to git and PyPI.
|
||||
#
|
||||
# Local / offline: set LOCAL_FRAMEWORK to a framework checkout on disk.
|
||||
# The script exports that checkout at HEAD with `git archive` (no network)
|
||||
# and reuses the checkout's existing venv, so it runs with no internet.
|
||||
#
|
||||
# Inputs (env var, or positional):
|
||||
# PLUGIN_DIR ($1) path to the plugin directory (holds manifest.json). Required.
|
||||
# FRAMEWORK_REF ($2) git ref to test against in CI mode. Default: main.
|
||||
# FRAMEWORK_URL framework git URL for CI mode.
|
||||
# Default: https://gitea.proudtech.net/ge-aerospace/shopdb-flask.git
|
||||
# LOCAL_FRAMEWORK path to an existing framework checkout. Set it to run offline.
|
||||
|
||||
set -eu
|
||||
|
||||
PLUGIN_DIR="${PLUGIN_DIR:-${1:-}}"
|
||||
FRAMEWORK_REF="${FRAMEWORK_REF:-${2:-main}}"
|
||||
FRAMEWORK_URL="${FRAMEWORK_URL:-https://gitea.proudtech.net/ge-aerospace/shopdb-flask.git}"
|
||||
LOCAL_FRAMEWORK="${LOCAL_FRAMEWORK:-}"
|
||||
|
||||
if [ -z "$PLUGIN_DIR" ]; then
|
||||
echo "ERROR: PLUGIN_DIR is required (env var or first argument)." >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ ! -f "$PLUGIN_DIR/manifest.json" ]; then
|
||||
echo "ERROR: $PLUGIN_DIR has no manifest.json - not a plugin directory." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
PLUGIN_ABS="$(cd "$PLUGIN_DIR" && pwd)"
|
||||
PLUGIN_NAME="$(sed -n 's/.*"name"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$PLUGIN_ABS/manifest.json" | head -n1)"
|
||||
if [ -z "$PLUGIN_NAME" ]; then
|
||||
PLUGIN_NAME="$(basename "$PLUGIN_ABS")"
|
||||
fi
|
||||
|
||||
WORKDIR="$(mktemp -d)"
|
||||
cleanup() { rm -rf "$WORKDIR"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
FRAMEWORK="$WORKDIR/framework"
|
||||
mkdir -p "$FRAMEWORK"
|
||||
|
||||
if [ -n "$LOCAL_FRAMEWORK" ]; then
|
||||
echo "==> Local mode: exporting framework from $LOCAL_FRAMEWORK (HEAD)"
|
||||
LOCAL_ABS="$(cd "$LOCAL_FRAMEWORK" && pwd)"
|
||||
git -C "$LOCAL_ABS" archive HEAD | tar -x -C "$FRAMEWORK"
|
||||
if [ -x "$LOCAL_ABS/venv/bin/python" ]; then
|
||||
PYTHON="$LOCAL_ABS/venv/bin/python"
|
||||
elif [ -x "$LOCAL_ABS/.venv/bin/python" ]; then
|
||||
PYTHON="$LOCAL_ABS/.venv/bin/python"
|
||||
else
|
||||
echo "ERROR: no venv found under $LOCAL_ABS (looked for venv/ and .venv/)." >&2
|
||||
exit 2
|
||||
fi
|
||||
else
|
||||
echo "==> CI mode: cloning $FRAMEWORK_URL @ $FRAMEWORK_REF"
|
||||
git clone --depth 1 --branch "$FRAMEWORK_REF" "$FRAMEWORK_URL" "$FRAMEWORK"
|
||||
echo "==> Creating venv and installing requirements"
|
||||
python3 -m venv "$WORKDIR/venv"
|
||||
PYTHON="$WORKDIR/venv/bin/python"
|
||||
"$PYTHON" -m pip install --upgrade pip >/dev/null
|
||||
"$PYTHON" -m pip install -r "$FRAMEWORK/requirements.txt"
|
||||
fi
|
||||
|
||||
echo "==> Linking plugin '$PLUGIN_NAME' into framework plugins/"
|
||||
rm -rf "$FRAMEWORK/plugins/$PLUGIN_NAME"
|
||||
ln -s "$PLUGIN_ABS" "$FRAMEWORK/plugins/$PLUGIN_NAME"
|
||||
|
||||
rm -f "$FRAMEWORK/instance/plugins.json"
|
||||
|
||||
RC=0
|
||||
|
||||
echo "==> Running framework contract tests"
|
||||
( cd "$FRAMEWORK" && "$PYTHON" -m pytest tests/test_plugin_contract.py -q ) || RC=1
|
||||
|
||||
if [ -d "$PLUGIN_ABS/tests" ]; then
|
||||
echo "==> Running plugin's own tests"
|
||||
( cd "$FRAMEWORK" && "$PYTHON" -m pytest "$PLUGIN_ABS/tests" -q ) || RC=1
|
||||
else
|
||||
echo "==> Plugin has no tests/ directory - skipping plugin test step"
|
||||
fi
|
||||
|
||||
if [ "$RC" -eq 0 ]; then
|
||||
echo "==> PASS: plugin '$PLUGIN_NAME' is compatible with framework ref '$FRAMEWORK_REF'"
|
||||
else
|
||||
echo "==> FAIL: plugin '$PLUGIN_NAME' - see output above" >&2
|
||||
fi
|
||||
exit "$RC"
|
||||
```
|
||||
|
||||
### What the harness does and does not check
|
||||
|
||||
The symlinked plugin is discovered and loaded by the plugin loader when the app
|
||||
starts under test. That validates the things that break a real install: the
|
||||
manifest parses, the `core_version` range admits the framework's
|
||||
`__contract_version__` (an out-of-range plugin makes startup fail loud, so the
|
||||
contract test run errors and the script exits nonzero), models expose
|
||||
`__tablename__`, and hooks return the right shapes.
|
||||
|
||||
The import-surface scan (`test_plugins_only_import_contract_surface`) covers
|
||||
symlinked plugins too: the scanner resolves each plugin directory before
|
||||
walking it, because `Path.rglob` alone does not descend symlinks (pinned by
|
||||
`test_import_scan_covers_symlinked_plugins`). You can additionally keep an
|
||||
equivalent import-surface assertion in your own `tests/`, so violations fail
|
||||
in your repo's CI even when run without the framework harness. A minimal
|
||||
version:
|
||||
|
||||
```python
|
||||
# tests/test_import_surface.py
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
ALLOWED = ('shopdb.api', 'shopdb.plugins.base')
|
||||
IMPORT_RE = re.compile(r'^\s*(?:from (shopdb[\w.]*) import|import (shopdb[\w.]*))', re.MULTILINE)
|
||||
|
||||
def test_only_contract_surface_imports():
|
||||
root = Path(__file__).resolve().parent.parent
|
||||
bad = []
|
||||
for path in root.rglob('*.py'):
|
||||
if '__pycache__' in path.parts or 'migrations' in path.parts:
|
||||
continue
|
||||
for m in IMPORT_RE.finditer(path.read_text()):
|
||||
mod = m.group(1) or m.group(2)
|
||||
if mod and not any(mod == a or mod.startswith(a + '.') for a in ALLOWED):
|
||||
bad.append(f'{path.name}: {mod}')
|
||||
assert not bad, 'core imports outside shopdb.api / shopdb.plugins.base: ' + '; '.join(bad)
|
||||
```
|
||||
|
||||
## GitHub Actions example
|
||||
|
||||
For a plugin repo hosted on GitHub, a workflow calling the harness against a
|
||||
pinned framework tag (config only, adjust URLs and ref to your setup):
|
||||
|
||||
```yaml
|
||||
name: plugin-contract
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
jobs:
|
||||
contract:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
FRAMEWORK_REF: v0.5.0
|
||||
FRAMEWORK_URL: https://gitea.proudtech.net/ge-aerospace/shopdb-flask.git
|
||||
steps:
|
||||
- name: Check out the plugin
|
||||
uses: actions/checkout@v4
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- name: Fetch the harness from the framework
|
||||
run: |
|
||||
git clone --depth 1 --branch "$FRAMEWORK_REF" "$FRAMEWORK_URL" /tmp/framework
|
||||
- name: Run the contract harness against this plugin
|
||||
run: |
|
||||
PLUGIN_DIR="$GITHUB_WORKSPACE" \
|
||||
FRAMEWORK_REF="$FRAMEWORK_REF" \
|
||||
FRAMEWORK_URL="$FRAMEWORK_URL" \
|
||||
bash /tmp/framework/scripts/test-external-plugin.sh
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART) - generate and install a plugin fast
|
||||
- [PLUGIN-HOOKS.md](PLUGIN-HOOKS) - the full hook and import-surface reference
|
||||
- [CONTRACT-STABILITY.md](CONTRACT-STABILITY) - what is settled vs still churning before 1.0
|
||||
- [ADR-002](ADR-002-plugin-versioning) - contract versioning and core_version ranges
|
||||
- [ADR-003](ADR-003-plugin-distribution) - the bundled vs external distribution model
|
||||
- [ADR-008](ADR-008-plugin-migration-ownership) - per-plugin migration ownership
|
||||
685
PLUGIN-GUIDE.md
Normal file
685
PLUGIN-GUIDE.md
Normal file
@@ -0,0 +1,685 @@
|
||||
# Building a ShopDB plugin: the measuringtools walkthrough
|
||||
|
||||
This guide builds one real plugin, `measuringtools`, from an empty directory to a
|
||||
running feature with its own list, detail, form, settings page, and report. It is
|
||||
the companion to [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART): the quickstart gets
|
||||
you moving with `flask plugin new`; this guide explains *why* each piece looks the
|
||||
way it does by walking the shipped code of the exemplar plugin.
|
||||
|
||||
`measuringtools` was chosen as the exemplar on purpose. It is the first plugin
|
||||
built after the framework matured (ADR-005 scoped it; ADR-008 changed how plugin
|
||||
migrations work; ADR-009 added frontend route gating). It exercises every current
|
||||
framework feature correctly, so it doubles as the reference implementation. When
|
||||
in doubt about how a plugin should do something, read `plugins/measuringtools/`.
|
||||
|
||||
The domain: measuring tools are gage-lab instruments (calipers, micrometers,
|
||||
thread gages, bore gages, height gages, indicators, Genspect heads) that *measure*
|
||||
parts, as opposed to equipment that *makes* parts. They live inside operation
|
||||
Locations (e.g. "0615 Blisk Inspection"). Their lifecycle is CALIBRATION, not
|
||||
maintenance: an interval, a last date, a next date, and a status derived from the
|
||||
next date. See [ADR-005](ADR-005-equipment-vs-measuringtools) for the split
|
||||
between `equipment` and `measuringtools`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Plugin anatomy
|
||||
|
||||
A bundled plugin is a Python package under `plugins/<name>/` plus a frontend that
|
||||
(for now) lives in core `frontend/src/` (see section 9 and ADR-009 for why). The
|
||||
backend tree for `measuringtools`:
|
||||
|
||||
```
|
||||
plugins/measuringtools/
|
||||
__init__.py # exports the plugin class
|
||||
manifest.json # metadata: name, version, core_version, api_prefix
|
||||
plugin.py # MeasuringToolsPlugin(BasePlugin): hooks + on_install
|
||||
models/
|
||||
__init__.py # re-exports the models + derive_status helper
|
||||
measuringtool.py # MeasuringTool, MeasuringToolType, derive_status
|
||||
api/
|
||||
__init__.py # exports the blueprint
|
||||
routes.py # the Flask blueprint (CRUD, types, report)
|
||||
migrations/
|
||||
env.py # delegates to the shared plugin Alembic runner
|
||||
script.py.mako # migration template (copied from a sibling plugin)
|
||||
versions/
|
||||
0001_measuringtools_baseline.py # REAL baseline: creates the tables
|
||||
```
|
||||
|
||||
Every plugin implements three required hooks (`meta`, `get_blueprint`,
|
||||
`get_models`) and overrides the optional hooks it needs. The base class is
|
||||
`shopdb/plugins/base.py`; the full hook catalog is
|
||||
[PLUGIN-HOOKS.md](PLUGIN-HOOKS).
|
||||
|
||||
---
|
||||
|
||||
## 2. Manifest and versioning
|
||||
|
||||
`plugins/measuringtools/manifest.json` is the single source of truth for the
|
||||
plugin's identity (ADR-002):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "measuringtools",
|
||||
"version": "1.0.0",
|
||||
"description": "Metrology and inspection instruments ... derived calibration status.",
|
||||
"author": "ShopDB Team",
|
||||
"dependencies": [],
|
||||
"core_version": ">=0.6.0,<1.0.0",
|
||||
"api_prefix": "/api/measuringtools",
|
||||
"default_enabled": false
|
||||
}
|
||||
```
|
||||
|
||||
Two fields deserve attention.
|
||||
|
||||
`core_version` is a semver range against the framework's `__contract_version__`
|
||||
(declared in `shopdb/__init__.py`, currently `0.6.0`). The loader refuses to load
|
||||
a plugin whose range excludes the running framework. We pin `>=0.6.0` because this
|
||||
plugin uses the `get_reports` hook, which was added to the contract in 0.6.0
|
||||
(see [PLUGIN-HOOKS.md](PLUGIN-HOOKS), "get_reports"). We cap at `<1.0.0` because
|
||||
the framework is pre-1.0 and the contract can still shift; a sister site forking
|
||||
the plugin gets an explicit failure rather than a silent break if it lands on a
|
||||
newer, incompatible core. This is the ADR-002 discipline: consume the lowest
|
||||
contract version you actually need, and cap below the next major.
|
||||
|
||||
`default_enabled: false` means installing the plugin does not turn it on. Sites opt
|
||||
in. This is the convention for any plugin that provisions extra tables or that not
|
||||
every site wants; a gage lab installs it, a site without one does not.
|
||||
|
||||
The `plugin.py` `meta` property just reads the manifest, so the manifest stays the
|
||||
one place these values live:
|
||||
|
||||
```python
|
||||
@property
|
||||
def meta(self) -> PluginMeta:
|
||||
return PluginMeta(
|
||||
name=self._manifest.get('name', 'measuringtools'),
|
||||
version=self._manifest.get('version', '1.0.0'),
|
||||
...
|
||||
core_version=self._manifest.get('core_version', '>=0.6.0,<1.0.0'),
|
||||
api_prefix=self._manifest.get('api_prefix', '/api/measuringtools'),
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Models, naming convention, and PLUGIN_TABLE_OWNERS
|
||||
|
||||
The plugin owns two tables (`plugins/measuringtools/models/measuringtool.py`):
|
||||
|
||||
- `measuringtooltypes` - the site-managed lookup (Caliper, Micrometer, ...) with a
|
||||
display color, the same shape as `machinetypes` / `computertypes`.
|
||||
- `measuringtools` - the one-to-one extension of a core `Asset`, carrying only the
|
||||
metrology domain fields.
|
||||
|
||||
```python
|
||||
class MeasuringTool(BaseModel):
|
||||
__tablename__ = 'measuringtools'
|
||||
|
||||
measuringtoolid = db.Column(db.Integer, primary_key=True)
|
||||
assetid = db.Column(
|
||||
db.Integer,
|
||||
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
||||
unique=True, nullable=False, index=True,
|
||||
)
|
||||
measuringtooltypeid = db.Column(
|
||||
db.Integer, db.ForeignKey('measuringtooltypes.measuringtooltypeid'),
|
||||
nullable=True)
|
||||
calibrationintervaldays = db.Column(db.Integer, nullable=True)
|
||||
lastcalibrationdate = db.Column(db.Date, nullable=True)
|
||||
nextcalibrationdate = db.Column(db.Date, nullable=True)
|
||||
calibrationprovider = db.Column(db.String(150), nullable=True)
|
||||
notes = db.Column(db.Text, nullable=True)
|
||||
```
|
||||
|
||||
Design points:
|
||||
|
||||
- **Identity lives on the Asset, not here.** Per ADR-001 and ADR-005,
|
||||
`Asset.assetnumber` is the gage tag, `Asset.serialnumber` is the vendor serial,
|
||||
and `Asset.gaugelabreference` is the gage-lab identifier. The extension table
|
||||
carries only what is specific to a measuring tool: its type and its calibration
|
||||
lifecycle. One extension row per asset, enforced by `unique=True` on `assetid`.
|
||||
|
||||
- **Naming convention (CONTRIBUTING.md).** DB tables and columns are lowercase and
|
||||
concatenated: `measuringtooltypeid`, `calibrationintervaldays`,
|
||||
`nextcalibrationdate`. No underscores, no camelCase, no banned shorthand. The
|
||||
pre-commit hook `scripts/check-naming-and-style.sh` fails the build otherwise.
|
||||
|
||||
- **Derived status is never stored.** Calibration status is computed from
|
||||
`nextcalibrationdate` at read time, mirroring the warranty plugin's pattern:
|
||||
|
||||
```python
|
||||
DUESOON_WINDOW_DAYS = 30
|
||||
STATUS_COLORS = {'overdue': '#F44336', 'duesoon': '#FF9800',
|
||||
'current': '#4CAF50', 'unknown': '#9E9E9E'}
|
||||
|
||||
def derive_status(nextcalibrationdate, today=None):
|
||||
if not nextcalibrationdate:
|
||||
return 'unknown'
|
||||
today = today or date.today()
|
||||
if nextcalibrationdate < today:
|
||||
return 'overdue'
|
||||
if nextcalibrationdate <= today + timedelta(days=DUESOON_WINDOW_DAYS):
|
||||
return 'duesoon'
|
||||
return 'current'
|
||||
```
|
||||
|
||||
Storing a status column would let it go stale the moment a due date passed with no
|
||||
write. Deriving it means the list badge, the detail badge, and the report are
|
||||
always correct without a nightly job.
|
||||
|
||||
`to_dict()` overlays the derived status and the type name onto the row and emits
|
||||
the two `Date` columns as `YYYY-MM-DD` (the framework's `BaseModel.to_dict` only
|
||||
iso-formats `datetime`, not plain `date`, so the plugin does it explicitly for
|
||||
predictable JSON).
|
||||
|
||||
**Register the tables in `PLUGIN_TABLE_OWNERS`.** The migration engine needs an
|
||||
explicit map of which tables each plugin owns
|
||||
(`shopdb/plugins/alembic_template.py`):
|
||||
|
||||
```python
|
||||
PLUGIN_TABLE_OWNERS: dict[str, Iterable[str]] = {
|
||||
...
|
||||
'measuringtools': ('measuringtooltypes', 'measuringtools'),
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
`tests/test_plugin_migrations.py::test_table_owners_match_declared_models` fails if
|
||||
this map drifts from the `__tablename__` declarations in either direction, so you
|
||||
cannot forget it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Per-plugin migrations: anchor vs real baseline
|
||||
|
||||
This is the part that differs most from the ten older bundled plugins, so read
|
||||
[ADR-008](ADR-008-plugin-migration-ownership) alongside this section.
|
||||
|
||||
Every table-owning plugin carries its own Alembic chain under
|
||||
`plugins/<name>/migrations/`, with a per-plugin version table
|
||||
`alembic_version_<name>` independent of the core `alembic_version`. The `env.py` is
|
||||
a three-line delegate to the shared runner:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ['PLUGIN_NAME'] = 'measuringtools'
|
||||
from shopdb.plugins.alembic_template import run_migrations
|
||||
run_migrations()
|
||||
```
|
||||
|
||||
The key distinction is the `0001` revision.
|
||||
|
||||
**Cutover plugins (the older ten) ship a no-op anchor.** ADR-008 froze a cutover
|
||||
at the core-chain head. The core Alembic chain had already created every table
|
||||
that existed at that point, including those ten plugins' tables. So their `0001`
|
||||
migration is a stamp-only no-op: `upgrade()` does `pass`, because the tables
|
||||
already exist. It exists only to give the plugin chain a base revision that
|
||||
`flask plugin upgrade-all` can stamp.
|
||||
|
||||
**`measuringtools` was built after the cutover, so its `0001` is a REAL baseline.**
|
||||
The core chain never knew about `measuringtooltypes` / `measuringtools`, so this
|
||||
per-plugin chain is their sole authoritative creator. `upgrade()` actually creates
|
||||
the tables:
|
||||
|
||||
```python
|
||||
revision = 'measuringtools0001baseline'
|
||||
down_revision = None
|
||||
|
||||
def upgrade():
|
||||
op.create_table('measuringtooltypes', ...)
|
||||
op.create_table('measuringtools',
|
||||
...,
|
||||
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'], ondelete='CASCADE'),
|
||||
sa.ForeignKeyConstraint(['measuringtooltypeid'],
|
||||
['measuringtooltypes.measuringtooltypeid']),
|
||||
...)
|
||||
op.create_index('idx_measuringtool_type', 'measuringtools', ['measuringtooltypeid'])
|
||||
```
|
||||
|
||||
Two things to know when you write a post-cutover baseline:
|
||||
|
||||
- **The shared `create_plugin_tables` helper cannot render a foreign key to a core
|
||||
table.** It builds a per-plugin `MetaData` filtered to only the plugin's own
|
||||
tables, so a foreign key to `assets` has no `assets` table to resolve against and
|
||||
Alembic raises `NoReferencedTableError` at compile time. Because these tables
|
||||
reference `assets.assetid`, the baseline uses explicit `op.create_table` ops (the
|
||||
same shape Alembic autogenerate produces) instead of the helper. If your plugin's
|
||||
tables have no FKs to core tables, `create_plugin_tables` works; if they do, hand
|
||||
the DDL to `op.create_table`. (This limitation surfaced building this exemplar
|
||||
and is a candidate framework fix: teach `_get_plugin_metadata` to pull in
|
||||
FK-referenced core tables.)
|
||||
|
||||
- **Charset.** The baseline emits plain `CREATE TABLE`, so the tables inherit the
|
||||
connection's default charset. On the documented utf8mb4 database that yields
|
||||
utf8mb4 tables, matching how the core chain creates its tables. No explicit
|
||||
`mysql_charset` is needed, and none of the unique columns exceed the InnoDB index
|
||||
prefix limit (the longest is `name VARCHAR(100)` = 400 bytes < 767).
|
||||
|
||||
The deploy sequence is unchanged: `flask db upgrade` (core chain) then
|
||||
`flask plugin upgrade-all` (stamps anchors and runs real baselines like this one).
|
||||
Both are idempotent.
|
||||
|
||||
**Adapting the guard test.** `tests/test_plugin_migrations.py` has a test that
|
||||
asserts each `0001` anchor is a pure no-op. That is true for the ten cutover
|
||||
plugins but deliberately false for `measuringtools`. The fix is to scope the no-op
|
||||
assertion to a frozen `CUTOVER_PLUGINS` list rather than to all discovered plugins,
|
||||
and to expect `measuringtools`'s real baseline revision in the upgrade-all test:
|
||||
|
||||
```python
|
||||
CUTOVER_PLUGINS = ('computers', 'employees', 'equipment', 'knowledgebase',
|
||||
'network', 'notifications', 'printers', 'slides', 'usb', 'warranty')
|
||||
|
||||
@pytest.mark.parametrize('plugin', CUTOVER_PLUGINS) # not all plugins
|
||||
def test_anchor_migration_is_noop(plugin):
|
||||
...
|
||||
```
|
||||
|
||||
Freezing the list (rather than deriving it) is intentional: a newly discovered
|
||||
plugin should not silently be treated as a cutover no-op.
|
||||
|
||||
---
|
||||
|
||||
## 5. Contract purity: import only from `shopdb.api`
|
||||
|
||||
A plugin may import from exactly two core places: `shopdb.plugins.base` (for
|
||||
`BasePlugin` / `PluginMeta`) and `shopdb.api` (everything else). Importing internal
|
||||
paths like `shopdb.core.models.*`, `shopdb.extensions`, or `shopdb.utils.*` is a
|
||||
contract violation. `measuringtools` gets its whole surface from `shopdb.api`:
|
||||
|
||||
```python
|
||||
from shopdb.api import (
|
||||
db, Asset, AssetType, AuditLog,
|
||||
success_response, error_response, paginated_response, ErrorCodes,
|
||||
get_pagination_params, paginate_query,
|
||||
require_permission,
|
||||
)
|
||||
from shopdb.api import db, BaseModel # in the models module
|
||||
```
|
||||
|
||||
`shopdb.api` is the versioned platform contract (ADR-001). Adding a name there is
|
||||
an additive minor bump; removing one is a breaking major bump. The guard test
|
||||
`tests/test_plugin_contract.py::test_plugins_only_import_contract_surface` scans
|
||||
every plugin's imports and fails the build if one reaches past the contract. Build
|
||||
against `shopdb.api` and your plugin travels cleanly to a sister site running a
|
||||
compatible core.
|
||||
|
||||
---
|
||||
|
||||
## 6. Blueprint: authz, responses, pagination
|
||||
|
||||
`plugins/measuringtools/api/routes.py` is a normal Flask blueprint. Three framework
|
||||
conventions run through it.
|
||||
|
||||
**Reads are `jwt_required(optional=True)`; writes require a permission.** This is
|
||||
the app-wide pattern: anonymous kiosks and unauthenticated internal users can read
|
||||
the asset catalog, but any state change needs a logged-in user with the right
|
||||
permission.
|
||||
|
||||
```python
|
||||
@measuringtools_bp.route('', methods=['GET'])
|
||||
@jwt_required(optional=True)
|
||||
def list_tools():
|
||||
...
|
||||
|
||||
@measuringtools_bp.route('', methods=['POST'])
|
||||
@jwt_required()
|
||||
@require_permission('measuringtools.create')
|
||||
def create_tool():
|
||||
...
|
||||
```
|
||||
|
||||
The `measuringtools.*` permissions belong to the plugin, not to core. The plugin
|
||||
declares them from the `get_permissions` hook (contract 0.10.0) so core never edits
|
||||
its catalog to accommodate a plugin:
|
||||
|
||||
```python
|
||||
class MeasuringToolsPlugin(BasePlugin):
|
||||
def get_permissions(self):
|
||||
return [
|
||||
('measuringtools.view', 'View measuring tools', 'measuringtools'),
|
||||
('measuringtools.create', 'Create measuring tools', 'measuringtools'),
|
||||
('measuringtools.edit', 'Edit measuring tools', 'measuringtools'),
|
||||
('measuringtools.delete', 'Delete measuring tools', 'measuringtools'),
|
||||
]
|
||||
```
|
||||
|
||||
Installing or enabling the plugin seeds these rows automatically, and
|
||||
`flask seed permissions` (which now seeds core plus every enabled plugin) is
|
||||
idempotent, so re-running it just adds any missing rows. The `admin` role bypasses
|
||||
every permission check, so an admin can operate the plugin before anyone grants the
|
||||
granular permissions. See `get_permissions` in `docs/PLUGIN-HOOKS.md` for the
|
||||
disabled-plugin edge case.
|
||||
|
||||
**Responses use the framework helpers.** `success_response`, `error_response` (with
|
||||
`ErrorCodes`), and `paginated_response` produce the standard envelope
|
||||
(`{status, data, meta}`) so every plugin's API looks the same to the frontend. The
|
||||
list endpoint pages with `get_pagination_params` + `paginate_query`.
|
||||
|
||||
**The write path merges asset core and extension in one payload**, mirroring
|
||||
`equipment` and `computers`. Create builds the `Asset` first, flushes to get the
|
||||
`assetid`, then builds the extension row and writes an audit log:
|
||||
|
||||
```python
|
||||
asset = Asset(assetnumber=data['assetnumber'], ..., assettypeid=assettypeid, ...)
|
||||
db.session.add(asset); db.session.flush()
|
||||
tool = MeasuringTool(assetid=asset.assetid, measuringtooltypeid=..., ...)
|
||||
db.session.add(tool); db.session.flush()
|
||||
AuditLog.log('created', 'MeasuringTool', entityid=tool.measuringtoolid,
|
||||
entityname=asset.assetnumber)
|
||||
db.session.commit()
|
||||
return success_response(_merged(tool), message='Measuring tool created', http_code=201)
|
||||
```
|
||||
|
||||
`_merged()` returns `asset.to_dict()` with the extension nested under
|
||||
`measuringtool`, so the GET, POST, and PUT responses all share one shape.
|
||||
|
||||
**The types resource has an in-use delete guard.** Deleting a type that tools still
|
||||
reference is refused with a 409, so a color/name in use cannot vanish out from
|
||||
under existing rows:
|
||||
|
||||
```python
|
||||
inuse = MeasuringTool.query.filter_by(measuringtooltypeid=type_id).count()
|
||||
if inuse:
|
||||
return error_response(ErrorCodes.CONFLICT,
|
||||
f'Cannot delete: {inuse} tool(s) still use this type', http_code=409)
|
||||
```
|
||||
|
||||
**The calibration report** mirrors the warranty report shape: counts plus lists
|
||||
bucketed by derived status.
|
||||
|
||||
```python
|
||||
@measuringtools_bp.route('/report/calibration', methods=['GET'])
|
||||
@jwt_required(optional=True)
|
||||
def calibration_report():
|
||||
buckets = {'overdue': [], 'duesoon': [], 'current': [], 'unknown': []}
|
||||
for tool in MeasuringTool.query.join(Asset).filter(Asset.isactive == True).all():
|
||||
buckets[derive_status(tool.nextcalibrationdate)].append(_merged(tool))
|
||||
return success_response({'counts': {k: len(v) for k, v in buckets.items()},
|
||||
'buckets': buckets, 'statuscolors': STATUS_COLORS})
|
||||
```
|
||||
|
||||
One deliberate note: the list endpoint's `calibrationstatus` filter is applied to
|
||||
the built rows, not as a SQL `WHERE`, because the status is derived and not a
|
||||
column. That is fine for a gage lab's worth of tools; it would need rethinking for
|
||||
a fleet the size of the PC estate.
|
||||
|
||||
---
|
||||
|
||||
## 7. on_install seeding
|
||||
|
||||
`on_install` runs once, when a site runs `flask plugin install measuringtools`. It
|
||||
seeds the plugin's own asset type and a set of starter tool types:
|
||||
|
||||
```python
|
||||
def on_install(self, app: Flask) -> None:
|
||||
with app.app_context():
|
||||
self._ensure_asset_type() # AssetType 'measuring_tool'
|
||||
self._ensure_starter_types() # Caliper, Micrometer, Thread Gage, ...
|
||||
db.session.commit()
|
||||
```
|
||||
|
||||
`_ensure_asset_type` registers `AssetType(assettype='measuring_tool',
|
||||
pluginname='measuringtools', tablename='measuringtools', icon='ruler')`. The asset
|
||||
type is what ties a core `Asset` to this plugin's extension table and drives the
|
||||
custom-fields lookup (section 10). The starter types come with colors drawn from
|
||||
the shared frontend palette so map markers and badges are legible out of the box.
|
||||
|
||||
Seeding is idempotent (each `_ensure_*` checks for existence first), so a reinstall
|
||||
does not duplicate rows.
|
||||
|
||||
---
|
||||
|
||||
## 8. Hooks, one by one
|
||||
|
||||
`measuringtools` implements four optional hooks and consciously skips two.
|
||||
|
||||
**`get_navigation_items`** puts "Measuring Tools" in the sidebar. Navigation is
|
||||
data-driven: the frontend renders whatever enabled plugins return, so no core file
|
||||
lists the link.
|
||||
|
||||
```python
|
||||
def get_navigation_items(self):
|
||||
return [{'name': 'Measuring Tools', 'icon': 'ruler',
|
||||
'route': '/measuringtools', 'position': 22}]
|
||||
```
|
||||
|
||||
`position: 22` slots it into the Assets band (10-29) after Network and Printers.
|
||||
The `icon` is a string key; the frontend maps `'ruler'` to a Lucide component
|
||||
(section 9).
|
||||
|
||||
**`get_reports`** (contract 0.6.0) contributes the calibration card to the Reports
|
||||
hub. Each card has a stable `id`, a `name`, a `description`, a `category`, and
|
||||
exactly one of `route` (a dedicated page) or `endpoint` (inline render). We use a
|
||||
route because the report is a full bucketed page:
|
||||
|
||||
```python
|
||||
def get_reports(self):
|
||||
return [{'id': 'calibration', 'name': 'Calibration Due',
|
||||
'description': 'Measuring tools bucketed by calibration status',
|
||||
'category': 'compliance', 'route': '/reports/calibration'}]
|
||||
```
|
||||
|
||||
`GET /api/reports` merges this after the core reports and drops it when the plugin
|
||||
is disabled.
|
||||
|
||||
**`get_models`** returns `[MeasuringTool, MeasuringToolType]` so migration tooling
|
||||
and admin views know the plugin's tables.
|
||||
|
||||
**`get_config_schema` returns `[]`, on purpose.** The setup wizard reads this to
|
||||
know what a plugin needs configured. Measuring tools are tracked by hand: there is
|
||||
no external system to authenticate against, no endpoint URL, no API key. So the
|
||||
schema is empty and the wizard shows nothing to configure. This is the intentional
|
||||
"no external creds" case. Contrast the printers plugin, which declares a Zabbix URL
|
||||
field. If your plugin talks to an external service, declare its (non-secret)
|
||||
settings here and mark credentials `secret: True` so the wizard emits an `.env`
|
||||
line instead of storing them in the database.
|
||||
|
||||
**Skipped: the collector hooks (`get_collector_schema` /
|
||||
`apply_collector_payload`).** These (ADR-006) exist for plugins fed by an automated
|
||||
agent pushing to `/api/collector/<name>`, like `computers` fed by the PXE pipeline.
|
||||
Measuring tools are entered by hand in the gage lab; there is no collector. Because
|
||||
`get_collector_schema` returns `None` (the base-class default), no collector
|
||||
endpoint is registered and `apply_collector_payload` is never called. The contract
|
||||
test `test_schema_declaring_plugins_implement_apply` only requires the upsert method
|
||||
when a schema is declared, so skipping both is clean. If a site later automates
|
||||
calibration imports (say from a cal-lab spreadsheet), that is the hook to add.
|
||||
|
||||
**Not skipped so much as not needed: dashboard widgets, services, CLI commands.**
|
||||
`get_dashboard_widgets` could add a "calibration due" tile later; `get_services`
|
||||
and `get_cli_commands` have no use here yet. Leaving them at their defaults keeps
|
||||
the plugin small.
|
||||
|
||||
---
|
||||
|
||||
## 9. Frontend integration
|
||||
|
||||
There is no frontend plugin system yet (see
|
||||
[ADR-009](ADR-009-frontend-plugin-gating), "Future direction"). A plugin's
|
||||
Vue routes and views ship in the core bundle. The plugin's job is to add them
|
||||
correctly and gate them.
|
||||
|
||||
**Route module with `meta.plugin` gating (ADR-009).** A new file
|
||||
`frontend/src/router/routes/measuringtools.js` is auto-discovered by the router's
|
||||
`import.meta.glob('./routes/*.js')`. Every route carries `meta.plugin =
|
||||
'measuringtools'`:
|
||||
|
||||
```js
|
||||
export default [
|
||||
{ path: 'measuringtools', name: 'measuringtools',
|
||||
component: () => import('../../views/measuringtools/MeasuringToolsList.vue'),
|
||||
meta: { plugin: 'measuringtools' } },
|
||||
{ path: 'measuringtools/new', name: 'measuringtool-new',
|
||||
component: () => import('../../views/measuringtools/MeasuringToolForm.vue'),
|
||||
meta: { requiresAuth: true, plugin: 'measuringtools' } },
|
||||
{ path: 'measuringtools/:id', ..., meta: { plugin: 'measuringtools' } },
|
||||
{ path: 'measuringtools/:id/edit', ..., meta: { requiresAuth: true, plugin: 'measuringtools' } },
|
||||
{ path: 'reports/calibration', ..., meta: { plugin: 'measuringtools' } },
|
||||
{ path: 'settings/measuringtooltypes', ...,
|
||||
meta: { requiresAuth: true, requiresAdmin: true, plugin: 'measuringtools' } },
|
||||
]
|
||||
```
|
||||
|
||||
`meta.plugin` makes the router guard redirect to the dashboard (instead of loading
|
||||
a broken shell) when the backend plugin is disabled. It fetches
|
||||
`GET /api/plugins/enabled` once, cached, and fails open. Note the auth pattern that
|
||||
matches the rest of the app: list and detail are public (no `requiresAuth`); the
|
||||
form is `requiresAuth`; the settings subtype page is `requiresAuth + requiresAdmin`.
|
||||
|
||||
**API client, addition only.** `frontend/src/api/index.js` gets a
|
||||
`measuringtoolsApi` object appended after `warrantyApi` (`list`, `get`, `create`,
|
||||
`update`, `remove`, `calibrationReport`, and a nested `types` CRUD). Do not
|
||||
reorganize the file; just add the block, mirroring `machinesApi`.
|
||||
|
||||
**Views mirror the master templates.** The frontend has master templates
|
||||
(`PrintersList.vue` for lists, `PrinterDetail.vue` for detail pages). `measuringtools` mirrors the equivalent equipment views:
|
||||
|
||||
- `views/measuringtools/MeasuringToolsList.vue` - table with search, a type filter,
|
||||
and a calibration-status filter; the status badge uses `utils/colorStyle` with
|
||||
the color the API derived.
|
||||
- `views/measuringtools/MeasuringToolDetail.vue` - hero + Identity card +
|
||||
Calibration card (with the derived badge) + Location card, plus the shared
|
||||
`CustomFieldsSection` and `WarrantyPanel` (section 10).
|
||||
- `views/measuringtools/MeasuringToolForm.vue` - asset core fields + type +
|
||||
location + the calibration fields, plus `CustomFieldsInputs`.
|
||||
- `views/reports/CalibrationReport.vue` - the four buckets (overdue / due soon /
|
||||
current / unknown), mirroring `WarrantyReport.vue`.
|
||||
|
||||
**Settings subtype page.** `views/settings/MeasuringToolTypesList.vue` mirrors
|
||||
`PCTypesList.vue`: add / edit / delete with a `ColorSwatchPicker`. It is linked from
|
||||
`settingsNav.js` with a "Measuring Tools" card group, so it appears in the settings
|
||||
rail and landing overview.
|
||||
|
||||
**iconMap, entry only.** The sidebar maps backend icon-name strings to Lucide
|
||||
components in `AppLayout.vue`. Add the import and one map entry:
|
||||
|
||||
```js
|
||||
import { ..., Ruler } from 'lucide-vue-next'
|
||||
const iconMap = { ..., 'ruler': Ruler }
|
||||
```
|
||||
|
||||
That is the only change to `AppLayout.vue`. This is exactly the kind of core-file
|
||||
edit ADR-009's future direction wants to replace with a registration API; for now
|
||||
it is a one-line addition.
|
||||
|
||||
---
|
||||
|
||||
## 10. Custom fields and warranty panel composition
|
||||
|
||||
Two cross-cutting core features compose onto the plugin's pages for free.
|
||||
|
||||
**Custom fields.** Sites define extra attributes per asset type in Settings. The
|
||||
detail page drops in `<CustomFieldsSection :assetid="tool.assetid" />` and the form
|
||||
drops in `<CustomFieldsInputs :assettypeid="assettypeid" :assetid="currentAssetId" />`,
|
||||
then calls `customFieldsRef.value.save(assetId)` after the tool saves. The one
|
||||
subtlety: `CustomFieldsInputs` needs the asset-type id. Rather than hardcode it (the
|
||||
equipment form hardcodes `EQUIPMENT_ASSETTYPEID = 1`), the measuringtools form
|
||||
resolves it dynamically from `GET /api/assets/types`, finding the row whose
|
||||
`assettype === 'measuring_tool'`. Dynamic lookup is preferred because seeded ids are
|
||||
not stable across sites.
|
||||
|
||||
**Warranty panel.** `<WarrantyPanel :assetid="tool.assetid" :items="warranties" />`
|
||||
renders any warranties covering the tool. Warranty is an asset-general plugin, so it
|
||||
works on any asset type with no coupling; the measuring-tool detail page composes it
|
||||
the same way the equipment and PC detail pages do.
|
||||
|
||||
**Map.** The asset map is data-driven off asset types and resolved positions
|
||||
(ADR-001). Once a measuring tool is typed and given a location or map coordinates,
|
||||
it appears on `/assets/map` with its type color, with no plugin-side map code. There
|
||||
is nothing to force; giving the type a color and the asset a position is enough.
|
||||
|
||||
---
|
||||
|
||||
## 11. Tests
|
||||
|
||||
`tests/test_plugins/test_measuringtools.py` covers the plugin end to end:
|
||||
|
||||
- **Derived status**, as pure unit tests against `derive_status`: all four buckets
|
||||
plus the 30-day boundary (exactly `+30` is still "due soon"; due today is "due
|
||||
soon", not "overdue").
|
||||
- **Type CRUD + the in-use delete guard** (409 when a tool references the type).
|
||||
- **Tool create/update in one merged payload**, asserting asset core fields land at
|
||||
the top level and extension fields nest under `measuringtool`, and that a past
|
||||
`nextcalibrationdate` derives `overdue`.
|
||||
- **The report shape** (counts and buckets agree; the four keys are present).
|
||||
|
||||
Because the plugin ships `default_enabled: false`, the shared session app does not
|
||||
register its blueprint. The test module builds its own app, registers the blueprint,
|
||||
runs `on_install` seeding, and restores the process-wide `plugin_manager` singleton
|
||||
afterward, the same snapshot/restore pattern `test_plugin_migrations.py` uses. The
|
||||
existing parametrized contract and migration tests pick the plugin up automatically
|
||||
from the filesystem and must stay green.
|
||||
|
||||
Run the suite:
|
||||
|
||||
```bash
|
||||
venv/bin/python -m pytest
|
||||
bash scripts/check-naming-and-style.sh
|
||||
cd frontend && npm run build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Install / enable lifecycle
|
||||
|
||||
The plugin is on disk but does nothing until a site installs it. Two commands, then
|
||||
a restart:
|
||||
|
||||
```bash
|
||||
flask plugin install measuringtools # runs the 0001 baseline, then on_install seeding
|
||||
flask plugin enable measuringtools # default_enabled is false, so enable explicitly
|
||||
```
|
||||
|
||||
`install` runs the migration (creating the tables), registers the plugin in
|
||||
`instance/plugins.json`, and calls `on_install`. Because `default_enabled` is false,
|
||||
it installs disabled; `enable` flips the registry flag. The registry entry looks
|
||||
like:
|
||||
|
||||
```json
|
||||
"measuringtools": { "name": "measuringtools", "version": "1.0.0",
|
||||
"enabled": true, "migrations_applied": [], "config": {} }
|
||||
```
|
||||
|
||||
**Restart is required.** Blueprints are registered at app startup; Flask forbids
|
||||
registering one after the app has served a request. So after enabling, restart the
|
||||
API so the loader registers `/api/measuringtools`. On a dev box:
|
||||
|
||||
```bash
|
||||
# kill the running server on :5001, then:
|
||||
setsid venv/bin/flask run --port 5001 --no-reload >/tmp/flask.log 2>&1 < /dev/null &
|
||||
```
|
||||
|
||||
For a deploy, the sequence is `flask db upgrade` then `flask plugin upgrade-all`
|
||||
(which runs this plugin's baseline on a fresh install and is a no-op on an existing
|
||||
one), then a service restart. See [PLUGINS.md](PLUGINS) and
|
||||
[ADR-008](ADR-008-plugin-migration-ownership).
|
||||
|
||||
---
|
||||
|
||||
## End checklist
|
||||
|
||||
When you build a plugin, confirm all of this before you call it done:
|
||||
|
||||
- [ ] `manifest.json`: name, version, `core_version` pinned to the lowest contract
|
||||
you need and capped below the next major, `api_prefix`, `default_enabled`.
|
||||
- [ ] Models declare tables with the naming convention; identity stays on `Asset`.
|
||||
- [ ] Derived state (if any) is computed at read time, never stored.
|
||||
- [ ] Tables registered in `PLUGIN_TABLE_OWNERS`.
|
||||
- [ ] Per-plugin migration chain: no-op anchor if your tables predate the cutover, a
|
||||
real baseline if the plugin is post-cutover. Hand-write `op.create_table` if
|
||||
you have FKs to core tables.
|
||||
- [ ] Imports only from `shopdb.api` and `shopdb.plugins.base` (contract test green).
|
||||
- [ ] Blueprint: jwt-optional reads, permission-gated writes; framework response and
|
||||
pagination helpers; audit logs on writes.
|
||||
- [ ] Permissions declared from the `get_permissions` hook; install/enable (or `flask seed permissions`) seeds them.
|
||||
- [ ] `on_install` seeds the asset type and any reference data, idempotently.
|
||||
- [ ] Hooks: navigation, reports, models implemented; config schema and collector
|
||||
implemented or consciously skipped with a reason.
|
||||
- [ ] Frontend: route module with `meta.plugin` gating, API client addition, views
|
||||
mirroring the master templates, settings subtype page + `settingsNav` entry,
|
||||
one `iconMap` entry.
|
||||
- [ ] Custom fields and warranty panel composed onto the detail/form pages.
|
||||
- [ ] Tests: unit + API, self-contained app if `default_enabled` is false; full
|
||||
suite and naming/style check green; `npm run build` green.
|
||||
- [ ] Install / enable / restart verified live; a demo record renders on every page.
|
||||
596
PLUGIN-HOOKS.md
Normal file
596
PLUGIN-HOOKS.md
Normal file
@@ -0,0 +1,596 @@
|
||||
# Plugin Hooks Reference
|
||||
|
||||
This is the canonical reference for the shopdb-flask plugin contract. Plugin authors implement `BasePlugin` and override the hooks they care about. Hooks marked `required` must be implemented; hooks marked `optional` have sensible defaults and can be left alone.
|
||||
|
||||
The contract is locked in [ADR-001](ADR-001-asset-as-platform-contract) and versioned per [ADR-002](ADR-002-plugin-versioning).
|
||||
|
||||
## Contract version
|
||||
|
||||
The framework declares its contract version in `shopdb/__init__.py`:
|
||||
|
||||
```python
|
||||
__contract_version__ = '0.11.0'
|
||||
```
|
||||
|
||||
Each plugin's `manifest.json` declares the range of contract versions it supports:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "yourplugin",
|
||||
"version": "1.0.0",
|
||||
"core_version": ">=0.1.0,<1.0.0",
|
||||
"dependencies": []
|
||||
}
|
||||
```
|
||||
|
||||
The plugin loader checks this at load time and refuses to load plugins outside the supported range.
|
||||
|
||||
## Plugin metadata
|
||||
|
||||
Each plugin ships a `manifest.json`. The dataclass `PluginMeta` is constructed from it.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "computers",
|
||||
"version": "1.0.0",
|
||||
"description": "Tracks shop-floor PCs and engineering workstations",
|
||||
"author": "shopdb-flask",
|
||||
"dependencies": [],
|
||||
"core_version": ">=0.1.0,<1.0.0",
|
||||
"api_prefix": "/api/computers"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Notes |
|
||||
|-------|----------|-------|
|
||||
| `name` | Yes | Lowercase concatenated, no underscores or dashes |
|
||||
| `version` | Yes | Plugin's own semver |
|
||||
| `description` | Yes | One sentence |
|
||||
| `dependencies` | No | List of plugin names that must load first |
|
||||
| `core_version` | Yes | Range of framework `__contract_version__` this plugin supports |
|
||||
| `api_prefix` | No | Defaults to `/api/<name>` |
|
||||
|
||||
## Required hooks
|
||||
|
||||
### `meta` -> `PluginMeta`
|
||||
|
||||
Returns the plugin's metadata. Convention is to construct from `manifest.json`:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
import json
|
||||
from shopdb.plugins.base import BasePlugin, PluginMeta
|
||||
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def __init__(self):
|
||||
manifestpath = Path(__file__).parent / 'manifest.json'
|
||||
with open(manifestpath) as f:
|
||||
self._manifest = json.load(f)
|
||||
|
||||
@property
|
||||
def meta(self) -> PluginMeta:
|
||||
return PluginMeta(
|
||||
name=self._manifest['name'],
|
||||
version=self._manifest['version'],
|
||||
description=self._manifest['description'],
|
||||
dependencies=self._manifest.get('dependencies', []),
|
||||
core_version=self._manifest.get('core_version', '>=0.1.0'),
|
||||
api_prefix=self._manifest.get('api_prefix'),
|
||||
)
|
||||
```
|
||||
|
||||
### `get_blueprint() -> Optional[Blueprint]`
|
||||
|
||||
Returns a Flask Blueprint with the plugin's API routes, or `None` if the plugin has no HTTP routes. The loader registers the blueprint at the `api_prefix` from the manifest.
|
||||
|
||||
```python
|
||||
from flask import Blueprint
|
||||
from .api import computers_bp
|
||||
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_blueprint(self):
|
||||
return computers_bp
|
||||
```
|
||||
|
||||
### `get_models() -> List[Type]`
|
||||
|
||||
Returns the SQLAlchemy model classes the plugin defines. Used by the migration runner and admin tooling.
|
||||
|
||||
```python
|
||||
from .models import Computer, ComputerSoftware
|
||||
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_models(self):
|
||||
return [Computer, ComputerSoftware]
|
||||
```
|
||||
|
||||
## Optional hooks
|
||||
|
||||
### `init_app(app, db) -> None`
|
||||
|
||||
Custom initialization. Called by the loader after the blueprint is registered and models are known. Use for Marshmallow schema registration, Caching configuration, secondary blueprint registration, or anything else the plugin needs.
|
||||
|
||||
```python
|
||||
class PrintersPlugin(BasePlugin):
|
||||
def init_app(self, app, db):
|
||||
from .api import printers_legacy_bp
|
||||
app.register_blueprint(printers_legacy_bp, url_prefix='/api/printers/legacy')
|
||||
```
|
||||
|
||||
### `get_cli_commands() -> List`
|
||||
|
||||
Returns a list of Click commands or command groups to register on the Flask CLI.
|
||||
|
||||
```python
|
||||
import click
|
||||
|
||||
@click.group()
|
||||
def computers_cli():
|
||||
pass
|
||||
|
||||
@computers_cli.command()
|
||||
def reset_computers():
|
||||
"""Reset all computer status flags."""
|
||||
...
|
||||
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_cli_commands(self):
|
||||
return [computers_cli]
|
||||
```
|
||||
|
||||
### `get_services() -> Dict[str, Type]`
|
||||
|
||||
Returns a dict of service-name to service-class. Another plugin obtains one via
|
||||
`plugin_manager.get_service('<name>')`, which searches enabled plugins and
|
||||
returns the registered class/factory (or None).
|
||||
|
||||
```python
|
||||
from .services import ZabbixService
|
||||
|
||||
class PrintersPlugin(BasePlugin):
|
||||
def get_services(self):
|
||||
return {'zabbix': ZabbixService}
|
||||
```
|
||||
|
||||
### `get_dashboard_widgets() -> List[Dict]`
|
||||
|
||||
Returns dashboard widget definitions for the home page.
|
||||
|
||||
```python
|
||||
class NotificationsPlugin(BasePlugin):
|
||||
def get_dashboard_widgets(self):
|
||||
return [{
|
||||
'name': 'recent_notifications',
|
||||
'component': 'NotificationsWidget',
|
||||
'endpoint': '/api/notifications/recent',
|
||||
'size': 'medium',
|
||||
'position': 1,
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/dashboard/widgets`, which merges widgets from all enabled
|
||||
plugins sorted by `position` (disabled plugins are skipped; a broken plugin is
|
||||
isolated in prod, re-raised in dev/test).
|
||||
|
||||
### `get_navigation_items() -> List[Dict]`
|
||||
|
||||
Returns navigation menu items.
|
||||
|
||||
```python
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_navigation_items(self):
|
||||
return [{
|
||||
'name': 'Computers',
|
||||
'icon': 'desktop',
|
||||
'route': '/computers',
|
||||
'position': 10,
|
||||
}]
|
||||
```
|
||||
|
||||
> Removed in contract 0.4.0: `get_searchable_fields`. Global search
|
||||
> (`/api/search`) is a core concern that queries the asset model directly and
|
||||
> already covers every bundled asset type; no plugin ever implemented the hook.
|
||||
> Search honors runtime plugin enable/disable.
|
||||
|
||||
### `get_reports() -> List[Dict]`
|
||||
|
||||
Returns report card definitions for the Reports hub. Added in contract 0.6.0.
|
||||
|
||||
Each entry has `id`, `name`, `description`, `category`, plus EXACTLY ONE of
|
||||
`route` (a frontend path for a dedicated report page) or `endpoint` (an API
|
||||
endpoint the hub renders inline).
|
||||
|
||||
```python
|
||||
class WarrantyPlugin(BasePlugin):
|
||||
def get_reports(self):
|
||||
return [{
|
||||
'id': 'warranty',
|
||||
'name': 'Warranty Report',
|
||||
'description': 'Assets bucketed by coverage: expired, expiring soon, active',
|
||||
'category': 'warranty',
|
||||
'route': '/reports/warranty',
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/reports`, which merges plugin cards after the static core
|
||||
reports sorted into category groups by the frontend (disabled plugins are
|
||||
skipped; a broken plugin is isolated in prod, re-raised in dev/test).
|
||||
|
||||
### `get_permissions() -> List`
|
||||
|
||||
Returns the RBAC permissions this plugin owns. Added in contract 0.10.0. A
|
||||
plugin declares the permission names its own routes enforce via
|
||||
`require_permission`, instead of core accumulating every plugin's permissions in
|
||||
one catalog (plugin-is-the-product).
|
||||
|
||||
Each entry is a `(name, description, category)` tuple, matching the core
|
||||
permission catalog shape (dicts with those keys are also accepted). Names follow
|
||||
the naming convention (lowercase dotted, e.g. `machines.edit`).
|
||||
|
||||
```python
|
||||
class MachinesPlugin(BasePlugin):
|
||||
def get_permissions(self):
|
||||
return [
|
||||
('machines.view', 'View machines', 'machines'),
|
||||
('machines.create', 'Create machines', 'machines'),
|
||||
('machines.edit', 'Edit machines', 'machines'),
|
||||
('machines.delete', 'Delete machines', 'machines'),
|
||||
]
|
||||
```
|
||||
|
||||
Consumed by the core helper `full_permission_catalog()` (core permissions plus
|
||||
every ENABLED plugin's `get_permissions()`), which backs three consumers:
|
||||
|
||||
- `flask seed permissions` seeds the full catalog.
|
||||
- The role-management grid (`GET /api/users/permissions`) lists it, grouped by
|
||||
category.
|
||||
- API-token scope validation (`ApiToken.unknown_scope_names`) accepts a plugin
|
||||
permission as a scope only while that plugin is enabled.
|
||||
|
||||
Plugin install and enable also seed the plugin's own permissions idempotently,
|
||||
so enabling a fresh plugin creates its `Permission` rows without a separate seed
|
||||
pass.
|
||||
|
||||
Disabled-plugin edge case: a disabled plugin is skipped by the catalog, so its
|
||||
permissions are no longer offered for new scope grants or new role assignments.
|
||||
The `Permission` ROWS already in the database are NOT deleted, so roles that
|
||||
already reference them keep working until an admin edits the role. A broken
|
||||
plugin is isolated in prod and re-raised in dev/test.
|
||||
|
||||
### `get_settings_cards() -> List[Dict]`
|
||||
|
||||
Returns settings-catalog card definitions. Added in contract 0.7.0 (ADR-010).
|
||||
Each card is merged into the settings rail and landing overview without the
|
||||
plugin hand-editing the core `settingsNav.js` catalog. `icon` is a string key
|
||||
mapped to a Lucide component core-side, exactly like `get_navigation_items`.
|
||||
|
||||
```python
|
||||
class MeasuringToolsPlugin(BasePlugin):
|
||||
def get_settings_cards(self):
|
||||
return [{
|
||||
'group': 'Measuring Tools', # rail group title (created if new)
|
||||
'to': '/settings/measuringtooltypes',
|
||||
'icon': 'ruler', # string key, mapped core-side
|
||||
'title': 'Measuring Tool Types',
|
||||
'description': 'Manage measuring-tool subtypes + map colors',
|
||||
'position': 22, # order within the group
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/pluginui/settings-cards`, which merges enabled plugins'
|
||||
cards into the core catalog (disabled plugins are skipped; a broken plugin is
|
||||
isolated in prod, re-raised in dev/test).
|
||||
|
||||
### `get_asset_panels() -> List[Dict]`
|
||||
|
||||
Returns asset-detail extension-panel definitions. Added in contract 0.7.0
|
||||
(ADR-010). A generic core `AssetPanel` component renders each panel on the
|
||||
matching detail pages, fetching the panel's `endpoint`. This replaces
|
||||
hand-composing a plugin panel component into each detail view.
|
||||
|
||||
```python
|
||||
class WarrantyPlugin(BasePlugin):
|
||||
def get_asset_panels(self):
|
||||
return [{
|
||||
'id': 'warranty',
|
||||
'title': 'Warranty',
|
||||
'assettypes': ['*'], # detail pages it appears on; ['*'] = all
|
||||
'endpoint': '/api/warranty/asset/{assetid}',
|
||||
'render': 'table', # 'keyvalue' | 'table' | 'badge'
|
||||
'position': 30,
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/pluginui/asset-panels?assetid=<id>`, which returns the
|
||||
panels whose `assettypes` match that asset's type (disabled plugins skipped;
|
||||
broken plugin isolated in prod, re-raised in dev/test). A panel that needs
|
||||
bespoke UI (a chart) is out of scope for this data-only hook.
|
||||
|
||||
### `get_map_overlays() -> List[Dict]`
|
||||
|
||||
Returns shop-floor map overlay/decoration definitions. Added in contract 0.7.0
|
||||
(ADR-010). The map stays data-driven off asset types + positions; an overlay
|
||||
adds decoration data (a badge or ring) plus an optional legend entry, with no
|
||||
plugin-side map code.
|
||||
|
||||
```python
|
||||
class MeasuringToolsPlugin(BasePlugin):
|
||||
def get_map_overlays(self):
|
||||
return [{
|
||||
'id': 'calibration-due',
|
||||
'label': 'Calibration due', # legend label
|
||||
'endpoint': '/api/measuringtools/map-overlay', # -> [{assetid, color, label}]
|
||||
'style': 'badge', # 'badge' | 'ring'
|
||||
'legend': True,
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/pluginui/map-overlays` (disabled plugins skipped; broken
|
||||
plugin isolated in prod, re-raised in dev/test).
|
||||
|
||||
### `get_asset_presentation() -> List[Dict]`
|
||||
|
||||
Returns asset-type presentation/routing definitions. Added in contract 0.7.0
|
||||
(ADR-010). Declares how a plugin-owned asset type renders in global-search rows
|
||||
and cross-links (which icon, which detail route), so core never hardcodes a
|
||||
plugin's route or icon.
|
||||
|
||||
```python
|
||||
class MeasuringToolsPlugin(BasePlugin):
|
||||
def get_asset_presentation(self):
|
||||
return [{
|
||||
'assettype': 'measuring_tool', # AssetType.assettype key the plugin owns
|
||||
'icon': 'ruler',
|
||||
'label': 'Measuring Tool',
|
||||
'route': '/measuringtools/{assetid}',
|
||||
}]
|
||||
```
|
||||
|
||||
Consumed by `GET /api/pluginui/asset-presentation` (disabled plugins skipped;
|
||||
broken plugin isolated in prod, re-raised in dev/test).
|
||||
|
||||
### `get_provisioning_note() -> Optional[Dict]`
|
||||
|
||||
Transparency note the setup wizard shows the moment a site checks this plugin
|
||||
during setup. Return `None` (the default) for plugins that need no special
|
||||
setup. Plugins that create extra tables beyond their asset-extension table
|
||||
(e.g. a self-hosted directory) return:
|
||||
|
||||
```python
|
||||
class EmployeesPlugin(BasePlugin):
|
||||
def get_provisioning_note(self):
|
||||
return {
|
||||
'tables': ['directoryemployees'],
|
||||
'note': 'Creates a local employee directory table in the shopdb database.',
|
||||
'docs': 'plugins/employees/README.md',
|
||||
}
|
||||
```
|
||||
|
||||
### `get_config_schema() -> List[Dict]`
|
||||
|
||||
Declares the config fields this plugin needs, so the setup wizard can prompt
|
||||
for them. Return `[]` (the default) if the plugin needs no configuration.
|
||||
Each field is a dict:
|
||||
|
||||
| Key | Meaning |
|
||||
|-----|---------|
|
||||
| `key` | the Setting key (non-secret) it maps to |
|
||||
| `label` | human label shown in the wizard |
|
||||
| `type` | `'text'` / `'number'` / `'password'` |
|
||||
| `secret` | `True` for credentials; NOT stored in the DB - the wizard emits an `.env` line for the operator instead |
|
||||
| `envvar` | (secret only) the `.env` variable name to emit |
|
||||
| `default` | optional placeholder |
|
||||
| `help` | optional hint |
|
||||
|
||||
```python
|
||||
class PrintersPlugin(BasePlugin):
|
||||
def get_config_schema(self):
|
||||
return [
|
||||
{'key': 'zabbix_url', 'label': 'Zabbix URL', 'type': 'text',
|
||||
'help': 'Base URL of the Zabbix server for supply lookups'},
|
||||
{'key': 'zabbix_token', 'label': 'Zabbix API token', 'type': 'password',
|
||||
'secret': True, 'envvar': 'ZABBIX_TOKEN'},
|
||||
]
|
||||
```
|
||||
|
||||
### `get_collector_schema() -> Optional[Dict]`
|
||||
|
||||
Declares the JSON Schema for an external collector pushing to `/api/collector/<pluginname>`. See [ADR-006](ADR-006-collector-contract) for the contract.
|
||||
|
||||
```python
|
||||
class ComputersPlugin(BasePlugin):
|
||||
def get_collector_schema(self):
|
||||
return {
|
||||
'identityfield': 'hostname',
|
||||
'fields': {
|
||||
'hostname': {'type': 'string', 'required': True},
|
||||
'macaddress': {'type': 'string'},
|
||||
'osname': {'type': 'string'},
|
||||
'osversion': {'type': 'string'},
|
||||
'currentuser': {'type': 'string'},
|
||||
'ipaddress': {'type': 'string'},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If the hook returns `None` (the default), no collector endpoint is registered.
|
||||
|
||||
### `apply_collector_payload(payload: Dict) -> Dict`
|
||||
|
||||
Companion to `get_collector_schema` (ADR-006). The generic
|
||||
`/api/collector/<pluginname>` endpoint calls this after the payload passes
|
||||
identity validation, to idempotently upsert an asset. Return a dict with at
|
||||
least `action` (`created` | `updated` | `noop`), `assetid`, and `warnings`
|
||||
(list).
|
||||
|
||||
This is a CONDITIONAL hook: it is only required when `get_collector_schema`
|
||||
returns non-None. The BasePlugin default raises `NotImplementedError` (the
|
||||
dispatcher turns that into a 500), so a plugin that declares a schema but
|
||||
forgets the upsert fails loud. Plugins with no collector schema never need it.
|
||||
The `test_schema_declaring_plugins_implement_apply` contract test enforces the
|
||||
pairing.
|
||||
|
||||
```python
|
||||
def apply_collector_payload(self, payload):
|
||||
host = payload['hostname']
|
||||
comp = Computer.query.filter(Computer.hostname.ilike(host)).first()
|
||||
action = 'updated' if comp else 'created'
|
||||
# ... create-or-update Asset + extension ...
|
||||
db.session.commit()
|
||||
return {'action': action, 'assetid': comp.assetid, 'warnings': []}
|
||||
```
|
||||
|
||||
## Lifecycle hooks
|
||||
|
||||
These run when the plugin's installation state changes. All optional.
|
||||
|
||||
| Hook | When | Use case |
|
||||
|------|------|----------|
|
||||
| `on_install(app)` | First time the plugin is installed via `flask plugin install` | Seed reference data, run plugin-specific migrations, register webhooks |
|
||||
| `on_uninstall(app)` | When the plugin is removed via `flask plugin uninstall` | Clean up reference data, deregister webhooks |
|
||||
| `on_enable(app)` | When the plugin is enabled at runtime | Subscribe to events, warm caches |
|
||||
| `on_disable(app)` | When the plugin is disabled at runtime | Unsubscribe, drain queues |
|
||||
|
||||
## The import surface (`shopdb.api`)
|
||||
|
||||
`shopdb.api` is the ONLY core module a plugin may import from (besides
|
||||
`shopdb.plugins.base` for `BasePlugin` / `PluginMeta`). Importing internal
|
||||
paths like `shopdb.core.models.*`, `shopdb.extensions`, or `shopdb.utils.*`
|
||||
is a contract violation and fails the test
|
||||
`tests/test_plugin_contract.py::test_plugins_only_import_contract_surface`.
|
||||
|
||||
What `shopdb.api` exposes:
|
||||
|
||||
- Infrastructure: `db`, `cache`
|
||||
- Model bases: `BaseModel`, `AuditMixin`
|
||||
- Core models: `Asset`, `AssetType`, `AssetStatus`, `Vendor`, `Model`,
|
||||
`Communication`, `CommunicationType`, `Location`, `Setting`, `AuditLog`,
|
||||
`Application`, `AppVersion`, `OperatingSystem`
|
||||
- Responses: `success_response`, `error_response`, `paginated_response`,
|
||||
`ErrorCodes`
|
||||
- Pagination: `get_pagination_params`, `paginate_query`
|
||||
- Authorization: `require_permission`, `require_role`,
|
||||
`service_token_authorized`
|
||||
(`service_token_authorized(scope)` returns True when the request carries a
|
||||
managed service token scoped for `scope` whose owner holds that permission -
|
||||
for unattended plugin endpoints like the GE-Enforce fetch API)
|
||||
- Helpers: `audit_log`, `resolve_asset_position`, `resolve_dualpath_pairs`,
|
||||
`dualpath_single_machine_enabled`
|
||||
- Import mode: `apply_import_timestamps`, `import_mode_active`,
|
||||
`parse_import_datetime`
|
||||
- Legacy employee directory: `employee_connection`
|
||||
|
||||
```python
|
||||
from shopdb.api import db, Asset, AssetType, success_response, paginate_query
|
||||
```
|
||||
|
||||
Adding a name to `shopdb.api` is an additive (minor) contract bump; removing
|
||||
one is breaking (major). See ADR-002.
|
||||
|
||||
## Helpers exposed to plugins
|
||||
|
||||
The framework provides helper APIs in `shopdb.api` (the public namespace).
|
||||
|
||||
### Audit logging
|
||||
|
||||
```python
|
||||
from shopdb.api import audit_log
|
||||
|
||||
audit_log(
|
||||
action='created',
|
||||
entitytype='Computer',
|
||||
entityid=computer.assetid,
|
||||
entityname=computer.hostname,
|
||||
changes={'before': {}, 'after': computer.to_dict()},
|
||||
)
|
||||
```
|
||||
|
||||
### Plugin-scoped settings
|
||||
|
||||
```python
|
||||
class PrintersPlugin(BasePlugin):
|
||||
def init_app(self, app, db):
|
||||
zabbix_url = self.get_setting('zabbix_url')
|
||||
if not zabbix_url:
|
||||
self.set_setting('zabbix_url', 'http://zabbix.example.com')
|
||||
```
|
||||
|
||||
Settings persist to the core `Setting` model and survive restarts.
|
||||
|
||||
### Position resolution
|
||||
|
||||
```python
|
||||
from shopdb.api import resolve_asset_position
|
||||
|
||||
position = resolve_asset_position(asset)
|
||||
# Returns dict: {'mapx': 234, 'mapy': 567, 'positionsource': 'self' | 'related' | 'location' | None}
|
||||
```
|
||||
|
||||
See [ADR-001](ADR-001-asset-as-platform-contract) for the position resolution algorithm.
|
||||
|
||||
### Dualpath single-machine collapse
|
||||
|
||||
A Dualpath relationship pair is one physical dual-bay machine recorded as two
|
||||
asset rows. When the site setting `dualpath_single_machine` is on (default), the
|
||||
machines list, dashboard/report counts, and the floor map show the pair as a
|
||||
single machine (the SECONDARY bay is hidden); the data model always keeps both
|
||||
rows and detail pages stay per-bay.
|
||||
|
||||
```python
|
||||
from shopdb.api import resolve_dualpath_pairs, dualpath_single_machine_enabled
|
||||
|
||||
collapse = resolve_dualpath_pairs()
|
||||
# collapse.secondaryassetids: set of the non-primary bay asset ids to hide
|
||||
# collapse.partnerbyasset: {assetid -> {'assetid', 'assetnumber'}} for every
|
||||
# pair member (primary and secondary), for banners
|
||||
|
||||
if dualpath_single_machine_enabled():
|
||||
# exclude the hidden bays and annotate the visible (primary) bay
|
||||
...
|
||||
```
|
||||
|
||||
PRIMARY is the pair member with the lower natural-sort assetnumber.
|
||||
`resolve_dualpath_pairs` ignores the toggle (so a detail-page sibling banner can
|
||||
show always); gate the collapse itself on `dualpath_single_machine_enabled()`.
|
||||
|
||||
### Import mode (legacy timestamp passthrough)
|
||||
|
||||
Bulk imports from the classic ASP shopdb need to preserve each row's original
|
||||
`createddate` / `modifieddate` instead of stamping "now". `apply_import_timestamps`
|
||||
does this, gated so it never affects normal traffic: it only acts when the
|
||||
caller is an admin AND sent the `X-Import-Mode: true` request header.
|
||||
|
||||
```python
|
||||
from shopdb.api import apply_import_timestamps
|
||||
|
||||
asset = Asset(assetnumber=data['assetnumber'], ...)
|
||||
db.session.add(asset)
|
||||
# In import mode, stamp legacy createddate/modifieddate from the payload.
|
||||
# No-op for normal callers, or when the payload omits the fields.
|
||||
apply_import_timestamps(asset, data)
|
||||
db.session.commit()
|
||||
```
|
||||
|
||||
`import_mode_active()` returns the same admin-plus-header predicate, for guarding
|
||||
other backdated behavior (for example accepting a historical `checkouttime`).
|
||||
`parse_import_datetime(value)` parses both ISO `2020-01-05T12:00:00` and legacy
|
||||
`YYYY-MM-DD HH:MM:SS` into naive UTC. See [docs/IMPORT-API.md](IMPORT-API) for
|
||||
the full migration operator manual.
|
||||
|
||||
## Removed hooks
|
||||
|
||||
The following hooks existed in early drafts and have been removed for v1:
|
||||
|
||||
| Hook | Reason |
|
||||
|------|--------|
|
||||
| `get_event_handlers` | Event bus deferred indefinitely. No real use case yet. Add via new ADR if needed. |
|
||||
|
||||
## Versioning your changes
|
||||
|
||||
When you change anything documented here, you must:
|
||||
|
||||
1. Bump `__contract_version__` per [ADR-002](ADR-002-plugin-versioning): major for removals or signature changes, minor for additive optional hooks, patch for docs.
|
||||
2. Update [ADR-001](ADR-001-asset-as-platform-contract) if the contract surface itself changed (or supersede with a new ADR).
|
||||
3. Add or update the test in `tests/test_plugin_contract.py` that asserts the new behavior.
|
||||
|
||||
The skill `defining-asset-contract` walks through the full checklist.
|
||||
191
PLUGIN-QUICKSTART.md
Normal file
191
PLUGIN-QUICKSTART.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# Plugin Quickstart
|
||||
|
||||
Build a working shopdb-flask plugin in 30 minutes. This walks through generating, customizing, installing, and testing a plugin from scratch.
|
||||
|
||||
For the full hook reference, see [PLUGIN-HOOKS.md](PLUGIN-HOOKS).
|
||||
For the architectural decisions behind the contract, see [docs/adr/](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/docs/adr/).
|
||||
|
||||
## Step 1: Generate the skeleton
|
||||
|
||||
```bash
|
||||
flask plugin new cameras --description "Tracks shop-floor surveillance cameras"
|
||||
```
|
||||
|
||||
Output: `plugins/cameras/` with manifest, plugin class, example model, example routes, schemas stub, tests, a README, and a paste-in `frontend-api-snippet.js`. When a `frontend/src/` tree is present, it also writes the frontend starting points: `frontend/src/views/cameras/CamerasList.vue`, `CamerasDetail.vue`, `CamerasForm.vue`, and the auto-discovered route file `frontend/src/router/routes/cameras.js`. (A plugin developed in its own repo, with no frontend tree, gets the backend skeleton plus the snippet only.)
|
||||
|
||||
The generated plugin already passes the framework's contract tests. Verify before editing:
|
||||
|
||||
```bash
|
||||
pytest plugins/cameras/tests/
|
||||
```
|
||||
|
||||
## Step 2: Edit the model
|
||||
|
||||
Open `plugins/cameras/models/cameras.py`. Replace the `examplefield` placeholder with your domain fields:
|
||||
|
||||
```python
|
||||
class Cameras(BaseModel):
|
||||
__tablename__ = 'cameras'
|
||||
|
||||
assetid = db.Column(
|
||||
db.Integer,
|
||||
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
|
||||
primary_key=True,
|
||||
)
|
||||
streamurl = db.Column(db.String(255), nullable=False)
|
||||
resolution = db.Column(db.String(20))
|
||||
fps = db.Column(db.Integer)
|
||||
poeport = db.Column(db.String(50))
|
||||
asset = db.relationship('Asset', backref=db.backref('cameras', uselist=False))
|
||||
|
||||
def to_dict(self):
|
||||
return {
|
||||
'assetid': self.assetid,
|
||||
'streamurl': self.streamurl,
|
||||
'resolution': self.resolution,
|
||||
'fps': self.fps,
|
||||
'poeport': self.poeport,
|
||||
}
|
||||
```
|
||||
|
||||
Note the naming convention: lowercase concatenated, no underscores (`streamurl`, not `stream_url`). See [CONTRIBUTING.md](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/CONTRIBUTING.md).
|
||||
|
||||
## Step 3: Add routes
|
||||
|
||||
Open `plugins/cameras/api/routes.py`. The scaffold provides list and detail endpoints. Add CRUD as needed:
|
||||
|
||||
```python
|
||||
@cameras_bp.route('', methods=['POST'])
|
||||
@jwt_required()
|
||||
def create_camera():
|
||||
data = request.get_json()
|
||||
|
||||
asset = Asset(assetnumber=data['assetnumber'], name=data['name'], ...)
|
||||
db.session.add(asset)
|
||||
db.session.flush()
|
||||
|
||||
camera = Cameras(
|
||||
assetid=asset.assetid,
|
||||
streamurl=data['streamurl'],
|
||||
resolution=data.get('resolution'),
|
||||
)
|
||||
db.session.add(camera)
|
||||
db.session.commit()
|
||||
|
||||
return success_response(camera.to_dict(), http_code=201)
|
||||
```
|
||||
|
||||
For audit logging, use the public helper:
|
||||
|
||||
```python
|
||||
from shopdb.api import audit_log
|
||||
|
||||
audit_log(action='created', entitytype='Camera', entityid=asset.assetid, entityname=asset.name)
|
||||
```
|
||||
|
||||
## Step 4: Install the plugin
|
||||
|
||||
```bash
|
||||
flask plugin install cameras
|
||||
flask db migrate -m "Add cameras plugin tables"
|
||||
flask db upgrade
|
||||
```
|
||||
|
||||
`install` runs the plugin's `on_install` hook (which seeds the AssetType row), registers it in the plugin registry, and runs migrations.
|
||||
|
||||
## Step 5: Verify it works
|
||||
|
||||
```bash
|
||||
flask plugin list
|
||||
```
|
||||
|
||||
You should see `cameras [Enabled]`.
|
||||
|
||||
Run the plugin's tests:
|
||||
|
||||
```bash
|
||||
pytest plugins/cameras/tests/
|
||||
```
|
||||
|
||||
Hit the API:
|
||||
|
||||
```bash
|
||||
curl http://localhost:5001/api/cameras
|
||||
```
|
||||
|
||||
## Step 6: Add hooks (optional)
|
||||
|
||||
Override hooks on the plugin class as needed. See [PLUGIN-HOOKS.md](PLUGIN-HOOKS) for the full list. Common ones:
|
||||
|
||||
| Hook | Adds |
|
||||
|------|------|
|
||||
| `get_navigation_items` | Plugin shows up in the sidebar nav |
|
||||
| `get_dashboard_widgets` | Plugin's dashboard widget appears on the home page |
|
||||
| `get_reports` | Plugin's report cards appear on the Reports hub |
|
||||
| `get_settings_cards` | Plugin's card joins the settings rail + landing (no `settingsNav.js` edit) |
|
||||
| `get_permissions` | Plugin's RBAC permissions join the catalog, seeding, role grid, and token scopes |
|
||||
| `get_asset_panels` | Plugin panel renders on matching asset-detail pages |
|
||||
| `get_map_overlays` | Plugin decorates shop-floor map markers + adds a legend entry |
|
||||
| `get_asset_presentation` | Plugin declares its asset type's search icon + detail route |
|
||||
| `get_collector_schema` + `apply_collector_payload` | Plugin accepts external pushes at `/api/collector/<name>` |
|
||||
|
||||
Each hook has a default that does nothing. Override only what your plugin needs.
|
||||
|
||||
## Step 7: Frontend (finish the generated starting points)
|
||||
|
||||
The scaffold generates the frontend starting points too: three views, a route file, and a paste-in api-client snippet (see Step 1). They build and run out of the box against the example model, so `npm run build` is green immediately after scaffolding. The list below is what you finish by hand once the views exist. Copy patterns from the closest bundled plugin (`network` is the cleanest) as you flesh them out, and work through these in order:
|
||||
|
||||
1. **View files** - the scaffold created `frontend/src/views/cameras/CamerasList.vue`, `CamerasDetail.vue`, `CamerasForm.vue` from the example model. Replace the `examplefield` columns and inputs with your domain fields. Keep the global `.filters` / `.form-control` / `.card` styles; do not invent per-page input styling.
|
||||
|
||||
2. **Route file** - the scaffold created `frontend/src/router/routes/cameras.js` exporting a route array. The router auto-discovers every file in `routes/` via `import.meta.glob`, so no registration edit is needed. Every route is already tagged with `meta: { plugin: 'cameras' }` - the ADR-009 guard redirects to the dashboard when the backend plugin is disabled - and the form routes already carry `requiresAuth: true`:
|
||||
|
||||
```js
|
||||
export default [
|
||||
{
|
||||
path: 'cameras',
|
||||
name: 'cameras',
|
||||
component: () => import('../../views/cameras/CamerasList.vue'),
|
||||
meta: { plugin: 'cameras' }
|
||||
},
|
||||
{
|
||||
path: 'cameras/:id/edit',
|
||||
name: 'cameras-edit',
|
||||
component: () => import('../../views/cameras/CamerasForm.vue'),
|
||||
meta: { requiresAuth: true, plugin: 'cameras' }
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
3. **API client** - the scaffolded views ship with an inline `camerasApi` client so they run standalone. To graduate to the shared module, paste the generated `plugins/cameras/frontend-api-snippet.js` block into `frontend/src/api/index.js`, then delete the inline const in each view and `import { camerasApi } from '../../api'` instead. The snippet already matches the existing blocks' shape (`list(params)`, `get(id)`, `create(data)`, `update(id, data)`, `remove(id)`).
|
||||
|
||||
4. **Sidebar entry** - implement `get_navigation_items` on the plugin class. No frontend edit: the sidebar builds itself from `/api/dashboard/navigation`.
|
||||
|
||||
5. **Report cards** (if any) - implement `get_reports` on the plugin class. No frontend edit: the Reports hub builds itself from `/api/reports`. Use `route` for a dedicated page (add it to your route file), or `endpoint` for inline rendering.
|
||||
|
||||
6. **Settings page** (if the plugin has subtypes) - add a route whose path starts with `settings/` (e.g. `settings/cameratypes`) to your route file; the router automatically nests it under the settings shell. Copy a types-list view from `frontend/src/views/settings/`.
|
||||
|
||||
7. **Verify** - `npm run build` must pass, then screenshot your pages against the dev servers: `venv/bin/python tools/shot.py /cameras`.
|
||||
|
||||
## Common errors
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `PluginNotFoundError: manifest.json` | Manifest deleted or moved | Restore `plugins/<name>/manifest.json` |
|
||||
| `PluginContractError: missing required field` | manifest.json incomplete | Re-add `name`, `version`, `description` |
|
||||
| `PluginVersionError: requires core_version X but framework is Y` | Framework upgraded past your range | Update `core_version` in manifest |
|
||||
| `Table 'cameras' is already defined` | Two models declared the same `__tablename__` | Pick a unique table name |
|
||||
| Index name collision | Two indexes share the same name (SQLite enforces global uniqueness) | Prefix index names with table: `idx_cameras_streamurl` |
|
||||
|
||||
## Next steps
|
||||
|
||||
- [PLUGIN-GUIDE.md](PLUGIN-GUIDE) - the full narrative walkthrough of building the `measuringtools` plugin end to end (models, per-plugin migration baseline, authz, hooks, frontend integration, tests). Read this after the quickstart when you want the exemplar that exercises every framework feature.
|
||||
- [PLUGIN-HOOKS.md](PLUGIN-HOOKS) for the full hook reference
|
||||
- [CONTRIBUTING.md](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/CONTRIBUTING.md) for naming conventions
|
||||
- [docs/adr/ADR-001-asset-as-platform-contract.md](ADR-001-asset-as-platform-contract) for what your plugin can rely on
|
||||
- [docs/adr/ADR-006-collector-contract.md](ADR-006-collector-contract) for accepting external collector input
|
||||
|
||||
## Distribution
|
||||
|
||||
If you are building a plugin for a specific GE Aerospace site (sister-site adoption), ship it as its own git repo. The site running shopdb-flask clones or symlinks your plugin into `<repo>/plugins/<name>/`. See [ADR-003](ADR-003-plugin-distribution).
|
||||
|
||||
For the full own-repo workflow (layout, symlink dev loop, CI recipe with `scripts/test-external-plugin.sh`, version pinning), see [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO). For what you can rely on staying stable before contract 1.0, see [CONTRACT-STABILITY.md](CONTRACT-STABILITY).
|
||||
72
PLUGINS.md
72
PLUGINS.md
@@ -0,0 +1,72 @@
|
||||
# Plugins
|
||||
|
||||
shopdb-flask is a framework. The plugins listed here are the pieces other GE Aerospace facilities can install, build, or skip per ADR-003. Bundled plugins ship in the framework repo. External plugins live in their own repos and drop into `<repo>/plugins/<name>/` at install time.
|
||||
|
||||
## Bundled (ship with the framework)
|
||||
|
||||
These plugins are in `plugins/` in this repo. Enable per site with `flask plugin install <name>`.
|
||||
|
||||
| Plugin | Tracks | Notes |
|
||||
|--------|--------|-------|
|
||||
| `machines` | Manufacturing machinery: 5-axis mills, lathes, broachers, heat treatment ovens | Manually entered. See [ADR-005](ADR-005-equipment-vs-measuringtools). Subtype tables for FOCAS / CLM / MTConnect controller protocols (planned). |
|
||||
| `computers` | Shop-floor PCs and engineering workstations | Fed by the PXE pipeline collector per [ADR-006](ADR-006-collector-contract). |
|
||||
| `printers` | Network and shop-floor printers | Optional Zabbix integration for supply tracking. Legacy `PrinterData` retiring per ADR-001. |
|
||||
| `network` | Switches, routers, access points, IDFs as locations | Asset-only; cleanest of the bundled set. |
|
||||
| `usb` | USB devices issued to shop-floor users | Lightweight checkout / check-in. |
|
||||
| `notifications` | Shop-floor notifications, recognitions, kiosk feed | Used by `ShopfloorDashboard.vue`. |
|
||||
| `measuringtools` | Metrology and inspection instruments: calipers, micrometers, thread/bore/height gages, indicators | Per [ADR-005](ADR-005-equipment-vs-measuringtools). Calibration lifecycle with derived status. First plugin built on the matured scaffold; its walkthrough is [PLUGIN-GUIDE.md](PLUGIN-GUIDE). Ships `default_enabled: false`. |
|
||||
|
||||
## Building your own
|
||||
|
||||
Guides:
|
||||
|
||||
- [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART) - generate, customize, install, and test a plugin in 30 minutes using `flask plugin new`.
|
||||
- [PLUGIN-GUIDE.md](PLUGIN-GUIDE) - the full narrative walkthrough of building the `measuringtools` plugin, the exemplar that exercises every current framework feature (models, per-plugin migrations, authz, hooks, frontend integration, tests).
|
||||
- [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO) - developing a plugin in its own repo per ADR-003: repo layout, symlink dev workflow, `core_version` pinning, and a runnable CI harness (`scripts/test-external-plugin.sh`) that tests the plugin against a pinned framework ref.
|
||||
- [CONTRACT-STABILITY.md](CONTRACT-STABILITY) - path to contract 1.0: what is settled vs still churning, the bump rules, and how much a sister site can safely build on today.
|
||||
|
||||
The contract is locked in [ADR-001](ADR-001-asset-as-platform-contract) and versioned per [ADR-002](ADR-002-plugin-versioning).
|
||||
|
||||
Quick path:
|
||||
|
||||
```bash
|
||||
flask plugin new cameras --description "Tracks shop-floor surveillance cameras"
|
||||
# edit plugins/cameras/models/cameras.py with your fields
|
||||
flask plugin install cameras
|
||||
```
|
||||
|
||||
## Migrations (per-plugin chains)
|
||||
|
||||
Each plugin that owns tables carries its own Alembic chain under
|
||||
`plugins/<name>/migrations/`, with a per-plugin version table
|
||||
`alembic_version_<name>` independent of the core `alembic_version`. Ownership is
|
||||
split at a fixed cutover (see [ADR-008](ADR-008-plugin-migration-ownership)):
|
||||
|
||||
- The core chain (`flask db upgrade`) created every table that existed through
|
||||
its head, including the bundled-plugin tables. Each bundled plugin's `0001`
|
||||
migration is a stamp-only no-op anchor recording that fact.
|
||||
- From the cutover forward, a change to a plugin's schema lands as
|
||||
`plugins/<name>/migrations/versions/000N_*.py`, never in the core chain. The
|
||||
core chain is reserved for core tables.
|
||||
- A plugin built AFTER the cutover (e.g. `measuringtools`) is different: the
|
||||
core chain never created its tables, so its `0001` is a REAL baseline that
|
||||
creates them, not a no-op anchor. See [PLUGIN-GUIDE.md](PLUGIN-GUIDE) for
|
||||
the anchor-vs-baseline distinction.
|
||||
|
||||
Deploys and upgrades run `flask db upgrade` then `flask plugin upgrade-all`.
|
||||
`upgrade-all` stamps every plugin anchor and applies any later plugin
|
||||
migrations; it is idempotent. The registry (`instance/plugins.json`) records
|
||||
which revisions each plugin has applied in `migrations_applied`.
|
||||
|
||||
## Distribution conventions
|
||||
|
||||
For sister-site plugins (per [ADR-003](ADR-003-plugin-distribution)):
|
||||
|
||||
- Plugin lives in its own git repo: `gitea.proudtech.net/<your-site>/<pluginname>`
|
||||
- Adopting site clones or symlinks into their `<repo>/plugins/<name>/`
|
||||
- Plugin manifest declares `core_version` range matching the framework version they target
|
||||
- Plugin readme explains: what it tracks, who maintains it, where to file issues
|
||||
|
||||
## Naming policy
|
||||
|
||||
Plugin names follow the framework's naming convention (lowercase concatenated, no underscores or dashes; full words preferred over acronyms). See [CONTRIBUTING.md](https://gitea.proudtech.net/ge-aerospace/shopdb-flask/src/branch/main/CONTRIBUTING.md). Plugin name collisions across sites are not enforced; the convention recommends prefixing site-specific plugins with the site code (e.g., `wjsf-shippingstation`) when there is risk of overlap.
|
||||
|
||||
648
PROPOSAL-ge-enforce-plugin.md
Normal file
648
PROPOSAL-ge-enforce-plugin.md
Normal file
@@ -0,0 +1,648 @@
|
||||
# Proposal: GE-Enforce as a shopdb plugin
|
||||
|
||||
Status: DRAFT / planning only. Not accepted, not built.
|
||||
Author: planning session 2026-07-12.
|
||||
|
||||
## 1. What this is
|
||||
|
||||
Today GE-Enforce is a PowerShell manifest engine that reads per-PC-type
|
||||
`manifest.json` files off an SMB share (`\\tsgwp00525.wjs.geaerospace.net\
|
||||
shared\dt\shopfloor\`). Each logon, a scheduled task running as SYSTEM mounts
|
||||
the share, reads the manifest for the machine's PC type, and installs or
|
||||
self-heals apps, files, drivers, registry values, and scripts. A parallel
|
||||
`preinstall.json` runs the same schema once at imaging.
|
||||
|
||||
This proposal turns the *manifest* into shopdb data: the authoritative manifest
|
||||
lives in the shopdb database, is edited through the shopdb UI (an expansion of
|
||||
`/settings/pctypemapping`), and is served to clients over HTTP as JSON. The
|
||||
*payloads* (MSI/EXE/PS1/config bytes) stay on SMB, on HTTP, or both, referenced
|
||||
by URL/path from the manifest rows. GE-Enforce.ps1 changes from "read a file on
|
||||
W:" to "GET a manifest from shopdb, then fetch each payload from wherever the
|
||||
row says."
|
||||
|
||||
The result: managing imaging PC types, their apps, scripts, files, registry
|
||||
rules, and version gates becomes a first-class shopdb feature instead of hand-
|
||||
edited JSON on a file share.
|
||||
|
||||
## 2. Why it fits shopdb
|
||||
|
||||
- shopdb already models the fleet (the collector ingests every PC's hostname,
|
||||
pctype, installed software, versions). Making shopdb *also* own what SHOULD be
|
||||
installed closes the loop: desired-state (manifest) and observed-state
|
||||
(collector) live in one system and can be diffed.
|
||||
- `/settings/pctypemapping` already maps `gea-shopfloor-*` PC types to
|
||||
`ComputerType`. That page becomes the entry point for full imaging-PC-type
|
||||
management.
|
||||
- The plugin contract (per-plugin models, migrations, API prefix, settings
|
||||
cards, collector hooks) is exactly the shape this needs.
|
||||
- ADR-004 (per-site instances) matches: each site's shopdb owns each site's
|
||||
manifest. No multi-tenant complication.
|
||||
|
||||
## 3. Grounding: the real manifest schema
|
||||
|
||||
Source of truth for these field names (do not invent others):
|
||||
|
||||
- Schema: `pxe-images/tsgwp00525-v2/shared/dt/shopfloor/_meta/manifest-schema.json`
|
||||
- Engine: `pxe-images/common/lib/Install-FromManifest.ps1`
|
||||
- Dispatcher: `.../shopfloor/common/GE-Enforce.ps1`
|
||||
- Architecture: `pxe/docs/ge-enforce-v2-architecture.md`
|
||||
|
||||
A manifest is `{ "Version": str, "_comment": str, "Applications": [entry, ...] }`.
|
||||
Only `Name` and `Type` are required per entry.
|
||||
|
||||
### Per-entry fields (complete set)
|
||||
|
||||
Identity / action:
|
||||
- `Name` (required, unique, also the status-key `<scope>/<Name>`)
|
||||
- `Type` (required): one of `MSI EXE CMD BAT PS1 INF File Registry`
|
||||
- `_comment` (documentation, heavily used in practice)
|
||||
|
||||
Type-specific payload references (sparse; depends on Type):
|
||||
- MSI/EXE/CMD/BAT/INF: `Installer` (relative path) + `InstallArgs`
|
||||
- PS1: `Script` (relative path, falls back to `Installer`) + `Args`
|
||||
- File: `Source` (relative) + `Destination` (absolute on-PC path)
|
||||
- Registry: `RegPath` + `RegName` + `RegValue` + `RegType`
|
||||
(`RegType` in `String DWord QWord MultiString ExpandString Binary`)
|
||||
- Optional `LogFile`, `WaitTimeoutSec` (EXE hang kill), `InUseCheck`
|
||||
|
||||
Detection (decides whether the action fires / self-heals):
|
||||
- `DetectionMethod`: one of
|
||||
`Registry File FileVersion Hash MarkerFile ValueMatches pnputil Always`
|
||||
- `DetectionPath`, `DetectionName`, `DetectionValue`, `DetectionPattern`
|
||||
- Note: `DetectionValue` is method-dependent - SHA256 for Hash, a 4-part
|
||||
version for FileVersion, a registry value for Registry, ignored for
|
||||
Always/File. Same column, different meaning per method.
|
||||
- No `DetectionMethod` = always installs.
|
||||
|
||||
Targeting filters (all ANDed; each is multi-value):
|
||||
- `PCTypes` (array; `"*"` = all; alias graph expands old<->new names)
|
||||
- `PCSubTypes` / subtype via `<pctype>-<subtype>` values
|
||||
- `TargetHostnames` (array; exact + `-like WJS-*` wildcards)
|
||||
- `TargetMachineNumbers` (array; per-bay)
|
||||
- `_CmmVersion` (scalar; per-entry PC-DMIS version gate, needs lib >= 2.6)
|
||||
|
||||
Nested:
|
||||
- `InUseCheck`: `{ Behavior, Processes: [{Name, ExePath, GracefulCloseTimeoutSec}] }`
|
||||
Behavior in `Defer CloseAndReopen ForceClose ScheduleForReboot`
|
||||
|
||||
Parsed-but-inert today (model them, mark inert):
|
||||
- `ApplyMode` (`Nightly Immediate ImmediateReboot`), `UpdateWindow` (`HH:MM-HH:MM`)
|
||||
|
||||
Preinstall-only extras (phase discriminator):
|
||||
- `PreEnrollment`, `KillAfterDetection`, `PCTypesStrict`, `_pcTypesNote`
|
||||
|
||||
### Load-bearing behaviors the model must preserve
|
||||
|
||||
1. **Array order IS execution order.** Config-restore entries are deliberately
|
||||
placed AFTER their vendor installer so a mid-cycle overwrite heals the same
|
||||
cycle (eMxInfo.txt after eDNC; udc_webserver_settings after UDC). We MUST
|
||||
store an explicit per-scope `sortorder`, not a set.
|
||||
2. **PCTypes alias graph** is many-to-many old<->new names resolved by set
|
||||
intersection, with a `PCTypesStrict` escape hatch. Not a simple FK.
|
||||
3. **Polymorphic entry by Type** - sparse column set per type. DECISION: one
|
||||
wide `manifestentries` table with an `entrytype` discriminator column and
|
||||
nullable per-type columns. NOT SQLAlchemy STI subclasses, NOT a JSON blob.
|
||||
Justification (section 4): the whole fleet is ~64 entries, so sparse columns
|
||||
cost nothing; real columns get validated, indexed, field-diffed, joined
|
||||
against collector data, and read in plain SQL by an IT tech - a JSON blob
|
||||
hides all of that, and class-per-type STI is expert ceremony for no gain. A
|
||||
`validate()` that switches on `entrytype` (mirroring the engine's own
|
||||
`switch ($App.Type)`) is ~40 obvious lines.
|
||||
4. **Two manifest phases** - runtime (self-heal, per logon) and preinstall
|
||||
(once at imaging) share the schema. One table with a `phase` discriminator.
|
||||
|
||||
## 4. Data model (new `geenforce` plugin)
|
||||
|
||||
Per-plugin Alembic chain (ADR-008). Tables (lowercase concatenated per naming
|
||||
convention). Sizing that shapes every decision here: the real fleet is 10
|
||||
runtime scopes = 43 entries, plus 1 preinstall manifest = 21 entries, so ~64
|
||||
rows total. That smallness is why this stays deliberately low-tech (one wide
|
||||
table, JSON-document snapshots, no row-mirroring) - the design target is an
|
||||
average site IT tech maintaining it, not a specialist.
|
||||
|
||||
- `manifestscopes` - one row per imaging PC type / scope.
|
||||
- `scopeid` PK
|
||||
- `scopename` (e.g. `gea-shopfloor-cmm`)
|
||||
- `phase` enum (`runtime` | `preinstall`)
|
||||
- UNIQUE (`scopename`, `phase`), NOT `scopename` alone: `common` exists in
|
||||
runtime, and a scope name can appear in both phases. Note the phases are
|
||||
shaped differently - runtime is many per-pctype scopes (one manifest file
|
||||
each), preinstall is ONE flat manifest gated internally by `PCTypes`, so
|
||||
preinstall is modeled as a single `phase=preinstall` scope, not per-pctype
|
||||
scopes.
|
||||
- `computertypeid` FK -> `computertypes` (this REPLACES the thin
|
||||
`pctypemap_<pxetype>` setting; the mapping becomes a column here).
|
||||
Runtime-scope only; null for the preinstall scope.
|
||||
- `measuringtooltypeid` FK -> `measuringtooltypes`, nullable (metrology
|
||||
scopes: what device this scope implies; keeps imaging + collector agreed,
|
||||
see section 11).
|
||||
- `manifestversion` (string, mirrors manifest `Version`)
|
||||
- `description`, `isactive`
|
||||
- `iscommon` bool (the `common/` fleet-wide scope)
|
||||
|
||||
- `manifestentries` - one row per Applications[] entry (the working/draft copy).
|
||||
- `entryid` PK, `scopeid` FK
|
||||
- `sortorder` int (preserves array order; the ordering contract)
|
||||
- `name`, `entrytype` (MSI/EXE/.../Registry), `comment`
|
||||
- payload columns (nullable, per type): `installer`, `installargs`,
|
||||
`scriptpath`, `scriptargs`, `sourcepath`, `destination`,
|
||||
`regpath`, `regname`, `regvalue`, `regtype`
|
||||
- `payloadsource` enum (`smb` | `http` | `inline`) + `payloadref`
|
||||
(see section 5)
|
||||
- `payloadsha256` - integrity hash of the payload bytes, INDEPENDENT of the
|
||||
detection method. Mandatory for `http`/`inline` payloads; optional for
|
||||
`smb`. Do NOT reuse `detectionvalue` for this - `detectionvalue` is a
|
||||
SHA256 only when `detectionmethod = Hash`; an MSI with `Registry`/
|
||||
`FileVersion` detection has no payload hash, so an HTTP fetch would
|
||||
otherwise run unverified bytes (see section 5).
|
||||
- `regvalue` stores the RAW JSON literal (`1` vs `"1"`) and is emitted
|
||||
verbatim on export. `RegValue` is untyped in the manifest schema and real
|
||||
entries carry numbers; the engine string-coerces for `ValueMatches` but
|
||||
`Set-ItemProperty -Type DWord` cares, so preserve the literal.
|
||||
- detection columns: `detectionmethod`, `detectionpath`, `detectionname`,
|
||||
`detectionvalue`, `detectionpattern`
|
||||
- gates: `cmmversion`, plus child tables for the multi-value filters
|
||||
- control: `logfile`, `waittimeoutsec`, `applymode`, `updatewindow`
|
||||
(`applymode`/`updatewindow` are parsed-but-INERT in the engine today; the
|
||||
UI must label them "not yet enforced" so a tech does not trust a dead gate)
|
||||
- preinstall flags: `preenrollment`, `killafterdetection`, `pctypesstrict`
|
||||
- `isactive`
|
||||
|
||||
- `manifestpublishedversions` - immutable published snapshots, SIMPLIFIED to
|
||||
freeze the rendered JSON DOCUMENT in a single `manifestjson` column (drop the
|
||||
row-mirrored `manifestpublishedentries` family the earlier draft proposed).
|
||||
The only consumer of a snapshot is the client, and it consumes exactly that
|
||||
document, so freezing the text makes immutability structural (no UPDATE path),
|
||||
rollback a one-flag `iscurrent` flip, serving a single-row read, and version
|
||||
diffing a plain text diff - all things average IT can debug; row-mirroring
|
||||
would add ~6 shadow tables and a copy routine that can drift. Columns:
|
||||
`publishedversionid`, `scopeid`, `versionnumber` (1,2,3 per scope),
|
||||
`manifestjson` (MEDIUMTEXT, verbatim), `publishedat`, `publishedby`,
|
||||
`iscurrent`, `notes`. Editing `manifestentries` never affects the fleet;
|
||||
"publish" freezes a new snapshot; the client is ALWAYS served the current
|
||||
snapshot, never the live draft. Rollback = flip `iscurrent` to an older
|
||||
version (the post-cutover safety net once the on-share JSON is retired).
|
||||
Mirrors today's `_meta/history/<date>-<scope>.json` backups, but authoritative.
|
||||
Revision history: every publish is a permanent, immutable revision kept
|
||||
indefinitely (snapshots are small JSON text, ~10 scopes - storage is a
|
||||
non-issue). An OPTIONAL retention policy (keep last M per scope, or prune
|
||||
older than N months) can be added later; default is keep-everything, off.
|
||||
|
||||
- Draft-edit audit trail (field-level history BETWEEN publishes): drafts
|
||||
(`manifestentries`) are not versioned - editing overwrites the working copy.
|
||||
To answer "who changed this entry and when" in the window between two
|
||||
published revisions, log every draft mutation through the EXISTING core audit
|
||||
system (no new table): on create/update/delete of a scope, entry, or child
|
||||
row, write an audit record with the actor, timestamp, entry name, and the
|
||||
changed field(s). This gives per-edit provenance for free and shows up in the
|
||||
same Audit Logs UI IT already uses; the published snapshots remain the
|
||||
coarse-grained "what the fleet actually got" record.
|
||||
|
||||
- `manifestentrypctypes`, `manifestentryhostnames`, `manifestentrymachinenumbers`
|
||||
- child rows for the ANDed multi-value filters (one value + a `sortorder` per
|
||||
row, wildcards stored verbatim as patterns)
|
||||
|
||||
- `manifestinusechecks` + `manifestinusecheckprocesses`
|
||||
- the nested InUseCheck object and its Processes[] child list (leave
|
||||
`gracefulclosetimeoutsec` nullable; do not bake the engine's default of 10
|
||||
into the row, emit it only when set)
|
||||
|
||||
- `manifestpayloads` - inline payload bytes for `payloadsource = inline`
|
||||
(`entryid`, `filename`, `contenttype`, `payloadbytes` LONGBLOB, `payloadsha256`,
|
||||
`uploadedat`). App-enforced size cap ~1 MB; the upload UI rejects larger with
|
||||
"use SMB for this" so nobody pastes an MSI into the database. Can ship empty
|
||||
and unused until P6.
|
||||
|
||||
- `pctypealiases` - a MIRROR of the old<->new name alias graph from
|
||||
`Install-FromManifest.ps1:463-475`, for server-side resolve/validate only.
|
||||
The engine lib stays the single source of truth (see section 10); shopdb
|
||||
never becomes the authority the client depends on for aliases.
|
||||
|
||||
The JSON the client receives is REBUILT from a published snapshot in exact
|
||||
array order. Parity with the current engine is proven by BEHAVIORAL equivalence,
|
||||
not byte-identity (see section 9): re-serialized JSON will differ in key order
|
||||
and whitespace, so the test is that both manifests parse to the same ordered
|
||||
entry set with the same detection/targeting/action semantics.
|
||||
|
||||
## 5. Payloads: SMB and/or HTTP (both supported)
|
||||
|
||||
The user asked whether payloads can be SMB and/or HTTP. Yes - per entry:
|
||||
|
||||
- `payloadsource = smb`: `payloadref` is the current relative path
|
||||
(`apps/eDNC_6-4-5.msi`); the client still mounts W: and resolves it against
|
||||
the scope root exactly as today. The engine is unchanged for these rows (the
|
||||
mount + scope-root resolution still happen; an HTTP-only site skips the mount
|
||||
because it has no `smb` rows). This is the default and the migration target
|
||||
for large binaries (MSIs are hundreds of MB; SMB streaming beats HTTP).
|
||||
- `payloadsource = http`: `payloadref` is a URL (absolute, or relative to a
|
||||
configured payload base). The client downloads to a local temp dir, verifies
|
||||
the Hash/FileVersion detection value, then runs it. Good for small
|
||||
config/script payloads and for sites with no SMB share.
|
||||
- `payloadsource = inline`: for small text payloads (a `.ps1`, a config file, a
|
||||
registry value), the bytes live in shopdb itself and are served in-band. No
|
||||
external store at all. Best for scripts and File-type config drops.
|
||||
|
||||
Manifest generation emits, per entry, whatever the client needs to fetch the
|
||||
bytes. The engine's existing "stage network EXE to local temp first" logic
|
||||
(SYSTEM access-denied workaround) generalizes cleanly to HTTP download.
|
||||
|
||||
Payload integrity uses the dedicated `payloadsha256` column, NOT `DetectionValue`.
|
||||
This is the correction to a subtle trap: `DetectionValue` is a SHA256 only when
|
||||
`DetectionMethod = Hash`. Most binaries detect by `Registry` or `FileVersion`
|
||||
and carry no payload hash at all, so relying on `DetectionValue` would let an
|
||||
HTTP/inline-fetched MSI run unverified. Instead, publishing an `http`/`inline`
|
||||
payload computes and stores `payloadsha256`, and the client verifies the fetched
|
||||
bytes against it BEFORE running, independent of how the entry detects install
|
||||
state. `smb` payloads may set it too (defense in depth) but the share ACL is
|
||||
their primary trust boundary. Detection stays a separate concern: it decides
|
||||
whether to act; the payload hash decides whether the bytes are trustworthy.
|
||||
|
||||
Transport security: the client fetches as SYSTEM, so the shopdb TLS cert must be
|
||||
trusted machine-wide. Sites with a self-signed or air-gapped shopdb need the CA
|
||||
in the machine trust store (provisioned by the same Azure DSC step that writes
|
||||
the token). Plain HTTP is acceptable only inside a trusted segment, and even
|
||||
then the `payloadsha256` check is what actually guarantees payload integrity.
|
||||
|
||||
## 6. API surface (`/api/geenforce/...`)
|
||||
|
||||
Two permissions via the plugin's `get_permissions()` hook (split so day-to-day
|
||||
techs can edit but only a lead ships to the fleet):
|
||||
- `geenforce.manage` - create/edit/reorder scopes, entries, drafts, payloads.
|
||||
- `geenforce.publish` - publish, rollback, export-to-share (the fleet-affecting
|
||||
actions).
|
||||
|
||||
Draft editing (`geenforce.manage`):
|
||||
- `GET/POST /scopes`, `GET/PUT/DELETE /scopes/<id>` - imaging PC types
|
||||
- `GET/POST /scopes/<id>/entries`, `PUT/DELETE /entries/<id>` - manifest entries
|
||||
- `PUT /scopes/<id>/entries/reorder` - the ordering contract; Move Up/Down in the
|
||||
UI (plain buttons + visible `sortorder`), not a drag-and-drop dependency
|
||||
- `POST /entries/<id>/payload` - upload an inline/http payload (multipart),
|
||||
compute + store its `payloadsha256` (the integrity hash; NOT `detectionvalue`)
|
||||
- `GET /scopes/<id>/preview` - the draft JSON a client WOULD receive on next
|
||||
publish; `GET /scopes/<id>/published` shows the currently-served snapshot
|
||||
- `GET /scopes/<id>/simulate?pctype=&subtype=&hostname=&machinenumber=&cmmversion=`
|
||||
- the "what would this PC get" simulator: runs the entry list through the same
|
||||
filter logic the engine uses and returns which entries apply and why the rest
|
||||
are filtered out. Reuses the P1 parity harness's filter engine, so it is
|
||||
nearly free, and it is the single most IT-empowering endpoint - it answers
|
||||
"why did/didn't app X install on PC Y" without reading a PowerShell log.
|
||||
|
||||
Publishing (`geenforce.publish`):
|
||||
- `POST /scopes/<id>/publish` - freeze the current draft into a new immutable
|
||||
`manifestpublishedversions` snapshot (this is what the fleet gets)
|
||||
- `POST /scopes/<id>/rollback/<version>` - mark an older snapshot current
|
||||
- `POST /scopes/<id>/export-share` (or a `flask geenforce export-share` CLI) -
|
||||
write the current published JSON to `<shareroot>/<scope>/manifest.json` after
|
||||
copying the existing file to `_meta/history/<date>-<scope>.json`. This is a
|
||||
first-class feature, not a footnote: it is the Milestone 1 product (author in
|
||||
shopdb, engine untouched) and the permanent break-glass path.
|
||||
|
||||
Client-facing (gated by a collector-style service token, `geenforce.fetch`
|
||||
scope, reusing the PAT + `X-API-Key` machinery already built for the collector):
|
||||
- `GET /manifest?pctype=<scope>&subtype=<s>&hostname=<h>&machinenumber=<n>`
|
||||
Returns the latest PUBLISHED snapshot for that scope (never the live draft).
|
||||
The server can pre-apply the PCTypes/hostname/machinenumber/cmmversion filters
|
||||
(thin client) OR return the full scope and let the engine filter (fat client,
|
||||
matches today). Start fat: return the scope manifest unchanged so the engine
|
||||
logic is untouched. Include the snapshot version + an ETag so the client can
|
||||
cache and no-op when unchanged.
|
||||
- Payload fetch for `http`/`inline` rows: `GET /payload/<entryid>` streaming the
|
||||
bytes; the client verifies them against `payloadsha256` from the manifest.
|
||||
|
||||
## 7. Frontend: expand `/settings/pctypemapping`
|
||||
|
||||
The current page (`PCTypeMappingSettings.vue`, "Collector PC Types") is a read-
|
||||
only-ish table of `pxetype -> ComputerType` dropdowns. It grows into the imaging-
|
||||
PC-type manager:
|
||||
|
||||
- **Scopes list**: add/rename/delete imaging PC types; each still carries its
|
||||
`ComputerType` mapping (that column moves from a setting into `manifestscopes`).
|
||||
A `phase` toggle (runtime vs preinstall). Common scope flagged.
|
||||
- **Scope detail / manifest editor**: an ordered list of entries with Move
|
||||
Up/Down buttons and a visible `sortorder` (the ordering contract made visible;
|
||||
NOT drag-and-drop - a drag library is the kind of dependency that breaks
|
||||
silently and average IT cannot fix; add drag later if wanted). Each entry is a
|
||||
typed form - the visible fields switch on `entrytype` (MSI shows
|
||||
Installer+InstallArgs; PS1 shows Script+Args; File shows Source+Destination;
|
||||
Registry shows the Reg* quartet), one line of help per detection method.
|
||||
Filter chips for PCTypes/hostnames/machine numbers. InUseCheck sub-editor.
|
||||
Payload source selector (smb/http/inline) with upload for the latter two.
|
||||
`applymode`/`updatewindow` sit behind an "Advanced (not yet enforced by the
|
||||
engine)" disclosure. Ship the editor in three usable-alone increments: (a)
|
||||
scope list + entry table, (b) the typed entry form, (c) publish + diff. That
|
||||
keeps the biggest chunk of the build from ballooning.
|
||||
- **Simulator ("what would this PC get")**: a small form (pctype, subtype,
|
||||
hostname, machine number, CMM version) that calls `GET /scopes/<id>/simulate`
|
||||
and lists which entries apply and why the rest are filtered. The single most
|
||||
IT-empowering piece of the UI.
|
||||
- **Draft, preview, publish**: editing changes only the draft; "publish" freezes
|
||||
an immutable snapshot (see section 4) and is what the fleet then gets. Show the
|
||||
draft-vs-published diff before publishing. Rollback republishes a prior
|
||||
snapshot.
|
||||
- **Desired vs observed (BUILT: observed-state reporting)**: rather than extend
|
||||
the collector, the plugin has its own reporting path. Each enforcement cycle a
|
||||
PC POSTs `POST /api/geenforce/report` (geenforce.report service token) with the
|
||||
published version it applied, the installed/skipped/failed/filtered counts, and
|
||||
per-entry outcomes. Stored in `manifestenforcementreports` (latest-per-host +
|
||||
history) and `manifestenforcementresults` (per-entry). Two payoffs fall out:
|
||||
RECEIVED - `receivedlatest` compares the applied version to the scope's current
|
||||
published version, so the fleet view shows which PCs picked up an update; and
|
||||
SELF-HEAL - each entry's action (installed = drift corrected, skipped = already
|
||||
good, failed) with any warning/error message. Admin reads: `GET /reports`
|
||||
(fleet compliance) and `GET /reports/<id>` (per-entry detail). This is the
|
||||
observed half that makes the manifest a closed desired-vs-observed loop.
|
||||
|
||||
This is an ADR-010 settings card contributed by the geenforce plugin, so it only
|
||||
appears when the plugin is enabled.
|
||||
|
||||
## 8. Client change (minimal, staged)
|
||||
|
||||
`GE-Enforce.ps1` today: mount W:, read `<scope>\manifest.json`, hand to
|
||||
`Install-FromManifest`. New path: GET the manifest from shopdb, write it to the
|
||||
same local location the engine reads, then run the engine unchanged. That is the
|
||||
smallest possible client delta - the engine, detection logic, self-heal, and
|
||||
SMB payload resolution all stay identical. Only the *source of the JSON* moves
|
||||
from file to HTTP.
|
||||
|
||||
Payloads: `smb` rows need no client change. `http`/`inline` rows need a small
|
||||
fetch-and-verify helper (download to temp, check SHA256, then the existing
|
||||
installer action runs against the local copy). The engine already stages network
|
||||
EXEs to temp, so this is an extension, not a rewrite.
|
||||
|
||||
Auth: the client already has SFLD credentials in
|
||||
`HKLM:\SOFTWARE\GE\SFLD\Credentials`. Add a shopdb service token (a
|
||||
`geenforce.fetch` PAT) provisioned the same way (Azure DSC writes it to
|
||||
registry), sent as `X-API-Key`. If shopdb is unreachable, the client falls back
|
||||
to the last-known-good manifest cached locally (fail-safe: never leave a PC
|
||||
unmanaged because the web app is down). This mirrors today's "creds missing =
|
||||
exit 0, retry next cycle" resilience.
|
||||
|
||||
## 9. Cutover strategy
|
||||
|
||||
The manifest is desired-state that runs as SYSTEM and installs software fleet-
|
||||
wide. A bad cutover = a fleet-wide mis-install. Stage it:
|
||||
|
||||
1. **Import + parity.** Write a one-shot importer that reads the current
|
||||
on-share manifests (common + every `gea-shopfloor-*` + preinstall.json;
|
||||
skip `.bak` / `.pre-mtconnect.bak` variants) into the new tables. Then
|
||||
generate JSON back out and prove BEHAVIORAL equivalence for every scope - do
|
||||
NOT chase byte-identity. Re-serialized JSON will differ in key order,
|
||||
whitespace, and `_comment` formatting, so a raw `diff` would never converge.
|
||||
The correct test: parse both the original and the regenerated manifest,
|
||||
normalize, and assert the same ordered entry list with identical
|
||||
detection/targeting/action fields per entry (ideally a small harness that
|
||||
mimics the engine's filter+detect decisions and confirms the same entries
|
||||
would fire in the same order on representative machine profiles). That, not
|
||||
byte equality, is what proves the model is lossless. (Same discipline as the
|
||||
ADR-001 data migration.)
|
||||
2. **Shadow mode.** shopdb serves the manifest at a new endpoint; a canary PC
|
||||
fetches from shopdb but ALSO reads the share, and logs any diff. No install
|
||||
behavior changes. Run across one of each PC type for a few cycles.
|
||||
3. **Read cutover, payloads still SMB.** Flip GE-Enforce to source the JSON from
|
||||
shopdb (payloads stay `smb`). The blast radius is only "where the JSON comes
|
||||
from"; the bytes and engine are unchanged. Keep the share manifests as the
|
||||
rollback (revert the dispatcher one-liner).
|
||||
4. **Payload migration (optional, per entry).** Move small scripts/configs to
|
||||
`inline`/`http` opportunistically. Leave big MSIs on SMB indefinitely - SMB
|
||||
is the right transport for them.
|
||||
5. **Author in shopdb.** Once read-cutover is stable, new manifest edits happen
|
||||
in the shopdb UI and the on-share JSON is retired (or auto-exported as a
|
||||
backup for break-glass).
|
||||
|
||||
Rollback during cutover (stages 2-4) is a one-line dispatcher revert, because
|
||||
the engine and payload layout never stop working from the share. AFTER the share
|
||||
JSON is retired (stage 5), that escape hatch is gone - post-cutover rollback is
|
||||
republishing a prior `manifestpublishedversions` snapshot (section 4). Both
|
||||
mechanisms must exist before stage 5, not just the dispatcher revert.
|
||||
|
||||
## 10. Risks / open questions
|
||||
|
||||
- **The engine is the contract.** Any drift between shopdb's generated JSON and
|
||||
what `Install-FromManifest.ps1` expects is a fleet-wide install bug. The
|
||||
byte-identical round-trip test (step 1) is non-negotiable, and the plugin must
|
||||
pin which engine lib version it targets (>= 2.6 for `_CmmVersion`).
|
||||
- **PCTypes alias graph** must be kept in sync with
|
||||
`Install-FromManifest.ps1:463-475`. The engine lib stays the single source of
|
||||
truth; shopdb only MIRRORS the map for server-side validation. Do NOT invert
|
||||
this to have the engine fetch aliases from shopdb - that would add exactly the
|
||||
availability coupling the next bullet warns against. When the lib's alias map
|
||||
changes, update shopdb's mirror as part of shipping that lib version.
|
||||
- **Availability coupling.** GE-Enforce currently depends only on SMB. Adding an
|
||||
HTTP dependency on shopdb means shopdb downtime could stall enforcement -
|
||||
hence the last-known-good local cache in section 8. Must be built in from day
|
||||
one, not bolted on. This is also why alias resolution and payloads stay
|
||||
independent of a live shopdb wherever possible.
|
||||
- **Transport trust.** The client runs as SYSTEM, so shopdb's TLS cert must be
|
||||
in the machine trust store (self-signed/air-gapped sites need the CA
|
||||
provisioned via the same DSC step as the token). `payloadsha256` verification
|
||||
is the real integrity guarantee and holds even over plain HTTP inside a
|
||||
trusted segment (section 5).
|
||||
- **Secrets in payloads.** Some config drops (site-config, credentials) may
|
||||
contain secrets. `inline` payloads live in the shopdb DB - those must respect
|
||||
the existing "secrets stay in .env, not the settings table" rule. Likely keep
|
||||
any secret-bearing payload on SMB with ACLs, never inline.
|
||||
- **Preinstall runner** is a separate consumer (`00-PreInstall-*` at imaging,
|
||||
before enrollment). It may not have a shopdb token yet at that point in the
|
||||
imaging sequence. Preinstall may need to stay share-sourced longer than
|
||||
runtime, or fetch a bootstrap manifest anonymously over HTTP.
|
||||
- **This is a big build.** Realistically phased: (P1) model + importer +
|
||||
behavioral-parity test; (P2) admin API + CRUD + publish/snapshot/rollback;
|
||||
(P3) frontend editor on /settings/pctypemapping; (P4) client fetch + shadow
|
||||
mode; (P5) read cutover; (P6) payload migration. P1 is the gating de-risk - if
|
||||
behavioral parity does not hold, stop. Snapshots (P2) must land before any
|
||||
client points at shopdb (P4), since serving the live draft to the fleet is
|
||||
unacceptable.
|
||||
|
||||
## 11. Relationship to existing work
|
||||
|
||||
- Replaces `plugins/computers/pctypemap.py` (the thin `pctypemap_<pxetype>`
|
||||
settings) - the pctype -> ComputerType mapping becomes the `computertypeid`
|
||||
column on `manifestscopes`. Two-source transition window: `pctype_mapping()`
|
||||
must keep reading the settings until the geenforce plugin is enabled, then
|
||||
fall back geenforce-table-first / settings-second, and only retire
|
||||
`seed_pctype_settings` + the settings at Milestone 1 close. Also reconcile the
|
||||
scope inventory: `pctypemap.py` lists `gea-shopfloor-display` but the share has
|
||||
no such manifest dir, and the share has a `main/` legacy dir the model ignores
|
||||
- the importer creates scopes only from what it finds (plus empty scopes for
|
||||
mapped-but-absent pctypes), and the P1 gate review reconciles the list with
|
||||
the floor team.
|
||||
- Also folds in the metrology mapping now living in `pctypemap.py`
|
||||
(`METROLOGY_TOOL_MAP`). The collector already auto-creates a MeasuringTool
|
||||
asset and a directional PC->tool `controls` relationship when it sees a
|
||||
metrology pctype (CMM / Keyence / Genspect / wax-and-trace); the PC stays a
|
||||
shopfloor PC. A metrology scope in the manifest model should carry the
|
||||
attached-measuring-tool type alongside its ComputerType so imaging and
|
||||
collector agree on what device the scope implies.
|
||||
- Reuses the collector's token machinery (PAT + `X-API-Key` + scopes) for the
|
||||
client-facing endpoints.
|
||||
- Reuses `get_permissions()` (contract 0.10.0) for `geenforce.manage` (edit
|
||||
drafts) / `geenforce.publish` (publish, rollback, export) / `geenforce.fetch`
|
||||
(the client service token).
|
||||
- Pairs with the collector: desired-state (this plugin) + observed-state
|
||||
(collector) enable a fleet compliance view.
|
||||
|
||||
## 12. Recommendation
|
||||
|
||||
Feasible and a strong architectural fit, but it is a multi-phase build with a
|
||||
fleet-wide blast radius. The single most important gate is P1: import the real
|
||||
manifests and prove BEHAVIORAL parity (same entries fire in the same order with
|
||||
the same detection/targeting), not byte-identity. Do not build the UI or touch a
|
||||
client until that parity holds. Three things separate a safe build from a
|
||||
dangerous one and must not be cut: behavioral-parity import (P1), immutable
|
||||
published snapshots with rollback before any client points at shopdb (P2/P4),
|
||||
and a dedicated `payloadsha256` for every HTTP/inline payload (section 5). If and
|
||||
when we proceed, this warrants a new ADR (ADR-012: GE-Enforce manifest
|
||||
ownership) capturing the desired-state model, the published-snapshot contract,
|
||||
the SMB/HTTP/inline payload + integrity model, and the fail-safe cache.
|
||||
|
||||
## 13. Execution plan (build order, gates, milestones)
|
||||
|
||||
Governing constraint: every step must be runnable and maintainable by average
|
||||
site IT, not just the original developer. Where an earlier draft implied expert
|
||||
machinery, this section simplifies it (and the model above already reflects
|
||||
those simplifications: one wide table, JSON-document snapshots, no row-mirroring).
|
||||
|
||||
### Phases and gates
|
||||
|
||||
- **P0 - Scaffold (S, ~0.5-1 day).** `flask plugin new geenforce`, structure
|
||||
copied from `plugins/measuringtools/`. Unlike bundled plugins' no-op migration
|
||||
anchors, this NEW plugin's `0001_geenforce_baseline` actually creates the
|
||||
tables and registers them in `PLUGIN_TABLE_OWNERS` (ADR-008). Deploy stays the
|
||||
standard `flask db upgrade` + `flask plugin upgrade-all`. Manifest:
|
||||
`api_prefix: /api/geenforce`, `default_enabled: false`, tight `core_version`.
|
||||
|
||||
- **P1 - Model + importer + parity harness (M, ~1-1.5 wk). THE GATE.** Order
|
||||
inside: tables -> `flask geenforce import-share` (reads common + every
|
||||
`gea-shopfloor-*` + preinstall.json, skips `.bak`, idempotent) -> exporter
|
||||
(rebuilds each scope's JSON from rows in `sortorder`) -> the parity harness
|
||||
(below). **GATE A:** `flask geenforce parity` prints PASS for all scopes. If
|
||||
it cannot pass, STOP the project. No API/UI/client work before Gate A.
|
||||
|
||||
- **P2 - Publish/snapshot/rollback + admin API + export-to-share (M, ~1.5-2 wk).**
|
||||
Publish freezes rendered JSON into `manifestpublishedversions`. CRUD per
|
||||
section 6. Plus `flask geenforce export-share` + an "Export to share" button
|
||||
that writes each scope's published JSON to the share after backing up the old
|
||||
file to `_meta/history/`. Engine, dispatcher, share layout, payloads, PCs all
|
||||
untouched. **GATE B = Milestone 1** (below).
|
||||
|
||||
- **P3 - Frontend editor (L, ~2-3 wk; parallel with P4 after P2 API freezes).**
|
||||
Expand `PCTypeMappingSettings.vue` per section 7, in three shippable
|
||||
increments; Move Up/Down not drag; the simulator.
|
||||
|
||||
- **P4 - Client fetch + shadow mode (M effort + soak time; needs P2, not P3).**
|
||||
Week-1 spike: a ~20-line PS1 on ONE canary PC proves SYSTEM-context HTTP auth +
|
||||
TLS trust before any real client change. Then `GE-Enforce.ps1` fetches JSON to
|
||||
a local cache and hands the file to `Install-FromManifest.ps1` unchanged;
|
||||
shadow mode installs from the share but logs any diff vs shopdb; ETag +
|
||||
last-known-good cache from day one. **GATE C:** zero shadow diffs across one PC
|
||||
of every pctype for >= 20 cycles.
|
||||
|
||||
- **P5 - Read cutover (S effort, M calendar).** Per-scope flip, canary first via
|
||||
`TargetHostnames`. Payloads stay `smb`. Rollback = dispatcher revert; share
|
||||
export continues as break-glass. **GATE D:** all scopes cut over.
|
||||
|
||||
- **P6 - Payload migration (S per entry, optional forever).** Small configs to
|
||||
`inline` (verified by `payloadsha256`); MSIs stay on SMB. Each entry
|
||||
independently revertible (flip `payloadsource`).
|
||||
|
||||
Hard ordering: P0 -> P1 -> P2 -> rest. **Snapshots (P2) MUST precede any client
|
||||
pointing at shopdb (P4).** P3 and P4 parallelize. Preinstall stays share-sourced
|
||||
through at least Milestone 1 (no token pre-enrollment; export writes
|
||||
`preinstall.json` too, so it is authored-in-shopdb for free with no client risk).
|
||||
|
||||
### The P1 parity harness (concrete, IT-re-runnable)
|
||||
|
||||
`plugins/geenforce/parity.py` + a CLI, also wrapped as a CI test. Two checks per
|
||||
scope, output one readable line per scope (`entries N/N identical profiles M/M
|
||||
same-fire PASS`), exit 0/1, prints the first differing entry/field on fail:
|
||||
|
||||
1. **Lossless field check (order-preserving).** Canonicalize each entry to
|
||||
exactly the fields the engine reads (Name, Type, the payload fields, all
|
||||
Detection*, the filter arrays, `_CmmVersion`, InUseCheck, preinstall flags);
|
||||
exclude `_comment` and key order (documentation, not behavior). Compare the
|
||||
ordered lists position by position.
|
||||
2. **Same-entries-fire-in-same-order.** Re-implement in ~120 lines of Python the
|
||||
engine's four filter functions exactly as written in `Install-FromManifest.ps1`
|
||||
(`Test-PCTypeMatches` incl. the alias groups at lines 463-475, `"*"`, and
|
||||
`<Type>-<SubType>`; `Test-HostnameMatches` exact + `-like`;
|
||||
`Test-MachineNumberMatches`; `Test-CmmVersionMatches`). For each machine-
|
||||
profile fixture, run BOTH manifests through it and assert the identical
|
||||
ordered list of entry names that pass all filters. Detection itself is not
|
||||
executed - check 1 already proved detection fields identical, so identical
|
||||
inputs to detection are guaranteed. This pair proves losslessness without
|
||||
byte-diffing.
|
||||
|
||||
Fixtures (`plugins/geenforce/parityfixtures.json`, ~16-18 profiles): one per
|
||||
pctype; CMM version variants `2016/2019/2026`/empty; collections machine-number
|
||||
variants (a credentialed bay, an MTConnect bay, neither); legacy-alias profiles
|
||||
(`Standard`+`Machine`, `CMM`) to exercise the alias graph both ways; a `WJS-*`
|
||||
hostname-wildcard profile; preinstall profiles including one that hits
|
||||
`PCTypesStrict`. Watch-items the harness must handle: empty `Applications: []`
|
||||
scopes (4 exist), entries with NO `DetectionMethod` (fire every run), and the
|
||||
`regvalue` literal typing.
|
||||
|
||||
### First slice: one vertical through `gea-shopfloor-cmm`
|
||||
|
||||
Only 4 entries but hits every hard part - MSI type, Registry detection with and
|
||||
without a pinned value, nested InUseCheck with Processes[], and the `_CmmVersion`
|
||||
gate. Tables: scopes, entries, entrypctypes, inusechecks + processes,
|
||||
publishedversions, pctypealiases. `flask geenforce import-share --scope
|
||||
gea-shopfloor-cmm`; `flask geenforce publish gea-shopfloor-cmm`; one endpoint
|
||||
`GET /api/geenforce/manifest?pctype=gea-shopfloor-cmm` serving the published
|
||||
snapshot (fat-client, ETag, collector-style `X-API-Key`/PAT auth reusing
|
||||
`shopdb/core/api/collector.py`). **Done =** parity PASS for cmm; the endpoint's
|
||||
JSON fed to `Install-FromManifest.ps1` on a bench CMM PC logs `4 skipped`
|
||||
identically to the share manifest; editing a draft does NOT change the served
|
||||
bytes but publishing does, and rollback restores the prior published bytes;
|
||||
unauth = 401, wrong-scope = 401.
|
||||
|
||||
### Milestone 1 (the recommended first stop)
|
||||
|
||||
End of P2 plus the publish/scope-list slice of P3: **manifests are authored and
|
||||
published in shopdb, exported to the share by a button, and the engine,
|
||||
dispatcher, share layout, payloads, and every PC are completely unchanged.**
|
||||
That delivers the real pain relief - validated editing instead of hand-edited
|
||||
JSON, version history, one-click rollback (republish + re-export), desired-state
|
||||
data sitting next to collector data - at ZERO client risk, with a rollback any
|
||||
IT tech already knows (restore the `_meta/history` backup file). Natural point to
|
||||
write ADR-012 with real experience behind it. P4/P5 (HTTP fetch, cutover) are a
|
||||
separately green-lit second milestone.
|
||||
|
||||
### Ranked risks / fail-fast
|
||||
|
||||
1. **Generated-JSON vs engine drift (fleet-wide mis-install).** Parity harness
|
||||
first; CI re-proves parity against checked-in real manifests on every
|
||||
exporter change; pin lib >= 2.6.
|
||||
2. **Serving a half-finished draft.** Structural: client reads only
|
||||
`iscurrent` snapshots; test asserts a draft edit leaves served bytes
|
||||
unchanged. Must exist before P4.
|
||||
3. **Availability coupling.** Last-known-good local cache in the first client
|
||||
prototype; shadow test blocks shopdb and confirms enforce-from-cache + WARN.
|
||||
4. **SYSTEM HTTP auth + TLS trust.** The ~20-line canary spike in P4 week 1,
|
||||
before the real client change. Hours of cost; if it fails, Milestone 1 still
|
||||
delivers full value.
|
||||
5. **Alias-graph drift.** Seed pins a lib version; harness legacy-name profiles
|
||||
fail loudly on divergence; new-lib runbook includes "update the alias seed".
|
||||
6. **Preinstall has no pre-enrollment token.** Keep share-sourced through
|
||||
Milestone 1/2; decide later.
|
||||
7. **Editor scope creep.** Three shippable increments; buttons over drag; reuse
|
||||
JSON preview.
|
||||
|
||||
### IT operability (day-to-day runbook, proving the design is manageable)
|
||||
|
||||
All in Settings > Imaging PC Types. No PowerShell, no SQL, no share edits.
|
||||
- **Add an app to a PC type:** open the PC type, Add Entry, pick Type (fields
|
||||
adapt), fill installer + detection + targeting, Move Up/Down to order, Preview
|
||||
(+ simulator), Publish with a note. PCs pick it up next 5-min cycle.
|
||||
- **Bump a version:** drop the new MSI in the scope's `apps/` on the share,
|
||||
update the entry's Installer + Detection value, Preview, Publish.
|
||||
- **Roll back a bad publish:** History -> pick last-good version -> Roll Back
|
||||
(during Milestone 1 also click Export to Share).
|
||||
- **Canary a risky change:** add the one test PC under Target Hostnames, Publish;
|
||||
when happy, remove the filter and Publish again.
|
||||
- **Check "did PC Y get app X":** the simulator with that PC's type/machine
|
||||
number/CMM version shows exactly which entries apply and why others are filtered.
|
||||
- **See revision history / who changed what:** the PC type's History tab lists
|
||||
every published version (date, author, note) with a Roll Back on each; the
|
||||
Audit Logs page shows the finer-grained draft edits (who touched which entry
|
||||
field, when) between publishes.
|
||||
31
README.md
Normal file
31
README.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Architecture Decision Records
|
||||
|
||||
Each ADR captures a single architectural decision: the context, the decision itself, the consequences, and the alternatives considered. ADRs are immutable once accepted. Superseded ADRs stay in this folder with a pointer to the newer ADR.
|
||||
|
||||
## Status definitions
|
||||
|
||||
- **PROPOSED**: drafted, awaiting decision
|
||||
- **ACCEPTED**: decision is in effect
|
||||
- **SUPERSEDED**: replaced by a later ADR (link forward)
|
||||
- **DEPRECATED**: no longer in effect, no replacement
|
||||
|
||||
## Index
|
||||
|
||||
| ADR | Title | Status |
|
||||
|-----|-------|--------|
|
||||
| [001](ADR-001-asset-as-platform-contract) | Asset model is the platform contract | ACCEPTED |
|
||||
| [002](ADR-002-plugin-versioning) | Plugin contract versioning (semver) | ACCEPTED |
|
||||
| [003](ADR-003-plugin-distribution) | Plugin distribution model | ACCEPTED |
|
||||
| [004](ADR-004-deployment-topology) | Deployment topology (per-site instances) | ACCEPTED |
|
||||
| [005](ADR-005-equipment-vs-measuringtools) | Equipment vs measuringtools plugin scope | ACCEPTED |
|
||||
| [006](ADR-006-collector-contract) | Plugin collector contract pattern | ACCEPTED |
|
||||
| [007](ADR-007-product-versioning-and-releases) | Product versioning and releases | ACCEPTED |
|
||||
| [008](ADR-008-plugin-migration-ownership) | Plugin migration ownership (per-plugin chains) | ACCEPTED |
|
||||
| [009](ADR-009-frontend-plugin-gating) | Frontend plugin route gating | ACCEPTED |
|
||||
| [010](ADR-010-frontend-plugin-hooks) | Frontend plugin hook contract | ACCEPTED |
|
||||
| [011](ADR-011-machines-rename) | Machines rename + modeltypes retyping | ACCEPTED |
|
||||
| [012](ADR-012-geenforce-manifest-ownership) | GE-Enforce manifest ownership in shopdb | ACCEPTED |
|
||||
|
||||
## Authoring
|
||||
|
||||
When proposing a new decision, copy the most recent ADR as a template, increment the number, and update this index. Do not edit accepted ADRs in place; supersede them with a new one.
|
||||
69
ROADMAP.md
69
ROADMAP.md
@@ -0,0 +1,69 @@
|
||||
# Roadmap
|
||||
|
||||
shopdb-flask is at `__contract_version__ = '0.11.0'` (pre-1.0; product `__version__ 0.7.0`, tags through v0.7.0). This document captures what stands between today and a stable `1.0.0` release. Maintained as scope evolves; supersedes nothing in the ADRs.
|
||||
|
||||
## Phase status
|
||||
|
||||
| Phase | Status | Commit |
|
||||
|-------|--------|--------|
|
||||
| 0 - Lock platform contract, naming, style enforcement | DONE | `d6725c0` |
|
||||
| 1 - pytest baseline, production hardening, pinned requirements | DONE | `2d1bb83` |
|
||||
| 2 - Plugin contract surface and compliance tests | DONE | `5fefb53` |
|
||||
| 3 - Manifest-first loader, shopdb.api namespace, auto-register blueprints | DONE | `6f085a1` |
|
||||
| 4 - Plugin scaffolding (`flask plugin new`) | DONE | `8eb9362` |
|
||||
| 5 - Alembic baseline, per-site deploy, ADRs to docs/adr | DONE | `d4e3ac9` |
|
||||
| 6 - Multi-site distribution readiness (settings-driven branding/ServiceNow/floor plan, security closeout, docs + Docker frontend build, release engineering) | DONE | v0.5.0-v0.7.0 |
|
||||
|
||||
The last big milestone before 1.0 is the legacy-ASP data import plus a production pilot deployment; the framework work below is what remains after that.
|
||||
|
||||
## What's left before tagging 1.0.0
|
||||
|
||||
### Must-have
|
||||
|
||||
- **Asset model fully wired**. `Asset.mapx, Asset.mapy` columns, `AssetRelationship.inheritsposition`, `AssetRelationship.propagatesthroughid` columns. Models match the locked ADR-001 surface that `resolve_asset_position` already targets.
|
||||
- **Equipment data migration script** for facilities migrating from legacy ASP shopdb. One-shot script under `scripts/migration/`. Per ADR-001, only `category='Equipment' AND machinenumber IS NOT NULL` migrates.
|
||||
- **Printers retirement**. Legacy `PrinterData` model, `printers_bp` legacy blueprint, and the frontend `PrinterForm.vue` references to `printer.printerdata.*` get removed in lockstep. Coordinated with the equipment migration.
|
||||
- **External plugin UI packaging**. The Vue-side hook contract ships (ADR-010: get_settings_cards / get_asset_panels / get_map_overlays / get_asset_presentation) and route gating is backend-driven (ADR-009), but plugin routes/views still live in core `frontend/src`. Let an external plugin ship its own Vue bundle so adopters can add UI without editing core.
|
||||
|
||||
### Nice-to-have
|
||||
|
||||
- **Full palette theming.** `brand_primary_color` and a few brand colors are settings-driven, but the rest of the CSS palette (surfaces, borders, accents) is still hardcoded in `style.css`. A complete theming pass would expose the palette as CSS variables a site can override, not just the brand colors.
|
||||
- Frontend scaffolding skill (the backend has `flask plugin new`; the frontend stub is currently manual copy-paste).
|
||||
- Marketplace listing site (PLUGINS.md is a one-pager; a proper listing with links to sister-site plugins becomes useful when there are more than three external plugins).
|
||||
- Plugin contract surface diff tooling. Today version bumps are manual judgment; a CI check that diffs the contract surface against the previous tag would catch missed bumps. See ADR-002.
|
||||
- Calibration cycles, maintenance windows, downtime tracking (domain extensions; would likely live in the `machines` and `measuringtools` plugins).
|
||||
|
||||
### Deferred (out of scope for 1.0)
|
||||
|
||||
- Multi-tenancy (rejected by ADR-004; revisit only if more than five sites adopt and operational overhead becomes painful).
|
||||
- Pip-installable plugins (deferred per ADR-003 v2). Filesystem distribution stays the v1 model.
|
||||
- Event bus on `BasePlugin` (removed per ADR-001; add via new ADR if a real use case appears).
|
||||
- Frontend rebuild beyond Vue 3 + Pinia + Vite (the existing stack is fine).
|
||||
|
||||
## What 1.0.0 means
|
||||
|
||||
Tagging `1.0.0` is a commitment that the contract surface is stable for at least the next minor version cycle. Plugin authors at sister sites can pin `core_version: ">=1.0.0,<2.0.0"` and trust their plugin will not break on framework patches.
|
||||
|
||||
Concretely, 1.0.0 ships when:
|
||||
|
||||
1. ADR-001's full contract surface is implemented in code, not just documented (`Asset.mapx`, `RelationshipType.propagatesthroughid`, etc.)
|
||||
2. The contract test suite covers every documented hook with both happy-path and a "broken plugin" failure-isolation test
|
||||
3. At least one external plugin (likely `measuringtools` from the scaffold canary) has been built end-to-end with no contract changes required mid-build
|
||||
4. The deploy runbook (DEPLOY.md) has been validated by an actual fresh-host deploy
|
||||
|
||||
## Decision log pointers
|
||||
|
||||
When a roadmap item gets prioritized, document the why in a new ADR and link from this file. The ADRs are the canonical source for design decisions; this file is the prioritized backlog.
|
||||
|
||||
- [ADR-001](ADR-001-asset-as-platform-contract) - Asset as platform contract
|
||||
- [ADR-002](ADR-002-plugin-versioning) - Plugin contract versioning
|
||||
- [ADR-003](ADR-003-plugin-distribution) - Plugin distribution model
|
||||
- [ADR-004](ADR-004-deployment-topology) - Deployment topology (per-site)
|
||||
- [ADR-005](ADR-005-equipment-vs-measuringtools) - Equipment vs measuringtools
|
||||
- [ADR-006](ADR-006-collector-contract) - Collector contract pattern
|
||||
- [ADR-007](ADR-007-product-versioning-and-releases) - Product versioning and releases
|
||||
- [ADR-008](ADR-008-plugin-migration-ownership) - Plugin migration ownership (per-plugin chains)
|
||||
- [ADR-009](ADR-009-frontend-plugin-gating) - Frontend plugin route gating
|
||||
- [ADR-010](ADR-010-frontend-plugin-hooks) - Frontend plugin hook contract
|
||||
- [ADR-011](ADR-011-machines-rename) - Machines rename + modeltypes retyping
|
||||
- [ADR-012](ADR-012-geenforce-manifest-ownership) - GE-Enforce manifest ownership
|
||||
|
||||
116
UPGRADE.md
116
UPGRADE.md
@@ -0,0 +1,116 @@
|
||||
# Upgrading an existing site
|
||||
|
||||
This is the procedure for moving a running shopdb-flask instance to a newer
|
||||
version. For a first-time install use [DEPLOY.md](DEPLOY) instead.
|
||||
|
||||
Each site is single-tenant (ADR-004), so an upgrade touches only that site's
|
||||
own stack. Read the ADRs added since your last update (`docs/adr/`) and the
|
||||
`CHANGELOG.md` before starting; a breaking ADR may require coordinated work.
|
||||
|
||||
## Step 0: Back up first
|
||||
|
||||
Never upgrade without a fresh backup you have tested. Take a full database dump
|
||||
and copy the `instance/` directory. See [BACKUP-RESTORE.md](BACKUP-RESTORE).
|
||||
|
||||
```bash
|
||||
# Docker:
|
||||
docker compose exec -T db mysqldump -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask | gzip > pre-upgrade-$(date +%F).sql.gz
|
||||
cp -a instance/ instance-backup-$(date +%F)/
|
||||
```
|
||||
|
||||
## Step 1: Get the new code / image
|
||||
|
||||
```bash
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
The application is distributed through the internal GE Aerospace Gitea; pull
|
||||
from there. There is no external image registry.
|
||||
|
||||
## Step 2: Rebuild
|
||||
|
||||
The Docker image builds the Vue frontend in-image, so a container rebuild picks
|
||||
up frontend changes automatically:
|
||||
|
||||
```bash
|
||||
docker compose build api
|
||||
docker compose up -d api
|
||||
```
|
||||
|
||||
Bare-metal / venv install: rebuild the frontend by hand and refresh Python
|
||||
dependencies:
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
cd frontend && npm ci && npm run build && cd ..
|
||||
```
|
||||
|
||||
## Step 3: Apply migrations
|
||||
|
||||
```bash
|
||||
# Docker:
|
||||
docker compose exec api flask db upgrade
|
||||
docker compose exec api flask plugin upgrade-all
|
||||
# venv:
|
||||
flask db upgrade
|
||||
flask plugin upgrade-all
|
||||
```
|
||||
|
||||
`flask db upgrade` applies any new migrations in the core Alembic chain.
|
||||
`flask plugin upgrade-all` then applies any new per-plugin migrations (each
|
||||
bundled plugin owns its schema going forward - see ADR-008). Both are
|
||||
idempotent; running them when already at head is a no-op.
|
||||
|
||||
## Step 4: Re-seed permissions and settings
|
||||
|
||||
New versions may add RBAC permissions or default Settings keys. Both seeders are
|
||||
idempotent - they add anything missing and leave existing rows untouched, so
|
||||
your site's customized values are preserved.
|
||||
|
||||
```bash
|
||||
# Docker:
|
||||
docker compose exec api flask seed permissions
|
||||
docker compose exec api flask seed settings
|
||||
# venv:
|
||||
flask seed permissions
|
||||
flask seed settings
|
||||
```
|
||||
|
||||
## Step 5: Restart
|
||||
|
||||
```bash
|
||||
docker compose restart api
|
||||
# venv: restart your process manager, e.g. pm2 restart shopdb-flask-api shopdb-flask-ui
|
||||
```
|
||||
|
||||
Confirm the app is healthy (login page renders, `/api/auth/login` returns a
|
||||
`VALIDATION_ERROR` for an empty body rather than a 500).
|
||||
|
||||
## Version-specific notes
|
||||
|
||||
### Upgrading to v0.5.0 or later: bundled West Jefferson floor plan removed
|
||||
|
||||
Versions before 0.5 shipped the West Jefferson facility floor-plan PNGs as the
|
||||
map default (`/static/images/sitemap2025-light.png` and `-dark.png`). v0.5+
|
||||
removes those bundled PNGs and ships a generic placeholder SVG instead.
|
||||
|
||||
If your instance's `map_blueprint_light` / `map_blueprint_dark` Settings still
|
||||
point at `/static/images/sitemap2025-*`, the map will 404 those images after the
|
||||
upgrade. Re-upload your own floor plan in **Settings > Floor Map**. Uploaded floor
|
||||
plans are stored under `instance/` and survive upgrades, so a site that already
|
||||
uploaded its own plan is unaffected. Only instances still using the old bundled
|
||||
default need to act.
|
||||
|
||||
To check what your instance points at:
|
||||
|
||||
```bash
|
||||
docker compose exec api flask shell -c "from shopdb.core.models.setting import Setting; print(Setting.query.filter(Setting.key.like('map_blueprint%')).all())"
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [BACKUP-RESTORE.md](BACKUP-RESTORE) - what to back up and how to restore
|
||||
- [CONFIG.md](CONFIG) - environment variables and Setting keys
|
||||
- [DEPLOY.md](DEPLOY) - first-time deploy runbook
|
||||
- `CHANGELOG.md` - what changed in each release
|
||||
|
||||
56
_Sidebar.md
56
_Sidebar.md
@@ -0,0 +1,56 @@
|
||||
## Docs
|
||||
|
||||
**Install and operate**
|
||||
|
||||
- [[INSTALL-WINDOWS-IIS]]
|
||||
- [[DEPLOY]]
|
||||
- [[DEPLOY-WINDOWS-IIS]]
|
||||
- [[PILOT-DEPLOY]]
|
||||
- [[CONFIG]]
|
||||
- [[UPGRADE]]
|
||||
- [[BACKUP-RESTORE]]
|
||||
|
||||
**Data import**
|
||||
|
||||
- [[IMPORT-API]]
|
||||
- [[IMPORT-ADOPTION]]
|
||||
|
||||
**Plugins**
|
||||
|
||||
- [[PLUGINS]]
|
||||
- [[PLUGIN-QUICKSTART]]
|
||||
- [[PLUGIN-GUIDE]]
|
||||
- [[PLUGIN-HOOKS]]
|
||||
- [[PLUGIN-EXTERNAL-REPO]]
|
||||
- [[CONTRACT-STABILITY]]
|
||||
|
||||
**Integrations**
|
||||
|
||||
- [[COLLECTOR-INTEGRATION]]
|
||||
- [[GE-ENFORCE]]
|
||||
- [[GE-ENFORCE-DEPLOY]]
|
||||
- [[GE-ENFORCE-CLIENT]]
|
||||
|
||||
**Project**
|
||||
|
||||
- [[ROADMAP]]
|
||||
|
||||
**ADRs**
|
||||
|
||||
- [[ADR-001-asset-as-platform-contract]]
|
||||
- [[ADR-002-plugin-versioning]]
|
||||
- [[ADR-003-plugin-distribution]]
|
||||
- [[ADR-004-deployment-topology]]
|
||||
- [[ADR-005-equipment-vs-measuringtools]]
|
||||
- [[ADR-006-collector-contract]]
|
||||
- [[ADR-007-product-versioning-and-releases]]
|
||||
- [[ADR-008-plugin-migration-ownership]]
|
||||
- [[ADR-009-frontend-plugin-gating]]
|
||||
- [[ADR-010-frontend-plugin-hooks]]
|
||||
- [[ADR-011-machines-rename]]
|
||||
- [[ADR-012-geenforce-manifest-ownership]]
|
||||
- [[README]]
|
||||
|
||||
**Proposals**
|
||||
|
||||
- [[PROPOSAL-ge-enforce-plugin]]
|
||||
|
||||
Reference in New Issue
Block a user