ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make main the lean product a site receives

Browse Source 
main was 16 days and 259 commits behind the line the product is actually built
on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It
was neither current nor curated - just stale. This brings it forward and, at the
same time, makes it mean something: main is what a receiving site installs, and
nothing else.

WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides,
knowledgebase, machines, network, notifications, printers, slides, tools,
warranty - plus applications, which is manifest-less core and always ships. The
only manifest dependency inside that set is endpoints on computers, and both are
in it, so the closure is self-contained.

WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb
stay on the site line. So do the Windows installer tree, the VM rehearsal
helpers, the agent tooling config, one site's data loader, the internal audit
and findings records, and the planner spreadsheet. 1,135 tracked files against
1,438.

A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped:

deploy/site-profile-universal.json still named all seventeen plugins. It is the
file docs/DEPLOY.md tells a site to provision from, so a fresh install would
have failed at `flask plugin apply-profile` with PluginError - the first thing a
receiving site does, broken by the branch meant to serve them. It now names the
twelve that exist, and its comment says the two must agree.

agent/ was never refreshed. Whoever populated this branch copied shopdb/,
plugins/, frontend/ and tests/ from the site line and left the agent tree at its
September state: 22 C# and PowerShell sources missing, including AgentCredentials,
EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the
site line had deleted still present. The old tree is internally consistent, so it
compiles, and nothing in CI builds it - every gate was green over a server that
speaks enrollment and per-PC credentials to a bundled agent source that cannot.
Resynced whole.

DECISIONS THIS ENCODES, rather than defers:

- The plugin-authoring lab builds printedparts, whose manifest depends on
  employees, so it could not stay. The lab remains on the site line and main
  ships no build-along tutorial. That is a real loss for an adopting developer
  and is recorded here rather than discovered later.
- Recognition, Training and Recertification notifications are no longer seeded.
  Each splits one notification into a card per employee and renders a photo,
  which needs a plugin this branch does not bundle. Seeding them would offer an
  author a type that cannot resolve anybody. A site that installs employees gets
  them from that plugin's own seed.
- Windows/IIS is documented as the legacy path throughout, and the README leads
  with Docker, because that is what a receiving site gets.

The API description is regenerated for THIS branch - 347 paths and 483
operations, down from 415 and 573 - so the published contract describes what is
here rather than what the site line serves. docs/CONFIG.md lost the two
measuring-tool label keys the code no longer declares, which two lanes had
decided opposite ways.

Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run
rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
cproudlock
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions
Download Patch File Download Diff File Expand all files Collapse all files

291
docs/ADOPTING-AT-ANOTHER-SITE.md Normal file
View File

@@ -0,0 +1,291 @@
<!-- docmeta
purpose: howto
audience: adopter
status: current
summary: Point the asset reporter and EventSaver at your own ShopDB, by Intune or by hand.
loadwith:
-->
# Adopting the fleet tools at another site
Two things run on shop-floor PCs and talk to a ShopDB instance: the **asset
reporter**, which posts what a PC is to the collector API, and **EventSaver**, a
screensaver that shows a slideshow served by ShopDB.
Neither is site-specific by design. What is site-specific is the URL, the
credential, and which PCs it runs on - and all three are inputs, not code.
**Getting them onto a PC is your site's job, not ShopDB's.** ShopDB serves the
data; something else lays the files down and keeps them there. This page covers
the two delivery paths that do not assume a tool you may not have: Microsoft
Intune, and doing it by hand. If you already run a configuration-enforcement
tool on the floor, the end state below is what it has to produce.
Read [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) first if you have not: it is one
page and it is what makes the rest of this make sense.
## Where these files come from
| Artifact | Where it is |
|---|---|
| `Report-AssetToShopDB.ps1` | `plugins/computers/client/`, so it versions with the collector contract it implements. It names no site: the server comes from `HKLM:\SOFTWARE\GE\ShopDB` `BaseUrl` or `-ApiUrl` |
| `Report-PrintersToShopDB.ps1` | `plugins/printers/client/`. Optional - it records what print queues a bay actually has |
| EventSaver (`EventSaver.cs`, `EventSaver.ini`) | `plugins/slides/client/`. One C# file that builds with the in-box .NET Framework compiler, so a site can rebuild it rather than trust a binary. The compiled `EventSaver.scr` is a release asset, not a file in the repository. Neither file names a site |
---
## Part 1: the asset reporter
### What it does
Runs as SYSTEM and POSTs what the PC is - hostname, BIOS serial, PC type,
logged-in user, make and model, OS, last boot, every routable NIC, the machine
number where one is configured - to
`POST https://<your-shopdb>/api/collector/computers`. The payload contract is
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md). A field your site does not collect
is simply absent; the server upserts on `hostname` and leaves the rest alone.
### What you must provide
| Input | Where it comes from |
|---|---|
| Server URL | `-ApiUrl`, or `HKLM:\SOFTWARE\GE\ShopDB` value `BaseUrl`. A script downloaded from Settings has your URL already in it |
| A credential | `HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey` for the shared key, or the PC's own credential once it has enrolled. Never stamped into the script. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| Your routable ranges | A **server** setting, `collector_corporate_ranges`. The bay reports every NIC and takes no view; the server picks which one is the PC's address |
The reporter still accepts `-AllowedRanges` and reads a `CollectorRanges`
registry value, and **ignores both**, logging a note when either is set. Choosing
the NIC on the bay was wrong - a bay cannot know a site's addressing - so it
moved to the server. The parameters remain only so an existing scheduled task
does not fail on an unknown argument. Set the server setting.
**Shortcut: download it pre-configured.** Settings > Computers > Asset reporter
generates the script with your server's URL already in the parameter defaults,
and shows its SHA-256. The credential is deliberately not included.
### The credential to start with
Mint a managed personal access token scoped to `collector.ingest` and nothing
else, so a token recovered off a shop-floor PC cannot read your asset register.
Plan to move off it onto per-PC credentials.
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole story and the only page that
describes it.
### Deploying it
The three paths below are alternatives - pick one. All of them end with the same
two facts on the PC: the script exists somewhere it can run from, and the
credential is in the registry.
#### Option A: Microsoft Intune
**A1. The key, as a platform script.** Devices > Scripts and remediations >
Platform scripts > Add > Windows 10 and later. Run as SYSTEM, do not run in the
64-bit context only.
```powershell
# Set-ShopdbCollectorKey.ps1 (Intune platform script, runs once per device)
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
exit 0
```
Putting a token in an Intune script body means anyone who can read the Intune
configuration can read the token. That is an argument for the `collector.ingest`
scope, not against Intune: the blast radius of that token is one API. Restrict
the registry key's ACL to SYSTEM and Administrators so an interactive user on the
bay cannot read it.
**A2. The reporter, as a remediation on a schedule.** Devices > Scripts and
remediations > Remediations. A remediation is the natural fit because it re-runs
on a schedule, which is exactly what reporting is.
*Detection script* - forces the remediation to run every cycle:
```powershell
# Always "not compliant": reporting is not a state to converge on, it is an
# event that should happen on every schedule tick.
exit 1
```
*Remediation script* - the reporter itself, with your URL:
```powershell
& "$PSScriptRoot\Report-AssetToShopDB.ps1" `
-ApiUrl 'https://shopdb.example.net/api/collector/computers'
exit 0
```
Assign it with a daily schedule. Run as SYSTEM, 64-bit.
**A3. Intune with DSC (Machine Configuration).** If your estate is already
governed by Azure Machine Configuration, express the *credential* as
configuration and leave the *reporting* to a scheduled task, because a report is
an event and DSC converges state:
```powershell
Configuration ShopdbCollector
{
Import-DscResource -ModuleName PSDesiredStateConfiguration
Node localhost
{
Registry CollectorKey
{
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
ValueName = 'CollectorKey'
ValueData = 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
ValueType = 'String'
Ensure = 'Present'
}
Registry CollectorUrl
{
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
ValueName = 'BaseUrl'
ValueData = 'https://shopdb.example.net'
ValueType = 'String'
Ensure = 'Present'
}
# ...and a Script resource whose SetScript registers the scheduled task
# below (option B has the three lines). Pass it no -ApiUrl: BaseUrl
# above is where the script reads the server from, and one source beats
# two that can disagree.
}
}
```
Do not add a `CollectorRanges` resource. That value is read and ignored; the
ranges are a server setting now.
#### Option B: manual
For a pilot, a single bay, or an estate with no management plane:
```powershell
# As administrator, once per PC.
New-Item -Path 'C:\ProgramData\ShopDB' -ItemType Directory -Force | Out-Null
Copy-Item .\Report-AssetToShopDB.ps1 'C:\ProgramData\ShopDB\'
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument @'
-NoProfile -ExecutionPolicy Bypass -File "C:\ProgramData\ShopDB\Report-AssetToShopDB.ps1" -ApiUrl "https://shopdb.example.net/api/collector/computers"
'@
$trigger = New-ScheduledTaskTrigger -Daily -At 7am
Register-ScheduledTask -TaskName 'ShopDB asset report' -Action $action -Trigger $trigger `
-User 'SYSTEM' -RunLevel Highest -Force
Start-ScheduledTask -TaskName 'ShopDB asset report'
```
Lock down `C:\ProgramData\ShopDB` if you put it there: a script a user can edit,
run by SYSTEM on a schedule, is a local privilege escalation. Break inheritance
and grant Administrators and SYSTEM only.
### Verifying, and what to do when it fails
Open Computers in the web UI and find the hostname: it should be there with its
serial, type and a last-reported timestamp within one schedule tick. The same
answer over the API needs a logged-in session, because the read side is
permissioned (`computers.view`) even though the ingest side is not:
```powershell
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>' `
-Headers @{ Authorization = "Bearer $jwt" }
```
The collector key does NOT work here - it grants `collector.ingest` and nothing
else, which is the point of it. The script also logs to `C:\Logs\Shopfloor`.
Every failure mode and its fix is the table in
[COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md).
---
## Part 2: EventSaver
### What you have to decide
[EVENTSAVER](EVENTSAVER.md) is what EventSaver is, how it picks slides and how to
diagnose it. Read it first; adopting it adds only three decisions:
1. **Your feed URL**, in `EventSaver.ini`:
`url=https://<your-shopdb>/api/slides/feed?surface=shopfloor`.
2. **Ship the ini with the binary.** There is no compiled-in fallback, so a
missing ini is a blank screen. Check any `.scr` you were handed rather than
built: an early build still has another site's file server baked in, which is
exactly why the fallback was removed (ADR-015).
3. **Which PCs get it.** A screensaver over a live display is a defect. Exclude
kiosks, lobby displays and metrology bays.
If your configuration tool enforces the ini by hash, your own hand edits on a
bay revert - that is the feature working. Edit the copy your tool deploys from,
and update the hash it detects on:
```powershell
(Get-FileHash .\configs\EventSaver.ini -Algorithm SHA256).Hash
```
### Deploying it
#### Option A: Intune
Package the three files as a Win32 app (`EventSaver.scr`, `EventSaver.ini`, and
the enable script) with this install command:
```powershell
# install.ps1 for the Win32 app
Copy-Item .\EventSaver.scr 'C:\Windows\System32\EventSaver.scr' -Force
Copy-Item .\EventSaver.ini 'C:\Windows\System32\EventSaver.ini' -Force
exit 0
```
Detection rule: file `C:\Windows\System32\EventSaver.scr` exists **and** matches
the expected SHA-256, so a changed binary reinstalls.
Enable it per user with a configuration profile (Settings catalog > Screen
saver) or a user-context script setting the three values under option B. With
Machine Configuration/DSC, the two files are `File` resources - but DSC runs as
SYSTEM, so the per-user setting needs the logged-on user's hive or a
default-user template, which is usually more trouble than a login script.
#### Option B: manual
```powershell
Copy-Item .\EventSaver.scr 'C:\Windows\System32\' -Force
Copy-Item .\EventSaver.ini 'C:\Windows\System32\' -Force # with YOUR url=
# Per user, in the user's own session:
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name SCRNSAVE.EXE -Value 'C:\Windows\System32\EventSaver.scr'
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveActive -Value '1'
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveTimeOut -Value '480'
```
Those three values are the whole "enable" step in every option above: a
screensaver is a per-user setting, so whatever deploys the binary still has to
reach the user hive.
### Verifying
Run `EventSaver.scr /c` on the PC - it reports which source it resolved and
where it is caching - then open that feed URL in a browser on the same PC. The
symptom table in [EVENTSAVER](EVENTSAVER.md) reads the answers.
---
## Which path to choose
| You have | Use |
|---|---|
| Intune across the estate | Option A. A remediation for the reporter, a Win32 app for EventSaver. |
| A configuration-enforcement tool on the floor | Its own idiom, producing the end state option B describes: the script on disk, the credential in the registry, a task as SYSTEM. |
| Neither, or a pilot of a few bays | Option B, then move to A once it earns its place. |
These are not exclusive, and on a real floor they usually are not. Where two
tools both converge the same PC, decide which owns which layer before deploying
both: the deciding question is which one you would look at first to answer "why
is this bay wrong".

89
docs/API-REFERENCE.md Normal file
View File

@@ -0,0 +1,89 @@
<!-- docmeta
purpose: reference
audience: integrator, developer
status: current
summary: Where the generated API spec is, what llms.txt is for, and the surface and auth orientation a spec cannot give.
loadwith: none
-->
# API reference
**The reference is generated and served by the app. This page is a pointer to
it, plus the two things a specification cannot tell you.**
There used to be a hand-written endpoint list here. It competed with the
generated spec, lost, and went stale in between - so it was cut rather than
maintained. Everything it listed is in the spec, derived from the code: every
operation with its auth, its query parameters, its response shape and
harvested examples. Counts are in [PROJECT-MAP](PROJECT-MAP.md), which is
generated; nothing of that kind belongs on this page.
## Where the generated reference is
| Reach it at | What it is |
|---|---|
| `GET /api/docs` | The reference, rendered with Redoc. The bundle is vendored in the image, so it renders with no network - which is the only reason it is usable on an air-gapped site. |
| `GET /api/docs/openapi.json` | The raw OpenAPI 3.1 spec. Also tracked as `docs/openapi.json`, so it can be read from a checkout without running anything; regenerate with `scripts/gen_openapi.py` after any API change. |
| `GET /api/docs/llms.txt` | The short version, for a machine caller. Also [llms.txt](llms.txt) in this corpus. |
| `GET /api/docs/try` | Swagger UI over the same spec, where Execute sends a real request as whoever is signed in on that browser. Off by default (`API_DOCS_TRYIT_ENABLED`) and not something a production site turns on without deciding to. |
[llms.txt](llms.txt) is not a second spec. It is the orientation a caller needs
before the spec is useful: the response envelope, how authentication is
obtained and sent, the pagination and filtering conventions, and the handful of
recipes people get wrong. Read it first, then look operations up in the spec.
## Which surface owns what
The spec lists every operation; it does not say which contract governs one.
There are three surfaces, and the third is everything else.
| Surface | Who calls it | Where its semantics live |
|---|---|---|
| **Fleet and client contracts** | Shop-floor PCs, the endpoint agent, kiosks, displays and printer installers - machines, not browsers | [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md), [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md), [COLLECTOR-AUTH](COLLECTOR-AUTH.md), [PRINTER-INSTALLER](PRINTER-INSTALLER.md), [EVENTSAVER](EVENTSAVER.md) |
| **Bulk import** | A script loading a site's data. There is no separate import surface: it drives the same endpoints the UI does, with a header that preserves the source rows' timestamps | [IMPORT-API](IMPORT-API.md) |
| **Core UI API** | The Vue frontend, and anything else holding a token | [CONTRACT-STABILITY](CONTRACT-STABILITY.md) for what is safe to depend on |
## How auth works at a glance
Four credentials, and which one an operation takes follows from its surface.
The spec states it per operation; this is the shape behind it.
- **JWT** - a login token or a Personal Access Token. The core UI API.
- **Scoped service token** - `X-API-Key` or `Authorization: Bearer`, minted and
revoked per purpose (`collector.ingest`, `collector.ingest.endpoints`). A
token can be bound to specific resources and is refused anything outside
them. This is what the fleet uses.
- **A per-PC credential** - an enrolled endpoint holds its own secret, and
revoking one endpoint costs nothing to the rest. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md)
and [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md).
- **A kiosk device token, or an address inside the kiosk's allowlist** - a
credential for a screen, not authorization for a person. It says which
cabinet is asking; the badge on the request says who is standing at it. Both
kiosks are off by default and answer 404, not 403, when they are off.
Two rules cover almost everything else:
- **Every read is behind a permission.** Reading is not free, and a site
decides what a signed-out visitor may see by granting the seeded `guest` role
the read permissions it chooses - in the same role grid it uses for staff.
- **Every mutation needs a JWT and a permission or role.** None are public.
## What is reachable with no credential
This is the inventory a firewall or a deployment reviewer asks for, and it is
the one question the spec answers badly - it is a question about the whole
surface, not about any one operation.
Do not read it off a page somebody maintains. It is enumerated, with a reason
per entry, in `tests/test_core/test_public_read_surface.py`, which fails the
build when a new public read appears without being added to the list. That file
is the answer, and it is the answer because it cannot go stale without turning
CI red.
## See also
- [CONFIG](CONFIG.md) - the environment variables and settings behind these
endpoints, including the public settings allowlist.
- [CONTRACT-STABILITY](CONTRACT-STABILITY.md) - what a sister site may depend
on, and how tightly to pin.
- [DEPLOY](DEPLOY.md) - the site-exposure surface a deploy reviewer needs.

158
docs/ASSET-COMPOSITION.md Normal file
View File

@@ -0,0 +1,158 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: When several devices share one identifier, file each under a parent asset - when to, and what it buys.
loadwith:
-->
# Composition: several devices under one parent
When more than one physical device answers to a single identifier, model each
device as its own asset and file it under a parent. This page is the pattern:
when to reach for it, what you get for free, and how to make the collector do it
without writing code. The mechanism itself is not new - `partof` and its
propagation rails are the ADR-001 platform contract; the recipe is.
## The problem it solves
A machine number is supposed to identify one thing. Sometimes it does not.
At the reference site, several Telesis part markers serve one operation number:
0613, 0615 and PRTMK01 each have more than one. Their configurations differ, most
often by COM port. Treating the operation as the device collapsed them into a
single record, and the damage was quiet:
- Their backups overwrote each other, so the history read as one device
flip-flopping between configurations that were really two devices.
- No question about an individual device could be asked at all - how many there
are, which port one is on, which one failed.
- Every PC driving one contested the single `controls` link to the operation,
so whichever reported last appeared to own it.
None of that announced itself. It surfaced as duplicate backup rows, weeks
later.
## When to use it
Reach for composition when ALL of these hold:
- Several physical devices share one identifier, and that is CORRECT - not a
numbering mistake. Two PCs carrying the same machine number by accident is a
fault to fix on the PC, not a model to build. `flask relationships
check-shared-machines` tells the two apart.
- The devices are individually interesting: they fail, get replaced, carry
their own configuration or calibration.
- Something already identifies each device. One device per PC is the easy case,
because the PC names it.
Do NOT reach for it when the parent is simply a location. A room holding six
printers wants a Location, not a parent asset.
## What you get
Filing a device `partof` a parent buys behaviour already built:
- **Control propagates.** `controls` propagates through `partof` (ADR-001, and
seeded by `flask seed reference-data`). A PC controlling a device therefore
controls its parent by inheritance, so the PC does NOT need - and must not
have - a direct link to the parent. That is what stops several devices
contesting a link only one can hold.
- **Position inherits.** Map-position resolution walks `partof` first, so a
device with no coordinates of its own shows at its parent's position.
- **History separates.** Anything keyed on the asset - backup revisions,
relationships, notes, audit - is per device instead of merged.
## Making the collector do it
A PC that drives a subordinate device declares it in
`plugins/computers/pctypemap.py`:
```python
SUBORDINATE_DEVICE_MAP = {
'gea-shopfloor-partmarker': {
'assettype': 'machine', # core AssetType for the device
'typename': 'Part Marker', # type within that vocabulary
'description': 'Telesis part marker',
'suffix': 'PARTMARKER', # device asset number = <PC>-<suffix>
'partof': True, # file under the reported machine number
'label': 'collector:partmarker',
},
}
```
`partof: True` is the switch that matters. It files the device under the
operation the PC reports AND stops that PC claiming the operation directly.
Leave it False for a device that does not share an identifier - a CMM is a
subordinate device too, but no two CMMs answer to one number, so it links to
its PC and nothing more.
`label` is the relationship origin marker. Only rows carrying it are archived
by a collector push, so links made by hand are never touched. **Do not change
an existing label**: those exact strings are in production databases.
A site adds or retargets an entry without a code change, per ADR-015:
```
Setting: subordinatedevice_<pctype> category: pctypemapping
Value: {"assettype": "machine", "typename": "Marking Laser",
"suffix": "LASER", "partof": true, "label": "collector:partmarker"}
```
A malformed override falls back to the built-in default rather than failing the
collector push - a bad setting must not stop a bay reporting its inventory.
## Making a backup kind follow the device
A backup posted by a PC that drives a device belongs to the DEVICE, not the
parent. `plugins/backups/services/registry.py` resolves this through
`markerforsource`: it finds the PC by the reported `sourcehostname`, follows the
PC's active device link, and files against what it finds. It falls back to the
machine number whenever the device cannot be resolved - no hostname on the
payload, a lean build without the computers plugin, or a PC that has not
reported yet - because filing under the parent beats rejecting a backup.
Two things that matter if you add a kind:
- **`sourcehostname` is load-bearing**, not informational. Without it a backup
files against the parent and merges two devices' histories.
- **A revision chain is (asset, kind, source hostname)**, so two devices on one
parent keep separate chains even before they become separate assets.
## Finding the next one
```
flask relationships check-shared-machines
```
Read-only. Lists every machine number claimed by more than one PC and separates
the two cases by whether child assets exist:
```
0615 11 PCs, 11 child asset(s) - modelled
2026 2 PCs, NO child assets - PCONE, PCTWO
```
The first is composition working. The second is two PCs carrying one number -
fix that on the PC.
## The parent is not disposable
Once devices are filed under it, the parent is doing a real job even though it
may look empty: it is the thing that says those devices belong together.
Deactivating it (`isactive = 0`, which is what "delete" does in the UI) breaks
filing, because the resolver requires an ACTIVE parent - every subsequent
report warns and files nothing.
A hard SQL `DELETE` is worse: `backuprevisions.assetid` is `ON DELETE CASCADE`,
so it destroys history still attached to the parent.
If the parent looks wrong in a list - an operation number appearing among
machines - that is a CLASSIFICATION question, not a deletion one.
## See also
- ADR-001 - relationship types, propagation rails, position inheritance
- ADR-015 - why the device map is settings-overridable
- [COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) - the collector contract
and the part-marker case end to end

52
docs/BACKUP-KINDS.md Normal file
View File

@@ -0,0 +1,52 @@
<!-- docmeta
purpose: reference
audience: developer
status: current
summary: How to add a backup kind to the backups plugin - the BackupKind members and what a new kind inherits for free.
loadwith:
-->
# Adding a fleet config backup kind
The recipe for adding a kind to the `backups` plugin - the per-PC CONFIG
backups the collector takes off shopfloor machines (NTLARS settings,
part-marker configs). Backing up the ShopDB database itself is
[BACKUP-RESTORE.md](BACKUP-RESTORE.md); why a Backup badge reads green or red
is [BACKUP-STALENESS.md](BACKUP-STALENESS.md), and you do not need it here.
---
A kind is one class in `plugins/backups/services/registry.py`, added to
`REGISTRY` at the bottom of that file:
```python
REGISTRY = {k.key: k for k in (NtlarsKind(), PartMarkerKind())}
```
Subclass `BackupKind` and override what applies. `NtlarsKind` is the fullest
example (parseable, renderable, has an info panel); `PartMarkerKind` is the
lean one.
| Member | What it is |
|--------|------------|
| `key` | Wire value. This is what lands in `backuprevisions.backupkind` and what the fleet table shows on the badge. Keep it short and lowercase. |
| `displayname` | Human label for the UI. |
| `storagebackend` | `'shopdb'` (bytes live in the DB, deduped on a semantic hash) or `'share'` (file lives on the SMB share, deduped on a raw byte hash). |
| `assettypes` | Which core asset types this kind can attach to; `['*']` for any. |
| `emptytext` | Text when an asset has no revisions of this kind. `None` HIDES the panel, which is the right default - a kind applies to an asset TYPE, but whether a given machine ever has that backup is per-machine. A part-marker panel on all 144 machines is noise. |
| `parse(raw)` | Opaque kinds return `None`; parseable kinds return a projection dict. |
| `formats()` / `render(...)` | Download formats, for `shopdb` kinds. |
| `resolveassetid(payload)` | Map a collector payload to the asset the backup belongs to. |
| `infopanel()` / `buildinfo(...)` | Optional at-a-glance card (ADR-010). Declared by the KIND, not hardcoded in the plugin, so a successor technology ships its own card by adding a class. |
| `sharedir(...)` | Conventional UNC directory for a `share` kind. Advisory - the authoritative path is whatever the collector reported, because the PC is what actually wrote the file. |
### What you do NOT have to touch
The **Backups stopped** dashboard card needs no change. It reads
`backuprevisions` generically: newest revision per chain, whatever the kind, and
shows `backupkind` as the detail line. A new kind inherits the card with no
work. The staleness rule needs no change either - it is time-based and
kind-agnostic ([BACKUP-STALENESS.md](BACKUP-STALENESS.md)).
Making a kind file against the DEVICE rather than the parent asset it hangs
under is [ASSET-COMPOSITION.md](ASSET-COMPOSITION.md).

202
docs/BACKUP-RESTORE.md Normal file
View File

@@ -0,0 +1,202 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Back up and restore a ShopDB instance - the database plus instance/ - on Docker or external MySQL.
loadwith:
-->
# Backup and restore the ShopDB database
This is the ShopDB instance itself. For the per-PC CONFIG backups the collector
takes off shopfloor machines, see [BACKUP-KINDS.md](BACKUP-KINDS.md).
Each site owns its own data (single-tenant, ADR-004), so backups are the site's
responsibility. A complete backup is two parts, and **a database-only backup is
not a backup**: restoring the database without the matching `instance/`
directory leaves the app pointing at floor plans and logos that no longer exist.
| Item | Location | Why |
|------|----------|-----|
| Database | MySQL `shopdb_flask` | All asset, user, audit and settings data. |
| `instance/branding/` | `instance/` dir | Uploaded logos and favicon. |
| `instance/modelimages/` | `instance/` dir | Uploaded vendor-model photos. |
| `instance/` floor plans | `instance/` dir | Uploaded map blueprints. |
| `instance/plugins.json` | `instance/` dir | Which plugins this site enabled. |
| `.env` | repo root (offline, secured) | Secrets needed to bring the stack back up. Store separately from the data backup, in a secrets manager. |
## The shipped Docker backup overlay
`docker-compose.backup.yml` is an opt-in per-site overlay that automates all of
the above. Turn it on with:
```bash
docker compose -f docker-compose.yml -f docker-compose.backup.yml \
--profile backup up -d
```
It adds three cooperating services: `backup-store`, a resident restic REST
server holding every snapshot in its own volume, deliberately NOT published to
the host so backups are reachable only over the compose network; `backup-watcher`,
which actuates the "Back up now" sentinel the api drops, because the api
container cannot run docker itself; and `backup`, the same one-shot run manually
(`run --rm backup`) for sites that prefer a host scheduler.
Each run captures one recoverable set: `db.sql.gz` (a non-blocking
`mysqldump --single-transaction`), a restic snapshot of `instance/`, and
`manifest.json` recording the core and plugin Alembic revisions plus the image
tag at capture time. That manifest is what makes a restore a CHECKED operation:
`scripts/backup/restore.sh` refuses to load a set onto code whose schema is
older than the set's, rather than leaving the site half-broken with rows
referencing columns the live schema has not applied. It does NOT roll back code
- restore the code first, run `flask db upgrade` and `flask plugin upgrade-all`,
then load the data.
Retention is restic-native: `--keep-daily 5 --keep-weekly 4 --keep-monthly 6`.
`RESTIC_REPOSITORY` defaults to the bundled store and can point at S3, Azure
Blob or a mounted share instead; credentials come from the site's env or secret
store, never committed.
## Backing up by hand
### Database (Docker)
```bash
docker compose exec -T db mysqldump \
-u root -p"${MYSQL_ROOT_PASSWORD}" \
--single-transaction --routines --triggers \
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
```
`--single-transaction` gives a consistent dump without locking the tables (InnoDB).
### Database (external MySQL, no container)
```bash
mysqldump -h <host> -u <user> -p \
--single-transaction --routines --triggers \
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
```
### instance directory
```bash
tar czf instance-$(date +%F).tar.gz instance/
```
Recommended cadence: nightly database dump to offsite storage, 14-day
retention; `instance/` captured on the same schedule (and always right before an
upgrade). Verify a restore quarterly.
## Restore
Restoring replaces the current database contents. Do it into a known-empty or a
throwaway target first if you are unsure.
### Step 1: Bring up the stack (or a fresh one)
```bash
cp .env.example .env # or restore your saved .env
# ensure MYSQL_* and DATABASE_URL match the dump's database name (shopdb_flask)
docker compose up -d db
```
Wait for the `db` container to report healthy (`docker compose ps`).
### Step 2: Load the database dump
```bash
gunzip -c shopdb-2026-07-10.sql.gz | \
docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask
```
For an external MySQL:
```bash
gunzip -c shopdb-2026-07-10.sql.gz | mysql -h <host> -u <user> -p shopdb_flask
```
If the target database does not exist yet, create it as utf8mb4 first (matching
the schema charset):
```sql
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
```
### Step 3: Restore the instance directory
**On Docker, `instance/` is a named volume, not the host directory.** The api
service mounts `instance_data:/app/instance`, so extracting the tarball into
`./instance` on the host restores nothing the application can see. The data has
to go into the volume:
```bash
tar xzf instance-2026-07-10.tar.gz # unpacks ./instance on the host
docker compose up -d api # creates the container and volume
docker compose cp instance/. api:/app/instance # copy into the volume
docker compose restart api
```
Check it landed before going on:
```bash
docker compose exec api ls /app/instance
```
`plugins.json` must be there. Without it the site comes back with its plugins
disabled and database rows pointing at uploaded files that are no longer
present.
A site that has replaced the named volume with its own bind mount
(`- ./instance:/app/instance`) restores by unpacking the tarball on the host and
nothing else. Check `docker-compose.yml` rather than assuming which one you have.
For a non-docker deploy, unpack the tarball over the application directory:
`instance/` is a plain directory there.
### Step 4: Reconcile migrations
```bash
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```
For a non-docker deploy, the same two commands without the prefix.
`flask db upgrade` is a safety net: if the dump predates the current code, this
applies only the core Alembic chain. `flask plugin upgrade-all` then applies any
newer per-plugin migrations (each bundled plugin owns its own chain, ADR-008);
without it, plugin-owned tables stay un-migrated. If the dump is at the same
version both are no-ops.
### Step 5: Verify
- Log in with a known account.
- Confirm the floor map renders (branding and map blueprints resolve from
`instance/`).
- Spot-check a few asset records and the audit log.
- `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .`
should return a `VALIDATION_ERROR`, not a 500.
## What is lost if `.env` is lost
`.env` is not in the database dump, and rebuilding it is not simply retyping it:
| Value | If it is lost |
|---|---|
| `SECRET_KEY`, `JWT_SECRET_KEY` | Every issued token and session becomes invalid. Users log in again; managed API tokens must be reissued, which means every collector client needs its key replaced. Recoverable, but it is a fleet-wide job. |
| `DATABASE_URL` password | Recoverable: reset the MySQL user's password and write the new one in. |
| `MYSQL_ROOT_PASSWORD` | Recoverable through MySQL's own reset procedure, which requires stopping the server. |
| `RESTIC_PASSWORD` | NOT recoverable. Every restic snapshot in the repository is encrypted with it; without it the backups are unreadable by anyone, including you. |
| `ZABBIX_TOKEN` and similar integration tokens | Reissue at the far end. Nothing else breaks. |
So back it up with the database, to somewhere as protected as the dump. A dump
without its `.env` restores the data and locks everyone out of it.
## See also
- [BACKUP-KINDS.md](BACKUP-KINDS.md) - the other backups: per-PC config
snapshots taken by the collector. A different subject entirely.
- [DEPLOY.md](DEPLOY.md) - first-time deploy
- [UPGRADE.md](UPGRADE.md) - upgrade procedure (back up first)
- [CONFIG.md](CONFIG.md) - environment variables and Setting keys

104
docs/BACKUP-STALENESS.md Normal file
View File

@@ -0,0 +1,104 @@
<!-- docmeta
purpose: explanation
audience: developer, operator
status: current
summary: Why the Backups-stopped card lists a chain, why it stays quiet about one, and why staleness is measured from lastseenat.
loadwith:
-->
# How the backups card decides good vs stale
The `backups` plugin's health verdict, as shown on the dashboard's **Backups
stopped** card. Read it when the card looks wrong, or before changing the rule.
Adding a new kind needs none of this - see [BACKUP-KINDS.md](BACKUP-KINDS.md).
## The trap this is built around
**A revision is only written when the config CHANGES.** Dedup compares against
the latest revision for the chain `(asset, kind, sourcehostname)`, so a machine
whose config has been stable for six months has a six-month-old newest revision
and is perfectly healthy. The question worth answering is not "when was the last
backup taken" - it is "is this still being checked".
So the timestamp the card uses is `backuprevisions.lastseenat`: the last time
the collector CONFIRMED this config, whether or not anything changed. It moves
on every successful collection; the revision does not.
It is a card of exceptions rather than a column of raw dates because a raw date
read as neglect. At the default `backups_intervalhours` of **24** the collector
only attempts once a day, so a day-old confirmation IS the healthy steady state -
and the date made the reader do arithmetic against a setting they would have to
go and look up.
## What it does now
`GET /api/backups/dashboard/stale` returns one row per chain that has gone
quiet, worst first:
| Field | Meaning |
|-------|---------|
| `assetid`, `assetnumber` | The asset the backup belongs to. |
| `backupkind` | Which kind stopped. Chains fail independently, so the kind is part of the identity, not a summary. |
| `sourcehostname` | The PC that was doing the collecting. |
| `lastseenat` | When this chain was last CONFIRMED still current (ISO). |
| `quietdays` | How long it has been quiet, which is what the card sorts on. |
`lastseenat` is reported rather than the capture time because they answer
different questions and one cannot stand for both: the first proves the check is
running, the second only says what is stored. Collapsing them was what made a
healthy machine look neglected.
## The chain is the unit, not the host
A marker bay runs both an NTLARS backup and a part-marker backup. Reporting one
verdict per host would let a healthy kind mask a dead one: the part-marker
capture could stop for a month and the host would look fine on the strength of
its NTLARS row. So the card is keyed on the CHAIN - asset, kind and source PC
together - because that is the unit that fails independently.
## Silence is not a verdict
A chain whose newest revision has never been confirmed - `lastseenat` NULL,
which means a backfilled row or one written before the column existed - is not
stale, it is UNMEASURABLE, and it is skipped rather than listed. Substituting
`collectedat` for it looked right and was wrong: on a real fleet it reported
every machine whose config had simply been stable for months as a stopped
backup, so the whole board lit up and said the site was broken when it was fine.
A chain becomes measurable the first time its PC posts, and the card says
nothing about it until then.
The card is equally silent about an asset that has NEVER been backed up. Whether
one SHOULD be is a question the asset record cannot answer - most machines carry
no NTLARS at all - and guessing would list a hundred healthy assets, which is
the noise that makes a board worth ignoring.
Known consequence: a device retired from a PC leaves its last revision behind,
so that chain stays listed until the revision is pruned. That is the honest
failure direction - it complains about a backup that genuinely is not running
any more, rather than going quiet about one.
The threshold is the backups plugin's own `backups_staledays` setting (default
**3**), read through `plugins.backups.services.staleness.staledays()` rather
than re-derived, so there is ONE definition of stale. `0` disables the card.
## Why time-based rather than per-kind
Every kind answers the same question the same way: something confirmed this
recently, or it did not. Making the rule per-kind would mean each new kind has
to define health before it can appear, for no gain. If a kind ever needs its own
window - a weekly backup that should not be judged on a 3-day rule - add a
threshold override on `BackupKind` and have the rule prefer it; the
skip-the-unmeasurable contract stays as it is.
## Where the code lives
- `plugins/backups/services/staleness.py` - `staledays()` (the shared
threshold), `latestperchain()` (newest revision per chain), `stalechains()`
(the verdict)
- `plugins/backups/api/routes.py` - `dashboard_stale`, a thin route over that
rule so it stays testable without an auth layer in the way
- `plugins/backups/plugin.py` - the `backups-stale` dashboard widget
declaration, including the `backups_staledays` setting
Adding a kind is [BACKUP-KINDS.md](BACKUP-KINDS.md); backing up the ShopDB
database itself is [BACKUP-RESTORE.md](BACKUP-RESTORE.md), a different subject.

292
docs/COLLECTOR-AUTH.md Normal file
View File

@@ -0,0 +1,292 @@
<!-- docmeta
purpose: reference
audience: adopter, integrator, developer
status: current
summary: The two ways a shop-floor PC authenticates to the collector API - its own per-PC credential, or the legacy shared key.
loadwith:
-->
# Collector authentication
**This page is the one authoritative description of how a fleet client proves
who it is.** Every other page links here rather than restating it.
There are two credentials, and which one a PC uses is a property of that PC, not
of the site:
| | Per-PC credential | Shared collector key |
|---|---|---|
| Status | **current** | **legacy - still works, do not build on it** |
| Header | `X-Endpoint-Credential` | `X-API-Key` (or `Authorization: Bearer`) |
| Identifies | this one PC | any holder of the key |
| Issued by | enrollment, once per PC | an admin, once per fleet |
| Revocable | per PC | only fleet-wide |
| Blast radius if it leaks | one PC's report | every PC's report |
A PC that has enrolled sends its credential and nothing else. A PC that has not
enrolled sends the shared key, exactly as the whole fleet used to. **Never both
on one request.**
Per-PC credentials need the `endpoints` plugin on the server and an endpoint
agent (or reporter script) that knows how to enroll. A site running older
clients is on the shared key and keeps working; that is the point of keeping
both.
> **Both halves are in this tree.** `plugins/endpoints/api/agentauth.py` resolves
> the credential and decides the shared-key fall-through,
> `plugins/endpoints/api/enrollment.py` carries enroll, approve, reject, revoke,
> allow-reenroll and the migration waves,
> `plugins/endpoints/models/credential.py` holds the three tables, and
> `shopdb/core/api/collector.py` reads the header on the generic collector
> route. The C# agent sends it
> (`agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs`) and both shipped
> PowerShell reporters fall back to the shared key on their own. The decision
> record is [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md); the wire
> contract is `plugins/endpoints/AGENT-CONTRACT.md` section 9, and the fleet
> rollout order is `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
---
## 1. Which endpoints take which
| Endpoint | Per-PC credential | Shared key |
|---|---|---|
| `POST /api/collector/computers` | yes | yes |
| `POST /api/collector/printers` | yes | yes |
| `POST /api/collector/endpoints` (and `/api/endpoints/collect`) | yes | yes |
| `POST /api/collector/<other plugin>` | only if that plugin's schema declares `acceptsagentcredential` | yes |
| `POST /api/collector/pc`, `/apps`, `/heartbeat`, `/bulk` | no | yes, see section 5 |
| `GET /api/collector/status` | no | yes |
| `GET /api/collector/_schemas` | no | no - JWT only, it is for humans |
A plugin's own unattended routes are outside this table. They authenticate with
a scoped managed service token, or with a source-IP allowlist, and they accept
neither a per-PC credential nor a collector key. That is the distinction people
get wrong: the collector credential says WHICH PC is writing a report, a service
token says WHAT an integration is allowed to do.
## 2. How the server decides
For a collector endpoint whose plugin schema declares `acceptsagentcredential`:
1. `X-Endpoint-Credential` present? Resolve it to an asset. Unknown or revoked
is `401 CREDENTIAL_INVALID`. The asset the credential resolves to is the asset
the request acts for, whatever hostname the body carries.
2. Otherwise fall through to shared-key auth (section 4).
3. If the shared key authenticated but the named PC **has enrolled**, the
request is refused `401 SHARED_KEY_REFUSED`. A PC with its own secret must
use it; a shared-key call in its name is an older agent or somebody else.
4. A payload that names a different asset than the credential is
`403 ASSET_MISMATCH`.
A PC that has NEVER enrolled has no credential row, is never refused by step 3,
and keeps reporting on the shared key indefinitely. That is what carries an
un-migrated fleet.
### Error codes
| HTTP | Code | Meaning | What the client does |
|---|---|---|---|
| 401 | `CREDENTIAL_INVALID` | unknown or revoked credential | delete the stored credential, retry once with the shared key, wake enrollment |
| 401 | `SHARED_KEY_REFUSED` | shared key sent for an enrolled PC, or the shared key is switched off | use the credential; if there is none, enroll |
| 403 | `ASSET_MISMATCH` | the credential's asset is not the one the request names | log and stop. Do not retry the same request |
| 401 | `Invalid API key` | the shared key is wrong or missing | fix the key |
| 500 | `Collector API key not configured` | server-side: no env key set and no managed token presented | a SERVER config gap, not a client problem |
`ASSET_MISMATCH` also covers a hostname that is on no computer record at all.
The code and status are the same because nothing changes for the client, but the
message says which of the two it is and names the hostname. An admin fixes it by
setting the hostname on the enrolled computer.
## 3. Per-PC credentials (current)
### Shape and storage
A credential is `sdbe_` plus 43 base64url characters (32 random bytes). The
server stores only its SHA-256 plus the first 12 characters for display; the full
secret exists exactly once, in the enrollment response.
On the PC it lives in
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
`LocalMachine`, in a folder ACL'd to SYSTEM and Administrators only. The file
records the `baseurl` it was issued for, and the credential must never be sent
anywhere else. `enroll.json` beside it holds the enrollment nonce in plain text.
### Enrolling
`POST {baseurl}/api/endpoints/enroll` with `hostname` and `serialnumber`
(required), plus an optional `deviceid`, an optional `agentversion`, and a
persisted `enrollnonce` of 32 to 128 characters from `A-Za-z0-9_-`.
The route never refuses a call for its authentication. A PC installed without the
shared key still records a request an admin can approve. The shared key only
decides whether a credential is issued on the spot.
Because it takes no auth and writes rows, it carries the two bounds an open door
needs: a per-IP fixed window of 300 calls a minute, sized for a whole shop floor
behind one address rather than for one PC, and a cap on how many PENDING rows
may exist at once (`plugin.endpoints.maxpendingenrollments`, default 500). The
cap turns away only a hostname and serial never seen before, and it drops
abandoned rows before it refuses a real PC, so neither bound can lock out a
machine already in the queue.
| Status | Body | What it means |
|---|---|---|
| 201 | `{"status":"enrolled","credential":"sdbe_...","assetid":123}` | issued. Store it and switch headers; no restart |
| 202 | `{"status":"pending","requestid":45,"reason":"serial mismatch"}` | queued for an admin. Retry on a backoff |
| 403 | `ENROLL_NONCE_MISMATCH` | the approved request belongs to a different nonce. An admin must reject and re-approve |
A credential is auto-issued only when ALL of these hold: the shared key
authenticated the call, `plugin.endpoints.autoenroll` is on, the hostname matches
exactly one computer, the serial on record matches (trimmed, case-insensitive),
and the asset has no active credential. Otherwise the request is pending with one
of: `unknown hostname`, `duplicate hostname`, `no serial on record`,
`serial mismatch`, `already enrolled`, `awaiting approval`, `nonce changed`,
`rejected`, `reenroll blocked`.
There is one request row per (hostname, serial number); repeated calls update it.
The nonce is what an approval binds to, so a second machine claiming the same
hostname and serial cannot collect the approved credential. A REJECTED row binds
to the identity instead of to the nonce: any nonce from that hostname and serial
stays rejected, so a reinstall cannot argue its way back into the queue, and
approving the request is the only way out of a rejection.
### Revoking, rotating, rolling back
Revoking sets `revokedat` on the asset's one credential row. Re-issuing updates
that same row, so an asset never carries two credentials.
By default a revoked PC **falls back to the shared key**. That is deliberate: a
revoke is also how an admin rotates a credential, and how a PC is rolled back to
an older agent that only knows the shared key. Denying by default would silence
that PC instead of rolling it back.
- `blockreenroll` on the revoke stops the PC enrolling again until an admin calls
`allow-reenroll`. That is what makes a revoke stick for a compromised PC.
- `plugin.endpoints.revokeddeniessharedkey` (default **off**) makes a revoke a
real eviction: the PC reports nothing until it enrolls again. Turn it on only
once the fleet is off the shared key, because it also closes the rollback path.
Admin API, for reference: `GET /api/endpoints/enrollment` (needs
`endpoints.fleet`), and the audited admin-role writes
`.../requests/<id>/approve`, `.../requests/<id>/reject`,
`.../assets/<assetid>/revoke`, `.../assets/<assetid>/allow-reenroll`.
### Server pins
A pinned agent accepts the TLS handshake if and only if the leaf certificate's
SubjectPublicKeyInfo SHA-256 is in its pin set; with no pins it requires a clean
Windows chain validation.
`HKLM:\SOFTWARE\GE\ShopDB` `ServerPins` (REG_MULTI_SZ, either registry view) is
authoritative whenever the value exists, **even when empty** - an empty value
means "registry authoritative, no pins". Only when it is absent from both views
do the `serverpins` entries in the agent's `appsettings.json` apply. The agent
logs which source is in effect at startup and on every change, and re-reads the
set within about a minute with no restart.
The Server Certificate settings page shows the pin; the client kit has
`Get-ShopDBServerPin.ps1` to compute one and `Set-ShopDBServerPins.ps1` to write
the set (`-None` for an empty authoritative value, `-Clear` to delete it).
## 4. The shared collector key (legacy)
### What counts as one
- A **managed API token** (`shopdb_pat_...`) scoped to ONLY `collector.ingest`,
or to `collector.ingest.<pluginname>` for one collector. This is the form to
use. It can be minted, rotated and revoked from Settings > API Tokens, and its
use shows in `lastusedat` and the audit log.
- An **environment key**: `COLLECTOR_API_KEY_<PLUGINNAME>` (per plugin,
uppercased) or `COLLECTOR_API_KEY` (shared fallback). Resolution is per-plugin
first, then shared. These stay supported as a bootstrap and legacy path.
An unscoped token, or one scoped to some other permission, is NOT a collector
credential and is rejected. `collector.ingest` gates no normal route and is
denied on every role-gated endpoint, so a scoped token that leaks can submit
collector payloads and do nothing else - not even with an admin owner, because
scoping suspends the admin bypass.
### Header only, fail-closed
The key travels in `X-API-Key` (Bearer also accepted for a managed token).
**The `?api_key=<key>` querystring form was removed on every collector
endpoint** - querystring keys leak into access logs, proxy history and browser
history. A caller still putting it in the URL gets 401.
If neither an env key nor a managed token is available server-side, every request
is refused. An unconfigured server never silently accepts unauthenticated data.
### Minting one
1. Settings > API Tokens > New Token.
2. Check **Restrict permissions**, tick only **Submit collector payloads (fleet
reporting)** (`collector.ingest`). Name it, optionally set an expiry, Create.
3. Copy the `shopdb_pat_...` secret, shown once, and deliver it to the fleet
(section 6).
If you prefer a non-admin owner, create a low-privilege user whose role holds
only `collector.ingest` and let it mint its own token; the scope ceiling then
caps anything it mints. The admin-minted route is simpler and equally contained.
### Rotating one
1. Mint the replacement.
2. Deliver it (section 6) - one value per site.
3. Watch the new token's `lastusedat` climb and the old one's go stale.
4. Revoke the old token. Revocation is immediate.
The endpoint agent's key has its own kill switch: Settings > Endpoint agent key
generates a key scoped to `collector.ingest.endpoints`, revokes any previous one
in the same request, and switches off the `.env` fallback for endpoints
(`plugin.endpoints.envkeyfallback`). Revoking it stops all endpoint-agent traffic
without touching the computers or printers collectors.
### Switching it off
`plugin.endpoints.sharedkeyallowed` (default on) gates the shared key for the
endpoints collector. Turning it off is the last step of a migration, not the
first: every PC that has not enrolled stops reporting the moment it flips.
## 5. The four legacy core routes
`POST /api/collector/pc`, `/apps`, `/heartbeat` and `/bulk` predate both the
generic collector contract and per-PC credentials. They take their target
hostname from the body and rewrite that PC's logged-in user, boot time, serial
number and installed apps, so the shared key is an **any-host write key** on
them and no per-PC credential narrows it. `plugin.endpoints.sharedkeyallowed`
does not reach them.
`collector.legacysharedkeyrefused` (default **false**, today's behaviour) closes
them to the shared key; a collector-scoped managed token still works. Every
shared-key use on these routes is logged whether the gate is on or off, so read
the "Legacy collector route used with the shared key" warnings for a while first
- whatever still calls them stops working the moment it is on. Neither shipped
reporter is affected; they post to `/api/collector/computers` and
`/api/collector/printers`.
Per ADR-006 the legacy `/pc` path is deprecated and slated for removal before
v1.0. New integrations target `POST /api/collector/<plugin>`.
## 6. Delivering the shared key to a PC
Do NOT hardcode a key in a script that lives on a file share: a literal key there
is published to everyone with share read. In order of preference:
1. **`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`**, written at imaging
time, by DSC, or by an Intune platform script. This is
where both shipped reporters look. Restrict the key's ACL to SYSTEM and
Administrators so an interactive or kiosk user cannot read it.
2. **A per-site config value** the machine account already reads -
`collectorApiKey` in `C:\Enrollment\site-config.json`. Rotation is one value
per site, no re-image and no script redeploy.
3. **A machine-local file** staged at imaging
(`C:\Enrollment\collector-api-key.txt`), ACL'd to SYSTEM. Tighter blast radius
- the key never sits on the share at all - but rotation means touching every
PC.
Never put a token in a scheduled task's `-Argument` string: task XML is
world-readable. Never put one in a manifest: manifests sync broadly.
A per-PC credential needs none of this. It is issued to the PC by enrollment and
never leaves it.

147
docs/COLLECTOR-CLIENT-SETUP.md Normal file
View File

@@ -0,0 +1,147 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Get the shipped asset and printer reporters running on a fleet, prove it, and fix it when a bay returns an error.
loadwith: COLLECTOR-AUTH.md
-->
# Running the collector reporters on a fleet
Follow this to put PC reporting on a fleet that does not have it yet, or to move
an existing fleet off a classic-ASP reporter. The payload contract is
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) and the credential choice is
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); you should not need either open to work
through this page.
Adopting the whole tool set at a plant that has none of it is a wider job -
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md).
---
## 1. Use the shipped reporters
Do not write your own. Two scripts ship in this repository and version with the
contract they implement:
| Script | Reports | Posts to |
|---|---|---|
| `plugins/computers/client/Report-AssetToShopDB.ps1` | hostname, BIOS serial, pc-type, logged-in user, make/model, OS, last boot, machine number, device id, every routable NIC | `POST /api/collector/computers` |
| `plugins/printers/client/Report-PrintersToShopDB.ps1` | every real print queue on the host | `POST /api/collector/printers` |
Both run as SYSTEM, always exit 0 so "last run result" stays clean, and log to
`C:\Logs\Shopfloor\`. Neither names a site: ADR-015 makes a site name in product
code a defect, and these ship everywhere.
Each reads its configuration in this order:
1. Its own parameters (`-ApiUrl`, `-ApiKey`), which a scheduled task's arguments
can supply for testing.
2. `HKLM:\SOFTWARE\GE\ShopDB` - `BaseUrl` for the server and `CollectorKey` for
the shared key. This is the secret store every shipped reporter reads, and
whatever images or configures the bay is what provisions it.
With neither, the script logs the reason and exits without posting.
**Which credential it sends** is decided on the PC, not in configuration: if the
PC holds a per-PC credential it is sent as `X-Endpoint-Credential` and the shared
key is not sent at all; otherwise the shared key goes as `X-API-Key`. If the
server refuses the credential as invalid or revoked, the reporter retries the
same POST once with the shared key, which is what makes a revoke a rollback
rather than an outage.
**Shortcut for a single site:** Settings > Computers > Asset reporter generates
`Report-AssetToShopDB.ps1` with your server's URL already in the parameter
defaults, and shows its SHA-256. The key is deliberately not included.
## 2. Prepare the server
1. Enable the `computers` plugin (and `printers`, if you want observed queues).
2. Mint the credential the fleet will start on - a managed token scoped to
`collector.ingest`. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 4 has the
steps. An un-enrolled fleet needs this even if you intend to move to per-PC
credentials, because enrollment itself auto-issues only to a call the shared
key authenticated.
3. Set `collector_corporate_ranges` to this site's routable CIDRs. Leave it blank
and the server falls back to the NIC the bay says carries the default route,
which is a guess on any bay with two NICs.
4. Map your imaging pc-types. Unmapped values only warn, but every report warns
until you do.
## 3. Deliver the key to the PCs
`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`, ACL'd to SYSTEM and
Administrators. How it gets there depends on what you run - an imaging task,
Intune, DSC or by hand - and the options with their trade-offs are in
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 6.
Two rules regardless: never bake the key into a script that lives on a file
share, and never put it in a manifest or a scheduled task's arguments.
## 4. Schedule the reporter
Reporting is an EVENT on a schedule, not a state to converge on. Two placements,
pick one:
- **As a scheduled task**, registered at imaging time, by Intune, or by hand.
SYSTEM, RunLevel Highest, at startup plus an interval.
- **As an entry in whatever enforces configuration on your floor**, run off a
share every cycle with no local copy. Order the key-provisioning step BEFORE
it so the key is present the same cycle the first report fires.
Either way, run it ALONGSIDE whatever reports today during rollout. Do not
remove the old path until the new one is proven.
## 5. Stage the rollout
1. Leave the existing reporting in place. Provision `CollectorKey` and enable the
plugins.
2. Add the reporter on a handful of pilot bays, one per pc-type. It cannot break
enforcement: it never throws and always exits 0.
3. Verify on the server. Pilot PCs appear or update, `action` is
`created`/`updated`, and every warning is understood - map any unmapped
`pctype`, add missing OS strings to the `operatingsystems` vocabulary.
Cross-check the client log's `OK` lines against the audit log.
4. Roll it out fleet-wide. From the `common` scope, every pc-type gets it.
5. Once ShopDB is the system of record and has survived a full imaging cycle,
retire the old reporter.
6. Migrate to per-PC credentials, then switch the shared key off. That is its
own rollout and deliberately comes last: read
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) for what a PC does to enroll and what an
admin approves, and `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`
for the order the shared key has to be cleared out of the places it lives.
Both shipped reporters already send a credential when one is present and fall
back to the shared key when it is not, so a PC that has not enrolled keeps
reporting and nothing has to be redeployed to start.
## 6. Verify one PC
```powershell
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>'
```
The PC should exist with its serial, type and last-reported timestamp. The
client log is `C:\Logs\Shopfloor\report-asset-YYYYMMDD.log` (printer queues:
`report-printers-YYYYMMDD.log`); it records the POST line and the server's
response verbatim, including which credential it used.
## 7. When a bay does not report
| Symptom | HTTP | Cause / fix |
|---|---|---|
| `Invalid API key` | 401 | The `X-API-Key` header is missing or wrong. Confirm the client key matches `COLLECTOR_API_KEY_COMPUTERS` (or the shared `COLLECTOR_API_KEY`) or is the managed token you minted. Check the client actually read it. The querystring form no longer works. |
| `Collector API key not configured` | 500 | Fail-closed: no env key set server-side and the caller presented no managed token. A SERVER config gap, not a client problem. |
| `Endpoint credential is unknown or revoked` | 401 | The PC's credential was revoked, or the database was restored past its issue. The reporter falls back to the shared key on its own; let the PC enroll again to restore it. |
| `The shared key is not accepted for this endpoint` | 401 | This PC has enrolled, so it must use its own credential - or the shared key is switched off site-wide. Check that `credential.bin` is present and readable by SYSTEM. |
| `The request names a different endpoint than the credential` | 403 | The hostname in the payload is on a different asset than the credential, or on no computer record at all. The message says which. Do not retry; fix the record. |
| `No collector registered for plugin computers` | 404 | The plugin is disabled or not loaded on that instance. Enable it. |
| `hostname is required` / `No data provided` | 400 | Empty body, non-JSON body, or missing identity field. Check `-ContentType 'application/json'`. |
| `Internal error processing collector payload` | 500 | Generic by design - the server does not leak the cause. The real error is in the server log (`Collector upsert failed for computers`). |
| `queues is required; send an empty array for a host with no queues` | 400 | The printers payload omitted `queues` entirely. Absent and empty are not the same thing here. |
| A bay's observed queues all vanished | 200 | It reported `queues: []`, which legitimately clears the host. Check the client log for `posting NOTHING` (a failed enumeration correctly sends nothing) versus an empty set. If the bay really has queues, fix the enumeration on that host. |
| Observed queues show as `unknown` | 200 | The queue matched no printer asset by port address or name. Add the printer to ShopDB with its IP and re-read - matching happens at read time, so the bay does not need to report again. |
| `warnings` present but `action` is created/updated | 200 | Soft issues only; the row WAS written. Common: unmapped `pctype`, unknown `osname`, unknown app name, `pcsubtype` not stored. No action needed unless the warning matters to you. |
| The wrong IP is recorded as primary | 200 | `collector_corporate_ranges` is blank or does not cover this bay. The bay reports every NIC and takes no view; set the ranges. |
Client-side logs are under `C:\Logs\Shopfloor\`. Server-side, the Flask app log
carries upsert failures and per-plugin schema failures.

85
docs/COLLECTOR-INTEGRATION.md Normal file
View File

@@ -0,0 +1,85 @@
<!-- docmeta
purpose: explanation
audience: operator, adopter, integrator, developer
status: current
summary: What the collector is, what it guarantees, and which of the three collector pages to open next.
loadwith:
-->
# The collector
The collector is how the shop-floor fleet pushes what it knows about itself into
ShopDB. It replaced the classic-ASP `api.asp?action=updateCompleteAsset` path.
This page is the shape and the guarantees. It is short on purpose: the detail
lives in three pages below, and you almost never need more than one of them.
## Which page you want
| You are | Open |
|---|---|
| Rolling reporting out to a fleet, or a bay is returning an error | [COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md) |
| Writing or changing a client, or answering "what does this field do" | [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) |
| Deciding how a PC proves who it is, minting or revoking a credential | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| Asking why the contract is shaped this way | [ADR-006](adr/ADR-006-collector-contract.md) |
Where all of this sits relative to EventSaver, the backup collectors and the
kiosks is [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md).
## The shape
One dynamic route, `POST /api/collector/<plugin>`, serves every enabled plugin
that returns a collector schema. A plugin declares its identity field and its
accepted fields; the dispatcher handles auth, parsing, the audit row and the
error envelope, and the plugin handles the upsert. Adding a collector is adding a
schema, not adding a route.
Two plugins use it on a normal fleet:
- `computers` - what this PC is. This is the record everything else hangs off.
- `printers` - what print queues this PC actually has.
The `endpoints` plugin adds a third, much larger payload for sites running the
endpoint agent.
## What it guarantees
**Idempotent.** The same report twice changes nothing. A PC is matched by its
identity field, so re-imaging a bay updates its record instead of duplicating it,
and existing asset relationships survive.
**Patch-style, where it says so.** On the computers payload an omitted field
leaves the stored value alone, so a report from a bay that could not read its
BIOS never blanks a serial number. The printers payload is deliberately the
opposite: the reported set REPLACES the host's queues, because "what does this
bay have right now" is not a question about history.
**Warnings, not failures, for soft problems.** An unmapped pc-type, an unknown OS
string or an unknown application name is reported back in `warnings` and the row
is still written. The alternative - refusing the report - loses the whole PC over
one unrecognised string.
**Observed is never assigned.** Nothing a PC reports becomes a statement of what
it SHOULD have. A bay that installed the wrong printer cannot make itself right
by reporting it; turning observed state into assigned state is always a person's
decision.
**The report says what the PC is, not where it is on the network.** The bay
reports every NIC and takes no view of which matters. Only the server knows the
site's addressing, so only the server picks.
**Fail-closed on authentication.** A server with no credential configured refuses
every request rather than silently accepting unauthenticated data. Credentials go
in a header, never a querystring, so they stay out of access logs and proxy
history.
## The one thing that changed recently
A PC used to authenticate with a key the whole fleet shared. It now
authenticates as ITSELF, with a per-PC credential issued by enrollment and
revocable one PC at a time. The shared key still works for PCs that have not
enrolled, and is refused for any PC that has.
If you are adopting this product today, plan on per-PC credentials and treat the
shared key as the bootstrap. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole
story.

399
docs/COLLECTOR-PAYLOADS.md Normal file
View File

@@ -0,0 +1,399 @@
<!-- docmeta
purpose: reference
audience: integrator, developer
status: current
summary: Field-by-field contract for the computers and printers collector payloads, and what the server does with each.
loadwith: COLLECTOR-AUTH.md
-->
# Collector payload contract
What a fleet client may send to `POST /api/collector/<plugin>` and what the
server does with each field. Authentication is a separate subject and lives in
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); the rationale for the contract shape is
[ADR-006](adr/ADR-006-collector-contract.md).
Source of truth is `get_collector_schema` in each plugin
(`plugins/computers/plugin.py`, `plugins/printers/plugin.py`). If you change a
payload, change it there and re-check the table here.
---
## 1. The generic endpoint
One dynamic route serves every enabled plugin that returns a collector schema.
For the PC fleet that is `POST /api/collector/computers`.
Request flow inside `generic_collect`:
1. Look up the plugin's schema. Unknown or disabled plugin -> `404 No collector
registered for plugin <plugin>`.
2. Authenticate (COLLECTOR-AUTH).
3. Parse the JSON body. Missing or non-object -> `400 No data provided`.
4. Resolve the identity field the schema names (`hostname` for both collectors
below). Missing or blank -> `400 <identityfield> is required`.
5. Idempotent upsert: the plugin's `apply_collector_payload` finds the existing
asset by identity and updates it, or inserts a new one. The same identity on a
later submission updates the same row - re-imaging a PC does not duplicate it,
and existing asset relationships are preserved.
6. Audit: every accepted submission writes an `AuditLog` row (`action`
created/updated, entity `Collector`, `details = collector:<plugin>
action=<action>`), then commits.
### Response
HTTP 200, in the standard envelope, with the collector result under `data`:
```json
{
"status": "success",
"data": {
"status": "ok",
"action": "created",
"assetid": 12345,
"identityvalue": "SHOPPC2335",
"warnings": ["unknown operating system: Windows 11 Enterprise 23H2 (build 22631)"]
},
"message": "computers collector created",
"meta": { "timestamp": "...", "requestid": "..." }
}
```
`action` is `created`, `updated` or `noop`. `warnings` is a list of soft problems
(unmapped pc-type, unknown OS, unknown app, un-stored pcsubtype) that did NOT
fail the request - the row was still written.
Errors use the same envelope with `status: "error"` and the detail under
`data.error`:
```json
{ "status": "error", "data": { "error": { "code": "UNAUTHORIZED", "message": "Invalid API key" } }, "meta": { } }
```
A plugin raises `ValueError` for controlled validation failures and its message
is returned verbatim (400). Any other exception is caught, logged server-side,
the transaction rolled back, and the caller gets a generic `500 Internal error
processing collector payload` with no internal detail - by design. Check the
server log to see what actually failed.
### Schema discovery
`GET /api/collector/_schemas` returns the schema for every enabled plugin. It is
JWT-protected, not key-protected: it is for humans and tooling introspecting what
payloads are accepted, not for the headless collectors themselves.
---
## 2. Computers payload
Identity field `hostname`. Every other field is optional and patch-style: an
omitted field leaves the existing value untouched, so a bare report never blanks
a column.
| Payload field | Type | Server behaviour (`apply_collector_payload`) |
|---|---|---|
| `hostname` (required) | string | Identity. Matches `Computer.hostname` (case-insensitive), then falls back to `Asset.assetnumber`. New asset created if no match. |
| `machinenumber` | string | Identifies the MACHINE this PC drives, never the PC. A new PC always takes `assetnumber = hostname`, and an existing PC's `assetnumber` is left alone. The number resolves a machine asset and builds a PC -> machine `controls` relationship; an unknown number warns rather than creating a machine. The placeholder `9999` and empty string link nothing. See section 3. |
| `deviceid` | string | Identifies the DEVICE hanging off this PC, read from `C:\Enrollment\asset-id.txt`. It names the asset and nothing else - it does NOT say what kind of device it is, because the pc-type already does - so the same file serves a part marker, a Keyence, a Genspect or anything declared later. SEPARATE from `machinenumber` on purpose: "which bay is this" and "which device is this" are different facts, and `machinenumber` is what a site's own bay-gated configuration keys off, so naming a device there would silently stop every bay-gated rule from matching. An unknown value warns rather than minting; a value resolving to the wrong KIND of asset is refused with the asset named. See section 4. |
| `measuringtoolid` | string | The 0.12.0 name for the same thing, read from `C:\Enrollment\measuringtool-id.txt`. Still accepted as an alias so a bay already staged with that file keeps reporting; `deviceid` wins when both arrive. Prefer `asset-id.txt` for anything new. |
| `pctype` | string | `gea-shopfloor-*` imaging type -> `Computer.computertypeid` via the configurable `pctypemap` settings. Unmapped value -> warning, not error. See section 5. |
| `pcsubtype` | string | Accepted but not stored yet -> warning. |
| `serialnumber` | string | `Asset.serialnumber`. |
| `loggedinuser` | string | `Computer.loggedinuser`. (`currentuser` is also accepted as a legacy alias.) |
| `lastboottime` | ISO-8601 datetime | `Computer.lastboottime`. Unparseable -> warning. |
| `lastcheckin` | ISO-8601 datetime | Accepted (heartbeat semantics). `lastreporteddate` is set server-side on every call regardless. |
| `interfaces` | array of `{ipaddress, macaddress, adaptername, isdefaultroute}` | EVERY connected hardware NIC with a routable IPv4, in no meaningful order. One `Communication` row per NIC. See "Which NIC is the PC's address" below. |
| `ipaddress` | string | The one-address form, kept for bays on an older collector. A payload carrying both wins on `interfaces`. |
| `vendorname` | string | `Computer.vendorid`. Vendor row auto-created if missing (free vocab). |
| `modelnumber` | string | `Computer.modelnumberid`, scoped to the vendor when known. Model row auto-created if missing. |
| `osname` | string | `Computer.osid`. Controlled vocab: looked up in `operatingsystems`, NOT auto-created. Unknown value -> warning (row still written, `osid` left unset). |
| `installedsoftware` | array of `{name, version}` | `ComputerInstalledApp` rows for applications shopdb already tracks. Unknown app name -> warning, skipped. |
| `accessprotocols` | array of strings | Remote-access protocols this PC actually exposes, by catalog name (`VNC`, `WinRM`, `RDP`). PRESENCE of the key drives the sync: reported protocols are activated, catalogued ones not reported are deactivated. Omit the key entirely to leave existing rows alone - most came from the legacy `isvnc`/`iswinrm` migration. |
`defaultprinter` and `printers` are **retired**. See section 6.
### Which NIC is the PC's address
The bay reports every NIC and takes no view. The server picks, in this order:
1. The first reported address inside a CIDR in the `collector_corporate_ranges`
setting.
2. Else the NIC the bay flagged `isdefaultroute`.
3. Else the first reported.
Step 3 is a guess, and configuring the ranges is what makes it stop being one.
Choosing on the bay was the wrong place for it: a bay cannot know a site's
addressing, so a CMM with a fast link to its controller reported the machine-side
address as the PC's and never sent the corporate one at all.
Rows are keyed on the MAC where the bay sent one, because an address moves
between adapters and a MAC does not, and fall back to the address for a bay that
reports none. Exactly one row is flagged primary, and only among the rows this
report touched. A NIC the bay did NOT report is left alone: a NIC that is down at
check-in is not evidence the record is wrong, and hand-entered detail lives on
those same rows.
## 3. PC -> machine links, and part markers
The reported `machinenumber` says which machine the PC drives. It is not the
PC's own identifier, and writing it as one is a defect this contract used to
describe: `assets.assetnumber` is uniquely indexed and the machine already holds
that value, so the insert failed and the bay got a 500 on every report, forever.
A PC takes its hostname as its asset number and keeps it.
- Link: PC -> machine as `controls`, tagged
`assetrelationships.label = 'collector:machine'`. Only tagged rows are ever
archived by a push, so a link made by hand is never touched.
- Unknown number: warns and links nothing. A machine is never invented, because
a mistyped number would create equipment nobody can account for.
- A second PC reporting a machine another PC holds is a CLAIM, not a handover.
A PC imaged for a machine carries that number before it reaches the floor, so
the PC currently running the machine keeps the link while it is still alive -
reporting within 24 hours (`MACHINE_CLAIM_QUIET_HOURS`) and still `In Use`.
The challenger is recorded as a dormant link, which also marks the claim as
already announced so it does not alert every cycle.
- Handover completes on its own once the incumbent has been quiet past that
window, or immediately when someone moves it off `In Use` - which is the
one-step way to force a swap at the moment it happens. The old link is
archived, never deleted, so "which PC ran this machine in June" stays
answerable. The retired PC's STATUS is deliberately not changed: the collector
cannot tell shelved from broken from re-imaged.
- Alerts for handover and contested claims are gated on the
`computers_machinelink_alerts` setting, which ships OFF. A site may run
several PCs on one machine number legitimately, and there the alerts fire on
correct data. Warnings ride in the collector response either way.
A PC reporting `pctype = gea-shopfloor-partmarker` is handled differently,
because several markers can serve one machine number and an operation holds any
number of them. Such a PC gets its own Part Marker machine asset
(`<HOSTNAME>-PARTMARKER`), `controls` it, and the marker is filed `partof` the
operation the PC reported. Both links are tagged `collector:partmarker`. The PC
does NOT claim the operation directly: `controls` propagates through `partof`,
so control still follows from controlling the marker, without two markers
contesting a link only one can hold. Backups from a marker PC file against the
marker rather than the operation.
That is one instance of a general pattern - several devices under one parent.
[ASSET-COMPOSITION](ASSET-COMPOSITION.md) covers when to use it, what
propagation buys you, and how to declare a new device type without a code change.
## 4. PC -> device links (`asset-id.txt`)
Some pc-types mean the PC drives an attached DEVICE: a part marker, or a
metrology instrument such as a CMM, Keyence, Genspect or wax-trace unit.
`SUBORDINATE_DEVICE_MAP` in `plugins/computers/pctypemap.py` declares which, and
a site can add its own through a `subordinatedevice_<pctype>` setting without a
code change.
**On this line only the part-marker case does anything.** The four metrology
pc-types declare `assettype: measuring_tool`, and the plugin that owned that
asset class is not bundled here (ADR-018: metrology instruments are machines).
The collector detects the absence, skips the link, and says so in the response
warnings - it does not fail the report, and it does not mint anything. File a
CMM as a machine and link it by hand.
`C:\Enrollment\asset-id.txt` names that device: one line, the device's
`assetnumber`. It does NOT record what kind of device it is - the pc-type already
does - so a bay declared later needs no new file and no client change.
Resolution runs most-stable-identity first, and ADOPTS before it mints:
1. **`deviceid`** - the asset named in `asset-id.txt`.
2. A prior collector link from this PC, reactivated.
3. An existing device this PC already controls that the collector did NOT
create - a legacy `MT-####` row, or one somebody made by hand. Adopting
stamps the label so it is ours from then on.
4. Mint, only when none of the above matched.
**Why the file exists, and why it is step 1.** Every other identity here is
derived from the PC. Step 2 looks for a link from THIS PC asset; step 4 mints
`<PC number>-<SUFFIX>`, and a PC's asset number is its hostname. All of them
survive a re-image and NONE of them survives a PC SWAP: a new hostname is a new
PC asset with no prior link and a predicted number that has never existed, so
the same physical device gets a second record while the first keeps its config
and backup history under a dead PC's name. That is how 43 legacy `MT-####`
tools ended up shadowed by minted twins, three records deep in places. Treat
anything minted as a placeholder to be reconciled.
Two guards on the file's contents:
- An **unresolvable** value warns and links nothing, rather than inventing a
phantom device nobody can account for.
- A value resolving to the **wrong kind** of asset is refused, naming the asset
it hit. A machine number pasted into `asset-id.txt` would otherwise file that
MACHINE under a device label the collector also owns - a link that reads as an
instrument, or as a marker, everywhere downstream.
A part marker additionally stays `partof` the operation from `machinenumber`,
because several markers serve one operation. Naming the marker in `asset-id.txt`
changes which marker asset the PC controls; it does not change which operation
that marker files under.
## 5. pc-type mapping (configurable per site)
`pctype` (e.g. `gea-shopfloor-cmm`) maps to a shopdb Computer Type through
`pctypemap_<pxetype>` settings. The built-in defaults in
`plugins/computers/pctypemap.py` are seeded on plugin install and the collector
reads them, so existing enrollment keeps working; add a setting of that name to
map an imaging type of your own. Unmapped pc-types produce a warning, not a
failure, because a bay must not stop reporting its inventory over a naming
question.
## 6. PC -> printer relationships are NOT set here (retired)
**A collector report has never been allowed to set a printer assignment, and as
of 2026-08-24 it no longer can.** `defaultprinter` and `printers` used to be
accepted on the computers payload and written as `defaultprinter`/`connectedto`
relationship rows. That contradicted "observed is not assigned" below, and this
contract carried both claims at once while the code followed the wrong one.
If you are still sending either key: **your push still succeeds** - they are
ignored, and the response carries a warning naming them. To report what a PC
actually has, use `POST /api/collector/printers`; to turn that into an
assignment, a person calls
`POST /api/printers/assignments/seed-from-observed/<assetid>` after reviewing the
comparison.
## 7. Printers payload (observed print queues)
`POST /api/collector/printers`. ShopDB has always known what a bay SHOULD have -
`GET /api/printers/for-host/<hostname>`, applied by `Set-ShopdbPrinters.ps1` -
and has never known what it actually has. This payload is that missing half.
Same dispatcher, same auth, same audit row; only the payload and the plugin
differ.
- Schema + apply: `get_collector_schema` / `apply_collector_payload` in
`plugins/printers/plugin.py`.
- Client: `plugins/printers/client/Report-PrintersToShopDB.ps1` (SYSTEM context,
`Type=PS1` / DetectionMethod `Always` manifest entry, logs to
`C:\Logs\Shopfloor\report-printers-YYYYMMDD.log`, always exits 0).
- Storage: `printerobservedqueues`, one row per observed queue, owned by the
printers plugin. A separate table from the assignment on purpose.
The identity field is `hostname`. Unlike the computers payload this one is NOT
patch-style: `queues` is required, and the reported set replaces everything
previously recorded for that host.
| Payload field | Type | Server behaviour |
|---|---|---|
| `hostname` (required) | string | Identity of the report. `COMPUTERNAME` or its FQDN; matched case-insensitively. Rows are keyed by the NAME, so a bay that reports before its PC record exists still records everything - the report resolves to an asset the moment that record appears. An unknown hostname is a warning, never an error. |
| `queues` (required) | array of objects | Every real print queue on the host. This REPLACES the host's previous set. An empty array is a valid report meaning "this bay has no queues" and clears the host's rows. A missing key is rejected (400), because one client bug that dropped the field would otherwise erase the fleet's observed state host by host. |
| `queues[].queuename` (required) | string | Windows printer name. A queue with no name is skipped with a warning; a repeated name within one report is dropped with a warning (Windows cannot hold two queues of one name). |
| `queues[].drivername` | string | Driver name verbatim, as the INF spells it. Compared against the assignment's driver to detect drift. |
| `queues[].portname` | string | Windows port name. Also tried as a match key, since a port created outside the client is usually named after the host address. |
| `queues[].portaddress` | string | `PrinterHostAddress` of a TCP/IP port - an IP or FQDN. The PRIMARY key for matching a queue to a printer asset. Omit it for a non-TCP port (USB, WSD, redirected); such a queue is still worth reporting and matches on name alone. |
| `queues[].isdefault` | boolean | True on the one queue that is the user's default. If several arrive true, the first is kept and the rest are cleared with a warning - a bay has exactly one default, and keeping both would leave a seed picking one at random. |
| `queues[].isshared` | boolean | True when the queue is shared off this PC. Recorded, not acted on. |
| `observedat` | ISO-8601 datetime | Accepted and ignored, as is any other client timestamp. The server stamps `observedat` at ingest, one value for the whole report, so a bay with a wrong clock cannot report itself fresh or stale. |
Response `data` is the standard collector shape: `action` is always `updated`
(this endpoint replaces rows and creates no asset - calling an identical
re-report `noop` would hide that the bay is still checking in), `assetid` is the
resolved PC or `null`, `extra.queuecount` is how many rows were stored, and
`warnings` carries the soft problems above.
```
POST /api/collector/printers
X-Endpoint-Credential: sdbe_...
Content-Type: application/json
{
"hostname": "WORKSTATION01",
"queues": [
{
"queuename": "Bay Label Printer",
"drivername": "Generic / Text Only",
"portname": "IP_192.0.2.40",
"portaddress": "192.0.2.40",
"isdefault": true
},
{
"queuename": "Office Laser",
"drivername": "HP Universal Printing PCL 6",
"portname": "IP_192.0.2.41",
"portaddress": "192.0.2.41",
"isdefault": false
}
]
}
```
### The latest report replaces the previous one
This is CURRENT STATE, not history. Each report deletes every row previously
recorded for that hostname and writes the reported set in its place, inside the
dispatcher's transaction. Nothing accumulates, "what does this bay have" is
never a question about time, and a queue removed from a bay disappears from
ShopDB on the next cycle without anyone tidying up.
That has one sharp edge, and it belongs to the client: an empty `queues` array
is a legitimate report that wipes the host's rows. A client whose enumeration
FAILED must therefore send nothing at all rather than an empty list. Reporting
nothing loses one cycle; reporting `[]` after a WMI hiccup deletes real state
and reads as a bay that lost its printers. `Report-PrintersToShopDB.ps1` tracks
this with an `$enumerated` flag and exits without posting when both the cmdlet
and the WMI fallback failed.
### Observed is not assigned
Stated plainly, because it is the whole point of keeping two tables:
**A collector report NEVER becomes an assignment.** Nothing on this path writes
a `usesprinter` or `defaultprinter` row. The moment a drifted bay's observed
state is treated as correct, enforcement stops meaning anything - a bay that
installed the wrong printer would make itself right simply by reporting it.
Observed state becomes assigned state only when a person asks for it, through
`POST /api/printers/assignments/seed-from-observed/<assetid>` (requires
`printers.edit`), after reviewing the comparison. That route refuses rather than
guesses: it will not seed from an ambiguous host, will not seed queues that
match no printer unless told to, and will not write an empty set.
### Matching, and what "unknown" means
Nothing is matched at ingest - the raw observed strings are stored as reported,
and resolution to a printer asset happens at READ time, so a printer added to
ShopDB tomorrow matches yesterday's report without the bay reporting again.
At read time each queue is matched by PORT ADDRESS first (an IP or FQDN names
one device unambiguously), then by queue name. There is no fuzzier fallback: a
queue that matches nothing is reported as `unknown` rather than guessed, because
a wrong match seeds a wrong assignment, which is worse than no assignment.
### Reading it back
| Endpoint | Purpose |
|---|---|
| `GET /api/printers/observed/<hostname>` | What the host last reported, every queue classified against the assignment it resolves to (`matching`, `drifted`, `extra`, `missing`, `unknown`), with `observedat`, a summary, and a `seedcandidate` preview. Read-only. Needs `printers.view`. |
| `POST /api/printers/assignments/seed-from-observed/<assetid>` | The one path from observed to assigned, human-triggered. Needs `printers.edit`. Normally posted against the MACHINE, so the assignment survives a reimage. |
Server-side the printers key resolves per-plugin first: set
`COLLECTOR_API_KEY_PRINTERS` if you scope env keys per collector, or rely on the
shared `COLLECTOR_API_KEY`. A site that has scoped the computers key per-plugin
and set no printers key gets 401 here until one of the two is in place. A
per-PC credential and a `collector.ingest` managed token both work here exactly
as they do for computers.
## 8. Porting from classic `api.asp`
The fleet's classic-ASP reporter posts form fields to
`api.asp?action=updateCompleteAsset`. Map them to the collector JSON as follows:
| Classic `updateCompleteAsset` form field | Collector field |
|---|---|
| `hostname` | `hostname` |
| `machineNo` | `machinenumber` |
| `deviceId` | `deviceid` |
| `pcType` | `pctype` |
| `serialNumber` | `serialnumber` |
| `loggedInUser` | `loggedinuser` |
| `lastBootUpTime` / `lastBootTime` | `lastboottime` |
| `manufacturer` | `vendorname` |
| `model` | `modelnumber` |
| `osVersion` | `osname` |
| `installedApps` | `installedsoftware` |
Not carried over (no current home in the computers schema): warranty fields, DNC
config, VNC/WinRM flags as booleans (report them through `accessprotocols`
instead). The classic reporter's full `networkInterfaces` array maps onto
`interfaces`.

352
docs/CONFIG.md Normal file
View File

@@ -0,0 +1,352 @@
<!-- docmeta
purpose: reference
audience: operator, adopter, developer
status: current
summary: Every environment variable and Settings key, what it defaults to and which one to reach for.
loadwith:
-->
# Configuration reference
**To find one setting: search this page for its exact key.** Every key appears
in exactly one table. Nothing here has to be read in order.
Two places, and the split is deliberate:
- **Environment variables** hold **secrets and deploy-time wiring**: database
credentials, signing keys, CORS origins, ports, API keys. Read once at boot by
`shopdb/config.py`, which is authoritative - where this page and that file
disagree, the file wins.
- **The Settings table** holds **site preferences**: branding, ServiceNow links,
search toggles, facility identity. Seeded by `flask seed settings`, edited in
the UI under Settings or the setup wizard, changeable at runtime with no
restart. `build_default_settings` in `shopdb/core/api/settings.py` is
authoritative for the defaults.
Rule of thumb: if leaking it would be a security incident, it is an environment
variable. Never put a secret in the Settings table.
**Part 1, environment variables:** [Flask core](#flask-core) |
[Database](#database) | [Auth rate limiting](#authentication-rate-limiting) |
[Collector ingest](#collector-ingest-adr-006) | [Zabbix](#zabbix-printer-supply-monitoring) |
[Employee directory DB](#employee-directory-database-optional-read-only) |
[docker-compose only](#docker-compose-only)
**Part 2, Settings keys:** [site](#site) (facility name, timezone, base URL,
retention) | [branding](#branding) | [printing](#printing) (QR and label
targets) | [map](#map) (superseded) | [integrations](#integrations)
(ServiceNow, Zabbix, Dell warranty) | [email](#email) | [audit](#audit) |
[auth](#auth) (SAML placeholders) | [tools](#tools) (Secure Send) |
[identifiers](#identifiers-dynamic) | [search](#search-dynamic) |
[Custom fields](#custom-fields)
---
## Part 1: Environment variables
Copy `.env.example` to `.env` and fill in values. In `production`
(`FLASK_ENV=production`), `ProductionConfig.validate()` refuses to boot if
`SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, or `CORS_ORIGINS` are missing or
set to the dev defaults.
### Flask core
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `FLASK_APP` | No | `wsgi.py` | Entry point for the `flask` CLI. |
| `FLASK_ENV` | Yes | `development` | `production` for live sites (triggers `validate()`). Other values: `development`, `testing`. |
| `SECRET_KEY` | Yes (prod) | dev default | Flask session/signing key. Generate: `python -c "import secrets; print(secrets.token_urlsafe(64))"`. |
| `JWT_SECRET_KEY` | Yes (prod) | dev default | JWT signing key. Different value from `SECRET_KEY`. |
| `JWT_ACCESS_TOKEN_EXPIRES` | No | `3600` | Access-token TTL in seconds. |
| `JWT_REFRESH_TOKEN_EXPIRES` | No | `2592000` | Refresh-token TTL in seconds (30 days). |
| `CORS_ORIGINS` | Yes (prod) | `http://localhost:5173` | Comma-separated explicit origins. Wildcard `*` is rejected in production. |
| `LOG_LEVEL` | No | `INFO` | Logging verbosity. |
### Database
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `DATABASE_URL` | Yes (prod) | dev localhost URL | `mysql+pymysql://<user>:<pass>@<host>:<port>/<db>?charset=utf8mb4`. Keep `?charset=utf8mb4`. |
### Authentication rate limiting
IP-based fixed-window limit on the login endpoint, defense-in-depth atop the
per-account lockout. Uses the cache extension, which is per-process, so the
limit is approximate across multiple gunicorn workers.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `AUTH_RATELIMIT_ENABLED` | No | `True` | Set `False` to disable (TestingConfig disables it). |
| `AUTH_RATELIMIT_MAX` | No | `30` | Max login attempts per source IP per window before 429. |
| `AUTH_RATELIMIT_WINDOW_SECONDS` | No | `300` | Window length in seconds. |
### Collector ingest (ADR-006)
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `COLLECTOR_API_KEY` | No | (empty) | Shared key for `/api/collector/*`. Endpoint fails closed (denies) when unset and no managed token is presented. Sent as the `X-API-Key` header. |
| `COLLECTOR_API_KEY_<PLUGIN>` | No | (empty) | Per-plugin override, e.g. `COLLECTOR_API_KEY_COMPUTERS`. Checked before the shared key. |
These env keys are the bootstrap/legacy fallback. A managed API token scoped to
`collector.ingest` is preferred and is minted, rotated and revoked from
Settings > API Tokens; see
[COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) (Managed collector tokens).
### Zabbix (printer supply monitoring)
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `ZABBIX_ENABLED` | No | `false` | Enable the Zabbix integration. |
| `ZABBIX_URL` | No | (empty) | Zabbix API URL. |
| `ZABBIX_TOKEN` | No | (empty) | Zabbix API bearer token. |
These are the boot-time wiring; the Settings entries (`zabbix_enabled`,
`zabbix_url`, `zabbix_token`) are preferred for runtime changes.
### Employee directory database (optional, read-only)
Separate HR lookup DB, read-only. This is where staff names come from: global
search resolves the employee domain through it, and the audit log uses it to
show a name instead of a bare SSO. Nothing here is required - leave it unset and
both surfaces fall back to SSOs. There is no safe default for the password; an
unset one fails loud rather than trying a guessed credential.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `EMPLOYEE_DB_HOST` | No | `localhost` | HR DB host. |
| `EMPLOYEE_DB_USER` | No | (empty) | HR DB user. |
| `EMPLOYEE_DB_PASSWORD` | No | (empty) | HR DB password. No safe default. |
| `EMPLOYEE_DB_NAME` | No | `wjf_employees` | HR DB name. |
### docker-compose only
Read by `docker-compose.yml`, not by the Flask app directly.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `MYSQL_ROOT_PASSWORD` | Yes | (none) | Root password for the bundled MySQL container. |
| `MYSQL_PASSWORD` | Yes | (none) | App-user password; must match the `DATABASE_URL` password. |
| `MYSQL_PORT` | No | `3306` | Host port for MySQL. Bound to `127.0.0.1` only. |
| `API_PORT` | No | `5001` | Host port for the API container. |
The opt-in backup overlay adds `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`,
`SHOPDB_SITE` and optional cloud credentials - see
[BACKUP-RESTORE.md](BACKUP-RESTORE.md).
---
## Part 2: Settings table keys
Seeded by `flask seed settings` (idempotent; re-running adds anything missing).
Values are stored as strings and typed by `valuetype`. Secrets in this table
(anything whose key contains `password`, `token`, or `secret`) are masked when
read back through the API.
### site
| Key | Default | Notes |
|-----|---------|-------|
| `setup_complete` | `false` | Set true once the first-run wizard finishes; gates the `/setup` route. |
| `employee_directory_mode` | `selfhosted` | `external` reads a separate HR database (see `EMPLOYEE_DB_*`). No plugin on this line keeps a self-hosted directory, so `selfhosted` resolves no names - set `external`, or accept bare SSOs. |
| `audit_retentiondays` | `0` | Delete audit entries older than this many days. 0 keeps everything forever. Saving a period deletes nothing by itself: purge from this page or run `flask audit purge --apply`. |
| `site_base_url` | (empty) | Public base URL for QR codes and absolute links. Scheme + host, optionally including the subpath a site is served under (`https://host/shopdb`). Blank = use the browsing origin. |
| `facility_name` | (empty) | Facility name in the dashboard header. Blank = frontend falls back to `ShopDB`. |
| `pc_access_domain` | `device.geaerospace.net` | Domain appended to a PC hostname for remote-access links. Blank = hostname as-is. |
| `employeeid_pattern` | `^\d{9}$` | Regex a search term must match to be treated as an employee id. Invalid regex falls back to the default and never 500s. |
| `printer_hostname_template` | `Printer-{ip}.printer.geaerospace.net` | Printer hostname template. `{ip}` is the dash-separated IP address. |
| `contact_email_domain` | `geaerospace.com` | Email domain appended to a support contact's SSO to build email (`sso@domain`) and Teams-chat links. Blank hides the contact action buttons. |
| `dualpath_single_machine` | `true` | Treat a Dualpath pair (a dual-bay machine with one controller) as a single machine in the machines list, dashboard/report counts, and the floor map (the secondary bay is hidden). The data model always keeps both bay records; detail pages stay per-bay with a sibling banner. `false` lists and counts both bays separately. |
| `site_timezone` | `America/New_York` | IANA timezone for the site. Notification times are stored UTC and converted for display by `frontend/src/utils/datetime.js`, so they are entered and shown in THIS zone, not the viewer's browser zone; daily-reset notification expiry (`expirymode=dailytime`) is computed here too. Public-readable so kiosks can resolve it. |
### branding
Blank values fall back to the shipped GE default asset, so an un-reconfigured
install still renders. Upload replacements at Settings > Branding, which saves
them under `instance/branding/`.
| Key | Default | Notes |
|-----|---------|-------|
| `site_logo` | `/ge-aerospace-logo.svg` | Header and login-page logo. |
| `qr_logo` | `/ge-monogram.svg` | Logo composited in printer QR labels. Blank = no overlay. |
| `badge_logo` | `/ge-aerospace-logo.svg` | Logo on the machine badge print page. |
| `site_favicon` | (empty) | Browser tab favicon. Blank = shipped `/favicon.svg`. |
| `brand_primary_color` | (empty) | Primary brand color as a CSS color value (maps to `--primary`). Blank = built-in theme color. |
| `brand_primary_dark_color` | (empty) | Primary hover/active color (maps to `--primary-dark`). Blank = auto-derived by darkening the primary color ~15%. |
| `brand_accent_color` | (empty) | Accent color for secondary buttons and badges (maps to `--secondary`). Blank = built-in theme color. |
| `brand_sidebar_color` | (empty) | Sidebar background color (maps to `--sidebar-bg`). Blank = built-in theme color. |
### printing
| Key | Default | Notes |
|-----|---------|-------|
| `qr_target_printer` | (empty) | Custom URL template for printer QR labels. Blank = link to the printer page on this instance. Placeholders: `{printerid}`, `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{ip}`, `{hostname}`. |
| `qr_target_machine` | (empty) | Custom URL template for machine labels. Blank = link to the machine page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
| `qr_target_computer` | (empty) | Custom URL template for computer labels. Blank = link to the computer page. Same placeholders as machine. |
| `qr_target_network_device` | (empty) | Custom URL template for network-device labels. Blank = link to the device page. Same placeholders as machine. |
| `label_default_style` | `card` | Default asset-label layout when a label first opens: `card` (badge with image and identity) or `plain` (just the code and a caption). |
| `label_default_codetype` | `qr` | Default asset-label code type when a label first opens: `qr` or `barcode` (CODE128). |
| `label_default_encodes_machine` | `assetnumber` | What a machine label encodes by default. |
| `label_default_encodes_computer` | `assetpage` | What a computer label encodes by default. |
| `label_default_encodes_printer` | `assetpage` | What a printer label encodes by default. |
| `label_default_encodes_network_device` | `assetpage` | What a network-device label encodes by default. |
Values for those four `label_default_encodes_*` keys: `assetpage`,
`assetnumber`, `serialnumber`, or `custom` (the matching `qr_target_<type>`
template). All of it is overridable on the label page itself.
There is no `measuring_tool` label type on this branch. Metrology instruments
are machines (ADR-018), so use the machine label keys, and the measuring-tools
plugin is not bundled here - `shopdb/core/api/settings.py` declares four
`qr_target_*` and four `label_default_encodes_*` keys to match.
| Route | What it does |
|---|---|
| `/print/asset-label/<assettype>/<id>` | One label. Public, like the other `/print/*` pages. `<assettype>` is `machine`, `computer`, `printer` or `network_device`; `<id>` is the asset's plugin id. |
| `/print/asset-label-batch/<assettype>` | A multi-selection of one type onto ULINE sheets: 6-up 3 in x 3 in, or a dense 72-up mini-label, with a start-cell offset for reusing partial sheets. Reached from the "Print Labels" button on each asset list page. |
### map
**SUPERSEDED as of 0.11.0. Editing these four keys changes nothing on the map.**
A blueprint and its pixel dimensions are properties of a LEVEL now (ADR-017).
They are retained only so a downgrade finds the blueprint it had; the levels
migration copied their values onto the first level. Manage plans in
**Settings > Buildings and levels** - see [FLOOR-MAP.md](FLOOR-MAP.md).
| Key | Default | Superseded by |
|-----|---------|-------|
| `map_blueprint_light` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintlight` |
| `map_blueprint_dark` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintdark` |
| `map_width` | `3300` | `maplevels.mapwidth` |
| `map_height` | `2550` | `maplevels.mapheight` |
### integrations
| Key | Default | Notes |
|-----|---------|-------|
| `servicenow_enabled` | `true` | Enable ServiceNow ticket recognition and links. Disabled = tickets render as plain text. |
| `servicenow_search_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. |
| `servicenow_ticket_prefixes` | `GEINC,GECHG,GERIT,GESCT` | Comma-separated prefixes recognized as ServiceNow tickets. |
| `servicenow_incident_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct incident URL if your instance has one. |
| `servicenow_change_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct change URL if your instance has one. |
| `zabbix_enabled` | `false` | Enable Zabbix for printer supply monitoring. |
| `zabbix_url` | (empty) | Zabbix API URL. |
| `zabbix_token` | (empty) | Zabbix API token (masked). |
| `warranty_dell_enabled` | `false` | Enable Dell warranty (service-tag) lookups. |
| `warranty_dell_clientid` | (empty) | Dell TechDirect API client id. |
| `warranty_dell_clientsecret` | (empty) | Dell TechDirect API client secret (masked). |
| `warranty_dell_tokenurl` | (empty) | Dell OAuth token URL. Blank = Dell default. |
| `warranty_dell_apiurl` | (empty) | Dell warranty API URL. Blank = Dell default. |
### email
| Key | Default | Notes |
|-----|---------|-------|
| `smtp_enabled` | `false` | Enable email notifications and alerts. |
| `smtp_host` | (empty) | SMTP server hostname. |
| `smtp_port` | `587` | SMTP port (587 TLS, 465 SSL, 25 plain). |
| `smtp_username` | (empty) | SMTP auth username. |
| `smtp_password` | (empty) | SMTP auth password (masked). |
| `smtp_use_tls` | `true` | Use TLS for the SMTP connection. |
| `smtp_from_address` | (empty) | From address for outgoing email. |
| `smtp_from_name` | `ShopDB` | From name for outgoing email. |
| `alert_recipients` | (empty) | Default alert/report recipients (comma-separated). |
The mailer (`shopdb/utils/mailer.py`) reads these settings-first, falling back to
matching `SMTP_*` environment variables only when at least one of them is set.
When `smtp_enabled` is false or `smtp_host` is blank, every send is a graceful
no-op that logs a warning, so an unconfigured site never crashes. The password
is never logged.
Three flows send mail, and **there is no scheduler in this app** - sending is
always on-demand:
- **Welcome email**, on `POST /api/users`. Best-effort: the account is created
even if the send fails (the response carries a `warning`). The new account is
flagged `mustchangepassword`, so the frontend forces `/change-password` before
the app; changing it clears the flag and resets lockout counters. Opt out with
`sendwelcome: false` in the create body.
- **Test email**, `POST /api/settings/test-email` (`settings.edit`), behind the
"Send Test Email" button. A real SMTP error is surfaced with the password
scrubbed out.
- **Alerts and report delivery**, `POST /api/reports/email` (`reports.export`),
taking `{subject, columns, rows, intro?, to?}` and mailing the rows as an HTML
table. Recipients default to `alert_recipients`. To automate a recurring send,
point an external cron job at this endpoint with a PAT scoped to
`reports.export`.
### audit
Retention is `audit_retentiondays`, in the [site](#site) section. There is only
one such key; anything writing `audit_retention_days` is writing to nothing.
### auth
**SAML is not implemented yet.** These keys exist so a site can be configured
ahead of the feature, and the settings page says so on screen. Nothing reads
them: there is no SAML login route and no assertion is accepted, so sign-in is
local accounts only whatever is set here. In particular `saml_allow_local_login`
does NOT disable password login - the toggle is locked on until there is another
way in, because a control that appears to close the local door while it stays
open is worse than no control.
| Key | Default | Notes |
|-----|---------|-------|
| `saml_enabled` | `false` | Reserved. Enable SAML SSO once it ships. |
| `saml_idp_metadata_url` | (empty) | Reserved. SAML IdP metadata URL. |
| `saml_entity_id` | (empty) | Reserved. SAML SP entity id. |
| `saml_acs_url` | (empty) | Reserved. SAML Assertion Consumer Service URL. |
| `saml_allow_local_login` | `true` | Reserved, and locked on. Local login cannot be turned off while it is the only way in. |
| `saml_auto_create_users` | `true` | Reserved. Auto-create users on first SAML login. |
| `saml_admin_group` | (empty) | Reserved. SAML group name that grants the admin role. |
**Personal API tokens (PATs) have no configuration here**, and
`JWT_ACCESS_TOKEN_EXPIRES` does not apply to them. They are created and revoked
from Settings > API Tokens; the token model is documented once, in
[IMPORT-API.md](IMPORT-API.md#1-authentication).
### tools
| Key | Default | Notes |
|-----|---------|-------|
| `securesend_allow_guests` | `false` | Let people with no account create Secure Send links. Public (readable before login) so the create page can show a guest the form instead of a sign-in wall. Off by default: a guest secret has no creator, so the audit trail can say when and from where but never who. Receiving a secret never needs an account either way. |
[SECURE-SEND.md](SECURE-SEND.md) is authoritative for the rest of it - guest
rate limiting, why the read fails closed, and why the size limit is not a tuning
knob.
### identifiers (dynamic)
One boolean key per asset identifier per asset type, keyed
`identifier_<name>_<assettype>_enabled` (default `true`). Admins choose which
optional identifiers show on which asset types. See ADR-001. The exact set is
generated from `IDENTIFIER_LABELS` x `IDENTIFIER_ASSETTYPES` in
`shopdb/core/api/settings.py`.
### search (dynamic)
One boolean key per search domain, keyed `search_<type>_enabled` (default
`true`). Toggles whether a domain appears in global search results. The set is
generated from `SEARCH_DOMAINS` in `shopdb/core/api/settings.py`.
Search terms are matched word-wise: a multi-word query returns rows containing
EVERY word, each word anywhere in the searched fields, in any order ("CSF Roles"
matches a row with "CSF" and "Roles" in different columns). Quoting does not
force a contiguous phrase.
## Custom fields
Site-defined extra attributes per asset type (Settings > Custom Fields, table
`customfields`). Each field has a `searchable` flag (default off). When on, the
field's stored values are matched by global search and a hit routes to the
owning asset's detail page. The asset's `search_<type>_enabled` domain toggle
still applies, so a custom-field hit on a computer only shows when the computer
search domain is enabled. Inactive or non-searchable fields are never matched.
---
## See also
- [DEPLOY.md](DEPLOY.md) - per-site deployment runbook
- [UPGRADE.md](UPGRADE.md) - upgrading an existing site
- [BACKUP-RESTORE.md](BACKUP-RESTORE.md) - backup and restore
- `GET /api/docs` - what endpoints exist and their auth, generated from the code

149
docs/CONTRACT-STABILITY.md Normal file
View File

@@ -0,0 +1,149 @@
<!-- docmeta
purpose: explanation
audience: adopter, integrator
status: current
summary: How stable each part of the plugin contract is today, and how tightly a sister site should pin.
loadwith: none
-->
# Path to contract 1.0
This page is an honest read of how stable the plugin contract is today, for a
sister site deciding how much to build on it. It is derived from the ADRs and
the live code, not aspiration. It does NOT define the contract - what each hook
takes and returns is in [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md), and the versioning
rules are in [ADR-002](adr/ADR-002-plugin-versioning.md). What it adds is a
judgement about how much weight each part will bear.
## Current version
The plugin contract version is declared in `shopdb/__init__.py` as
`__contract_version__`, and the current value is in
[PROJECT-MAP.md](PROJECT-MAP.md), which is generated. It is pre-1.0, which under
semver means any 0.x minor bump is allowed to break the contract, and this
project has used that latitude - see 0.19.0 below.
The product release version (`__version__`) is a separate series with its own
bump rules; see [ADR-007](adr/ADR-007-product-versioning-and-releases.md).
Do not pin against it for compatibility - pin against `__contract_version__`.
### 0.x history
Recorded in the comment block in `shopdb/__init__.py`:
| Version | Change | Kind |
|---------|--------|------|
| 0.3.0 | `shopdb.api` expanded to the full plugin import surface (db, cache, model bases, core models, response + pagination helpers, `employee_connection`) so plugins stop importing internal core paths | additive (minor) |
| 0.4.0 | Removed the never-implemented `get_searchable_fields` hook (search is a core concern over the asset model) and wired `get_dashboard_widgets` to a real consumer (`/api/dashboard/widgets`) | pre-1.0 contract reduction |
| 0.6.0 | Added the `get_reports` hook, consumed by `GET /api/reports` to merge plugin report cards into the Reports hub | additive optional hook (minor) |
| 0.7.0 | Added the four ADR-010 frontend-contribution hooks (`get_settings_cards`, `get_asset_panels`, `get_map_overlays`, `get_asset_presentation`), consumed by the `GET /api/pluginui/*` endpoints | additive optional hooks (minor) |
| 0.9.0 | Exposed the dualpath pair-resolution helpers on `shopdb.api` for the machines plugin | additive surface (minor) |
| 0.10.0 | Added the `get_permissions` hook so plugins declare their own RBAC permissions; the catalog is resolved dynamically from core + enabled plugins | additive optional hook (minor) |
| 0.11.0 | Added `service_token_authorized(scope)` to `shopdb.api` so a plugin's unattended endpoints can authorize a scoped managed service token without importing core token internals | additive surface (minor) |
| 0.12.0 | Added the mailer helpers (`send_email`, `send_alert`) to `shopdb.api` | additive surface (minor) |
| 0.13.0 | Added the `User` model to the `shopdb.api` surface | additive surface (minor) |
| 0.14.0 | Added `send_webhook` to the `shopdb.api` surface | additive surface (minor) |
| 0.15.0 | Added `authorized_service_token` / the `SupportTeam` model to the `shopdb.api` surface | additive surface (minor) |
| 0.16.0 | Added the `get_settings_defaults` hook so a plugin declares the Setting rows it owns; the framework seeds them at install, at enable, and on `flask plugin upgrade-all`, files a first-time write under the declared category, and honours `public: True` for pages that render before login | additive optional hook (minor) |
| 0.17.0 | Added `DashboardDefault` to the `shopdb.api` surface, so a plugin can resolve a display without reaching into core | additive surface (minor) |
| 0.18.0 | Added `DISPLAY_ROLES`, `DISPLAY_ROLE_PATHS` and `normalize_display_role`, and finally exported `DashboardDefault`, which 0.17.0 imported but left out of `__all__`. The role vocabulary became the kiosk's own - `Dashboard`, `Lobby`, `3DPrintRoom` - so a plugin holding its own copy of that map reads core's instead of drifting from it | additive surface (minor) |
| 0.19.0 | **BREAKING.** `get_dashboard_widgets` returns DATA AND SHAPE, not a component name. The old shape (`name` + `component` + `size`) named a Vue component per widget, which cannot survive a lean build - a plugin's component may never be staged into the frontend bundle (ADR-013) - and five plugins were declaring widgets that pointed at components nobody had written. A card now declares `id` / `title` / `endpoint` / `render` / `severity` / `permission` / `empty` / `position`; see PLUGIN-HOOKS.md. Also added `send_upload` so a plugin serving user-supplied bytes gets the headers that keep an SVG from running as script | **contract change (minor, pre-1.0)** |
| 0.20.0 | The `Asset` model gained `levelid` (which floor plan its `mapx`/`mapy` are pixels of) and `mapverifiedat`; `Location` gained `levelid` too, and `resolve_asset_position` now returns the `levelid` belonging to whichever source supplied the coordinates. A plugin that writes a map position MUST write the level with it - a position without one renders as "level unknown" rather than being drawn on the default drawing, because a marker on the wrong floor plan looks correct (ADR-017) | additive surface (minor) |
| 0.21.0 | Added the `get_guest_grantable_permissions` hook: a plugin names the WRITE permissions a site may grant to the seeded `guest` role, which is who anyone without a token now holds. Reads (`.view`/`.read`) are grantable without a plugin saying anything; anything else is refused. Also exposed `require_any_permission` on `shopdb.api` for a read two different rights should reach | additive optional hook + surface (minor) |
| 0.22.0 | Added the `register_import_hook` surface: a plugin owns a VIRTUAL COLUMN on an importable CSV table - a header the bulk importer accepts but never writes - plus a post-write row hook that turns the value into a plugin concept. The computers plugin uses it for `machineassetnumber` on `computers.csv`, which builds the PC->machine controls link, so a site writes the machine number in the PC's own row instead of hand-building a separate relationships sheet. Core never learns what the column means. `RowWarning` is exported with it so a hook can report a soft failure without aborting the import | additive surface (minor) |
| 0.23.0 | Added `mint_service_token`, `list_service_tokens` and `revoke_service_tokens` so a plugin can provision, list and revoke a single-scope managed service token without importing core token internals. The endpoints plugin uses them for a UI-managed, regenerable agent key scoped to `collector.ingest.endpoints` alone - narrow on purpose, so rotating or revoking it cannot disturb the computers collector, and revocation is an instant kill switch on agent ingest | additive surface (minor) |
| 0.24.0 | Added `collector_authorized(pluginname)` so a plugin can gate an agent-facing route that lives OUTSIDE the generic collector blueprint on the same per-plugin key and kill switch the ingest route uses. The endpoints command channel needs it: the agent long-polls `/api/endpoints/commands/next` and posts results back, and those are agent callbacks rather than JWT routes. Without it a plugin would reach into core collector internals to authorize them | additive surface (minor) |
| 0.25.0 | Added `caller_holds(permission)`, which answers whether the current caller holds a permission WITHOUT refusing the request - the complement to `require_permission` for a route that serves everyone but should only offer what the caller can act on. The endpoints services panel is the case: it emitted Start/Stop row actions to every reader, because the emission never consulted the caller. Clicking them was always refused server-side, so nothing was exploitable, but offering a read-only viewer controls over a shop-floor PC is the wrong signal, and it becomes a real problem once a site grants that panel to guests. Guests are included in the answer, since a signed-out visitor holds whatever the guest role was granted | additive surface (minor) |
| 0.26.0 | Dashboard cards may declare `shortname`. The dashboard merged its per-source cards into one worst-first queue, and a card's `title` is written as a heading ("Endpoints with crashes or shutdowns") which as a row badge ran wider than the hostname it qualified. Optional, defaults to the title, so a plugin that does not declare one is unaffected | additive field (minor) |
| 0.27.0 | Added `Script` so a plugin can read the ShopDB-hosted script library that backs endpoint `run_script` delivery without importing core model internals. The endpoints plugin resolves a delivered script by name and serves its body by digest to the agent | additive surface (minor) |
| 0.29.0 | Added an optional `group` to `get_asset_panels`, naming which detail-page tab a panel belongs to: `details` (the default) / `monitors` / `software` / `activity`. The detail pages are tabbed, and a page that placed panels by knowing their ids would defeat the hook. Additive: an absent or unrecognised group resolves to `details` server-side, so every existing plugin keeps working with no change | additive surface (minor) |
| 0.28.0 | Added `client_ip` and `strip_port` so a plugin does not hand-roll caller-IP extraction. Two plugins had, and both read the first `X-Forwarded-For` hop, which the caller supplies - so a rate limit keyed on it was defeated by rotating the header. The helper resolves from `remote_addr`, which ProxyFix has already filled from the header when `TRUSTED_PROXY_COUNT` says a proxy is really in front | additive surface (minor) |
| 0.31.0 | The asset-panel vocabulary a plugin may declare grew four optional things: two more tab groups, `serviceprocess` and `reports`, beside the 0.29.0 set; a `format: 'datetime'` on a column or field, so a panel stops shipping pre-formatted date strings the browser cannot re-localise; a `map.body` key, so a keyvalue field can name where its value lives in a nested payload; and a `tabs` section that carries `columns`/`rows`/`empty` and renders as a table, so one card can hold two tables that share no key (volume capacity and physical drive health are the case). All optional and all resolved server-side when absent, so a plugin declaring none of them is unaffected | additive surface (minor) |
The source comment block documents 0.3.0, 0.4.0, 0.6.0, 0.7.0, 0.9.0, 0.10.0,
0.11.0, 0.16.0, 0.18.0 and 0.19.0; 0.12.0 through 0.15.0 and 0.17.0 are recorded
in this table only.
**0.19.0 is the one to read before pinning.** It is the only entry in this
series that took something away, and it shipped before it was recorded here -
which is precisely the reason to pin tight rather than to trust that a minor
bump is safe. Earlier points
(0.1.x / 0.2.x) predate that recorded rationale; `PluginMeta`'s fallback
`core_version` default of `>=0.2.0,<1.0.0` is the only remaining trace of the
0.2 baseline.
## Settled surface
These are unlikely to break before 1.0. A plugin can depend on them with
reasonable confidence; a breaking change to any would be a major bump and would
land with a new or amended ADR. What each one does is in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md); this is only the stability judgement.
- The required hooks and the manifest schema: `meta` / `PluginMeta`,
`get_blueprint`, `get_models`.
- Plugin plumbing: `init_app`, `get_cli_commands`, `get_services`, the settings
helpers (`get_setting` / `set_setting`), and the four lifecycle hooks.
- The UI-contribution hooks: `get_navigation_items`, `get_dashboard_widgets`,
`get_reports`, `get_permissions`, and the ADR-010 set (`get_settings_cards`,
`get_asset_panels`, `get_map_overlays`, `get_asset_presentation`).
- The collector pair, `get_collector_schema` + `apply_collector_payload`, per
[ADR-006](adr/ADR-006-collector-contract.md).
- The setup-wizard declarations: `get_provisioning_note`, `get_config_schema`.
- The `shopdb.api` import surface itself: adding a name is minor, removing one
is major.
The asset model itself - `Asset`, `AssetType`, `AssetStatus`,
`AssetRelationship`, and the shared reference models - is the platform contract
locked in [ADR-001](adr/ADR-001-asset-as-platform-contract.md).
## Expected churn before 1.0
Known-unstable areas. Building on these means expecting rework.
| Area | Status | Reference |
|------|--------|-----------|
| Frontend renderers (residual) | The four data-only hooks and their `/api/pluginui/*` consumers are settled (0.7.0). The generic core renderers are landing incrementally: the settings-cards rail/landing renderer ships with 0.7.0; the asset-panel, map-overlay, and search-presentation renderers are wired opt-in per the ADR adoption plan. Real component-backed panels (bespoke charts, custom overlays) remain deferred to Option C (build-time glob discovery). | [ADR-010](adr/ADR-010-frontend-plugin-hooks.md) (ACCEPTED) |
| Per-plugin migrations | Brand new. The per-plugin Alembic engine exists and every bundled plugin now carries a chain, but the pattern has one release of production mileage, not years. | [ADR-008](adr/ADR-008-plugin-migration-ownership.md) (2026-07-10) |
| Pip distribution | Deferred to v2. External plugins install by clone / submodule / symlink; there is no entry-point discovery and no automatic update path yet. | [ADR-003](adr/ADR-003-plugin-distribution.md) |
## Bump rules
From [ADR-002](adr/ADR-002-plugin-versioning.md), applied to `__contract_version__`:
| Bump | Trigger |
|------|---------|
| major | Breaking change to the `BasePlugin` ABC, the `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`). Removing a name from `shopdb.api` is major. |
| minor | Additive change: a new optional hook, a new field on a contract model with a default, a new name added to `shopdb.api`. |
| patch | Bug fix with no change to the contract surface. |
### Deprecation policy
ADR-002 defines the bump classification above but is **silent** on any
deprecation window or notice period for pre-1.0 removals. In practice a removal
is simply a major (or, pre-1.0, a breaking minor) bump: the hook or name is
gone, and the loader fails loud in dev or excludes the mismatched plugin in prod
(see the load-time table in [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md)).
Because the ADR says nothing about a grace period, this document proposes none;
the honest guidance for sister sites is the mitigation that already exists - pin
a tight `core_version` range and re-test before widening it.
## Criteria for declaring 1.0
Provisional. This is the maintainer's working list, not a committed checklist,
and it will move. ADR-002's own open question ("When does the framework declare
1.0.0?") ties 1.0 to the `Machine` retirement from ADR-001 and the framework
being "ready for sister sites"; the items below expand that intent.
- Frontend hook contract defined via its own ADR (asset-detail panels, map
markers, search results), closing the biggest churn item above.
- `Machine` / legacy-model retirement complete per ADR-001, so the asset model
is the only contract.
- At least two external plugins running in production at a second site - the
bar ADR-003 already sets before pip distribution and sister-site readiness
are considered justified.
- The per-plugin migration pattern (ADR-008) proven across real upgrades, not
just fresh installs.
- No contract bump needed for N consecutive product releases (N to be fixed
when the list is firmed up), showing the surface has actually settled.
Until then: pre-1.0, pin tight, re-test on every framework bump.

139
docs/CSV-IMPORT.md Normal file
View File

@@ -0,0 +1,139 @@
<!-- docmeta
purpose: howto
audience: adopter, operator
status: current
summary: Load a new site's starting data from generated CSV templates, with no developer and no source database.
loadwith:
-->
# Loading a site's data from spreadsheets
**Four import pages.** Spreadsheet and no developer: this page. The HTTP wire
contract: [IMPORT-API](IMPORT-API.md). How to shape a per-site loader around
it: [IMPORT-ADOPTION](IMPORT-ADOPTION.md). Source is the classic-ASP shopdb:
[IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) on top of the contract.
---
## The short version
```bash
cd C:\shopdb-flask # or your install directory
venv\Scripts\flask csv templates --out csv-templates
```
Fill in the templates. Then:
```bash
venv\Scripts\flask csv import --dir csv-templates
```
That **checks only** and changes nothing. It tells you what it would create and
what is wrong. When you are happy:
```bash
venv\Scripts\flask csv import --dir csv-templates --commit
```
---
## Write names, not numbers
This is the part that makes the difference. Every column that points at another
table accepts the **name** of the thing:
```
assetnumber,name,assettypeid,statusid,locationid
CMM-01,Zeiss Contura,Machine,Active,Gage Lab
MILL-07,Haas VF-2,Machine,Active,Bay 3
```
`assettypeid` gets `Machine`. `locationid` gets `Gage Lab`. You never
have to import a file, read back the numbers it generated, and paste them into
the next one.
The column keeps its database name so it matches the rest of the system, but
the value is whatever you actually know. Numeric ids still work if you have
them - useful when re-importing something this system exported.
Names resolve **across files in the same run**, so `assets.csv` can reference a
location that only exists because `locations.csv` was loaded moments earlier.
## Nothing is half-imported
Every row is checked before anything is written. If one row is wrong, nothing is
written at all - you fix the file and run it again. A mistake on line 400 never
leaves 399 rows loaded.
## Running it twice is safe
Each file is matched on a natural key - `assetnumber` for assets, `locationname`
for locations, and so on. Re-importing an edited file **updates** those rows
rather than creating second copies. Correcting a spreadsheet and re-running is
the expected workflow, not a mistake.
## What the errors look like
```
assets: 0 new, 0 updated, 1 problem(s)
line 4, column 'locationid': nothing in locations is named 'Bay 9'
- add it to locations.csv, or import that file first
```
Line, column, value, and what to do. Not a foreign key constraint violation.
---
## What you can import
Fourteen tables. `flask csv templates` writes one file per table and `--dir`
imports them in dependency order, which is the order the generated folder lists
them in: assetstatuses, assettypes, locationtypes, modeltypes, computertypes,
machinetypes, businessunits, locations, vendors, models, operatingsystems,
assets, computers, machines. The authoritative list, with each table's natural
key, is `IMPORTABLE` in `shopdb/core/services/csvimport.py`.
You only need the files you have; skip any you do not care about. Use `--file`
with `--table` for a single file out of order.
**User accounts are deliberately not importable.** Passwords do not belong in a
spreadsheet, in either direction. Create the first administrator through the
first-run page and the rest in the application.
## The templates are generated, not maintained
`flask csv templates` builds them from the live database schema each time, so
every column offered exists, every required one is marked, and every foreign key
says which file it refers to. Do not hand-maintain a template set: the one that
was tried had invented columns on seven of eleven tables and named a table that
does not exist, while looking entirely plausible. A test fails the build if the
generated set ever drifts from the schema.
## Editing the files
- **UTF-8**, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
- Lines starting with `#` are ignored, so the notes at the top of each template
can stay where they are.
- **The example rows are not comments. Delete them.** Each template carries a
worked example set as LIVE rows, so that a freshly downloaded folder imports
cleanly as a demo. Import it with your own data still in it and you get
`PC-001`, `CNC-014`, `Dell` and `Bay 3` in your database. The template says so
in its own header.
- Booleans are `1` or `0`.
- Dates are `YYYY-MM-DD` (`2026-08-04`). `YYYY-MM-DD HH:MM:SS` also works, as do
`DD/MM/YYYY` and `MM/DD/YYYY`.
- Leave a cell **empty** for "no value". Not `NULL`, not `N/A`.
- Quote anything containing a comma: `"Bay 3, North"`.
## If it will not run
**"the 'assets' table does not exist in this database"** - the schema has not
been created. Run `flask db upgrade` first, and check `DATABASE_URL` points at
the site you meant.
**"unknown column(s): ..."** - a column that does not exist, usually from an
older template. Regenerate with `flask csv templates`; the message lists what
the table does accept.
**"required column(s) missing: ..."** - a column that must be present has been
deleted from the header. Regenerate and copy your data across.

108
docs/DEPENDENCY-SECURITY.md Normal file
View File

@@ -0,0 +1,108 @@
<!-- docmeta
purpose: howto
audience: developer
status: current
summary: Answer "are we exposed to this advisory" from the shipped SBOM, and what can and cannot fail a build.
loadwith:
-->
# Dependency security
Two scripts, one question each.
| question | script | output |
|---|---|---|
| what are we running | `scripts/generate_sbom.py` | `sbom.cdx.json`, CycloneDX 1.6, shipped with every release |
| are we exposed | `scripts/scan_sbom.py` | a report, and an exit code |
The SBOM was already generated and installed onto every server. Nothing read
it, so "are we exposed to this advisory" was still a person going and looking.
The scanner closes that.
## Running it
```bash
venv/bin/python scripts/scan_sbom.py # report, exit 0
venv/bin/python scripts/scan_sbom.py --gate # exit 1 on a shipped finding
venv/bin/python scripts/scan_sbom.py --json out.json # machine-readable too
```
With no `--sbom`, it generates one first, so there is never a stale input.
It needs outbound HTTPS to `api.osv.dev`. **Run it where the repository lives,
not on a customer's server** - a deployed instance is frequently air-gapped and
has nothing to query. That is also why the SBOM ships as a file: it is the
artefact a security team scans on their own terms.
## Why OSV
It takes a purl and returns the advisories that apply. No account, no API key,
no agent installed anywhere, no vulnerability database to keep current. The
SBOM is already the input it wants, so this is a lookup against what shipped
rather than a second scan of the tree that could disagree with it.
## Shipped and build-only are not the same finding
Two thirds of the components in the SBOM are build tooling - bundlers, test
runners, their transitive dependencies. They never reach a server, and the SBOM
already marks them `scope: excluded`.
Both are reported. **Only shipped components can fail a build.**
This is deliberate. At the time of writing, the highest-severity match in the
whole tree is a CRITICAL against the test runner, which is not installed
anywhere a release reaches. A gate that fails on that teaches everybody to pass
`--no-verify` within a month, and then it catches nothing at all.
## Severity is taken, not computed
Where an advisory records a plain word - GHSA gives LOW, MODERATE, HIGH,
CRITICAL - that word is used. Where there is only a CVSS vector, the report says
UNKNOWN rather than parsing the vector into a score it might get wrong.
UNKNOWN is shown and never gates. A guessed score is worse than an honest gap,
because somebody would act on it.
## The allowlist is a record, not a mute button
`scripts/vulnerability-allowlist.json` holds advisories that were reviewed and
accepted. An entry carries the advisory id, the reason, who accepted it, the
date, and what would invalidate the reasoning.
A good reason names the precondition that is absent. The file has held exactly
one entry so far, and how it ended is the point.
A PyJWT advisory rated HIGH - algorithm confusion, forged HS256 tokens - needs a
verifier configured with both symmetric and asymmetric algorithms AND a raw JSON
JWK passed as the key. This application configures neither: there is no
`JWT_ALGORITHM` or `JWT_DECODE_ALGORITHMS` anywhere in the tree, so
flask-jwt-extended runs HS256-only against a string secret. That was written
down, with `invalidatedby` recording what would make it untrue - SSO issuing
tokens verified against a JWK.
Then PyJWT was upgraded, the advisory stopped matching, and the entry was
deleted. The allowlist is empty again, which is the state to aim for.
`invalidatedby` is the field that earns its keep. A reason stops being true when
the code that made it true changes, and nobody re-reads an allowlist looking for
that. Write down what would break it.
Upgrading is almost always cheaper than defending a decision to skip it. Do not
add an entry to make a build pass.
## What the report tells you to do
The report ends with a per-component upgrade rollup: the smallest upgrade that
clears every advisory against that component, and how many it clears.
"Smallest" is doing real work there. An advisory patched on several release
branches lists a fix per branch - nanoid's is fixed in both 3.3.18 and 5.1.16 -
and taking the highest recommends a major upgrade when a patch on the current
branch would do. Where the package is transitive that major may not even be
reachable: nanoid arrives under postcss, pinned `^3.3.11`, a range 5.x cannot
satisfy. So the target is the lowest published fix above the version in use.
Fixes on older branches are ignored rather than offered as a downgrade.
A target on a different major version is still labelled `MAJOR`, but now that
label means something: no fix exists on the current branch, so the upgrade is
genuinely a different job - it can be ESM-only, or drop an API.

180
docs/DEPLOY-AIRGAP.md Normal file
View File

@@ -0,0 +1,180 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build the stack on a connected box, carry one tarball in, and run a Docker site with no internet at all.
loadwith: none
-->
# Air-gapped Docker deploy
> **Topology: Docker, at a site with no internet at all.** The connected
> equivalent is [DEPLOY](DEPLOY.md).
> **Where this sits:** **install** -> operate -> upgrade.
> Later versions: [Upgrading to a new version](#upgrading-to-a-new-version) on
> this page, not [UPGRADE](UPGRADE.md), which assumes a checkout to pull.
The site cannot `pip install`, `npm ci`, or `docker pull`, so nothing is built
or pulled there. You build a fully self-contained image on a **connected** box,
ship it as one tarball, and `docker load` + run it at the site.
`docker-compose.airgap.yml` is the counterpart to `docker-compose.yml` and
differs in four ways that matter:
- **`image:` not `build: .`** - the site runs a loaded image; it never builds.
- **No `./plugins` bind mount** - the image already carries every plugin baked
in. Binding a host `./plugins` (which does not exist at an image-only site)
would mask the baked plugins with an empty dir and load **zero** plugins.
- **A one-shot `migrate` service** runs `db upgrade` + `plugin upgrade-all` +
seed before `api` starts, so `up -d` alone brings up a working site.
- **No `./certs` mount and no TLS flags** - `api` runs the image's own gunicorn
command and serves plain http on 5001. Put TLS in front of it, or in the
container the way `docker-compose.yml` does.
**A ready-made package exists too.** `scripts/build-channel-installer.ps1`
wraps this same tarball with `docker-compose.airgap.yml`, a pre-filled `.env`
and an `Install-ShopDB.ps1`, for an operator who has no repository at all. Read
the script's own header for what it expects on the build box.
---
## 1. Build the bundle (connected box)
On a box with clean access to Docker Hub + PyPI + npm, from the repo root:
```powershell
pwsh scripts/build-offline-bundle.ps1 -Version <version>
```
This builds `shopdb-flask:<version>` (frontend + all Python deps baked in), pulls
`mysql:8.4`, and writes:
- `shopdb-stack-<version>.tar.gz` - both images in one archive
- `shopdb-stack-<version>.tar.gz.sha256` - checksum to verify after transfer
The image is self-contained: **no pip/npm/registry access is needed at the
site.**
### Building behind an inspecting proxy
If the build box is itself behind an inspecting corporate proxy, the in-build
`pip`/`npm` fails with `CERTIFICATE_VERIFY_FAILED`: the build container has its
own certificate store, and the host's `PIP_CERT` / `NODE_EXTRA_CA_CERTS` do NOT
carry into a `docker build`. Easiest fix is to build from a box with clean
internet. Otherwise the corporate root CA must be trusted **inside** the build,
which is a Dockerfile change and is not wired today - see
[DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md) section 0b for the all-roots PEM
bundle.
---
## 2. Transfer + verify
Carry `shopdb-stack-<version>.tar.gz` (+ the `.sha256`), plus
`docker-compose.airgap.yml` and `.env.example`, to the site on approved media.
Verify the archive survived the trip:
```powershell
# PowerShell
(Get-FileHash shopdb-stack-<version>.tar.gz -Algorithm SHA256).Hash.ToLower()
# compare against the .sha256 file
```
```bash
# Linux site
sha256sum -c shopdb-stack-<version>.tar.gz.sha256
```
---
## 3. Load + run (air-gapped site)
```bash
# 1) Load both images into the local Docker.
docker load -i shopdb-stack-<version>.tar.gz
docker image ls | grep -E 'shopdb-flask|mysql' # confirm both present
# 2) Configure the site.
cp .env.example .env
# Edit .env - REQUIRED:
# IMAGE_TAG=<version> # MUST match the loaded image tag
# MYSQL_ROOT_PASSWORD=... # strong, unique
# MYSQL_PASSWORD=... # strong, unique (the app's db user)
# SECRET_KEY=... # 32+ random bytes
# JWT_SECRET_KEY=... # 32+ random bytes, different from SECRET_KEY
# CORS_ORIGINS=https://shopdb.site.example # the site's browser origin(s)
# Optional: API_PORT (default 5001), LOG_LEVEL, ZABBIX_URL/ZABBIX_TOKEN.
# 3) Bring it up. The migrate one-shot runs the schema + seed, then api starts.
docker compose -f docker-compose.airgap.yml up -d
# 4) Watch the one-shot finish (it exits 0 when the schema + seed are done).
docker compose -f docker-compose.airgap.yml logs -f migrate
```
`IMAGE_TAG` in `.env` must equal the loaded tag; otherwise compose looks for an
image that was never loaded and `api`/`migrate` will not start.
`docker-compose.airgap.yml` falls back to `0.7.0` when `IMAGE_TAG` is unset,
which is almost certainly not the version you just built - so set it, rather
than relying on the default.
---
## 4. Create the first admin
Seeding creates permissions, settings, and reference data, but not a login. Make
one admin (choose the password; it is not automatable):
```bash
docker compose -f docker-compose.airgap.yml exec api \
flask seed admin --username <username> --email <email> --password <password>
# Omit --password to have a strong one generated and printed once.
```
---
## 5. Verify
```bash
docker compose -f docker-compose.airgap.yml ps # db + api "running", migrate "exited (0)"
docker compose -f docker-compose.airgap.yml logs api # gunicorn started, no tracebacks
curl -sf http://localhost:${API_PORT:-5001}/api/dashboard/health # or browse the site origin
```
Then log in at the site origin with the admin created in step 4.
---
## Upgrading to a new version
Build a new bundle on the connected box (`-Version <newversion>`), transfer, then at the
site:
```bash
docker load -i shopdb-stack-<newversion>.tar.gz
# set IMAGE_TAG=<newversion> in .env
docker compose -f docker-compose.airgap.yml up -d
```
The `migrate` one-shot re-runs `db upgrade` + `plugin upgrade-all` + seed (all
idempotent) against the existing data before the new `api` starts. Back up first
([BACKUP-RESTORE](BACKUP-RESTORE.md)); the `db_data` volume persists across
upgrades.
---
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| `service api is not running` / `manifest ... not found` | The image was not loaded, or `IMAGE_TAG` in `.env` does not match a loaded image. `docker image ls`, fix `IMAGE_TAG`. Also: never use `docker-compose.yml` here - its `build: .` needs internet. |
| Site loads but **no plugins / empty nav** | You used the wrong compose file. `docker-compose.yml` bind-mounts `./plugins` (absent here) over the baked plugins. Use `docker-compose.airgap.yml`. |
| `api` never starts, `migrate` shows an error | Read `logs migrate`. A DB-connection error means `db` is not healthy yet (`logs db`) or `MYSQL_PASSWORD` in `.env` differs from what the `db` volume was first initialised with. A fresh site with a stale `db_data` volume needs the volume removed (`docker compose ... down -v` - DESTROYS data). |
| `SECRET_KEY must be set` (and similar) on `up` | A required `.env` var is empty. Fill every REQUIRED key in step 3. |
| Build fails on the connected box at `pip`/`npm` | Proxy certificate - see "Building behind an inspecting proxy" above. |
## See also
- [DEPLOY](DEPLOY.md) - the connected stack, the environment variables, and the
reverse-proxy and client-address reference
- [BACKUP-RESTORE](BACKUP-RESTORE.md) - back up before every image swap
- [CONFIG](CONFIG.md) - what each `.env` value means

500
docs/DEPLOY.md Normal file
View File

@@ -0,0 +1,500 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Stand up a new site on Docker at a connected facility, from clone to first admin to TLS.
loadwith: none
-->
# Deploy a site with Docker (connected facility)
> **Topology: Docker, at a site with network access. This is the path a new
> site takes,** and the only one this line ships. No internet at all ->
> [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md).
> **Where this sits:** **install** -> operate -> update -> upgrade.
> Forward: [UPGRADE](UPGRADE.md) for later versions,
> [BACKUP-RESTORE](BACKUP-RESTORE.md) for day-to-day custody of the data.
shopdb-flask is single-tenant per ADR-004. Each adopting facility runs its own
stack: own database, own users, own enabled plugins, own secrets. This page is
the runbook for a fresh site deploy that has network access, and it is also the
reference for the environment variables and for how the application is published
and learns who is calling it - other pages link here rather than restate it.
### Which compose file
| File | For |
|---|---|
| `docker-compose.yml` | this page: a site that can build the image from source |
| `docker-compose.prebuilt.yml` | the same stack built from an already-compiled `frontend/dist/`, for a box where npm cannot reach the registry. See [DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) |
| `docker-compose.airgap.yml` | [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md): runs a loaded image, never builds, and migrates itself with a one-shot `migrate` service |
| `docker-compose.pull.yml` | dev and pilot workstations that pull a published channel image instead of building |
| `docker-compose.channel.yml` | overlay adding `SHOPDB_ENV_LABEL` so a stable and an edge stack can run side by side |
| `docker-compose.backup.yml` | opt-in backup overlay (restic repository, watcher, one-shot). See [BACKUP-RESTORE](BACKUP-RESTORE.md) |
## Prerequisites
- Docker 24+ and Docker Compose v2 (or equivalent container runtime)
- A reverse proxy with TLS termination (nginx, traefik, Caddy, a corporate load balancer), or the certificate pair described in Step 2 - the application itself is not a TLS terminator by design
- A MySQL backup destination (offsite recommended)
- A clone of the repository (the product is distributed as source, not as a public image)
## Step 1: Clone and configure
```bash
git clone <your-remote>/shopdb-flask.git
cd shopdb-flask
cp .env.example .env
```
Edit `.env`:
| Variable | Required | Notes |
|----------|----------|-------|
| `FLASK_ENV` | Yes | `production` for live sites |
| `SECRET_KEY` | Yes | `python -c "import secrets; print(secrets.token_urlsafe(64))"` |
| `JWT_SECRET_KEY` | Yes | Same generation, different value |
| `DATABASE_URL` | venv only | `mysql+pymysql://shopdb:PASSWORD@db:3306/shopdb_flask?charset=utf8mb4`. Under this compose file the services build their own from `MYSQL_PASSWORD` and never read this line - see the note below |
| `CORS_ORIGINS` | Yes | Comma-separated explicit origins. Wildcard rejected. |
| `MYSQL_ROOT_PASSWORD` | Yes | Container only |
| `MYSQL_PASSWORD` | Yes | Container only, must match `DATABASE_URL` password |
| `MYSQL_PORT` | No | Default 3306 |
| `API_PORT` | No | Default 5001 |
| `LOG_LEVEL` | No | Default INFO |
| `ZABBIX_URL`, `ZABBIX_TOKEN` | No | Only if printers plugin uses Zabbix |
| `COLLECTOR_API_KEY` | No | Shared key for `/api/collector/*` ingest. Required only if unattended collectors push data. Endpoint fails closed (denies) when unset. |
| `COLLECTOR_API_KEY_<PLUGIN>` | No | Per-plugin override (e.g. `COLLECTOR_API_KEY_COMPUTERS`), checked before the shared key (ADR-006) |
| `EMPLOYEE_DB_HOST/USER/PASSWORD/NAME` | No | Read-only HR directory for notifications + kiosks. No safe default for the password. |
| `TRUSTED_PROXY_COUNT` | No | venv/IIS only. Under compose it is pinned to 1 for the bundled nginx. See [Step 7](#step-7-tls-and-the-proxy-that-is-already-there) |
[CONFIG](CONFIG.md) and `shopdb/config.py` are authoritative for every other
variable and its default.
**`.env` is not handed to the container wholesale.** None of the compose files
use `env_file:`; compose substitutes `.env` values into the `environment:` block
it declares, and only the variables named there reach the application. As
shipped, the `api` service forwards `SECRET_KEY`, `JWT_SECRET_KEY`,
`CORS_ORIGINS`, `LOG_LEVEL`, `ZABBIX_URL`, `ZABBIX_TOKEN`, `COLLECTOR_API_KEY`,
`COLLECTOR_API_KEY_ENDPOINTS` and `SHOPDB_ENV_LABEL`. Anything else in this
table - a per-plugin collector key for some other plugin, the `EMPLOYEE_DB_*`
set, the `CMMC_USB_DB_*` set, `PLUGIN_TRUSTED_KEYS` - needs a line adding to
that block as well, or it is set in a file the application never reads.
Two entries are set by the compose file itself and ignored from `.env`:
- **`DATABASE_URL`** is composed in-file as
`mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask?charset=utf8mb4`,
on both `api` and `migrate`. `MYSQL_PASSWORD` is therefore the value that
matters; editing `DATABASE_URL` in `.env` changes nothing under compose.
- **`TRUSTED_PROXY_COUNT`** is pinned to 1 because the stack contains its own
nginx. See [Step 7](#step-7-tls-and-the-proxy-that-is-already-there).
`SHOPDB_ENV_LABEL` defaults to `DEV` here on purpose: this file is the
source-build template, and a raw source build is by definition not a release.
Set it to the site's label for a production stack.
## Step 2: Bring up the stack
```bash
docker compose build
docker compose up -d
```
### What comes up
Four services, and knowing which is which saves most of the debugging:
| Service | Does | Listens |
|---|---|---|
| `db` | MySQL 8.4, utf8mb4 server-wide, volume `db_data` | `127.0.0.1:${MYSQL_PORT:-3306}` - loopback only, for `mysqldump` and a local client. `api` reaches it over the compose network regardless |
| `migrate` | schema, plugin bootstrap and seeds, then exits. `restart: "no"` | nothing |
| `api` | gunicorn, gevent workers, serves the compiled SPA | `127.0.0.1:5002` - debugging bypass only |
| `nginx` | **terminates TLS** and proxies to `api` | `${API_PORT:-5001}` -> 443 |
So the address a browser uses is `https://<host>:${API_PORT}`, served by nginx,
and `5002` is a back door for `curl` on the host.
**TLS lives in nginx, not in gunicorn.** `nginx.conf` reads
`/etc/nginx/certs/server.crt` and `server.key`, and `./certs` is mounted there
read-only. The repository ships a self-signed pair and `certs/openssl.cnf`;
replace both files with the site's own certificate and
`docker compose restart nginx`. The gunicorn command carries no `--certfile`,
by design - one TLS terminator, one place to rotate a key.
The reason there is TLS at all rather than plain http: browser APIs that require
a secure context, `navigator.clipboard` among them, refuse to run over plain
http on a non-localhost origin, so a stack reached by hostname needs real TLS
somewhere.
**Why gevent and not threads.** The endpoint agent long-polls
`/api/endpoints/commands/next` and re-polls immediately on a 204, so every agent
holds a connection essentially continuously and fleet size IS the concurrency.
Under threads that was 4 x 8 = 32 slots for the whole site, and a worker's
threads filled with sleeping agents while the CPU sat idle. A held connection
under gevent is a greenlet, so idle is nearly free. Before changing `--workers`,
read the connection arithmetic in the comment beside it: workers x (pool size +
overflow) has to stay under the MySQL `max_connections`, and at the shipped 4
workers it is already about 90 of 151.
### Building where npm cannot reach the registry
The image builds the Vue frontend in a first stage and copies the compiled SPA
in, so `docker compose build` needs no separate Node step. Behind a proxy that
re-signs HTTPS (Zscaler and the like), that first stage is where the build dies:
npm inside the container does not trust the corporate root CA, and the failure
reads `npm ci failed` or `Exit handler never called`.
Two ways out, in order of preference:
1. **Give the build the corporate root CA.** Export it to `corp-root-ca.pem` in
the repository root - [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md#0b-corp-network-ssl-cert---if-you-are-behind-a-gezscaler-proxy)
has the one-line PowerShell that bundles every trusted root - then
`docker compose build --no-cache`.
2. **Build the frontend on the host and skip npm in the image entirely.** That
is `docker-compose.prebuilt.yml` plus `Dockerfile.prebuilt`, and it is also
two to three minutes instead of fifteen. See
[DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md).
(For a bare-metal venv install instead: `cd frontend && npm ci && npm run build`,
which writes `frontend/dist/` for Flask to serve.)
The MySQL container initializes its volume on first run; its healthcheck probes
over TCP rather than the socket, because the socket answers during init while
the server is still about to restart. Watch the bring-up with
`docker compose logs -f migrate api`: if `ProductionConfig.validate()` raises,
the container exits with the offending env-var named in the log. Fix `.env` and
`docker compose up -d` again.
## Step 3: The schema, and what the `migrate` service already did
**On this stack you do not run these by hand.** The `migrate` service runs them
on every `up`, in one place and one order, and `api` will not start until it
exits 0:
```
flask db upgrade && /app/scripts/bootstrap-plugins.sh && flask plugin upgrade-all
&& flask seed permissions && flask seed settings && flask seed reference-data
```
`up -d api` still migrates first: compose honours
`service_completed_successfully` even when a single service is named. A failing
migration exits once and leaves the error as the last line of
`docker compose logs migrate`, rather than crash-looping under a restart policy
and burying it.
The rest of this section says what each command covers, because you will need
that when one of them fails, when you are on a venv install with no `migrate`
service, or when you are pruning a lean site (below). To run one by hand:
`docker compose exec api flask <command>`.
`flask db upgrade` applies the core Alembic chain: the baseline migration plus
every later migration, which together create all core AND bundled-plugin tables
through the chain head. `flask plugin upgrade-all` then stamps each bundled
plugin's own migration chain (the `alembic_version_<plugin>` tables) and applies
any plugin-specific migrations added after the ownership cutover. Both commands
are idempotent, so re-running them is safe. See ADR-008 for why plugin schema
splits into per-plugin chains from the cutover forward.
**Lean sites (ADR-014):** the core chain creates every bundled plugin's tables,
so a site that ships only some plugins still has the others' (empty) tables. To
carry only core + chosen-plugin tables, prune the rest once, at initial
provisioning, after the two commands above:
```bash
docker compose exec api flask plugin prune-schema # dry-run, review
docker compose exec api flask plugin prune-schema --yes --force
```
It drops the tables of every plugin not installed on this site. `--force` is
needed because the core chain seeds a few plugin reference tables (default
access protocols, etc.); at first provisioning those hold only seeded defaults,
before any site data. It refuses to drop a table that holds rows without
`--force`, so it is safe to leave out of routine upgrades - run it only when
provisioning a lean site or after deliberately removing a plugin. Installing a
pruned plugin later recreates its tables automatically.
**Charset:** the schema is utf8mb4 (`utf8mb4_unicode_ci`). The docker-compose `db` service sets `--character-set-server=utf8mb4`, so the auto-created `shopdb_flask` database is utf8mb4. If you point at an external MySQL instead of the bundled container, create the database as utf8mb4 first, or it inherits the server default (often latin1) and the schema silently drifts:
```sql
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
```
`DATABASE_URL` must keep `?charset=utf8mb4` so the connection matches. On MySQL older than 5.7 also enable `innodb_large_prefix=ON` + `innodb_file_format=Barracuda`, or the utf8mb4 indexes exceed the 767-byte prefix limit (error 1071). MySQL 5.7+ and 8.0 need no extra config.
## Step 4: Seed permissions, settings, and reference data
The `migrate` service runs all three on every `up`, and they are idempotent, so
this is a no-op once a site is current and is how a later release's new
permission or new default setting reaches an existing site. They are listed here
because you need to know what they cover, and because a venv install runs them
itself:
```bash
docker compose exec api flask seed permissions
docker compose exec api flask seed settings
docker compose exec api flask seed reference-data
```
- `seed permissions` - creates the RBAC permission rows and the default roles
the app checks with `@require_permission`. Run this before anyone logs in or
permission checks have nothing to match.
- `seed settings` - writes the default Setting rows (branding, ServiceNow
integration, floor-map placeholders, search toggles, site identity). A site
overrides these later in Settings or the setup wizard.
- `seed reference-data` - creates default `ModelType`, `AssetStatus`,
`LocationType`, `CommunicationType`, `OperatingSystem`, `RelationshipType` rows
seeded with the platform contract values (`partof`, `controls`, `connectedto`).
(`Vendor`, `Location`, and `BusinessUnit` are not seeded here; they come from
`seed demo`.)
## Step 5: Pick plugins to enable
The image bundles every plugin in the tree. [PROJECT-MAP](PROJECT-MAP.md) lists
them with their versions, generated from the code; [PLUGINS](PLUGINS.md) is the
page for deciding which ones a site should carry. Only enabled plugins are
loaded.
`scripts/bootstrap-plugins.sh`, inside the `migrate` service, has already
installed a starting set on a site with no `instance/plugins.json`: backups,
computers, endpoints, guides, knowledgebase, machines, network, notifications,
printers, slides, tools, warranty. It installs them **disabled**, and it skips
itself entirely once the registry file exists, so it never fights a site's own
choices.
That is every bundled plugin, so on this line the starting set and the shelf
are the same list - a site turns one on rather than installing it:
```bash
docker compose exec api flask plugin list
docker compose exec api flask plugin enable warranty
```
Or put the whole list in a profile and keep that in version control, which is
the thing to do when more than one site is involved:
```bash
docker compose exec api flask plugin apply-profile deploy/site-profile-universal.json
```
`deploy/site-profile.example.json` is the template to copy for a site of your
own. Enabling a plugin needs `docker compose restart api` to take effect.
To install a sister-site or third-party plugin (per ADR-003), drop its directory into `<repo>/plugins/<name>/` (the docker-compose mounts this read-only into the container) and run `flask plugin install <name>`.
## Step 6: Create the first admin (setup wizard)
The primary path is the first-run setup wizard. Once the stack is up and the DB
is seeded, browse to the site (through the reverse proxy configured in Step 7,
or directly at the API port during bring-up) and go to `/setup`. The wizard
creates the first admin account and captures site identity (facility name,
optional logo). It runs only while `setup_complete` is false; after it finishes
the route redirects to the app.
Headless alternative (no browser, e.g. automated provisioning):
```bash
docker compose exec api flask seed admin --username admin --email admin@facility.example.com
# Password is generated and printed once. Store in your password manager.
```
Subsequent users are managed through the UI.
## Step 7: TLS, and the proxy that is already there
**The stack ships its own reverse proxy.** The `nginx` service terminates TLS on
`${API_PORT:-5001}` and proxies to `api` over the compose network, forwarding
`X-Real-IP`, `X-Forwarded-For`, `X-Forwarded-Proto` and `Host`, with 90-second
proxy timeouts so an agent's long poll is not cut mid-hold. For most sites
Step 7 is therefore: drop your certificate into `./certs`, set `API_PORT`, and
stop.
Put a corporate load balancer or an existing nginx in FRONT of that only if the
site already has one. The shape then is proxy -> nginx -> api, which is two
proxies, and `TRUSTED_PROXY_COUNT` has to say so (below). A site that would
rather terminate TLS in its own proxy can instead remove the `nginx` service and
publish `api` directly - a configuration like this one:
```nginx
server {
listen 443 ssl;
server_name shopdb.facility-a.example.com;
ssl_certificate /etc/ssl/certs/shopdb.crt;
ssl_certificate_key /etc/ssl/private/shopdb.key;
location / {
proxy_pass http://localhost:5001;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
(That block proxies to plain http, so it belongs on the same host as the
container or on a trusted network segment.)
### Real client addresses: `TRUSTED_PROXY_COUNT`
Behind a proxy the application sees the proxy's connection, not the browser's.
`TRUSTED_PROXY_COUNT` is the number of proxies **actually** in front of this
instance, and the application then enables ProxyFix, which resolves the client
address, scheme, host and prefix from the `X-Forwarded-*` headers.
**`docker-compose.yml` already sets it to 1 on the `api` service**, because the
bundled nginx is that one proxy. It is not read from `.env` there - the value
belongs to the stack's shape, not to the site. Change it in the compose file,
not in `.env`, and only when you change the shape: 2 if you put a corporate load
balancer in front of nginx, 0 if you delete the nginx service and publish `api`
directly with nothing ahead of it.
- **It is 0 by default, deliberately.** ProxyFix makes the application trust
those headers, and any caller can send them. Enabled with nothing in front,
a caller can claim a different source address or scheme - and the login rate
limit and the account lockout are keyed on the client address.
- **Leave it at 0 behind a proxy** and the application is not wrong about the
client so much as blind to it: every request looks like it came from the
proxy. The whole site then shares one rate-limit bucket, and the audit log
records the proxy for every action.
- Getting the scheme wrong has its own symptom: an absolute redirect built from
the proxy's view sends an https page to an http URL the browser refuses. That
is what made a redirect on the endpoints list unfollowable behind IIS.
- `.env.example` also carries `TRUSTED_PROXY_COUNT`, defaulting to 0. That line
is for a venv or IIS install, where nothing sets it for you; under this
compose file the compose value wins and the `.env` line is inert.
`CORS_ORIGINS` is an exact origin match (scheme + host + port, no path), so it
has to name the address the browser actually uses, not the container's.
## Step 8: Backups
Per-site backups are the site's responsibility: nightly dump to offsite storage,
14-day retention, a restore rehearsed quarterly. The database is not the whole
backup - `instance/` holds uploaded floor plans, branding, `plugins.json` and
tokens that are not in MySQL, and under compose it is the `instance_data` named
volume. [BACKUP-RESTORE](BACKUP-RESTORE.md) is the procedure for both, including
the compose backup overlay.
```bash
docker compose exec -T db mysqldump -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask | gzip > backup-$(date +%F).sql.gz
docker compose run --rm -v "$PWD:/backup" api tar czf /backup/instance-$(date +%F).tar.gz -C /app/instance .
```
## Step 9: Updates
Moving this site to a newer version is [UPGRADE](UPGRADE.md) - back up, pull,
rebuild, migrate, re-seed, restart, plus the version-specific notes. One thing
belongs here rather than there, because it is about how this stack is built:
### The `instance_data` volume, and stacks that predate it
`up -d api` REPLACES the container. Everything the site has written lives in the
`instance_data` volume for exactly this reason: the enabled-plugin list
(`plugins.json`), uploaded floor plans and branding, model and application
images, employee photos, warranty proofs, slides, printed-part files, and the
Dell OAuth token.
**`docker compose down` is safe; `docker compose down -v` is not.** The `-v`
deletes the named volumes, which is both `instance_data` and `db_data`: the
site's files and its database in one command. There is no confirmation prompt.
If your stack predates that volume, those files are in the old container's
writable layer and an update discards them. Move them across ONCE, before the
next rebuild:
```bash
docker compose cp api:/app/instance ./instance-rescued # BEFORE pulling new code
docker compose up -d api # creates the volume
docker compose cp ./instance-rescued/. api:/app/instance
docker compose exec -u root api chown -R shopdb:shopdb /app/instance
docker compose restart api
docker compose exec api flask plugin list # the site's plugins, enabled again
```
Two things decide whether this works.
- **The `chown` is not optional.** `docker compose cp` writes the files with the
copying user's numeric uid, which is only `shopdb` by coincidence if your host
account happens to be uid 1000. Get it wrong and the site reads its restored
files fine and cannot write new ones. Every copy into a container in these
pages ends with this same `chown`, for this reason.
- **Restore `instance/` BEFORE `plugin upgrade-all`.** That command works from
`plugins.json`, so against an empty instance directory it upgrades nothing and
reports success.
The symptom of having missed it is a site that comes back with its plugins
disabled and image URLs that 404: the MySQL rows survived, the files did not.
Re-enabling by hand works, but `flask plugin apply-profile <profile.json>` puts
the same list back in one command and is the thing to keep in version control.
## Importing an existing database
A site coming from an earlier ShopDB, or a test stack loading a copy of
production, restores the dump and then brings the schema forward:
```bash
docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask < dump.sql
docker compose up -d migrate # re-runs upgrade, bootstrap and the seeds
docker compose restart api
```
The order matters and so does the second line. A restored dump carries plugin
TABLES but not `instance/plugins.json`, which is where the enabled-plugin list
lives (ADR-008 and the note in Step 9). Without a bootstrap pass the site comes
up with a full schema and no plugins registered, which reads as "all my data is
gone". `flask plugin apply-profile` puts a known list back in one command.
Restore `instance/` at the same time if you have it - see Step 9.
## Common issues
| Symptom | Cause | Fix |
|---------|-------|-----|
| `ConfigError: SECRET_KEY is required in production` | `.env` missing or blank | Set `SECRET_KEY` in `.env`, re-up |
| `ConfigError: CORS_ORIGINS must be a comma-separated allowlist` | `.env` has `*` | Set explicit origins |
| `PluginVersionError: requires core_version X but framework is Y` | Plugin pinned a too-narrow range | Update `manifest.json` `core_version` or pin framework version |
| 500s after `flask db upgrade` | Migration ran but app cached old schema | `docker compose restart api` |
| Cannot reach API after restart | Reverse proxy not pointing at the container's exposed port | Confirm `API_PORT` and proxy config |
| `npm ci failed` / `Exit handler never called` during build | A proxy re-signs HTTPS and npm in the build stage does not trust the corporate root CA | Stage `corp-root-ca.pem` in the repo root and `docker compose build --no-cache`, or build the frontend on the host ([DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md)) |
| `database is unhealthy`, or `db` never goes green | MySQL still initializing, or the published port is taken on the host | `docker compose logs db`; if the port clashes, set `MYSQL_PORT=3307` in `.env`. The mapping is loopback-only and does not affect `api` |
| `Access denied for user 'shopdb'` | `MYSQL_PASSWORD` changed after the `db_data` volume was initialized | MySQL sets the app user's password only on first init. Change it in MySQL, or drop the volume on a site with nothing in it yet |
| `Could not connect to database` from `api` or `migrate` | Something is pointing at `localhost` instead of the `db` service | Inside the stack the host is `db`. `localhost` in a container is that container |
| `ERR_CERT_AUTHORITY_INVALID` in the browser | The shipped `certs/server.crt` is self-signed | Expected on a first bring-up. Replace the pair in `certs/` with the site's own certificate and `docker compose restart nginx` |
| A plugin is installed but nothing appears in the nav | Installed is not enabled, and enabling needs a restart | `flask plugin enable <name>` then `docker compose restart api` |
| The first build takes 15 minutes | npm install and the Vite build inside the image | Expected once; later builds hit the layer cache. To avoid npm in the image entirely see [DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) |
## Health check
```bash
# Through nginx, on API_PORT. -k because the shipped certificate is self-signed
# until you replace it.
curl -sk -X POST -H "Content-Type: application/json" \
-d '{}' https://localhost:5001/api/auth/login | jq .
# Expect: {"status": "error", "data": {"error": {"code": "VALIDATION_ERROR", ...}}}
curl -sk https://localhost:5001/api/dashboard/health # no authentication needed
```
`api` has its own healthcheck over plain http inside the stack, and the
debugging bypass on the host is `curl -s http://127.0.0.1:5002/api/dashboard/health`.
Comparing the two separates "the app is broken" from "nginx or the certificate
is broken": bypass green and `:5001` red means the problem is TLS or the proxy.
If both are red or return no JSON, `docker compose logs api`.
## References
- [UPGRADE](UPGRADE.md) - moving this site to a newer version
- [DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) - building
the image where npm cannot reach the registry
- [DEV-BUILD-TEST](DEV-BUILD-TEST.md) - pulling a prebuilt stack onto a Windows
box to look at it
- [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md) - the same stack where nothing can be pulled
- [BACKUP-RESTORE](BACKUP-RESTORE.md) - backup and restore procedure
- [CONFIG](CONFIG.md) - every environment variable and every Setting key
- [PLUGIN-QUICKSTART](PLUGIN-QUICKSTART.md) - building a custom plugin for your
site
- [ADR-004](adr/ADR-004-deployment-topology.md) - per-site instances rationale;
[ADR-003](adr/ADR-003-plugin-distribution.md) bundled vs external plugins;
[ADR-006](adr/ADR-006-collector-contract.md) per-plugin collector endpoints
- [shopdb/config.py](../shopdb/config.py) - all the env vars in one place

56
docs/DEV-BUILD-TEST.md Normal file
View File

@@ -0,0 +1,56 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Stand up a prebuilt ShopDB stack on a Windows box with Docker and one PowerShell script, no source build.
loadwith: none
-->
# ShopDB dev build - quick test
Two dev builds you can stand up on a Windows box with Docker Desktop. You run
one PowerShell script: it checks your dependencies (offers to winget-install
anything missing), then PULLS a prebuilt image from GHCR and starts the stack in
Docker. Nothing is compiled on your machine - no Node, no npm, no source build -
so it works on a box that has only Docker installed.
This is for LOOKING at a build, not changing one. To edit the code, you want a
real dev environment: [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md).
## What you need
- Windows with Docker Desktop running.
- Git for Windows (`winget install Git.Git`).
- A GitHub login for this private repo/registry already stored on the box. If
you have cloned this repo before, or signed in to GitHub in Visual Studio or
VS Code, you already have it - the script uses that stored credential
automatically. If Docker cannot pull the image, sign in once with
`docker login ghcr.io` (use a GitHub username and a personal access token that
has `read:packages`), then re-run the launcher.
## Get the launcher
You will be sent `Start-ShopDBDev.ps1` directly. Save it anywhere (for example
your Downloads folder) and open PowerShell in that folder.
## Bleeding edge (main, the trunk)
powershell -ExecutionPolicy Bypass -File .\Start-ShopDBDev.ps1 -Channel edge
Comes up at https://localhost:5011. Red DEV badge in the sidebar. Pulls the
`edge` image.
## Stable (newest release tag)
powershell -ExecutionPolicy Bypass -File .\Start-ShopDBDev.ps1 -Channel stable
Comes up at https://localhost:5001. No DEV badge. Pulls the `stable` image.
## Notes
- Both stacks can run at the same time - they use different ports and separate
Docker volumes, so they do not collide and each keeps its own database.
- Each launch pulls the latest image for the channel, so re-running the launcher
is how you pick up a new build.
- First page load shows a self-signed cert warning (expected for a dev box).
- Stop a stack: `docker compose -p shopdb-stable down` (or `shopdb-edge`).

390
docs/DEVELOPMENT-SETUP.md Normal file
View File

@@ -0,0 +1,390 @@
<!-- docmeta
purpose: howto
audience: developer
status: current
summary: Clone to a running dev site, the three gates you run before pushing, and how to contribute a change back.
loadwith: none
-->
# Development setup: from clone to first change
The goal of this page: a new developer clones the repo and has a working dev
site plus a change they can see in the browser, in one sitting. Reference
material lives elsewhere - naming rules in `CONTRIBUTING.md`, every config
variable in the CONFIG guide, plugin authoring in the PLUGIN docs - this is
just the on-ramp.
**Most developers here are on Windows in VS Code** - commands below are
PowerShell first, with the bash equivalent in a comment where they differ.
Install **Git for Windows** (it ships Git Bash, which VS Code and the git
hooks use to run the shell-based naming check) and **VS Code** with the
extensions this repo recommends (you'll be prompted - section 2c).
Two ways to run it. **Docker** (Docker Desktop on Windows) is the fastest to a
working site. **Manual (venv + Node)** is the daily driver - frontend
hot-reloads, backend restarts on save. Do Docker once to confirm the box is
sane, then use manual for day-to-day work.
If you only want to LOOK at a running site and are not going to change the
code, do not do any of this: [DEV-BUILD-TEST](DEV-BUILD-TEST.md) pulls a
prebuilt image and needs nothing but Docker.
---
## 0. Prerequisites
| Need | Version | Check |
| --- | --- | --- |
| Python | 3.14 (64-bit) - matches CI, the container image and the Windows installer wheelhouse | `python --version` |
| Node.js | 18+ | `node --version` |
| MySQL | 8.4 LTS (or Docker, below) | `mysql --version` |
| Git | any recent | `git --version` |
On Windows, install all of them with winget (accept each license, then
reopen the terminal so PATH updates):
```powershell
winget install Git.Git # includes Git Bash (naming hook needs it)
winget install Python.Python.3.14
winget install OpenJS.NodeJS.LTS # LTS; may install v24, fine for this SPA
winget install Microsoft.VisualStudioCode
winget install Oracle.MySQL # or Docker.DockerDesktop for the DB
```
CI runs Node 20; the LTS package may be newer. This Vite/Vue frontend builds
identically across 20-24, so it does not matter. To pin exactly:
`winget install CoreyButler.NVMforWindows` then `nvm install 20; nvm use 20`.
---
## 0b. Corp network (SSL cert) - if you are behind a GE/Zscaler proxy
A proxy that inspects HTTPS (Zscaler on GE PCs) re-signs every connection
with a corporate root CA. `git`, `npm`, `pip`, and Node each keep their own
trust store and do not trust that CA by default, so downloads fail:
| Tool | Symptom |
| --- | --- |
| npm | `UNABLE_TO_GET_ISSUER_CERT_LOCALLY` |
| git | `SSL certificate problem: unable to get local issuer certificate` |
| pip | `SSLError` / `CERTIFICATE_VERIFY_FAILED` |
Fix once - export the corp root CA, point every tool at it. PowerShell
mangles multi-line pastes, so each step below is **one physical line**: paste
it, press Enter, then the next. Do not paste both at once.
```powershell
# 1) Bundle EVERY trusted root into one PEM (one line). Guessing which single cert is the proxy's is fragile; bundling all always includes it.
$sb = New-Object System.Text.StringBuilder; Get-ChildItem Cert:\LocalMachine\Root | ForEach-Object { [void]$sb.AppendLine("-----BEGIN CERTIFICATE-----"); [void]$sb.AppendLine([Convert]::ToBase64String($_.RawData,'InsertLineBreaks')); [void]$sb.AppendLine("-----END CERTIFICATE-----") }; [IO.File]::WriteAllText("$HOME\corp-root-ca.pem", $sb.ToString())
```
Confirm it has many certs (dozens, not 1):
`(Select-String "BEGIN CERTIFICATE" $HOME\corp-root-ca.pem).Count`
```powershell
# 2) Point every tool at it (one line, persistent). NODE_EXTRA_CA_CERTS also fixes Vite / npm run dev.
git config --global http.sslCAInfo "$HOME\corp-root-ca.pem"; npm config set cafile "$HOME\corp-root-ca.pem"; setx NODE_EXTRA_CA_CERTS "$HOME\corp-root-ca.pem"; setx PIP_CERT "$HOME\corp-root-ca.pem"
```
Reopen the terminal so `setx` takes effect. Quick unblock if you cannot
export right now (skips verification - use briefly, then set back):
`npm config set strict-ssl false`, `git config --global http.sslVerify false`.
---
## 1. Get the code
```powershell
git clone https://github.com/ge-aero/shopdb-flask.git
cd shopdb-flask
```
Never work on `main`. Branch for your change:
```powershell
git checkout -b feat/<short-description>
```
---
## 2a. Fast path - Docker (a working site in one command)
```powershell
copy .env.example .env
# Edit .env: set SECRET_KEY, JWT_SECRET_KEY, and the MYSQL_* passwords.
# Generate a secret: python -c "import secrets;print(secrets.token_urlsafe(64))"
docker compose up -d --build # MySQL + the app (frontend built in-image)
# Schema + platform data (idempotent, safe to re-run):
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
docker compose exec api flask seed permissions
docker compose exec api flask seed settings
docker compose exec api flask seed reference-data
docker compose exec api flask seed admin --username admin --email you@example.com
docker compose exec api flask seed demo # OPTIONAL: sample data (undo: flask seed demo-clear)
```
The app is on the port the compose file maps (see `docker-compose.yml`). Good
for a smoke test; for active development use the manual path so the frontend
hot-reloads.
---
## 2b. Manual path - venv + Node (the daily driver)
### Database
Either point at an existing MySQL 8, or bring one up with just the db service
from compose:
```powershell
docker compose up -d db # MySQL on 127.0.0.1:3306
```
The mapping is loopback-only and `MYSQL_PORT` moves it. If the box already runs
a local MySQL on 3306, set `MYSQL_PORT=3307` in `.env` and put the same 3307 in
`DATABASE_URL` - a mismatch here is the single most common "cannot connect"
in development, because the error names 3306 and looks like the container is
down when it is simply somewhere else.
Create the database + app user (skip if compose already did via env):
```sql
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'shopdb'@'%' IDENTIFIED BY 'devpassword';
GRANT ALL PRIVILEGES ON shopdb_flask.* TO 'shopdb'@'%';
FLUSH PRIVILEGES;
```
### Backend
```powershell
python -m venv venv
venv\Scripts\Activate.ps1 # bash/mac: source venv/bin/activate
# If PowerShell blocks the activate script (execution policy), run once:
# Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
pip install -r requirements-dev.txt
copy .env.example .env # bash/mac: cp .env.example .env
# Edit .env - set SECRET_KEY, JWT_SECRET_KEY, and
# DATABASE_URL=mysql+pymysql://shopdb:devpassword@127.0.0.1:3306/shopdb_flask?charset=utf8mb4
# CORS_ORIGINS=http://localhost:5173
$env:FLASK_APP = "shopdb" # bash/mac: export FLASK_APP=shopdb
flask db upgrade # core schema
flask plugin upgrade-all # per-plugin schema (ADR-008)
flask seed permissions
flask seed settings
flask seed reference-data
flask seed admin --username admin --email you@example.com # password printed once
flask seed demo # OPTIONAL: sample assets across the plugins (undo: flask seed demo-clear)
```
Enable the plugins you want visible. To turn on everything for development:
PowerShell:
```powershell
foreach ($p in "backups","computers","endpoints","guides","knowledgebase",
"machines","network","notifications","printers","slides",
"tools","warranty") {
flask plugin install $p; flask plugin enable $p
}
```
(bash/mac: a `for p in ...; do flask plugin install "$p"; ...; done` loop.)
Run the backend ON PORT 5001 - the frontend dev server proxies `/api` and
`/static` there (a bare `flask run` uses 5000 and nothing will load). Use the
launcher in the repository root rather than `flask run`:
```powershell
python run_dev.py
```
**Use `run_dev.py`, not `flask run`.** `flask run` reloads by spawning a
subprocess, and the subprocess does not inherit what `python-dotenv` loaded from
`.env` - so the reloaded backend comes up with no `DATABASE_URL` and tries
127.0.0.1:3306 with default credentials. `run_dev.py` loads `.env` BEFORE
importing Flask, then runs `flask run --host=0.0.0.0 --port=5001 --no-reload`.
It prints the `DATABASE_URL` and `SHOPDB_BACKUPS_ROOT` it resolved, which is the
fastest check that `.env` was actually read. The cost is that backend edits need
a manual restart.
**Leave this running.** It does not return to a prompt - that is correct, not a
hang. The server holds this terminal until you stop it. Do NOT press Ctrl+C to
move on; that kills the backend. Open the frontend in a separate terminal (next
section) and leave this one alone. Ctrl+C only when you are done for the day.
### Frontend (a second terminal - leave the backend running)
```powershell
cd frontend
npm install
npm run dev # http://localhost:5173
```
Open http://localhost:5173, log in as `admin` with the printed password.
To point the dev proxy at a backend on another machine, set the environment
variable rather than editing `vite.config.js`:
```bash
VITE_DEV_API_TARGET=https://10.0.0.5:5001 npm run dev
```
Certificate verification is off for an `https` target, because those boxes
normally carry a self-signed certificate; `VITE_DEV_SECURE=true` turns it back
on.
> Convenience: instead of two terminals you can run both under a process
> manager (pm2, honcho, foreman). Keep the backend on 5001.
### Working from a copy of a site's database
Restoring a dump instead of seeding is the fastest way to reproduce a real bug,
and it has two traps:
```powershell
docker exec -i <db-container> mysql -u root -p"<MYSQL_ROOT_PASSWORD>" shopdb_flask < dump.sql
flask db upgrade
flask plugin upgrade-all
```
1. **The dump has plugin tables but no plugin registry.** The enabled-plugin
list lives in `instance/plugins.json`, not in MySQL, so a restored database
comes up with every table and an empty nav. Install and enable the plugins
(above), or `flask plugin apply-profile deploy/site-profile-universal.json`.
2. **You will not know the admin password.** Mint a fresh hash and write it
over the row, clearing the lockout counters at the same time:
```powershell
python -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('admin'))"
```
```sql
UPDATE users SET passwordhash = '<paste>', failedlogins = 0, lockeduntil = NULL
WHERE username = 'admin';
```
Do this only against a development copy. Never paste a hash from a document
into a real site - one that has been published is a password everybody has.
If the backups feature is in play, `SHOPDB_BACKUPS_ROOT` has to name a path that
exists on the development box. Unset, it defaults to a POSIX path and Windows
answers `Access is denied: '/backups'`.
---
## 2c. VS Code (turnkey)
The repo ships shared VS Code config in `.vscode/` (personal `settings.json`
stays git-ignored):
- **Recommended extensions** - on first open VS Code offers to install them
(Python + Pylance, Vue Volar, ESLint, Docker). Accept.
- **Run the dev site** - Command Palette > "Tasks: Run Task" >
**Dev site (backend + frontend)** starts both servers in parallel (backend
on 5001, frontend on 5173). Individual tasks exist too. If the backend task
invokes `flask run` rather than `run_dev.py`, expect the `.env` problem
described in 2b and start the backend by hand instead.
- **Debug the backend** - the Run panel's **Flask API (:5001)** config runs
the app under the debugger (breakpoints in routes/services, full
stepping); **Pytest (current file)** debugs the open test file.
- **The CI gate** - task **Check: naming + tests + build** runs the same
three checks CI runs, before you commit.
Prerequisite: the venv and `npm install` from 2b must be done first (the
tasks call `venv/` and `frontend/node_modules`).
---
## 3. The development loop
1. Make a change. Backend: `flask run` auto-reloads. Frontend: Vite hot-reloads.
2. Before committing, run the three gates. Easiest: in VS Code, Command
Palette > "Tasks: Run Task" > **Check: naming + tests + build**. By hand
in PowerShell:
```powershell
venv\Scripts\python -m pytest tests/ -q # backend
cd frontend; npx vitest run; npm run build; cd ..
bash scripts/check-naming-and-style.sh # naming - runs via Git Bash
```
CI runs all three on every push and pull request, so running them locally
only saves the round trip. No hook is installed for you; to have the naming
check run on every `git commit`, enable the shipped one once per clone with
`git config core.hooksPath .githooks`. The naming check is a shell script,
so it needs Git Bash (installed with Git for Windows) on Windows.
3. Commit in small, working steps. Subject: short, present tense, plain
English; body says WHY. Read `CONTRIBUTING.md` before naming anything - the
naming hook will reject snake_case DB columns, banned shorthand, and
non-ASCII.
Seeing a change in the real app (not just tests) is the bar for "done" -
drive the actual flow in the browser.
---
## 4. Your first change (suggested)
Add a field to an existing list page. Or build a plugin end to end, starting at
[PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART.md).
---
<a id="contributing"></a>
## 5. Contributing back
This is the whole contributor workflow, for a core change and for a plugin
alike.
1. **Branch, never work on `main`** (section 1), and commit in working steps -
each commit a checkpoint that passes its own tests.
2. **Run the three gates before every push** (section 3).
3. **Push and open a Pull Request against `main`.**
```powershell
git push -u origin feat/<short-description>
```
In the description: what changed, which plugin hooks it implements, any
contract additions (these need a `__contract_version__` bump plus a
`docs/PLUGIN-HOOKS.md` update in the SAME pull request - a docs-drift test
fails the build otherwise), and any deliberate security posture. An endpoint
that is open on purpose needs a decision record, not a comment.
4. **What a reviewer looks for:**
- contract purity - core imported only via `shopdb.api` and
`shopdb.plugins.base` (there is a guard test)
- the naming convention, per `CONTRIBUTING.md`
- a per-plugin migration chain, its `PLUGIN_TABLE_OWNERS` entry, and its
expected-head declaration in `tests/test_plugin_migrations.py`
- permissions both DECLARED and ENFORCED
- tests for the invariants, not just the happy path
- `default_enabled` correct for the plugin's nature
5. **After approval**, the maintainer lands the change on the mainline and the
next published release commit includes it; your pull request is then closed
as merged. Day-to-day history lives on the development mainline and the
published repository carries the release line, so do not be surprised when
your commits arrive squashed or folded into a release commit.
---
## Common setup problems
| Symptom | Cause / fix |
| --- | --- |
| Frontend loads but every API call fails / CORS error | backend not on 5001 (`flask run --port 5001`), or `CORS_ORIGINS` missing `http://localhost:5173`. |
| App refuses to boot in production config | a required `.env` var (`SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, `CORS_ORIGINS`) missing or a dev default. |
| `flask db upgrade` / `flask plugin upgrade-all` error 1071 (key too long) | MySQL 5.6 on the default Antelope file format, where an index prefix caps at 767 bytes - one utf8mb4 `VARCHAR(255)` is 1020. Start mysqld with `--innodb-file-format=Barracuda --innodb-large-prefix=ON` (both are also settable live with `SET GLOBAL`), keep `?charset=utf8mb4` on `DATABASE_URL`, and note that tables CREATED before those flags were on are silently `ROW_FORMAT=Compact` rather than `DYNAMIC` - 5.6 downgrades the row format with a warning instead of an error, so they keep the 767-byte cap until rebuilt with `ALTER TABLE <t> ROW_FORMAT=DYNAMIC`. MySQL 5.7+ and 8.0 need none of this. |
| Nav missing Machines/PCs/... | plugins not installed/enabled (step 2b), or the backend not restarted after enabling. |
| "No time zone found with key America/New_York" | `tzdata` not installed - `pip install -r requirements.txt` includes it. |
| npm/git/pip SSL error (`UNABLE_TO_GET_ISSUER_CERT_LOCALLY`, `unable to get local issuer certificate`) | corp proxy (Zscaler) intercepts HTTPS - point each tool at the corp root CA. See section 0b. |
| Naming hook rejects a commit | you used snake_case on a DB-mirrored field or a banned acronym - see `CONTRIBUTING.md`. |
| Plugin toggle throws an internal error | app cannot write `instance/` (the plugin registry lives there) - fix directory permissions. |
| `Can't connect to MySQL on 127.0.0.1:3306` while the container is clearly up | the backend never read `.env` (started with `flask run` instead of `run_dev.py`), or `MYSQL_PORT` moved the container and `DATABASE_URL` still says 3306. |
| `Access is denied: '/backups'` on Windows | `SHOPDB_BACKUPS_ROOT` unset, so the backups code fell back to a POSIX path. Point it at a directory on this box and create it. |
| Backups can be requested but never run | expected outside Docker. The API queues a backup with a sentinel file; the work is done by a sidecar container, so a venv-only box queues and nothing executes. |

144
docs/EVENTSAVER.md Normal file
View File

@@ -0,0 +1,144 @@
<!-- docmeta
purpose: reference
audience: operator, developer
status: current
summary: What EventSaver reads, how it picks the running order, and the first thing to check when a bay shows the wrong slides.
loadwith:
-->
# EventSaver
A rotating-image screensaver for shop-floor PCs. Idle bays show the slides
managed in ShopDB, so a safety notice or a shift message reaches the floor
without anyone walking to each machine.
It is a small C# WinForms screensaver (`.scr`), not part of the ShopDB server.
It is documented here because ShopDB serves its content and because a site
adopting the fleet tools needs to know how it is configured.
For how to deploy it at a new site, see
[ADOPTING-AT-ANOTHER-SITE.md](ADOPTING-AT-ANOTHER-SITE.md). This page is what it
is, how it decides what to show, and how to diagnose it.
## Two sources, one of them preferred
EventSaver reads `EventSaver.ini` from the directory it lives in, **on every
launch**. Retargeting it never needs a recompile.
```ini
# HTTP mode: pull the playlist from ShopDB and cache it locally.
url=https://shopdb.example.net/api/slides/feed?surface=shopfloor
# Folder mode: used only when url is blank. SMB or local path.
# folder=\\fileserver\shopfloor\tv
interval=10 # seconds per image, unless a slide carries its own time
shuffle=0 # 1 = random order, 0 = ordered
fadems=600 # crossfade length in ms, reserved (v1 hard-cuts)
```
**HTTP mode is the one to use.** Slides are managed in ShopDB (Slides), the
feed is public so the screensaver needs no credentials, and each PC caches what
it fetched under its own `LocalApplicationData`. A bay that cannot reach the
server keeps showing the last set it saw instead of going black - which matters,
because the failure would otherwise be visible to the whole floor.
**Folder mode** predates the server and stays as a fallback for a site with no
ShopDB instance yet, or for content nobody wants in the database.
> If `EventSaver.ini` is missing, or both `url` and `folder` are blank, there is
> no source to read and the screensaver shows nothing. That is deliberate: the
> compiled-in fallback used to be the path of the site it was first built for, so
> a missing ini silently pointed a new site at someone else's file server. It is
> now empty, and failing visibly beats displaying the wrong site's slides. Ship
> the ini.
## What decides the running order
| Situation | Behaviour |
|---|---|
| No `order.txt` | Every image shows, sorted by filename. Number them `001_`, `002_` to sequence. |
| `shuffle=1` | Random order, ignored when a playlist is present. |
| `order.txt` present | ONLY the listed files show, in the listed order, looped. |
`order.txt` is the strict-sequence option and is the whole playlist - a file not
listed does not appear. Per-slide duration is appended with a pipe:
```
# Lines starting with # or ; are ignored.
001_welcome.jpg
002_safety_week.jpg|15
003_quality_board.jpg
```
In HTTP mode you do not write `order.txt` by hand: ShopDB's feed carries the
order and each slide's own seconds, and EventSaver writes the equivalent
`order.txt` into its cache so the same playlist logic runs either way.
A running screensaver rescans about every 30 seconds, so a slide change reaches
the floor without touching any PC.
## Command-line switches
Windows calls a screensaver with these, and EventSaver answers all three:
| Switch | Meaning |
|---|---|
| `/s` | Show fullscreen. What Windows uses on idle. |
| `/c` | Configuration. Reports the resolved source and cache directory, and points at the ini. |
| `/p <hwnd>` | Preview pane. Deliberately a no-op - it keeps the Settings dialog happy without drawing a thumbnail nobody looks at. |
`EventSaver.scr /c` is the first diagnostic: it says which source it resolved
and where it is caching, which answers most "why is it showing the wrong thing"
questions immediately.
## Where it should and should not run
A screensaver over a live display is a defect, not a feature. The reference
fleet excludes:
- **Kiosks and lobby displays**, which show a live dashboard.
- **Metrology bays** - CMM, Genspect, Keyence, wax-trace - where an operator
watches a measurement run.
- **Bays that sleep**, where the screensaver never wins anyway.
Targeting is by whatever converges shop-floor state at your site - a group
assignment in Intune, a DSC configuration, or an imaging-time task. Decide it
deliberately: the cost of getting it wrong is a screensaver
covering something someone needed to see.
## Building it
The source is `plugins/slides/client/EventSaver.cs`. No SDK required - it
compiles with the in-box .NET Framework compiler on any Windows 10 or 11
machine:
```
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe ^
/target:winexe /out:EventSaver.scr ^
/reference:System.dll,System.Drawing.dll,System.Windows.Forms.dll ^
EventSaver.cs
```
Ship `EventSaver.scr` and `EventSaver.ini` together. A `.scr` is an executable
with a different extension, so it goes to `C:\Windows\System32\`.
## Diagnosing it
| Symptom | Cause to check first |
|---|---|
| Blank screen, no images | Source unreachable and cache empty. Run `/c` and open the `url` in a browser on that PC. |
| Old slides only | Feed reachable at first run, not since. The cache is doing its job; fix the network path. |
| Wrong content | Pointing at another surface, or another site's server. `/c` reports which. |
| Local ini edits revert | Correct behaviour if the config is enforced by hash. Edit the copy your configuration tool deploys from, and update the hash it detects on. |
| Screensaver never starts | Per-user setting missing. Check `HKCU:\Control Panel\Desktop` for `SCRNSAVE.EXE`, `ScreenSaveActive` and `ScreenSaveTimeOut`. |
The feed is a plain public endpoint, so it can always be checked from the PC
itself:
```
https://shopdb.example.net/api/slides/feed?surface=shopfloor
```
An empty `slides` array means no slides have been uploaded for that surface -
the PC is configured correctly and there is nothing to show.

144
docs/FLEET-ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,144 @@
<!-- docmeta
purpose: explanation
audience: operator, adopter, developer
status: current
summary: How the server and the things that run on shop-floor PCs relate, and which page to open next.
loadwith:
-->
# How the pieces fit together
ShopDB is a server. Around it are the things that run on shop-floor PCs, each
documented in its own page, and each of those pages assumes you already know how
it relates to the others. This page is that relationship.
Read this one first, then open ONE of the pages in the table below. They are
written to be read alone once you have this page's shape in your head.
## The shape
```
+-------------------------------+
| ShopDB (one per site) |
| Flask API + Vue UI + MySQL |
+-------------------------------+
^ ^ ^ |
reports | | | | serves
| | | v
+----------------+ | | | +------------------+
| asset reporter |---+ | | | kiosks, displays |
| on every PC | | | | (browser, HTTPS) |
+----------------+ | | +------------------+
| |
+----------------+ | | +------------------+
| backup |----------+ +----| EventSaver |
| collectors | config revisions | screensaver |
+----------------+ +------------------+
pulls the slide feed
+----------------+
| endpoint agent | enrols, reports health, runs queued commands
| service, SYSTEM|
+----------------+
```
Every arrow is one direction on purpose. **Nothing on the server reaches out to
a PC.** The server holds state and answers questions; the fleet asks. That is
what lets a bay behind a firewall, asleep, or on a different network segment be
merely out of date rather than broken - and it is why every one of these tools
degrades to "stale" instead of "failed".
## What each piece is for
| Piece | Direction | What it does | Page |
|---|---|---|---|
| **Asset reporter** | PC to server | Says what this PC is: hostname, serial, type, user, NICs, machine number. Creates the PC record the rest of the system hangs off. | [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md) |
| **Endpoint agent** | PC to server | Enrols the PC for its own credential, reports health, and runs the commands the server has queued for it. | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| **Backup collectors** | PC to server | Post machine configuration (NTLARS/DNC, CMM, part marker, UDC) as revisions, so a controller can be restored and a change can be seen. | [BACKUP-RESTORE](BACKUP-RESTORE.md) |
| **EventSaver** | PC to server | Pulls the slide playlist and shows it when a bay is idle. | [EVENTSAVER](EVENTSAVER.md) |
| **Kiosks and displays** | PC to server | A browser in kiosk mode on a ShopDB route. No client software beyond the browser. | [EVENTSAVER](EVENTSAVER.md) |
| **Printer installer** | PC to server | Reads the printer map and installs the network printers a bay should have. Laid down by a manifest entry. | [PRINTER-INSTALLER](PRINTER-INSTALLER.md) |
**ShopDB does not lay software on a bay.** Getting the reporters and the
screensaver onto a PC in the first place is the job of whatever already
converges shop-floor state at your site - an imaging task, Intune, DSC, or a
configuration-enforcement tool of your own. ShopDB serves the data those tools
and those clients read; it never pushes.
Setting all of this up at a plant that does not have it yet is
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md).
## The three ways a PC talks to the server
**The collector API**, `POST /api/collector/*`. Used by the asset reporter, the
printer-queue reporter and the endpoint agent. A PC authenticates as ITSELF with
a per-PC credential where it has enrolled, and with a shared fleet key where it
has not. Idempotent by design: the same report twice changes nothing.
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the one page that defines that choice;
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) is the payload contract, and ADR-006
is the rationale.
**Scoped service tokens.** An unattended integration that is not a single PC -
a build job, a site tool, another system - authenticates with a managed token
limited to named permissions, optionally bound to specific resources. This path
does NOT take per-PC credentials: it is a scope talking to the server, not a PC.
Minting, binding and revoking one is Settings > API Tokens.
**Plain public reads.** The slide feed, the display-role lookup and the kiosk
dashboard routes need no credentials at all, because a screensaver and a wall
display have nowhere to keep one. They are read-only and carry nothing a floor
visitor should not see.
## Where the credentials live
Both ends of every credential, in one table, because a credential described from
only one end is the kind that gets provisioned twice. The collector row is
defined in full by [COLLECTOR-AUTH](COLLECTOR-AUTH.md).
| Credential | On the server | On the PC | Rotate by |
|---|---|---|---|
| Per-PC endpoint credential | One row per asset, sha256 only. Issued by enrollment, revocable per PC | `%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI LocalMachine, ACL'd to SYSTEM and Administrators | Revoking it and letting the PC enroll again. One PC at a time, no fleet touch |
| Shared collector key (legacy) | A managed token scoped to `collector.ingest` only. `COLLECTOR_API_KEY` in `.env` also works and is simpler for a pilot, but cannot be revoked per fleet | `HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey` | Replacing the value the fleet reads from, so every PC picks it up on its next cycle. Never put it in a manifest - manifests sync broadly |
| Scoped service token | A managed token limited to named permissions, optionally resource-bound | Wherever the integration keeps its secrets; on a PC, `HKLM:\SOFTWARE\GE\ShopDB` value `ApiToken` | Minting a bound replacement and revoking the old one |
| Server URL | n/a | `HKLM:\SOFTWARE\GE\ShopDB` value `BaseUrl` | Setting it once at install; nothing should overwrite a bay's own value |
| Server pins | The leaf certificate's SPKI SHA-256, shown on the Server Certificate settings page | `HKLM:\SOFTWARE\GE\ShopDB` value `ServerPins` (REG_MULTI_SZ) | Writing the new pin set; the agent re-reads it within about a minute |
| Application secrets | `.env`: `SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, integration tokens | n/a | See [BACKUP-RESTORE](BACKUP-RESTORE.md) - losing these invalidates every issued token |
Two rules behind that table. A shop-floor PC is not a trusted place, so what it
holds is scoped to exactly what it does: a credential that leaks buys the reader
the ability to file ONE PC's report, not to read the asset register. And a
credential is delivered, never typed - a value that has to be entered per machine
is a value that is wrong on some machine.
## What owns what
The question that decides most arguments: **who is allowed to change this?**
- **ShopDB owns the record.** What a PC is, what it drives, what it should have.
- **Your configuration tool owns the PC's configuration.** Whatever converges
shop-floor state at your site puts back anything edited on the bay, on its own
schedule. ShopDB records; it does not enforce.
- **The PC owns its own identity.** Serial, hostname, installed software: the
server records what it is told rather than deciding. The one exception is
which NIC is the PC's address, which moved to the server because only the
server knows the site's addressing.
Where two systems both converge state - Intune DSC and a shop-floor enforcement
tool, say - decide which owns which layer before deploying both, or they will
fight in a way that is very hard to see.
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md) has the recommended
split.
## When something looks wrong
| Symptom | The piece to look at first |
|---|---|
| A PC is missing from ShopDB entirely | Asset reporter: never ran, or no credential and no collector key |
| A PC is there but its details are old | Asset reporter: stopped running. The dashboard's "PCs not reporting" card is this |
| A PC reports 401 after an admin action | Its credential was revoked. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| A PC has the wrong software | Not ShopDB: check whatever converges configuration at your site. ShopDB records what the PC reported, it does not decide it |
| A bay shows the wrong thing on screen | EventSaver config, or the display-type dashboard default - both are per PC type |
| A machine has no recent config backup | Backup collector on the controlling PC. "Backups that have stopped" on the dashboard |
| Two PCs claim one machine number | Neither: it is a claim in progress, or a typo. The dashboard card names both |
Each row is a different program on a different schedule. Knowing which one to
open is most of the diagnosis.

137
docs/FLOOR-MAP.md Normal file
View File

@@ -0,0 +1,137 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Load floor plans, place markers, and move every marker safely when a plan is redrawn.
loadwith:
-->
# The floor map: buildings, levels and marker positions
The map shows where things physically are. This page is the operator's view for
whoever maintains it: loading a floor plan, adding a second floor or a second
building, and moving the markers when the plan changes. The reasoning behind the
model is [ADR-017](adr/ADR-017-buildings-and-levels.md).
## The model in three sentences
A **building** holds one or more **levels**. A level owns a floor-plan image (one
per theme) and the pixel size of that image. A marker's position is an X and Y in
the pixels of ONE level, so every position names the level it belongs to.
That last point is the whole design. Coordinates alone are meaningless: `1200,900`
is a spot on the ground floor and a different spot on the second floor. A marker
whose level is unknown shows as **level unknown** and is not drawn, because
drawing it on the default plan would look completely correct while pointing at
the wrong part of the building.
## Loading a floor plan
**Settings > Buildings and levels.**
1. Add a building if you do not have one. A single-building site needs exactly
one, and it can be called anything.
2. Add a level. Give it a name people use out loud - "Ground floor", "Mezzanine",
"Second floor" - and a sort order that puts the list in the order someone
would walk it. Name and order are separate on purpose: levels are not reliably
numbered, and leaving gaps in the order lets you insert one later without
renumbering everything.
3. Upload the plan, per theme. A light-theme image is enough to start; if there
is no dark version, the light one is used in both.
**The level takes its pixel size from the image you upload, while the level is
still empty.** Once markers are on it, an upload whose size does not match reports
the mismatch and changes nothing. That is deliberate - see below.
Supported: PNG, JPEG, GIF, WEBP and SVG. An SVG is served with headers that stop
it executing as script, so a plan exported from a CAD tool is safe to use.
## Placing markers
Two ways, both fine:
- **One at a time**, from the asset's own form: the map-position picker opens on
the level the marker is already on, and you can switch level inside the picker.
The saved position states which drawing it is on.
- **Many at once**, in the map editor. Filter to what you are working through -
unplaced, needs-review, or one level - search for a specific thing, place it,
and confirm. Placing a marker by hand counts as confirming it.
A location can carry a position too, which acts as the default for assets in it.
## When the floor plan changes
This is the case that needs care, and the reason the tooling exists.
**A new plan of the SAME dimensions**: upload it and you are done. Markers keep
their coordinates and still line up.
**A new plan of DIFFERENT dimensions, or a real move**: the coordinates are pixels
of the old image, so every marker is now in the wrong place relative to the
drawing. Do NOT fix this by editing the level's width and height - that just
reinterprets every existing position against a new scale.
Use **Recalibrate** in the map editor:
1. Pick two or more **landmarks** - features you can find on both the old and the
new plan. Corners of the building, a doorway, a column. Two is the minimum;
three or four spread across the plan is better.
2. For each, give the old coordinates and the new ones.
3. Review the **dry run**. It lists every marker's old and new position and flags
anything that would land off the drawing. Nothing has changed yet.
4. Apply it. The previous positions are snapshotted first.
The transform is worked out separately for the X and Y axes from your landmarks,
and never guessed from the image dimensions. That matters in the exact case this
was built for: a plan that gained a floor below it got taller without anything
being rescaled, and a size-derived scale would have stretched everything
vertically.
After a recalibration every affected marker is marked **needs review**, because a
transform is a good starting guess and nothing in the coordinates can tell you
which machines actually moved. Work the needs-review filter, correct what is off,
and confirm as you go.
## Undo
Every bulk change snapshots the positions first, and a restore snapshots before
it restores, so an undo is itself undoable. The history is in the map editor:
what caused each snapshot, when, and whether it has been restored. A restore puts
back the level and the review state, not only the coordinates.
Assets deleted since a snapshot are reported and skipped rather than failing the
whole restore.
## Adding a second building or a second floor
Add it, upload its plan, then move the markers that belong on it. In the map
editor, markers on other levels are badged, so you can see what still needs
moving.
Do not stack two floors onto one tall image. It looks simpler and it makes the
level implicit in the Y coordinate - "above 2550 means upstairs" - which nothing
can validate and which breaks the moment either plan is redrawn.
## Things that are not levels
- **Locations** are where something is in words ("Cell 4", "Gage lab"). A level is
which drawing. A location can name a level, but they are different things.
- **Business units** are organisational, not physical.
## Known limits
- The printer-installer map (the standalone program that installs printers by
clicking a floor plan) has one plan compiled into it and does not know about
levels. It is accurate for the default level only. See
[PRINTER-INSTALLER](PRINTER-INSTALLER.md) section 6.
- The first-run setup wizard still asks for one site-wide blueprint; those
settings no longer drive the map. Use Settings > Buildings and levels.
- Map PDF export covers the level you are looking at.
## For integrators
Reading levels is public and writing them needs an admin; repositioning markers
needs `assets.edit`. The rules and the parameters are both in the generated
spec at `/api/docs`. If you import positions
from another system, send `levelid` with every coordinate - see
[IMPORT-API](IMPORT-API.md).

237
docs/GUIDES.md Normal file
View File

@@ -0,0 +1,237 @@
<!-- docmeta
purpose: reference
audience: developer, integrator
status: current
summary: The guides block model, API, publishing rules and limits, for creating guides from a script.
loadwith:
-->
# Guides: writing procedures in ShopDB, and generating them from elsewhere
A guide is a step-by-step procedure stored in ShopDB: images, working
checklists, version history, and printable output. This is the reference for the
data model and the API, aimed at somebody generating guides programmatically -
from a script, an importer, or a model.
Not the **knowledgebase** plugin, which is a link registry pointing at documents
held somewhere else. That one references documents; this one holds them. They
share only a tag vocabulary and the global search box.
---
## 1. The shape of a guide
A guide is an ordered list of typed **blocks**, stored as JSON. Not HTML, and
that is load-bearing:
- a tickable step needs a **stable id** to be remembered between visits
- **dark mode** needs semantic markup, not pasted colours
- **print** needs to know where a step begins so it does not split one across a
page
Free-form HTML gives none of those, and pasted styling fights the theme.
### Block types
| type | Fields | Notes |
|---|---|---|
| `heading` | `text` | Section heading |
| `paragraph` | `text` | Body text |
| `step` | `title`, `text` | Numbers itself by position among steps |
| `checklist` | `items[]` | The tickable one. Items need their own ids |
| `bullets` | `items[]` | Unordered list |
| `numbers` | `items[]` | Ordered list |
| `callout` | `text`, `title`, `variant` | `variant`: `info`, `warning`, `success` |
| `code` | `text`, `language`, `caption` | Never autocorrected; wraps rather than clips |
| `image` | `url`, `alt`, `caption` | `url` comes from the image upload endpoint |
| `table` | `header[]`, `rows[][]` | Simple grid |
Every block needs:
```json
{ "id": "b1", "type": "paragraph", "text": "..." }
```
**Ids must be unique and stable.** They are what a reader's tick is stored
against, so a save without them is refused, and changing one loses that item's
progress. Checklist items carry their own ids for the same reason:
```json
{ "id": "b4", "type": "checklist",
"items": [ { "id": "i1", "text": "Confirm the Verified date" } ] }
```
An unknown block type renders as nothing rather than an error, so a guide
written against a newer version stays readable on an older one.
### Inline text
Block text supports a deliberately small subset: `**bold**`, `*italic*`,
`` `code` ``, `[links](https://example.com)`, and line breaks. Nothing else -
no fonts, sizes or colours. Emphasis is semantic and the theme decides how it
looks, which is what makes dark mode work and keeps a guide written next year
matching one written today.
Text is escaped **before** formatting is applied, so no markup in the source can
reach the DOM. `javascript:` and `data:` links are dropped.
---
## 2. Creating a guide from a script
Every endpoint is listed in `docs/api-inventory.json`; this is the workflow.
```bash
# 1. Create - always starts as a draft
curl -X POST http://localhost:5001/api/guides \
-H "Authorization: Bearer $TOK" -H 'Content-Type: application/json' \
-d '{
"title": "Replace a part marker controller",
"summary": "Restoring patterns and settings after a swap",
"tags": "partmarker, backups",
"blocks": [
{"id":"b1","type":"paragraph","text":"The new unit arrives **empty**."},
{"id":"b2","type":"callout","variant":"warning","title":"Before you start",
"text":"Check the marking head."},
{"id":"b3","type":"checklist","items":[
{"id":"i1","text":"Copy the snapshot path"},
{"id":"i2","text":"Confirm the pattern count"}]}
]
}'
# 2. Publish - readers see nothing until this happens
curl -X POST http://localhost:5001/api/guides/12/publish \
-H "Authorization: Bearer $TOK"
```
`$TOK` is a login JWT or a scoped API token; they work the same way here.
### What the server rejects
Validation happens at the door rather than at render time, because a malformed
guide stored is a page that fails every time somebody opens it, long after the
client bug that caused it:
- a block with no `id`, or a duplicate `id`
- an unknown block `type` (the error lists the known ones)
- a checklist item with no `id`
- more than 500 blocks, or more than 2 MB of blocks
- text longer than 20,000 characters in one block
Errors name the offending block by position: `block 3: needs a stable id`.
---
## 3. Drafts, versions and publishing
Three separate things, and the distinction is what stops a live procedure
changing under somebody following it:
| | What it is |
|---|---|
| **Working copy** | The `guides` row. What the editor loads and saves |
| **Revisions** | Immutable snapshots in `guiderevisions`, written when content changes |
| **Published pointer** | Which revision READERS get |
- `PUT /api/guides/<id>` saves the working copy and writes a revision **only if
the content hash changed**. An autosave that changed nothing records nothing,
so the history stays readable. The response carries `saved: false`.
- The hash covers title, summary, tags **and** blocks - renaming or retagging is
a change a reader notices.
- `POST /api/guides/<id>/publish` **promotes an existing revision**. It does not
snapshot, so a publish cannot ship unsaved edits nobody reviewed.
- `POST /api/guides/<id>/restore/<revisionid>` brings old content back as a
**new** revision. History is never rewritten.
- Drafts are invisible to readers, to `GET /api/guides/<slug>` (404, not 403),
and to global search.
---
## 4. Images
```bash
curl -X POST http://localhost:5001/api/guides/images \
-H "Authorization: Bearer $TOK" -F file=@screenshot.png -F guideid=12
```
Returns `{"url": "/api/guides/images/5", ...}` for use as an image block's
`url`. PNG, JPEG, GIF and WebP, up to 8 MB. `guideid` is optional because an
image is pasted before a new guide has an id.
Bytes live in the database, not on disk: a row travels with a database backup,
whereas a file under `instance/` has been lost to a container rebuild on this
deployment before.
**Known gap:** images uploaded to a guide that is then abandoned are never
pruned.
---
## 5. Proofing
The editor runs checks continuously and warns rather than blocking. They exist
because none of these are spelling mistakes:
- a **smart quote or long dash in a code block** - autocorrect does this, the
command then fails, and it still looks right to the author (one-click fix)
- an empty checklist item, which reads as a step somebody forgot to write
- a heading with nothing under it, left behind by a reorganise
- an image with no alt text
- a leftover `TODO`, `TBD` or `FIXME`
- no summary or tags, which makes a guide hard to find
Publishing with unresolved errors asks for confirmation; it does not refuse. A
procedure that exists imperfectly beats one nobody wrote.
---
## 6. Printing
Print always renders **light**, whatever the screen theme, because a dark page
lays a full sheet of toner down and knocks the text out in white. Buttons and
progress bars are dropped, steps and code blocks are kept whole across page
breaks, code wraps rather than clips, and checkboxes print **empty** even where
ticked on screen - a printout is a worksheet somebody ticks by hand. The footer
carries the version and the print date, so a stale copy on a noticeboard is
spottable.
---
## 7. Permissions
| Permission | Allows |
|---|---|
| `guides.view` | Read guides |
| `guides.create` | Create, and edit your OWN guide while it is unpublished |
| `guides.edit` | Edit any guide, including published ones |
| `guides.publish` | Publish or withdraw |
| `guides.delete` | Delete |
The split between `edit` and `publish` is deliberate: a technician can write a
procedure up while somebody else decides it is fit for the floor.
---
## 8. Converting existing documents
There is no importer; conversion is a script you write against section 2. What
to expect from a source format:
- **Structured HTML and Markdown convert cleanly** - headings, fenced code,
lists, tables and callout divs all have matching block types. The GE Aerospace
KB template's `alert alert-warning` divs map to warning callouts, `code-block`
to code blocks, `step` divs to steps.
- **Plain-text runbooks convert poorly.** ASCII underlines and hand-indented
lists carry no structure, so everything lands as paragraphs and a person marks
up the steps afterwards. Worth doing anyway - the text is then searchable and
versioned - but budget the cleanup.
- Generate ids yourself and keep them stable across re-imports, or every
re-import resets readers' progress.
---
## See also
- `docs/api-inventory.json` - every endpoint with auth, params and an example
- `plugins/guides/services/blocks.py` - the validation rules, and why each exists
- `plugins/guides/services/authoring.py` - the single write path

96
docs/IMPORT-ADOPTION.md Normal file
View File

@@ -0,0 +1,96 @@
<!-- docmeta
purpose: explanation
audience: adopter, developer
status: current
summary: How to build the per-site loader that reads your source database and drives the import API.
loadwith: IMPORT-API.md
-->
# Building a per-site import loader
**Four import pages.** Shaping the loader: this page. The HTTP wire contract it
drives: [IMPORT-API](IMPORT-API.md), which this page assumes you have read and
does not restate. Spreadsheet and no developer: [CSV-IMPORT](CSV-IMPORT.md).
Source is the classic-ASP shopdb: [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md).
## Two layers, and why the split holds
Every adopting site has its own source database, and it will not match another
site's schema. So the import is split:
1. **The import API is the stable contract.** Whatever your source looks like,
you create records through the same documented REST endpoints. This layer is
the product; it is schema-agnostic and documented once, in IMPORT-API.
2. **A per-site loader is thin glue.** It reads *your* source database and POSTs
to those endpoints. Nobody runs another site's loader - you copy the pattern.
Keeping the two apart is what stops one site's judgement calls from becoming
everyone's contract.
## The shape of a loader
The reference implementation lives under `scripts/site_imports/<site>/`. Read it
alongside this page.
- `harness.py` - builds the app against the target `DATABASE_URL`, mints an
unscoped admin PAT in-process, and drives the real endpoints through the app
test client with `Authorization: Bearer <pat>` + `X-Import-Mode: true`. This
exercises the same routes, authorization and validation an HTTP client would,
with no running server needed. It also holds read-only access to the source DB
and a JSON `IdMap` of legacy-id to new-id crosswalks.
- `run.py` - ordered `stage_*` functions. Each reads a slice of the source,
POSTs it, and records the crosswalk later stages resolve foreign keys against.
Post-import fixups that re-point existing assets (for example
`scripts/reclassify_servers_to_network.py`, which moved servers imported as PCs
to network devices in place) belong in the site loader's verify stage, never in
the API layer.
### The crosswalk is the keystone
Reference and lookup tables first so foreign keys resolve, then the entity hub,
then dependents, then links:
```
reference -> catalog -> assets (persist the source-id -> assetid crosswalk)
-> dependents (installs, warranties, notifications, ...) -> relationships
```
Capture every legacy id to new id as you create rows, and resolve foreign keys
through it in later stages. New autoincrement ids will not match the source's.
The full ordering rule is section 4 of [IMPORT-API.md](IMPORT-API.md).
## Producing the mapping
You do not have to hand-derive the source-to-target mapping. Point an
agent-assisted workflow at a source database plus the API contract and it emits
a per-table mapping (source columns to endpoint fields, transforms, what is
importable and what is out of scope) plus a loader skeleton. That is the
repeatable onboarding path, and
[IMPORT-LEGACY-ASP.md](IMPORT-LEGACY-ASP.md) is a finished example of the output.
## Running it, against a THROWAWAY import database
1. Build a fresh target: `flask db upgrade`, `flask plugin upgrade-all`, then
`flask seed permissions/settings/reference-data`. Enable every bundled plugin
you need - some ship disabled, and a plugin's routes only register when it is
enabled at app start.
2. Load your source dump into a scratch database the loader can read.
3. Run the loader stages in order, dry-running and spot-checking as you go.
4. Verify: row-count and foreign-key-resolution audit against the source, then a
UI spot-check - log in and eyeball the lists, the map, a detail page.
5. Only then point a real instance at the imported database.
## What a mature loader ends up handling
From the reference implementation, as a checklist of what to expect:
- Fanning one source "machine" table out to several asset types
(computer/machine/network/measuring-tool) by a routing rule, with the
duplicate, placeholder and skip decisions applied.
- Synthesizing a natural key where the source lacks one (printers to `PRN-{id}`).
- Folding a primary IP onto an asset, pairing a check-in/out event log into
checkouts, deduping colliding names, reversing an inverse relationship type.
- The handful of narrow gaps the API cannot cover (there is no bulk
communications endpoint, for instance) handled as documented direct-ORM
writes.

283
docs/IMPORT-API.md Normal file
View File

@@ -0,0 +1,283 @@
<!-- docmeta
purpose: reference
audience: developer, integrator, adopter
status: current
summary: The stable HTTP import contract - PAT auth, X-Import-Mode timestamps, and the lookup-then-upsert recipe.
loadwith:
-->
# Import API: loading a site's data through HTTP
**Four import pages.** The HTTP wire contract: this page. Spreadsheet and no
developer: [CSV-IMPORT](CSV-IMPORT.md). How to shape a per-site loader around
this contract: [IMPORT-ADOPTION](IMPORT-ADOPTION.md). Source is the classic-ASP
shopdb: [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) on top of this page.
This page is schema-agnostic: it says how to authenticate, how to preserve a
legacy row's real timestamps, and how to make a run rerunnable. It does NOT map
anybody's source columns - that is the loader's job.
Every row is created through a documented endpoint, so authorization,
validation, auditing and plugin hooks run exactly as they do for a human
operator. No direct writes to the application database are needed or wanted.
Contents: [1 Authentication](#1-authentication) |
[2 Import mode](#2-import-mode-the-x-import-mode-header) |
[3 Reference-data seed](#3-reference-data-seed) |
[4 Order of operations](#4-order-of-operations) |
[5 The idempotent upsert recipe](#5-the-idempotent-upsert-recipe) |
[6 Verifying a run](#6-verifying-a-run)
---
## 1. Authentication
Examples on this page use `http://localhost:5001`, a development instance.
Substitute your own base URL; every import call targets `/api/...` on it.
Every write needs authentication, and import mode additionally needs an admin.
A large import can outlast a login JWT: `access_token` expires after one hour,
so a long run dies mid-import with 401s. Use a **personal API token (PAT)**
instead. A PAT never expires (unless you set an expiry), acts as the user that
created it, and is sent exactly like a JWT: `Authorization: Bearer
shopdb_pat_...`. Create one as an admin, from Settings > API Tokens or the API:
```bash
# Bootstrap: a short login JWT is fine just to mint the long-lived PAT.
JWT=$(curl -s http://localhost:5001/api/auth/login \
-H 'Content-Type: application/json' \
-d '{"username":"<admin>","password":"<password>"}' | jq -r '.data.access_token')
# The full secret (shopdb_pat_...) is returned ONCE. Save it now.
curl -s http://localhost:5001/api/apitokens \
-H "Authorization: Bearer $JWT" \
-H 'Content-Type: application/json' \
-d '{"name":"legacy import runner"}' | jq -r '.data.secret'
```
Revoke it from the same Settings page (or `DELETE /api/apitokens/<id>`) when
the import is done. Minting one needs the `apitokens.create` permission, which
admins have by default.
Use an **unscoped** token. A token may optionally carry a scopes list that
limits it to specific permissions; a scoped token suspends the admin bypass and
is denied on role-gated endpoints AND on import mode, so it cannot run an
import. Leave the "Restrict permissions" option off, which is the default.
## 2. Import mode: the `X-Import-Mode` header
By default the server stamps `createddate`/`modifieddate` to "now" on every
create and update, which would erase a migrated row's real history. To preserve
it, send the request header:
```
X-Import-Mode: true
```
When (and only when) the caller is an admin AND that header is present:
- create/update endpoints on timestamped entities accept optional
`createddate` and `modifieddate` fields in the JSON body and store them
verbatim (naive UTC). Both `2020-01-05T12:00:00` (ISO) and the legacy
`2020-01-05 12:00:00` (MySQL) forms are parsed. A bare `2020-01-05` works too.
Without the header, or for a non-admin caller, those fields are silently ignored
and the server behaves exactly as it does normally. This is enforced centrally
by `shopdb/utils/import_mode.py` (`import_mode_active`, `apply_import_timestamps`,
`parse_import_datetime`), exposed on the plugin contract surface `shopdb.api`.
Timestamped entities that honor `createddate`/`modifieddate`: assets (every
type plugin), vendors, models, modeltypes, businessunits, locations, operating
systems, applications, knowledge base, asset relationships.
Entities that carry history in domain fields instead (createddate passthrough is
a no-op there, by design): notifications (`starttime`/`endtime`), warranties
(`startdate`/`enddate`/`lastcheckeddate`). Set those fields directly in the
payload; they are already accepted.
## 3. Reference-data seed
Before importing, seed the reference tables that have no CRUD endpoint of their
own (communication types such as IP/Serial/USB, default statuses, canonical
relationship types, permissions, settings):
```bash
flask seed permissions
flask seed settings
flask seed reference-data
```
`communicationtypes` is populated here, so a primary-IP mapping can resolve
`comtype='IP'`. There is no CRUD endpoint for it; the seed is the only way in.
---
## 4. Order of operations
Import in dependency order so foreign keys always resolve. Each step is a
lookup-then-upsert loop (section 5); rerunning any step is safe.
1. **Reference and lookup data first**
1. Vendors (`/api/vendors`)
2. Model types (`/api/modeltypes`)
3. Models (`/api/models`) - needs vendors and model types
4. Business units (`/api/businessunits`)
5. Location types, then locations (`/api/locations/types`, `/api/locations`)
6. Operating systems (`/api/operatingsystems`)
7. Asset statuses (`/api/assets/statuses`)
8. Relationship types (`/api/assets/relationshiptypes`)
9. Notification types (`/api/notifications/types`)
10. Per-plugin subtypes: computer, machine, printer and network-device types
11. Support teams and their contacts (`/api/supportteams`,
`/api/supportteams/{id}/contacts`) - BEFORE applications, because
`applications.supportteamid` points at a team
12. Applications (`/api/applications`) and their versions
2. **Assets, per type.** Each create makes the core Asset row plus its
extension: computers, machines, printers, network devices.
3. **Communications.** The primary IP is set through the asset payload's
`ipaddress` field in step 2. There is no bulk-communications endpoint, so
secondary interfaces have no target; carry them as custom fields or defer
them.
4. **Relationships** (`/api/assets/relationships`): needs both endpoint assets
and the relationship types to exist already.
5. **Installed applications** (`/api/computers/{id}/apps`): needs computers and
applications.
6. **Knowledge base, notifications and warranties**, including backdated
history.
7. **Custom fields** for any source column with no home in the target schema:
`POST /api/customfields` once per field, then
`PUT /api/customfields/asset/{assetid}`. Custom-field values are not
timestamped, so they carry no history.
### A map position needs its level
`mapx`/`mapy` are absolute pixels in the native coordinate space of ONE level
(ADR-017), so an import that sends coordinates without `levelid` produces
markers the map renders as "level unknown" rather than drawing them on the
default level. A source that predates levels has one floor plan, so the right
value is that site's default level:
```bash
curl -s http://server/api/maplevels | python -c "import json,sys; print(json.load(sys.stdin)['data']['defaultlevelid'])"
```
Send that `levelid` on every row carrying coordinates. If an import already ran
without it, the levels migration's backfill is not retroactive - assign the rows
with `POST /api/mappositions/positions`, which requires `levelid` per row for
exactly this reason. Loading and recalibrating the plans themselves is
[FLOOR-MAP.md](FLOOR-MAP.md).
---
## 5. The idempotent upsert recipe
The endpoints are NOT upserts. The idempotent unit is a two-step recipe that
composes with import mode:
1. **Look up** the row by its natural key using the exact-match list filter.
2. If found, **PUT** to update it; if not, **POST** to create it.
Each import-relevant list endpoint has an exact-match filter for its natural key
(added for exactly this purpose):
| entity | lookup |
|---|---|
| assets (all 5 plugins) | `GET /api/{plugin}?assetnumber=<n>` |
| vendors | `GET /api/vendors?vendor=<name>` |
| models | `GET /api/models?modelnumber=<m>&vendor=<vendorid>` |
| model types | `GET /api/modeltypes?modeltype=<name>` |
| business units | `GET /api/businessunits?businessunit=<name>` |
| locations | `GET /api/locations?locationname=<name>` |
| operating systems | `GET /api/operatingsystems?osname=<name>` |
| applications | `GET /api/applications?appname=<name>` |
| knowledge base | `GET /api/knowledgebase?linkurl=<url>` |
| warranties | `GET /api/warranty?servicetag=<tag>&vendor=<name>` |
| notifications | `GET /api/notifications?ticketnumber=<t>` |
### Worked example
The loop, with the two headers that matter and a dry run. A single bad row
reports and keeps going: aborting a multi-hour run on row 4000 is worse than
finishing with a list of failures.
```python
import os
import requests
BASE = "http://localhost:5001"
class ImportClient:
def __init__(self, token=None, dryrun=False):
self.session = requests.Session()
self.dryrun = dryrun
self.session.headers.update({
# A PAT (shopdb_pat_...) does not expire like a login JWT, so it
# survives a long import. X-Import-Mode makes the createddate /
# modifieddate passthrough take effect.
"Authorization": f"Bearer {token or os.environ['SHOPDB_TOKEN']}",
"X-Import-Mode": "true",
})
def lookup(self, path, params):
resp = self.session.get(f"{BASE}{path}", params=params)
resp.raise_for_status()
rows = resp.json().get("data") or []
return rows[0] if rows else None
def upsert(self, path, idfield, lookupparams, payload):
"""Lookup by natural key; PUT if found, else POST. Returns the row."""
existing = self.lookup(path, lookupparams)
if self.dryrun:
print(f"[dry-run] {'PUT' if existing else 'POST'} {path} {lookupparams}")
return existing or payload
if existing:
resp = self.session.put(f"{BASE}{path}/{existing[idfield]}", json=payload)
else:
resp = self.session.post(f"{BASE}{path}", json=payload)
if resp.status_code >= 400:
print(f"ERROR {resp.status_code} {path}: {resp.text[:200]}")
return None
return resp.json()["data"]
# One entity, in the order of section 4. Legacy history is preserved because
# X-Import-Mode is set.
client.upsert(
"/api/vendors",
idfield="vendorid",
lookupparams={"vendor": row["vendor"]},
payload={"vendor": row["vendor"],
"createddate": row.get("dateadded"),
"modifieddate": row.get("lastupdated")},
)
```
Keep a `legacy_id -> new_id` dictionary for every entity as you import it; you
need it to remap foreign keys (a machine's `businessunitid`, a checkout's
`machineid`, a relationship's `machineid`/`related_machineid`, and so on).
---
## 6. Verifying a run
Compare counts after each phase. The new side exposes a total in the pagination
metadata, so one call per entity answers it:
```bash
curl -s "http://localhost:5001/api/vendors?per_page=1" \
-H "Authorization: Bearer <token>" | jq '.meta.pagination.total'
```
Count the same thing on the source however that source is queried, and
investigate any gap. Where your loader routes one source table to several target
plugins, the source-side count has to be filtered the same way the routing was,
or the two numbers are not measuring the same set.
Beyond counts: resolve-rate on foreign keys (nothing silently landed NULL), then
a UI spot-check - log in, open a list, the map, and one detail page.
The wider API surface and its auth model is the generated spec at `GET /api/docs`
(`docs/openapi.json`); [llms.txt](llms.txt) is the short version of the envelope
and the auth model.

163
docs/IMPORT-LEGACY-ASP.md Normal file
View File

@@ -0,0 +1,163 @@
<!-- docmeta
purpose: reference
audience: adopter, developer
status: historical
summary: Record of one classic-ASP shopdb migration: the fan-out routing, the judgement calls and what was left behind.
loadwith: IMPORT-API.md
-->
# Migrating the classic-ASP shopdb: the record
**Read this only if your source is the classic-ASP shopdb.** It is one site's
completed migration, kept for the parts a second site would otherwise rederive
and for the parts that still matter after the import is over: how the `machines`
hub was fanned out, which judgement calls were made, and which legacy tables
were deliberately left behind.
It is NOT the product contract. Authentication, `X-Import-Mode`, the seed, the
ordering rule, the idempotent upsert recipe and the verification method are in
[IMPORT-API.md](IMPORT-API.md), which is current and schema-agnostic. Read that
first; this page assumes it. The mechanical column-by-column mapping that the
run was driven from is not reproduced here - the source and target column names
match closely enough to re-derive, and a copy would only drift.
What dates it, and what you must re-check against your own database rather than
trust here: the `machinetypeid` integers below are ids from ONE site's
`machinetypes` lookup, not a shared vocabulary; the row counts are that
database's; and the routing is a recommended default that a site may
legitimately depart from.
---
## The `machines` hub fans out into the asset plugins
`machines` (885 rows) is the central legacy asset table. Two columns drive the
fan-out: `machinetypeid` (what the asset physically is) and `pctypeid` (a
computer's sub-type). Route each row by `machinetypeid`:
| legacy `machinetypeid` | `machinetypes.machinetype` | target plugin | subtype source |
|---|---|---|---|
| 1 | LocationOnly (also `islocationonly=1`) | core **Locations** (NOT an asset) | `locationtype` |
| 33 | PC | **computers** | `computertype` from `pctype.typename` via `machines.pctypeid` |
| 20 | Server | **computers** | computer type "Server" |
| 15 | Printer | **printers** | `printertype` (authoritative source is the legacy `printers` table) |
| 16 Access Point / 17 IDF / 18 Camera / 19 Switch / 46 Firewall | | **network** | `networkdevicetype` |
| 44 | USB Device | **usb** | (usb device) |
| 23 Measuring Machine / 3 CMM / 48 Spline Checker / 8 Eddy Current / 47 Inspection | | **measuringtools** (gage-lab judgment call; ADR-005) | `measuringtooltype` |
| 2,4,5,6,7,9,10,11,12,13,14,21,22,24,25,45 (lathes, mills, welders, grinders, ...) | | **machines** | `machinetype` |
Decide the routing table once, up front: it is what every later count, report
and parity check is measured against. A site may re-route a `machinetypeid` -
sending CMM to `machines` rather than `measuringtools`, for example.
Non-obvious column mappings on the hub, the ones that are not a same-name copy:
- `machinenumber` is the `assetnumber` (the business identifier and the natural
key); `alias` or `hostname` becomes the layperson `name`.
- `mapleft`/`maptop` become `mapx`/`mapy`, which are pixels of ONE level. An
import that omits `levelid` produces markers that are never drawn - see
[IMPORT-API.md](IMPORT-API.md#a-map-position-needs-its-level).
- `machinestatusid`, `businessunitid`, `osid` are remapped through the lookups
imported before the assets.
- `dateadded`/`lastupdated` become `createddate`/`modifieddate`, honored in
import mode only.
- `ipaddress1` is the primary IP, folded onto the asset as `ipaddress`.
## Lookups that are not a straight copy
Most legacy lookup tables map name-to-name onto the obvious endpoint. These do
not:
| legacy table | target | the catch |
|---|---|---|
| `machinetypes` | `POST /api/modeltypes` | becomes the MODEL type vocabulary (ADR-011), and needs a `category` (Equipment/Computer/...) the legacy row does not carry |
| `controllertypes` | `vendors` + `models` | one legacy row splits in two: "Fanuc" becomes a Vendor, the controller model becomes a Model, then set `controllervendorid`/`controllermodelid` on the machine |
| `pctype` | `POST /api/computers/types` | `typename` becomes `computertype`; also drives the computer subtype in the fan-out above |
| `subnets` | `POST /api/network/subnets` | create the VLAN first (`POST /api/network/vlans`), then reference it; `subnettypeid` resolves to a `subnettype` NAME, not an id |
| `dashboarddefaults` | `POST /api/dashboarddefaults` | resolution is FQDN-first with IP fallback, so import both keys when the source has them |
| `comstypes` | `communicationtypes` | seeded, not imported. `flask seed reference-data` creates IP/Serial/USB/... and there is no CRUD endpoint |
| `topics` | `POST /api/applications` | `topics` is a near-clone of `applications` and `knowledgebase.appid` points at it, so each topic imports as an Application or the KB links resolve against nothing |
| `appowners` | `POST /api/supportteams/{id}/contacts` | folded into contacts: each `supportteams` row carries one `appownerid`, which becomes ONE contact on that team |
**Model images** import as a plain URL string only. Binary photos are not part
of the import payload; upload them afterwards with
`POST /api/models/<modelid>/image`. Employee photos work the same way
(`POST /api/employees/<sso>/photo`), and only self-hosted mode owns people at
all - in external mode the photo is a read-only pass-through from the HR
database.
## Relationships
`machinerelationships` becomes `POST /api/assets/relationships`, with
`machineid`/`related_machineid` resolved through the legacy-id crosswalk to
`sourceassetid`/`targetassetid`. The legacy type names do not survive as they
are; this is the mapping that was used:
Controls and Master-Slave become **Controls**; Controlled By becomes **Controls**
with source and target reversed; Cluster Member and Contains become **partof**;
Connected To becomes **connectedto**; Dualpath, Backup For and Stored At keep
their own types. Create the types before the links.
## Data quirks worth knowing about
- **`notifications.endtime` carries the sentinel `2099-00-03 09:52:32`**, which
is not a valid date. Drop or clamp it; a straight copy fails the parse.
- **`installedapps` only makes sense for computer-class assets.** Skip rows
whose `machineid` did not route to a computer.
- **`warranties` needs a `vendor`** that the legacy row may not carry. It is
required on create.
- **USB history replays as events, not rows**: create the device
(`POST /api/usb`), then replay each `usbcheckouts` row as a checkout and, if
it was returned, a checkin, carrying `checkouttime`/`checkintime`. Those
overrides are honored in import mode only.
- Any legacy column with no target field (`machines.logicmonitorurl`,
`machines.fqdn`, `printers.printerpin`) becomes a custom field - the recipe is
step 7 of [IMPORT-API.md](IMPORT-API.md#4-order-of-operations).
---
## Tables with no target
These legacy tables have no import target. The dispositions are DECIDED, not
open questions.
- **`dncconfig` and `commconfig`** - intentionally NOT migrated. DNC
communication settings drift constantly, so a one-shot import of stale data
has little value. The plan is a future DNC feature fed live by the GE-Enforce
collector rather than a historical import. When that is built, the expected
ingestion pattern is one attribute at a time across the whole facility (sweep
every machine's baud rate in one pass, then ports, and so on), so the design
should favor per-field fleet-wide updates over per-machine full-record
imports.
- **`compliance`, `compliancescans`** - 0 rows. A future compliance plugin would
own them. Skip.
- **`ednc_installations`, `ednc_logs`** - 0 rows, and they belong to the eDNC
tooling rather than the asset catalog. Skip.
- **`distributiongroups`** (2 rows) - email distribution lists referenced by
`businessunits.distributiongroupid`. No target; skip, or attach as a business
unit custom field.
- **`functionalaccounts`** (7 rows) - service-account concept referenced by
`pctype`/`machinetypes`; no equivalent in the new schema. Skip, or capture as
a computer-type custom field.
- **`skilllevels`** (2 rows) - orphaned lookup (no FK from `machines`). Skip.
## Row-count parity
The method is section 6 of [IMPORT-API.md](IMPORT-API.md). What is specific to
this source is the legacy-side `WHERE` clause, because the fan-out means one
legacy table feeds several targets. Adjust each to match the routing you chose:
| entity | legacy count | new count |
|---|---|---|
| computers | `machines WHERE machinetypeid IN (33,20)` | `GET /api/computers` total |
| machines | `machines WHERE machinetypeid IN (2,4,5,6,7,9,10,11,12,13,14,21,22,24,25,45)` | `GET /api/machines` total |
| network devices | `machines WHERE machinetypeid IN (16,17,18,19,46)` | `GET /api/network` total |
| USB devices | `machines WHERE machinetypeid=44` | `GET /api/usb` total |
| printers | `printers WHERE isactive` | `GET /api/printers` total |
| applications | `applications` | `GET /api/applications?showhidden=true` total |
| knowledge base | `knowledgebase WHERE isactive` | `GET /api/knowledgebase` total |
| relationships | `machinerelationships WHERE isactive` | per-asset `GET /api/assets/{id}/relationships` |
| USB checkouts | `usbcheckouts` | `GET /api/usb/checkouts` |
Vendors, models, business units and the other straight lookups compare
one-to-one and need no clause.

178
docs/INDEX.md Normal file
View File

@@ -0,0 +1,178 @@
<!-- docmeta
purpose: reference
audience: operator, adopter, developer, integrator
status: current
summary: Every task these docs answer, the smallest page set that answers it, and what that costs to read.
loadwith: none
-->
# Documentation index
Find the row that matches what you are about to do, read the pages in the
"Read" column, and skip everything else. That is the whole intent: these docs
are larger than any one person or agent can hold, so the only useful thing an
index can do is tell you which few pages you actually need.
**Lines** is the total size of that row's page set. It is there so you can see
the cost before you pay it. A one-page row is one page on purpose.
**Derived, not authored.** Every row comes from the `docmeta` block at the top
of the page it names - its `purpose`, `status` and `summary` - and the page
itself is the source of truth. So do not improve a row here: fix the `docmeta`
on the page and bring the row into line with it. `tests/test_docs_index.py`
fails when this file and the pages disagree, when a current page is reachable
from no task row, when a row misstates what it costs to read, and when a link
here or in START-HERE points at a file that is not there.
New to the project and not sure which row is yours: [START-HERE](START-HERE.md)
is shorter and picks for you. An agent should read [llms.txt](llms.txt) first.
## Tasks
### Run a site
Docker is the topology this product line ships, and the only one. There is no
Windows installer here and no IIS procedure to follow.
| Task | Read | Lines |
|---|---|---|
| Stand up a new site on Docker at a connected facility | [DEPLOY](DEPLOY.md) | 500 |
| Stand up a site with no internet at all | [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md) | 180 |
| Build the image where npm cannot reach the registry | [DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) | 87 |
| Try the product on a Windows box without building it | [DEV-BUILD-TEST](DEV-BUILD-TEST.md) | 56 |
| Upgrade a Docker or venv site to a newer version | [UPGRADE](UPGRADE.md) | 150 |
| Back up a site, or restore one | [BACKUP-RESTORE](BACKUP-RESTORE.md) | 202 |
| Find a setting, an environment variable or its default | [CONFIG](CONFIG.md) | 356 |
| Decide which plugins this site should carry | [PLUGINS](PLUGINS.md) | 134 |
| Answer whether this site is exposed to a published advisory | [DEPENDENCY-SECURITY](DEPENDENCY-SECURITY.md) | 108 |
| Hand somebody a password without leaving it in a chat | [SECURE-SEND](SECURE-SEND.md) | 157 |
### Fill a site with data
| Task | Read | Lines |
|---|---|---|
| Load starting data from spreadsheets, with no developer | [CSV-IMPORT](CSV-IMPORT.md) | 139 |
| Load starting data from a source database | [IMPORT-API](IMPORT-API.md) + [IMPORT-ADOPTION](IMPORT-ADOPTION.md) | 379 |
| Look up the import wire contract only | [IMPORT-API](IMPORT-API.md) | 283 |
| Load the floor plans, or move every marker onto a redrawn plan | [FLOOR-MAP](FLOOR-MAP.md) | 137 |
| Write a procedure technicians will follow on the floor | [GUIDES](GUIDES.md) | 237 |
| File several devices that share one asset number | [ASSET-COMPOSITION](ASSET-COMPOSITION.md) | 158 |
### Shop-floor PCs
| Task | Read | Lines |
|---|---|---|
| Understand what runs on a bay and which page to open next | [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) | 144 |
| Understand what the collector is before choosing a page | [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md) | 85 |
| Get the shipped reporters running on a fleet, and prove it | [COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md) + [COLLECTOR-AUTH](COLLECTOR-AUTH.md) | 439 |
| Write a client that posts to the collector API | [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) + [COLLECTOR-AUTH](COLLECTOR-AUTH.md) | 691 |
| Decide how a PC authenticates - per-PC credential or shared key | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) | 292 |
| Fix a bay showing the wrong slides | [EVENTSAVER](EVENTSAVER.md) | 144 |
| Install printers from ShopDB, or read the installer endpoints | [PRINTER-INSTALLER](PRINTER-INSTALLER.md) | 140 |
| Point the shop-floor tools at a different site's ShopDB | [ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md) | 290 |
| Explain why a host's backup is listed as stopped | [BACKUP-STALENESS](BACKUP-STALENESS.md) | 104 |
### Build on the platform
| Task | Read | Lines |
|---|---|---|
| Write and install a first plugin | [PLUGIN-QUICKSTART](PLUGIN-QUICKSTART.md) + [PLUGIN-HOOKS](PLUGIN-HOOKS.md) | 1059 |
| Look up one hook, manifest field or shopdb.api import | [PLUGIN-HOOKS](PLUGIN-HOOKS.md) | 866 |
| Check a plugin against the conventions, and the traps that have bitten people | [PLUGIN-GUIDE](PLUGIN-GUIDE.md) + [PLUGIN-HOOKS](PLUGIN-HOOKS.md) | 1060 |
| Ship a plugin from its own git repository | [PLUGIN-EXTERNAL-REPO](PLUGIN-EXTERNAL-REPO.md) + [PLUGIN-HOOKS](PLUGIN-HOOKS.md) | 1152 |
| Choose how tightly to pin a sister site to the contract | [CONTRACT-STABILITY](CONTRACT-STABILITY.md) | 149 |
| Sign a plugin artifact, or make a site verify signatures | [PLUGIN-SIGNING](PLUGIN-SIGNING.md) | 149 |
| Add a backup kind to the backups plugin | [BACKUP-KINDS](BACKUP-KINDS.md) | 52 |
| Set up a dev environment and contribute a change back | [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md) | 390 |
| Call the API: where the generated spec is, which surface owns what, how auth works | [API-REFERENCE](API-REFERENCE.md) | 89 |
| Look up current versions, plugin list or Alembic heads | [PROJECT-MAP](PROJECT-MAP.md) | 94 |
| Find out what has shipped, and what is left before 1.0 | [ROADMAP](ROADMAP.md) | 210 |
| Find which architectural decision governs a question | [adr/README](adr/README.md) | 69 |
| Find the design record behind a shipped feature | [proposals/README](proposals/README.md) | 27 |
**There is no build-along lab on this line.** The five-part tutorial was written
around a plugin this product does not carry, and a tutorial that walks a reader
through a directory they do not have is worse than none. Learn the contract from
PLUGIN-QUICKSTART and PLUGIN-HOOKS, scaffold with `flask plugin scaffold`, and
read `plugins/warranty/` - the smallest bundled plugin that owns tables,
permissions, an asset panel and a dashboard card.
Looking for one endpoint? Do not look in a page. The generated spec is served
at `GET /api/docs` on any running instance and tracked as `docs/openapi.json`;
[llms.txt](llms.txt) is the short version - auth, the response envelope, and
the recipes people get wrong. The API-REFERENCE row above is the pointer to
both, plus the surface and auth orientation a spec cannot give.
### Decisions
One ADR answers one question. Read the one row that matches and stop -
nothing here is a prerequisite for anything else. [adr/README](adr/README.md)
is the authoritative ADR index and the only place that records whether a
decision still holds; the lines below are the pages' own summaries, for
routing.
| Question it settles | Read | Lines |
|---|---|---|
| Why Asset, not the legacy Machine model, is the surface plugin authors may depend on | [ADR-001](adr/ADR-001-asset-as-platform-contract.md) | 200 |
| How the plugin contract is versioned and what core_version a plugin must declare | [ADR-002](adr/ADR-002-plugin-versioning.md) | 68 |
| How a plugin reaches a site: bundled in-tree, or dropped into plugins/ from its own repo | [ADR-003](adr/ADR-003-plugin-distribution.md) | 76 |
| Why every site runs its own instance instead of one multi-tenant deployment | [ADR-004](adr/ADR-004-deployment-topology.md) | 100 |
| How a plugin declares a collector schema so the framework wires the ingest endpoint and auth | [ADR-006](adr/ADR-006-collector-contract.md) | 156 |
| How the product is versioned and released, separately from the plugin contract | [ADR-007](adr/ADR-007-product-versioning-and-releases.md) | 123 |
| Why each plugin owns its own Alembic chain instead of adding tables to the core chain | [ADR-008](adr/ADR-008-plugin-migration-ownership.md) | 131 |
| Why the SPA gates plugin routes, so a disabled plugin URL redirects instead of breaking | [ADR-009](adr/ADR-009-frontend-plugin-gating.md) | 156 |
| The hook contract a plugin uses to contribute UI without hand-editing core Vue files | [ADR-010](adr/ADR-010-frontend-plugin-hooks.md) | 326 |
| Why the equipment domain was renamed to machines and machinetypes became modeltypes | [ADR-011](adr/ADR-011-machines-rename.md) | 75 |
| How a site gets only the plugins it chose, and why curation plus signing is the security model | [ADR-013](adr/ADR-013-plugin-catalog-and-lean-builds.md) | 435 |
| Why a lean site stops creating database tables for plugins it never installed | [ADR-014](adr/ADR-014-schema-lean-per-site.md) | 150 |
| Where a site's own hostnames, paths and defaults are allowed to live, and where they are not | [ADR-015](adr/ADR-015-site-specific-configuration.md) | 156 |
| Why the floor map became buildings and levels instead of one image with stacked floors | [ADR-017](adr/ADR-017-buildings-and-levels.md) | 151 |
| Why metrology instruments are machines, so this product has no measuring-tool asset class | [ADR-018](adr/ADR-018-metrology-instruments-are-machines.md) | 132 |
| Why every endpoint gets its own enrolled secret instead of one fleet-wide shared key | [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md) | 217 |
## Archive - do not follow these
Nothing below is current guidance and nothing above depends on it. A **legacy**
page is still true but superseded: do not start new work from it. A
**historical** page is a finished migration, a design proposal, or a decision
about something this product line no longer carries, kept only as the record of
why something looks the way it does.
They are listed so you can recognise one if you land on it, and so nobody has
to read one to find out it does not apply. Together they are 1,799 lines that
almost nobody needs.
| Page | Status | What it is | Lines |
|---|---|---|---|
| [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) | historical | Record of one classic-ASP shopdb migration: the fan-out routing, the judgement calls and what was left behind. | 163 |
| [ADR-005](adr/ADR-005-equipment-vs-measuringtools.md) | historical | The split of machinery from metrology tools into two plugins. Superseded by ADR-018. | 163 |
| [ADR-012](adr/ADR-012-geenforce-manifest-ownership.md) | historical | Why ShopDB owned GE-Enforce manifests as data. That plugin is not bundled here; ADR-013 and the API-token resource scopes trace back to this decision. | 137 |
| [ADR-016](adr/ADR-016-credential-delivery.md) | historical | The plan for ShopDB-resolved credential targeting. Never built; superseded by ADR-019. | 204 |
| [deployment/PRODUCTION-READINESS](deployment/PRODUCTION-READINESS.md) | historical | The 2026-09-07 assessment of why deployment was hard. Its gap list became the distributed-release item on the roadmap; its recommendation of Windows/IIS was reversed. | 372 |
| [proposals/dashboard-live-fleet](proposals/dashboard-live-fleet.md) | historical | The case for a worst-first dashboard queue. Built; kept as the record of why. | 179 |
| [proposals/device-identity](proposals/device-identity.md) | historical | One durable identity for the device a PC drives. Phase 1 is built; phases 2 and 3 are not started. | 235 |
| [proposals/printer-assignment](proposals/printer-assignment.md) | historical | The design making printers a property of the machine. Shipped, and the shipped behaviour is in PRINTER-INSTALLER. | 346 |
## What the docmeta fields mean
Every page in `docs/` opens with an HTML comment like this one, and this index
is built from those blocks:
```
<!-- docmeta
purpose: howto
audience: operator
status: current
summary: One sentence saying what question this page answers.
loadwith: PAGE-A.md
-->
```
- **purpose** - `tutorial` teaches by doing, `howto` is followed under
pressure, `reference` is looked something up in, `explanation` says why.
- **audience** - `operator`, `adopter`, `developer`, `integrator`.
- **status** - `current`, `legacy` (true but superseded), `historical` (a
record, not guidance).
- **summary** - what this index shows instead of the page.
- **loadwith** - the pages that page is not complete without. Usually `none`.
Where a row above lists two pages, that is where the second one came from.

286
docs/PLUGIN-EXTERNAL-REPO.md Normal file
View File

@@ -0,0 +1,286 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build, pin and CI-test a plugin that lives in your own git repo and drops into a site's plugins/ directory.
loadwith: PLUGIN-HOOKS.md
-->
# Developing a plugin in its own repo
This guide is for a sister GE Aerospace site (or any third party) building a
shopdb-flask plugin in a git repo it owns, outside the framework tree. It is
the "external plugin" path in [ADR-003](adr/ADR-003-plugin-distribution.md):
the framework ships a bundled set, and you drop your own plugin into
`<framework>/plugins/<name>/` by clone, submodule, or symlink. No pip packaging
is required for v1 (pip distribution is deferred to v2 per ADR-003).
> **Windows:** use `venv\Scripts\python` and `$env:FLASK_APP="shopdb"` (not
> `export`). Onboarding: [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md).
If you have not written a plugin before, start with
[PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART.md) and the hook reference in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md). This document only covers the parts that are
different when the plugin lives in its own repo.
## Recommended repo layout
Your repo root holds the plugin directory contents directly, so the whole repo
can be dropped in at `<framework>/plugins/<name>/`. A plugin named `shipping`
in a repo named `sitecode-shipping` looks like this:
```
sitecode-shipping/ # your git repo root == the plugin directory
manifest.json # required: name, version, description, core_version
plugin.py # required: your BasePlugin subclass
__init__.py
models/
__init__.py # exports every model
shipping.py
api/
__init__.py
routes.py # Flask Blueprint returned by get_blueprint()
migrations/ # per-plugin Alembic chain (ADR-008), if you own tables
env.py
script.py.mako
versions/
0001_shipping_baseline.py
tests/ # your own tests (the CI harness runs these)
test_shipping.py
README.md # what it tracks, who maintains it, where to file issues
```
Notes:
- `manifest.json` is the single source of truth for metadata (ADR-002). The
`name` field must match the directory name the site installs it under and
follows the framework naming convention (lowercase concatenated, no
underscores or dashes). Prefix site-specific names with the site code when a
collision across sites is possible, e.g. `sitecode-shipping`
(see [PLUGINS.md](PLUGINS.md) naming policy and ADR-003).
- `migrations/` is only needed if your plugin owns tables. A plugin built
outside the tree never had its tables created by the framework's core chain,
so its `0001` is a REAL baseline that creates them, not a stamp-only anchor.
That distinction is explained once, in [PLUGINS.md](PLUGINS.md#migrations).
- Import core code only through `shopdb.api` and `shopdb.plugins.base`. The
surface, and what happens when you reach past it, are in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#import-surface).
## Local dev workflow
Symlinking lets you edit in your own repo while the framework loads the plugin
live. The loader discovers a symlinked directory the same as a real one.
```bash
# 1. Clone the framework and your plugin repo side by side.
git clone https://github.com/ge-aero/shopdb-flask.git
git clone https://github.com/ge-aero/sitecode-shipping.git
# 2. Symlink your repo into the framework's plugins/ directory.
# The link name is the plugin name from your manifest.json.
cd shopdb-flask
ln -s ../../sitecode-shipping plugins/shipping
# (use an absolute path if you prefer: ln -s "$(pwd)/../sitecode-shipping" plugins/shipping)
#
# Windows: use a directory junction instead of ln -s. In an ADMIN prompt
# (or with Developer Mode on) from the shopdb-flask dir:
# mklink /D plugins\shipping ..\..\sitecode-shipping
# The plugin loader treats a junction the same as a real directory.
# 3. Set up the framework as usual.
python3 -m venv venv
venv/bin/pip install -r requirements-dev.txt
# 4. Install (enable) your plugin.
venv/bin/flask plugin install shipping
# 5. If your plugin owns tables, apply its migrations.
venv/bin/flask db upgrade # core chain
venv/bin/flask plugin upgrade-all # plugin chains, including yours
# 6. Run tests.
venv/bin/python -m pytest tests/test_plugin_contract.py
venv/bin/python -m pytest ../sitecode-shipping/tests
```
Edits in `../sitecode-shipping` are picked up on the next framework restart, because
the symlink points back at your working tree.
### If your plugin owns tables, step 5 needs a change to the framework
This is the one place an external plugin is not self-contained, and it is better
said plainly than discovered at the first migration.
`flask plugin upgrade-all` builds each plugin's metadata from
`PLUGIN_TABLE_OWNERS` in `shopdb/plugins/alembic_template.py`, and raises rather
than guessing when a plugin has no entry:
```
RuntimeError: PLUGIN_TABLE_OWNERS has no entry for plugin 'shipping'.
Update shopdb/plugins/alembic_template.py.
```
So a plugin that owns tables needs its table names registered in that dictionary
in the framework repository. It is deliberate - the registry is what stops one
plugin's migration touching another's tables, and `tests/test_plugin_migrations.py`
tests it - but it does mean a table-owning external plugin is a two-repository
change: yours, plus a one-line addition upstream.
Two ways to live with it:
- **Send the entry upstream.** One line in `PLUGIN_TABLE_OWNERS` plus one in
`EXPECTED_HEAD_REVISION`, and your plugin is a normal citizen from then on.
- **Own no tables.** A plugin that stores nothing of its own - a report, a
dashboard card, a settings page over existing models - has nothing to
register and stays entirely in your repository. More plugins fit this than
expect to.
## core_version pinning for sister sites
The framework is pre-1.0. Under semver, any 0.x minor bump is allowed to break
the contract, and this project uses that latitude (see the history in
[CONTRACT-STABILITY.md](CONTRACT-STABILITY.md)). So pin a TIGHT range in your
`manifest.json`, per ADR-002 (pip-style `>=,<`), admitting only the contract
minor you tested against and not the whole 0.x line.
Read the current contract version from [PROJECT-MAP.md](PROJECT-MAP.md), which
is generated from `shopdb/__init__.py` - a version typed into this page is
wrong within a fortnight, and a plugin pinned to a stale one is refused at
startup. Take that minor and the next one up: if the map says `0.31.0`, your
range is `>=0.31.0,<0.32.0`:
```json
{
"name": "shipping",
"version": "1.0.0",
"description": "Tracks shipping-station scanners and label printers",
"core_version": ">=<the map's minor>,<<the next minor>",
"dependencies": []
}
```
Do NOT pin the loose `>=0.2.0,<1.0.0` default that `PluginMeta` falls back to.
That default exists so bundled plugins keep loading across minor bumps; an
external plugin should be deliberate and re-test before widening its range.
What happens at load time on a mismatch (ADR-002):
| Environment | Behavior on core_version mismatch |
|-------------|-----------------------------------|
| dev / test | The loader re-raises. Startup fails loud so you notice immediately. |
| production | The loader logs an error, marks the plugin incompatible, and excludes it from registration. The rest of the app still starts. |
When you move a site to a newer framework, bump your `core_version` upper bound
only after the harness below passes against the new ref.
## CI recipe
`scripts/test-external-plugin.sh` (in the framework repo) stands up a throwaway
framework at a pinned ref, drops your plugin in as a symlink, and runs the
framework contract tests plus your own `tests/`. It has two modes:
- **CI / remote** (default): clones the framework at `FRAMEWORK_REF` and builds
a fresh venv. Needs network access to git and PyPI.
- **Local / offline**: set `LOCAL_FRAMEWORK` to a framework checkout on disk.
The script exports that checkout at HEAD with `git archive` and reuses its
existing venv, so it runs with no internet. Useful for air-gapped verification.
```bash
# CI: test the plugin in the current repo against a pinned tag
PLUGIN_DIR=. FRAMEWORK_REF=v0.5.0 scripts/test-external-plugin.sh
# Offline: test against a framework checkout already on disk
LOCAL_FRAMEWORK=/opt/shopdb-flask PLUGIN_DIR=. scripts/test-external-plugin.sh
```
Its header comment documents every input (`PLUGIN_DIR`, `FRAMEWORK_REF`,
`FRAMEWORK_URL`, `LOCAL_FRAMEWORK`); read it in the file rather than from a copy
on a wiki page, because a copied script is one that stops matching the script
that runs. In order it resolves the plugin name from `manifest.json`, stands up
the throwaway framework, symlinks your plugin into its `plugins/`, clears
`instance/plugins.json`, runs the framework's contract tests, then runs your
plugin's own `tests/` if it has any. Nonzero exit means the plugin is not
compatible with that framework ref.
### What the harness does and does not check
The symlinked plugin is discovered and loaded by the plugin loader when the app
starts under test. That validates the things that break a real install: the
manifest parses, the `core_version` range admits the framework's
`__contract_version__` (an out-of-range plugin makes startup fail loud, so the
contract test run errors and the script exits nonzero), models expose
`__tablename__`, and hooks return the right shapes.
The import-surface scan (`test_plugins_only_import_contract_surface`) covers
symlinked plugins too: the scanner resolves each plugin directory before
walking it, because `Path.rglob` alone does not descend symlinks (pinned by
`test_import_scan_covers_symlinked_plugins`). You can additionally keep an
equivalent import-surface assertion in your own `tests/`, so violations fail
in your repo's CI even when run without the framework harness. A minimal
version:
```python
# tests/test_import_surface.py
import re
from pathlib import Path
ALLOWED = ('shopdb.api', 'shopdb.plugins.base')
IMPORT_RE = re.compile(r'^\s*(?:from (shopdb[\w.]*) import|import (shopdb[\w.]*))', re.MULTILINE)
def test_only_contract_surface_imports():
root = Path(__file__).resolve().parent.parent
bad = []
for path in root.rglob('*.py'):
if '__pycache__' in path.parts or 'migrations' in path.parts:
continue
for m in IMPORT_RE.finditer(path.read_text()):
mod = m.group(1) or m.group(2)
if mod and not any(mod == a or mod.startswith(a + '.') for a in ALLOWED):
bad.append(f'{path.name}: {mod}')
assert not bad, 'core imports outside shopdb.api / shopdb.plugins.base: ' + '; '.join(bad)
```
## GitHub Actions example
For a plugin repo hosted on GitHub, a workflow calling the harness against a
pinned framework tag (config only, adjust URLs and ref to your setup):
```yaml
name: plugin-contract
on:
push:
pull_request:
jobs:
contract:
runs-on: ubuntu-latest
env:
FRAMEWORK_REF: v0.5.0
FRAMEWORK_URL: https://github.com/ge-aero/shopdb-flask.git
steps:
- name: Check out the plugin
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Fetch the harness from the framework
run: |
git clone --depth 1 --branch "$FRAMEWORK_REF" "$FRAMEWORK_URL" /tmp/framework
- name: Run the contract harness against this plugin
run: |
PLUGIN_DIR="$GITHUB_WORKSPACE" \
FRAMEWORK_REF="$FRAMEWORK_REF" \
FRAMEWORK_URL="$FRAMEWORK_URL" \
bash /tmp/framework/scripts/test-external-plugin.sh
```
## See also
- [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART.md) - generate and install a plugin fast
- [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md) - the full hook and import-surface reference
- [CONTRACT-STABILITY.md](CONTRACT-STABILITY.md) - what is settled vs still churning before 1.0
- [ADR-002](adr/ADR-002-plugin-versioning.md) - contract versioning and core_version ranges
- [ADR-003](adr/ADR-003-plugin-distribution.md) - the bundled vs external distribution model
- [ADR-008](adr/ADR-008-plugin-migration-ownership.md) - per-plugin migration ownership

194
docs/PLUGIN-GUIDE.md Normal file
View File

@@ -0,0 +1,194 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: The conventions and traps a real plugin proves out, and the checklist to walk before calling one done.
loadwith: PLUGIN-HOOKS.md
-->
# Building a ShopDB plugin: conventions and traps
This page is the judgement layer. [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md) says what
each hook TAKES AND RETURNS; this says what a plugin that works actually DOES
with them - the conventions every bundled plugin follows, the traps that have
cost someone an afternoon, and the checklist to walk before calling a plugin
done. It defines nothing: where it shows a shape, the contract page is right.
Elsewhere: [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART.md) to get a plugin running
today; [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md) to ship one from its
own repo.
> **Windows:** use `venv\Scripts\python` and `$env:FLASK_APP="shopdb"` (not
> `export`). Onboarding: [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md).
**Read `plugins/warranty/` alongside this page.** It is the reference
implementation: own tables, own permissions, a derived status, an asset panel, a
dashboard widget and reports - most of the contract exercised correctly, and
small enough to read in one sitting. `plugins/network/` is the cleanest
asset-only plugin, `plugins/backups/` the one to copy for a revision ledger fed
by the collector, and `plugins/knowledgebase/` for a plugin that owns no assets
at all.
---
## Conventions the contract does not state
**Identity lives on the `Asset`, not in your table** (ADR-001). The asset
carries the number, the serial, the location and the status; your extension
table carries only what is specific to your domain, keyed one-to-one by
`assetid` with `unique=True` and `ondelete='CASCADE'`. A plugin that adds its
own "asset number" column has forked the platform contract.
**Derived state is computed at read time, never stored.** Calibration status,
warranty coverage and low-stock are all derived from a date or a count when
they are read. A stored status column goes stale the moment a due date passes
with no write, and then the list badge, the detail badge and the report
disagree. Deriving costs one thing, and it is worth knowing before you choose:
you cannot filter it in SQL. `warranty` applies its coverage filter to the built
rows instead of as a `WHERE`, which is fine for the number of warranties a site
holds and would need rethinking at the size of the PC estate.
**One payload, one shape.** Create and update take the asset core fields and
the extension fields in ONE body, and every response returns the asset's
`to_dict()` with the extension nested under the plugin's key, so GET, POST and
PUT are the same shape to the frontend. `BaseModel.to_dict` iso-formats
`datetime` but not plain `date`, so a model with `Date` columns formats them
itself (`YYYY-MM-DD`) rather than shipping whatever the driver returned.
**Reads are `jwt_required(optional=True)`; writes require a permission.**
Anonymous kiosks and signed-out staff read the catalog; any state change needs
a logged-in user holding the right permission, declared by your own
[`get_permissions`](PLUGIN-HOOKS.md#hook-get_permissions) so core never edits
its catalog to accommodate a plugin. The `admin` role bypasses every check, so
an admin can drive the plugin before anyone grants the granular permissions.
Every write also writes an `AuditLog` row.
**Use the framework response helpers** (`success_response`, `error_response`
with `ErrorCodes`, `paginated_response`, `get_pagination_params`,
`paginate_query`) so every plugin's API looks the same to the frontend.
**Guard a delete that something still references.** Deleting a subtype that
rows still point at is refused with a 409 and a count, not allowed to blank out
existing rows.
**Seed reference data in `on_install`, idempotently.** Each `_ensure_*` helper
checks for existence first, so a reinstall does not duplicate rows. Do NOT seed
`Setting` rows this way - declare them from
[`get_settings_defaults`](PLUGIN-HOOKS.md#hook-get_settings_defaults), which
also reaches sites that installed an earlier version.
**Names are lowercase and concatenated** - `nextcalibrationdate`, not
`next_calibration_date`. `scripts/check-naming-and-style.sh` fails the build
otherwise. The rules are in `CONTRIBUTING.md`.
**Skipping a hook is a decision worth recording.** `warranty` declares no
`get_config_schema` because warranties are entered by hand: no external system,
no endpoint, no API key. The collector hooks are skipped for the same reason. A
reviewer asks about both, so say why in the plugin class rather than leaving a
silent gap.
---
## Traps
**`create_plugin_tables` cannot render a foreign key to a core table.** It
builds a per-plugin `MetaData` filtered to the plugin's own tables, so an FK to
`assets` has nothing to resolve against and Alembic raises
`NoReferencedTableError` at compile time. If your tables reference core tables,
hand the DDL to `op.create_table` (the same shape autogenerate produces); the
helper is only for plugins whose tables stand alone. This is a candidate
framework fix - teaching `_get_plugin_metadata` to pull in FK-referenced core
tables - not a rule you should design around.
**Two registries must know about your plugin**, and the failures are far apart:
`PLUGIN_TABLE_OWNERS` in `shopdb/plugins/alembic_template.py` (missing entry =
`flask plugin upgrade-all` raises rather than guessing) and
`EXPECTED_HEAD_REVISION` in `tests/test_plugin_migrations.py` (missing entry =
a `KeyError` from the test suite). A plugin that later adds `0002` moves its
expected-head line to the new revision. Whether your `0001` creates tables or
is a stamp-only anchor is decided in [PLUGINS.md](PLUGINS.md#migrations).
**Charset comes from the connection.** A plain `CREATE TABLE` inherits the
database default, which on the documented utf8mb4 database is what you want and
matches how the core chain creates its tables. Check any unique column against
the InnoDB index prefix limit before you widen it (utf8mb4 `VARCHAR(255)` is
1020 bytes, over the 767-byte cap on old MySQL - see the setup-problems table
in [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md)).
**Enabling a plugin does not serve it - restart the API.** Blueprints are
registered at app startup and Flask refuses to register one after the app has
served a request. `flask plugin enable <name>` flips a registry flag; until the
service restarts, `/api/<name>` is still a 404.
**Never hardcode an AssetType id.** Seeded ids are not stable across sites.
Resolve the type at runtime from `GET /api/assets/types` by its `assettype`
key. The machines form hardcoded one and has since moved to the lookup.
**`default_enabled: false` means the shared test app never registers your
blueprint.** The plugin's test module builds its own app, registers the
blueprint, runs `on_install` seeding, and restores the process-wide
`plugin_manager` singleton afterwards - the same snapshot/restore pattern
`tests/test_plugin_migrations.py` uses. The parametrized contract and migration
tests still pick the plugin up from the filesystem on their own.
---
## The three core files a bundled plugin still edits
Everything the sidebar, the Reports hub, the settings rail, asset panels and
the map show is data-driven off hooks, so a plugin contributes UI without core
knowing it exists. Three edits remain, and a review expects exactly these:
1. `PLUGIN_TABLE_OWNERS` - the migration registry above.
2. `frontend/src/api/index.js` - ADD your `<name>Api` block, mirroring
`machinesApi`. Do not reorganize the file. (The scaffold generates the block
for you and ships the views with an inline client so they run before you
paste it.)
3. `frontend/src/views/AppLayout.vue` - one `iconMap` entry mapping the icon
string your hooks return to a Lucide component. ADR-009 wants this replaced
by a registration API eventually.
Your route module carries `meta.plugin` on every route so the router gates it
when the plugin is absent; the staging rules are in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#frontend). Follow the app's auth pattern:
list and detail public, the form `requiresAuth`, the settings subtype page
`requiresAuth + requiresAdmin`.
Three core features compose onto your pages for free, and there is nothing to
wire: `CustomFieldsSection` / `CustomFieldsInputs` (call
`customFieldsRef.value.save(assetId)` after your own save), `WarrantyPanel`
(asset-general, works on any type), and the floor map, which draws any typed
asset that has a position (ADR-001, ADR-017).
---
## End checklist
Before you call a plugin done:
- [ ] `manifest.json`: name, version, `core_version` pinned to the lowest
contract you need and capped below the next major, `api_prefix`,
`default_enabled`.
- [ ] Models follow the naming convention; identity stays on `Asset`.
- [ ] Derived state (if any) is computed at read time, never stored.
- [ ] Tables in `PLUGIN_TABLE_OWNERS`; expected head in
`tests/test_plugin_migrations.py`.
- [ ] Migration chain correct for your side of the cutover; hand-written
`op.create_table` if you have FKs to core tables.
- [ ] Imports only from `shopdb.api` and `shopdb.plugins.base` (the contract
test scans for this).
- [ ] Blueprint: jwt-optional reads, permission-gated writes, framework
response and pagination helpers, audit logs on writes.
- [ ] Permissions declared from `get_permissions`; install/enable seeds them.
- [ ] `on_install` seeds the asset type and reference data, idempotently;
settings declared from `get_settings_defaults`.
- [ ] Hooks: navigation, reports and models implemented; config schema and
collector implemented or consciously skipped with a reason.
- [ ] Frontend: route module with `meta.plugin`, API client addition, views
mirroring the master templates, settings subtype page, one `iconMap`
entry.
- [ ] Custom fields and warranty panel composed onto the detail and form pages.
- [ ] Tests: unit + API, self-contained app if `default_enabled` is false; the
three gates green ([DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md)).
- [ ] Install / enable / RESTART verified live, and a demo record renders on
every page.

866
docs/PLUGIN-HOOKS.md Normal file
View File

@@ -0,0 +1,866 @@
<!-- docmeta
purpose: reference
audience: developer, integrator
status: current
summary: The plugin contract itself: every BasePlugin hook, the manifest fields, and the shopdb.api import surface.
loadwith: none
-->
# Plugin Hooks Reference
**This page is the contract, and it is the only page that defines it.** Every
hook a plugin may implement, every manifest field, and every name a plugin may
import is specified here. PLUGIN-QUICKSTART and PLUGIN-GUIDE TEACH - they show
one plugin making one set of choices, and they link back here for the shape of
anything they use. If one of them disagrees with
this page, this page is right and the other page is a defect: fix it here and
delete the copy there.
Plugin authors implement `BasePlugin` and override the hooks they care about.
Required hooks must be implemented; optional hooks have defaults and can be
left alone. Every section carries a stable anchor (`#hook-get_reports` and so
on) so another page links to exactly the hook it means.
The contract is locked in [ADR-001](../docs/adr/ADR-001-asset-as-platform-contract.md) and versioned per [ADR-002](../docs/adr/ADR-002-plugin-versioning.md).
## Hook index
Jump straight to the one you need; you should never have to read this page end
to end.
| Hook | | Contributes |
|---|---|---|
| [`meta`](#hook-meta) | required | The plugin's identity, built from `manifest.json` |
| [`get_blueprint`](#hook-get_blueprint) | required | The Flask blueprint mounted at `api_prefix` |
| [`get_models`](#hook-get_models) | required | The SQLAlchemy models the plugin owns |
| [`init_app`](#hook-init_app) | optional | Anything else at startup: schemas, a second blueprint |
| [`get_cli_commands`](#hook-get_cli_commands) | optional | Click commands on the Flask CLI |
| [`get_services`](#hook-get_services) | optional | Named services another plugin can obtain |
| [`get_dashboard_widgets`](#hook-get_dashboard_widgets) | optional | Cards on the dashboard queue |
| [`get_navigation_items`](#hook-get_navigation_items) | optional | The sidebar entry |
| [`get_reports`](#hook-get_reports) | optional | Cards on the Reports hub |
| [`get_permissions`](#hook-get_permissions) | optional | The RBAC permissions the plugin's routes enforce |
| [`get_guest_grantable_permissions`](#hook-get_guest_grantable_permissions) | optional | Which of its WRITES a site may grant to `guest` |
| [`get_settings_cards`](#hook-get_settings_cards) | optional | Cards in the settings rail |
| [`get_asset_panels`](#hook-get_asset_panels) | optional | Panels on asset detail pages |
| [`get_map_overlays`](#hook-get_map_overlays) | optional | Badges/rings and a legend entry on the floor map |
| [`get_asset_presentation`](#hook-get_asset_presentation) | optional | Icon and detail route for an asset type it owns |
| [`get_provisioning_note`](#hook-get_provisioning_note) | optional | What the setup wizard warns a site it will create |
| [`get_config_schema`](#hook-get_config_schema) | optional | Config fields the setup wizard prompts for |
| [`get_settings_defaults`](#hook-get_settings_defaults) | optional | The `Setting` rows the plugin owns |
| [`get_collector_schema`](#hook-get_collector_schema) | optional | JSON Schema for an external collector push |
| [`apply_collector_payload`](#hook-apply_collector_payload) | conditional | The upsert behind that collector |
| [`on_install` / `on_uninstall` / `on_enable` / `on_disable`](#lifecycle-hooks) | optional | State-transition work |
| [`get_setting` / `set_setting`](#helpers) | provided | Plugin-scoped settings (you call these, not override) |
The rest of the contract: [manifest fields](#manifest) -
[frontend contribution](#frontend) - [the `shopdb.api` import
surface](#import-surface) - [helpers](#helpers) - [removed
hooks](#removed-hooks) - [changing the contract](#versioning) - [the bulk-import
journal](#import-journal).
**One rule holds for every list-returning hook below.** The endpoint that
consumes it skips disabled plugins, and isolates a plugin that raises: logged
and dropped in production, re-raised in dev and test so you see it.
<a id="contract-version"></a>
<a id="manifest"></a>
## Plugin metadata
Each plugin ships a `manifest.json`, the single source of truth for its
identity; the dataclass `PluginMeta` is constructed from it.
```json
{
"name": "computers",
"version": "1.0.0",
"description": "Tracks shop-floor PCs and engineering workstations",
"author": "shopdb-flask",
"dependencies": [],
"core_version": ">=0.1.0,<1.0.0",
"api_prefix": "/api/computers"
}
```
| Field | Required | Notes |
|-------|----------|-------|
| `name` | Yes | Lowercase concatenated, no underscores or dashes |
| `version` | Yes | Plugin's own semver |
| `description` | Yes | One sentence |
| `dependencies` | No | List of plugin names that must load first |
| `core_version` | Yes | Range of framework `__contract_version__` this plugin supports |
| `api_prefix` | No | Defaults to `/api/<name>` |
| `display_name` | No | Human label for UIs; defaults to the title-cased `name` |
| `default_enabled` | No | `false` means installing does not turn the plugin on |
`core_version` is checked at load time and a plugin outside the range is
refused. The framework declares its own version in `shopdb/__init__.py` as
`__contract_version__`; the current value is in
[PROJECT-MAP.md](PROJECT-MAP.md), which is generated from the code. This page
does not restate it, because a version copied into prose is stale within a
fortnight and a plugin pinned against a stale one is refused at startup. How
tight to pin, and what happens on a mismatch, are in
[PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md).
<a id="required-hooks"></a>
## Required hooks
<a id="hook-meta"></a>
### `meta` -> `PluginMeta`
Returns the plugin's metadata. Read it from `manifest.json` rather than typing
the values twice - the scaffold (`flask plugin new`) writes this for you:
```python
from shopdb.plugins.base import BasePlugin, PluginMeta
class ComputersPlugin(BasePlugin):
@property
def meta(self) -> PluginMeta:
return PluginMeta(
name=self._manifest['name'],
version=self._manifest['version'],
description=self._manifest['description'],
dependencies=self._manifest.get('dependencies', []),
core_version=self._manifest.get('core_version', '>=0.1.0'),
api_prefix=self._manifest.get('api_prefix'),
)
```
<a id="hook-get_blueprint"></a>
### `get_blueprint() -> Optional[Blueprint]`
Returns a Flask Blueprint with the plugin's API routes, or `None` if the plugin has no HTTP routes. The loader registers the blueprint at the `api_prefix` from the manifest.
```python
from flask import Blueprint
from .api import computers_bp
class ComputersPlugin(BasePlugin):
def get_blueprint(self):
return computers_bp
```
<a id="hook-get_models"></a>
### `get_models() -> List[Type]`
Returns the SQLAlchemy model classes the plugin defines. Used by the migration runner and admin tooling.
```python
from .models import Computer, ComputerSoftware
class ComputersPlugin(BasePlugin):
def get_models(self):
return [Computer, ComputerSoftware]
```
<a id="optional-hooks"></a>
## Optional hooks
<a id="hook-init_app"></a>
### `init_app(app, db) -> None`
Custom initialization. Called by the loader after the blueprint is registered and models are known. Use for Marshmallow schema registration, Caching configuration, secondary blueprint registration, or anything else the plugin needs.
```python
class PrintersPlugin(BasePlugin):
def init_app(self, app, db):
from .api import printers_legacy_bp
app.register_blueprint(printers_legacy_bp, url_prefix='/api/printers/legacy')
```
<a id="hook-get_cli_commands"></a>
### `get_cli_commands() -> List`
Returns a list of Click commands or command groups to register on the Flask CLI.
```python
import click
@click.group()
def computers_cli():
pass
@computers_cli.command()
def reset_computers():
"""Reset all computer status flags."""
...
class ComputersPlugin(BasePlugin):
def get_cli_commands(self):
return [computers_cli]
```
<a id="hook-get_services"></a>
### `get_services() -> Dict[str, Type]`
Returns a dict of service-name to service-class. Another plugin obtains one via
`plugin_manager.get_service('<name>')`, which searches enabled plugins and
returns the registered class/factory (or None).
```python
from .services import ZabbixService
class PrintersPlugin(BasePlugin):
def get_services(self):
return {'zabbix': ZabbixService}
```
<a id="hook-get_dashboard_widgets"></a>
### `get_dashboard_widgets() -> List[Dict]`
Returns dashboard card definitions for the home page.
A card declares DATA AND SHAPE, never a component name. Core owns a small set of
generic renderers and draws the card; the plugin says what to show, where it
comes from, and how to link it.
**A card using the pre-0.19.0 shape (`'component': 'SomeWidget.vue'`) is
ignored and renders nothing** - if a card of yours is silently absent, that is
why. The reasoning is in the 0.19.0 row of
[CONTRACT-STABILITY.md](CONTRACT-STABILITY.md).
```python
class WarrantyPlugin(BasePlugin):
def get_dashboard_widgets(self):
return [{
'id': 'warranty-expiring', # stable, unique across plugins
'title': 'Warranties expiring',
'endpoint': '/api/warranty/dashboard/expiring',
'render': 'exceptions', # a core renderer, not a component
'severity': 'info', # orders cards on the page
'permission': 'warranty.view', # hidden without it
'empty': 'hide', # say nothing when there is nothing
'position': 50,
'viewall': '/reports/warranty', # optional link behind the heading
'map': {'title': 'hostname', 'detail': 'state'},
}]
```
`empty: 'hide'` is not cosmetic. A card that reports "nothing wrong" every day
teaches people to stop reading the page.
Consumed by `GET /api/dashboard/widgets`, which merges every enabled plugin's
widgets sorted by `position`.
<a id="hook-get_navigation_items"></a>
### `get_navigation_items() -> List[Dict]`
Returns navigation menu items. A plugin owns its own sidebar entry here, so it
appears when the plugin is installed and disappears when it is not (including in
a lean per-site build that omits the plugin).
```python
class ComputersPlugin(BasePlugin):
def get_navigation_items(self):
return [{
'name': 'Computers',
'icon': 'desktop',
'route': '/computers',
'position': 10,
}]
```
**Placement.** `position` (int) sets both the sort order and which section the
item lands in - the core sidebar (`AppLayout.vue:buildNavItems`) assigns section
headers by position range:
| `position` | section |
|-----------|---------|
| `< 10` | top, above any header (Dashboard is 0, Map is 4) |
| `10-29` | **Assets** |
| `30-49` | **Information** |
| `>= 50` | trailing, below Information |
Lower number sorts higher within a section. An explicit `'section':
'information'` forces the Information group regardless of position. Only these
two named sections exist; a new section needs a core edit to `buildNavItems`.
The **Displays** group (kiosk/TV links) is hardcoded in `AppLayout.vue`, not
plugin-driven.
`icon` is a string key mapped to a Lucide component core-side (same idea as
`get_settings_cards`); an unknown key renders with no icon.
<a id="hook-get_reports"></a>
### `get_reports() -> List[Dict]`
Returns report card definitions for the Reports hub. Added in contract 0.6.0.
Each entry has `id`, `name`, `description`, `category`, plus EXACTLY ONE of
`route` (a frontend path for a dedicated report page) or `endpoint` (an API
endpoint the hub renders inline).
```python
class WarrantyPlugin(BasePlugin):
def get_reports(self):
return [{
'id': 'warranty',
'name': 'Warranty Report',
'description': 'Assets bucketed by coverage: expired, expiring soon, active',
'category': 'warranty',
'route': '/reports/warranty',
}]
```
Consumed by `GET /api/reports`, which merges plugin cards after the static core
reports; the frontend groups them by category.
<a id="hook-get_permissions"></a>
### `get_permissions() -> List`
Returns the RBAC permissions this plugin owns. Added in contract 0.10.0. A
plugin declares the permission names its own routes enforce via
`require_permission`, instead of core accumulating every plugin's permissions in
one catalog (plugin-is-the-product).
Each entry is a `(name, description, category)` tuple, matching the core
permission catalog shape (dicts with those keys are also accepted). Names follow
the naming convention (lowercase dotted, e.g. `machines.edit`).
```python
class MachinesPlugin(BasePlugin):
def get_permissions(self):
return [
('machines.view', 'View machines', 'machines'),
('machines.create', 'Create machines', 'machines'),
('machines.edit', 'Edit machines', 'machines'),
('machines.delete', 'Delete machines', 'machines'),
]
```
Consumed by the core helper `full_permission_catalog()` (core permissions plus
every ENABLED plugin's `get_permissions()`), which backs three consumers:
- `flask seed permissions` seeds the full catalog.
- The role-management grid (`GET /api/users/permissions`) lists it, grouped by
category.
- API-token scope validation (`ApiToken.unknown_scope_names`) accepts a plugin
permission as a scope only while that plugin is enabled.
Plugin install and enable also seed the plugin's own permissions idempotently,
so enabling a fresh plugin creates its `Permission` rows without a separate seed
pass.
Disabled-plugin edge case: a disabled plugin is skipped by the catalog, so its
permissions are no longer offered for new scope grants or new role assignments.
The `Permission` ROWS already in the database are NOT deleted, so roles that
already reference them keep working until an admin edits the role. A broken
plugin is isolated in prod and re-raised in dev/test.
<a id="hook-get_guest_grantable_permissions"></a>
### `get_guest_grantable_permissions() -> List[str]`
Names the WRITE permissions this plugin allows a site to grant to the `guest`
role. Added in contract 0.21.0.
Anyone with no account holds the `guest` role. It is seeded with nothing, and a
site opens a door by granting it a permission in the role grid - which replaced
the per-feature booleans (`securesend_allow_guests`, one per kiosk) that used to
answer the same question one feature at a time.
Any `.view` or `.read` permission is grantable without a plugin saying anything.
This hook is only for the exception: a write that is genuinely meant to work for
somebody who never signs in. Naming one here does not open it - it makes the
checkbox available, and a site still has to tick it.
```python
class ToolsPlugin(BasePlugin):
def get_guest_grantable_permissions(self):
# Sending a password to a vendor: the recipient has no account here,
# and often neither does the sender. The unguessable token is the
# authorisation and the link is one-view.
return ['tools.secrets']
```
The bar for naming one: could this action be attributed to nobody, forever, and
still be defensible in the audit log? A guest write records no creator - the
trail says when and from where, never who. Return `[]` (the default) and none of
this plugin's writes can ever be made anonymous; anything not named is refused
by the API when an admin tries to grant it, not merely warned about.
Consumed by `guest_grantable_permissions()`, which backs the role API's guard on
the guest role and the greyed-out checkboxes in the role grid. A plugin may only
bless permissions that already exist in the catalog, so a typo widens nothing.
<a id="hook-get_settings_cards"></a>
### `get_settings_cards() -> List[Dict]`
Returns settings-catalog card definitions. Added in contract 0.7.0 (ADR-010).
Each card is merged into the settings rail and landing overview without the
plugin hand-editing the core `settingsNav.js` catalog. `icon` is a string key
mapped to a Lucide component core-side, exactly like `get_navigation_items`.
```python
class PrintersPlugin(BasePlugin):
def get_settings_cards(self):
return [{
'group': 'Printers', # rail group title (created if new)
'to': '/settings/printer-alerts',
'icon': 'bell', # string key, mapped core-side
'title': 'Low-Toner Alerts',
'description': 'Who gets warning and critical toner emails',
'position': 48, # order within the group
}]
```
Consumed by `GET /api/pluginui/settings-cards`, which merges enabled plugins'
cards into the core catalog.
<a id="hook-get_asset_panels"></a>
### `get_asset_panels() -> List[Dict]`
Returns asset-detail extension-panel definitions. Added in contract 0.7.0
(ADR-010). A generic core `AssetPanel` component renders each panel on the
matching detail pages, fetching the panel's `endpoint`. This replaces
hand-composing a plugin panel component into each detail view.
```python
class WarrantyPlugin(BasePlugin):
def get_asset_panels(self):
return [{
'id': 'warranty',
'title': 'Warranty',
'assettypes': ['*'], # detail pages it appears on; ['*'] = all
'endpoint': '/api/warranty/asset/{assetid}',
'render': 'table', # 'keyvalue' | 'table' | 'badge'
'position': 30,
}]
```
Consumed by `GET /api/pluginui/asset-panels?assetid=<id>`, which returns the
panels whose `assettypes` match that asset's type. A panel that needs bespoke UI
(a chart) is out of scope for this data-only hook.
<a id="hook-get_map_overlays"></a>
### `get_map_overlays() -> List[Dict]`
Returns shop-floor map overlay/decoration definitions. Added in contract 0.7.0
(ADR-010). The map stays data-driven off asset types + positions; an overlay
adds decoration data (a badge or ring) plus an optional legend entry, with no
plugin-side map code.
**No bundled plugin implements this hook on this line,** so the example below is
written out rather than pointed at. `GET /api/pluginui/map-overlays` returns an
empty list until a plugin declares one, which is the correct behaviour and not a
fault. Take the shape from here and from
[ADR-010](adr/ADR-010-frontend-plugin-hooks.md).
```python
class CamerasPlugin(BasePlugin):
def get_map_overlays(self):
return [{
'id': 'camera-offline',
'label': 'Camera offline', # legend label
'endpoint': '/api/cameras/map-overlay', # -> [{assetid, color, label}]
'style': 'badge', # 'badge' | 'ring'
'legend': True,
}]
```
Consumed by `GET /api/pluginui/map-overlays`.
An overlay keys on `assetid` and never on coordinates, so it needs to know
nothing about buildings or levels (ADR-017): it decorates whichever marker that
asset already has, on whatever level that marker is on. Do not return `mapx` /
`mapy` / `levelid` from an overlay endpoint - a position belongs to the asset,
and a second copy of it in an overlay is one that can disagree.
<a id="hook-get_asset_presentation"></a>
### `get_asset_presentation() -> List[Dict]`
Returns asset-type presentation/routing definitions. Added in contract 0.7.0
(ADR-010). Declares how a plugin-owned asset type renders in global-search rows
and cross-links (which icon, which detail route), so core never hardcodes a
plugin's route or icon.
**No bundled plugin implements this hook on this line either,** so this example
is also written out. `GET /api/pluginui/asset-presentation` returns an empty
list until a plugin declares one; core then falls back to its own icon and
route for the asset type.
```python
class CamerasPlugin(BasePlugin):
def get_asset_presentation(self):
return [{
'assettype': 'camera', # AssetType.assettype key the plugin owns
'icon': 'video',
'label': 'Camera',
'route': '/cameras/{assetid}',
}]
```
Consumed by `GET /api/pluginui/asset-presentation`.
<a id="hook-get_provisioning_note"></a>
### `get_provisioning_note() -> Optional[Dict]`
Transparency note the setup wizard shows the moment a site checks this plugin
during setup. Return `None` (the default) for plugins that need no special
setup. Plugins that create extra tables beyond their asset-extension table
return:
No bundled plugin declares one on this line; the shape is:
```python
class CamerasPlugin(BasePlugin):
def get_provisioning_note(self):
return {
'tables': ['camerastreams'],
'note': 'Creates a stream-credential table in the shopdb database.',
'docs': 'plugins/cameras/README.md',
}
```
<a id="hook-get_config_schema"></a>
### `get_config_schema() -> List[Dict]`
Declares the config fields this plugin needs, so the setup wizard can prompt
for them. Return `[]` (the default) if the plugin needs no configuration.
Each field is a dict:
| Key | Meaning |
|-----|---------|
| `key` | the Setting key (non-secret) it maps to |
| `label` | human label shown in the wizard |
| `type` | `'text'` / `'number'` / `'password'` |
| `secret` | `True` for credentials; NOT stored in the DB - the wizard emits an `.env` line for the operator instead |
| `envvar` | (secret only) the `.env` variable name to emit |
| `default` | optional placeholder |
| `help` | optional hint |
```python
class PrintersPlugin(BasePlugin):
def get_config_schema(self):
return [
{'key': 'zabbix_url', 'label': 'Zabbix URL', 'type': 'text',
'help': 'Base URL of the Zabbix server for supply lookups'},
{'key': 'zabbix_token', 'label': 'Zabbix API token', 'type': 'password',
'secret': True, 'envvar': 'ZABBIX_TOKEN'},
]
```
<a id="hook-get_settings_defaults"></a>
### `get_settings_defaults() -> List[Dict]` (0.16.0)
Declares the `Setting` rows this plugin owns. Return `[]` (the default) if it
owns none. Each entry is a dict:
| Key | Meaning |
|-----|---------|
| `key` | the Setting key |
| `value` | default value in string form |
| `valuetype` | `'string'` / `'boolean'` / `'integer'` / `'json'` |
| `category` | grouping the plugin's settings page filters on |
| `description` | what the setting does |
| `public` | `True` if an unauthenticated caller may read it; default `False` |
```python
class BackupsPlugin(BasePlugin):
def get_settings_defaults(self):
return [
{'key': 'backups_intervalhours', 'value': '24', 'valuetype': 'integer',
'category': 'backups', 'public': True,
'description': 'Minimum hours between collection attempts on a PC'},
]
```
`public: True` there is load-bearing, not tidiness: the collecting script reads
that value from `/api/settings/public` BEFORE it has any credential. Left
private, the setting looked configurable and was not.
The framework seeds declared keys at install, at enable, and on every
`flask plugin upgrade-all`, so a key added in a later plugin version reaches a
site that installed an earlier one. Existing values are never overwritten.
Declaring a key is also what tells the settings API which category and type to
use when an admin's save creates the row for the first time. Do not seed
settings by hand in `on_install` / `on_enable`: those hooks fire only on a state
transition, so a hand-seeded key added later never reaches an existing site, and
the row the first save creates lands in the placeholder `plugin` category where
the plugin's own settings page (which filters by category) cannot see it.
`public: True` puts the key on the unauthenticated read allowlist of
`GET /api/settings/<key>` and `GET /api/settings`. Use it only for cosmetic
values that a page rendering before login needs (a kiosk, a print page). Never
mark a credential, a hostname, or an integration URL public.
<a id="hook-get_collector_schema"></a>
### `get_collector_schema() -> Optional[Dict]`
Declares the JSON Schema for an external collector pushing to `/api/collector/<pluginname>`. See [ADR-006](../docs/adr/ADR-006-collector-contract.md) for the contract.
```python
class ComputersPlugin(BasePlugin):
def get_collector_schema(self):
return {
'identityfield': 'hostname',
'fields': {
'hostname': {'type': 'string', 'required': True},
'macaddress': {'type': 'string'},
'osname': {'type': 'string'},
'osversion': {'type': 'string'},
'currentuser': {'type': 'string'},
'ipaddress': {'type': 'string'},
}
}
```
If the hook returns `None` (the default), no collector endpoint is registered.
<a id="hook-apply_collector_payload"></a>
### `apply_collector_payload(payload: Dict) -> Dict`
Companion to `get_collector_schema` (ADR-006). The generic
`/api/collector/<pluginname>` endpoint calls this after the payload passes
identity validation, to idempotently upsert an asset. Return a dict with at
least `action` (`created` | `updated` | `noop`), `assetid`, and `warnings`
(list).
This is a CONDITIONAL hook: it is only required when `get_collector_schema`
returns non-None. The BasePlugin default raises `NotImplementedError` (the
dispatcher turns that into a 500), so a plugin that declares a schema but
forgets the upsert fails loud. Plugins with no collector schema never need it.
The `test_schema_declaring_plugins_implement_apply` contract test enforces the
pairing.
```python
def apply_collector_payload(self, payload):
host = payload['hostname']
comp = Computer.query.filter(Computer.hostname.ilike(host)).first()
action = 'updated' if comp else 'created'
# ... create-or-update Asset + extension ...
db.session.commit()
return {'action': action, 'assetid': comp.assetid, 'warnings': []}
```
<a id="frontend"></a>
## Frontend contribution
A plugin's UI is part of the plugin, not part of core. This section is the
contract for it; ADR-013 has the reasoning.
**Where the files live.** `plugins/<name>/frontend/`, holding `routes.js` and
`views/`. Nothing is written into `frontend/src/`. `scripts/stage-frontend.mjs`
(run by npm predev/prebuild) copies each chosen plugin's `frontend/` into
`frontend/src/.plugins-staged/<name>/` and codegens
`frontend/src/router/routes.gen.js`, which the router imports. Both outputs are
generated and gitignored.
**What a lean build stages.** `SITE_PLUGINS=machines,printers npm run build`
stages only those plugins; unset, it stages every plugin that has a
`frontend/routes.js`. The exception, and the rest of the lean-build model, is
in [PLUGINS.md](PLUGINS.md#lean-builds).
**The route module.** `routes.js` default-exports an array of routes and may
also export `toplevel` for full-screen routes that render outside `AppLayout`
(a kiosk, a print page). `toplevel` is optional: the codegen imports each
staged module as a namespace, so a module without it is not a build error.
Every route carries `meta: { plugin: '<name>' }`; the router guard redirects to
the dashboard when the backend plugin is disabled, so a route never loads a
shell over a plugin that is not there.
**One import rule, and staging is the reason.** Anything from the CORE tree
uses the `@` alias (`@/api`, `@/components/PaginationBar.vue`, `@/stores/auth`);
anything inside your own plugin is relative (`./views/...`). A relative path
into the core tree such as `../../api` is correct where the file sits in your
plugin and wrong once staged into `.plugins-staged/<name>/`. The failure is a
build error, not anything subtle.
**Menus follow the build, not a flag.** The sidebar nav, the settings rail and
the Displays links gate on whether your route was actually staged, so an
omitted plugin's nav item, settings cards and kiosk links vanish together -
your hooks cannot leave a menu entry pointing at a page that is not there.
<a id="lifecycle-hooks"></a>
## Lifecycle hooks
These run when the plugin's installation state changes. All optional.
| Hook | When | Use case |
|------|------|----------|
| `on_install(app)` | First time the plugin is installed via `flask plugin install` | Seed reference data, run plugin-specific migrations, register webhooks |
| `on_uninstall(app)` | When the plugin is removed via `flask plugin uninstall` | Clean up reference data, deregister webhooks |
| `on_enable(app)` | When the plugin is enabled at runtime | Subscribe to events, warm caches |
| `on_disable(app)` | When the plugin is disabled at runtime | Unsubscribe, drain queues |
<a id="import-surface"></a>
## The import surface (`shopdb.api`)
`shopdb.api` is the ONLY core module a plugin may import from (besides
`shopdb.plugins.base` for `BasePlugin` / `PluginMeta`). Importing internal
paths like `shopdb.core.models.*`, `shopdb.extensions`, or `shopdb.utils.*`
is a contract violation and fails the test
`tests/test_plugin_contract.py::test_plugins_only_import_contract_surface`.
What `shopdb.api` exposes. The machine-readable truth is `__all__` in
`shopdb/api/__init__.py`; this list groups the same names by what they are for.
**Infrastructure and model bases** - `db`, `cache`, `BaseModel`, `AuditMixin`.
**Core models (the asset contract, ADR-001)** - `Asset`, `AssetType`,
`AssetStatus`, `AssetRelationship`, `RelationshipType`, `Vendor`, `Model`,
`Location`, `Communication`, `CommunicationType`, `Setting`, `AuditLog`,
`Application`, `AppVersion`, `OperatingSystem`, `Script` (the hosted script
library), `User`, `Role`, `SupportTeam` (carries a `webhookurl`, so an alerting
plugin can route to a team's webhook).
**Displays** - `DashboardDefault`, `DISPLAY_ROLES`, `DISPLAY_ROLE_PATHS`,
`normalize_display_role`. The role vocabulary is core's, so a plugin that
reports on displays reads it instead of keeping a copy that drifts.
**Responses and pagination** - `success_response`, `error_response`,
`paginated_response`, `ErrorCodes`, `get_pagination_params`, `paginate_query`.
**Authorization** - `require_permission`, `require_any_permission` (a read two
different rights should reach), `require_role`, and `caller_holds(permission)`,
which ANSWERS whether the caller holds a permission without refusing the
request - for a route that serves everyone but should only offer what the
caller can act on.
**Service and collector tokens** - `service_token_authorized(scope)` returns
True when the request carries a managed service token scoped for `scope` whose
owner holds that permission; `authorized_service_token(scope)` returns the
`ApiToken` itself so a plugin can honor the token's optional resource binding
(`token.resourcescopelist`, NULL = unrestricted); `mint_service_token`,
`list_service_tokens` and `revoke_service_tokens` provision, list and revoke a
single-scope token from a plugin's own UI; `collector_authorized(pluginname)`
gates an agent-facing route that lives outside the generic collector blueprint
on the same per-plugin key and kill switch the ingest route uses.
**Serving uploads** - `send_upload`, `harden_upload_response`. Use them for any
user-supplied bytes: they set the headers that keep an SVG or an HTML file from
running as script in this origin.
**Caller IP** - `client_ip`, `strip_port`. Do not hand-roll this. The obvious
version reads the first `X-Forwarded-For` hop, which the caller supplies, so a
rate limit keyed on it is defeated by rotating the header.
**Request-argument and value parsing** - `intarg`, `intargs`, `QueryArgError`;
`parse_date`, `parse_date_fields`, `DateValueError`; `badge_kind`,
`normalise_badge`, `BadgeShapeError` (one badge vocabulary for the site, since
both kiosks read the same physical badges).
**Helpers** - `audit_log`, `resolve_asset_position`, `resolve_dualpath_pairs`,
`dualpath_single_machine_enabled`.
**Mail and webhooks** - `send_email(to, subject, html, text=None)`,
`send_alert(subject, html, text=None)` (settings-first, no-op safe when email
is unconfigured; targets the site's alert_recipients) and
`send_webhook(title, text)` (Teams Incoming Webhook / Workflow, or generic JSON
via the `alert_webhook_format` setting; best-effort, no-op when unset).
`send_alert` fans out to the webhook automatically alongside email.
**Bulk import** - `apply_import_timestamps`, `import_mode_active`,
`parse_import_datetime`; `register_import_hook` and `RowWarning` (a plugin owns
a VIRTUAL COLUMN on an importable CSV table plus a post-write row hook; core
never learns what the column means); and the journal models under
[Reversible bulk-import journal](#import-journal).
**External directories** - `employee_connection`, a read-only handle on a
site's existing HR database. Core global search and audit-log name resolution
use it; a plugin may too. It returns nothing when `EMPLOYEE_DB_*` is unset, so
guard on that rather than assuming a site has one.
```python
from shopdb.api import db, Asset, AssetType, success_response, paginate_query
```
Adding a name to `shopdb.api` is an additive (minor) contract bump; removing
one is breaking (major). See ADR-002.
<a id="helpers"></a>
## Helpers exposed to plugins
Everything importable is listed above; this section is only the semantics you
cannot guess from the name.
**`audit_log(action, entitytype, entityid=None, entityname=None, changes=None,
details=None)`** - write an audit row on every mutation. `changes` takes
`{'before': ..., 'after': ...}`. The current user, IP and timestamp are
captured for you.
**`get_setting(key, default=None)` / `set_setting(key, value, valuetype='string',
description=None)`** - methods ON your plugin class, reading and writing the
core `Setting` store under the namespaced key `plugin.<pluginname>.<key>`, so
two plugins cannot collide. Use them to READ; declare the rows themselves from
[`get_settings_defaults`](#hook-get_settings_defaults) rather than writing them
in `on_install`.
**`resolve_asset_position(asset)`** - returns
`{'mapx', 'mapy', 'levelid', 'positionsource'}`, where `positionsource` is
`'self' | 'related' | 'location' | None`. The resolution order is in
[ADR-001](../docs/adr/ADR-001-asset-as-platform-contract.md); the `levelid`
belongs to whichever source supplied the coordinates, and a plugin that writes
a position MUST write its level with it (ADR-017) - a marker drawn on the wrong
floor plan looks correct.
**`resolve_dualpath_pairs()` / `dualpath_single_machine_enabled()`** - a
Dualpath pair is one physical dual-bay machine recorded as two asset rows. When
the site setting `dualpath_single_machine` is on (the default), lists,
dashboard/report counts and the map show the pair as one machine by hiding the
SECONDARY bay; both rows always exist and detail pages stay per-bay. PRIMARY is
the pair member with the lower natural-sort assetnumber.
`resolve_dualpath_pairs()` returns `secondaryassetids` (the ids to hide) and
`partnerbyasset` (`assetid -> partner`, for both members, for banners) and
IGNORES the toggle, so a detail-page sibling banner can always show; gate the
collapse itself on `dualpath_single_machine_enabled()`.
**`apply_import_timestamps(row, data)` / `import_mode_active()` /
`parse_import_datetime(value)`** - bulk imports from the classic ASP shopdb
preserve each row's original `createddate` / `modifieddate` instead of stamping
"now". `apply_import_timestamps` is a no-op unless the caller is an admin AND
sent `X-Import-Mode: true`, so it never affects normal traffic; call it after
`db.session.add(row)` and before the commit. `import_mode_active()` is the same
predicate, for guarding other backdated behaviour (accepting a historical
`checkouttime`, say). `parse_import_datetime` reads both ISO
`2020-01-05T12:00:00` and legacy `YYYY-MM-DD HH:MM:SS` into naive UTC. The
operator manual is [IMPORT-API.md](IMPORT-API.md).
<a id="removed-hooks"></a>
## Removed hooks
The following hooks existed in early drafts and have been removed for v1:
| Hook | Reason |
|------|--------|
| `get_event_handlers` | Event bus deferred indefinitely. No real use case yet. Add via new ADR if needed. |
| `get_searchable_fields` | Removed in 0.4.0. Global search (`/api/search`) is a core concern over the asset model, already covering every bundled asset type, and honours runtime enable/disable. No plugin ever implemented it. |
<a id="versioning"></a>
## Versioning your changes
When you change anything documented here, you must:
1. Bump `__contract_version__` per [ADR-002](../docs/adr/ADR-002-plugin-versioning.md): major for removals or signature changes, minor for additive optional hooks, patch for docs.
2. Update [ADR-001](../docs/adr/ADR-001-asset-as-platform-contract.md) if the contract surface itself changed (or supersede with a new ADR).
3. Add or update the test in `tests/test_plugin_contract.py` that asserts the new behavior.
4. Update this page in the same change. `tests/test_docs_contract.py` fails the
build when a public `BasePlugin` hook is not mentioned here.
<a id="import-journal"></a>
## Reversible bulk-import journal
Contract version 0.30.0.
A plugin that drives a bulk import - a site asset adoption, an integration sync -
records the primitive changes it makes so the whole batch can be reversed later.
The journal is import-agnostic: it stores primitive mutations (set a cell, insert
a row) tied to a batch id, and a generic undo replays them in reverse. Nothing in
core learns what the import means.
Exposed on `shopdb.api`: `ImportBatch` (one row per applied import; `batchid` is
the identity the approval step, the write and the undo all reference; `status`
is `BATCH_APPLIED` or `BATCH_UNDONE`) and `ImportJournalEntry` (one reversible
mutation; `operation` is `OP_UPDATE`, carrying `oldvalue`/`newvalue`, or
`OP_INSERT`, a row this batch created).
**A plugin READS these and does not write them.** Its import UI lists applied
batches and offers "Undo import" per batch. The write and undo logic lives with
the import writer (`scripts/site_imports/journal.py`), which journals in the
SAME transaction as the mutation it describes and reverses conservatively: an
update is put back only if the cell still holds the value the import wrote (a
value a human has since changed is left alone and reported), and an insert is
hard-deleted only when nothing outside the batch references the created row.
The tables are core-owned (migration `7d37_import_batches_and_journal`), so they
exist regardless of which plugins are installed.

193
docs/PLUGIN-QUICKSTART.md Normal file
View File

@@ -0,0 +1,193 @@
<!-- docmeta
purpose: tutorial
audience: developer
status: current
summary: Scaffold, edit, install and test your first plugin in about 30 minutes using flask plugin new.
loadwith: PLUGIN-HOOKS.md
-->
# Plugin Quickstart
Build a working shopdb-flask plugin in about 30 minutes: generate it, put your
fields in it, install it, see it in the UI. Every shape used below is defined
in [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md), the contract and the only page that
defines it; read shapes there, not here.
> **Windows:** use `venv\Scripts\python` and `$env:FLASK_APP="shopdb"` (not
> `export`). Onboarding: [DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md).
## Step 1: Generate the skeleton
```bash
flask plugin new cameras --description "Tracks shop-floor surveillance cameras"
```
Output: `plugins/cameras/` with manifest, plugin class, example model, example
routes, schemas stub, tests, a README, a paste-in `frontend-api-snippet.js`,
and the frontend starting points - `plugins/cameras/frontend/routes.js` and
`frontend/views/CamerasList.vue`, `CamerasDetail.vue`, `CamerasForm.vue`.
Nothing is written into `frontend/src/`; the plugin owns its frontend, which is
what makes it prunable. The staging rules are in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#frontend). A plugin developed in its own repo
gets exactly this layout and starts working the moment the directory is dropped
into a host repo's `plugins/`.
The generated plugin already passes the framework's contract tests. Verify
before editing:
```bash
pytest plugins/cameras/tests/
```
## Step 2: Edit the model
Open `plugins/cameras/models/cameras.py`. Replace the `examplefield`
placeholder with your domain fields:
```python
class Cameras(BaseModel):
__tablename__ = 'cameras'
assetid = db.Column(
db.Integer,
db.ForeignKey('assets.assetid', ondelete='CASCADE'),
primary_key=True,
)
streamurl = db.Column(db.String(255), nullable=False)
resolution = db.Column(db.String(20))
fps = db.Column(db.Integer)
poeport = db.Column(db.String(50))
asset = db.relationship('Asset', backref=db.backref('cameras', uselist=False))
def to_dict(self):
return {
'assetid': self.assetid,
'streamurl': self.streamurl,
'resolution': self.resolution,
'fps': self.fps,
'poeport': self.poeport,
}
```
Note the naming convention: lowercase concatenated, no underscores
(`streamurl`, not `stream_url`). See [CONTRIBUTING.md](../CONTRIBUTING.md).
## Step 3: Add routes
Open `plugins/cameras/api/routes.py`. The scaffold provides list and detail
endpoints. Add CRUD as needed:
```python
@cameras_bp.route('', methods=['POST'])
@jwt_required()
def create_camera():
data = request.get_json()
asset = Asset(assetnumber=data['assetnumber'], name=data['name'], ...)
db.session.add(asset)
db.session.flush()
camera = Cameras(
assetid=asset.assetid,
streamurl=data['streamurl'],
resolution=data.get('resolution'),
)
db.session.add(camera)
db.session.commit()
return success_response(camera.to_dict(), http_code=201)
```
Everything you may import comes from `shopdb.api` - the list is in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#import-surface). For audit logging:
```python
from shopdb.api import audit_log
audit_log(action='created', entitytype='Camera', entityid=asset.assetid, entityname=asset.name)
```
## Step 4: Install the plugin
First add `plugins/cameras/migrations/` with a per-plugin Alembic chain that
creates the plugin's tables, and register those tables in
`PLUGIN_TABLE_OWNERS`. Your plugin is post-cutover, so its `0001` really
creates the tables; see [PLUGINS.md](PLUGINS.md#migrations) for the
anchor-vs-baseline rule. Then:
```bash
flask plugin install cameras
flask plugin upgrade-all
```
`install` runs the plugin's `on_install` hook (which seeds the AssetType row),
registers it in the plugin registry, and runs the plugin's own migration chain.
`flask db migrate` / `flask db upgrade` is reserved for core tables and must not
be used for plugin schema.
## Step 5: Verify it works
```bash
flask plugin list # cameras [Enabled]
pytest plugins/cameras/tests/
curl http://localhost:5001/api/cameras
```
## Step 6: Add hooks (optional)
Override hooks on the plugin class as needed. Each has a default that does
nothing, so override only what your plugin needs. The catalog, with the exact
shape each one returns, is
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#optional-hooks) - do not copy shapes from
this page or any other, because there is only one that is kept correct.
Four that most new plugins start with: `get_navigation_items` (sidebar entry),
`get_permissions` (its own RBAC permissions), `get_reports` (a card on the
Reports hub) and `get_settings_cards` (a card in the settings rail).
## Step 7: Finish the generated frontend
The scaffolded views build and run out of the box against the example model, so
`npm run build` is green immediately after scaffolding. What you finish by hand,
in order. Copy patterns from the closest bundled plugin - `network` is the
cleanest.
1. **Views** - replace the `examplefield` columns and inputs in
`plugins/cameras/frontend/views/*.vue` with your domain fields. Keep the
global `.filters` / `.form-control` / `.card` styles; do not invent per-page
input styling. Mind the import rule in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md#frontend): `@` for core, relative for your
own plugin.
2. **Routes** - `plugins/cameras/frontend/routes.js` is already generated with
`meta: { plugin: 'cameras' }` on every route and `requiresAuth` on the form
routes. No registration edit is needed.
3. **API client** - the scaffolded views ship with an inline `camerasApi` so
they run standalone. To graduate to the shared module, paste the generated
`plugins/cameras/frontend-api-snippet.js` block into
`frontend/src/api/index.js`, delete the inline const in each view, and
`import { camerasApi } from '@/api'` instead.
4. **Sidebar, reports, settings** - implement the hooks from step 6. No
frontend edit: those surfaces build themselves from what the hooks return.
5. **Settings subtype page** (if the plugin has subtypes) - add a route whose
path starts with `settings/` (e.g. `settings/cameratypes`); the router
nests it under the settings shell automatically. Copy a types-list view from
`frontend/src/views/settings/`.
6. **Verify** - `npm run build` must pass, then look at the pages against the
dev servers.
## Common errors
| Symptom | Cause | Fix |
|---------|-------|-----|
| `PluginNotFoundError: manifest.json` | Manifest deleted or moved | Restore `plugins/<name>/manifest.json` |
| `PluginContractError: missing required field` | manifest.json incomplete | Re-add `name`, `version`, `description` |
| `PluginVersionError: requires core_version X but framework is Y` | Framework upgraded past your range | Update `core_version` in manifest |
| `Table 'cameras' is already defined` | Two models declared the same `__tablename__` | Pick a unique table name |
| Index name collision | Two indexes share the same name (SQLite enforces global uniqueness) | Prefix index names with table: `idx_cameras_streamurl` |
## Next
- [PLUGIN-GUIDE.md](PLUGIN-GUIDE.md) - the conventions and traps of a real plugin, and the end checklist.
- [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md) - shipping a plugin from its own repo, per [ADR-003](adr/ADR-003-plugin-distribution.md).
- [CONTRACT-STABILITY.md](CONTRACT-STABILITY.md) - what you can safely build on before contract 1.0.

149
docs/PLUGIN-SIGNING.md Normal file
View File

@@ -0,0 +1,149 @@
<!-- docmeta
purpose: howto
audience: adopter, integrator
status: current
summary: How a curator signs a plugin artifact and how a site pins keys, verifies it and enforces signatures.
loadwith: none
-->
# Plugin signing and packaging (curator guide)
ADR-013 Phase 1. How a plugin becomes a signed, verifiable artifact and how a
site trusts it. The signature proves an artifact is EXACTLY what a curator
reviewed and signed - it does not prove the code is safe. Human review before
signing is the actual safety control; the signature makes that review's verdict
tamper-evident all the way to the point of execution.
Requires the `cryptography` package (already a dependency).
## One-time: create the publisher key pair
```
flask plugin keygen --out ./keys --name curator
```
Writes `keys/curator.key` (PRIVATE) and `keys/curator.pub` (public).
- Keep the `.key` OFFLINE with the curator. It is the only thing that can sign a
trusted artifact. Never put it on the plugin shelf or in the repo.
- Distribute the `.pub` with each site's deployed config and pin it (below).
- Rotation: generate a new pair, pin BOTH public keys on sites for an overlap
window (`verify` accepts any trusted key), then retire the old one.
## Per plugin: review, then pack
1. Review the plugin's source. This is the security gate - read what it does.
2. Validate and package in one step:
```
flask plugin pack printers --key ./keys/curator.key --publisher yoursite
```
`pack` refuses to sign a directory that does not validate (manifest schema,
name/dir match, core_version, dependencies on disk). On success it writes
`printers-<version>.shopdbplugin` - a zip of the plugin plus:
- `PROVENANCE.json`: name, version, publisher, created, and a sorted
`{file: sha256}` map of every packaged file.
- `PROVENANCE.sig`: a detached ed25519 signature over the exact
`PROVENANCE.json` bytes.
3. Publish the artifact to the shelf (a SharePoint-synced or copied folder).
Transport is untrusted; the signature is what makes it safe.
## Verify an artifact
```
flask plugin validate dist/printers-1.0.0.shopdbplugin --pubkey ./keys/curator.pub
```
Checks, fail-closed: signature against the trusted key(s), every file's hash,
no unexpected files, manifest schema, and that the plugin's `core_version`
admits this framework's contract version. Any changed byte in any file fails
the hash check; a signature from an untrusted key fails the signature check.
## Pin trusted keys on a site
Set `PLUGIN_TRUSTED_KEYS` to one or more public-key PEM paths, separated by the
OS path separator (`:` on Linux, `;` on Windows), in the site's environment:
```
PLUGIN_TRUSTED_KEYS=/etc/shopdb/keys/curator.pub:/etc/shopdb/keys/curator-next.pub
```
Keys are read only from this deployed config, never from the shelf - a folder an
attacker could write must not also carry the keys that authenticate it. With no
keys set, `validate` on an artifact fails closed (unverifiable).
## Enforce signatures (Phase 2)
By default nothing is enforced - plugins load unsigned, as before. To require
signatures on a site:
1. Stamp the plugins the image ships with, so verify-at-load applies to them
too (run at image build with the site/build key):
```
flask plugin stamp-bundled --key ./keys/curator.key
```
This writes `PROVENANCE.json` + `PROVENANCE.sig` into each in-tree plugin.
2. Pin the public key(s) and turn enforcement on (site config):
```
PLUGIN_TRUSTED_KEYS=/etc/shopdb/keys/curator.pub
PLUGIN_REQUIRE_SIGNED=true
```
Now a plugin only loads or migrates when its tree matches a trusted signature.
Under enforcement a `sys.meta_path` guard verifies EVERY `plugins.<name>.*`
import (not just `plugin.py`) - including the `from plugins.<name>.models import
...` that core request handlers do - against the plugin's signed provenance, and
executes the exact bytes it hashed (never a `.pyc`). An unsigned, tampered, or
wrong-key plugin is refused, fail-closed. `PLUGIN_DEV_TRUST_DIRS` exempts named
directories, but ONLY under DEBUG/TESTING (the external-repo dev workflow);
production ignores it.
Because every `plugins.*` import is verified, `stamp-bundled` must cover EVERY
plugin directory present (its no-argument form does), not only the enabled ones
- core code can import a disabled plugin's module, and an unstamped one would be
refused.
Defense in depth - set filesystem permissions so the app's runtime user CANNOT
write the `plugins/` directory (owned by the deploy user). Import-time
verification closes the "attacker drops a file, a request imports it" path; a
strict read-only `plugins/` also closes the narrow verify-vs-migrate race where
an attacker with concurrent write to `plugins/` swaps a migration script between
the check and alembic re-reading it.
## The shelf and adopt (Phase 2)
A shelf is a read-only folder of artifacts plus a signed index. The app reads
`PLUGIN_SHELF_DIR`; it never talks to SharePoint - a sync (or robocopy/USB)
populates that folder, and the signature makes the transport untrusted and
interchangeable.
Publish (curator, after packing artifacts into the shelf folder):
```
flask plugin shelf-build --dir /srv/shelf --key ./keys/curator.key --serial 3
```
The index carries a monotonic `serial` (a site refuses an index older than the
last it saw) and a `revoked` list (carried forward across builds). Bump
`--serial` on every publish.
On a site:
```
flask plugin shelf-list # browse (verifies index + serial)
flask plugin adopt printers # or printers==1.2.0
flask plugin audit # warn if an installed version is revoked
```
`adopt` verifies the shelf index and the artifact (signature + every file
hash), unpacks into a staging area, re-verifies, then atomically moves it into
place and installs + enables it with its dependency closure. It refuses a
downgrade unless `--force-downgrade`. Run `flask plugin upgrade-all` and restart
afterward so migrations apply and routes register.

134
docs/PLUGINS.md Normal file
View File

@@ -0,0 +1,134 @@
<!-- docmeta
purpose: reference
audience: adopter, operator, developer
status: current
summary: What each bundled plugin tracks, how per-plugin migrations are owned, and how a site builds lean.
loadwith: none
-->
# Plugins
shopdb-flask is a framework. The plugins listed here are the pieces other GE Aerospace facilities can install, build, or skip per ADR-003. Bundled plugins ship in the framework repo. External plugins live in their own repos and drop into `<repo>/plugins/<name>/` at install time.
## Bundled (ship with the framework)
These plugins are in `plugins/` in this repo. Enable per site with `flask plugin
install <name>`. None of them ships `default_enabled: false`, so the shelf and
the catalog are the same list here: a site turns one on rather than fetching
it. [PROJECT-MAP](PROJECT-MAP.md) carries their versions and migration heads,
generated from the code.
| Plugin | Tracks | Notes |
|--------|--------|-------|
| `machines` | Manufacturing machinery: 5-axis mills, lathes, broachers, heat treatment ovens, and metrology instruments | Manually entered. Metrology instruments are machines here, not a class of their own - see [ADR-018](adr/ADR-018-metrology-instruments-are-machines.md). Subtype tables for FOCAS / CLM / MTConnect controller protocols (planned). |
| `computers` | Shop-floor PCs and engineering workstations | Fed by the PXE pipeline collector per [ADR-006](adr/ADR-006-collector-contract.md). |
| `printers` | Network and shop-floor printers | Public installer map page + fleet install contract (`/api/printers/install-list`, `/pc-default`, `/install-batch`; see [PRINTER-INSTALLER.md](PRINTER-INSTALLER.md)). Optional Zabbix integration for supply tracking. Legacy `PrinterData` retiring per ADR-001. |
| `network` | Switches, routers, access points, IDFs as locations | Asset-only; cleanest of the bundled set. |
| `notifications` | Shop-floor notifications, recognitions, kiosk feed | Used by `ShopfloorDashboard.vue`. |
| `knowledgebase` | Knowledge Base articles linking to external resources | Lightweight article store. |
| `slides` | Slides for the lobby display and shop-floor screensaver | Upload / reorder / delete per surface. Management gated on the `slides.manage` permission, grantable to non-admin curators; display routes are public. |
| `tools` | Technician utilities: barcode and QR label generation, unit conversions, Secure Send | Mostly browser-side by design, so an air-gapped bench gets them with no network. Secure Send is the exception and is why this plugin has a blueprint, a table and a migration chain: see [SECURE-SEND.md](SECURE-SEND.md). |
| `warranty` | Asset warranty tracking | Manual entry now, Dell / Lenovo / HP provider lookups later. Derived coverage status with report buckets. |
| `guides` | Step-by-step procedures written in ShopDB: typed blocks, pasted screenshots, live checkboxes, tags, drafts and revision history | Content is stored as typed blocks rather than markup, so the same guide renders to screen, dark mode and paper. Readers see the PUBLISHED revision only - a technician following a procedure must not have it change under them mid-task. Checkbox ticks are per-viewer and never edit the guide. See [GUIDES.md](GUIDES.md). |
| `endpoints` | Live endpoint telemetry for shop-floor PCs: liveness, services, processes, inventory, and curated Windows event records, reported by the ShopDB agent | Depends on `computers` - an endpoint is telemetry ABOUT a PC asset, never a second record of it. Adds the Fleet console plus the live panels on a PC's detail page. Four separate permissions: `endpoints.view` to read, `endpoints.fleet` for the fleet-wide console, `endpoints.command` to act on one endpoint, and `endpoints.restart` for a reboot. They are split because read access is grantable to guests and the command channel is not. Ingest is not a route in this plugin: agent reports arrive on the generic collector route and authorize on a service token scoped `collector.ingest.endpoints`, not a user JWT, per [ADR-006](adr/ADR-006-collector-contract.md). That scope doubles as the kill switch for the agent key. See [FLEET-ARCHITECTURE.md](FLEET-ARCHITECTURE.md). |
| `backups` | Per-asset configuration backups with revision history: NTLARS/DNC registry and part marker | Kind-pluggable per [BACKUP-KINDS.md](BACKUP-KINDS.md); a kind is one class in `services/registry.py`. Parseable kinds store a JSON projection and re-render to the native format on download; opaque vendor formats stay on the SFLD share with ShopDB holding metadata and the UNC pointer. Fed by the collector, so ingest authorizes on a `collector.ingest` service token, not a user JWT. `backups.view` and `backups.download` are separate permissions because a `.reg` is a restorable artifact. Not to be confused with [BACKUP-RESTORE.md](BACKUP-RESTORE.md), which is the ShopDB database itself. |
## Plugin permissions
Plugins may register their own permissions (for example `slides.manage`). Admins
implicitly hold them; grant them to specific roles or users to delegate curation
without admin. Each plugin's registered permissions appear in its `plugin.py`
`get_permissions()`.
## Building your own
Guides:
- [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md) - **the contract**, and the only page that defines it: every hook, every manifest field, every importable name.
- [PLUGIN-QUICKSTART.md](PLUGIN-QUICKSTART.md) - start here: generate, customize, install and test a plugin in about 30 minutes.
- [PLUGIN-GUIDE.md](PLUGIN-GUIDE.md) - the conventions and traps of a real plugin, and the checklist to walk before calling one done.
- [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md) - developing a plugin in its own repo per ADR-003: repo layout, symlink dev workflow, `core_version` pinning, and the CI harness.
- [PLUGIN-SIGNING.md](PLUGIN-SIGNING.md) - packaging a plugin as a signed artifact and trusting it at a site.
- [CONTRACT-STABILITY.md](CONTRACT-STABILITY.md) - how much of the contract is settled, and how tightly to pin.
The contract is locked in [ADR-001](adr/ADR-001-asset-as-platform-contract.md) and versioned per [ADR-002](adr/ADR-002-plugin-versioning.md).
Quick path:
```bash
flask plugin new cameras --description "Tracks shop-floor surveillance cameras"
# edit plugins/cameras/models/cameras.py with your fields
flask plugin install cameras
```
<a id="migrations"></a>
## Migrations (per-plugin chains)
This is the one place the anchor-vs-baseline rule is written down; other pages
link here.
Each plugin that owns tables carries its own Alembic chain under
`plugins/<name>/migrations/`, with a per-plugin version table
`alembic_version_<name>` independent of the core `alembic_version`. Ownership is
split at a fixed cutover (see [ADR-008](adr/ADR-008-plugin-migration-ownership.md)):
- The core chain (`flask db upgrade`) created every table that existed through
its head, including the bundled-plugin tables. Each bundled plugin's `0001`
migration is a stamp-only no-op anchor recording that fact.
- From the cutover forward, a change to a plugin's schema lands as
`plugins/<name>/migrations/versions/000N_*.py`, never in the core chain. The
core chain is reserved for core tables.
- A plugin built AFTER the cutover (`endpoints` and `guides` are the bundled
examples), and every plugin built from now on, is different: the core chain
never created its tables, so its `0001` is a REAL baseline that creates them,
not a no-op anchor.
Deploys and upgrades run `flask db upgrade` then `flask plugin upgrade-all`.
`upgrade-all` stamps every plugin anchor and applies any later plugin
migrations; it is idempotent. The registry (`instance/plugins.json`) records
which revisions each plugin has applied in `migrations_applied`.
## Distribution conventions
For sister-site plugins (per [ADR-003](adr/ADR-003-plugin-distribution.md)):
- Plugin lives in its own git repo: `<git-host>/<your-site>/<pluginname>`
- Adopting site clones or symlinks into their `<repo>/plugins/<name>/`
- Plugin manifest declares `core_version` range matching the framework version they target
- Plugin readme explains: what it tracks, who maintains it, where to file issues
<a id="lean-builds"></a>
## Lean per-site builds
A site ships only the plugins it chose; a site that never wants warranty /
guides / network never carries that code (see
[ADR-013](adr/ADR-013-plugin-catalog-and-lean-builds.md) and
[ADR-014](adr/ADR-014-schema-lean-per-site.md)). Three layers make a build lean:
- **Backend code** - `scripts/build-site.sh <profile>` stages `shopdb/core` plus
only the chosen plugins' directories (and their hard-dependency closure). A
plugin a site did not choose is absent from the backend tree.
- **Frontend code** - `SITE_PLUGINS=machines,printers npm run build` (via
`scripts/stage-frontend.mjs`) stages only those plugins' `frontend/` dirs and
codegens the route table. **Exception:** a `plugins/<name>/frontend/` dir with
**no `manifest.json`** is a CORE feature (e.g. `applications`), not a per-site
plugin, and is ALWAYS staged regardless of `SITE_PLUGINS` - otherwise a lean
build would lose a core page.
- **Database** - the shared core Alembic baseline creates every plugin's tables,
so a lean site provisions them and then drops the ones it does not use with
`flask plugin prune-schema` (ADR-014). Run it once at provisioning, after
`flask db upgrade` and `flask plugin upgrade-all`; see
[DEPLOY.md](DEPLOY.md).
**Menus follow the build, not a plugin flag.** The sidebar nav, the settings
rail, and the Displays links all gate on whether the target route was actually
staged into this build (the router's own route table), not on a registry
"enabled" flag. So a lean site never shows a menu entry that dead-ends on a
blank page - an omitted plugin's nav item, settings cards, and kiosk links all
disappear together. Shopfloor Dashboard is a core view but is gated on the
notifications plugin (its only data source), so it drops when notifications is
not in the build.
## Naming policy
Plugin names follow the framework's naming convention (lowercase concatenated, no underscores or dashes; full words preferred over acronyms). See [CONTRIBUTING.md](../CONTRIBUTING.md). Plugin name collisions across sites are not enforced; the convention recommends prefixing site-specific plugins with the site code (e.g. `sitecode-shippingstation`) when there is risk of overlap.

140
docs/PRINTER-INSTALLER.md Normal file
View File

@@ -0,0 +1,140 @@
<!-- docmeta
purpose: reference
audience: developer, integrator
status: current
summary: The three printer install endpoints, the frozen text field order the shipped installer reads, and the level gap.
loadwith:
-->
# Printer installer map and install endpoints
How the shop-floor fleet installs network printers from shopdb-flask, replacing
the classic ASP `apiprinters.asp` / `apipcdefaultprinter.asp` / `installprinter.asp`
contract. Shopfloor 2.0 PCs cannot run unsigned `.bat` maps, so a signed
installer EXE (and the public web map page) drives installs from three endpoints
in the printers plugin.
- Server code: `plugins/printers/api/asset_routes.py`
(`printer_install_list`, `pc_default_printer`, `printer_install_batch`)
- Consumed by whatever lays software on a bay - a fleet configuration tool, an
imaging task, or a scheduled task. ShopDB serves the data; it does not push
the installer. Display kiosks do not get it: they print nothing.
All three endpoints are `@jwt_required(optional=True)`: an anonymous fleet
client works, and a logged-in browser (the public map page) works too.
---
## 1. The public map page
`PrinterInstallerMap` is a public (no-login) frontend page: the floor map with
printer hotspots positioned at each printer's `mapx` / `mapy`. The user clicks
the printers they want, and the page requests an install batch. The PC's default
printer is preselected via `pc-default`.
---
## 2. `GET /api/printers/install-list`
Flat, unpaginated list of active NETWORK printers. A printer counts as network
only if it has a hostname or a non-USB IP; USB-only printers are excluded.
Fields per row:
| Field | Notes |
|---|---|
| `printerid` | Printer id (the token `install-batch` takes). |
| `name` | Asset name, else asset number. |
| `machinenumber` | The asset number. |
| `windowsname` | Standardized Windows printer name. |
| `sharename` | Share / CSF name. |
| `hostname` | Print-queue host. |
| `ipaddress` | Primary IP (falls back to any communication row). |
| `vendorname` | Direct vendor, else the model's vendor. |
| `modelnumber` | Model name. |
| `installpath` | Installer path for this printer (see install-batch). |
| `iscsf` | CSF flag. |
| `locationname` | Location name, if the asset has one. |
| `mapx` / `mapy` | Floor-map hotspot position, in the native pixels of `levelid`. |
| `levelid` | Which level (drawing) those pixels belong to. Null when unplaced. |
`?format=text` returns a pipe-delimited line per printer, one printer per line,
with a fixed field order so the Inno / Pascal installer does a `split()` instead
of parsing JSON:
```
printerid|windowsname|vendorname|modelnumber|hostname|ipaddress|mapx|mapy|levelid
```
Any pipe or newline inside a value is neutralized to a space so the field count
stays fixed. The web map uses the default JSON.
**Fields 0-7 are frozen.** The shipped installer reads them positionally
(`GetField(Line, 6)` is mapx), so a new field goes on the END and nowhere else -
inserting one shifts every later field and the installer keeps running while
reading a model number as a coordinate. `levelid` is field 8 for that reason, and
installers built before levels existed ignore it.
An installer that ignores `levelid` draws every printer on whichever single
blueprint it ships, which is correct only while a site has one level. See
ADR-017 and section 5.
---
## 3. `GET /api/printers/pc-default?machine=NNNN`
The PC's default printer, by machine (asset) number persisted at PXE enrollment.
Parity with classic `apipcdefaultprinter.asp`: the installer preselects a PC's
default-printer hotspot on the site-map wizard. The link is resolved through the
`defaultprinter` asset relationship (PC asset -> printer asset), so it stays
inside the contract surface (no cross-plugin model import).
Returns `{printerid, windowsname}`, or `{}` when the machine is unknown or has no
active default printer set.
`?format=text` returns one pipe-delimited line (`printerid|windowsname`), or an
EMPTY body when there is no default (so the installer's split yields nothing).
---
## 4. `GET /api/printers/install-batch?printerids=1,2,3`
Returns a self-deleting Windows `.bat` that installs the selected printers,
grouped the same way classic `installprinter.asp` grouped them:
- HP / Xerox: one universal `PrinterInstaller.exe /PRINTER="a,b,c"` call.
- Printers with a `.exe` `installpath`: a PowerShell `Invoke-WebRequest` download
(using the caller's Windows credentials, against the site base URL + the
IIS-served `/installers` folder) followed by running it `/SILENT`.
- No `installpath`, or a non-`.exe` payload (e.g. a `.zip`): listed as a manual
install rather than run blindly.
`printerids` is required, comma-separated; non-numeric tokens are ignored. An
empty / missing list is a validation error.
The install name preference is `windowsname`, else `sharename`, else the asset
name / number.
---
## 5. The shipped installer predates levels (known gap)
The Pascal installer in the `inno` repo (`PrinterInstallerMap`) still assumes one
drawing per site, in two places that must change together:
- `MAP_SOURCE_W = 3300` / `MAP_SOURCE_H = 2550` are compiled-in constants, and
every hotspot is scaled by them. They are the dimensions of ONE level.
- A downsized copy of that level's blueprint is bundled into the installer
(`880x680`), so the picture is fixed at build time.
Two consequences, neither of which the installer can detect:
1. It ignores field 8, so printers on any level are drawn on the bundled image.
Coordinates from a different drawing land somewhere plausible and wrong.
2. When a level's blueprint is replaced with one of different dimensions, the
constants and the bundled image are both stale and every hotspot shifts.
The API side is ready: `GET /api/maplevels` is deliberately unauthenticated so
the installer can fetch level dimensions and blueprint URLs at runtime rather
than compiling them in, which is what fixes both. Until the installer is
rebuilt against it, treat its map as accurate for the default level only.

94
docs/PROJECT-MAP.md Normal file
View File

@@ -0,0 +1,94 @@
<!-- docmeta
purpose: reference
audience: developer, adopter
status: current
summary: Generated inventory - versions, plugins, Alembic heads, ADR index and codebase size.
loadwith: none
-->
<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
Regenerate with: venv/bin/python scripts/gen_project_map.py -->
# Project map
Derived from the code on every regeneration. If something here is wrong, the
code changed and the map was not regenerated - fix it by running the generator,
never by editing this file.
## Versions
| series | value | governed by |
|---|---|---|
| product `__version__` | `0.12.0` | ADR-007 |
| plugin contract `__contract_version__` | `0.35.0` | ADR-002 |
They move independently. A contract bump is not a release.
## Alembic chains (ADR-008)
Core runs with `flask db upgrade`; every plugin chain runs
with `flask plugin upgrade-all`. Both are needed on a deploy.
| chain | head |
|---|---|
| core | `7d51_endpoint_logcollections` |
| backups | `backups0003clearlastseen` |
| computers | `computers0006holdindefinite` |
| endpoints | `endpoints0024bootstraptokens` |
| guides | `guides0001baseline` |
| knowledgebase | `knowledgebase0001anchor` |
| machines | `machines0002rename` |
| network | `network0003prefix` |
| notifications | `notifications0006fleetalerts` |
| printers | `printers0005observedqueues` |
| slides | `slides0001anchor` |
| tools | `tools0001securesend` |
| warranty | `warranty0002proof` |
## Bundled plugins (12)
| plugin | version | core_version | owns migrations |
|---|---|---|---|
| backups | 1.0.0 | >=0.16.0,<1.0.0 | yes |
| computers | 1.0.0 | >=0.20.0,<1.0.0 | yes |
| endpoints | 0.1.0 | >=0.34.0,<1.0.0 | yes |
| guides | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| knowledgebase | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| machines | 1.0.0 | >=0.20.0,<1.0.0 | yes |
| network | 1.0.0 | >=0.20.0,<1.0.0 | yes |
| notifications | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| printers | 1.0.0 | >=0.20.0,<1.0.0 | yes |
| slides | 2.0.0 | >=0.2.0,<1.0.0 | yes |
| tools | 1.0.0 | >=0.16.0,<1.0.0 | yes |
| warranty | 1.0.0 | >=0.2.0,<1.0.0 | yes |
Manifest-less directories under `plugins/` are core frontend surface and always ship: `applications`.
## Architecture decisions
| ADR | title | status |
|---|---|---|
| ADR-001-asset-as-platform-contract.md | ADR-001: Asset model is the platform contract | current |
| ADR-002-plugin-versioning.md | ADR-002: Plugin contract versioning | current |
| ADR-003-plugin-distribution.md | ADR-003: Plugin distribution model | current |
| ADR-004-deployment-topology.md | ADR-004: Deployment topology (per-site instances) | current |
| ADR-005-equipment-vs-measuringtools.md | ADR-005: Equipment plugin scope vs measuringtools plugin | historical |
| ADR-006-collector-contract.md | ADR-006: Plugin collector contract pattern | current |
| ADR-007-product-versioning-and-releases.md | ADR-007: Product versioning and releases | current |
| ADR-008-plugin-migration-ownership.md | ADR-008: Plugin migration ownership (per-plugin chains from the cutover) | current |
| ADR-009-frontend-plugin-gating.md | ADR-009: Frontend plugin gating | current |
| ADR-010-frontend-plugin-hooks.md | ADR-010: Frontend plugin hook contract | current |
| ADR-011-machines-rename.md | ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes | current |
| ADR-012-geenforce-manifest-ownership.md | ADR-012: GE-Enforce manifest ownership in shopdb | historical |
| ADR-013-plugin-catalog-and-lean-builds.md | ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds | current |
| ADR-014-schema-lean-per-site.md | ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables) | current |
| ADR-015-site-specific-configuration.md | ADR-015: Where a site's own data is allowed to live | current |
| ADR-016-credential-delivery.md | ADR-016: Credential delivery to the fleet | historical |
| ADR-017-buildings-and-levels.md | ADR-017: Buildings and levels as the map model | current |
| ADR-018-metrology-instruments-are-machines.md | ADR-018: Metrology instruments are machines | current |
| ADR-019-per-pc-endpoint-credentials.md | ADR-019: Per-PC endpoint credentials | current |
## Size
- test functions defined: **2034** (parametrised cases collect higher)
- documented API paths: **347** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)

210
docs/ROADMAP.md Normal file
View File

@@ -0,0 +1,210 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Every major capability the product has shipped so far, and the two things left before a stable 1.0.0.
loadwith: none
-->
# Roadmap
**Hand-written, not generated, and it drifts.** Current contract and product
versions, the plugin inventory, every Alembic chain head and the codebase size
are in [PROJECT-MAP](PROJECT-MAP.md), which IS generated; nothing of that kind
belongs here. This page is a different thing: what the product can do today, and
the short list of what is left. It supersedes nothing in the ADRs.
shopdb-flask is pre-1.0 on both series.
## What has shipped
Read this as the answer to "does it already do X". Each row names the page that
covers it, so this stays a map rather than a second copy of the docs.
### The platform
| Capability | Where it is documented |
|---|---|
| One instance per site, single-tenant, with its own database, users, secrets and enabled plugins | [ADR-004](adr/ADR-004-deployment-topology.md) |
| `Asset` as the one model a plugin may build on, with relationships, propagation and map position on it | [ASSET-COMPOSITION](ASSET-COMPOSITION.md), [ADR-001](adr/ADR-001-asset-as-platform-contract.md) |
| A versioned plugin contract, with `core_version` ranges enforced at load | [CONTRACT-STABILITY](CONTRACT-STABILITY.md), [ADR-002](adr/ADR-002-plugin-versioning.md) |
| Manifest-first plugin loader, `shopdb.api` as the only import surface, auto-registered blueprints | [PLUGIN-HOOKS](PLUGIN-HOOKS.md) |
| Frontend hooks - settings cards, asset panels, map overlays, asset presentation - so a plugin contributes UI without editing core Vue | [ADR-010](adr/ADR-010-frontend-plugin-hooks.md) |
| Backend-driven route gating, so a disabled plugin's URL redirects instead of breaking | [ADR-009](adr/ADR-009-frontend-plugin-gating.md) |
| Per-plugin Alembic chains, owned by the plugin, applied by `flask plugin upgrade-all` | [ADR-008](adr/ADR-008-plugin-migration-ownership.md) |
| Lean per-site builds: a site carries only the plugins it chose, and `plugin prune-schema` drops the rest of the tables | [ADR-013](adr/ADR-013-plugin-catalog-and-lean-builds.md), [ADR-014](adr/ADR-014-schema-lean-per-site.md) |
| Plugin signing and a verification gate a site can switch on | [PLUGIN-SIGNING](PLUGIN-SIGNING.md) |
| Plugins shipped from their own repositories, dropped into `plugins/` | [PLUGIN-EXTERNAL-REPO](PLUGIN-EXTERNAL-REPO.md), [ADR-003](adr/ADR-003-plugin-distribution.md) |
| `flask plugin new` scaffolding, with an external-plugin harness to prove one loads | [PLUGIN-QUICKSTART](PLUGIN-QUICKSTART.md), [PLUGIN-EXTERNAL-REPO](PLUGIN-EXTERNAL-REPO.md) |
| RBAC with declared-and-enforced permissions, account lockout and login rate limiting | [CONFIG](CONFIG.md) |
| Every read behind a permission, and `guest` as a seeded role rather than a switch per feature - a site opens a door in the same role grid it uses for everybody else, and what a guest may be granted is bounded | [CONFIG](CONFIG.md), [PLUGINS](PLUGINS.md) |
| An append-only audit log with a retention period, a size-and-impact preview, and a purge that runs only when asked | [CONFIG](CONFIG.md) |
| Generated OpenAPI served at `/api/docs`, with `llms.txt` for machine callers | [llms.txt](llms.txt) |
| A dependency inventory (SBOM) and advisory scanning shipped with every release | [DEPENDENCY-SECURITY](DEPENDENCY-SECURITY.md) |
### The domain
| Capability | Where it is documented |
|---|---|
| PCs, printers, machines, network devices, applications, warranties | [PLUGINS](PLUGINS.md) |
| Metrology instruments filed as machines, with the measuringtools split retired | [ADR-018](adr/ADR-018-metrology-instruments-are-machines.md) |
| The equipment domain renamed to machines, and the models catalog retyped by modeltype | [ADR-011](adr/ADR-011-machines-rename.md) |
| Staff names resolved from the site's own HR database rather than copied into ShopDB - global search and audit-log attribution | [CONFIG](CONFIG.md) |
| Several devices under one asset number, as composition rather than duplicate records | [ASSET-COMPOSITION](ASSET-COMPOSITION.md) |
| Floor map as buildings and levels, with landmark-pair transforms to move markers onto a redrawn plan | [FLOOR-MAP](FLOOR-MAP.md), [ADR-017](adr/ADR-017-buildings-and-levels.md) |
| Procedures as typed, checkbox-able blocks inside ShopDB instead of documents on a share | [GUIDES](GUIDES.md) |
| A knowledge base, notifications and boards, slides and display kiosks | [PLUGINS](PLUGINS.md), [EVENTSAVER](EVENTSAVER.md) |
| Secure Send - a one-view expiring link for handing somebody a password | [SECURE-SEND](SECURE-SEND.md) |
| Technician utilities that work on an air-gapped bench: barcode and QR label generation, two-sided labels, unit conversions | [PLUGINS](PLUGINS.md) |
| Search across the inventory from one box, with each domain answering only to a caller holding that domain's permission | [CONFIG](CONFIG.md) |
| Retiring a record from the interface as a soft delete, so a scrapped tool leaves the lists and keeps its history | [ASSET-COMPOSITION](ASSET-COMPOSITION.md) |
| A vendor and model catalog with hygiene commands - dedupe, merge and rename - for the twins legacy data carries in | [CSV-IMPORT](CSV-IMPORT.md) |
| A worst-first dashboard queue with filters | [proposals/dashboard-live-fleet](proposals/dashboard-live-fleet.md) |
| Backups as a first-class feature, with kinds, a staleness card and a restore path | [BACKUP-RESTORE](BACKUP-RESTORE.md), [BACKUP-STALENESS](BACKUP-STALENESS.md) |
### The fleet
| Capability | Where it is documented |
|---|---|
| A generic collector contract: a plugin declares a schema and gets an ingest endpoint with auth | [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md), [ADR-006](adr/ADR-006-collector-contract.md) |
| Shipped PowerShell reporters for PC inventory and printer queues | [COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md) |
| A C# endpoint agent with a command channel, a tray app and staged upgrades | [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) |
| **Per-PC credentials**: each endpoint enrolls, holds its own secret, and can be revoked alone | [COLLECTOR-AUTH](COLLECTOR-AUTH.md), [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md) |
| Server public-key pinning on the agent, with the pin set authoritative from the registry | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| Printer installation driven from ShopDB, with printers as a property of the machine | [PRINTER-INSTALLER](PRINTER-INSTALLER.md) |
| Display kiosks and the event saver | [EVENTSAVER](EVENTSAVER.md), [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) |
| Adopting the shop-floor tools at a site that is not this one | [ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md) |
### Getting a site running and filled
| Capability | Where it is documented |
|---|---|
| Docker deployment: build, migrate, seed and serve behind the stack's own nginx | [DEPLOY](DEPLOY.md) |
| Air-gapped deployment from an image tarball, nothing pulled | [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md) |
| Pre-built frontend builds, for a box where npm cannot reach the registry | [DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) |
| Upgrading a running site, with the schema moved forward before the app serves | [UPGRADE](UPGRADE.md) |
| A first-run setup wizard that creates the first admin and captures site identity | [DEPLOY](DEPLOY.md), step 6 |
| Settings-driven branding, ServiceNow integration and site identity, so nothing is hardcoded per site | [ADR-015](adr/ADR-015-site-specific-configuration.md) |
| CSV import for sites with no developer, and a scripted import API for sites with a source database | [CSV-IMPORT](CSV-IMPORT.md), [IMPORT-API](IMPORT-API.md) |
| The completed classic-ASP migration, as a worked mapping | [IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) |
| Guides that travel: a procedure exports with its images in one file and imports at another site, and an existing HTML article converts into one | [GUIDES](GUIDES.md) |
## What is left
Two items. Everything else that was on this list is either done and listed
above, or deferred below.
### 1. Finalize a distributed release
The product runs at the site that built it. Making it something another site
receives, rather than something that is stood up by the people who wrote it, is
the remaining work.
**Docker is the shape a receiving site gets,** and on this line it is the only
one: the Windows/IIS installer topology is not carried here. What is missing:
- **Published images, pulled rather than built.** `docker-compose.pull.yml`
and `docker-compose.airgap.yml` already run a stack from an image instead of
a source build, and the image-publishing workflow exists with its channels
decided: a release tag publishes `:stable`, the trunk publishes `:edge`, and
both carry a short-SHA tag for rollback. It is not switched on. Its push
triggers are commented out and it runs only by hand, because nobody has
confirmed the registry package exists under this owner or decided who may
pull it. Turning that on, or producing an offline bundle for a site that
cannot reach a registry, with the checksum and the upgrade path written as
one procedure rather than three, is the work.
[DOCKER-PREBUILT-APPROACH](deployment/DOCKER-PREBUILT-APPROACH.md) ends with
the shape of it.
- **One command to stand a site up.** Today a first deploy is a `.env` to
fill, a certificate to place, a plugin profile to apply and an admin to
create. Each step is documented and none is hard; together they are the
reason an adopting site needs us on the call.
- **A receiving site's upgrade story, end to end.** [UPGRADE](UPGRADE.md)
covers a site that has the source. A site that was handed an image needs the
same page written for the artifact it actually holds.
- **Prove it by adopting at a site that is not this one.**
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md) is written and has not
been walked by somebody with no access to the authors.
The 2026-09-07
[production readiness assessment](deployment/PRODUCTION-READINESS.md) is the
record of why this is the item, and several of the gaps it names have closed
since.
### 2. Setup wizard polish
The wizard exists and is the primary path to a first admin: `/setup` runs while
`setup_complete` is false, creates the first administrator behind a setup token
or a loopback request, seeds starter vendors, and captures site identity.
Polishing it is what turns a working wizard into the thing an adopting site is
handed:
- Carry the steps that are still command-line - choosing and enabling plugins,
the floor plan, the collector key - into the wizard, so a site that never
opens a terminal gets a complete instance.
- Say what each choice costs and what can be changed later. A first-run screen
that quietly makes an irreversible decision is worse than a prompt.
- Make it re-runnable for the parts that are safe to re-run, so a site that
skipped a step is not stuck with the CLI as the only way back.
- Finish it against a genuinely empty instance, repeatedly, and on the air-gap
path where there is no network to lean on.
## Deferred, deliberately
Out of scope for 1.0. Each has a reason, and the reason is the point.
- **Multi-tenancy.** Rejected by [ADR-004](adr/ADR-004-deployment-topology.md).
Revisit only if more than five sites adopt and the per-site overhead becomes
painful.
- **Pip-installable plugins.** Deferred per
[ADR-003](adr/ADR-003-plugin-distribution.md). Filesystem distribution stays
the v1 model.
- **An event bus on `BasePlugin`.** Removed per
[ADR-001](adr/ADR-001-asset-as-platform-contract.md); add it via a new ADR if
a real use case appears.
- **A frontend rebuild** beyond Vue 3 + Pinia + Vite. The stack is fine.
- **Full palette theming.** Brand colors are settings-driven; the rest of the
palette is still in `style.css`. Worth doing, not worth blocking on.
- **A marketplace listing site.** [PLUGINS](PLUGINS.md) is a one-pager, and
that is enough until there are more than three external plugins.
- **Contract surface diff tooling.** Version bumps are manual judgement today.
A CI check that diffs the surface against the previous tag would catch a
missed bump - see [ADR-002](adr/ADR-002-plugin-versioning.md).
- **File sharing by expiring link.** The secret half shipped as
[SECURE-SEND](SECURE-SEND.md); the file half was dropped on purpose, and
dropping it removed the export-control question entirely, along with blob
storage and the need to authenticate a recipient. What would have to be
settled before it comes back, so nobody rediscovers it: a FILE recipient
should be logged in too, which costs nothing on an intranet-only server where
everyone with the link already has an account; sharing a file wants its own
permission separate from sharing a secret, because the risk classes differ;
and zero-knowledge encryption is right for a credential but questionable for
a controlled document, where "the server cannot say what was in it" is a
liability rather than a feature. Export control should confirm internal link
sharing in writing before anything here carries drawings or NC programs.
- **Calibration cycles, maintenance windows, downtime tracking.** Domain
extensions that would live in the `machines` plugin, not framework work.
## What 1.0.0 means
Tagging `1.0.0` is a commitment that the contract surface is stable for at least
the next minor version cycle: a plugin author at another site can pin
`core_version: ">=1.0.0,<2.0.0"` and trust the plugin will survive framework
patches.
Concretely, it ships when:
1. A site that did not build the product can receive a release, install it and
upgrade it without us - the distributed release item above.
2. The setup wizard takes a site from empty to usable without a terminal.
3. The contract test suite covers every documented hook with both a happy path
and a broken-plugin failure-isolation test.
4. At least one external plugin has been built end to end, from its own
repository, with no contract change needed mid-build.
## Decision log pointers
When something here gets prioritized, the why goes in a new ADR and this file
links it. The ADRs are canonical for design decisions and
[adr/README](adr/README.md) is their index, including which ones still hold.
This file is only the prioritized backlog.

157
docs/SECURE-SEND.md Normal file
View File

@@ -0,0 +1,157 @@
<!-- docmeta
purpose: explanation
audience: operator, adopter, developer
status: current
summary: How the one-view expiring credential link works, what the server cannot see, and who may create or read one.
loadwith:
-->
# Secure Send
Hand somebody a password without leaving it in a chat window.
Secure Send makes a link that carries one credential, dies when it is opened,
and expires on its own if nobody opens it. It lives in **Tech Tools** at
`/tools/secure-send`, and the person receiving it opens `/s/<token>`. It is the
one tool in that section that is not purely browser-side, because a link that
dies on first read needs somewhere to keep the ciphertext until it is claimed.
## What the server can and cannot see
The secret is encrypted **in the sender's browser** with AES-GCM, and the key
travels in the **URL fragment** - the part after the `#`. Browsers never send a
fragment to a server, so the key reaches the recipient's browser and nowhere
else: not this server, not its logs, not a proxy in between.
What the `securesendsecrets` table holds is a random token, base64 ciphertext,
an IV, and the rules for the row's life. Reading the database yields nothing.
That cuts both ways, and it is worth being explicit about it:
- **Nobody can recover a secret that goes missing.** Not an administrator, not
the database owner. If the link is lost, the secret is lost. Create another.
- **The server cannot tell a password from a page of a drawing.** It cannot
look at what it holds, so it cannot enforce a rule about content. The only
lever it has is SIZE, which is the next section.
- **The audit trail records the fact, never the content.** Who created a
secret, when, and when it was read are all recorded. What was in it is not
recoverable by anyone.
## This is for credentials, not documents
The ciphertext limit is **4096 characters** (about 2800 characters of plaintext,
which the create page enforces first with a friendlier message). That limit is
not a tuning knob. In a design where the server cannot read what it stores,
size is the entire server-side control over what this feature is for, and
raising it quietly turns a password tool into an unaudited document transfer.
Everything else the create page does - counting lines, spotting file
signatures, refusing a pasted file - runs in the browser. It guides an honest
person and is bypassable by anyone willing to POST directly. The size limit is
the part that holds.
If somebody needs to move a file, the answer is a different tool, not a bigger
number here.
## Who may create, who may read
**Two different doors, on purpose.**
**Creating** normally needs a login plus the `tools.secrets` permission (admins
hold it by default; grant it to other roles from Settings > Users & Roles). The
login is not a formality: anonymous creation would make this an unattributable
write endpoint that stores arbitrary ciphertext, and the creator is the only
person these rows can ever name.
**Reading needs neither an account nor a permission.** The recipient usually has
no account here, and the link is normally handed over on a call or in a chat.
The unguessable token IS the authorisation, which is why it is 128 bits of
randomness and why the row is spent on the first successful read.
### Guest creation
A site may open creation to people with no account with the
`securesend_allow_guests` setting (Settings > Tools > Secure Send). It is
**off by default**, because a guest secret has no creator and the audit trail
can then say when and from where but never who. The create page itself is served
without a login either way, so a guest at a site that allows it is not stopped
by a sign-in wall; the server, not the page, decides whether the write lands.
When it is on, guest creation is rate limited to 10 links per address per 10
minutes. That is not a security boundary - a determined person has more
addresses - but it stops an open endpoint being an easy place to park data or
fill a disk.
The setting is read straight from the settings row rather than the cache, and
**fails closed**: if it cannot be read at all, the answer is no. An error must
never be the thing that opens an unauthenticated write endpoint.
Receiving a secret never needs an account either way.
## Lifetime
| Control | Default | Range |
|---------|---------|-------|
| Expiry | 24 hours | 1 to 168 hours |
| Views | 1 | 1 to 100 |
Both apply, whichever comes first. A spent row is deleted outright, including
its ciphertext.
Expired secrets are swept opportunistically - whenever anybody creates a secret,
and whenever a dead link is clicked - rather than on a schedule. A secret nobody
ever opens would otherwise be removed by nothing, and an air-gapped site is not
going to have a cron job anybody remembers to add.
Missing, expired and spent all answer identically. Distinguishing them would
confirm to anyone holding a guessed token whether it ever named anything.
## Redeeming is a POST
`POST /api/tools/secrets/<token>` hands over the ciphertext and spends the view.
A GET would be wrong here: chat and mail clients prefetch links to build
previews, and a one-view secret consumed by a preview window is a secret the
recipient never gets to see.
There is a `GET /api/tools/secrets/<token>` as well, but it reports availability
only and never spends a view. The read page asks it first so it can say "this
secret is waiting, and opening it will destroy it" instead of consuming the only
view to find out whether there was anything there.
## Before anybody relies on it
**Send a link to yourself through whatever your site uses to pass it on -
Teams, Outlook, a chat client - and check the key is still there after the `#`.**
Some mail and chat security products rewrite links. If one strips or mangles the
fragment, the secret is unrecoverable and the server cannot help, which is the
whole point of the design. Find that out on a test secret rather than on a
production credential.
## Getting the link right
The link is built from the site's configured `site_base_url`, not from the
address the sender's browser happens to be using. Somebody creating a secret may
have reached the app by an IP, a short hostname, or a name that only resolves
inside one VLAN, while the recipient has to open the link from wherever they
are. Blank falls back to the browsing origin.
A site served under a subpath (`/shopdb`, `/ops`) needs that subpath reachable
in the link. Setting it as part of `site_base_url` works, and so does leaving it
out; the mount is added only when it is not there already.
## Where it lives
| Piece | Path |
|-------|------|
| Plugin | `plugins/tools/` |
| Table | `securesendsecrets` |
| API | `POST /api/tools/secrets`, `POST` and `GET /api/tools/secrets/<token>` |
| Create page | `/tools/secure-send` |
| Read page | `/s/<token>` |
| Setting | `securesend_allow_guests` (public, boolean, category `tools`) |
| Permission | `tools.secrets` |
The `tools` settings section is in [CONFIG.md](CONFIG.md); the response envelope
these routes use is in [llms.txt](llms.txt) and in the generated spec at
`/api/docs`.

106
docs/START-HERE.md Normal file
View File

@@ -0,0 +1,106 @@
<!-- docmeta
purpose: reference
audience: operator, adopter, developer, integrator
status: current
summary: The shortest correct path for the handful of things people actually arrive here to do.
loadwith: none
-->
# Start here
ShopDB is a per-site asset platform: one instance per plant, plugins for what
each site tracks. These pages document the server, the plugin contract, and the
tools that run on shop-floor PCs.
This page covers the handful of paths people actually arrive on, and the
warnings worth reading before you pick one. Anything not here is in
[INDEX](INDEX.md), which lists every task, the smallest page set that answers
it, and how many lines that costs. Do not read both: find your row in one of
them and go.
Each path below is the shortest correct one, not everything that exists.
## I am standing up ShopDB at a new site
**Docker is the path, and the only one this line ships.**
1. [DEPLOY](DEPLOY.md) - clone to first admin on Docker at a connected site.
No internet at all: [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md).
2. Then load the site's data: [CSV-IMPORT](CSV-IMPORT.md) from spreadsheets, or
[IMPORT-API](IMPORT-API.md) if there is a source database to script against.
3. Later versions: [UPGRADE](UPGRADE.md), after
[BACKUP-RESTORE](BACKUP-RESTORE.md).
Just want to look at a build without standing one up?
[DEV-BUILD-TEST](DEV-BUILD-TEST.md) pulls a prebuilt stack onto a Windows box
with Docker Desktop.
## I am deploying the shop-floor tools
Read [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) first - it is one page and it
is what makes the rest make sense. It ends with a symptom table that says which
piece to open when a bay misbehaves.
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md) has worked examples for
Intune and for manual installation, and says where each artifact comes from.
ShopDB serves the data these tools read; getting them onto a bay is your site's
configuration tooling, not ShopDB's.
## I am writing a plugin
1. [PLUGIN-QUICKSTART](PLUGIN-QUICKSTART.md) - scaffold and first run.
2. [PLUGIN-HOOKS](PLUGIN-HOOKS.md) - the contract. Authoritative, and the one
page every other plugin page defers to.
3. [PLUGIN-GUIDE](PLUGIN-GUIDE.md) - the conventions every bundled plugin
follows and the traps that have actually bitten someone. Walk its checklist
before you call a plugin done. Read `plugins/warranty/` beside it: it is the
smallest bundled plugin that exercises most of the contract.
If it lives in its own repository, [PLUGIN-EXTERNAL-REPO](PLUGIN-EXTERNAL-REPO.md),
and [CONTRACT-STABILITY](CONTRACT-STABILITY.md) before you choose a version pin.
## I am integrating with the API
1. [llms.txt](llms.txt) (also served at `/api/docs/llms.txt`) - auth, the
response envelope, common recipes. Short, and the envelope section is the
part people get wrong.
2. `GET /api/docs` on any running instance - the full spec, browsable.
3. [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md) if you are feeding data in
from a PC, and [COLLECTOR-AUTH](COLLECTOR-AUTH.md) for how that PC proves
who it is.
## Something is broken
- Server: `docker compose logs api`, then the health checks at the end of
[DEPLOY](DEPLOY.md).
- A PC or a bay: the symptom table at the end of
[FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md).
- An upgrade that went in and then misbehaved: [UPGRADE](UPGRADE.md), with
[BACKUP-RESTORE](BACKUP-RESTORE.md) if data has to come back too.
## I am trying to understand why something is built this way
[The ADRs](adr/README.md). They are the decision record, they say what was
rejected and why, they say whether each decision still holds, and they are the
fastest way to avoid relitigating a settled question. One ADR answers one
question; nothing there is a prerequisite for anything else.
## Three things people do not know exist
- [GUIDES](GUIDES.md) - procedures live in ShopDB itself, as typed blocks with
checkboxes a technician ticks as they work, instead of a Word document on a
share.
- [SECURE-SEND](SECURE-SEND.md) - a one-view expiring link for handing somebody
a password. Read its "Before anybody relies on it" section first: there is one
thing to test at your site before the first real credential goes through it.
- [DEPENDENCY-SECURITY](DEPENDENCY-SECURITY.md) - every release installs a full
inventory of third-party components, so a security team can answer "are we
exposed to this advisory" without access to anything here.
## Before you read anything else here
Every page opens with a `docmeta` comment saying what it is for and whether it
is still current. A page marked `legacy` is true but superseded, and a page
marked `historical` is a record of a finished migration or a proposal. Neither
is guidance. [INDEX](INDEX.md) lists all of them in one place, at the bottom,
so you never have to read one to find out it does not apply to you.

150
docs/UPGRADE.md Normal file
View File

@@ -0,0 +1,150 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Move a running Docker or venv site to a newer version, plus the notes for versions that need attention.
loadwith: none
-->
# Upgrading an existing site
> **Topology: Docker, or a bare-metal venv install.** An air-gapped Docker site
> upgrades by loading a newer image tarball -
> [DEPLOY-AIRGAP](DEPLOY-AIRGAP.md#upgrading-to-a-new-version), not this page.
> **Where this sits:** install -> operate -> **upgrade**.
> Back: [DEPLOY](DEPLOY.md), which is also the page for a first-time install.
Each site is single-tenant (ADR-004), so an upgrade touches only that site's
own stack. Read the ADRs added since your last update (`docs/adr/`) and the
`CHANGELOG.md` before starting; a breaking ADR may require coordinated work.
## Step 0: Back up first
Never upgrade without a fresh backup you have tested: a full database dump and a
copy of `instance/`. [BACKUP-RESTORE](BACKUP-RESTORE.md) is the procedure.
```bash
docker compose exec -T db mysqldump -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask | gzip > pre-upgrade-$(date +%F).sql.gz
cp -a instance/ instance-backup-$(date +%F)/
```
## Step 1: Get the new code
```bash
git pull <your-remote> <your-branch>
```
The application is distributed as source, and a production site builds its own
image from that checkout. Sites do not pull a published image: CI publishes
channel images to a container registry for dev and pilot workstations only
(`docker-compose.pull.yml`, `.github/workflows/publish-images.yml`).
## Step 2: Rebuild
The Docker image builds the Vue frontend in-image, so a container rebuild picks
up frontend changes automatically:
```bash
docker compose build api
docker compose up -d api
```
Bare-metal / venv install: rebuild the frontend by hand and refresh Python
dependencies:
```bash
source venv/bin/activate
pip install -r requirements.txt
cd frontend && npm ci && npm run build && cd ..
```
`up -d api` REPLACES the container, so everything the site has written must be
in the `instance_data` volume. If your stack predates that volume, do the
one-time rescue in [DEPLOY](DEPLOY.md#the-instance_data-volume-and-stacks-that-predate-it)
**before** this step, not after.
## Step 3: Migrate and re-seed
**Under compose this is automatic.** `docker compose up -d api` will not start
`api` until the `migrate` service has run `db upgrade`, the plugin bootstrap,
`plugin upgrade-all` and the three seeders and exited 0. Watch it:
```bash
docker compose logs migrate # the last line is the error, if there was one
```
A venv install has no `migrate` service and runs them itself:
```bash
flask db upgrade
flask plugin upgrade-all
flask seed permissions
flask seed settings
flask seed reference-data
```
All of them are idempotent, so running them at head or on an already-seeded
database is a no-op; the seeders add anything a new version introduced and leave
your site's customized values alone. `plugin upgrade-all` is the one people
skip - each plugin owns its own Alembic chain (ADR-008), and missing it surfaces
days later as a 1054 "Unknown column" on the page that uses the new field.
[DEPLOY](DEPLOY.md#step-3-the-schema-and-what-the-migrate-service-already-did)
explains what each command covers.
## Step 4: Restart and confirm
```bash
docker compose restart api
# venv: restart your process manager, e.g. pm2 restart shopdb-flask-api shopdb-flask-ui
docker compose exec api flask db current # must match `flask db heads`
```
Then log in: the page renders, and `/api/auth/login` returns a
`VALIDATION_ERROR` for an empty body rather than a 500.
## Version-specific notes
### Upgrading to v0.11.0: the floor map became buildings and levels
Nothing to do, and nothing moves on screen - the ordinary Step 3 migration
handles it. `7d33_buildings_and_levels` creates one building ("Main") and one
default level ("Ground floor") carrying the blueprint paths and dimensions from
your `map_*` settings, then assigns every already-placed asset and location to
that level. Unplaced things stay unplaced. The old settings rows are
deliberately left in place, so a downgrade still finds the blueprint.
Two things to know afterwards:
- **A second floor or building is a Settings task**, not a migration: add it under
Settings > Buildings and levels, upload its blueprint, then move the markers
that belong on it. `POST /api/mappositions/transform` moves a whole level's
markers onto a redrawn plan from landmark pairs, dry run by default.
- **Replacing a blueprint with one of DIFFERENT dimensions moves every marker on
that level**, because the coordinates are pixels of the old size. Upload it, then
recalibrate with landmarks rather than editing the level's width and height by
hand. The API refuses to adopt a new size on a level that already has markers,
for this reason.
If you wrote a plugin that stores map positions, see CONTRACT-STABILITY.md: the
contract is 0.20.0 and a position must now carry its `levelid`.
### Upgrading from before v0.5.0: the bundled floor plan was removed
Only affects an instance whose `map_blueprint_light` / `map_blueprint_dark`
Settings still point at `/static/images/sitemap2025-*`. Those bundled PNGs are
gone from v0.5 on, so the map 404s them after the upgrade. Re-upload your own
plan in **Settings > Floor Map**. Uploaded plans live under `instance/` and
survive upgrades, so a site that already uploaded one is unaffected.
```bash
docker compose exec -T api flask shell <<'PY'
from shopdb.core.models.setting import Setting
print(Setting.query.filter(Setting.key.like('map_blueprint%')).all())
PY
```
## See also
- [BACKUP-RESTORE](BACKUP-RESTORE.md) - what to back up and how to restore
- [CONFIG](CONFIG.md) - environment variables and Setting keys
- [DEPLOY](DEPLOY.md) - first-time deploy runbook
- `CHANGELOG.md` - what changed in each release

200
docs/adr/ADR-001-asset-as-platform-contract.md Normal file
View File

@@ -0,0 +1,200 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why Asset, not the legacy Machine model, is the surface plugin authors may depend on.
loadwith: none
-->
# ADR-001: Asset model is the platform contract
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
shopdb-flask is being shaped as a framework that sister GE Aerospace facilities can adopt. The framework defines a stable core; sites install plugins for the asset classes they care about (equipment, computers, printers, measuring tools, network gear, etc.).
The codebase ran two parallel object models:
1. **Legacy `Machine` model** in `shopdb/core/models/machine.py`. Original schema inherited from the classic-ASP shopdb. Tables: `machines`, `pctypes`, `machinetypes`. Plugins like printers stored extension data via `PrinterData` keyed by `machineid`.
2. **New `Asset` model** in `shopdb/core/models/asset.py`. Generic asset abstraction with `AssetType`, `AssetStatus`, `AssetRelationship`. Plugins also exposed asset-based tables keyed by `assetid`.
For the framework to be adoptable, the platform contract has to be one model, documented, versioned, and stable.
## Decision
**`Asset` is the platform contract.** Plugin authors target the asset-based API. The `Machine` model and its dependents (`PCType`, `MachineType`, `PrinterData` keyed by `machineid`) are legacy and will be retired through a tracked migration.
### Platform contract surface
The following are the public, versioned surface. Plugin authors may depend on them. Breaking changes require a major version bump per ADR-002.
#### Models
- `Asset` (core entity)
- `AssetType` (asset classification, registered by plugins)
- `AssetStatus` (active / inactive / decommissioned / retired)
- `AssetRelationship` + `RelationshipType` (cross-plugin links)
- `Vendor`, `Location`, `LocationType`, `BusinessUnit`, `Model`, `OperatingSystem` (shared reference data)
#### Helpers
- `AuditLog` API: `audit_log(action, entitytype, entityid, ...)` for plugins to record audit entries with consistent schema
- `Setting` API: `plugin.get_setting(key)` and `plugin.set_setting(key, value)` for plugin-scoped config persisted via the core `Setting` model
- `resolve_asset_position(asset)` for the documented position-resolution algorithm
#### Import surface (`shopdb.api`) - expanded in __contract_version__ 0.3.0
`shopdb.api` is the ONLY core module plugins may import (plus `shopdb.plugins.base`
for the ABC). Deep imports of `shopdb.core.*`, `shopdb.extensions`, or
`shopdb.utils.*` are contract violations enforced by the
`test_plugins_only_import_contract_surface` test. The surface re-exports:
- Infrastructure: `db`, `cache`
- Model bases: `BaseModel`, `AuditMixin`
- Core models: `Asset`, `AssetType`, `AssetStatus`, `Vendor`, `Model`,
`Communication`, `CommunicationType`, `Location`, `Setting`, `AuditLog`,
`Application`, `AppVersion`, `OperatingSystem`
- Responses: `success_response`, `error_response`, `paginated_response`, `ErrorCodes`
- Pagination: `get_pagination_params`, `paginate_query`
- Helpers: `audit_log`, `resolve_asset_position`; legacy `employee_connection`
Adding a name here is a minor (additive) contract change; removing one is major.
#### Plugin contract
- `BasePlugin` ABC and its hooks (navigation, dashboard widgets, collector schema + `apply_collector_payload`). Note: `get_searchable_fields` was removed in contract 0.4.0 - global search is a core concern over the asset model, not a per-plugin hook.
#### Excluded from the contract for v1
- Event bus (`get_event_handlers` removed from `BasePlugin`). Add later if a real use case appears.
### Relationship types (seeded core values)
`RelationshipType` is seeded with three rows. Plugin authors do not add new types in v1.
| Type | Meaning | Boundary rule |
|------|---------|---------------|
| `partof` | Composition, siblings, sub-assemblies | Parent dies without it |
| `controls` | One asset has operational authority over another | PC commands a machine, measuring tool, etc. |
| `connectedto` | Network or data link without operational authority | Cable, switch port, NAS mount |
### AssetRelationship columns
```
AssetRelationship
- relationshipid PK
- sourceassetid FK to assets
- targetassetid FK to assets
- relationshiptypeid FK to relationshiptypes (one of partof, controls, connectedto)
- label text, free description (e.g., "ethernet PoE", "DNC feed")
- inheritsposition bool, true means resolved-position walk follows this edge
- propagatesthroughid FK to relationshiptypes, nullable, see propagation below
- notes text
- isactive bool
- createddate, modifieddate
```
Free-text `label` carries the domain nuance (`controls` with label "DNC feed" vs `controls` with label "operator workstation"). Avoids inflating the type list.
### Sibling propagation
When a relationship is created or deleted, the framework checks `RelationshipType.propagatesthroughid`. If set, it finds all assets related to the source via the propagation type and applies the same change.
Default seeding:
| RelationshipType | propagatesthroughid |
|------------------|---------------------|
| `partof` | null (the propagation rail itself) |
| `controls` | `partof` (controls relationships propagate across siblings) |
| `connectedto` | null (network paths don't propagate) |
Cycle protection: max walk depth 3, visited-set during traversal.
### Position resolution
Resolved map position for any asset follows this priority chain:
1. Asset-specific override (`assets.mapx`, `assets.mapy` if non-null)
2. Walk relationships where `inheritsposition=true`, ordered by relationship type priority (`partof` first, then `controls`), recursively resolve the related asset's position with cycle detection
3. Fall back to the asset's location coords (`locations.mapx`, `locations.mapy` via `assets.locationid`)
4. If none of the above, asset is "unplaced" and rendered in a tray, not on the map
API exposes resolved position with a source indicator: `{"mapx": 234, "mapy": 567, "positionsource": "self|related|location|none"}`.
### Hierarchical locations
`locations` is extended with `parentlocationid` (self-FK, nullable). Sites define their own location tree (cells, sub-cells, network closets, meeting rooms, labs).
`locationtypes` lookup is added with seeded values: `section`, `cell`, `subcell`, `meetingroom`, `lab`, `office`, `storage`, `hallway`, `networkcloset`, `building`. Sites can extend.
Asset-to-location is the existing `assets.locationid` FK only (single primary location for v1). Multi-location and transient placement are deferred.
## Migration scope (narrowed)
Only one class of legacy data migrates: physical manufacturing equipment with a `machinenumber` (5-axis mills, lathes, broachers, heat treatment, CMMs, etc.).
Migration filter:
```sql
SELECT * FROM legacy.machines
WHERE category = 'Equipment'
AND machinenumber IS NOT NULL
```
Rows without `machinenumber`: **skipped.** Add manually post-migration if any matter.
Migrated rows become `assets` with `assettype='Equipment'` plus a row in the equipment plugin's `equipment` table. Legacy `machinetypeid` is preserved on the equipment row to enable later reclassification (some "Equipment" rows are actually measuring tools, see ADR-005).
Skipped from migration:
- PCs (rebuilt via PXE collector pipeline, see ADR-006)
- Printers, USB devices, network gear, notifications, KB articles (operational or re-collectable, decision per class deferred until needed)
## Consequences
### Positive
- Plugin authors at sister sites have a single, documented contract to target
- Cross-plugin features (relationships, search, map) work uniformly
- Three relationship types are easy to learn; free-text label captures nuance
- Sibling propagation handles dual-path machines without code changes per use case
- Position resolution gives users one map without per-plugin map code
### Negative
- Migration of equipment data must run cleanly; `migrating-asset-schema` skill owns the procedure
- Frontend `views/machines/` will be repointed or replaced; effort tracked in Phase 5
- Legacy `core/api/machines.py` (641 LOC) becomes deprecated; deletion or shim deferred until equipment is migrated
### Neutral
- `Machine`, `PCType`, `MachineType`, `PrinterData` remain in the schema until the migration completes, behind a deprecation notice. New code does not touch them.
## Alternatives considered
1. **Keep both models indefinitely.** Doubles test surface, confuses contract docs, breaks cross-plugin features. Rejected.
2. **Make `Machine` the contract; retire `Asset`.** `Machine` has shop-floor-specific assumptions that don't generalize. Rejected.
3. **Define a higher abstraction above both.** Yet another layer. `Asset` is already the abstraction. Rejected.
4. **More relationship types** (`controls`, `operates`, `monitors`, `dncfeeds`, `mountedon`, `connectedvia`, `inspects`, `siblingbay`). Eight types proved unwieldy. Collapsed to three with free-text labels and per-row inheritance/propagation flags.
## Open questions deferred
- Lifecycle relationship type (`replaces`, `replacedby`). Add when first needed.
- Multi-location asset placement, transient placement (calibration trip, off-site repair). Add when first needed.
- Map editing UI (drag locations onto floor plan). Phase 6 polish.
- Plugin-extensible relationship types beyond the three seeded. Add when a real cross-plugin case can't be expressed via labels.
## References
- `shopdb/core/models/asset.py`
- (core `Machine` model retired per this decision; machine data now owned by the machines plugin at `plugins/machines/models/machine.py`)
- `shopdb/plugins/base.py`
- ADR-002 (versioning of the surface)
- ADR-003 (plugin distribution)
- ADR-004 (deployment topology)
- ADR-005 (equipment vs measuring tools split)
- ADR-006 (collector contract pattern)

68
docs/adr/ADR-002-plugin-versioning.md Normal file
View File

@@ -0,0 +1,68 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: How the plugin contract is versioned and what core_version a plugin must declare.
loadwith: none
-->
# ADR-002: Plugin contract versioning
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
Once sister sites start writing their own plugins (or pulling community plugins), the framework's plugin contract becomes a public API. Without a versioning story, any change to `BasePlugin` or the core platform models can silently break installed plugins at remote sites.
The existing `BasePlugin` and `PluginMeta` already declare a `core_version` field (default `">=1.0.0"`), but it is not enforced anywhere. The plugin loader does not check it before instantiation.
## Decision
The framework adopts **semantic versioning for the plugin contract**, declared in two places:
1. **Framework version** (`shopdb/__init__.py`): a single `__contract_version__` constant. This is the version of the platform contract as defined in ADR-001. Bumped according to semver:
- **Major**: breaking change to `BasePlugin` ABC, `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`).
- **Minor**: additive change (new optional hook, new field on a contract model with default).
- **Patch**: bug fix, no contract surface change.
2. **Plugin requirement** (`plugins/<name>/manifest.json`): the existing `core_version` field, expressed as a semver range (e.g., `">=1.0.0,<2.0.0"`).
The plugin loader (`shopdb/plugins/loader.py`) checks `core_version` against `__contract_version__` at load time. Mismatch in dev = re-raise (fail loud). Mismatch in prod = log error, mark plugin as incompatible, exclude from registration.
The `__contract_version__` starts at **`1.0.0`** when ADR-001 is accepted and the `Machine` retirement migration is complete (whichever comes later). Until then, the framework is pre-1.0; plugins should declare `core_version: ">=0.1.0,<1.0.0"`.
## Consequences
### Positive
- Sister sites can pin a known-good framework version. They will not be silently broken when the framework is upgraded.
- Plugin authors know what counts as a breaking change because the contract surface is enumerated in ADR-001.
- The loader fails predictably: a mismatched plugin is reported, not silently disabled.
### Negative / cost
- Discipline required: every change to the contract surface must be classified (major / minor / patch). Adding a `version-bump` skill (or a check in code review) reduces the chance of mis-classification.
- `__contract_version__` becomes a coupling point. Forgetting to bump it after a breaking change means downstream plugins crash silently at runtime instead of failing at install.
### Neutral
- Existing plugins (`plugins/printers/`, etc.) ship as part of the framework, so their `core_version` is always the current `__contract_version__`. The discipline matters mostly for external / sister-site plugins.
## Alternatives considered
1. **No versioning, just trust.** Works for an in-tree-only world. Fails the moment a sister site ships its own plugin. Rejected.
2. **Calendar versioning** (e.g., `2026.05.0`). Easier to bump, harder to communicate breaking changes. Rejected; semver is the industry standard for library-like contracts.
3. **Per-hook versioning.** Each hook has its own version. Too granular; plugins still couple to multiple hooks. Rejected.
## Open questions
- When does the framework declare `1.0.0`? Tied to ADR-001 (Asset retirement of Machine) and the framework being deemed "ready for sister sites". Best-effort target: end of Phase 5 in the refactor plan.
- Should `core_version` accept commercial-grade ranges (`^1.0.0`) or stick to PEP 440 / npm-style ranges? Recommend pip-style (`>=,<`) to match Python ecosystem.
## References
- `shopdb/plugins/base.py` (PluginMeta declaration)
- `shopdb/plugins/loader.py` (where the version check belongs)
- ADR-001 (defines what is in the contract)

76
docs/adr/ADR-003-plugin-distribution.md Normal file
View File

@@ -0,0 +1,76 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How a plugin reaches a site: bundled in-tree, or dropped into plugins/ from its own repo.
loadwith: none
-->
# ADR-003: Plugin distribution model
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
## Context
Sister sites adopting shopdb-flask need a way to:
1. Install the framework
2. Pick which plugins they want
3. Build their own plugins for site-specific equipment
4. Receive updates to both framework and plugins
Today, every plugin lives in the `plugins/` directory of the framework repo. There is no separation between framework code and plugin code, no install / uninstall, and no way for a site to develop a plugin without forking the whole repo.
Three viable distribution models:
| Model | How a site installs a plugin |
|---|---|
| **In-tree only** | Fork the framework repo, add plugin under `plugins/`, run their own deploy. No separation. |
| **Pip-installable plugins** | Each plugin published as a Python package. Site does `pip install shopdb-printers shopdb-network` etc. Discovery via Python entry points. Framework loads any installed plugin that registers itself. |
| **Git-based plugins** | Each plugin lives in its own git repo. Site clones / submodules into `plugins/<name>/`. Loader picks them up from the directory. |
## Decision
Use a **hybrid model** with two clearly-labeled paths.
1. **Bundled plugins**: a small set of plugins ships with the framework, in-tree at `plugins/`. These are the reference implementations and the default install (printers, computers, network, equipment, usb, notifications). A site that wants only what's bundled needs no extra work.
2. **External plugins**: sister sites or third parties build plugins in their own git repos. The site running the framework drops the plugin into `plugins/<name>/` (clone, submodule, or symlink) and runs `flask plugin install <name>`. No pip packaging required for v1.
Pip-installable plugins (Python entry-point discovery) are deferred to v2. The complexity is not justified until at least two sites are running their own plugins.
## Consequences
### Positive
- v1 is simple: filesystem-based discovery (already implemented in `shopdb/plugins/loader.py`), works for both bundled and external plugins.
- Sites can develop plugins without changing the framework repo.
- The `plugins/` directory is already the canonical location, so no architectural change is needed.
### Negative / cost
- No automatic update path for external plugins. Sites must `git pull` in each plugin directory manually. Acceptable for v1; revisit when plugin count grows.
- Multiple plugin authors writing in parallel can collide on namespace (e.g., two plugins both registering an `AssetType` named "Equipment"). Need a naming policy: plugin names and asset-type names should be prefixed with the site or org if not in the bundled set.
### Neutral
- The existing in-tree pattern keeps working. This decision just formalizes it and clarifies the path for outside-the-tree plugins.
## Alternatives considered
1. **Pip-installable from day one.** Cleaner for the long term but adds packaging, entry-point registration, and CI steps. Premature for current scale (one site running, no sister-site plugins yet).
2. **In-tree only forever.** Forces every site to fork. Doesn't scale beyond two or three sites.
3. **Submodules only.** Forces git-submodule discipline on every adopting site. Submodules are notoriously fiddly. Rejected.
## Open questions
- For external plugins, should there be a manifest field (`source_url`) declaring where the plugin can be cloned from, so `flask plugin install` could pull it for the site? Defer; manual clone is fine for v1.
- Naming convention for non-bundled plugin directory names: prefix with site? (`gea-wjsf-shipping`)? Adopt if and when we hit a name collision.
## References
- `shopdb/plugins/loader.py` (filesystem discovery)
- `shopdb/plugins/cli.py` (plugin install / uninstall command)
- ADR-001 (defines what plugins target)
- ADR-002 (defines plugin version compatibility)

100
docs/adr/ADR-004-deployment-topology.md Normal file
View File

@@ -0,0 +1,100 @@
<!-- docmeta
purpose: explanation
audience: adopter
status: current
summary: Why every site runs its own instance instead of one multi-tenant deployment.
loadwith: none
-->
# ADR-004: Deployment topology (per-site instances)
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
## Context
shopdb-flask manages shop-floor inventory. If multiple GE Aerospace sites adopt it, the deployment can take one of two shapes:
| Model | How it works |
|---|---|
| **Per-site instances** | Each site runs its own Flask + MySQL + Vue stack. Each site has its own DB, its own users, its own enabled-plugin list, its own deploy. Sites are isolated. |
| **Multi-tenant single instance** | One central Flask + MySQL + Vue stack serves all sites. A `siteid` foreign key on every asset partitions data. Auth distinguishes which site a user belongs to. |
The codebase today is single-tenant per deployment. There is no `siteid` column, no tenant filter, no cross-site auth model. Plugins can be enabled / disabled but only globally for the running instance.
## Decision
**Per-site instances.** Each adopting site runs its own dedicated stack. The framework does not support multi-tenancy.
Each site:
- Owns its database (own credentials, own backup policy, own retention). The
database charset is part of the contract: it must be **utf8mb4**
(`utf8mb4_unicode_ci`). The migration chain creates every table utf8mb4, and
the connection pins `?charset=utf8mb4`. A site that creates the database with
a different default charset (older MySQL defaults to latin1) gets a schema
that silently diverges from every other site. See `docs/DEPLOY.md`.
- Picks its own enabled plugins
- Configures its own JWT secret, CORS allowlist, Zabbix integration, Active Directory binding
- Deploys at its own cadence
The framework provides:
- A `Dockerfile` and `docker-compose.yml` template suitable for a single-site deploy
- A `.env.example` listing all required environment variables
- A `docs/DEPLOY.md` walking through a fresh-site install
## Consequences
### Positive
- Simpler code: no tenant filter on every query, no cross-tenant auth, no shared-state partitioning bugs.
- Sites are independent. A schema change at one site does not affect another. A plugin crash at one site does not blast radius to other sites.
- Clear ownership: each site's IT team owns their own stack and data. Compliance and audit boundaries match operational boundaries.
- Aligns with how GE Aerospace sites already operate (independent IT, independent shop floors).
### Negative / cost
- No cross-site reporting out of the box. If GE corporate ever wants a fleet-wide view, it has to be built on top (e.g., a roll-up dashboard that queries each site's API). That layer is out of scope for the framework.
- Each site administers its own stack. Higher operational overhead than a single central instance, but each site already runs its own infrastructure.
- Updates require visiting each site's deploy. Fine for the current adoption model; revisit if dozens of sites adopt.
### Neutral
- No `siteid` column needed. The existence of one DB per site is the partition.
## Alternatives considered
1. **Multi-tenant single instance.** Lower operational overhead at scale, easier cross-site reporting, but adds significant code complexity and risk: every query needs a tenant filter, auth gets complex, schema migrations affect every site at once, and a bug at one site can leak data across sites. Rejected for v1; revisit if and only if more than five sites adopt and operational overhead becomes painful.
2. **Hybrid: per-site DB but central app server.** Adds the operational complexity of multi-tenancy without isolating the failure domain (one app crash = all sites down). Rejected.
## Migration strategy (resolved)
Deploys run two commands: `flask db upgrade` then `flask plugin upgrade-all`
(lean/ADR-014 sites add an optional `flask plugin prune-schema` at initial
provisioning). The core Alembic chain applied by `flask db upgrade` creates the
full core AND bundled-plugin schema through the chain head (this includes
migration `7c04_fold_plugin_schema`). But every bundled plugin still carries its
own Alembic chain per ADR-008: `flask plugin upgrade-all` stamps each plugin's
own chain (the `alembic_version_<plugin>` tables) and applies any plugin-specific
migrations added after the ownership cutover. The earlier Phase 7B state that
folded everything into core with no per-plugin chains was superseded by ADR-008's
per-plugin ownership. A fresh `flask db upgrade` reproduces the live core schema
exactly (verified on a scratch DB).
External (out-of-tree) plugins per ADR-003 ship their own migrations too; the
framework runs the same per-plugin chain mechanism (ADR-008) for both bundled and
external plugins.
## Open questions
- Should the framework provide an optional **read-only fleet roll-up** mode where a "central" instance can pull aggregate metrics from each site's API? Defer. Out of scope for v1.
- Backup strategy per site: framework recommendation, or each site decides? Framework should publish a recommended backup runbook (mysqldump + offsite copy) but not enforce.
- Auth federation: each site has its own user table, or sites can share an LDAP / SSO? Recommend documenting the LDAP config knob in `.env.example` so sites can plug in their own auth without code change.
## References
- `shopdb/config.py` (currently single-tenant, no `siteid`)
- ADR-001 (asset model is per-site, not cross-site)
- ADR-003 (plugin distribution per site)

163
docs/adr/ADR-005-equipment-vs-measuringtools.md Normal file
View File

@@ -0,0 +1,163 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026 split of machinery from metrology tools into two plugins. Superseded by ADR-018.
loadwith: none
-->
# ADR-005: Equipment plugin scope vs measuringtools plugin
- **Status:** SUPERSEDED by [ADR-018](ADR-018-metrology-instruments-are-machines.md)
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
> **Do not build against this.** ADR-018 (2026-09-04) retired the
> `measuringtools` plugin and made metrology instruments machines. The split
> below was sound for the population it described, and that population never
> arrived: the plugin held zero rows. Kept as the record of why the split was
> made and why it was undone.
## Context
ADR-001 narrowed the migration to physical manufacturing equipment with a `machinenumber`. In practice, the legacy `category='Equipment'` rows contain two distinct asset classes:
1. **Manufacturing machinery** (5-axis mills, lathes, broachers, heat treatment ovens). These produce parts.
2. **Metrology and inspection instruments** (CMMs, Keyence vision systems, wax-and-trace surface profilometers, GenSpec instruments). These measure parts.
Both share `Asset` properties (vendor, model, location, controller). They differ in domain fields (axes vs measurement accuracy, cycle time vs calibration interval, controller protocol vs measurement software).
Mixing them under one plugin pollutes the schema and confuses cross-plugin queries ("show me all measuring tools" requires an enumeration of measuring-instrument equipmenttype values, which scales badly).
## Decision
Two plugins, separate concerns, shared platform contract.
### `equipment` plugin
Tracks manufacturing machinery. Bundled, in-tree.
Schema (per ADR-001 contract):
```
equipment
- assetid FK to assets, PK
- equipmenttypeid FK to equipmenttypes (5-axis mill, lathe, broacher, heat treat, ...)
- vendorid FK to vendors (platform)
- modelid FK to models (platform)
- controllertypeid FK to controllertypes (equipment plugin)
- controllerosid FK to controlleros (equipment plugin)
- (other shared fields: spindle count, axes, max workpiece size, ...)
equipmenttypes (lookup, equipment plugin)
- equipmenttypeid, name (5-axis mill, lathe, broacher, heat treat, ...)
controllertypes (lookup, equipment plugin)
- controllertypeid, name (Fanuc 31i, Siemens 840D, Mitsubishi M70, Heidenhain TNC640, ...)
- vendorid (FK to vendors)
controlleros (lookup, equipment plugin - separate from PC OS)
- controllerosid, name (FAPT, VxWorks, embedded Windows, Linux RT, ...)
equipmentfocas (subtype, optional, present only when FOCAS-equipped)
- assetid PK, FK to equipment
- focasipaddress text
- focasport integer
- focasversion text
- focasmachinenumber text
equipmentclm (subtype, optional, present only when CLM-equipped)
- assetid PK, FK to equipment
- (CLM-specific: address, port, station ID - finalize when plugin is built)
equipmentmtconnect (subtype, optional, present only when MTConnect-equipped)
- assetid PK, FK to equipment
- mtconnectagenturl text
- mtconnectdevicename text
```
The `equipment.protocol` enum field is deliberately **not** included. Presence or absence of a subtype row indicates which protocol applies. Avoids a denormalized field that can drift out of sync.
### `measuringtools` plugin
Tracks metrology and inspection instruments. Bundled, in-tree (built in Phase 3-4 of the refactor as the first new plugin built using the framework scaffold).
Schema (initial draft, refined when plugin is built):
```
measuringtools
- assetid FK to assets, PK
- measuringtooltypeid FK to measuringtooltypes (CMM, vision system, profilometer, surface tester, ...)
- vendorid FK to vendors (platform)
- modelid FK to models (platform)
- measurementaxes integer (e.g., 3 for a 3-axis CMM)
- accuracyspec text (e.g., "+/-0.5um")
- calibrationintervaldays integer
- lastcalibrationdate date
- nextcalibrationdate date (computed)
- (other domain fields as needed)
measuringtooltypes (lookup, measuringtools plugin)
- measuringtooltypeid, name (CMM, vision system, surface profilometer, gage block, ...)
```
Future extension: subtype tables for measurement-software integrations (PC-DMIS, Keyence, GenSpec). Same pattern as equipment subtype tables.
### Subtype-table pattern (general)
Both plugins use the same pattern for protocol- or software-specific fields:
- Core plugin table carries shared, common fields
- Optional subtype tables (one per protocol or software) hold extension fields
- Each subtype table is keyed by `assetid` (PK), one-to-one with the parent
- Subtype row exists if and only if the asset uses that protocol or software
- Sister sites add new subtype tables for their own integrations without touching core
## Reclassification of legacy data
ADR-001's migration moves all legacy `category='Equipment' AND machinenumber IS NOT NULL` rows to `assets` with `assettype='Equipment'` and into the equipment plugin's `equipment` table. This includes both manufacturing machinery and measuring tools.
After the equipment migration, when the measuringtools plugin is built:
1. Build a mapping table: legacy `machinetypeid` values that are measuring tools (CMM type, Keyence type, etc.)
2. Run a reclassification script:
- For each `assets` row where the original `machinetypeid` is in the measuring-tool mapping
- Change `assets.assettype` to `'MeasuringTool'`
- Move the row from `equipment` to `measuringtools`
- Map domain fields where they differ (e.g., legacy `axes` field maps to `measurementaxes`)
3. Verify counts pre- and post-reclassification
4. Audit log entry per reclassified row
Reclassification is one-shot, run once, archived. Like the original migration script.
## Consequences
### Positive
- Manufacturing machinery and measuring tools are first-class plugins, each with appropriate domain fields
- Sister sites can install one or both depending on what they track
- Subtype-table pattern is the canonical example for protocol-specific data and extends naturally to other plugins
- Building `measuringtools` mid-refactor validates the plugin scaffold tooling against a real new plugin
### Negative
- Reclassification is a second migration step. Lower risk than the initial migration because it is data-only (no schema change beyond moving rows between two tables that share the same `assetid` link).
- Sites that adopt the framework before `measuringtools` ships need to either keep measuring tools in `equipment` (workable but suboptimal) or wait for the plugin
### Neutral
- Legacy `machinetypeid` is preserved on the equipment row during migration to enable reclassification
## Alternatives considered
1. **Single equipment plugin with sub-typed assets.** Use `equipment.equipmenttypeid` to discriminate manufacturing vs metrology. Rejected: domain fields differ enough that a single table is wide and full of NULLs.
2. **Migrate split (build mapping before initial migration).** Cleaner end state but requires the `measuringtools` plugin to exist before the migration runs, which delays Phase 5. Rejected.
3. **JSON blob for protocol data instead of subtype tables.** Considered for both plugins. Rejected: weak typing, awkward queries, no schema validation.
## References
- ADR-001 (Asset is platform contract)
- ADR-002 (versioning of the surface)
- `plugins/machines/` (equipment plugin, renamed per ADR-011)
- `plugins/computers/` (existing example of plugin pattern)

156
docs/adr/ADR-006-collector-contract.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer, integrator
status: current
summary: How a plugin declares a collector schema so the framework wires the ingest endpoint and auth.
loadwith: none
-->
# ADR-006: Plugin collector contract pattern
- **Status:** ACCEPTED
- **Date:** 2026-05-08
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
PC inventory data was collected by PowerShell scripts pushing to `/api/collector/pc` (`shopdb/core/api/collector.py`, ~374 LOC). The endpoint is hardcoded for PCs: it accepts a fixed schema and writes to the legacy `Machine` model.
Per ADR-001, `Machine` is being retired in favor of `Asset`. Per the project shift to PXE-driven imaging, PC inventory is moving to a new collection pipeline (PXE / GE-Enforce / manifest engine produces JSON about each PC). Other asset classes may want similar collector pipelines (printers via Zabbix, network gear via SNMP scan).
This calls for a generalizable contract: any plugin that wants to accept external collector input declares a JSON schema, and the framework wires the endpoint, auth, and idempotency.
## Decision
`BasePlugin` gets two new hooks (added in __contract_version__ 0.2.x -> the surface is carried at 0.3.0):
```python
def get_collector_schema(self) -> Optional[dict]:
"""Return JSON Schema describing the collector payload for this plugin.
Return None if the plugin does not accept collector input.
The schema must include:
- 'identityfield': name of the field that uniquely identifies an asset
across submissions (e.g., 'hostname' for PCs, 'macaddress' for network
devices). Used for idempotent upsert.
- 'fields': JSON Schema definitions for the rest of the payload.
"""
return None
def apply_collector_payload(self, payload: dict) -> dict:
"""Idempotently upsert an asset from a validated collector payload.
Called by /api/collector/<pluginname> after identity validation.
CONDITIONAL hook: required only when get_collector_schema returns
non-None. Default raises NotImplementedError (the dispatcher returns
500) so a schema-without-upsert fails loud. Returns a dict with at
least 'action' ('created'|'updated'|'noop'), 'assetid', 'warnings'.
"""
raise NotImplementedError
```
The pairing (schema present => apply implemented) is enforced by the
`test_schema_declaring_plugins_implement_apply` contract test.
A single dynamic dispatch route `/api/collector/<pluginname>` serves every
plugin that returns a schema (rather than registering a blueprint per plugin),
because Flask forbids `register_blueprint` after the first request and plugins
can be enabled at runtime. Auth is API-key, separate from JWT. Per-plugin keys via env vars:
- `COLLECTOR_API_KEY_<PLUGINNAME>` (preferred, plugin-specific)
- `COLLECTOR_API_KEY` (fallback, shared)
### Idempotent upsert
The endpoint uses the `identityfield` to find an existing `Asset` for the same identity. Found = update. Not found = insert. Existing relationships are preserved on update.
### Response contract
```json
{
"status": "ok",
"action": "created" | "updated" | "noop",
"assetid": 12345,
"identityvalue": "PC-1234",
"warnings": []
}
```
### Audit logging
Every collector submission produces an audit log entry: `{action, plugin, identityvalue, before/after diff}`. Audit retention per site policy.
### Schema discovery
The framework exposes the registered schemas at `/api/collector/_schemas` (read-only, JWT-protected) so external collector authors can introspect what payloads are accepted by which plugins.
## Concrete first user: computers plugin
The `computers` plugin is the first to implement `get_collector_schema`. The PXE pipeline conforms.
Initial computers collector schema (sketch, finalized when plugin is built):
```json
{
"identityfield": "hostname",
"fields": {
"hostname": "string, required",
"macaddress": "string, optional, secondary identity",
"osname": "string",
"osversion": "string",
"lastboottime": "datetime",
"currentuser": "string",
"ipaddress": "string",
"memorygb": "number",
"cputype": "string",
"imagename": "string (PXE image deployed)",
"imageappliedat": "datetime",
"installedsoftware": "array of {name, version}"
}
}
```
The PC re-image case is handled by the identity field: a freshly imaged PC keeps its hostname, so the existing `Asset` row is updated rather than duplicated. Existing `AssetRelationship` rows pointing at that PC (e.g., `controls` to a machine) are preserved across re-images.
## Migration of the existing endpoint
`shopdb/core/api/collector.py` (`/api/collector/pc`) is **deprecated** in v1 and **removed** before v1.0.
Migration path:
1. Implement `get_collector_schema` on the `computers` plugin. New endpoint `/api/collector/computers` is auto-registered.
2. Run both endpoints in parallel for one cycle of PXE imaging across the floor. PXE pipeline switches to `/api/collector/computers`.
3. Remove `shopdb/core/api/collector.py` and the legacy blueprint registration.
## Consequences
### Positive
- Generalizable across plugins. Sister sites adopting `printers`, `network`, etc. can wire their own collectors with no core change.
- Identity-based idempotency makes PC re-imaging safe by default.
- Audit logging is uniform across plugins.
- Schema discovery enables external tools to validate before submission.
### Negative
- Plugin authors must write a JSON schema. Slight learning curve, but JSON Schema is widely understood and the framework can ship a few examples.
- The `/api/collector/_schemas` endpoint plus per-plugin endpoints expand the public API surface; minor maintenance cost.
### Neutral
- API-key auth pattern stays as it is today (separate from JWT). Sites manage their own collector keys per plugin via env vars.
## Alternatives considered
1. **Keep `/api/collector/pc` and add new plugin-specific endpoints alongside.** Two ways to send PC data, plugin authors confused. Rejected.
2. **Use JWT for collectors instead of API key.** Collectors are headless processes (PXE pipeline, scripts), not interactive users. JWT lifecycle (refresh tokens, expiry) is the wrong tool. API key is simpler. Rejected.
3. **Plugins write directly to the database, no collector endpoint.** Skips audit logging and schema validation. Rejected.
## References
- `shopdb/core/api/collector.py` (legacy endpoint to be removed)
- `shopdb/plugins/base.py` (`get_collector_schema` + `apply_collector_payload` hooks)
- ADR-001 (asset model the collectors target)
- ADR-002 (collector schema is part of plugin contract; changes to the hook signature are major bumps)
- The PXE project (the PXE imaging project) which feeds the computers collector

123
docs/adr/ADR-007-product-versioning-and-releases.md Normal file
View File

@@ -0,0 +1,123 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How the product is versioned and released, separately from the plugin contract version.
loadwith: none
-->
# ADR-007: Product versioning and releases
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
ADR-002 established semantic versioning for the plugin contract via
`__contract_version__` in `shopdb/__init__.py`. That number answers one
question only: is a given plugin compatible with this platform build. It
says nothing about the state of the product as a whole. A sister site
standing up its own instance needs a different answer: which release am I
running, and what changed since the last one.
Until now the product had no version, no tags, no changelog, and no CI.
ADR-002's pinning model implicitly assumes that a downstream site can
pin a known-good build, but there were no tags to pin to. Adopters had no
release record to read before upgrading, and no automated gate confirming
that a given commit builds and passes tests.
## Decision
The product carries its own release version, separate from the plugin
contract version.
1. **Product version** (`shopdb/__init__.py`): a single `__version__`
constant. This is the version of the shopdb-flask product as a whole.
It follows semantic versioning of the product's user-visible and
operator-visible behavior. It is deliberately NOT part of the
`shopdb.api` contract surface, so it is never re-exported through
`shopdb.api`; exporting it would itself be a contract change under
ADR-002.
2. **Plugin contract version** (`__contract_version__`): unchanged from
ADR-002. It moves only when the plugin contract surface changes, per
the major/minor/patch rules in ADR-002.
The two are distinct series with independent bump rules. They happen to
coincide at `0.5.0` for this release; that is a coincidence of timing,
not a coupling. A product release that changes no contract surface bumps
`__version__` while leaving `__contract_version__` fixed, and vice versa.
3. **Release record** (`CHANGELOG.md`, repo root): the canonical,
human-readable record of what changed in each release, in
Keep-a-Changelog format. Every release has an entry; work in flight
accumulates under `## [Unreleased]`.
4. **Git tags**: each product release is tagged `vX.Y.Z`, where `X.Y.Z`
matches `__version__` at the tagged commit. Tags are what ADR-002's
downstream-pinning model pins to.
5. **Frontend version** (`frontend/package.json`): kept in lock-step with
`__version__` so the shipped single-page app reports the same product
version as the backend that serves it.
### Release procedure
To cut release `X.Y.Z`:
1. Bump `__version__` in `shopdb/__init__.py` to `X.Y.Z`.
2. Bump `version` in `frontend/package.json` to the same `X.Y.Z`.
3. Move the accumulated `## [Unreleased]` notes in `CHANGELOG.md` into a
new `## [X.Y.Z] - YYYY-MM-DD` section, leaving a fresh empty
`## [Unreleased]` above it.
4. If the plugin contract surface changed this release, bump
`__contract_version__` per ADR-002 (independently of `__version__`).
5. Commit, then tag: `git tag -a vX.Y.Z -m "shopdb-flask X.Y.Z"`.
6. Push the commit and the tag.
## Consequences
### Positive
- Adopters have a real release to name in bug reports and a changelog to
read before upgrading.
- ADR-002's pinning story finally has tags to pin to.
- Product and contract can evolve at their own pace without one dragging
the other into a misleading bump.
### Negative / cost
- Two version numbers to keep straight. The comment in
`shopdb/__init__.py` and this ADR exist to keep the distinction clear.
- Release discipline: the changelog and both version constants must be
updated together, or the tagged build misreports itself.
### Neutral
- The internal CI workflow runs the backend tests, the naming/style
gate, and the frontend build on push and PR. It is best-effort: the internal CI
Actions availability on the host is unverified, so the workflow is
config-only until a runner is confirmed.
## Alternatives considered
1. **Reuse `__contract_version__` as the product version.** Conflates two
independent concerns; a docs-only or UI-only release would either
falsely bump the contract or leave the product looking unchanged.
Rejected.
2. **Calendar versioning for the product** (e.g. `2026.07.0`). Easy to
bump but poor at signaling breaking operator-facing changes. Rejected
for the same reasons ADR-002 rejected it for the contract.
3. **No product version, rely on git SHAs.** Opaque to adopters and
unpinnable in any human-meaningful way. Rejected.
## References
- ADR-001 (defines the contract surface)
- ADR-002 (plugin contract versioning; `__contract_version__` bump rules)
- `shopdb/__init__.py` (`__version__`, `__contract_version__`)
- `CHANGELOG.md` (release record)
- `frontend/package.json` (frontend version, kept in lock-step)
- the internal CI workflow (CI gate)

131
docs/adr/ADR-008-plugin-migration-ownership.md Normal file
View File

@@ -0,0 +1,131 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why each plugin owns its own Alembic chain instead of adding tables to the core chain.
loadwith: none
-->
# ADR-008: Plugin migration ownership (per-plugin chains from the cutover)
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** the "Migration strategy (resolved)" section of ADR-004
## Context
ADR-004 resolved a Phase 7B footgun by folding every bundled plugin's tables
into the single core Alembic chain (migration `7c04_fold_plugin_schema`), and
later core migrations (`7d04`..`7d16`) kept adding plugin schema directly to the
core chain. At the time this was the safe choice: bundled-plugin baselines and
the core baseline had both been creating the same tables, so
`flask plugin upgrade-all` would collide with `flask db upgrade`.
That resolution left the per-plugin Alembic engine
(`shopdb/plugins/migrations.py`, `shopdb/plugins/alembic_template.py`, the
`alembic_version_<plugin>` version tables, and `flask plugin upgrade-all`) fully
built but unused for the bundled plugins. The framework is the product (per the
project's charter); a plugin that cannot own its own schema is not really a
plugin. Sister sites that adopt or fork a plugin need its schema history to
travel with the plugin, not be entangled in the host's core chain. Keeping every
future plugin table change in the core chain also means the core chain grows
without bound and a plugin can never be cleanly removed.
The blocker ADR-004 worried about (double table creation) only exists while a
plugin's own migration tries to CREATE tables the core chain already created.
That is avoidable: history is immutable, so the tables already built by the core
chain stay owned by the core chain; only NEW schema needs a new home.
## Decision
Ownership splits at a fixed cutover, the current core-chain head.
1. **Core chain owns history through its head.** The core Alembic chain
(baseline `68b3947ae14f` .. head `7d16_directoryemployees`) remains
authoritative for every table that exists at the cutover, including the
bundled-plugin tables it created. Those migrations are immutable and are not
rewritten. `flask db upgrade` continues to reproduce the full schema.
2. **Plugin chains own plugin schema going forward.** From the cutover forward,
any change to a plugin's schema lands as
`plugins/<name>/migrations/versions/000N_*.py` in that plugin's own chain,
never in the core chain. The core chain is reserved for core tables.
3. **Every table-owning bundled plugin gets a `0001` anchor.** Each such plugin
carries a migration chain whose first revision is a stamp-only no-op:
`upgrade()` does nothing because the core chain already created the tables.
The anchor exists so the plugin chain has a base that
`flask plugin upgrade-all` can stamp into the per-plugin version table
`alembic_version_<plugin>`. Blueprint-only plugins that own no tables get no
chain.
4. **Deploy and upgrade sequence.** A deploy runs `flask db upgrade`
(core chain, creates everything through the head) then
`flask plugin upgrade-all` (stamps every plugin anchor and applies any later
per-plugin migrations). The same two commands upgrade an existing install;
both are idempotent. The registry (`instance/plugins.json`) tracks each
plugin's applied revisions in `migrations_applied`.
5. **Table-ownership registry.** `PLUGIN_TABLE_OWNERS` in
`shopdb/plugins/alembic_template.py` is the explicit map of which tables each
plugin owns; it is kept in sync with the plugins' `__tablename__` declarations
and pinned by `tests/test_plugin_migrations.py`.
External (out-of-tree) plugins per ADR-003 already shipped their own chains;
this ADR brings the bundled plugins onto the same model, so there is one rule
for all plugins.
## Consequences
### Positive
- A plugin's schema history travels with the plugin. Adopting or forking sites
get the plugin's migrations, not a slice of someone else's core chain.
- The core chain stops accreting plugin schema; it stays about core tables.
- A plugin can be evolved (or, with its own downgrade, removed) independently.
- The long-built per-plugin Alembic engine is finally exercised on every deploy,
so it cannot silently rot.
### Negative / cost
- Two migrate commands per deploy instead of one. Documented in `docs/DEPLOY.md`
and `docs/UPGRADE.md`; both are idempotent so the cost is one extra safe call.
- A plugin author must now put new tables in the plugin chain and register them
in `PLUGIN_TABLE_OWNERS`, rather than autogenerating into the core chain. The
`flask plugin new` guidance and this ADR spell that out.
- The cutover is a discontinuity: tables created before it are core-owned,
tables created after it are plugin-owned. The line is the core-chain head at
this ADR's date, recorded here so it is unambiguous.
### Neutral
- No schema changes and no data migration: the anchors are no-ops. A fresh
install and an existing install converge to the same state.
## Alternatives considered
1. **Keep everything in the core chain (status quo per ADR-004).** Simplest
operationally but defeats the plugin-as-product goal: plugin schema cannot
travel, the core chain grows without bound, and plugins can never be cleanly
removed. Rejected.
2. **Rewrite history so plugin tables move out of the core chain into plugin
`0001` CREATE migrations.** Would make each plugin chain self-contained from
empty, but breaks the immutability rule, forces every existing site to
re-run a rewritten chain, and re-introduces the exact double-creation footgun
ADR-004 fixed. Rejected.
3. **Anchor that CREATEs tables with `IF NOT EXISTS` guards.** Lets a from-empty
install build plugin tables from the plugin chain, but then two chains both
claim the same tables and drift can diverge silently. The no-op anchor keeps
a single authoritative creator (the core chain) for cutover-era tables.
Rejected.
## References
- ADR-003 (plugin distribution; external plugins already ship chains)
- ADR-004 (deployment topology; this ADR supersedes its migration-strategy note)
- `shopdb/plugins/alembic_template.py` (`PLUGIN_TABLE_OWNERS`, shared env runner)
- `shopdb/plugins/migrations.py`, `shopdb/plugins/cli.py` (`upgrade-all`)
- `plugins/<name>/migrations/` (per-plugin chains and `0001` anchors)
- `tests/test_plugin_migrations.py` (ownership + chain + idempotency guards)
- `docs/DEPLOY.md`, `docs/UPGRADE.md`, `docs/PLUGINS.md` (deploy sequence)

156
docs/adr/ADR-009-frontend-plugin-gating.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why the SPA gates plugin routes, so a disabled plugin URL redirects instead of breaking.
loadwith: none
-->
# ADR-009: Frontend plugin gating
- **Status:** ACCEPTED
- **Date:** 2026-07-10
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
The backend plugin system (ADR-002, ADR-003) lets an operator disable a
plugin. Disabling it unregisters the plugin's API blueprint, drops its
rows from global search (see `test_search_disabled`), and removes its
entry from `get_navigation_items()`, so the disabled feature's sidebar
link disappears.
The frontend told a different story. Every plugin's Vue routes and views
ship inside the core bundle. `frontend/src/router/index.js` auto-discovers
them with `import.meta.glob('./routes/*.js')`, so a route like `/usb` or
`/printers/3` is registered regardless of whether the owning backend
plugin is enabled. A user who typed the URL, followed a stale bookmark,
or clicked a cross-link reached a page whose API calls all 404, landing
on a broken shell instead of a clean redirect. The navigation was already
data-driven; direct-URL reachability was the gap.
There is no frontend plugin system yet. The product is packaged as one
core Flask app plus backend-only plugins; the Vue app is monolithic and
build-time static. Any gating has to live in core frontend code because
that is the only place the plugin routes exist.
## Decision
### Step 1 (this ADR, implemented): route-level gating
Plugin-owned frontend routes are gated against the backend's enabled-plugin
list. This is the whole of what ships now.
1. **Enabled-list endpoint.** A new `GET /api/plugins/enabled`
(`jwt_required(optional=True)`) returns a flat JSON array of enabled
plugin name strings and nothing else. It is a cheap registry read
(`registry.get_enabled_plugins()`), no database access. Exposing it to
anonymous callers is safe because `GET /api/dashboard/navigation`
already leaks the same enabled/disabled signal, and unauthenticated
kiosk routes (`/tv`) need the answer too. It carries no metadata, so it
reveals strictly less than the equally-anonymous but metadata-carrying
`GET /api/plugins` (also `jwt_required(optional=True)`; only the
`PUT /api/plugins/<name>` toggle is admin-gated).
2. **Route tagging.** Every plugin-owned route carries `meta.plugin =
'<pluginname>'`. This covers the per-plugin route modules
(`routes/computers.js`, `routes/equipment.js`, ...) and the
plugin-owned routes that physically live in core files: the
PC-relationships and toner reports, the slide manager and `/tv`
dashboard (slides), the printer-QR and USB-label print pages, and the
employee-detail page. Genuinely core routes (dashboard, search, map,
applications, reference-data settings) stay untagged and are never
gated.
3. **Cached fetch, fail-open.** A composable
(`composables/enabledPlugins.js`) fetches the list exactly once behind
a cached promise. If the fetch fails or returns a non-array, the code
fails **open**: every plugin is treated as enabled. A transient API
error must never brick navigation. The cost is that a disabled
plugin's page is briefly reachable during an outage, which is
acceptable because its API calls would fail anyway and the next
successful fetch closes the gap.
4. **Router guard.** `router.beforeEach` awaits the cached fetch when the
target route has `meta.plugin`. If that plugin is not enabled it
redirects to `/` and raises an info toast. The endpoint is
jwt-optional, so the guard works for both authenticated pages and the
unauthenticated `/tv` kiosk route.
This is intentionally a thin layer. It does not change the plugin
contract surface, so `__contract_version__` does not move: adding a core
HTTP endpoint and tagging core-shipped routes are not plugin-contract
changes. Plugins still ship no frontend code of their own.
### Non-goals for step 1
- No build-time or runtime loading of plugin-authored Vue code.
- No per-permission or per-role route gating (that stays with the
existing `requiresAuth` / `requiresAdmin` meta flags).
- No removal of disabled routes from the route table; they remain
registered and are intercepted by the guard. Keeping them registered
avoids a rebuild when a plugin is toggled and keeps the redirect path
simple.
## Future direction (PROPOSED, not implemented)
Step 1 gates routes that core already owns. The longer-term goal is a
real frontend-plugin contract where a plugin ships its own frontend and
core discovers it, mirroring the backend model. Sketch:
1. **Plugin-owned frontend tree.** Each plugin gains a
`plugins/<name>/frontend/` directory holding its route module, views,
and any plugin-specific components. Core stops carrying `views/usb`,
`views/printers`, and so on.
2. **Build-time discovery.** The Vite build discovers plugin frontends
with a glob over `plugins/*/frontend/routes.js` (analogous to today's
`import.meta.glob('./routes/*.js')`), so a plugin's presence in the
tree is what puts its routes in the bundle. Combined with the step-1
enabled-list gate, a plugin that is absent from the build ships no
code and a plugin that is present-but-disabled is route-gated at
runtime.
3. **Shared component + registration contract.** Plugins register into
named extension points instead of editing core files: an `iconMap`
registration for nav/asset icons, asset-detail panels, map-marker
renderers, and search-result renderers (the "Frontend hook contract"
already listed as deferred in docs/ROADMAP.md). Core exposes a
stable set of shared components (form controls, detail-page shells,
table primitives) as the plugin frontend's only allowed core imports,
the frontend analogue of the `shopdb.api` namespace.
4. **Versioned frontend contract.** The shared-component and
registration surface would be versioned the same way the backend
contract is (ADR-002), so a plugin frontend can declare the core
frontend range it needs.
### Tradeoffs of the future direction
- **Pro:** true plugin self-containment; a site can drop in or remove a
plugin (frontend and backend together) without patching core; smaller
core; clearer ownership.
- **Con:** significant build-system work (per-plugin Vite entry
discovery, code-splitting, dev-server HMR across the plugin tree); a
new versioned frontend contract to maintain and document; a migration
that moves ten plugins' worth of views out of core; risk of a leaky
shared-component surface becoming an accidental contract. The payoff
only matters once external (out-of-tree) plugins with their own
frontends are a real requirement. Until then, step 1's route gating
delivers the user-visible correctness (no reachable dead pages) at a
fraction of the cost.
## Consequences
- Disabling a backend plugin now makes its frontend routes redirect to
the dashboard instead of loading a broken shell. Behavior matches the
already-dynamic navigation.
- One extra lightweight request at app boot (`GET /api/plugins/enabled`),
cached for the session.
- Fail-open means gating is a UX guardrail, not a security control. It is
not a substitute for backend authorization: the API still enforces auth
and the disabled plugin's endpoints are simply unregistered. Never rely
on route gating to protect data.
- The future frontend-plugin contract remains open work; this ADR records
the direction and its cost so a later decision can pick it up.

326
docs/adr/ADR-010-frontend-plugin-hooks.md Normal file
View File

@@ -0,0 +1,326 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: The hook contract a plugin uses to contribute UI without hand-editing core Vue files.
loadwith: none
-->
# ADR-010: Frontend plugin hook contract
- **Status:** ACCEPTED
- **Date:** 2026-07-11
- **Accepted:** 2026-07-11
- **Deciders:** ShopDB maintainers
- **Supersedes:** none
## Context
The backend plugin contract is settled (ADR-002 surface, ADR-006 collector,
ADR-008 migrations, ADR-009 route gating). The one undefined piece before 1.0 is
the frontend: how a plugin contributes UI without hand-editing core Vue files.
CONTRACT-STABILITY.md names this the single biggest churn item ("no server-side
hook for asset-detail panels, map markers, or search-result rendering; a plugin
that needs custom UI still hand-edits the Vue frontend").
The `measuringtools` plugin (bundled 2026-07-11, whose construction is narrated
in docs/PLUGIN-GUIDE.md) is the fresh evidence. Integrating it needed exactly
**two hand edits to core frontend files**:
1. `frontend/src/api/index.js` - an api-client block appended after
`warrantyApi` (PLUGIN-GUIDE.md section 9). The scaffolder now emits a
paste-in snippet, but it is still a hand edit to a shared, churn-heavy file.
2. `frontend/src/views/settings/settingsNav.js` - a card entry in the hardcoded
`settingsGroups` catalog (the "Measuring Tools" group, ~line 51), consumed by
`SettingsLayout` (left rail) and `SettingsIndex` (landing overview).
Everything else integrated with **zero** core edits, because a mechanism already
existed for it:
| Capability | Existing mechanism |
|---|---|
| Views / routes | Auto-discovery: `frontend/src/router/index.js` globs `router/routes/*.js` via `import.meta.glob`; a route starting `settings/` auto-nests under the settings shell. |
| Disabled-plugin gating | `meta: { plugin: '<name>' }` per ADR-009; router guard redirects when the backend plugin is off. |
| Sidebar / dashboard / reports | Declarative backend hooks rendered by core: `get_navigation_items`, `get_dashboard_widgets`, `get_reports` on `BasePlugin` (shopdb/plugins/base.py), merged by `shopdb/core/api/dashboard.py` and `shopdb/core/api/reports.py`. |
| Map placement | Data-driven off asset types + resolved positions (ADR-001); a typed, positioned asset appears on the map with its type color and no plugin-side map code. |
Capabilities a plugin **cannot have at all today** - no hook exists, and adding
one would mean forking a core view:
3. **Asset-detail extension panels.** A warranty-coverage section on a PC or
printer detail page, a calibration-status card on a measuring tool. Today
`WarrantyPanel.vue` is composed in by hand-editing each detail view.
4. **Map marker / overlay contributions.** A calibration-due badge on the
shop-floor map (`frontend/src/views/MapView.vue`). The map draws type
colors but has no plugin decoration path.
5. **Search-result rendering / routing for plugin asset types.** Global search
returns assets, but core hardcodes how each type renders and where its detail
link points; a plugin asset type has no way to declare its icon or route.
There is also an unsolved distribution wrinkle from ADR-003: external plugins
symlink into `plugins/<name>/` backend-only, but any frontend file they carry
must be physically copied into `frontend/src/`, because Vite compiles the tree
at build time and cannot reach outside it.
## Options considered
### A. Runtime dynamic component registration
Plugins ship real Vue components that core loads and mounts at runtime into
named extension points (an `iconMap` registration, a panel registry, a marker
renderer registry). This is the richest model and the one ADR-009's "Future
direction" sketched.
- Pro: a plugin can render anything; no core generic-renderer ceiling.
- Con: requires runtime module loading of plugin-authored code (dynamic import
of built chunks), a versioned shared-component surface that becomes an
accidental contract the moment it leaks, and it does nothing about the
build-time-only reach of Vite for external plugins. It is the most code for
the least near-term payoff. ADR-009 already priced this and deferred it.
### B. Declarative data-only hooks rendered by core generic components
Plugins return plain dicts from new `BasePlugin` hooks; a generic core component
renders them. This is the exact precedent already proven three times:
`get_navigation_items`, `get_dashboard_widgets`, and `get_reports` (0.6.0) all
return dicts that a core consumer endpoint merges and a core Vue component
renders. No plugin ships frontend code.
- Pro: additive, minor-bump changes under ADR-002; identical access pattern to
the existing consumers (skip disabled, fail-loud in dev, isolate in prod);
works untouched for external symlink-only plugins, because the data crosses
the wire and core owns the renderer.
- Con: bounded to what a generic renderer can draw. A panel that needs a bespoke
chart or a custom-interaction map overlay does not fit.
### C. Build-time file-convention discovery
Extend the `import.meta.glob` precedent from `./routes/*.js` to a plugin-owned
frontend tree (`plugins/*/frontend/`), so a plugin's real components (full
list/detail/form views, its api-client module) live with the plugin and the
build picks them up by convention. This is ADR-009 "Future direction" steps 1-2.
- Pro: true self-containment for genuine components; the natural home for the
api-client block (friction 1) and full views.
- Con: significant build-system work (per-plugin Vite entry discovery,
code-splitting, dev-server HMR across the tree) and it does not by itself
solve the external-plugin wrinkle - a symlinked out-of-tree `frontend/` is
still outside Vite's compiled root. Only pays off once out-of-tree plugins
with frontends are a real requirement.
### Hybrid
The evidence splits cleanly. The five friction points fall into two buckets:
surfaces where a generic renderer fed by plugin data is sufficient (2, 3, 4, 5),
and surfaces where a real component is genuinely unavoidable (full views, and
the api-client module in friction 1). Option B fits the first bucket exactly and
is cheap and additive. Option C is the right long-term answer for the second but
is expensive and, per ADR-009, gated on external-plugin demand. Option A buys
nothing B does not, at the highest cost. The decision is B now, C deferred, A
not pursued.
## Decision
**DECISION:** adopt a hybrid. Add **data-only declarative hooks** (Option B) for
the four presentation surfaces a generic core renderer can serve, and keep
**file-convention glob discovery** (Option C) as the deferred mechanism for the
residual cases where a real component is unavoidable. Do not pursue runtime
component registration (Option A).
### New hooks (Option B, data-only, additive)
Each is a new optional `BasePlugin` method returning a list of dicts, merged by
a core consumer endpoint using the same pattern as `get_reports` (inject
`plugin` name, skip disabled, re-raise in dev/test, log-and-isolate in prod),
and rendered by a generic core component. Icon values are string keys mapped to
Lucide components core-side, exactly like `get_navigation_items`.
**`get_settings_cards`** - resolves friction point 2 (settingsNav.js hand edit).
```
{
'group': 'Measuring Tools', # rail group title (created if new)
'to': '/settings/measuringtooltypes',
'icon': 'ruler', # string key, mapped core-side
'title': 'Measuring Tool Types',
'description': 'Manage measuring-tool subtypes + map colors',
'position': 22, # order within the group
}
```
Consumer: a new `GET /api/settings/cards` merges enabled plugins' cards into the
core `settingsGroups` catalog; `SettingsLayout` and `SettingsIndex` read the
merged catalog instead of the hardcoded JS array. The catalog's own core groups
stay in `settingsNav.js`; plugin groups are appended.
**`get_asset_panels`** - resolves friction point 3 (asset-detail panels).
```
{
'id': 'calibration',
'title': 'Calibration',
'assettypes': ['measuring_tool'], # detail pages it appears on; ['*'] = all
'endpoint': '/api/measuringtools/{assetid}/calibration-panel',
'render': 'keyvalue', # 'keyvalue' | 'table' | 'badge'
'position': 20,
}
```
Consumer: `GET /api/assets/{assetid}/panels` returns the panels whose
`assettypes` match that asset's type; a generic `AssetPanel` component on the
detail page fetches each `endpoint` and renders it in the declared style. This
covers the warranty-coverage and calibration-status cases. A panel that needs
bespoke UI (a chart) is out of scope for the data-only hook and falls to the
deferred component mechanism below - stated honestly, not hidden.
**`get_map_overlays`** - resolves friction point 4 (map marker/overlay badges).
```
{
'id': 'calibration-due',
'label': 'Calibration due', # legend label
'endpoint': '/api/measuringtools/map-overlay', # -> [{assetid, color, label}]
'style': 'badge', # 'badge' | 'ring'
'legend': True,
}
```
Consumer: `MapView.vue` fetches enabled plugins' overlay endpoints and decorates
the already-placed markers; legend entries append to the existing legend. The
map stays data-driven; plugins add decoration data, not map code.
**`get_asset_presentation`** - resolves friction point 5 (search rendering /
routing) and, as a bonus, removes the `AppLayout.vue` iconMap hand edit called
out in PLUGIN-GUIDE.md section 9.
```
{
'assettype': 'measuring_tool', # AssetType.assettype key the plugin owns
'icon': 'ruler',
'label': 'Measuring Tool',
'route': '/measuringtools/{assetid}', # detail-route pattern
}
```
Consumer: `GET /api/assets/presentation` returns the type-to-presentation map;
global-search result rows and any asset cross-link use it to pick the icon and
build the detail link, so core never hardcodes a plugin's route or icon.
### Friction map
| Friction point | Mechanism | Hook / change |
|---|---|---|
| 1. api-client block in api/index.js | C (deferred) | plugin-owned `frontend/` tree, glob-discovered; scaffolder snippet is the near-term mitigation |
| 2. settingsNav.js card entry | B (now) | `get_settings_cards` |
| 3. asset-detail panels | B (now) | `get_asset_panels` (bespoke panels -> C, deferred) |
| 4. map markers / overlays | B (now) | `get_map_overlays` (bespoke overlays -> C, deferred) |
| 5. search-result rendering / routing | B (now) | `get_asset_presentation` |
| views (full list/detail/form) | C (deferred) | already glob-discovered under core `routes/*.js`; long-term move to `plugins/*/frontend/` per ADR-009 |
### Deferred: file-convention frontend tree (Option C)
Extending `import.meta.glob` from `./routes/*.js` to `plugins/*/frontend/` (so
views and the api-client module live with the plugin) is ADR-009 "Future
direction" steps 1-2. It is deferred for the same reasons ADR-009 gave: it is
build-system-heavy and the payoff only lands once out-of-tree plugins with their
own frontends are a real requirement. Until then, friction 1 stays mitigated by
the scaffolder snippet, and full views keep shipping in the core bundle and
gated by ADR-009 `meta.plugin`.
This is also the honest limit on the external-plugin wrinkle. The four data-only
hooks need **zero** frontend files from a plugin, so an external symlink-only
(backend) plugin gets settings cards, detail panels, map overlays, and search
presentation with no copy-into-`frontend/src/` step at all. The wrinkle survives
only for the residual component-backed cases, which is exactly the deferred
Option C work; pushing external-repo frontend distribution to that later ADR
matches ADR-003's posture of deferring out-of-tree packaging until two sites run
their own plugins.
### Contract-version impact (ADR-002)
Each of the four hooks is a new optional `BasePlugin` method - an additive
change, so a **minor** bump per ADR-002, the same classification `get_reports`
took at 0.6.0. Landing all four together is a single minor bump (proposed
**0.7.0**); landing them incrementally is one minor bump each. Adding the core
consumer endpoints and the generic renderer components is core-internal and does
not itself move `__contract_version__`. The deferred Option C introduces a
separate, versioned **frontend** contract (ADR-009 step 4), tracked apart from
the backend `__contract_version__`; it is not part of this proposal's bumps.
## Consequences
### Positive
- Closes the CONTRACT-STABILITY.md "single biggest gap" for the four surfaces a
generic renderer can serve, using the already-proven declarative pattern - low
risk, low cost, additive-only.
- External symlink-only plugins get four presentation surfaces with no
copy-into-core step, shrinking (not yet eliminating) the ADR-003 wrinkle.
- Removes three of the two-plus hand edits the exemplar needed (settingsNav
card, plus the AppLayout iconMap edit from section 9), moving them to data.
- Docs-drift guard forces documentation: `tests/test_docs_contract.py` fails if
any new public `BasePlugin` hook is missing from `docs/PLUGIN-HOOKS.md`, so the
hooks cannot ship undocumented.
### Negative
- Four more hooks and four more core consumer endpoints to maintain, each with
the skip-disabled / fail-loud-in-dev / isolate-in-prod discipline.
- The generic renderers set a ceiling: bespoke panels and overlays still have no
home until Option C lands, so the contract is honest-but-partial, not total.
- One extra request per surface on the pages that use it (a detail page fetches
its panels; the map fetches overlays), on top of ADR-009's enabled-list fetch.
### Neutral
- Bundled plugins keep their current in-core views; migrating them to the hooks
is opt-in and incremental (see Adoption plan), not a flag-day rewrite.
- `meta.plugin` route gating (ADR-009) is unchanged and still gates the views;
these hooks add presentation, not routing.
- Contract tests (`tests/test_plugin_contract.py`) already assert every public
hook is exercised; the new hooks slot into that harness.
## Adoption plan
Prove the contract on bundled plugins before declaring it settled for sister
sites.
1. **`get_asset_panels` first, via `warranty`.** Warranty is asset-general and
already composes `WarrantyPanel.vue` onto multiple detail pages by hand, with
a clean per-asset endpoint behind it. Migrating it to a declarative
`get_asset_panels` entry rendered by the generic `AssetPanel` is the
lowest-risk proof and immediately removes hand edits from every detail view
that shows warranty. This is the recommended first migration.
2. **`get_settings_cards`, `get_map_overlays`, `get_asset_presentation` via
`measuringtools`.** The guide exemplar already needs a settings card, a
calibration-due map badge, and search routing for its `measuring_tool` type,
so it exercises all three at once and its PLUGIN-GUIDE.md walkthrough becomes
the reference for the new hooks.
3. Only after both plugins run on the hooks in a real build: bump
`__contract_version__` to 0.7.0, document the hooks in
`docs/PLUGIN-HOOKS.md`, and mark this ADR ACCEPTED.
## Open questions
- Should `get_asset_panels` `render` styles stay a small closed set
(`keyvalue` / `table` / `badge`), or grow? A closed set keeps the renderer
generic; growth pressure is the signal that a case actually needs Option C.
- Should the four consumer endpoints collapse into one bundled
`GET /api/plugins/frontend-contributions` call to save round-trips, or stay
separate per surface for cache locality? Defer until the per-surface request
cost is measured.
- When Option C lands, does the api-client module move under
`plugins/*/frontend/` or get replaced entirely by a generated client from the
backend blueprint? Out of scope here; belongs to the ADR-009 frontend-contract
follow-up.
## References
- ADR-001 (asset model as the map/search data source)
- ADR-002 (bump classification for the new hooks)
- ADR-003 (external-plugin distribution posture)
- ADR-009 (route gating; "Future direction" is the deferred Option C)
- docs/PLUGIN-GUIDE.md sections 9-10 (the measuringtools frontend hand edits)
- docs/CONTRACT-STABILITY.md (the expected-churn line this ADR answers)
- shopdb/plugins/base.py (existing declarative hooks this pattern extends)
- shopdb/core/api/dashboard.py, shopdb/core/api/reports.py (consumer precedent)

75
docs/adr/ADR-011-machines-rename.md Normal file
View File

@@ -0,0 +1,75 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why the equipment domain was renamed to machines and machinetypes became modeltypes.
loadwith: none
-->
# ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes
- **Status:** ACCEPTED
- **Date:** 2026-07-11
- **Deciders:** ShopDB maintainers
- **Relates to:** ADR-005 (equipment vs measuringtools scope), ADR-008 (plugin migration ownership)
## Context
ADR-005 split the old "equipment" concept: gage-lab instruments moved to the
measuringtools plugin, leaving the equipment plugin holding exactly one thing -
shop-floor machines (lathes, mills, CMMs, grinders, furnaces). The UI, the
routes (`/machines`), the frontend view folder (`views/machines/`), and even
the data vocabulary (asset numbers headed "MACHINE #") already said machines.
Only the code identity still said equipment: the plugin name, the API prefix,
the `equipment`/`equipmenttypes` tables, and the `assettype='equipment'` data
value. Every future contributor would pay a small permanent translation tax.
A display-label-only swap was shipped first (2026-07-11) and immediately showed
the dissonance: UI Machines over code equipment forever.
Separately, the legacy `machinetypes` table survived the Machine-model
retirement for one real reason: it types the vendor **models catalog**
(`models.machinetypeid`, 95 of 127 models) with fine-grained, cross-category
types (Lathe, Switch, Laser Printer). Its name was a misnomer - it never typed
machines specifically - and it squatted on the name the machines plugin's
subtype table wants.
The economics favor doing this now and never again: the plugin contract is
pre-1.0 (breaking changes are cheap per ADR-002), no sister site has adopted
yet, there is no production instance, and the test suite plus CI give the
strongest safety net the project has had.
## Decision
Rename the domain in one coordinated pass:
| Current | New | Rationale |
|---|---|---|
| `plugins/equipment/`, `/api/equipment`, `equipment.*` perms | `plugins/machines/`, `/api/machines`, `machines.*` | plugin identity matches the domain |
| `equipment` table, `equipmentid` | `machines`, `machineid` | main entity |
| `equipmenttypes`, `equipmenttypeid`, `equipmenttype` | `machinetypes`, `machinetypeid`, `machinetype` | subtypes take the freed name |
| legacy `machinetypes`, `models.machinetypeid` | `modeltypes`, `models.modeltypeid` | role-accurate: it types the models catalog, losslessly (no collapse to the 4 asset types) |
| `assettypes.assettype = 'equipment'` (data) | `'machine'` | the string compare sites flip with it |
Migration placement follows ADR-008: the modeltypes rename and the data flips
land in the core chain (core tables, core data); the plugin table renames land
in the machines plugin's own chain, idempotently guarded so fresh installs
(whose core chain still creates the old names at `7c04`) rename on first
`flask plugin upgrade-all`. The `alembic_version_equipment` version table is
renamed to `alembic_version_machines` by the core migration so the plugin chain
resumes seamlessly on existing installs. The plugin registry auto-migrates an
`equipment` key to `machines` on load.
Historical migrations and accepted ADRs are not rewritten; old ids and ADR-005
wording remain as history.
## Consequences
- Upgrading instances run `flask db upgrade` then `flask plugin upgrade-all`,
same as every deploy; both migrations are idempotent.
- `/api/equipment` and `equipmentApi` cease to exist - a breaking change made
while there is nobody to break. After contract 1.0 this class of rename
would be a major-version event; this is the last cheap moment.
- The "Machine Types" settings card becomes "Model Types" and finally says
what it does; the machines plugin's subtype card owns the machine-types name.
- Contributors stop translating between UI vocabulary and code vocabulary.

137
docs/adr/ADR-012-geenforce-manifest-ownership.md Normal file
View File

@@ -0,0 +1,137 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: historical
summary: The record of why ShopDB owned GE-Enforce manifests as data. The plugin it decided is not bundled here.
loadwith: none
-->
# ADR-012: GE-Enforce manifest ownership in shopdb
- **Status:** ACCEPTED
- **Date:** 2026-07-13
- **Deciders:** ShopDB maintainers
- **Relates to:** ADR-002 (plugin contract versioning), ADR-004 (per-site
deployment), ADR-006 (collector contract), ADR-008 (per-plugin Alembic chains)
> **Not bundled on this line.** The `geenforce` plugin this ADR decided is not
> part of this product build, so nothing here is a procedure to follow. It is
> kept because it is a decision record and because ADR-013 and the resource-scope
> design on API tokens both trace back to it. The plugin, its client and its
> pages live on the site line that still runs them.
## Context
GE-Enforce is a desired-state enforcement system for shopfloor PCs: a PowerShell
engine (`Install-FromManifest.ps1`) reads per-PC-type `manifest.json` files off
an SMB share every logon and installs / self-heals what they declare. Authoring
those manifests today means hand-editing JSON on a file share, and there is no
central view of what each PC actually did.
We want shopdb to own the manifests as data (author, version, publish, roll
back) and to observe fleet compliance, while NOT taking on the GE-Enforce engine
itself (which is the GE-Enforce framework's, maintained separately) and NOT
dictating any site's imaging path (per ADR-004, each site is single-tenant with
its own provisioning - PXE at the reference site, OOBE provisioning packages at
others).
The manifests are an enforcement PROGRAM, not an application inventory: entry
`Type` is not an app/config discriminator and entry `Name` is a manifest label,
not a Windows ARP DisplayName. Any design that treats them as an app catalog is
wrong.
## Decision
Build a bundled `geenforce` plugin that owns the manifest as shopdb data, with a
client kit and a deployment bootstrap. Specifically:
1. **Data model.** One wide `manifestentries` table with an `entrytype`
discriminator and nullable per-type columns (not SQLAlchemy STI, not a JSON
blob - the fleet is ~64 entries, so sparse columns are free and stay
queryable). Scopes are `manifestscopes`, unique on `(scopename, phase)`;
runtime is per-pctype scopes, preinstall is one flat scope. Multi-value gates
(PCTypes / hostnames / machine numbers) and the nested InUseCheck are child
tables. `sortorder` is the execution-order contract. RegValue is stored as a
raw JSON literal so DWord-vs-string typing survives. Per-plugin Alembic chain
(ADR-008).
2. **Published snapshots.** Editing touches a DRAFT only. Publish freezes the
rendered JSON document into an immutable `manifestpublishedversions` row; the
client is ALWAYS served the current published snapshot, never the draft;
rollback flips `iscurrent` to an older version. Freezing the document (not
row-mirroring) makes immutability structural.
3. **Behavioral-parity gate, not byte-identity.** A DB-free harness
(`parity.py`) imports each real manifest and renders it back, then proves
BEHAVIORAL equivalence (same ordered entries with identical detection /
targeting, and the same entries fire across machine-profile fixtures) - never
byte equality, which re-serialization would never satisfy. This gates any
build that touches the model.
4. **Filter mirror; engine is the single source of truth.** `filters.py`
mirrors the engine's four gate functions and alias graph for the "what would
this PC get" simulator and parity. The engine lib stays authoritative;
shopdb mirrors it (never the reverse). PCTypesStrict is honored only for the
preinstall phase, matching the runners.
5. **Payload integrity is separate from detection.** For `http`/`inline`
payloads a dedicated `payloadsha256` is verified before running - independent
of `DetectionMethod` (DetectionValue is a hash only for `Hash` detection).
`smb` payloads keep the share ACL as their trust boundary. Large binaries
stay on SMB; small config/scripts may move to http/inline later.
6. **Observed-state reporting.** Each PC POSTs its enforcement result;
`manifestenforcementreports` (+ results) records the applied version
(received-latest) and per-entry self-heal / failure. Status derives from
explicit self-heal flags only, never the raw installed count (Always/no-
detection scripts install every cycle without being drift corrections).
7. **Service-token auth.** Client endpoints authorize via managed service
tokens scoped `geenforce.fetch` / `geenforce.report`, through a new
`service_token_authorized(scope)` on the `shopdb.api` contract surface
(contract 0.11.0). Admin CRUD uses `geenforce.manage` / `geenforce.publish`.
8. **Client + deployment, engine referenced not vendored.** shopdb ships the
fetch/report kit (`plugins/geenforce/client/`) and a site-neutral bootstrap
(`Install-GEEnforce.ps1`) that provisions a PC's identity
(`C:\Enrollment\pc-type.txt` etc. - what determines the PC type; there is no
auto-detection, the provisioner supplies it), the shopdb registry config, and
the scheduled task. The GE-Enforce ENGINE is referenced (`-EngineSource`),
not carried by shopdb. Deployment is provisioning-path independent (PXE step,
OOBE ppkg, Intune, manual); the runtime task is fail-safe.
9. **Milestone 1 = export to share; staged cutover.** Until a site cuts its
client over to shopdb-sourced manifests, the plugin publishes and EXPORTS the
manifest to the share (with a `_meta/history` backup, atomic write); the
unchanged engine picks it up. Cutover is staged: shadow mode (fetch from
shopdb AND read the share, log diffs, install from share) then read cutover.
10. **No application auto-seeding.** The core Applications catalog already
tracks these apps (from the classic-shopdb migration) with version
histories; auto-creating Applications from manifest labels produced
duplicates and misclassified config drops. Application linkage, if wanted, is
a curated manifest-entry -> existing-Application link, not label scraping.
## Consequences
- **Positive.** Manifests become validated, versioned, publishable data with
one-click rollback and a fleet-compliance view; desired-state and observed-
state live in one system. The parity gate + published snapshots + separate
payload hash make a fleet-wide-SYSTEM system safe to author. The plugin is
provisioning-agnostic, so any GE Aerospace site can adopt it regardless of
imaging path. Validated end to end: parity green against the real manifests,
and the client kit + installer proven on a Windows VM (PS 5.1) and Linux
pwsh 7.
- **Boundaries / risks.** The engine remains the GE-Enforce framework's, so
shopdb's parity mirror must be kept in sync with the lib (guarded by the parity
fixtures; the plugin pins lib >= 2.6 for `_CmmVersion`). Provisioning writes
the PC identity - shopdb cannot set a PC's type at imaging (a PC is unknown
until it enrolls and reports). Manifest-label vs ARP-name mismatch means the
catalog link, when built, needs a curated alias layer.
- **Deferred.** Desired-vs-observed per-entry compliance (needs a collector
installedVersions field); curated manifest-entry -> Application linking; the
live client cutover (a site operational decision); inline payload upload.
The design note, the concepts page, the fetch/report contract and the client
deployment page were all written against this plugin and stay with it on the
site line that carries it. None of them is in this repository.

435
docs/adr/ADR-013-plugin-catalog-and-lean-builds.md Normal file
View File

@@ -0,0 +1,435 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: How a site gets only the plugins it chose, and why curation plus signing is the security model.
loadwith: none
-->
# ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds
- Status: ACCEPTED
- Date: 2026-07-18
- Deciders: ShopDB maintainers
- Relates to: ADR-002 (contract versioning), ADR-003 (plugin distribution), ADR-004 (per-site instances), ADR-008 (per-plugin migrations), ADR-009 (frontend plugin gating), ADR-010 (frontend hook contract)
## Context
Every site today ships identical code. The backend image contains all 13 plugin
directories (Dockerfile COPY at line 55; the header comment listing "eleven core
plugins" is stale), and the SPA compiles every plugin's routes and views via a
static glob (frontend/src/router/index.js:11) plus hardcoded imports. A site's
"chosen set" exists only as runtime enable flags in instance/plugins.json.
Disabled is not absent: a site that never wants printedparts/usb/network still
ships, and can execute, that code.
Distribution per ADR-003 is "drop a directory into plugins/ by hand". There is
no artifact format, no signing, no catalog, no validate gate, and the loader
trusts whatever it finds on disk (loader.py:61-73 discovers any folder with a
plugin.py; loader.py:185-224 loads it; migrations.py:29-62 runs its DDL with
full DB privileges). Plugins run in-process with the full shopdb.api surface
including db, so the only tenable security model on air-gapped GE networks is
curation plus cryptographic provenance, enforced everywhere code can execute,
not sandboxing.
Known defects this ADR also resolves:
- upgrade_all_plugins checks hasattr(registry, 'list_installed') which never
exists (registry has only get_all/get_enabled_plugins), so it always falls
back to migrating every folder on disk, adopted or not (__init__.py:94).
- PILOT-DEPLOY.md enables plugins that were never installed; enable refuses.
There is no declarative "apply this chosen set" operation.
- Reverse-dependency checks on uninstall/disable read only LOADED plugin
instances, so an installed-but-unloaded dependent is invisible.
- Dependency install/enable is check-only; nothing computes a closure, and
install-a-dependent can fail at its own load step because the dependency was
installed disabled (default_enabled=false on employees).
- The dependency sort has no cycle detection.
- Soft couplings (geenforce -> computers, notifications -> employees) are
invisible to the manifest graph.
Frontend reality check (this drove the design below): plugin UI is NOT one
folder per plugin. Routes live in routes/<plugin>.js (slides has none), inside
the shared core.js (computers report, printers toner report, employees detail,
slides settings), and as six hardcoded top-level imports in index.js itself
(/parts-kiosk, /tv, /print/printer-qr x2, /print/usb-labels,
/print/printedparts-labels). View dirs mismatch plugin names (computers ->
views/pcs). Plugin settings cards sit in shared views/settings/
(DellWarrantySettings, ZabbixSettings, SlideManager, EmployeeDirectory,
MeasuringToolTypesList, PrintedPartsSettings), plugin print views in shared
views/print/, and some views span plugins (AssetLabel.vue serves five asset
types; PCDetail imports WarrantyPanel). Any lean-frontend design that only
moves routes/*.js and views/<plugin>/ fails the build the moment a plugin is
pruned. This ADR scopes that work honestly instead of calling it mechanical.
## Decision
### 1. Tiering: mandatory core is the core package; all plugins are catalog-optional
- The mandatory core is the non-plugin shopdb/core/ package (auth, users,
assets, locations, vendors, models, settings, audit, dashboard, search,
reports, plugin management). It already survives every plugin being absent
via hasattr/lazy-import guards. No plugin is promoted into it.
- New optional manifest field `tier: "core" | "optional"`, default "optional".
All 13 existing manifests are unchanged and unchanged in meaning. The
lifecycle gains a guard: uninstall_plugin and disable_plugin refuse a
tier:core plugin (alongside the reverse-dependency checks at
__init__.py:269-278 and :351-360). No plugin ships tier:core initially; the
field and guard exist so a future curation decision is a manifest edit, not a
framework change.
- Per-site mandates live in the site profile (section 5): a `locked` list the
profile applier refuses to remove. This preserves ADR-004 site autonomy: a
wing site can mandate usb without the framework mandating it fleet-wide.
- New manifest field `optional_dependencies: []` (names only, loader-ignored).
Declared for the verified soft couplings: geenforce lists computers
(service.py:32-43 loses app-detection gates without it), notifications lists
employees (routes.py:151-171 loses name/photo enrichment). Catalog listing
and adopt WARN on unmet optional deps; nothing blocks.
- Hard `dependencies` gains optional PEP440 ranges ("employees>=1.1").
validate/adopt honor ranges; the runtime loader keeps name-only semantics
(specifier stripped) so no loader behavior changes. The single existing hard
edge printedparts -> employees stays as-is; whether it can relax to optional
(badges.py has an external HR fallback) is a follow-up product question, not
blocked on this ADR.
- Dependency plumbing fixes: _sort_by_dependencies gains cycle detection
(raise PluginDependencyError on a back edge); reverse-dependency checks read
manifests of ALL installed plugins from disk, not loaded instances.
### 2. Packaging: signed, versioned artifacts
Artifact: `<name>-<version>.shopdbplugin` (a zip of the plugin directory:
manifest.json, plugin.py, api/, models/, migrations/, and frontend/ once
section 6 lands) plus two members generated at pack time:
- `PROVENANCE.json`: plugin name, version, publisher id, build timestamp, and
a sorted map of every packaged file path to its SHA-256. PROVENANCE.json is
not listed in its own map, so there is no circular-hash problem and no zip
canonicalization needed; determinism comes from sorted per-file hashes.
- `PROVENANCE.sig`: detached ed25519 signature over the exact PROVENANCE.json
bytes.
New CLI:
- `flask plugin pack <name> --key <path>` (producer side): runs validate on the
directory, then emits the artifact.
- `flask plugin validate <dir|artifact>` (the missing pre-publish gate),
fail-closed pipeline: signature (artifact mode) -> per-file hashes -> manifest
against a new docs/plugin-manifest.schema.json -> name == directory ->
core_version parses as a specifier and admits the target contract version ->
static import-surface scan reusing tests/test_plugin_contract.py logic ->
alembic versions parse. The schema types the known fields (name, version,
description, dependencies, optional_dependencies, tier, core_version,
api_prefix, display_name, default_enabled, provides, settings) and PERMITS
additional properties, so all 13 existing manifests pass unmodified.
The import-surface scan is documented as a lint, not a security control; it is
trivially bypassed by dynamic import. The security control is human review
before signing (section 4).
### 3. The shelf: a read-only folder, transport-agnostic by design
- One config knob: `PLUGIN_SHELF_DIR`. The app only ever reads this folder. It
never speaks SharePoint, OneDrive, or any network protocol.
- Transport is explicitly out of scope and explicitly untrusted. On networks
that can reach corporate M365, a SharePoint document library sync populates
the folder. On strictly air-gapped floors where no sync agent can run, the
folder is populated by robocopy/USB. Both are equally supported and equally
untrusted, because every decision-bearing byte is signed: swapping transport
changes nothing about the trust model.
- Layout: `<shelf>/<name>/<name>-<version>.shopdbplugin` plus
`shelf-index.json` and `shelf-index.sig`.
- The index is SIGNED with the same publisher key and carries a monotonically
increasing `serial` plus a `revoked` list of name-version pairs. Each site
records the last-seen serial in instance state and refuses an index with a
lower serial (anti-rollback of the catalog itself). The index also carries
per-entry version/tier/core_version so `flask plugin shelf-list` can display
compatibility without unpacking, but the index is a BROWSE layer only:
adopt reads dependencies, tier, and core_version from the signed manifest
inside the verified artifact, never from the index.
- Trusted keys: `PLUGIN_TRUSTED_KEYS` is a list of pinned public keys delivered
out-of-band in the site's deployed config/image. Keys are NEVER read from the
shelf; a folder that can be written by an attacker must not also carry the
keys that authenticate it. Multiple pinned keys allow overlap rotation.
Revocation of an artifact rides the signed index `revoked` list; a
`flask plugin audit` command warns when an installed version appears there.
- Partial-sync robustness: adopt copies the artifact to a temp location,
verifies signature and every file hash there, then unpacks to
plugins/.staging/<name> and renames into place atomically. OneDrive
placeholder stubs, zero-byte files, or an index referencing not-yet-synced
artifacts all fail closed with a clear "artifact not fully synced/verified"
error.
- `flask plugin adopt <name>[==version]`: resolve version from the shelf,
verify, compute the hard-dependency closure from signed manifests, then for
each closure member in topological order: unpack, INSTALL, and ENABLE (not
install-only; the load gate at loader.py:201-206 checks is_enabled, so an
install-only closure with default_enabled=false deps would fail its own
load). Migrations run via the unchanged per-plugin chain (ADR-008). Refuses
to adopt a version lower than the installed one unless
`--force-downgrade` is given interactively. Prints the restart notice.
- Adopt/install/uninstall remain CLI-only. The admin HTTP surface stays a
read-only catalog view plus the existing enable/disable toggle; because
Flask cannot register blueprints after the first request, any adopt or
enable takes full effect only on restart, and the UI says so. There is no
"install button" that pretends otherwise.
### 4. Trust model: verify at adopt AND at every load and migrate
Signing that gates only adoption is bypassable through every other write path
into plugins/ (git clone, symlink, USB drop) and defeated by post-adoption
tampering. Therefore verification is enforced where code executes:
- Adoption leaves PROVENANCE.json and PROVENANCE.sig inside plugins/<name>/
and records publisher + artifact hash in the registry entry.
- load_plugin verifies the signature against PLUGIN_TRUSTED_KEYS and re-hashes
the plugin tree against the provenance file map BEFORE importing plugin.py
(new step ahead of loader.py:185). Missing or invalid provenance is a
fail-closed refusal in production.
- run_plugin_migrations performs the same verification before executing any
revision, so a routine `flask plugin upgrade-all` can never run DDL from an
unverified folder.
- upgrade_all_plugins iterates registry.get_all() (fixing the phantom
list_installed fallback at __init__.py:94), so unadopted on-disk folders are
never migrated as a side effect of deploys.
- Development and the ADR-003 external-repo/symlink workflow (including
scripts/test-external-plugin.sh) are preserved via `PLUGIN_DEV_TRUST_DIRS`,
honored ONLY when DEBUG or TESTING is set. Production ignores it.
- Cost: hashing 13 small plugin trees at boot is milliseconds; accepted.
What signing does NOT claim: a valid signature proves the artifact is exactly
what a curator reviewed and signed, nothing more. Plugins remain in-process
Python with full DB access. The actual safety control is the human review
before signing; the signature makes that review's verdict tamper-evident all
the way to execution.
### 5. Declarative site profiles and lean backend builds
- `site-profile.json` per site (kept in the site's deploy config): site name,
list of chosen plugins, optional `locked` list. `flask plugin apply-profile
<file>` resolves the closure, installs AND enables in dependency order, runs
migrations, reports which changes need a restart. This replaces the
imperative CLI sequences in DEPLOY.md/PILOT-DEPLOY.md and fixes the
enable-without-install bug.
- Lean backend image: `scripts/build-site.sh` reads the profile and stages
only core + chosen plugin directories into the Docker build context
(correcting the Dockerfile COPY and its stale header comment). Discovery
needs no change; it already scans whatever exists.
- Prerequisite the naive version misses: core hardcodes plugin imports.
shopdb/core/api/search.py (~15 sites), reports.py, assets.py, collector.py,
applications.py, auditlogs.py, and shopdb/cli/__init__.py import
plugins.<name>.* lazily. Some already guard ImportError; ALL must, with
graceful degradation, before any site prunes a folder. This is audited and
enforced by a new CI job that deletes one plugin directory and runs the full
test suite (repeated per plugin). Longer term these aggregators should move
to registry-driven contract hooks (get_search_providers/get_report_sources)
so a new catalog plugin can join search/reports without core edits; that is
scoped as follow-up work, not a blocker for lean builds.
- Schema-lean is DEFERRED to its own ADR. The core baseline 68b3947ae14f
unconditionally creates the 10 pre-cutover plugins' tables, and lifting them
into plugin baselines collides with cross-plugin foreign keys (the
computers-owned installedapps table FKs machines.machineid while computers
declares no dependency on machines). Reversing the cutover would either
introduce undeclared hard deps or drop FKs; neither is decided here. A lean
site therefore carries a handful of empty pre-cutover tables. Accepted.
### 6. Frontend delivery: Path C for rich UIs, Path A for simple ones, Path B rejected
Path B (runtime-loaded JS / module federation) is REJECTED: it moves executable
UI delivery from a signed, statically auditable build artifact to runtime
fetching, which is exactly the wrong direction for an air-gapped,
review-then-sign posture, for zero benefit given restarts are already required.
Path A (declarative JSON UI over generic renderers) is COMMITTED and scheduled
EARLY: the three unwired ADR-010 endpoints (pluginui.py asset-panels:62,
map-overlays:88, asset-presentation:100) get generic core renderers, joining
the already-consumed settings-cards. After this, a simple plugin ships JSON-only
UI with zero frontend build involvement. Sequencing this before the relocation
gives every plugin an escape hatch during the migration instead of after it.
Path C (self-contained plugin frontend) is the primary mechanism, scoped
against the real code, not the idealized layout:
- Canonical home: plugins/<name>/frontend/ containing routes.js (the plugin's
complete route array, INCLUDING routes currently embedded in index.js and
core.js), views/, and settings views.
- A pre-Vite staging step (scripts/stage-frontend.mjs, run by build-site.sh
and the dev script) copies the CHOSEN plugins' frontend/ into
frontend/src/.plugins-staged/<name>/ (gitignored) and generates two files
inside the Vite root: routes.gen.js (aggregated plugin routes) and
meta.gen.js (plugin-supplied icon names, title spellings, settings-standalone
flags, replacing the hardcoded iconMap/TITLE_SPELLINGS/SETTINGS_STANDALONE in
AppLayout.vue, settingsCatalog.js, and index.js). This exists because
import.meta.glob requires a static literal inside the project root and
cannot select a per-site subset by itself.
- ONE-TIME core-router surgery, done first and called what it is: the six
hardcoded plugin-view imports in index.js (PartsKiosk, TVDashboard,
PrinterQRBatch/Single, USBLabelBatch, PrintedPartsLabels) and the plugin
routes embedded in core.js move into their owning plugins' routes.js. Without
this, pruning slides/printers/usb/printedparts fails the Vite build on
unresolvable imports; no amount of glob work fixes it.
- Per-plugin relocation PRs (13), each REAL WORK, not a file move: carve routes
out of shared files, move views (handling name mismatches like computers ->
views/pcs), move the plugin's settings views out of shared views/settings/,
and rewrite relative ../../ imports of core shared code to the @/ alias
(relative paths break at the staged depth). A lint rule enforces alias-only
core imports in plugin frontend code from then on.
- Shared plugin-aware code STAYS CORE and ships to every site: AssetLabel.vue
(spans five asset types), views/print helpers (assetLabel.js, qrLogo.js),
MachineBadge.vue, and cross-plugin panels like WarrantyPanel used by
PCDetail. These already null-guard or gate via isPluginEnabled and must keep
degrading when a peer plugin is absent; over time they migrate to ADR-010
asset-panels so the data becomes plugin-supplied. Lean v1 therefore prunes
plugin-EXCLUSIVE code; a small plugin-aware core remainder is accepted and
shrinks as Path A absorbs it.
- Dual-location transition: the staging step unions legacy locations
(routes/*.js glob, views/<plugin>/) with plugins/<name>/frontend/ until each
plugin has moved. The SPA builds green at every commit; each plugin's move is
independently revertable until the legacy glob is removed at the end.
- Nav and settings cards are already server-driven (dashboardApi.navigation,
settings-cards); the remaining hardcoded plugin entries in settingsNav.js
(/settings/zabbix, /settings/dellwarranty) move to those plugins'
get_settings_cards so pruning leaves no dead links.
### 7. Effect on the 13 existing plugins
- Backend: ZERO code changes required. tier/optional_dependencies/provenance
are additive; pack zips the directory as-is; all plugins keep passing
tests/test_plugin_contract.py. Bundled plugins in a site's image get
provenance stamped at build time by pack, so verify-at-load applies to them
identically.
- Frontend: one relocation PR each, of the honest scope above. Until a
plugin's PR lands it keeps working from its legacy location.
- Operationally nothing changes for a site that does nothing: default builds
remain all-plugins, apply-profile is opt-in, and enable/disable semantics
(including the restart requirement) are unchanged.
## Consequences
### Positive
- A real catalog: sites declare their set in site-profile.json and apply it in
one idempotent command; the chosen set drives backend image, SPA bundle, and
runtime state from one source of truth.
- Curated marketplace with end-to-end provenance: review -> sign -> any
transport -> verify at adopt, at load, and at migrate. Transport (SharePoint
sync or sneakernet) is untrusted and interchangeable, which is exactly right
for air-gapped sites.
- Lean per-site builds: unchosen plugins exist in neither the image nor the
bundle, shrinking attack surface and download size.
- Fixes shipped along the way: upgrade-all migrating unadopted folders,
PILOT-DEPLOY install/enable ordering, reverse-dep checks blind to unloaded
plugins, missing cycle detection, missing dependency closure, hardcoded
frontend plugin metadata.
- Path A completion makes simple plugins UI-capable with no build glue, which
is the cheapest possible marketplace onboarding.
### Negative
- Key management is a per-site operational burden: pinned keys delivered
out-of-band, rotation is a config change everywhere. Accepted as the price of
not trusting the distribution folder.
- The frontend re-org is the long pole: one core-router surgery plus 13
non-trivial PRs. It is sequenced to be always-green and per-plugin
revertable, but it is weeks of work, not a rename.
- Schema is not lean: pre-cutover plugin tables still appear at every site
until the deferred baseline re-org ADR.
- Restarts remain required after adopt/enable (Flask blueprint constraint);
the marketplace UX is honest about it rather than working around it.
- Boot adds a signature + tree-hash check per enabled plugin (milliseconds,
but nonzero).
### Risks
- Key compromise or curation failure: a signature proves provenance, not
safety; a compromised pinned key or a rubber-stamp review signs malware that
every gate will happily pass. Mitigations: multi-key pinning with overlap
rotation, signed revocation list with monotonic index serial, and keeping the
signing key offline with the curator. The static import scan is a lint and
must never be presented as a boundary.
- Rollback/downgrade: mitigated three ways: index serial monotonicity, adopt
refusing version downgrades without interactive --force-downgrade, and the
signed revoked list. Residual risk: a site that never syncs a newer index
cannot learn of revocations; `flask plugin audit` at deploy time narrows the
window.
- Version skew across ADR-004 sites: one shelf serves sites at different
contract versions. Adopt checks core_version from the signed manifest against
the site's own __contract_version__ (authoritative); shelf-list shows an
advisory compatibility column from the index. Incompatible artifacts are
listable but not adoptable.
- Partial/placeholder sync files: fail closed on hash verification; the error
message distinguishes "not fully synced" from "tampered" only by wording,
intentionally, since the app cannot tell.
- Dev-trust misuse: PLUGIN_DEV_TRUST_DIRS silently ignored outside
DEBUG/TESTING; a prod config carrying it gets a startup warning.
- Frontend closure drift: plugin views importing cross-plugin components is a
graph the manifest does not model. The lint rule (plugin frontend may import
core @/ paths and its own tree only, never another plugin's) prevents new
edges; existing shared plugin-aware code is explicitly core-owned.
## Implementation phases
- Phase 0, groundwork (small, days): upgrade_all_plugins uses
registry.get_all(); reverse-dep checks read installed manifests from disk;
cycle detection in _sort_by_dependencies; shopdb/plugins/manifest_schema.json +
`flask plugin validate` (directory mode); `flask plugin apply-profile` with
install+enable closure ordering; fix Dockerfile stale comment. All additive,
zero risk to running sites.
- Phase 1, packaging and signing (medium, about a week): PROVENANCE format,
`flask plugin pack`, validate artifact mode, PLUGIN_TRUSTED_KEYS config,
ed25519 signing tooling and curator docs. No runtime behavior change yet.
- Phase 2, shelf and enforcement (medium-large, one to two weeks):
PLUGIN_SHELF_DIR, signed shelf-index with serial + revoked list,
`flask plugin shelf-list` / `adopt` / `audit` with atomic verified unpack;
verify-at-load in load_plugin and verify-at-migrate in
run_plugin_migrations, fail-closed in prod; PLUGIN_DEV_TRUST_DIRS for
dev/test and the external-repo harness; tier:core lifecycle guard;
provenance stamping of bundled plugins at build. This phase completes the
security model; everything after it is delivery optimization.
- Phase 3, Path A completion (medium, one to two weeks): generic renderers for
asset-panels, map-overlays, asset-presentation; migrate settingsNav.js
hardcoded plugin cards to get_settings_cards. Done BEFORE relocation so
JSON-only UI is available during the migration.
- Phase 4, frontend re-org (large, the long pole, several weeks elapsed):
stage-frontend.mjs staging + routes.gen.js/meta.gen.js codegen; ONE core PR
moving the six index.js hardcoded plugin imports and the core.js-embedded
plugin routes into plugin route files; then 13 per-plugin relocation PRs
(views, settings views, name-mismatch dirs, @/ alias rewrite) under the
dual-location union; lint rule for plugin frontend imports. Always-green,
per-plugin revertable.
- Phase 5, lean builds end to end (medium, about a week after Phase 4):
build-site.sh staging backend dirs + frontend staging from site-profile.json;
core lazy-import guard audit finished, enforced by the delete-a-plugin CI
matrix; remove the legacy glob; pilot one real lean site (a location without
printedparts/usb/network) and diff its image and bundle against a full build.
Deferred, each to its own future decision: schema-lean core-baseline re-org
(blocked on the installedapps -> machines FK question), pip/entry-point
distribution (ADR-003 v2), hook-based search/report aggregation contract, and
any revisit of Path B.
## Amendment, 2026-08-14
Status corrected from PROPOSED to ACCEPTED. The lean-build half of this decision
shipped some time ago - `scripts/build-site.sh`, `SITE_PLUGINS` staging,
`flask plugin prune-schema` and `default_enabled: false` are all live, and
ADR-014 was accepted on top of it - while the record still said the decision was
under consideration. A decision that has been implemented and depended upon is
not proposed, whatever the header says, and leaving it that way makes every
other status in the index worth less.
The catalog and signed-artifact parts of the decision remain unbuilt. That is
the ordinary state of an accepted decision: accepted means settled, not
delivered.
## Note for this product line, 2026-09-19
This line bundles no plugin it withholds: the tree and the shelf are the same
set. So the omitted set a lean build leaves out is chosen by the receiving
SITE, not by this repository, and there is no bundled-but-unshipped code here
for a gate to catch.
That changes what the CI lean-build job can prove. It now builds a genuine
SUBSET of this tree's own plugins and asserts the rest are absent, so what it
exercises is the SUBSETTING MECHANISM - `scripts/build-site.sh`,
`scripts/resolve_plugin_closure.py` and the frontend staging - rather than one
site's chosen set. The omitted names are picked to have no dependants, so the
closure stays empty and the assertion stays honest.

150
docs/adr/ADR-014-schema-lean-per-site.md Normal file
View File

@@ -0,0 +1,150 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Why a lean site stops creating database tables for plugins it never installed.
loadwith: none
-->
# ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables)
- Status: ACCEPTED
- Date: 2026-07-19
- Deciders: ShopDB maintainers
- Relates to: ADR-008 (per-plugin migration ownership), ADR-013 (plugin catalog + lean per-site builds), ADR-001 (asset model)
## Context
ADR-013 delivered lean per-site builds for plugin CODE (backend tree + frontend
bundle carry only chosen plugins). One residual was left, explicitly deferred:
the DATABASE. The core Alembic baseline (68b3947ae14f) creates EVERY table,
including ~30 plugin-owned tables (PLUGIN_TABLE_OWNERS). Each plugin's own
baseline is a stamp-only no-op (the core chain already made its tables, per
ADR-008). So a lean site that omits a plugin still creates that plugin's tables,
empty and unused.
The deferral cited a blocker: "the computers-owned installedapps table FKs
machines.machineid while computers declares no dependency on machines; reversing
the cutover would introduce undeclared hard deps or drop FKs; neither is
decided." Investigation refined this:
The cross-boundary foreign keys into the machines plugin table are ALL held by
DEAD legacy tables/columns that predate the asset model (ADR-001) and the
per-plugin cutover (ADR-008), and are queried nowhere in the codebase:
- `machinerelationships` (child/parentmachineid -> machines) - superseded by
`assetrelationships`. No model, no query.
- `printerdata` (machineid -> machines) - the pre-cutover printers table,
superseded by the printers plugin. No model, no query.
- `installedapps` (machineid -> machines) - a standalone machine-app link table;
the live relationship is `computerinstalledapps` (FK to computers only). The
standalone table has no model, no query.
- `communications.machineid` (-> machines) - a legacy column on the core
communications table (which is now assetid-based). Not read anywhere.
No LIVE plugin table hard-FKs another plugin's table. computerinstalledapps FKs
only computers.computerid (intra-plugin). So the blocker is dead cruft, not
live design.
## Decision
Two phases, both leaving existing databases correct.
### Phase 1: retire the dead cross-boundary cruft - ALREADY DONE
Investigation found this is already accomplished by existing migrations:
`7a01_adr001_position_contract` and `7c01_drop_legacy_machine` drop
`machinerelationships`, `printerdata`, `installedapps`, and
`communications.machineid` (with its FK). The current schema (verified on the
dev database) has none of them. So the cross-plugin FK blocker ADR-013 cited no
longer exists in the live schema - only in the baseline's transient
create-then-later-drop. No new migration is needed for Phase 1.
Precedent: ADR-001 dropped a cross-plugin FK the same way
(usbcheckouts.machineid -> machines became a soft sentinel).
### Enabling change (executed now): idempotent create_plugin_tables
`shopdb/plugins/alembic_template.py:create_plugin_tables` now skips any table
that already exists (inspects the bind first) instead of raising. This is the
mechanism Phase 2 needs: a plugin anchor can create its tables on a fresh lean
install AND be a safe no-op on an existing database that already has them from
the pre-cutover core baseline. Correct and inert regardless of Phase 2 (no
current caller creates against a populated schema). Verified against the
plugin-migration suite.
### Phase 2 (executed): prune not-installed plugin tables after upgrade
Two mechanisms were weighed to make a lean site's database carry only
core + chosen-plugin tables:
- **Relocate** (rejected): pull every plugin-table create/alter out of the core
chain into the plugin baselines, so the core chain never creates a
not-installed plugin's table. Measurement killed this: plugin tables are
created and altered across ~15 released core migrations (baseline plus 7c04,
7d05, 7d08, 7d13, 7d15, 7d16, 7d17, ...), not just the baseline. Because the
whole core chain runs before any plugin chain, removing a table's create from
core while a later core migration still alters it breaks FULL installs too, so
relocation means surgically rewriting ~15 released migrations - the highest
blast radius in the project - for a purely cosmetic gain (the omitted tables
are empty and the lean CODE build already never loads the plugin).
- **Prune-after-upgrade** (chosen): leave the entire core chain untouched. Add
`flask plugin prune-schema`, which drops the tables of every plugin in
PLUGIN_TABLE_OWNERS that is not installed on this site. Run once at deploy,
after `flask db upgrade` and `flask plugin upgrade-all`. Same end state
(core + chosen tables) with near-zero blast radius: no released migration is
edited, and an existing full site is unaffected because it never runs the
command.
`prune-schema` drops by table name (no plugin-code import), so it works on a
lean image where the omitted plugin's directory is absent. It is a dry-run by
default and refuses to drop a table that holds rows unless `--force`, so a
misfire on a populated site cannot silently delete data. Because the core chain
seeds a few plugin reference tables (e.g. 7d05 inserts default access
protocols), initial lean provisioning uses `--force` - at that point the tables
hold only migration-seeded defaults, before any site data exists.
The idempotent `create_plugin_tables` (enabling change above) is what lets a
lean site later ADD an omitted plugin: its anchor recreates the pruned tables.
Verified end to end on MySQL: fresh full install (86 tables) then prune is a
no-op; fresh lean install (machines + printers) then prune drops the other 19
plugin tables, leaving core + chosen; second prune is a no-op; the non-empty
guard refuses without `--force`. Four SQLite regression tests pin the behavior
(tests/test_plugin_prune_schema.py), running in the backend CI job via the real
CLI runner: drop-only-not-installed, full-site no-op, refuse-non-empty, and
force-drops-non-empty.
## Consequences
### Positive
- A lean site's database contains only core + chosen-plugin tables, with no edit
to any released migration (near-zero blast radius).
- The cross-plugin FK blocker ADR-013 cited is gone (dead cruft, dropped by
existing migrations), so plugin schemas are already FK-independent.
- Adding an omitted plugin to a lean site later just works: the idempotent
anchor recreates its tables.
### Negative / risk
- prune-schema is destructive by nature; the row-count guard + dry-run default +
required `--force` for non-empty tables contain that. It is a deploy-time
provisioning step, not something to run casually on a live populated site.
- A lean fresh install still transiently creates then drops the omitted plugins'
tables (the core chain builds them, prune removes them). Harmless and one-time
at provisioning; the trade for not touching the released baseline.
## Implementation
- Phase 1: nothing to do - the dead cross-boundary FK objects were already
dropped by existing migrations `7a01_adr001_position_contract` and
`7c01_drop_legacy_machine`; verified absent on a fresh full MySQL upgrade.
- Enabling change: `create_plugin_tables` made idempotent
(`shopdb/plugins/alembic_template.py`).
- Phase 2: `flask plugin prune-schema` (`shopdb/plugins/cli.py`), dry-run by
default, `--yes` to execute, `--force` for non-empty tables. Deploy order:
`flask db upgrade` -> `flask plugin upgrade-all` -> `flask plugin prune-schema
--yes --force`. Regression tests in `tests/test_plugin_prune_schema.py` (run in
the backend CI job).

156
docs/adr/ADR-015-site-specific-configuration.md Normal file
View File

@@ -0,0 +1,156 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Where a site own hostnames, paths and defaults are allowed to live, and where they are not.
loadwith: none
-->
# ADR-015: Where a site's own data is allowed to live
- Status: ACCEPTED
- Date: 2026-08-11
- Deciders: ShopDB maintainers
- Relates to: ADR-004 (per-site instances, not multi-tenant), ADR-013 (plugin catalog + lean per-site builds), ADR-014 (schema-lean per-site builds)
## Context
ShopDB is built as a product several GE Aerospace sites can adopt (ADR-004:
each site runs its own instance). It is also developed at one site, against
that site's live fleet. Every feature therefore arrives
carrying that site's hostnames, networks, share paths and vocabulary, and the
fastest way to ship any of them is to write the value where the code needs it.
That is not a hypothetical drift. A scan of `plugins/` and `shopdb/` for a site
host, a site FQDN or a site network returns 19 hits, of which roughly 8 are a
shipped default or a hardcode rather than a comment:
- `plugins/geenforce/seed_display_scope.py` baked one site's own ShopDB URL and
`device.geaerospace.net` into an inline PowerShell payload. A sister site
could not override either without editing the source. FIXED 2026-08-14: the
kiosk base URL now comes from the `site_base_url` setting at seed time and
from HKLM at run time, and the dispatcher refuses rather than guessing when
neither is set.
- `plugins/backups/services/registry.py` defaults the backup share root to one site's own
UNC path.
- `shopdb/core/api/settings.py` ships `device.geaerospace.net` and
`Printer-{ip}.printer.geaerospace.net` as SETTING DEFAULTS, so another site
inherits that site's domain silently until someone notices.
- `shopdb/core/models/dashboarddefault.py` holds the same domain a second time
as a module constant, and `plugins/computers/api/routes.py` a third time as a
literal fallback.
The problem is not that site-specific values exist - they must. It is that the
repository already has three good mechanisms for them and no rule about which to
use, so each new value lands wherever was convenient. The result reads as drift
because it is unmanaged, not because any single choice was wrong.
Nothing here is a security matter: these are internal hostnames on an air-gapped
network, and `tests/test_docs_publishable.py` already gates what reaches the
public wiki. This is about whether a sister site can adopt the product without
reading our source.
## Decision
A site's own data lives in one of three places, in this order of preference.
**1. A setting, with a NEUTRAL default.** Anything an operator could plausibly
change: hostnames, domains, share paths, networks, base URLs, label prefixes.
The default that ships must be blank or generic, never one site's value. Blank
must be a working state - code reads the setting and degrades sensibly, the way
`display_fqdn_domain` already treats blank as "use the hostname as-is". A
site-specific default is the failure mode this ADR exists to stop: it is
invisible, it looks deliberate, and it is wrong everywhere except here.
**2. A site-namespaced directory.** Anything that is genuinely one site's and
will never be another's, such as the reference site's import loader at
`scripts/site_imports/wjf/`. Never reached by another site's build.
**3. Seed data, not code.** Vocabularies and catalogs, loaded by `flask seed`
and editable per site afterwards.
Fleet-wide convention is NOT site-specific. The `gea-shopfloor-*` imaging types
are a GE fleet taxonomy, not one site's data, and they already carry a per-site
override through the `pctypemap_*` settings. They are out of scope here.
Comments and documentation may name a site where it explains WHY - "part markers
share a machine number at the reference site" is a fact a maintainer needs. Prefer
phrasing the rule generally and the example specifically.
## Enforcement
`scripts/check-naming-and-style.sh`, which already runs in CI, greps `plugins/`
and `shopdb/` for a site host, a site FQDN or a site network, excluding tests
and `site_imports/`.
It shipped in REPORT-ONLY mode, because turning a 19-item backlog into a hard
gate on the day it lands blocks every unrelated commit until someone works
through it. See the amendment below: it now fails the build.
## Consequences
Positive:
- A sister site can adopt without reading our source for values it must change.
- The backlog is visible and counted rather than felt.
- New site-specifics are caught at commit time, in the same place the naming
rules are already enforced, so there is nothing new to remember.
Negative:
- Report-only means the rule is advisory until someone flips it, and an
advisory rule can be ignored.
- A neutral default is more code than a working one: every read needs a blank
case, which is a real cost paid at every call site.
- The pattern list is a denylist and will miss a site value it has not been
taught. It catches drift, not everything.
## Backlog
CLEARED - see the amendment below. Kept as the record of what the decision was
answering:
- `plugins/geenforce/seed_display_scope.py` - kiosk base URL, FQDN domain, and
the legacy-shortcut sweep pattern
- `plugins/backups/services/registry.py` - `DEFAULTSHAREROOT`
- `shopdb/core/api/settings.py` - `display_fqdn_domain` and the printer
hostname pattern defaults
- `shopdb/core/models/dashboarddefault.py` - `DEFAULT_DISPLAY_FQDN_DOMAIN`
- `plugins/computers/api/routes.py` - literal fallback duplicating that default
The domain appears three times in three files, which is its own defect: one
setting, one default, one read.
Cosmetic mentions in comments and UI placeholder text are acceptable under this
ADR and are not part of the backlog, with one exception worth fixing when
convenient: the GE-Enforce settings page shows one site's real CIDRs as its
placeholder, which every site sees.
## Amendment, 2026-08-14
The backlog above is cleared and the scanner now ENFORCES: it fails the build,
and `SITE_LITERALS_ENFORCE=0` is the escape hatch rather than the default. The
report-only period ended for the reason the Consequences section predicted - the
hit count did not move for weeks, because a rule that only prints is read as no
rule.
Three things changed beyond clearing the list:
- The scan covers PowerShell, the installer, the seeds, generated JSON and the
frontend, case-insensitively, across `plugins/`, `shopdb/`, `scripts/`,
`deploy/`, `frontend/src/` and `tools/`. Its original scope - three file types
under two directories - missed every literal that actually reached a second
site, because those were all in the client scripts and the installer.
- A line may declare itself deliberate with a trailing `ADR-015-OK: <reason>`
marker. That is for an organisation-wide default which is right at every site
and configurable anyway, such as the corporate device domain. It is not a
way to defer work: the marker puts the claim in the diff where a reviewer
sees it.
- Both publication gates - the export scrub and the docs publishability test -
carry the site patterns, which neither did. The ADR said this was not a
security matter because the values were internal to an air-gapped network;
that was true of the fleet and false of the public mirror, which had been
carrying a production hostname and a share path.
This amendment is appended rather than edited into the text above, because what
the decision looked like when it was taken is the part worth keeping.

204
docs/adr/ADR-016-credential-delivery.md Normal file
View File

@@ -0,0 +1,204 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 plan for ShopDB-resolved credential targeting. Never built; superseded by ADR-019.
loadwith: none
-->
# ADR-016: Credential delivery to the fleet
- Status: SUPERSEDED by [ADR-019](ADR-019-per-pc-endpoint-credentials.md)
- Date: 2026-08-11
- Deciders: ShopDB maintainers
- Relates to: ADR-006 (collector contract), ADR-012 (GE-Enforce manifest ownership), ADR-015 (site-specific configuration)
> **Do not build against this. Nothing in it was built.** The model below has a
> PC ask ShopDB for credentials chosen by server-side targeting rules. The
> `GET /api/collector/credentials` endpoint, the `credentials.fetch` and
> `credentials.manage` permissions and the rule store do not exist in the code,
> and a reader searching for them will not find them.
>
> What shipped instead is the opposite shape: the endpoint holds ONE secret of
> its own, obtained by enrolling, and there are no targeting rules to resolve.
> [ADR-019](ADR-019-per-pc-endpoint-credentials.md) records that decision.
>
> The PROBLEM stated below is still the real problem, and ADR-019 cites it.
> Kept as the record of a design that was decided and then not taken.
## Context
Shopfloor bays need credentials they cannot prompt for. A collections bay loses
the Windows Credential Manager entry for its controller subnet periodically -
suspected Defender or Intune scrub - and without it, controller-side
connections need an interactive prompt, so an unattended bay simply stops
working. A machine whose NTLARS config points at a program share needs a
credential to reach that share for the same reason.
Today this is solved by `Set-ControllerCredential.ps1` on the SFLD share:
```powershell
$Target = '192.168.1.1'
$Username = 'CHANGEME'
$Password = 'CHANGEME'
```
A person edits the values into the file, GE-Enforce runs it every cycle, and
`cmdkey /add` re-applies it. It works, and it has three problems.
The password is **cleartext in a file on a share**, readable by anything that
can read the share - a wider exposure than the machine-local secret it becomes.
Rotating it means editing that file and hoping: there is no way to ask which
bays have picked up the new value. And it is **one value for everything**,
while the requirement is per-bay variation: sites will have machines needing
different credentials from their neighbours.
Per-bay variation is what rules out the obvious alternative. GE-Enforce scopes
are per PC TYPE, so expressing per-machine credentials as manifest scopes means
a scope per bay, which is not a model. The manifest can deliver a value
(`Registry` entries with inline payloads already do this for the display
scope), but it cannot decide WHICH value a given PC should get.
## Decision
ShopDB owns credential definitions and their targeting. A PC asks for its own
credentials and receives a decided answer; it never receives rules to evaluate.
### Targeting
A credential rule carries a targeting expression using the SAME axes
GE-Enforce already filters on, plus those the asset model knows:
| Axis | Source | Notes |
|---|---|---|
| hostname | reported | exact or `-like` glob (`WJRP*`) |
| machine | PC -> machine link | the machine the PC drives |
| model | asset model | e.g. every Okuma of one model |
| controller OS | asset / reported | version-valued, like the existing CmmVersion gate |
| PC type | ComputerType | with GE-Enforce ALIAS EXPANSION |
| GE-Enforce profile | manifest scope | the imaging profile |
| DNC/NTLARS setting | stored backup projection | e.g. any machine whose `DataHost` is a given UNC |
| site default | setting | the fallback |
The DNC axis is not a special case bolted on: the NTLARS projection ShopDB
already stores contains `DataHost`, `DataPath` and `MarkMasterPath`, so "every
machine that talks to \\ntshare" is a selector over data already held. It is
also the axis that answers "which machines even need a credential", which
nothing answers today.
### Precedence
Overlapping matches are guaranteed, not exceptional - a hostname glob and a
per-model rule will both hit the same bay. Precedence is therefore EXPLICIT,
never emergent:
1. Each rule carries an integer `priority`. Higher wins. This is the only
thing an operator needs to reason about.
2. Ties break on scope specificity, in the order of the table above (hostname
most specific, site default least).
3. Remaining ties break on the lowest rule id, so the result is stable rather
than dependent on row order.
A resolve endpoint must be able to EXPLAIN itself - "this PC got rule 12,
matched on model, beating rule 4 on priority" - or nobody will trust it. The
GE-Enforce simulator (`applicable_entry_names`) already set this precedent.
### Reuse the existing matcher
`plugins/geenforce/filters.py` already mirrors the engine's targeting: alias
groups, hostname globs, the rule that a machine-number filter with no machine
number EXCLUDES rather than defaults. The resolver reuses it. It must not
become a third implementation of the same matching logic - there are already
two (the PowerShell engine and its Python mirror), kept honest by a parity
harness, and a third would drift silently.
Alias expansion in particular is not optional: if a rule says `PCTypes:
Standard`, it must mean what it means in GE-Enforce, or the two systems will
disagree about who a rule covers.
### Targeting is not detection
Who gets a credential and whether it is already applied are separate
questions, exactly as `PCTypes` and `DetectionMethod` are separate in the
manifest. The client decides the second locally - is this `cmdkey` entry
already present and correct - and rewrites only on drift. Conflating them
rewrites the credential every cycle instead of healing it, and turns the audit
trail into noise.
### Storage
- Encrypted at rest with Fernet. `cryptography` is already a dependency.
- The key lives OUTSIDE the database - instance file or environment, like
`SECRET_KEY`. A key in the database it protects is not encryption.
- Write-only through the API. Masked on read, as `smtp_password` already is. A
fleet PC receives a value; a human never reads one back.
- A dedicated `credentials.manage` permission for authoring.
### Delivery
```
GET /api/collector/credentials X-API-Key: <credentials.fetch token>
```
- A DEDICATED `credentials.fetch` scope, not `collector.ingest`. A leaked
collector key must not yield controller passwords.
- The caller is resolved by hostname; it receives only what it is entitled to,
and never the rule set.
- Every fetch is audited: which PC, which credential, when. The audit log
exists; this belongs in it.
- The client half is a shared helper alongside `ShopdbBackupClient.psm1`:
fetch, compare, `cmdkey /add` on drift, log once per state.
## Consequences
Positive:
- The cleartext credential leaves the share.
- Rotation becomes a settings change plus a report of who has picked it up,
instead of a file edit and a hope.
- Per-bay, per-model and per-share credentials become expressible, which is the
actual requirement.
- "Which machines need a credential" becomes answerable from the DNC data
already stored.
Negative, and these are real:
- **ShopDB becomes a credential store.** That is a different security posture
from an asset database, and it earns the obligations above rather than
choosing them.
- **The key is a new single point of loss.** Restore the database without it
and every stored credential is unrecoverable. `docs/BACKUP-RESTORE.md` must
say so IN THE SAME CHANGE, not later.
- A PC that cannot reach ShopDB gets no credential. The client must keep the
last known good value rather than clearing a working entry on a failed fetch.
- Targeting complexity is real complexity. Eight axes with priorities will
produce a rule set someone has to debug, which is why the explain endpoint is
a requirement and not a nicety.
## Alternatives rejected
**Keep editing the script on the share.** Cannot express per-bay values, keeps
the cleartext, no rotation story.
**A manifest scope per bay.** Per-machine credentials as GE-Enforce scopes is a
spreadsheet, not a model, and the manifest cannot resolve WHICH value applies.
**Encrypted payload in the manifest.** DPAPI is per-machine, and one manifest
serves many machines, so the payload cannot be encrypted to its readers.
**Registry-only provisioning (no ShopDB).** A worthwhile FIRST STEP - it moves
the secret off the share immediately and needs no new endpoint - but it leaves
provisioning per-bay by hand and offers no rotation. Recommended as tier one
regardless, since the client helper is the same either way.
## Implementation status
Never implemented, and now superseded. The fetch endpoint and the
`credentials.*` permissions described above do not exist in the code.
The note that stood here until 2026-09-18 said the decision still stood and a
plugin author designing against it was designing correctly. That is no longer
true: design against
[ADR-019](ADR-019-per-pc-endpoint-credentials.md). What credentials the fleet
uses today, and where they live, is in
[FLEET-ARCHITECTURE](../FLEET-ARCHITECTURE.md).

151
docs/adr/ADR-017-buildings-and-levels.md Normal file
View File

@@ -0,0 +1,151 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: Why the floor map became buildings and levels instead of one image with stacked floors.
loadwith: none
-->
# ADR-017: Buildings and levels as the map model
- Status: ACCEPTED
- Date: 2026-08-17
- Deciders: ShopDB maintainers
- Relates to: ADR-001 (asset as the platform contract), ADR-004 (per-site
instances), ADR-010 (frontend plugin hooks), ADR-015 (site-specific
configuration)
- Supersedes: the site-wide `map_blueprint_light` / `map_blueprint_dark` /
`map_width` / `map_height` settings
## Context
A site's floor map is one image. Four settings describe it - two blueprints
(light and dark) and the native pixel width and height - and `assets.mapx` /
`assets.mapy` are absolute pixel coordinates in that image's space.
That model has one floor in one building, which was true when it was written and
is no longer. The reference site is adding a second level, and a second building
is likely within a year or two. The immediate trigger is a new blueprint at
3308x4000 where the old one was 3300x2550.
The obvious cheap answer is to stack levels on one canvas: draw the second level
below the first and keep one image. It works, and it was seriously considered.
It was rejected because it makes the level an inference rather than a fact:
- "Which level is this asset on" becomes `mapy > 2550`. Every per-level filter,
count, report and export has to know that constant, and re-exporting the
drawing at a different height silently changes the answer everywhere. Not
broken - wrong, which is worse.
- Distance between two markers becomes computable and meaningless, so anything
doing nearest-asset or clustering quietly answers nonsense across levels.
- One canvas forces one scale. A mezzanine drawn at a different scale than the
floor below cannot be expressed at all.
- 3308x4000 is portrait for a building whose floor is landscape. At a zoom where
a marker is clickable, roughly half a level is visible, and "fit to level" is
not expressible.
- A third level makes each of these worse, and adds a second threshold.
Timing decided it. Because the level layout changed and machines moved, every
marker is going to be repositioned anyway. Introducing levels now costs one pass
over the positions; stacking now and splitting later costs two, and in between
every position placed encodes the threshold into real data - which a later
migration would then have to un-guess by comparing Y against it.
## Decision
Two tables, and assets reference the level.
```
buildings buildingid, buildingname, sortorder, isactive
maplevels levelid, buildingid, levelname, sortorder,
blueprintlight, blueprintdark, mapwidth, mapheight,
isdefault, isactive
assets levelid (nullable; the default level for existing rows)
```
**1. An asset references the level, never the building.** The building derives
from the level, so the two cannot disagree. Storing both would be a fact
recorded twice with no difference in granularity to justify it.
**2. Blueprints and native dimensions belong to the level.** This is what the
site-wide settings could not express: two levels in one building may be drawn at
different sizes and scales, and two buildings certainly are.
**3. Name and order are separate.** `levelname` is text and `sortorder` is an
integer. Levels are not reliably numbered - basement, ground, mezzanine, roof,
tunnel - and sort order gives adjacency and up/down navigation without
pretending the names are ordinal. It also lets a mezzanine be inserted between
two existing levels without renumbering anything.
**4. `mapx` / `mapy` keep their meaning, scoped to the level.** They stay
absolute pixels in the native coordinate space of the level's blueprint. No
normalisation to fractions: pixels are what the drawing tools produce, what an
operator can read off an image, and what the existing data already holds.
**5. A level is required to render a position, and absence is not a default.**
Where a payload carries a position without a level, the UI renders "level
unknown" rather than falling back to the default level. Falling back draws one
building's ground floor with a marker positioned for another building's
mezzanine: it renders perfectly and points at the wrong place, and nothing about
the result looks wrong. A visible gap is worth more than a confident wrong
answer.
**6. The level list is readable without authentication.** The printer installer
map runs optional-auth, before anyone logs in, and it needs a blueprint. This
follows the precedent already set for printer install-list and the slide feed.
Blueprint paths and level names are not secrets; the positions of assets on them
already render on public kiosk pages.
**7. Levels are not Locations.** Assets carry `locationid` already, and reusing
it is tempting. A Location answers "which operation owns this"; a level answers
"which drawing renders it, at what native size". Overloading Location with
blueprint images and pixel dimensions makes both concepts worse. They coexist:
an asset on level 2 in operation 0613.
## Migration
The existing four settings become one building and one level, marked default,
and every asset with a position points at it. Nothing renders differently on the
day it lands. The settings keys are then retired rather than left as a second
source of truth that can disagree with the rows.
## Consequences
Positive:
- A second building costs a row and a blueprint. So does a third level.
- Per-level export, per-level zoom, and per-level calibration all become
expressible. Calibrating one level against its own landmarks is more correct
than transforming a whole site at once.
- Level becomes queryable - counts and filters per level or per building are
ordinary queries rather than coordinate arithmetic.
Negative:
- Every surface that draws a marker learns there is more than one drawing. That
is two components, seven views, four asset position pickers, the printer
installer map, the PDF export, and eight API payloads that must emit `levelid`
beside `mapx`. A payload that forgets it produces the silent-wrong-map failure
described above, which is why rule 5 exists and why a build gate checks that
the two fields travel together.
- `get_map_overlays` gains level context, so this is a plugin contract change
(0.20.0) and plugins rendering overlays need to declare which level they are
for.
- One more admin surface: levels and buildings have to be managed somewhere, and
the existing floor-map settings page becomes that.
## Alternatives considered
**Stack levels on one canvas.** Rejected above. Zero cost today, and it makes
the level an inference over a magic number.
**Normalise coordinates to fractions of the image.** Would make a rescaled
blueprint self-correcting, which is genuinely attractive. Rejected for now
because it converts every existing integer position through a lossy division,
and because it does not help the actual problem: the levels changed and machines
moved, so the positions need human review regardless. Worth revisiting
independently.
**A `level` string on the asset, with one blueprint per level in settings.**
Rejected: settings keyed by level name is a table with extra steps, and it gives
no place for per-level dimensions or ordering.

132
docs/adr/ADR-018-metrology-instruments-are-machines.md Normal file
View File

@@ -0,0 +1,132 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why metrology instruments are machines and the measuringtools plugin is retired.
loadwith: none
-->
# ADR-018: Metrology instruments are machines
- Status: ACCEPTED
- Date: 2026-09-04
- Deciders: ShopDB maintainers
- Relates to: ADR-001 (asset as the platform contract), ADR-006 (collector
contract), ADR-008 (a plugin owns its tables and its migration chain)
- Supersedes: ADR-005 (equipment plugin scope vs measuringtools plugin)
## Context
ADR-005 split manufacturing machinery from metrology instruments into two
plugins on the grounds that they are two asset classes: machines make parts,
measuring tools measure them. It named the metrology population as "CMMs,
Keyence vision systems, wax-and-trace surface profilometers, GenSpec
instruments", and the `measuringtools` plugin was built against a docstring
naming a different population again: "calipers, micrometers, thread gages,
bore gages, height gages".
Those two lists are not the same thing, and the difference is the whole
decision. One is bay-mounted capital equipment with a controlling PC. The other
is hand tools in a drawer. The plugin was built for the second and populated
with neither.
Sixteen months on, the evidence is in:
- `measuringtools` holds **zero rows**, against 204 machines, 305 computers,
73 network devices and 53 printers.
- The metrology inventory actually waiting to be entered is 134 instruments:
35 CMM, 32 form tracer, 23 vision, 18 wax trace, 14 measuring, 8 Genspect,
2 eddy current, 1 spline. Every one is bay-mounted with a controlling PC.
- Not one caliper, micrometer, thread gage, bore gage or height gage. The
population the plugin's own docstring justifies it with does not exist at
the reference site and is not planned.
- `CMM` already exists as a `machinetype`, and `Part Marker` is already
classed as a machine while `Form Tracer` is classed as a measuring tool.
Two bay instruments with a controlling PC, split across two plugins, with
the line drawn somewhere no operator can predict.
The split has therefore been paying an ongoing cost - a duplicate type page, a
settings group, a nav section, a migration chain, a searcher, dashboard cards,
map colours, label printing and four collector `pctypemap` entries - to model a
distinction that the inventory does not make.
## Decision
Metrology and inspection instruments are machines. The `measuringtools` plugin
is retired and the `measuring_tool` asset type is removed.
### Calibration moves to machines, and is not lost
`machines` gains the calibration lifecycle as nullable columns:
```
machines
- calibrationintervaldays integer, null
- lastcalibrationdate date, null
- nextcalibrationdate date, null
- calibrationprovider varchar(150), null
```
Calibration status stays **derived at read time** from `nextcalibrationdate`
(`overdue` / `duesoon` / `current` / `unknown`, 30-day window), exactly as
`measuringtools` derived it and for the same reason: a stored status is stale
the day after it is written. The helper moves across unchanged.
This is a net gain rather than a port. A machine could not record calibration
at all before, and a part marker with a certified fixture, or a machine with a
calibrated probe, had nowhere to put it.
### `machinetypes.ismetrology` answers ADR-005's objection
ADR-005 made one concrete technical argument, and it deserves a concrete
answer rather than silence:
> "show me all measuring tools" requires an enumeration of measuring-instrument
> equipmenttype values, which scales badly.
That is correct about enumeration and it is why `machinetypes` gains a boolean
`ismetrology`. The metrology view stays a first-class single-column filter -
`WHERE mt.ismetrology = 1` - not a hardcoded list of type names that drifts the
first time someone adds an instrument type. The calibration report, the gage
lab view and any compliance export all hang off that flag.
### What does not change
- `assets.gaugelabreference` stays on the core asset. The gage tag was never a
`measuringtools` column, so it survives untouched and stays searchable for
every asset type.
- The collector's device identity file and the `measuringtoolid` payload alias
stay accepted. Bays staged with `measuringtool-id.txt` keep reporting; only
the asset type they resolve to changes.
- Instrument types keep their names. A Form Tracer is still a Form Tracer, now
as a `machinetype` with `ismetrology = 1`.
## Consequences
**The migration is free today and expensive later.** `measuringtools` has zero
rows, so retirement is a `DROP TABLE`, not a data migration. Once the 134
staged instruments are entered they carry relationships, audit history, map
positions and collector links, and every one of those would have to be
rewritten. This ADR is being taken now specifically because the window is open.
**A site that did populate it needs a real migration.** The reference site did
not and the sister site shows no sign of having done so - its branch contains
no deliberate work on the plugin in its entire history - but that is inference,
not a query against their database. Confirm before the removal ships to another
site. If rows exist, the retirement migration must copy them into `machines`
with `ismetrology` types rather than drop them.
**Four collector `pctypemap` entries change asset type.** `gea-shopfloor-cmm`,
`-keyence`, `-genspect` and `-waxtrace` resolve to `machine` instead of
`measuring_tool`. The `collector:measuringtool` relationship label is kept as a
recognised origin marker so links a collector already created are still
archived correctly rather than orphaned.
**One fewer of everything.** A duplicate type page, a settings group, a plugin
migration chain, a gated searcher, a set of dashboard cards, a map colour entry
and a label-printing branch all go. That is the point.
**ADR-005 is superseded, not reversed in principle.** Its reasoning was sound
for the population it described. The population never arrived. If a site ever
does track hand gages in quantity, the answer is a `handgages` plugin against
that real inventory - not the reinstatement of this one.

217
docs/adr/ADR-019-per-pc-endpoint-credentials.md Normal file
View File

@@ -0,0 +1,217 @@
<!-- docmeta
purpose: explanation
audience: developer
status: current
summary: Why every endpoint gets its own enrolled secret instead of one fleet-wide shared key.
loadwith: none
-->
# ADR-019: Per-PC endpoint credentials
- Status: ACCEPTED
- Date: 2026-09-18
- Deciders: ShopDB maintainers
- Relates to: ADR-006 (collector contract), ADR-012 (GE-Enforce manifest
ownership), ADR-015 (site-specific configuration)
- Supersedes: ADR-016 (credential delivery to the fleet)
## Context
The endpoints plugin authenticates every agent-facing call with ONE key. The
same `X-API-Key` value is on every PC in the fleet, and the server cannot tell
one caller from another: a request names its own hostname and is believed.
That key is not only an inventory-ingest key. It is also the key on the command
channel (`AGENT-CONTRACT.md` section 8), where the server hands an agent running
as LocalSystem an allowlisted action to execute - including `run_script`, which
fetches a reviewed PowerShell body and runs it. Anyone holding the key can claim
commands as any PC and post results as any PC.
And the key is readable. The agent installer writes it to
`HKLM\SOFTWARE\GE\ShopDB` `CollectorKey`, and the DACL that subkey inherits by
default grants `BUILTIN\Users` read, so any logged-on user on any bay could read
the fleet key (the reason is recorded in `plugins/endpoints/api/siteagent.py`
beside the code that now locks the key down). The same value also lived in
`appsettings.json` in the agent directory, in a text file on the GE-Enforce
share, and in the `Args` of the manifest entry that wrote it.
One fleet-wide secret, readable by a standard user on any bay, authenticating a
SYSTEM command channel. Rotating it meant touching every one of those places at
once, and nothing could say which PCs had picked the new value up.
ADR-016 answered a different question about credentials - which controller
password a given bay should receive, decided by server-side targeting rules - and
was never built. The question underneath it is "which PC is this", and it needs
no targeting at all.
## Decision
Each endpoint holds its OWN secret, obtained by enrolling, and the server
derives the calling asset from the secret.
### The credential
- `X-Endpoint-Credential: sdbe_` + 43 characters of base64url (32 random bytes).
- The server stores only the SHA-256 of the secret plus its first 12 characters
as a display prefix (`EndpointCredential` in `plugins/endpoints/models/credential.py`).
The full value exists once, in the enroll response.
- One row per asset, so an asset never carries two credentials. Re-issuing after
a revoke updates the same row.
- A request whose credential resolves to one asset while naming another is
refused `403 ASSET_MISMATCH`. A PC can no longer report as another PC.
### Enrollment
`POST /api/endpoints/enroll` never refuses a call for its authentication. The
shared key, if present and still allowed, only decides whether the credential is
issued automatically. Auto-issue requires all of: the shared key authenticated
the call, `plugin.endpoints.autoenroll` is on, the hostname matches exactly one
computer record, the serial on record matches, and the asset has no active
credential. Anything else records a PENDING request an admin approves in the UI.
The agent generates an enrollment nonce once and persists it. The nonce is
recorded on first sight and an approval is bound to it, so a second machine
claiming the same hostname and serial cannot collect the approved credential. A
rejected request binds to the identity instead, so a reinstall cannot argue its
way back into the queue.
### Storage on the endpoint
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
`LocalMachine`, in a directory ACLed to SYSTEM and Administrators only. The tray
holds no secret at all: it runs as the interactive operator and asks the SYSTEM
service over a local named pipe. `HKLM\SOFTWARE\GE\ShopDB` is locked to SYSTEM
and Administrators in both registry views.
### Server pinning
The agent pins the server's public key: the lowercase hex SHA-256 of the leaf
certificate's SubjectPublicKeyInfo, in `ServerPins` (REG_MULTI_SZ, both views),
authoritative whenever the value exists even when empty. Locking the registry
key is what makes the registry safe to treat as authoritative, since a standard
user can no longer write a pin.
### The shared key stays, deliberately
`plugin.endpoints.sharedkeyallowed` keeps the shared key working for PCs that
have NOT enrolled. It is always refused for a PC that holds an active credential
(`401 SHARED_KEY_REFUSED`). A PC whose credential was REVOKED falls back to the
shared key, which is what makes a rollback to an older agent possible;
`plugin.endpoints.revokeddeniessharedkey` turns that off and is off by default,
because it is only safe once the fleet is off the shared key.
## Alternatives rejected
**Client certificates, 2026-09-14.** This was not a thought experiment: it was
built and then removed three days later. `client_cert.py`, certificate renewal,
expiry monitoring, an unauthenticated `/test/client-cert` route, nginx
client-certificate forwarding and an agent certificate script all shipped on a
branch and were all deleted.
Three reasons, and the first is fatal on its own:
1. **The header was forgeable.** Behind IIS the client certificate reaches the
app as a request header, and nothing stopped a caller from simply sending
that header itself.
2. **The private key was readable by users** on the endpoint, so the certificate
was no better protected than the key it replaced.
3. **The shared key remained underneath it** as a fallback, so the weakest path
was still open and the certificate bought nothing.
It also cost more than it looked. The attempt put five underscore-spelled
certificate columns on the CORE `Asset` model for a plugin's feature and
retracted them again, with no contract bump in either direction - an ADR-001 and
ADR-002 breach inside an unreleased branch. Its migrations
(`endpoints0021clientcerts`, `0022`) survive as retired no-ops so a database
that ran them still upgrades; `0023` drops the columns where a database has
them.
**ShopDB-resolved credential targeting (ADR-016).** Decided 2026-08-11 and never
implemented. It gave a PC a credential chosen by rules over eight targeting axes
with an explicit priority order and an explain endpoint to debug the result. The
targeting was the whole design, and the exposure that needed closing was not a
targeting problem: the fleet needed identity, not a rule engine. ADR-016 is
superseded rather than deleted because the problem it opens with - a cleartext
controller password on a share, with no rotation story - is still unsolved. This
ADR does not solve it.
**Keep one key and just lock down where it lives.** The registry ACL is part of
this decision anyway, and it is worth having on its own. It does not help: one
secret shared by every PC still means the server cannot identify a caller, a
revoked bay cannot be evicted without rotating the whole fleet, and any
administrator or any process running as SYSTEM on any bay still reads the key
that drives every other bay's command channel.
## Consequences
**The shared key remains until every consumer is migrated.** That is the main
cost of this decision and it is not short. The key lives in a GE-Enforce
manifest entry's `Args`, in `CollectorKey` in both registry views, in a text file
in the share's `common/configs`, in `appsettings.json` in the agent directory, in
every generated `install.ps1` that was kept, and on display kiosks. Each one has
to be cleared in a specific order - retire the entry that rewrites the value
before deleting the value - and the agent never deletes the shared key itself,
because other SYSTEM scripts may still read it. The runbook
(`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`) carries the table.
**Turning the shared key off is wider than the agent.** It is also refused on
`/api/collector/computers` and `/api/collector/printers`, so a PC with reporter
scripts but no enrolled agent goes silent at cutover.
**Enrollment is a new admin queue.** Anything that cannot be auto-issued waits
for a person: unknown or duplicate hostname, missing or mismatched serial, a
one-off PC nobody wrote a manifest entry for. A dashboard row counts PCs waiting
and PCs blocked from re-enrolling.
**Pins have to arrive before the agent does.** An agent that pins with no pin set
falls back to plain Windows chain validation, which a self-signed or internal-CA
server may fail, and a pin set after the fact is a per-bay debugging job.
Rotating the server key becomes a two-phase fleet operation: publish old and new
pins everywhere, switch the certificate, then retire the old pin. Switching
first cuts off every pinned PC, including its route to the corrected pin list.
**GE-Enforce can undo an upgrade.** Agents installed from the share are held to a
file hash by a manifest entry. A store upgrade that lands a newer agent while the
entry still carries the old hash is reverted on the next cycle, so the entries
have to be re-stamped or removed before the waves reach those PCs.
**Revoke means two different things depending on one setting.** With
`revokeddeniessharedkey` off, revoking drops a PC back to the shared key, which
is how rollback and credential rotation work. With it on, revoking is a real
eviction and the PC reports nothing until it enrolls again.
**A PC that cannot reach ShopDB keeps its credential.** Unlike ADR-016's fetch
model, there is nothing to re-fetch per cycle: the secret is local and long-lived
until an admin revokes it.
## Implementation status
**Built, and in this tree.** `plugins/endpoints/api/agentauth.py`,
`api/enrollment.py` and `models/credential.py` are all present,
`shopdb/core/api/collector.py` reads `X-Endpoint-Credential` on the generic
collector route, `AGENT-CONTRACT.md` carries section 9, and
`tests/test_plugins/test_endpoints_credentials.py` covers the auth decisions.
The C# agent enrolls and sends the header, and both shipped PowerShell reporters
send it when a credential is present and fall back to the shared key when it is
not.
Proven on a Windows test PC against a test server: self-enrollment, the shared
key refused for an enrolled PC, a request naming another asset refused, pins
enforced, the tray pipe working for a non-admin user, both SYSTEM reporters
sending with the per-PC credential, revoke followed by re-enrollment, rollback
to the previous agent, and enrolled PCs still working with the shared key
switched off.
What is NOT finished is the fleet migration, which is the consequence this ADR
opens with: every PC that has not enrolled is still on the shared key by design,
and the key still lives in the places listed above. The runbook is
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
## References
- `plugins/endpoints/api/agentauth.py` - credential and shared-key authentication
- `plugins/endpoints/api/enrollment.py` - enroll, approve, revoke, migration waves
- `plugins/endpoints/models/credential.py` - credential, enroll request, wave
- `plugins/endpoints/AGENT-CONTRACT.md` section 9 - the wire contract
- `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md` - the rollout runbook
- ADR-016 - the superseded targeting model

69
docs/adr/README.md Normal file
View File

@@ -0,0 +1,69 @@
<!-- docmeta
purpose: reference
audience: developer, adopter
status: current
summary: Which architectural decision governs your question, and whether it still holds.
loadwith: none
-->
# Architecture Decision Records
An ADR captures one decision: the context, the decision, the consequences and
the alternatives. ADRs are immutable once accepted. A superseded ADR stays in
this folder, marked at the top, pointing forward.
**Read the index, not the folder.** Find the one line that matches your
question, open that ADR, and skip the rest. Nothing here is a prerequisite for
anything else.
## Which decision governs your question
| ADR | The question it answers | Holds? |
|---|---|---|
| [001](ADR-001-asset-as-platform-contract.md) | What may a plugin depend on? What is the versioned platform surface? | In force |
| [002](ADR-002-plugin-versioning.md) | How is that surface versioned, and what does a plugin's `core_version` mean? | In force |
| [003](ADR-003-plugin-distribution.md) | How does a plugin reach a site - bundled, or from its own repo? | In force. ADR-013 added the catalog, artifact and signing on top |
| [004](ADR-004-deployment-topology.md) | One instance per site, or one multi-tenant instance for all sites? | In force, except its "Migration strategy" section, replaced by ADR-008 |
| [005](ADR-005-equipment-vs-measuringtools.md) | Are metrology instruments a separate asset class from machinery? | **Superseded by [018](ADR-018-metrology-instruments-are-machines.md).** Answer reversed |
| [006](ADR-006-collector-contract.md) | How does a plugin accept data pushed in from outside? | In force |
| [007](ADR-007-product-versioning-and-releases.md) | What version is the PRODUCT, as distinct from the plugin contract? | In force |
| [008](ADR-008-plugin-migration-ownership.md) | Whose Alembic chain owns a plugin's tables? | In force |
| [009](ADR-009-frontend-plugin-gating.md) | What happens when someone opens a URL belonging to a disabled plugin? | In force |
| [010](ADR-010-frontend-plugin-hooks.md) | How does a plugin add UI without editing core Vue files? | In force |
| [011](ADR-011-machines-rename.md) | Why is the code called `machines` and what happened to `machinetypes`? | In force |
| [012](ADR-012-geenforce-manifest-ownership.md) | Does ShopDB own GE-Enforce manifests, and does it own the engine? | **Record only on this line.** The plugin it decided is not bundled here; ADR-013 and the API-token resource scopes trace back to it |
| [013](ADR-013-plugin-catalog-and-lean-builds.md) | How does a site ship only the plugins it chose, and how is plugin code trusted? | In force |
| [014](ADR-014-schema-lean-per-site.md) | Does a lean site still create tables for plugins it never installed? | In force |
| [015](ADR-015-site-specific-configuration.md) | Where may one site's hostnames, shares and defaults live? | In force |
| [016](ADR-016-credential-delivery.md) | How does a bay get a controller credential without a cleartext file on a share? | **Superseded by [019](ADR-019-per-pc-endpoint-credentials.md), and never built.** Do not build against it |
| [017](ADR-017-buildings-and-levels.md) | How does the floor map hold more than one floor and more than one building? | In force |
| [018](ADR-018-metrology-instruments-are-machines.md) | Where do CMMs and vision systems live now that `measuringtools` is retired? | In force, and this line is the one it describes: there is no measuring-tool asset class |
| [019](ADR-019-per-pc-endpoint-credentials.md) | How does the server know WHICH PC is calling, and what replaced the fleet-wide key? | In force |
Related decisions that are not ADRs, kept as records of why something was built:
[docs/proposals](../proposals/README.md).
## Status vocabulary
The `Status:` line at the top of each ADR uses these, and nothing else:
- **PROPOSED** - drafted, awaiting decision.
- **ACCEPTED** - in effect.
- **SUPERSEDED by ADR-NNN** - replaced. The link forward is part of the status,
and the superseded ADR carries a note at the top saying not to build against
it and what changed.
- **DEPRECATED** - no longer in effect, with no replacement.
## Authoring
1. Copy the most recent ADR as a template and increment the number.
2. Start the file with a `docmeta` comment, before the H1. ADRs are
`purpose: explanation`.
3. Add a row to the table above: the QUESTION it answers, not its title.
4. Add it to [docs/INDEX.md](../INDEX.md) too - under "Decisions" while it is
current, under "Archive" once it is superseded. It has to be a table ROW
with its real line count, not a mention in a paragraph:
`tests/test_docs_index.py` fails on an ADR no row routes to, and on a row
whose line count is stale.
5. Never edit an accepted ADR's decision. Supersede it with a new one, and mark
the old one at the top - ADR-005 and ADR-016 are the worked examples.

13107
docs/api-inventory.json Normal file
View File

File diff suppressed because it is too large Load Diff

87
docs/deployment/DOCKER-PREBUILT-APPROACH.md Normal file
View File

@@ -0,0 +1,87 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build the Docker image where npm cannot reach the registry, by compiling the frontend on the host first.
loadwith: none
-->
# Building the image with a pre-built frontend
> **Where this sits:** a variant of Step 2 of [DEPLOY](../DEPLOY.md), not a
> separate deployment. Everything else on that page still applies.
The shipped `Dockerfile` is a multi-stage build: stage 1 runs `npm ci` and
`npm run build` under Node, stage 2 copies the compiled SPA into the Python
image. Behind a proxy that re-signs HTTPS, stage 1 is where it dies - npm inside
the container does not trust the corporate root CA, and the failure reads
`npm ci failed` or the unhelpful `Exit handler never called`.
The fix is to build the frontend where npm already works - the host, which has
the corporate CA in its trust store - and give the image the finished
`frontend/dist/`.
It is also much faster: two to three minutes instead of fifteen. A site with no
proxy problem at all may want it for that reason alone.
## Do it
```bash
# 1. Build the frontend on the host
cd frontend
npm ci
npm run build # writes frontend/dist/
cd ..
# 2. Build and start the stack from the pre-built dist
docker compose -f docker-compose.prebuilt.yml build
docker compose -f docker-compose.prebuilt.yml up -d
```
The rest is [DEPLOY](../DEPLOY.md) unchanged: the `migrate` service runs the
schema, the plugin bootstrap and the seeders before `api` starts, nginx
terminates TLS on `API_PORT`, and the first admin comes from `/setup` or
`flask seed admin`.
## The files this uses
| File | What it is |
|---|---|
| `Dockerfile.prebuilt` | single-stage: Python dependencies plus a copy of `frontend/dist/`. No Node, no npm |
| `docker-compose.prebuilt.yml` | the same stack, building from `Dockerfile.prebuilt` |
| `.dockerignore.prebuilt` | permits `frontend/dist/` to be copied, which the normal `.dockerignore` excludes |
## What it costs
- **npm has to exist on the build host.** That is the whole trade: the trust
problem moves from the container to a machine that has already solved it.
- **It is two steps, and forgetting the first one ships a stale UI.** The image
copies whatever is in `frontend/dist/` at build time. A frontend change that
was not rebuilt is simply not in the image, and nothing warns you. If the UI
does not show your change, rebuild `dist/` before debugging anything else.
- **`frontend/dist/` has to reach the build.** Either it is committed, or the
pipeline builds it immediately before the image.
## The version of this that scales
The pre-built approach is a workaround for building at the deployment site. The
end state is not building there at all:
```yaml
# CI: build the frontend, build the image, push it
- run: cd frontend && npm ci && npm run build
- run: docker build -f Dockerfile.prebuilt -t shopdb:$GIT_SHA .
- run: docker push <registry>/shopdb:$GIT_SHA
```
A site then pulls a tagged image and runs it - seconds, no build step, no npm
anywhere near the facility. `docker-compose.pull.yml` is the compose file for
that shape today, and `docker-compose.airgap.yml` is the same idea for a site
that loads an image tarball off removable media
([DEPLOY-AIRGAP](../DEPLOY-AIRGAP.md)). Finishing that path is the distributed
release item on [ROADMAP](../ROADMAP.md).
| Approach | Build time | Behind a re-signing proxy | Where it belongs |
|---|---|---|---|
| Multi-stage `Dockerfile` | 15+ min | fails unless the CA is staged | a developer box with a clean network |
| Pre-built frontend (this page) | 2-3 min | works | a site that must build locally |
| Pull a published image | seconds | works | where every site should end up |

372
docs/deployment/PRODUCTION-READINESS.md Normal file
View File

@@ -0,0 +1,372 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: historical
summary: The 2026-09-07 assessment of why deployment was hard. Its gaps became the roadmap; its recommendation was reversed.
loadwith: none
-->
# Production readiness assessment, 2026-09-07
> **This is a record, not guidance.** Read it for WHY the deployment story
> needed work, not for what to do. Two things have moved since:
>
> - **Its recommendation was reversed.** It recommends Windows/IIS for the
> reference site and Docker only for new ones. The decision since is the
> other way round: Docker is the path for every site and Windows/IIS is the
> legacy topology that will be deprecated.
> - **Its "critical path" became the roadmap.** Pre-built images in a registry,
> one-command setup, automatic plugin registration and a simpler backup mode
> are now the distributed-release item on [ROADMAP](../ROADMAP.md). Several
> of its gaps are already closed: the `migrate` service runs the schema, the
> plugin bootstrap and the seeders with no manual step, and
> `docker-compose.prebuilt.yml` removes npm from the image build.
>
> Current pages: [DEPLOY](../DEPLOY.md) to stand a site up,
> [DOCKER-PREBUILT-APPROACH](DOCKER-PREBUILT-APPROACH.md) to build it behind a
> proxy, [ROADMAP](../ROADMAP.md) for what is left.
## Executive Summary
[CRITICAL] **NOT PRODUCTION READY** in current state without significant operational investment.
The Docker deployment requires extensive manual setup, multiple services, and assumes expertise with Docker Compose, restic backup systems, and Flask plugin architecture. This assessment documents current gaps and provides recommendations.
## Current State: Deployment Complexity
### What Works
[OK] **Flask application code** - Stable, tested, 17 plugins functional
[OK] **Database schema** - Migrations work, data model is sound
[OK] **Frontend build** - Vue 3 SPA compiles and serves correctly
[OK] **Docker images** - Build successfully (with caveats below)
[OK] **Plugin architecture** - Clean, documented, extensible
### Critical Gaps
[FAIL] **One-command deployment** - Requires 7-10 manual steps
[FAIL] **Corporate proxy handling** - Zscaler cert extraction needed for npm
[FAIL] **Backup system complexity** - 3 containers + restic + scheduler
[FAIL] **TLS certificate generation** - Manual OpenSSL commands required
[FAIL] **Plugin initialization** - Must run `apply-profile` after first start
[FAIL] **Password reset** - No default admin password, requires manual SQL
[FAIL] **Build time** - 15+ minutes with corporate proxy
[FAIL] **Documentation scattered** - 3 different setup guides (Docker, Local, Backup)
## Why Is This So Complex?
### 1. Corporate Network Reality
**Problem:** Behind Zscaler proxy, npm can't trust npm registry
**Impact:** Docker build fails without extracted corporate root CA
**Workaround:** PowerShell script to extract cert before building
**Production Fix:** Pre-build images in CI/CD with cert, publish to registry
### 2. Plugin Architecture Design
**Problem:** Plugin state in `instance/plugins.json`, not database
**Impact:** Importing prod database doesn't register plugins
**Workaround:** Manual `flask plugin apply-profile` after first start
**Production Fix:** Auto-detect installed plugins from schema on boot
### 3. Backup System Architecture
**Problem:** Designed for restic with 3-container orchestration
**Impact:** Requires backup-store + backup-watcher + backup containers
**Workaround:** All documented, but complex to deploy
**Production Fix:** Simpler mysqldump-only mode with direct file storage
### 4. Multi-Stage Docker Build
**Problem:** Frontend build needs Node 20, backend needs Python 3.14
**Impact:** Multi-stage build is slow, complex, hard to debug
**Workaround:** Build works, but takes 15 minutes
**Production Fix:** Pre-build frontend on CI, copy dist/ into Python-only image
## Deployment Models (Ordered by Simplicity)
### Option 1: IIS + Windows Services (SIMPLEST)
**Best for:** Single Windows server deployment
**Complexity:** Low
**Maintenance:** Medium
**Setup:**
1. Install Python 3.13+
2. Install MySQL (or use existing instance)
3. Run installer (creates Windows service for Flask)
4. Configure IIS reverse proxy
5. Install plugins via UI
**Pros:**
- Native Windows deployment
- Familiar IIS management
- No Docker complexity
- Fast startup
**Cons:**
- Windows-only
- Manual updates
- No containerization benefits
**Production Ready:** [WARN] **YES** (with installer improvements)
### Option 2: Docker Pre-Built Images (RECOMMENDED)
**Best for:** Sites with Docker but no build capability
**Complexity:** Low-Medium
**Maintenance:** Low
**Setup:**
1. Pull pre-built image from registry
2. Run docker-compose with .env file
3. Access UI, enable plugins
**Pros:**
- No build step
- Fast deployment (2 minutes)
- Consistent environment
- Easy updates (pull new image)
**Cons:**
- Requires Docker
- Need image registry
- Initial registry setup
**Production Ready:** [OK] **YES** (requires CI/CD setup)
### Option 3: Docker Source Build (CURRENT)
**Best for:** Development, customization
**Complexity:** High
**Maintenance:** High
**Setup:**
1. Extract Zscaler cert
2. Generate TLS certs
3. Build Docker images (15+ min)
4. Configure .env with 15+ variables
5. Start docker-compose
6. Run migrations
7. Install plugins
8. Reset admin password
9. Configure backup system
10. Test everything
**Pros:**
- Full source control
- Customizable
- No external dependencies
**Cons:**
- Extremely complex
- Long build times
- Many failure points
- Requires Docker expertise
**Production Ready:** [CRITICAL] **NO** (too complex for operations)
### Option 4: Kubernetes (FUTURE)
**Best for:** Multi-site deployment, high availability
**Complexity:** Very High
**Maintenance:** Medium (with proper GitOps)
**Status:** Not implemented
**Effort:** 2-3 weeks of work
## What Would Make This Production Ready?
### Critical Path (Must Have)
1. **Pre-Built Images in Registry**
- Build images in CI/CD
- Push to container registry
- Sites just `docker pull`
- **Effort:** 1 week
2. **One-Command Setup**
- `./deploy.sh` does everything
- No manual cert extraction
- No manual plugin installation
- **Effort:** 3 days
3. **Automated Plugin Detection**
- Scan database schema on boot
- Auto-register plugins in JSON
- No manual `apply-profile`
- **Effort:** 2 days
4. **Simplified Backup (Option)**
- Simple mysqldump + tar mode
- Single container, no restic
- Good enough for 80% of sites
- **Effort:** 1 week
5. **Windows Installer Improvements**
- One-click install
- Includes Python runtime
- Creates service automatically
- **Effort:** 1 week
### Nice to Have (Future)
- Helm chart for Kubernetes
- Monitoring/metrics built-in
- Health check endpoints
- Automated testing in CI
- Rolling update strategy
## Honest Recommendations
### For the reference site
**Recommended:** Option 1 (IIS + Windows Services)
*Reversed since. See the note at the top of this page.*
**Why (as argued at the time):**
- The site already had Windows infrastructure
- Simpler to operate
- Faster to deploy
- The team knew IIS
**Steps:**
1. Build Windows installer
2. Deploy to Windows Server
3. Configure IIS reverse proxy
4. Import prod database
5. Done
**Timeline:** 2-3 days including testing
### For New Sites
**Recommended:** Option 2 (Docker Pre-Built Images)
**Prerequisites:**
- Set up CI/CD to build images
- Set up container registry (Docker Hub, Harbor, or AWS ECR)
- Create `deploy.sh` script
**Then sites can:**
```bash
# 1. Copy .env.example to .env and edit
cp .env.example .env
# 2. Run deployment
./deploy.sh
# 3. Access UI
https://your-server:5001
```
**Timeline for CI/CD setup:** 1-2 weeks
### For Current Docker Source Build
**Status:** Development/customization only, not for production
**Use when:**
- Testing new features
- Developing plugins
- Debugging issues
- Learning the system
**Don't use when:**
- Deploying to operations
- Training new ops staff
- Time-sensitive deployments
## Current Documentation Issues
### Too Many Guides
- `LOCAL-DEV-SETUP.md` - Windows development (port 3307, run_dev.py)
- `DOCKER-SETUP.md` - Docker source build (complex, 15+ steps)
- `PRODUCTION-READINESS.md` - This file (meta/assessment)
- `scripts/backup/README.md` - Backup system (3 containers, restic)
*Closed since: the first two were folded into `DEVELOPMENT-SETUP` and `DEPLOY`,
and `INDEX.md` is now the single entry point this section asks for.*
**Problem:** Which one do I follow?
**Solution:** Create decision tree:
- Developer on Windows -> LOCAL-DEV-SETUP.md
- Ops deploying to prod -> (DOESN'T EXIST YET - need simplified guide)
- Custom development -> DOCKER-SETUP.md
- Backup questions -> scripts/backup/README.md
### Missing Guides
- **PRODUCTION-DEPLOY.md** - Step-by-step for operations (IIS or Docker pull)
- **QUICK-START.md** - 5-minute version for evaluation
- **TROUBLESHOOTING.md** - Common issues, not scattered across files
- **UPGRADE-GUIDE.md** - How to update between versions
## What We Did Today (Honest Assessment)
[OK] **Merged the upstream development branch down** - 98 commits
[OK] **Fixed local dev** - `run_dev.py` works for Windows development
[OK] **Documented Docker** - Comprehensive but too complex
[OK] **Added backup docs** - System works but needs expertise
[WARN] **Docker "production ready"** - Technically works, operationally complex
[FAIL] **Simple deployment** - Still requires too many manual steps
## Next Steps (Priority Order)
### Immediate (This Week)
1. **Create QUICK-START.md**
- 5-minute eval version
- Uses pre-built image (if available) OR local dev
- Gets people to working system fast
2. **Test Windows Installer**
- Build the installer for the deployment branch
- Deploy to test Windows Server
- Document any issues
3. **Create deployment decision tree**
- Flowchart: what's my situation -> which method to use
- Don't make users read 4 docs to figure out where to start
### Short Term (Next Sprint)
4. **Set up CI/CD for image builds**
- GitHub Actions or Jenkins
- Build on every push to the deployment branch
- Push to internal registry
5. **Create `deploy.sh` script**
- One command: `./deploy.sh --env prod`
- Handles certs, plugins, migrations automatically
6. **Simplify backup system**
- Add "simple mode" - just mysqldump + tar
- Keep restic for advanced sites
- Document tradeoffs
### Long Term (Next Month)
7. **Auto-plugin detection**
- Boot-time schema scan
- No manual `apply-profile`
8. **Consolidate documentation**
- One entry point
- Clear paths for each audience
- Tested by someone who hasn't seen the system
## Conclusion
**Is ShopDB production-ready?**
- **The code:** Yes, it's solid
- **Docker source build deployment:** No, too complex
- **Windows installer deployment:** Yes, with improvements
- **Docker pre-built images deployment:** Yes, but needs CI/CD setup
**Bottom line:** The application is mature. The deployment story needs work. Focus should shift from "making Docker work" to "making deployment simple."
**Recommendation:** Invest 2-3 weeks in deployment automation (CI/CD, pre-built images, deploy.sh) rather than continuing to document complex manual processes.

125
docs/llms.txt Normal file
View File

@@ -0,0 +1,125 @@
# ShopDB - guide for an AI agent
ShopDB is a plugin-based asset-management system (PCs, printers, machines,
network devices, applications, knowledge base, guides, notifications, slides,
warranties, backups, endpoint telemetry) for manufacturing sites. Every site
runs its own instance.
This file is the machine entry point, served at `/api/docs/llms.txt`. It covers
the API, which is what most callers need, and routes everything else.
## Routing - read this before opening any document
`docs/INDEX.md` is the routing table. Rows are TASKS, each row gives the
smallest set of pages that answers it, and each row says how many lines that
costs. Find the row, read those pages, skip the rest. The corpus is far larger
than any one context window and almost none of it applies to a given question.
Every page opens with an HTML comment naming `purpose`
(tutorial/howto/reference/explanation), `audience`, `status` and a one-sentence
`summary`, plus `loadwith` for the pages it is not complete without. Filter on
that metadata before opening anything: `status: legacy` means true but
superseded (do not follow it for new work) and `status: historical` means a
finished migration or a proposal kept as a record, so neither is an answer to
"how do I do this today".
`docs/START-HERE.md` is the same routing for a human, and shorter.
Not an API question? Still start from the index rather than guessing at a
filename: its rows cover installing and running a site, loading its data, the
shop-floor PCs, and building on the platform. For "why is it built this way",
`docs/adr/README.md` says which decision governs the question and whether it
still holds. For writing a plugin, `docs/PLUGIN-HOOKS.md` is the contract and
everything else defers to it.
## Answering "how do I install or fix this server"
Docker is the topology this line ships, and the only one. `docs/DEPLOY.md` is a
new site at a connected facility, `docs/DEPLOY-AIRGAP.md` is the same stack
where nothing can be pulled, and `docs/UPGRADE.md` moves a running site to a
newer version. There is no Windows installer here.
Before diagnosing anything, ask for `docker compose ps` and
`curl -s http://127.0.0.1:5002/api/dashboard/health`. Comparing that bypass with
the site's real URL separates "the app is broken" from "nginx or the certificate
is broken". `docs/DEPLOY.md` has the rest of the health surface, and
`docs/CONFIG.md` is where every environment variable and Setting key is defined.
## Bulk-loading a site's data
Two routes, and the right one depends on what the site has:
- SPREADSHEET, no developer (the common case): `flask csv templates --out <dir>`
generates templates FROM THE LIVE SCHEMA, then `flask csv import --dir <dir>`
checks and `--commit` applies. Do NOT hand-write the templates - generate
them. `docs/CSV-IMPORT.md`.
- A SOURCE DATABASE to script against: the HTTP import API,
`docs/IMPORT-API.md`, with `docs/IMPORT-ADOPTION.md` for the loader around it.
User accounts are deliberately not CSV-importable.
## Base URL
Each site runs its own instance (ADR-004), so there is no single production URL:
it is whatever that site serves, e.g. `https://shopdb.example.net/shopdb`.
All API paths are under `/api` (e.g. `<base>/api/assets`). Dev: `http://localhost:5001`.
## Auth
Three schemes:
- **Bearer JWT** - most endpoints. Get one by logging in, or use a managed
Personal Access Token (PAT). Send `Authorization: Bearer <token>`.
- Login: `POST /api/auth/login` `{ "username": "...", "password": "..." }`
-> `data.access_token`. Refresh: `POST /api/auth/refresh`.
- PATs are minted in the UI (Settings > API Tokens); a *scoped* PAT is limited
to named permissions and suspends the admin bypass.
- **X-API-Key** - unattended/service endpoints (collector ingest, and any
plugin's own unattended routes). Send `X-API-Key: <managed-token>`. How a shop-floor PC gets its own
credential is `docs/COLLECTOR-AUTH.md`.
- **Public** - some read endpoints (e.g. printer install-list, the slide feed,
display dashboards) need no auth.
Auth level per endpoint is in the OpenAPI `security` field: `bearerAuth`,
`apiKeyAuth`, or none. Admin-only and permission-gated routes both use bearer.
## Response envelope
JSON endpoints return `{ "status": "success", "data": <payload>, "meta": {...} }`.
Errors nest under `data`, NOT at the top level:
`{ "status": "error", "data": { "error": { "code": "...", "message": "...", "details": {...} } } }`
with an HTTP 4xx/5xx. Reading `message` or `code` off the root gives `undefined`.
Lists carry `meta.pagination.{page, perpage, total, pages}` - again nested, not
`meta.total`. A few feed endpoints (screensaver, some installer text formats)
return raw text/JSON without the envelope - noted per route.
## Common recipes
- Search everything: `GET /api/search?q=<term>` (multi-word = AND across words).
- List assets on the map: `GET /api/assets/map`.
- List a type: `GET /api/printers`, `/api/computers`, `/api/machines`,
`/api/network` (paginated: `?page=&perpage=`). A plugin that is disabled at
that site serves none of its routes.
- Get one: `GET /api/printers/<id>` etc.
- Create (bearer): `POST /api/printers` `{assetnumber, windowsname, vendorid, ...}`.
- Reports: `GET /api/reports` (list), `GET /api/reports/pc-relationships` (PC<->machine).
- Printer installer data: `GET /api/printers/install-list` (public; add
`?format=text` for a pipe-delimited variant); `GET /api/printers/pc-default?machine=<n>`.
- Collector ingest (X-API-Key): `POST /api/collector/computers`,
`/api/collector/printers`, `/api/collector/endpoints`. How a PC gets its own
credential is `docs/COLLECTOR-AUTH.md`.
- Import (admin PAT, preserves timestamps with `X-Import-Mode`): `docs/IMPORT-API.md`.
## Conventions
- DB-mirrored params/fields use lowercase concatenated names (no underscores):
`locationid`, `vendorid`, `windowsname` - match them exactly.
- IDs in paths are integers.
- Plugin endpoints live under the plugin's prefix (`/api/<plugin>/...`).
## Full reference
- Machine spec: `GET /api/docs/openapi.json` (OpenAPI 3.1). Operation and
plugin counts live in `docs/PROJECT-MAP.md`, which is generated.
- Interactive: `GET /api/docs` (Redoc).
- Human reference: the same spec at `GET /api/docs`, and `docs/INDEX.md` for everything else.

48940
docs/openapi.json
View File

File diff suppressed because it is too large Load Diff

27
docs/proposals/README.md Normal file
View File

@@ -0,0 +1,27 @@
<!-- docmeta
purpose: reference
audience: developer
status: current
summary: What each proposal designed, whether it was built, and which ADR decided it.
loadwith: none
-->
# Proposals
A proposal is a design written before the work, kept afterwards as the record of
why the thing looks the way it does. A proposal is NOT a decision record: where
a proposal and an ADR disagree, the ADR wins.
**None of these is current guidance.** Read one to understand why something
exists, not to learn how it works today - for that, read the page named in the
right-hand column.
| Proposal | What it designed | Built? | Read instead |
|---|---|---|---|
| [dashboard-live-fleet](dashboard-live-fleet.md) | A dashboard that shows what needs doing instead of row counts | Yes | [ADR-010](../adr/ADR-010-frontend-plugin-hooks.md) for the contract it introduced |
| [device-identity](device-identity.md) | One durable identity for the device a PC drives, ending in the registry | Phase 1 built (the collector `deviceid` field). Phases 2 and 3 not started | The proposal itself, for the remaining phases |
| [printer-assignment](printer-assignment.md) | Printers as a property of the machine, installed by the PC | Yes, v0.11.3 and v0.12.0 | [PRINTER-INSTALLER](../PRINTER-INSTALLER.md), [COLLECTOR-INTEGRATION](../COLLECTOR-INTEGRATION.md) |
A proposal that is adopted says so in its own status line, and names what
adopted it. A proposal nobody built stays here anyway: the alternatives it
rejected are the part worth keeping.

179
docs/proposals/dashboard-live-fleet.md Normal file
View File

@@ -0,0 +1,179 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 case for a worst-first dashboard queue. Built; kept as the record of why.
loadwith: none
-->
# Proposal: a dashboard that shows the fleet, not the row count
- Status: IMPLEMENTED (a proposal, not a decision record - the
contract it introduced is ADR-010 and contract 0.19.0)
- Date: 2026-08-11
- Author: ShopDB maintainers
- Relates to: ADR-010 (frontend plugin hooks), ADR-013 / ADR-014 (lean per-site builds), ADR-006 (collector contract), ADR-012 (GE-Enforce manifest ownership)
## The problem
The dashboard reports totals: assets, machines, PCs, network devices, printers,
measuring tools. Those numbers are true every day and actionable on none of
them. Nobody opens the page to learn there are 417 assets.
Meanwhile the system already knows things worth acting on and shows them
nowhere:
- a bay returned 500 to every collector report for a day and a half, and the
only evidence was a log file on the PC
- eleven part markers' backups were overwriting each other for weeks
- a PC can stop reporting entirely and nothing says so
- GE-Enforce records enforcement failures, stale manifests and repeated
self-heals, and none of it surfaces outside a report modal
Every one of those was found by someone going to look. The dashboard should be
where they find you.
## What already exists
Two discoveries shaped this proposal.
**The plugin mechanism is built and dead.** `BasePlugin.get_dashboard_widgets`
exists, `GET /api/dashboard/widgets` aggregates across enabled plugins, skips
disabled ones, isolates a broken plugin in production and sorts by position.
Five plugins - notifications, network, machines, computers, printers - already
declare widgets.
**Nothing consumes it.** The frontend never calls the endpoint, and not one of
the declared components (`PrinterStatusWidget`, `ComputerStatusWidget`,
`MachineStatusWidget`, `NotificationsWidget`, `NetworkStatusWidget`) exists.
The declarations point at components that were never written.
So this is less "build a dashboard" than "finish one", plus a contract fix.
**The data is already collected.** Nothing here needs a new collector field:
| Signal | Source |
|---|---|
| PC reporting / silent | `computers.lastreporteddate` |
| Machine has a controlling PC | `controls` links, `collector:machine` |
| Backed up, stale, never | `backuprevisions` per asset and kind |
| Enforcement failed | `manifestenforcementreports.status`, `failedcount` |
| Which entry failed, and why | `manifestenforcementresults.action`, `exitcode`, `message` |
| Stale manifest | report `appliedversion` versus the scope's published version |
| Drift that will not stick | repeated `selfhealed` results for one entry |
| Enforcer version spread | `manifestenforcementreports.enforcerversion` |
| Installed software and versions | `computerinstalledapps` |
| Toner critical / low | printer supplies, Zabbix-fed |
| Warranty expiring | warranty plugin |
| Mis-numbered bays | the `check-shared-machines` query |
## Proposed change to the widget contract
The existing contract names a **component per widget**. That does not scale and
does not survive a lean build: every widget needs a bespoke Vue component, and a
plugin-provided component has to be staged into the frontend bundle to exist at
all.
ADR-010 already solved this for asset panels: a plugin declares `render: 'list'`
or `'keyvalue'` or `'tabs'` plus a `map` of fields, and a GENERIC core component
renders it. Do the same here. A plugin declares data and shape; core owns the
rendering.
```python
{
'id': 'geenforce-failures',
'title': 'Enforcement failures',
'endpoint': '/api/geenforce/dashboard/failures',
'render': 'exceptions', # generic renderer
'severity': 'critical', # critical | warning | info
'permission': 'geenforce.view',
'position': 10,
'empty': 'hide', # hide | line
'map': {
'title': 'hostname',
'detail': 'entryname',
'meta': [{'key': 'message'}, {'key': 'exitcode'}],
'link': '/pcs/{computerid}',
},
}
```
Three renderers cover everything listed above:
- `exceptions` - a list of things that are wrong, each linking to the thing
- `metric` - a single number with a trend or threshold, for the cases where the
count IS the story (toner critical: 3)
- `list` - recent items, for notifications and KB adds
## Principles
**An exception board, not a stat board.** The default view is what needs a
person today. A count appears only where the count is the story.
**Empty cards must shrink.** A card that says "nothing wrong" every day trains
people to stop reading the page. That is exactly how `ntlars-backup.log` reached
3,234 lines with 17 that mattered. `empty: 'hide'` is the default; `'line'` for
the few where absence is itself news.
**Every row links to the thing.** A dashboard that says three printers are low
without linking to them is a worse version of a report.
**Cards respect RBAC.** A card declares a `permission`; a user without it never
sees the card or its endpoint. Toner levels are not sensitive, but "which PCs
expose VNC" is, and the dashboard must not become a way around role gating.
**Thresholds are settings with neutral defaults** (ADR-015): quiet window, toner
percentage, warranty horizon. Every site will disagree with the numbers.
**One endpoint per card, lazily loaded, failing independently.** A hung Zabbix
call must not blank the page.
## Wave one
Cards whose data is already reliable, ordered by how actionable they are:
1. **Enforcement failures** - current reports with `status = failed`, showing
the failing entry and exit code.
2. **PCs not reporting** - `lastreporteddate` beyond the quiet window, and PCs
with no GE-Enforce report at all, which is a different and worse case.
3. **Backups missing or stale** - machines with no revision of a kind their
PC type should produce, and revisions older than the interval.
4. **Toner critical, then low** - links to the printer, then to its admin page.
5. **Warranties expiring** - within the configured horizon.
6. **Mis-numbered bays** - `check-shared-machines`, promoted from a CLI command
nobody will remember to run.
Then, lower and quieter: stale manifest versions, repeated self-heals, enforcer
version spread, active notifications, recent application and KB additions.
## Wave two: desired versus observed
The highest-value card and the most work. The manifest declares what a PC type
should have; `computerinstalledapps` records what it does; `filters.py` already
decides which entries apply to a given PC. Diffing those answers "which bays are
missing something they should have" - which nothing in the system answers today,
and which is the thing GE-Enforce exists to guarantee.
Deferred to wave two because it needs the resolver wired into a query path, and
because wave one needs nothing new.
## Risks
**Aggregate queries on the most-visited page.** These are fleet-wide scans on
every load. Cache 30-60 seconds, and log slow cards - a lazily-loaded card can
be slow for months before anyone mentions it.
**Cards become the new noise.** Six cards that are always empty are as useless
as one number that never changes. If a card is empty for a month, delete it.
**Plugin gating must be real.** A lean site without `printers` must render no
toner card. The existing endpoint already skips disabled plugins; the frontend
must not hardcode a card list alongside it.
## Non-goals
- Per-user dashboard customisation. Later, if asked for.
- Historical charting. This is a "what needs doing now" board; trends belong in
reports.
- Replacing the shopfloor TV dashboard, which is a different audience with
different needs.

235
docs/proposals/device-identity.md Normal file
View File

@@ -0,0 +1,235 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Plan for one durable identity for the device a PC drives. Phase 1 is built here; phases 2 and 3 are not started.
loadwith: none
-->
# Proposal: one identity for the device a PC drives, ending in the registry
Status: PHASE 1 BUILT AND IN THIS TREE - `plugins/computers/plugin.py` declares
`deviceid` on the computers collector schema and prefers it over the older
per-family fallbacks. Phases 2 and 3 not started.
Author: planning session 2026-08-20.
## 1. What this is
A shop-floor PC often drives something: a CMM, a Keyence, a Genspect, a
wax-trace unit, a part marker. ShopDB needs to know WHICH one, and the answer
has to survive the PC being replaced.
Today that answer is assembled from whatever each bay family happens to have,
and two of the three sources are derived from the PC itself, so they do not
survive a swap. This proposal converges them on one identity, moves it from
files to the registry, and does it in three phases so no bay is stranded.
## 2. Why PC-derived identity fails
Two mechanisms stand in for a real identity today:
- **Reuse**: look for a prior collector link from THIS PC asset.
- **Mint**: build `<PC assetnumber>-<SUFFIX>`, and a PC's asset number is its
hostname.
Both survive a re-image. Neither survives a PC SWAP. A new hostname is a new PC
asset with no prior link and a predicted number that has never existed, so the
same physical device gets a SECOND record while the first keeps its config and
its backup history under a dead PC's name.
That is not hypothetical: minting from the hostname left 43 legacy `MT-####`
tools shadowed by minted twins, three records deep in places.
## 3. The rail that must not be crossed
`machinenumber` answers "WHICH BAY is this" and is what GE-Enforce
`TargetMachineNumbers` gates on. The device identity answers "WHICH DEVICE is
this". They are different facts and must stay in different fields.
Putting a device id into the machine-number channel would silently stop every
bay-gated manifest entry from matching - a failure that looks like nothing
happening, on every cycle, with no error anywhere.
## 4. Where identity comes from today
| Bay family | Identity source | Survives a PC swap |
|---|---|---|
| DNC / collections | `MachineNo` in the eDNC/NTLARS registry | yes |
| CMM | `C:\Enrollment\cmm\cmmid.txt` | yes, but see below |
| Keyence, Genspect, wax-trace | nothing, before phase 1 | no |
| Part markers | nothing, before phase 1 | no |
**`cmmid.txt` is not only an identity file.** It is a PATH KEY, and that is why
it cannot simply be retired:
| Consumer | Use |
|---|---|
| `resolve-cmm-bay-config.ps1` | writes it |
| `Verify-And-Heal-Staging.ps1` | stages `installers-post\cmm\backups\<cmmid>` |
| `09-Setup-CMM.ps1` | locates the bay's staged backup set |
| `Restore-CMM.ps1` | restores from `<BackupRoot>\<cmmid>\`, same-bay by design |
| collector | reads it as `machineNo` |
Only the last row is an identity use. The rest address directories by it.
It is also doing the job badly: for a CMM bay it reports the INSTRUMENT's id in
the `machinenumber` channel, and the server then resolves it back to a measuring
tool at step 4 of its own fallback chain. That round trip is what phases 2 and 3
remove.
## Phase 1 - one field, one file (DONE)
The collector accepts a single `deviceid`, read from `C:\Enrollment\asset-id.txt`
and resolved BEFORE anything PC-derived, for every device family.
- One file for every bay with no `MachineNo` registry to identify it.
- It names the asset and NOT its type: the pc-type already says what kind of
device it is, through `SUBORDINATE_DEVICE_MAP`, so a device type declared
later through `subordinatedevice_<pctype>` needs no new file and no client
change.
- Wire field is `deviceid`, not `assetid`, because `assetid` is already this
contract's RESPONSE field for a PC's integer primary key.
- `measuringtoolid` (0.12.0) is accepted as an alias and the older file is still
read, so a bay staged in that window keeps reporting.
- Guards: an unresolvable value warns rather than minting a phantom, and a value
resolving to the WRONG KIND of asset is refused with the asset named.
Nothing was removed. Every previous fallback still runs, one place lower.
## Phase 2 - imaging writes it, and `cmmid.txt` stops being an identity
Repository: the PXE/imaging repo, not this one. **Needs deploying to the live
server, not only committing** - the live boot image is authoritative.
1. `resolve-cmm-bay-config.ps1` writes `C:\Enrollment\asset-id.txt` ALONGSIDE
`cmmid.txt`. Two files, two questions: the identity, and the backup-set key.
`cmmid.txt` keeps every path use it has today.
2. Every other device pc-type writes `asset-id.txt` at imaging from its bay
config. Bays with no device write nothing, and the file's absence stays
meaningful.
3. The collector demotes `cmmid.txt` to last place in the machine-number chain
and stops being the intended path for CMM identity. It is still read, so a
grandfathered bay that never gets re-imaged keeps reporting.
4. The value written is the device's `assetnumber` as ShopDB holds it. Where a
CMM's asset number is already its bay id (`CMM3`), phase 2 is a no-op in
content and a change only in which file carries it.
Exit criterion: a report from a re-imaged bay of each device family carries
`deviceid`, and no bay of those families relies on the `cmmid.txt` fallback.
The collector response warnings are the check: an adopted device is silent, a
minted one is not.
## Phase 3 - identity moves to the registry
`HKLM:\SOFTWARE\GE\ShopDB\DeviceId`.
This is the destination because the registry survives the profile cleanup and
disk hygiene that eats `C:\Enrollment`, is readable as SYSTEM with no file ACL
to get wrong, and is already how a DNC bay is identified. It needs no new key
and no new read path: the collector already opens
`HKLM:\SOFTWARE\GE\ShopDB` (and the `WOW6432Node` variant) for `BaseUrl` and
`ApiToken`, and `Install-GEEnforce.ps1` already writes that key.
**One value, every family.** `DeviceId` is a single registry value, not a key
per device type. The client sends one field and the pc-type decides which sync
consumes it, exactly as `asset-id.txt` works in phase 1, so:
| Bay | `DeviceId` holds | Replaces |
|---|---|---|
| CMM | the instrument's `assetnumber` | `cmmid.txt` as IDENTITY only |
| Keyence | the instrument's `assetnumber` | nothing; had none |
| Genspect | the instrument's `assetnumber` | nothing; had none |
| wax-trace | the instrument's `assetnumber` | nothing; had none |
| part marker | the marker's `assetnumber` | minting from the hostname |
A device family added later through `subordinatedevice_<pctype>` is covered on
the day it is declared, with no client change and no new registry value.
Resolution order after phase 3:
1. `HKLM:\SOFTWARE\GE\ShopDB\DeviceId`
2. `C:\Enrollment\asset-id.txt`
3. `C:\Enrollment\measuringtool-id.txt` (0.12.0 window)
4. `C:\Enrollment\cmm\cmmid.txt` as `machineNo` (grandfathered CMM bays)
Then, and only then, the file branches can be deleted one at a time, newest
first, each when no bay has reported through it for a full inventory cycle.
**`cmmid.txt` survives phase 3 regardless**, as the CMM backup-set path key. It
is not on the deletion list; only its identity use is.
### PXE and GE-Enforce prefer it too, not just the collector
Phase 3 is not only "the collector reads a registry value". All three writers
and readers move together:
| Component | Change |
|---|---|
| PXE / imaging | writes `DeviceId` at enrollment instead of (or as well as) the file |
| GE-Enforce | resolves device identity registry-first, mirroring how it already resolves the machine number |
| collector | reads the registry ahead of the file |
**GE-Enforce already made this exact decision for the machine number, and wrote
down why.** `Install-FromManifest.ps1` resolves `TargetMachineNumbers` gating
from the eDNC registry BEFORE `C:\Enrollment\machine-number.txt`, because:
> the imaging-time `machine-number.txt` is written ONCE by `startnet.cmd` at the
> PXE menu and is NOT updated on reassignment, so it goes stale.
**`asset-id.txt` inherits that defect exactly.** It is written once at imaging
and nothing updates it when a device is swapped, moved between bays, or
replaced. The bay then reports an identity that was true on imaging day, and the
collector - which now trusts that value FIRST, ahead of every other source -
adopts the wrong device with no warning, because the value still resolves.
That makes phase 3 a correctness fix rather than a tidy-up. A file that only
imaging writes cannot be the top of a resolution order for something that
changes during a PC's life.
It also implies a reassignment path, the way `Set-MachineNumber` /
`Update-MachineNumber` exist for the bay number: something that rewrites
`DeviceId` when a device is swapped, and migrates any per-device state with it.
Without that, phase 3 moves the staleness from a file to a registry value rather
than curing it.
### Doing phase 3 early is cheaper than doing it late
If the registry read lands before imaging is changed, phase 2 can write the
REGISTRY for new bays and skip the file era entirely for them. The file then
exists only for bays staged in the window between phase 1 and phase 2, and the
eventual cleanup is deleting a code branch rather than sweeping the fleet.
The cost of adding the read now is one branch in the client and one row in the
docs. The cost of adding it after every bay has a file is a migration.
## 5. What this does not change
- `machinenumber` and its GE-Enforce gating. Untouched in all three phases.
- A part marker still files `partof` the operation from `machinenumber`. The
device identity says WHICH marker the PC controls, not which operation that
marker belongs to.
- Minting. It stays as the last resort for a bay that names nothing, because a
device nobody declared is still better recorded than not recorded. Anything
minted should be treated as a placeholder to be reconciled.
## 6. Before phase 2: find what is already duplicated
Phase 1 stops NEW duplicates. It does not merge the ones a PC swap has already
created. Before imaging starts writing files, run a read-only query for device
assets whose controlling PC no longer exists, or whose asset number carries a
hostname that is not a current PC. That list is the reconciliation backlog, and
it is much easier to read before the fleet starts reporting new identities over
the top of it.
## 7. Rollback
Each phase is independently reversible because nothing is removed until the
phase after it proves the replacement:
- Phase 1: stop sending `deviceid` and every previous fallback still runs.
- Phase 2: stop writing `asset-id.txt` and bays fall back to what they used
before.
- Phase 3: stop writing the registry value and bays fall back to the file.
The only irreversible step is deleting a fallback branch, which is deliberately
outside all three phases.

346
docs/proposals/printer-assignment.md Normal file
View File

@@ -0,0 +1,346 @@
<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-08 design making printers a property of the machine. Shipped in v0.11.3 and v0.12.0.
loadwith: none
-->
# Proposal: assign printers to a machine in ShopDB, let the PC install them
Status: SHIPPED AND RELEASED. Built out 2026-08-18/19; the client script is
built too. The feature itself released in v0.11.3, the observation half and
`seed-from-observed` adoption in v0.12.0.
Author: planning session 2026-08-18.
One thing this proposal did NOT anticipate, resolved 2026-08-24 in `adc7ef4`:
the computers collector carried a `_sync_printer_links` that wrote
`defaultprinter` / `connectedto` rows straight from a PC report, which is the
ASSIGNED-vs-OBSERVED split this proposal rests on, broken from the other end.
It is retired; `defaultprinter` and `printers` are now ignored (not rejected) on
the collector payload. See [COLLECTOR-INTEGRATION.md](../COLLECTOR-INTEGRATION.md).
## 1. What this is
Today a printer reaches a shop-floor PC because a person walks up to it, opens
the printer installer, finds the printer on a floor plan and clicks it. That is
fine for someone choosing a printer, and wrong for a bay whose printers are a
property of the bay.
This proposal makes the assignment data. The printers belong to the MACHINE, not
to the box currently driving it: tick the printers that belong on the machine,
mark one default, and the assignment reaches whichever PC controls that machine.
The PC converges on its next GE-Enforce cycle - installing what is missing and
setting the default - and keeps converging, so a reimaged bay comes back with
its printers and a bay that drifts is corrected.
The point of putting the assignment on the machine is that a reimaged PC needs
no backup and no restore step. The asset register is the source of truth, and a
replacement PC that inherits the `controls` edge inherits the printers with it.
The map installer stays, for the case it is actually good at: a person at an
unmanaged or office PC picking a printer that nobody assigned.
## 2. Why it is worth doing
- **The assignment becomes a record.** "Which printers does that bay have" is a
question ShopDB can answer, and today it cannot.
- **A reimage stops costing a visit.** The bay reinstalls its own printers, from
the machine's record, with nothing saved off the old PC.
- **Swapping the PC keeps the printers.** They were never the PC's.
- **Drift is corrected, not just detected.** A queue deleted by a user comes
back.
- **It removes the walk-up from the common case.** The installer's map remains
for the uncommon one.
## 3. What already exists
Most of the model is in place, which is why this is a small feature rather than
a project.
| piece | state |
|---|---|
| PC to printer link | `defaultprinter` asset relationship, seeded by `flask seed reference-data` |
| Propagation mechanism | `RelationshipTypePropagation` (ADR-001): "type X propagates through connections of type Y" |
| A working precedent | `resolve_asset_position` walks `partof` then `controls` to give a PC the machine's map position |
| Default lookup | `GET /api/printers/pc-default?machine=NNNN` |
| Host lookup | `GET /api/computers/by-hostname/<hostname>` |
| Printer model | `Printer.modelnumberid` - populated for 44 of 44 printers at the reference site |
| Driver record | `PrinterDriver` (name, `location` as SMB path or URL, optional `modelnumberid`) |
| Per-printer install path | `Printer.installpath` |
| Batch install | `GET /api/printers/install-batch?printerids=1,2,3` |
| Client transport | GE-Enforce manifest entries, `Type=PS1`, running as SYSTEM every cycle |
| Silent driver staging | Proven in `PrinterInstaller.iss`: trust the catalog's signing cert, then `pnputil /add-driver` |
## 4. The data model
### 4.1 One new relationship type
`usesprinter`, directional, source -> printer, meaning "this printer is
installed here". It is seeded next to `defaultprinter`, which already exists and
means "which of them is the default". Both are seed data, not a migration, which
is how every other relationship type shipped.
### 4.2 Two propagation rails, consumed at READ time
`usesprinter` propagates through `controls`, and so does `defaultprinter`. Both
are rows in `relationshiptypepropagations`, the same mechanism map positions
use. Inventing a second mechanism for this was the alternative, and it was
rejected.
The rails are inert at write time on purpose. The create-time fan-out
(`propagate_relationship`) skips directional through-types, and `controls` is
directional, so assigning a printer to a machine does not copy rows onto its PC.
The walk happens when something asks, which is what makes the next rule possible.
### 4.3 Resolution order for a PC
1. The PC's OWN active `usesprinter` / `defaultprinter` rows, if it has any.
2. Otherwise, one hop out along its `controls` edges to the machines it drives,
and those machines' rows instead, tagged as inherited.
Own beats inherited, whole set at a time: a PC with its own assignment is
overriding the bay, not adding to it. An office PC controls no machine and still
works, because step 1 is the normal case for it.
The override is a real trap and the UI has to say so. A tech who "fixes" a bay
by editing the PC has shadowed the machine's record, and the machine will keep
disagreeing until someone clears the PC's own rows.
### 4.4 One default, optional, and never dangling
The unique constraint is `(source, target, type)`, which happily accepts two
different defaults. So the rule is enforced in the API on write:
- Exactly one `defaultprinter` per asset. Setting a default replaces the
existing one.
- A default is OPTIONAL. A bay with three printers and no default is valid.
- The default must be one of the assigned printers. Unassigning the printer that
is currently default clears the default rather than leaving it dangling.
### 4.5 One column on `printerdrivers`
`drivername` - the driver's exact name as the INF declares it, e.g.
`HP Universal Printing PCL 6`. `Add-PrinterDriver` matches on that string, not
on `name`, which is ours to choose, and a mismatch is the usual failure.
Deriving it by parsing the INF on hundreds of bays is fragile; a human
confirming it once in ShopDB is not.
It is a plugin-chain migration (`printers0003drivername`), nullable, guarded so a
re-run is a no-op. `printerdrivers` was created by a core migration but its DDL
moved to the printers chain at the ADR-008 cutover.
The `installmethod` column (`pnputil` or `dpinst`) proposed earlier is NOT being
built. See section 8: if production confirms no Brother printers, everything is
`pnputil` and the column has no second value to hold.
## 5. What has to be built
### 5.1 A resolution helper in core
The read-time walk of section 4.3, beside `resolve_asset_position` and exported
on the `shopdb.api` contract surface (an additive minor bump). It has to live in
core because `RelationshipTypePropagation` is not on the contract surface, and a
plugin may not reach past it (ADR-002).
The through-type comes from the seeded rails, not from a hardcoded `'controls'`,
so a site that adds a rail gets the behaviour without a code change.
### 5.2 Two endpoints
```
GET /api/printers/for-host/<hostname>
```
The desired printer set for one PC, resolved per section 4.3, each entry with
what a client needs to install it: queue name, host or IP, port, driver name,
driver location, and which one is default.
Resolved by hostname, not machine number: the collector already upserts PCs by
hostname, and an office PC has no machine number. Matched case-insensitively -
`COMPUTERNAME` is uppercase and MySQL forgives that where SQLite does not.
An unknown host, a site without the computers plugin, or nothing assigned all
return an empty set. That is the client's designed no-op and it must stay
indistinguishable from "assigned nothing".
```
PUT /api/printers/assignments/for-asset/<asset_id>
```
The whole assignment for one asset - machine or PC - in one call:
`{printerassetids: [...], defaultprinterassetid: N|null}`. It reconciles rather
than inserting: rows that went away are soft-deleted, rows that come back
REACTIVATE the soft-deleted row (the unique constraint spans inactive rows, so a
blind insert is an integrity error on assign, unassign, re-assign), new rows are
created, and the single default is replaced.
The rules in 4.4 hold here or nowhere. Row-at-a-time writes through the generic
relationships path leave two-default windows and know nothing of the subset rule.
**Removing an assignment NEVER uninstalls anything.** Server-side the row simply
goes: no cascade, no side effects, nothing queued for the client to undo.
**Per-PC assignments must NOT go in the manifest.** Manifests are keyed by scope
and PC type and sync broadly; putting per-PC rows there would leak every bay's
configuration to every bay and grow without limit. One manifest entry runs one
script that asks the API what THIS host gets - the mirror image of
`Report-AssetToShopDB.ps1`.
### 5.3 UI on the PC form
A printer multi-select plus a default dropdown whose options are only the
currently selected printers, clearing itself when its printer is deselected.
Saved through the reconcile endpoint against the PC's asset.
The computers plugin does not depend on the printers plugin and must not start:
the section hides itself when the printers API is not there.
The machine-side picker is out of scope for now, which means the machine's
assignment is editable only through the generic relationships card. That is the
side the design says is primary, so it is the obvious next piece of UI.
### 5.4 One client script, in two contexts
`Set-ShopdbPrinters.ps1`, shipped in `plugins/printers/client/` beside the
contract it consumes, and run as a manifest entry with `DetectionMethod=Always`.
*As SYSTEM, every cycle:*
1. `GET /api/printers/for-host/$env:COMPUTERNAME`
2. For each assigned printer with no queue: trust the driver catalog's cert,
`pnputil /add-driver`, create the port, create the queue
3. Write the desired default to `HKLM:\SOFTWARE\GE\ShopDB DefaultPrinter`
4. Ensure the per-user task exists
*In the user's context, at logon and on a repeat:*
5. Read that value, compare with the current default, set it if it differs, and
clear "Let Windows manage my default printer" - otherwise Windows silently
overrides the choice the next time someone prints elsewhere
The default printer is per-user state, which is the only reason this needs two
contexts. Everything else is machine state and belongs to the cycle.
Converge, do not reinstall: when the state matches, the script does nothing.
Nothing here needs the manifest to know when a printer changes, because the
desired state is fetched, not declared.
## 6. Desired state and observed state are not the same thing
Everything above is DESIRED state: what SHOULD be installed on a PC. Nothing in
this feature knows what IS installed on it. The client reads the desired state,
converges toward it, and reports nothing back.
Reporting the observed state is the obvious next feature and is deliberately not
this one. If the collector sent the installed queues per host - name, port,
driver, which is default - then comparing that against the resolved assignment
gives drift detection for free: "this bay is missing the label printer", "this
PC has three queues nobody assigned", "the default is not the assigned one".
Keeping them apart is a rule, not a preference:
- **Observed data never writes `usesprinter` rows.** A register that learns from
what it finds mirrors the drift instead of correcting it, and the fault
becomes the desired state.
- **Observed data belongs on the computer record, timestamped**, like the rest
of the collector payload. It is an observation with an age, not a decision.
- **An empty answer from the API means "nothing assigned", not "nothing
installed"**, which is exactly why section 5.2 refuses to make removal
uninstall anything.
- The two can disagree indefinitely and that is a report to read, not an error
to resolve automatically.
## 7. Decisions to take before writing the client
1. **Never remove a queue by default.** A transient API failure would otherwise
strip printers fleet-wide. Deletion is an explicit opt-in, per PC.
2. **Enforced or set-once for the default?** Re-applying every cycle overrides a
user who chose their own default - correct for a locked bay, irritating on an
office PC. Set-once is Active Setup or RunOnce. Make it a per-PC-type flag
rather than one global answer.
3. **Failure is silent and safe**: unreachable API means change nothing, log,
exit 0 - the convention `Report-AssetToShopDB.ps1` already follows.
Open on the server side, and each one changes the response contract:
4. **How a universal driver resolves.** `PrinterDriver` links to a printer by
exact `modelnumberid`, and the target state is roughly four rows dominated by
HP UPD and Xerox GPD, which match no single model. Either the driver row
gains a vendor, or a `modelnumberid IS NULL` row matches on the printer's
resolved vendor name. Until this is settled, `for-host` returns no driver for
41 of 44 printers.
5. **What `port` means when it is null.** RAW 9100 is the obvious default; whose
job it is to apply it - server or script - has to be written down once.
6. **Who may read `for-host`.** `pc-default` and `install-list` are anonymous;
the collector and the GE-Enforce fetch use scoped service tokens. This one
discloses per-PC configuration keyed by hostname.
7. **Two inherited defaults.** A PC can legitimately control both bays of a
dual-bay machine, or several machines. The union of assigned printers is
easy; the default needs a deterministic rule, or none when it is ambiguous.
8. **Legacy `defaultprinter` rows have no `usesprinter` row**, because they
predate the type. Either an active default implies assignment on read
(zero-touch, preferred) or a one-time backfill writes the missing rows.
Otherwise existing defaults vanish from `for-host` while still showing in
`pc-default`.
9. **Deletions through the generic relationships card bypass the reconcile
endpoint** and can strand an active default pointing at an unassigned
printer. Either the resolver drops dangling defaults or the core delete path
learns the rule.
## 8. What the fleet data says, and the one prerequisite
The reference site's 44 printers are HP 26, Xerox 15, Zebra 1, HID 1, Epson 1.
- **HP and Xerox are 41 of 44, and both have true universal drivers** (HP UPD,
Xerox Global Print Driver). One driver record each serves every queue of that
make.
- **There are no Brother printers at all**, yet the installer carries 208 files
of per-model Brother MFC-J inkjet drivers. Those are host-based GDI devices
with no Printer-class INF, which is the only reason a second staging method
(DPInst) exists. If production confirms no Brother, that payload and that code
path can both go - and with them the `installmethod` column.
- **Zebra, HID and Epson are one printer each**, and the HP DesignJet plotter is
a fourth special case - a PostScript device the UPD does not cover.
**Prerequisite: populate `printerdrivers`.** It currently holds ONE row, and it
points at a per-model folder (`HP LaserJet Pro M607 Driver`) rather than the
universal driver - the opposite of how a UPD should be used. The table needs
roughly four rows: HP UPD, Xerox GPD, one per oddity, and DesignJet when its
payload is restored. Each needs `drivername` copied verbatim from its INF.
Nothing in this proposal works until a printer can resolve to a driver.
## 9. Deployment constraint that shapes the design
**The SFLD share is mounted only during GE-Enforce's cycle.** Any work touching
a share path must run as a manifest entry inside that cycle, never as its own
scheduled task. The failure is silent - the task reports 0 processed, 0
installed, 0 failed - and it has cost a session before.
This is why driver staging belongs in the cycle even though the per-user default
does not, and why "the assignment script schedules a task that installs drivers"
is the wrong shape.
## 10. Upgrading an existing site
Three steps, and the third is the one that gets forgotten:
1. `flask db upgrade` - no core migration in this feature, but a deploy runs it.
2. `flask plugin upgrade-all` - applies `drivername`. Skipping it is the classic
1054 unknown-column error.
3. `flask seed reference-data` - REQUIRED. Without it the `usesprinter` type and
both propagation rails do not exist, and `for-host` resolves nothing, quietly,
because empty is also the healthy answer.
Pair the upgrade with a smoke check against a known bay. A site with reversed
legacy `controls` rows (machine -> PC) should run
`flask relationships fix-controls-direction` first, or inheritance resolves for
none of those PCs.
## 11. What this does not change
- The printer installer keeps working, for walk-up and self-service.
- Nothing about how printers are modelled, mapped or reported.
- The collector contract. Section 6 would change it; this feature does not.
- Sites not running GE-Enforce: the same endpoint suits an Intune remediation or
a DSC `Script` resource, since it is a plain HTTP GET and a PowerShell script.