diff --git a/docs/proposals/ge-enforce-plugin.md b/docs/proposals/ge-enforce-plugin.md index 0a3ca09..b3a80d4 100644 --- a/docs/proposals/ge-enforce-plugin.md +++ b/docs/proposals/ge-enforce-plugin.md @@ -99,15 +99,26 @@ Preinstall-only extras (phase discriminator): store an explicit per-scope `sortorder`, not a set. 2. **PCTypes alias graph** is many-to-many old<->new names resolved by set intersection, with a `PCTypesStrict` escape hatch. Not a simple FK. -3. **Polymorphic entry by Type** - sparse column set per type. Model as - single-table with nullable columns, or a typed-payload child. STI is simpler. +3. **Polymorphic entry by Type** - sparse column set per type. DECISION: one + wide `manifestentries` table with an `entrytype` discriminator column and + nullable per-type columns. NOT SQLAlchemy STI subclasses, NOT a JSON blob. + Justification (section 4): the whole fleet is ~64 entries, so sparse columns + cost nothing; real columns get validated, indexed, field-diffed, joined + against collector data, and read in plain SQL by an IT tech - a JSON blob + hides all of that, and class-per-type STI is expert ceremony for no gain. A + `validate()` that switches on `entrytype` (mirroring the engine's own + `switch ($App.Type)`) is ~40 obvious lines. 4. **Two manifest phases** - runtime (self-heal, per logon) and preinstall (once at imaging) share the schema. One table with a `phase` discriminator. ## 4. Data model (new `geenforce` plugin) Per-plugin Alembic chain (ADR-008). Tables (lowercase concatenated per naming -convention): +convention). Sizing that shapes every decision here: the real fleet is 10 +runtime scopes = 43 entries, plus 1 preinstall manifest = 21 entries, so ~64 +rows total. That smallness is why this stays deliberately low-tech (one wide +table, JSON-document snapshots, no row-mirroring) - the design target is an +average site IT tech maintaining it, not a specialist. - `manifestscopes` - one row per imaging PC type / scope. - `scopeid` PK @@ -144,28 +155,49 @@ convention): SHA256 only when `detectionmethod = Hash`; an MSI with `Registry`/ `FileVersion` detection has no payload hash, so an HTTP fetch would otherwise run unverified bytes (see section 5). + - `regvalue` stores the RAW JSON literal (`1` vs `"1"`) and is emitted + verbatim on export. `RegValue` is untyped in the manifest schema and real + entries carry numbers; the engine string-coerces for `ValueMatches` but + `Set-ItemProperty -Type DWord` cares, so preserve the literal. - detection columns: `detectionmethod`, `detectionpath`, `detectionname`, `detectionvalue`, `detectionpattern` - gates: `cmmversion`, plus child tables for the multi-value filters - control: `logfile`, `waittimeoutsec`, `applymode`, `updatewindow` + (`applymode`/`updatewindow` are parsed-but-INERT in the engine today; the + UI must label them "not yet enforced" so a tech does not trust a dead gate) - preinstall flags: `preenrollment`, `killafterdetection`, `pctypesstrict` - `isactive` -- `manifestpublishedversions` + `manifestpublishedentries` - immutable - published snapshots. Editing `manifestentries` never affects the fleet; a - "publish" action freezes the current draft into a new numbered snapshot. The - client is ALWAYS served the latest published snapshot for a scope, never the - live draft, so a half-finished edit can never reach a PC. Rollback = mark an - older snapshot current (this is the post-cutover safety net that replaces the - "revert the dispatcher" rollback once the on-share JSON is retired). Mirrors - the current `_meta/history/-.json` backups, but authoritative. +- `manifestpublishedversions` - immutable published snapshots, SIMPLIFIED to + freeze the rendered JSON DOCUMENT in a single `manifestjson` column (drop the + row-mirrored `manifestpublishedentries` family the earlier draft proposed). + The only consumer of a snapshot is the client, and it consumes exactly that + document, so freezing the text makes immutability structural (no UPDATE path), + rollback a one-flag `iscurrent` flip, serving a single-row read, and version + diffing a plain text diff - all things average IT can debug; row-mirroring + would add ~6 shadow tables and a copy routine that can drift. Columns: + `publishedversionid`, `scopeid`, `versionnumber` (1,2,3 per scope), + `manifestjson` (MEDIUMTEXT, verbatim), `publishedat`, `publishedby`, + `iscurrent`, `notes`. Editing `manifestentries` never affects the fleet; + "publish" freezes a new snapshot; the client is ALWAYS served the current + snapshot, never the live draft. Rollback = flip `iscurrent` to an older + version (the post-cutover safety net once the on-share JSON is retired). + Mirrors today's `_meta/history/-.json` backups, but authoritative. - `manifestentrypctypes`, `manifestentryhostnames`, `manifestentrymachinenumbers` - - child rows for the ANDed multi-value filters (one value per row, wildcards - stored verbatim as patterns) + - child rows for the ANDed multi-value filters (one value + a `sortorder` per + row, wildcards stored verbatim as patterns) - `manifestinusechecks` + `manifestinusecheckprocesses` - - the nested InUseCheck object and its Processes[] child list + - the nested InUseCheck object and its Processes[] child list (leave + `gracefulclosetimeoutsec` nullable; do not bake the engine's default of 10 + into the row, emit it only when set) + +- `manifestpayloads` - inline payload bytes for `payloadsource = inline` + (`entryid`, `filename`, `contenttype`, `payloadbytes` LONGBLOB, `payloadsha256`, + `uploadedat`). App-enforced size cap ~1 MB; the upload UI rejects larger with + "use SMB for this" so nobody pastes an MSI into the database. Can ship empty + and unused until P6. - `pctypealiases` - a MIRROR of the old<->new name alias graph from `Install-FromManifest.ps1:463-475`, for server-side resolve/validate only. @@ -219,19 +251,37 @@ then the `payloadsha256` check is what actually guarantees payload integrity. ## 6. API surface (`/api/geenforce/...`) -Admin CRUD (gated by a new `geenforce.manage` permission via the plugin's -`get_permissions()` hook): +Two permissions via the plugin's `get_permissions()` hook (split so day-to-day +techs can edit but only a lead ships to the fleet): +- `geenforce.manage` - create/edit/reorder scopes, entries, drafts, payloads. +- `geenforce.publish` - publish, rollback, export-to-share (the fleet-affecting + actions). + +Draft editing (`geenforce.manage`): - `GET/POST /scopes`, `GET/PUT/DELETE /scopes/` - imaging PC types - `GET/POST /scopes//entries`, `PUT/DELETE /entries/` - manifest entries -- `PUT /scopes//entries/reorder` - the ordering contract, drag-to-reorder +- `PUT /scopes//entries/reorder` - the ordering contract; Move Up/Down in the + UI (plain buttons + visible `sortorder`), not a drag-and-drop dependency - `POST /entries//payload` - upload an inline/http payload (multipart), compute + store its `payloadsha256` (the integrity hash; NOT `detectionvalue`) +- `GET /scopes//preview` - the draft JSON a client WOULD receive on next + publish; `GET /scopes//published` shows the currently-served snapshot +- `GET /scopes//simulate?pctype=&subtype=&hostname=&machinenumber=&cmmversion=` + - the "what would this PC get" simulator: runs the entry list through the same + filter logic the engine uses and returns which entries apply and why the rest + are filtered out. Reuses the P1 parity harness's filter engine, so it is + nearly free, and it is the single most IT-empowering endpoint - it answers + "why did/didn't app X install on PC Y" without reading a PowerShell log. + +Publishing (`geenforce.publish`): - `POST /scopes//publish` - freeze the current draft into a new immutable `manifestpublishedversions` snapshot (this is what the fleet gets) - `POST /scopes//rollback/` - mark an older snapshot current -- `GET /scopes//preview` - the draft JSON a client WOULD receive on next - publish (review before publish); `GET /scopes//published` shows the - currently-served snapshot +- `POST /scopes//export-share` (or a `flask geenforce export-share` CLI) - + write the current published JSON to `//manifest.json` after + copying the existing file to `_meta/history/-.json`. This is a + first-class feature, not a footnote: it is the Milestone 1 product (author in + shopdb, engine untouched) and the permanent break-glass path. Client-facing (gated by a collector-style service token, `geenforce.fetch` scope, reusing the PAT + `X-API-Key` machinery already built for the collector): @@ -254,14 +304,23 @@ PC-type manager: - **Scopes list**: add/rename/delete imaging PC types; each still carries its `ComputerType` mapping (that column moves from a setting into `manifestscopes`). A `phase` toggle (runtime vs preinstall). Common scope flagged. -- **Scope detail / manifest editor**: an ordered, drag-reorderable list of - entries (the ordering contract made visible). Each entry is a typed form - - the visible fields switch on `entrytype` (MSI shows Installer+InstallArgs; - PS1 shows Script+Args; File shows Source+Destination; Registry shows the Reg* - quartet). Detection block with a method dropdown that reveals only the - relevant Detection* fields. Filter chips for PCTypes/hostnames/machine numbers. - InUseCheck sub-editor. Payload source selector (smb/http/inline) with upload - for the latter two. +- **Scope detail / manifest editor**: an ordered list of entries with Move + Up/Down buttons and a visible `sortorder` (the ordering contract made visible; + NOT drag-and-drop - a drag library is the kind of dependency that breaks + silently and average IT cannot fix; add drag later if wanted). Each entry is a + typed form - the visible fields switch on `entrytype` (MSI shows + Installer+InstallArgs; PS1 shows Script+Args; File shows Source+Destination; + Registry shows the Reg* quartet), one line of help per detection method. + Filter chips for PCTypes/hostnames/machine numbers. InUseCheck sub-editor. + Payload source selector (smb/http/inline) with upload for the latter two. + `applymode`/`updatewindow` sit behind an "Advanced (not yet enforced by the + engine)" disclosure. Ship the editor in three usable-alone increments: (a) + scope list + entry table, (b) the typed entry form, (c) publish + diff. That + keeps the biggest chunk of the build from ballooning. +- **Simulator ("what would this PC get")**: a small form (pctype, subtype, + hostname, machine number, CMM version) that calls `GET /scopes//simulate` + and lists which entries apply and why the rest are filtered. The single most + IT-empowering piece of the UI. - **Draft, preview, publish**: editing changes only the draft; "publish" freezes an immutable snapshot (see section 4) and is what the fleet then gets. Show the draft-vs-published diff before publishing. Rollback republishes a prior @@ -379,7 +438,15 @@ mechanisms must exist before stage 5, not just the dispatcher revert. - Replaces `plugins/computers/pctypemap.py` (the thin `pctypemap_` settings) - the pctype -> ComputerType mapping becomes the `computertypeid` - column on `manifestscopes`. Migrate those settings in, then retire them. + column on `manifestscopes`. Two-source transition window: `pctype_mapping()` + must keep reading the settings until the geenforce plugin is enabled, then + fall back geenforce-table-first / settings-second, and only retire + `seed_pctype_settings` + the settings at Milestone 1 close. Also reconcile the + scope inventory: `pctypemap.py` lists `gea-shopfloor-display` but the share has + no such manifest dir, and the share has a `main/` legacy dir the model ignores + - the importer creates scopes only from what it finds (plus empty scopes for + mapped-but-absent pctypes), and the P1 gate review reconciles the list with + the floor team. - Also folds in the metrology mapping now living in `pctypemap.py` (`METROLOGY_TOOL_MAP`). The collector already auto-creates a MeasuringTool asset and a directional PC->tool `controls` relationship when it sees a @@ -389,8 +456,9 @@ mechanisms must exist before stage 5, not just the dispatcher revert. collector agree on what device the scope implies. - Reuses the collector's token machinery (PAT + `X-API-Key` + scopes) for the client-facing endpoints. -- Reuses `get_permissions()` (contract 0.10.0) for `geenforce.manage` / - `geenforce.fetch`. +- Reuses `get_permissions()` (contract 0.10.0) for `geenforce.manage` (edit + drafts) / `geenforce.publish` (publish, rollback, export) / `geenforce.fetch` + (the client service token). - Pairs with the collector: desired-state (this plugin) + observed-state (collector) enable a fleet compliance view. @@ -407,3 +475,151 @@ and a dedicated `payloadsha256` for every HTTP/inline payload (section 5). If an when we proceed, this warrants a new ADR (ADR-012: GE-Enforce manifest ownership) capturing the desired-state model, the published-snapshot contract, the SMB/HTTP/inline payload + integrity model, and the fail-safe cache. + +## 13. Execution plan (build order, gates, milestones) + +Governing constraint: every step must be runnable and maintainable by average +site IT, not just the original developer. Where an earlier draft implied expert +machinery, this section simplifies it (and the model above already reflects +those simplifications: one wide table, JSON-document snapshots, no row-mirroring). + +### Phases and gates + +- **P0 - Scaffold (S, ~0.5-1 day).** `flask plugin new geenforce`, structure + copied from `plugins/measuringtools/`. Unlike bundled plugins' no-op migration + anchors, this NEW plugin's `0001_geenforce_baseline` actually creates the + tables and registers them in `PLUGIN_TABLE_OWNERS` (ADR-008). Deploy stays the + standard `flask db upgrade` + `flask plugin upgrade-all`. Manifest: + `api_prefix: /api/geenforce`, `default_enabled: false`, tight `core_version`. + +- **P1 - Model + importer + parity harness (M, ~1-1.5 wk). THE GATE.** Order + inside: tables -> `flask geenforce import-share` (reads common + every + `gea-shopfloor-*` + preinstall.json, skips `.bak`, idempotent) -> exporter + (rebuilds each scope's JSON from rows in `sortorder`) -> the parity harness + (below). **GATE A:** `flask geenforce parity` prints PASS for all scopes. If + it cannot pass, STOP the project. No API/UI/client work before Gate A. + +- **P2 - Publish/snapshot/rollback + admin API + export-to-share (M, ~1.5-2 wk).** + Publish freezes rendered JSON into `manifestpublishedversions`. CRUD per + section 6. Plus `flask geenforce export-share` + an "Export to share" button + that writes each scope's published JSON to the share after backing up the old + file to `_meta/history/`. Engine, dispatcher, share layout, payloads, PCs all + untouched. **GATE B = Milestone 1** (below). + +- **P3 - Frontend editor (L, ~2-3 wk; parallel with P4 after P2 API freezes).** + Expand `PCTypeMappingSettings.vue` per section 7, in three shippable + increments; Move Up/Down not drag; the simulator. + +- **P4 - Client fetch + shadow mode (M effort + soak time; needs P2, not P3).** + Week-1 spike: a ~20-line PS1 on ONE canary PC proves SYSTEM-context HTTP auth + + TLS trust before any real client change. Then `GE-Enforce.ps1` fetches JSON to + a local cache and hands the file to `Install-FromManifest.ps1` unchanged; + shadow mode installs from the share but logs any diff vs shopdb; ETag + + last-known-good cache from day one. **GATE C:** zero shadow diffs across one PC + of every pctype for >= 20 cycles. + +- **P5 - Read cutover (S effort, M calendar).** Per-scope flip, canary first via + `TargetHostnames`. Payloads stay `smb`. Rollback = dispatcher revert; share + export continues as break-glass. **GATE D:** all scopes cut over. + +- **P6 - Payload migration (S per entry, optional forever).** Small configs to + `inline` (verified by `payloadsha256`); MSIs stay on SMB. Each entry + independently revertible (flip `payloadsource`). + +Hard ordering: P0 -> P1 -> P2 -> rest. **Snapshots (P2) MUST precede any client +pointing at shopdb (P4).** P3 and P4 parallelize. Preinstall stays share-sourced +through at least Milestone 1 (no token pre-enrollment; export writes +`preinstall.json` too, so it is authored-in-shopdb for free with no client risk). + +### The P1 parity harness (concrete, IT-re-runnable) + +`plugins/geenforce/parity.py` + a CLI, also wrapped as a CI test. Two checks per +scope, output one readable line per scope (`entries N/N identical profiles M/M +same-fire PASS`), exit 0/1, prints the first differing entry/field on fail: + +1. **Lossless field check (order-preserving).** Canonicalize each entry to + exactly the fields the engine reads (Name, Type, the payload fields, all + Detection*, the filter arrays, `_CmmVersion`, InUseCheck, preinstall flags); + exclude `_comment` and key order (documentation, not behavior). Compare the + ordered lists position by position. +2. **Same-entries-fire-in-same-order.** Re-implement in ~120 lines of Python the + engine's four filter functions exactly as written in `Install-FromManifest.ps1` + (`Test-PCTypeMatches` incl. the alias groups at lines 463-475, `"*"`, and + `-`; `Test-HostnameMatches` exact + `-like`; + `Test-MachineNumberMatches`; `Test-CmmVersionMatches`). For each machine- + profile fixture, run BOTH manifests through it and assert the identical + ordered list of entry names that pass all filters. Detection itself is not + executed - check 1 already proved detection fields identical, so identical + inputs to detection are guaranteed. This pair proves losslessness without + byte-diffing. + +Fixtures (`plugins/geenforce/parityfixtures.json`, ~16-18 profiles): one per +pctype; CMM version variants `2016/2019/2026`/empty; collections machine-number +variants (a credentialed bay, an MTConnect bay, neither); legacy-alias profiles +(`Standard`+`Machine`, `CMM`) to exercise the alias graph both ways; a `WJS-*` +hostname-wildcard profile; preinstall profiles including one that hits +`PCTypesStrict`. Watch-items the harness must handle: empty `Applications: []` +scopes (4 exist), entries with NO `DetectionMethod` (fire every run), and the +`regvalue` literal typing. + +### First slice: one vertical through `gea-shopfloor-cmm` + +Only 4 entries but hits every hard part - MSI type, Registry detection with and +without a pinned value, nested InUseCheck with Processes[], and the `_CmmVersion` +gate. Tables: scopes, entries, entrypctypes, inusechecks + processes, +publishedversions, pctypealiases. `flask geenforce import-share --scope +gea-shopfloor-cmm`; `flask geenforce publish gea-shopfloor-cmm`; one endpoint +`GET /api/geenforce/manifest?pctype=gea-shopfloor-cmm` serving the published +snapshot (fat-client, ETag, collector-style `X-API-Key`/PAT auth reusing +`shopdb/core/api/collector.py`). **Done =** parity PASS for cmm; the endpoint's +JSON fed to `Install-FromManifest.ps1` on a bench CMM PC logs `4 skipped` +identically to the share manifest; editing a draft does NOT change the served +bytes but publishing does, and rollback restores the prior published bytes; +unauth = 401, wrong-scope = 401. + +### Milestone 1 (the recommended first stop) + +End of P2 plus the publish/scope-list slice of P3: **manifests are authored and +published in shopdb, exported to the share by a button, and the engine, +dispatcher, share layout, payloads, and every PC are completely unchanged.** +That delivers the real pain relief - validated editing instead of hand-edited +JSON, version history, one-click rollback (republish + re-export), desired-state +data sitting next to collector data - at ZERO client risk, with a rollback any +IT tech already knows (restore the `_meta/history` backup file). Natural point to +write ADR-012 with real experience behind it. P4/P5 (HTTP fetch, cutover) are a +separately green-lit second milestone. + +### Ranked risks / fail-fast + +1. **Generated-JSON vs engine drift (fleet-wide mis-install).** Parity harness + first; CI re-proves parity against checked-in real manifests on every + exporter change; pin lib >= 2.6. +2. **Serving a half-finished draft.** Structural: client reads only + `iscurrent` snapshots; test asserts a draft edit leaves served bytes + unchanged. Must exist before P4. +3. **Availability coupling.** Last-known-good local cache in the first client + prototype; shadow test blocks shopdb and confirms enforce-from-cache + WARN. +4. **SYSTEM HTTP auth + TLS trust.** The ~20-line canary spike in P4 week 1, + before the real client change. Hours of cost; if it fails, Milestone 1 still + delivers full value. +5. **Alias-graph drift.** Seed pins a lib version; harness legacy-name profiles + fail loudly on divergence; new-lib runbook includes "update the alias seed". +6. **Preinstall has no pre-enrollment token.** Keep share-sourced through + Milestone 1/2; decide later. +7. **Editor scope creep.** Three shippable increments; buttons over drag; reuse + JSON preview. + +### IT operability (day-to-day runbook, proving the design is manageable) + +All in Settings > Imaging PC Types. No PowerShell, no SQL, no share edits. +- **Add an app to a PC type:** open the PC type, Add Entry, pick Type (fields + adapt), fill installer + detection + targeting, Move Up/Down to order, Preview + (+ simulator), Publish with a note. PCs pick it up next 5-min cycle. +- **Bump a version:** drop the new MSI in the scope's `apps/` on the share, + update the entry's Installer + Detection value, Preview, Publish. +- **Roll back a bad publish:** History -> pick last-good version -> Roll Back + (during Milestone 1 also click Export to Share). +- **Canary a risky change:** add the one test PC under Target Hostnames, Publish; + when happy, remove the filter and Publish again. +- **Check "did PC Y get app X":** the simulator with that PC's type/machine + number/CMM version shows exactly which entries apply and why others are filtered.