The reports table 500'd on every load: the backup lookup ordered with
ORDER BY lastseenat DESC NULLS LAST, which SQLite accepts and MySQL
rejects outright. Every test passed and the real database refused the
query - the tests run on SQLite, so the dialect difference was invisible.
Sorting in Python removes the dependency for nothing: the rows are one per
host per kind. The regression test pins which revision wins, including
that one never confirmed does not, and says why the sort lives here so it
does not get helpfully moved back into SQL.
The enforcement reports needed to name what a display IS, and reached
straight into shopdb.core.models.dashboarddefault to do it. Plugins may
only touch core through shopdb.api, and the contract test said so.
The role belongs on the surface rather than behind it: it lives in core,
no plugin owns it, and a plugin reporting on displays has no other way to
resolve it. Added there and the version bumped, which the docs test pins.
A report row carried a hostname and some counts. Everything an operator
wants next hangs off the asset behind that host, and none of it was there.
Host now links to its PC page. Beside it, what the PC IS or DRIVES: the
machine number for a bay or part-marker PC, the measuring tool for a tool
PC, the role for a display - each linking to that asset's own page. A map
pin appears only when the asset has coordinates and hovers to the same
floor-plan preview the asset pages use; an icon that opened an empty map
would be worse than no icon.
The backup column is LAST CONFIRMED, not last changed. Dedup means an
unchanged config writes no revision, so a machine stable for six months has
a six-month-old newest revision and is perfectly healthy - the one to worry
about is the machine whose backup stopped running. It reads lastseenat and
names the kind rather than assuming ntlars, since udc/file kinds on the
share are coming.
Resolution is bulk, never per row: this table shows the whole fleet, so a
lookup inside the loop would be one query per PC. It reads the collector's
existing 'controls' relationship rather than re-deriving which machine a PC
drives - that same resolution living in two places is what put a wrong
subtype filter on the map. Every plugin it touches is optional, so each
lookup is ImportError-guarded and a lean build renders the table without
those columns. A host ShopDB has no asset for still shows: the enforcement
result is real even when the inventory is behind.
The engine treats a minor-newer manifest as backward compatible and carries
on. That holds for additions which WIDEN behaviour - an old lib skips a Type
it does not know - and inverts for one that NARROWS it. _CmmVersion arrived
in lib 2.6 as a minor bump, so a PC on 2.5 does not recognise the field,
reads every gated entry as unfiltered, and installs every PC-DMIS version it
cannot detect, on every CMM, within one cycle.
The share runbook already says push the lib first. A runbook is not a
control, and the failure is silent, fleet-wide and about five minutes fast.
ShopDB already had the evidence and was not using it: every enforcement
report carries the enforcer version, and publish_scope had no gate at all.
It now compares the scope's manifest version against the versions PCs
actually report for that scope and refuses when any is behind, naming the
hosts. force=True for someone who knows why. A report with no or an
unreadable version counts as behind - that field arrived with the
summary-emitting engine, so its absence IS an old lib, and treating unknown
as safe is precisely how this fails open.
A scope nobody has reported for still publishes, or a fresh site could
never publish anything. Versions compare numerically, since as text '2.10'
sorts below '2.9'.
Also exposed as a preflight endpoint so the UI can warn before someone
clicks publish, and as a 409 with the offending hosts rather than a 500.
Wave one of the dashboard proposal, built as a vertical slice so the contract
is proven by something real before the other five cards follow.
GET /api/geenforce/dashboard/failures lists entries that FAILED on their PC's
most recent enforcement cycle. Per ENTRY, not per report: "three PCs failed" is
a number, while "Install OpenText failed with exit 1603 on WJSF1234" is
something a person can act on. Only current reports count, so a failure that
has since been fixed clears itself instead of needing dismissing. Hostnames
resolve to computerids in one query so each row links to the PC, and a PC
shopdb does not know still appears - the failure is real even when the
inventory is behind, and that is the bay most likely to be misconfigured.
The data has been there all along. The only way to see any of it was to open
one PC's report modal, one PC at a time.
The widget declaration is the contract change. The old shape named a Vue
component per widget, which cannot survive a lean build where a plugin's
component may never be staged into the bundle - which is exactly why five
plugins declare widgets pointing at components nobody ever wrote. This declares
data, a generic renderer, a permission and a link template, the way ADR-010
already does for asset panels. A test asserts no 'component' key, so the old
shape cannot creep back.
empty: hide is part of the contract, not decoration. A card reporting "nothing
wrong" daily teaches people to stop reading the page, which is how a fleet log
reached 3,234 lines with 17 that mattered.
Frontend rendering comes next; the endpoint and declaration stand alone and
change nothing that exists.
CLIENT IP / SPOOFABILITY. docs/geenforce-api-cutover.md claimed that removing the
IIS rewrite rule made the allowlist fail closed and that it does NOT become
spoofable. The opposite is true. IIS never sets X-Forwarded-For on its own; the
rule is the only thing that does. Remove it and IIS still forwards whatever
X-Forwarded-For the CALLER sent, waitress trusts it because it arrives from
127.0.0.1, and remote_addr becomes attacker-controlled - so a token-less caller
can fetch manifests from anywhere on the network. The document and the
_trusted_client_ip docstring now say so, waitress runs with
--trusted-proxy-count=1, and stage 5 checks the rule is actually live rather than
assuming it. The wizard question is rephrased to something an operator can verify
with their network team instead of guessing at.
NON-ASCII. The style gate only ever checked .py/.vue/.js/.ts, so documentation
accumulated em-dashes, arrows and box-drawing characters against this repo's own
convention - including in files added this week. Cleaned, and the gate now uses
INCLUDES_ALL so Markdown, JSON and YAML are covered.
PLUGIN DEFAULTS. The wizard pre-ticked measuringtools and printedparts, both of
which ship default_enabled=false, so every site taking the defaults installed and
enabled them against their manifests. Inno has no JSON parser so the list must be
hardcoded, but tests/test_installer_defaults.py now fails when it drifts.
UPGRADES. The payload copy merges, so a plugin dropped from a site's profile kept
its code forever - which defeats a lean build and leaves core's optional-import
guards succeeding for a plugin the site no longer has. Stale plugin directories
are now deregistered and removed before the copy.
add-plugin used 'plugin install', which for the five default_enabled=false
plugins left them installed but DISABLED - and printed a green success line
anyway. It now goes through apply-profile, and the success line is gated on the
exit code. Invoke-Flask records its own exit status, because $LASTEXITCODE keeps
a stale value when flask.exe is missing and no native command runs.
CHARSET. The utf8mb4 compiler hook lived inline in migrations/env.py, so it
covered the CORE chain only: plugin baselines inherited the server default, which
on a latin1 server means two charsets in one database. It is now
shopdb/utils/mysql_charset.py, imported by both, and preflight reports the
database's default charset.
BACKUP HONESTY. The dump was described as 'all of your asset data'. Uploaded
branding and floor-map images live in instance\ on disk, not in the database, so
a restore from the .sql alone comes back with no map. backup now archives
instance\ alongside it and says both are needed.
VERSIONING. AppVersion was hardcoded at 0.9.0 while the product, the frontend and
the newest tag said 0.7.0 - and 0.9.0 collides with a retired contract version.
Both builders now generate version.iss from shopdb/__init__.py.
Smaller: rollback overwrites .env before deleting it, as uninstall already did;
appcmd unlocks are scoped to this site's location rather than server-wide, with
the wide unlock as a fallback; DEVELOPMENT-SETUP says Python 3.14; the README
plugin list gains printedparts; prune-schema --force is documented as
first-provisioning-only; HTTPS is documented as not-the-default with the steps to
add it; the DBA SQL is on the wizard's database page; the features page says
unticking does not remove an installed feature; and the installer README states
that bundle-lock cannot vouch for the exe itself - that needs signing or an
out-of-band hash, neither of which is wired up.
- allowlist auth uses remote_addr, not the spoofable first X-Forwarded-For hop
(adds _trusted_client_ip + a regression test); rate-limit path unchanged
- client psm1: fix Set-StrictMode crashes reading absent keys in Get-ShopdbConfig
(token-less mode) and Resolve-ShopdbPayloads (no-payload entries); validate
the manifest response is JSON before overwriting the last-known-good cache
- runner: pass the engine its required -InstallerRoot/-LogFile; create the log
directory so enforce logging is not silently lost on a fresh kiosk
- display scope: dispatcher writes an all-users Startup shortcut instead of
Start-Process (SYSTEM cannot show a window in session 0), resolves the base
URL from HKLM, and adds an always-on power/no-lock entry; tests updated for
the 6-entry scope
Fleet PCs on a trusted (vaulted) network can now reach the GE-Enforce client
endpoints (manifest, payload, report) without a per-PC token: the auth path
accepts a valid geenforce.fetch/report token OR a source IP in the configured
allowlist (setting geenforce_allowed_cidrs). Fail-closed; an empty allowlist
means the token stays the only path, so existing deployments are unchanged.
Rationale: the client token lives in HKLM on every kiosk, so it does not
defend against a compromised kiosk anyway - network-perimeter trust is the
same practical strength with far less provisioning + no token-rotation churn
on a DB wipe. Documented in-UI that this is perimeter trust, not per-device
identity.
- _ip_allowlisted() (ipaddress, X-Forwarded-For-aware via _client_ip)
- /geenforce/config GET/PUT extended with allowedcidrs, server-validated +
normalized (bad CIDR -> 400)
- new GE-Enforce > Settings tab (GeEnforceSettings.vue) to edit the allowlist
in admin, no SQL
- 3 regression tests (allow by IP, reject outside list, empty = token required)
A geenforce.fetch token can now be pinned to specific manifest scopes so a
fleet-wide key (a display's, delivered by DSC or baked into the image) is not a
skeleton key for the whole content store. NULL binding = unrestricted, so every
existing service token keeps working.
Core:
- ApiToken.resourcescopes column + resourcescopelist property (migration
7d30_apitoken_resourcescopes; NULL = unrestricted).
- apitokens API create/update accept + persist an optional resourcescopes list
(a resource-name allowlist; not permission-catalog names).
- New contract helper authorized_service_token(scope): same check as
service_token_authorized but returns the ApiToken so a plugin can read its
binding. Contract 0.14.0 -> 0.15.0; also export SupportTeam.
GE-Enforce enforcement:
- get_manifest: a bound token requesting a scope outside its allowlist -> 403.
- get_payload: a bound token may only pull a blob its own scope(s) reference
(service.blob_referenced_by_scopes); anything else -> 404 (no hash probing).
- Decorator stashes the authorized token on g for the route to read.
Also fixes a pre-existing contract-surface violation: the printers/printedparts
alert helpers imported shopdb.core.models / shopdb.extensions directly; now
via shopdb.api (SupportTeam newly exported). Docs: GE-ENFORCE-DISPLAY.md
provisioning note, PLUGIN-HOOKS.md, CLAUDE.md.
9 new resource-binding tests; full suite 1131 passing.
Get GE-Enforce closer to running on credential-less Intune/Entra display PCs
that pull manifest + payloads over HTTPS instead of SMB.
Server (plugins/geenforce/api/routes.py):
- Rate-limit + 512MB served-size ceiling on GET /payload/<sha256> (reuses the
login limiter's cache pattern, config-overridable via GEENFORCE_PAYLOAD_*).
- New tests: payload hardening, manifestblobs model-vs-migration parity, and a
report-contract test locking the lowercase per-entry report keys.
PS client (plugins/geenforce/client/):
- Fix New-ShopdbReport per-entry key casing to lowercase (name/action/selfhealed/
exitcode/message) to match what the server reads; the engine emits PascalCase.
- Enforce TLS 1.2 in the network functions.
- Fetch + merge the fleet-wide common scope alongside the pctype scope
(pctype wins on conflict; -NoCommon opt-out).
- Normalize whatever the engine returns into a well-formed summary.
- Make the empty-cache fail-safe observable: event-log entry + report ping
instead of a silent exit 0.
Manifest (plugins/geenforce/seed_display_scope.py + docs/GE-ENFORCE-DISPLAY.md):
- Seed a gea-shopfloor-display scope: 4 Edge kiosk drift-heal registry entries
+ 1 data-driven dispatcher (Dashboard/Lobby/3DPrintRoom via display-type.txt).
Kiosk EXEs stay image-baked; the manifest heals policy/config drift only.
- Documents the common SMB-payload audit (entries needing http/inline before a
share-less display can inherit common).
Migration registry (shopdb/plugins/alembic_template.py + test):
- Register the pre-existing manifestblobs and the new printersupplyalerts tables
in PLUGIN_TABLE_OWNERS; update EXPECTED_HEAD_REVISION for geenforce (0002blobs),
printers (0002supplyalerts), and printedparts (0004txnrev) which had drifted.
Lets share-less (Intune/local-account) PCs pull installers the manifest
references over HTTPS instead of SMB - the general capability the whole fleet
migrates toward. New ManifestBlob registry (migration 0002) with bytes on disk
at instance/geenforce/payloads/<sha256> (deduped by content); service.store_blob
+ blob_path; client-facing GET /api/geenforce/payload/<sha256> (geenforce.fetch
token, ETag=hash, serves the blob store or an inline DB payload by hash). The
serializer now emits PayloadSource/PayloadSha256/PayloadRef for http/inline
entries only (smb entries round-trip unchanged - parity green). CLI
'flask geenforce add-payload <file>' registers a blob and prints its sha256.
This is the shopdb half (B1); the PS client/engine fetch is B2.
Fleet-install compliance for app-linked manifest entries: new service
compliance_for_scope + GET /geenforce/scopes/<id>/compliance count active
ComputerInstalledApp rows by curated appid (null-safe when computers plugin
absent). ManifestEditor gains a compliance panel. Curated appid stays shopdb
metadata and never enters manifest JSON, so behavioral parity is unaffected.
Inline manifest payloads: store_inline_payload (sha256, 1MB cap,
payloadsource='inline') + POST/GET /geenforce/entries/<id>/payload; editor
gains an upload control. Entry payload metadata surfaced in _entry_payload.
Frontend test harness: extract the editor's entry-form logic into pure
entryForm.js (buildEntryPayload, describeEntry, availableEntryTypes, scope
gates, ...) and cover it with 45 vitest tests. ManifestEditor now imports
those helpers, so the tests exercise the shipped code path (no duplication).
908 backend tests pass; vitest 45 pass; frontend build green; naming green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The honest replacement for the backed-out auto-seeding: instead of scraping
manifest labels into duplicate Application rows, an entry can be LINKED to an
existing catalog Application, cross-referencing what shopdb already tracks.
- Model: manifestentries.appid (nullable soft ref to core applications; in the
0001 baseline). It is shopdb METADATA, deliberately NOT a manifest field - it
never appears in the rendered manifest JSON, so enforcement + parity are
unaffected (test asserts it stays out of the preview manifest).
- API: _entry_payload returns appid + resolved appname; create/update accept an
optional appid (validated, unknown id ignored, null unlinks) via _apply_app_link;
GET /geenforce/applications is the picker source (id + name).
- Editor: a "Tracked application (optional)" select in the entry modal, and the
entry summary line notes the linked app ("...; tracked: eDNC").
- Foundation for a future desired-vs-observed compliance view.
889 tests green (incl. the link test + parity/migration unaffected); build +
naming green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A deep Fable review of the real manifest corpus (READ-ONLY reference) showed the
manifests are an ENFORCEMENT PROGRAM, not an application inventory, and that
auto-seeding the Applications catalog from entry Type + Name was wrong:
- The catalog ALREADY tracks these apps from the classic-shopdb migration, with
version histories (PC - DMIS, UDC x11 versions, eMX / eDNC, CLM, CSF, Oracle
Database, FormTracePak). Seeding from manifest labels created DUPLICATES under
different names (PC-DMIS 2016 vs PC - DMIS; eDNC (bundles NTLARS) vs eMX / eDNC;
OpenText HostExplorer ShopFloor vs CSF). It also misclassified config drops
(eMxInfo.txt) as apps and could never match a PC's reported ARP name.
So the seed-applications command + service are removed. Properly linking
manifest entries to the EXISTING catalog is a curated feature, not label-scraping.
Two REAL bugs the review found are fixed and kept:
- Report status (R4): every healthy cycle runs Always/no-detection scripts the
engine counts as "installed", so keying self-heal off installed>0 marked the
common scope selfhealed forever and made 'ok' unreachable. Status now derives
from explicit per-entry self-heal flags only; the stored flag no longer infers
from action=='installed'; the client kit doc reflects it.
- PCTypesStrict (R5): the runtime engine has no strict handling (preinstall
runner only). filters.matches_pctype now applies strict only when phase ==
'preinstall'; simulate + parity thread the scope phase through; the strict test
uses a preinstall scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consolidated fixes from a three-dimension adversarial review.
Data-loss (HIGH): the manifest entry editor stripped fields the form did not
expose, because PUT /entries is a full reset-then-apply. The form now captures
everything - InUseCheck processes as structured name/ExePath/timeout rows (not
just names), LogFile, and the three preinstall flags as checkboxes; the dead
payload-source control (never wired) is removed. New regression test proves an
edit preserves ExePath/timeout/LogFile/PreEnrollment/PCTypesStrict.
Update-entry crash (found by that regression test): replacing an entry's
one-to-one InUseCheck (unique entryid) collided with the old row mid-flush ->
IntegrityError -> 400. update_entry now frees the old InUseCheck (delete+flush)
before populate re-inserts it.
Export truncation (MEDIUM): export_scope_to_share used a plain truncating open,
so a failed/partial write left the live on-share manifest (every PC reads it)
empty. Now writes a temp file in the same dir and os.replace() atomically.
Report dedup case bug (MEDIUM, confirmed by scratch test): the iscurrent demote
matched hostname case-sensitively while the read path uses ilike, so a PC
reporting different casing left two iscurrent rows and double-counted. Demote is
now case-insensitive; regression test added.
Simulator fidelity (MEDIUM): PCTypesStrict was captured but ignored by the
filter mirror, so the simulator wrongly matched a collections-only strict entry
to a nocollections PC via the shared Standard alias group. matches_pctype now
honors PCTypesStrict (disables alias expansion); test added.
Hardening: removed the dead/unscoped GEENFORCE_API_KEY env fallback (never wired
into config; tokens are the only path); create/update entry return 400 on a
duplicate Name instead of 500; parity now asserts scope-level Version/Site; a
new test guards real-manifest field lengths against column limits (the DB-free
parity harness can't see truncation); error handling added to the previously
unguarded editor + reports API calls.
Full suite green; naming + frontend build green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rounds out the Milestone 1 admin experience: author + publish in shopdb, push
to the share by a button, and see what the fleet actually did.
Export to share:
- GET/PUT /api/geenforce/config stores the on-share export root (Setting
geenforce_share_root); POST /scopes/<id>/export-share writes the current
published JSON to <shareroot>/<scope>/manifest.json (preinstall.json for the
preinstall phase), backing up the existing file to _meta/history first.
geenforce.publish gated. The engine and PCs are untouched - this is the safe
Milestone 1 push whose rollback is restoring the history backup.
- Editor: a share-root config row + an "Export to Share" button per scope.
- 3 tests (config roundtrip, export writes the file, second export backs up).
Fleet-compliance UI (Settings > Enforcement Reports):
- New page over GET /reports + /reports/<id>: latest report per PC with
received (applied vs latest published version), status (ok/selfhealed/failed),
and install/skip/fail counts; row detail shows per-entry outcomes with
self-heal flags, exit codes, and messages. Hostname/PC-type filters.
- ADR-010 settings card + ADR-009 plugin-gated route.
Full suite 883 green; frontend build + naming green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full HTTP admin surface behind the manifest editor (geenforce.manage for edits,
geenforce.publish for shipping):
- Scopes: POST/GET/PUT/DELETE /scopes[/<id>] (create imaging PC types, edit the
ComputerType/MeasuringToolType mapping + metadata, delete).
- Entries: POST /scopes/<id>/entries, PUT/DELETE /entries/<id>. Payloads use the
manifest Applications[] shape; populate_entry (refactored out of build_entry)
updates an entry in place, resetting omitted fields and replacing children.
- Reorder: PUT /scopes/<id>/entries/reorder enforces the ordering contract
(body must list exactly the scope's entry ids).
- Simulate: GET /scopes/<id>/simulate?pctype&subtype&hostname&machinenumber&
cmmversion returns which entries apply and which filter excluded the rest,
reusing the engine-mirror filters. The "what would this PC get" tool.
- Publish lifecycle: POST /scopes/<id>/publish (records publishedby from JWT),
GET /scopes/<id>/versions, GET .../versions/<n> (frozen manifest),
POST /scopes/<id>/rollback.
Entry type validated against ENTRY_TYPES; 8 CRUD tests. JWT+permission gated so
the authz sweep covers them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PCs now report enforcement results back to shopdb, closing the desired-vs-observed
loop.
- POST /api/geenforce/report (geenforce.report service token): each cycle a PC
posts the published version it applied, install/skip/fail/filtered counts, and
per-entry outcomes.
- Two tables: manifestenforcementreports (latest-per-host + history: applied
version, enforcer version, counts, derived status ok/selfhealed/failed) and
manifestenforcementresults (per entry: action installed/skipped/failed,
selfhealed flag, exit code, warning/error message).
- RECEIVED: reports carry the applied version; the admin view derives
receivedlatest by comparing it to the scope's current published version, so
the fleet view shows which PCs picked up an update.
- SELF-HEAL: per-entry action captures drift correction (installed when it
should already be present) vs skipped (already good) vs failed, with messages.
- Admin reads: GET /reports (fleet compliance rollup) and GET /reports/<id>
(per-entry detail). New geenforce.report permission.
- Tables added to the (undeployed) 0001 baseline; geenforce.post_report is a
service-token endpoint so it is exempt from the JWT authz sweep, like the
collector blueprint. 8 reporting tests; full suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First execution phases of docs/proposals/ge-enforce-plugin.md. The GE-Enforce
manifest becomes shopdb data.
P0 scaffold: new geenforce plugin (api_prefix /api/geenforce, default_enabled
false, core_version >=0.7.0). Registered in PLUGIN_TABLE_OWNERS (ADR-008); its
0001 baseline really creates the tables.
P1a model: one wide manifestentries table + entrytype discriminator (not STI,
not JSON blob), manifestscopes (UNIQUE scopename+phase), the three multi-value
filter child tables, inusechecks + processes, immutable manifestpublishedversions
(frozen rendered JSON), manifestpayloads (inline, capped), pctypealiases
(mirror of the engine lib's alias graph). regvalue stored as its raw JSON
literal so DWord typing survives.
P1c importer + exporter: parse common + gea-shopfloor-* + preinstall.json into
draft rows and rebuild the JSON verbatim from rows in sortorder.
P1d parity harness (GATE A): filters.py mirrors the engine's four filter
functions + alias graph; parity.py proves import+export is behaviorally lossless
(field-identical + same-entries-fire across 18 machine-profile fixtures) WITHOUT
byte-diffing. Verified PASS against all 11 real reference manifests (64 entries)
and a synthetic site-neutral fixture covering every type/filter (the CI gate).
First slice (gea-shopfloor-cmm shape): service layer (import/publish/rollback/
export-to-share), CLI (parity, import-share, publish, export-share), and the
client endpoint GET /api/geenforce/manifest serving the current published
snapshot (never the draft) with ETag/304. Split permissions
geenforce.manage/publish/fetch. Tests prove import->publish->serve, draft edits
never change served bytes, publish+rollback, and auth (401 unauth/wrong-scope).
Contract 0.11.0: added service_token_authorized(scope) to shopdb.api so plugin
service endpoints authorize a scoped managed token without importing core token
internals. Documented in PLUGIN-HOOKS.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
