ShopDB is a product several sites adopt, developed at one site against that
site's live fleet, so every feature arrives carrying West Jefferson's
hostnames, networks and share paths. A scan of plugins/ and shopdb/ returns 19
site literals, of which about 8 are a shipped default or a hardcode rather than
a comment: a kiosk URL baked into a PowerShell payload a sister site cannot
override, a WJ UNC path as the backup share default, and device.geaerospace.net
shipped as a SETTING DEFAULT - inherited silently by anyone who adopts.
The mechanisms already existed; the rule did not, so each value landed wherever
was convenient. The rule: a setting with a NEUTRAL default first, a
site-namespaced directory second, seed data third. Blank must be a working
state. Fleet-wide vocabulary such as the gea-shopfloor-* imaging types is not
site data and is out of scope - it already has a per-site override.
Enforced by an addition to check-naming-and-style.sh, which CI already runs, in
REPORT-ONLY mode. Making a 19-item backlog a hard gate the day it lands blocks
every unrelated commit until someone clears it; SITE_LITERALS_ENFORCE=1 turns
it into a gate once the listed hardcodes are done. The ADR carries that backlog,
including that the display FQDN domain is defined three times in three files.
COLLECTOR-INTEGRATION.md told integrators that machinenumber maps to
Asset.assetnumber and that a real value updates it on an existing PC. That is
precisely the behaviour that returned 500 to every bay reporting a machine
ShopDB already knew, and it is the document another site reads to build its
reporter.
Rewritten to say what the field is for: it identifies the machine, the PC keeps
its hostname, and the number builds a controls link. Adds a section covering
the claim-versus-handover rule and its quiet window, the alert gate defaulting
off, and how a part-marker PC gets its own marker asset under the operation.
Three defects, all found on printedparts_label_prefix, all one root cause:
nothing in the framework knew that setting existed.
The parts kiosk runs logged out. An unauthenticated read of a setting is
limited to an allowlist, the key was not on it, so the kiosk got a 404 and
fell back to no prefix. An admin previewing the same page while logged in saw
the prefix, which is why it looked like it worked.
The same setting also looked like it would not save. The row did not exist on
a site that installed the plugin before the setting was added, so the first
save created it - under the placeholder category the settings API uses for
keys it does not recognise, where the plugin's settings page, which lists by
category, could no longer see it. The value was in the database the whole
time.
And the row was missing in the first place because seeding ran from
on_install / on_enable, which fire only on a state transition. Neither runs
again on an upgrade, so a setting added in a later plugin version never
reached a site that installed an earlier one. The comment claiming enable ran
every upgrade cycle was simply wrong.
A plugin now declares the settings it owns in get_settings_defaults(): key,
default, type, category, description, and whether a logged-out page may read
it. The framework seeds declared keys at install, at enable, and on every
flask plugin upgrade-all; files a first-time write under the declared
category; re-homes any row left in the placeholder category, value untouched;
and answers an anonymous read for keys marked public. Core carries no list of
any plugin's keys.
Contract 0.16.0 (additive optional hook). printedparts and printers move to
the hook and floor their core_version at 0.16.0. The dev database had two rows
in the misfiled state (printedparts_alert_email, employee_db_host); the first
repairs itself on the next upgrade pass.
The installer on the share was stamped 0.8.0 and contained sixteen commits that
were not in the v0.8.0 tag. A build that misreports its own version is exactly
what the version check in docs/RELEASING-WINDOWS.md exists to prevent, and it
would have left two sites unable to say what they were running.
Everything in 0.8.1 landed after v0.8.0 was tagged this morning, driven by two
sites entering real data for the first time: the blank-code 500, the two-slide
display that never rotated, modals discarding a part-filled form, filters
returning an empty page, model photos that could not be saved, and the shared
equipment catalog that lets a new site start with vendors, models and printer
supply part numbers already present.
CHANGELOG gains a 0.8.1 section, and the OpenAPI document follows __version__
rather than being restated.
The exe is attached to a release, never committed: most forges reject files over
100 MB inside a repository while allowing release assets far larger, and a
committed binary would sit in every future clone forever.
Also records that tags must be pushed explicitly. A plain push of the branch
does not carry them, so a release had nothing to hang off.
The Windows installer has never shipped under a version: v0.7.0 was tagged
before any of it existed, so every build handed out so far stamped a server with
0.7.0. Two servers running different builds were indistinguishable, and the
installer logged each upgrade as "same version already installed" rather than
recording what changed. This cuts the release that fixes that.
0.8.0 rather than a patch: the air-gapped installer is a new capability, and
pre-1.0 semantic versioning puts that in the minor slot (ADR-007).
CHANGELOG gains a 0.8.0 section covering the twelve defects a real Windows
Server 2019 install surfaced, the move from inferring "is this a re-run of my
install?" to recording it, and the operator documentation.
deploy/site-profile-universal.json is now in the repository. Released builds
were being produced from a profile in a temporary directory, so the next release
could not have been reproduced once that file was cleaned up.
docs/RELEASING-WINDOWS.md points at the committed profile and says why.
scripts/gen_openapi.py reads __version__ out of shopdb/__init__.py instead of
restating it. Its hardcoded copy had already drifted a release behind, which is
the same mistake that once shipped an installer stamped with the wrong version.
Waiting for a certificate from the organisation's own certificate authority
rather than buying one from a public CA. Every server this installer runs on is
centrally managed and already trusts that root, so an internally issued
Authenticode certificate removes the unknown-publisher warning exactly where it
matters; a public certificate would buy trust on machines this software never
reaches.
Notes the interim measure that costs nothing: publish the SHA-256 through a
channel separate from the installer, since a hash beside the file is only as
trustworthy as write access to that location.
Wording avoids naming internal infrastructure, since docs/ is published.
Two audiences, two documents. Both were only in people's heads.
UPDATES-WINDOWS.md is for whoever runs a server: updates arrive as one
self-contained exe, an update takes two to four minutes, the site is down for
that time, .env and data and any hand-edited web.config are kept, unticking a
feature never removes it, the database is backed up and verified first, and a
downgrade is refused because migrations only go forwards. It covers both kinds
of security release, application and third-party, and explains that the
CycloneDX inventory staged on every server is what answers a published
vulnerability question. It also says plainly that the exe is not signed and the
checksum is the integrity check to rely on today.
It answers one question the existing docs did not address at all: the effect on
other sites sharing the same IIS server. The application pool is isolated and
the configuration is scoped to its own path, so other sites keep their own
handlers. What IS shared gets named rather than glossed: installing the IIS
modules and writing server-level configuration recycles application pools
across the server, which can drop requests in flight and clears in-memory
session state, though IIS is never stopped and no iisreset is issued. The two
IIS modules and the single permitted rewrite server variable are machine-wide
and stay behind on uninstall, deliberately, since another site may have come to
depend on them. The bundled database option collides on port 3306 with an
existing MySQL.
RELEASING-WINDOWS.md is for whoever builds releases: the three kinds of change
and the commands for each, why bundle-lock.json must be committed, the two
dependency traps that have each already cost a release, which generated files
must never be hand-edited, and the pre-release checks. It records the two known
gaps honestly - no code signing, and compiling still requires Windows and a
person.
UPGRADE.md and OPERATE-WINDOWS.md link to the operator document.
Reported from the Server 2019 test: the Existing database page rendered a
truncated "Username:" label and no input boxes at all below it, so there was no
way to enter connection details.
CreateInputQueryPage lays its fields out BELOW the description text. The
description had grown to include a five-line CREATE DATABASE / CREATE USER /
GRANT block, added so a DBA could be handed the exact SQL. With five fields
underneath, the last two fell past the bottom of the page surface, where they
are simply not drawn - no error, no scrollbar, just missing controls.
The description is back to three lines. The SQL moves to
docs/INSTALL-WINDOWS.md, which is where someone would look for it anyway and
where it can be copied without being retyped from a wizard page.
Wizard page descriptions are a fixed budget: anything long enough to be worth
reading twice belongs in the guide, not on the page.
Adopting a site means getting its asset register in. The HTTP import API suits a
site with a source system and someone to script against it; a sister site with a
spreadsheet and no developer needs something else, and that is the common case.
FOREIGN KEYS TAKE NAMES. This is the whole design. A CSV row has to say where an
asset is, and the database stores locationid, an integer. Requiring the number
means importing locations, reading back the generated ids and pasting them into
the asset sheet - a workflow nobody finishes. Every foreign key here accepts
either a numeric id or the referenced row's name:
assetnumber,assettypeid,statusid,locationid
CMM-01,Measuring Tool,Active,Gage Lab
The column keeps its database name, per CONTRIBUTING.md; the value is whatever
the operator actually knows. Names resolve across files in one run, so
assets.csv can reference a location that only exists because locations.csv was
read moments earlier. A name that does not resolve is reported with its line,
column and value, not as a foreign key violation from three layers down.
Dry run is the default, and writes go into the transaction either way - the
rollback is what makes it a dry run. Skipping the writes instead made every
cross-file reference fail, which is the one thing a folder-wide check exists to
verify. Validation covers every row before anything is written, so a typo on
line 400 cannot leave 399 rows imported. Files are matched on a natural key, so
correcting a spreadsheet and re-running updates rather than duplicates.
TEMPLATES ARE GENERATED, NOT MAINTAINED. "flask csv templates" builds them from
the live schema, annotated with required/optional and which file each foreign
key refers to. The prompt for this was a hand-written template set that had
invented columns on seven of eleven tables and named a table that does not
exist, while looking entirely plausible - and described an import mechanism
(a Data Import page, a flask import-csv command) that had never existed. A test
fails the build if a generated template ever offers a column the schema lacks.
User accounts are deliberately not importable: passwords do not belong in a
spreadsheet in either direction.
Verified end to end against MySQL 5.6 - a folder dry run catching one bad
reference, the fix, the commit, and a re-run reporting updates rather than
inserts. 16 tests.
The bundled-database option could not actually be built. Stage 0 looks for
mysql\mysql-8.0.x-winx64.msi, and Oracle no longer publishes a standalone server
MSI for 8.0 - every 8.0.x returns 404. What remains for 8.0 is the MySQL
Installer bundle, which is an installer-manager: 'msiexec /i INSTALLDIR=' would
install THAT rather than a database, and stage 0 would then fail on a missing
mysqld.exe.
MySQL 8.0 also reached end of life in April 2026, so bundling it would have put
an unsupported database on every new site.
8.4 LTS still ships the standalone MSI (129MB, which is what the '125MB' note in
stage 0 was written against) and is supported into 2032. Defaults follow it:
install root MySQL Server 8.4, service MySQL84. The operator console still looks
for an 8.0 install path as a fallback, for sites already running one.
Also bundles mysqlclient\ - mysql.exe and mysqldump.exe with the two OpenSSL
DLLs they actually import, 20MB rather than the 51MB of debug and auth-plugin
libraries the archive ships. Stage 2 stages it onto the server, so a site whose
database is on ANOTHER host can still take the pre-upgrade backup that every
upgrade depends on. That was the gap the preflight had started warning about.
Bundle is now 221MB.
docs/ is excluded from the code bundle and its scrub gate, because it goes to
the GitHub wiki instead - via a generator that has no gate at all. So the one
part of the repository written in prose, by people, about internal
infrastructure, was the one part nothing checked.
What was reaching a public wiki: the internal git server's URL and hostname,
.gitea workflow paths, developer home directories in the GE-Enforce cutover
reference, and a dev database root password inside a copy-pasteable command in
the import guide.
All replaced with neutral equivalents. tests/test_docs_publishable.py is now the
gate, at the source, in CI - a wiki page cannot be un-published, so catching this
after the fact is not good enough.
PROJECT-REVIEW.md also referred to internal tooling by name throughout; those
references are generalised. It remains an internal candid assessment of this
project that is nonetheless published, which is worth a separate decision.
CLIENT IP / SPOOFABILITY. docs/geenforce-api-cutover.md claimed that removing the
IIS rewrite rule made the allowlist fail closed and that it does NOT become
spoofable. The opposite is true. IIS never sets X-Forwarded-For on its own; the
rule is the only thing that does. Remove it and IIS still forwards whatever
X-Forwarded-For the CALLER sent, waitress trusts it because it arrives from
127.0.0.1, and remote_addr becomes attacker-controlled - so a token-less caller
can fetch manifests from anywhere on the network. The document and the
_trusted_client_ip docstring now say so, waitress runs with
--trusted-proxy-count=1, and stage 5 checks the rule is actually live rather than
assuming it. The wizard question is rephrased to something an operator can verify
with their network team instead of guessing at.
NON-ASCII. The style gate only ever checked .py/.vue/.js/.ts, so documentation
accumulated em-dashes, arrows and box-drawing characters against this repo's own
convention - including in files added this week. Cleaned, and the gate now uses
INCLUDES_ALL so Markdown, JSON and YAML are covered.
PLUGIN DEFAULTS. The wizard pre-ticked measuringtools and printedparts, both of
which ship default_enabled=false, so every site taking the defaults installed and
enabled them against their manifests. Inno has no JSON parser so the list must be
hardcoded, but tests/test_installer_defaults.py now fails when it drifts.
UPGRADES. The payload copy merges, so a plugin dropped from a site's profile kept
its code forever - which defeats a lean build and leaves core's optional-import
guards succeeding for a plugin the site no longer has. Stale plugin directories
are now deregistered and removed before the copy.
add-plugin used 'plugin install', which for the five default_enabled=false
plugins left them installed but DISABLED - and printed a green success line
anyway. It now goes through apply-profile, and the success line is gated on the
exit code. Invoke-Flask records its own exit status, because $LASTEXITCODE keeps
a stale value when flask.exe is missing and no native command runs.
CHARSET. The utf8mb4 compiler hook lived inline in migrations/env.py, so it
covered the CORE chain only: plugin baselines inherited the server default, which
on a latin1 server means two charsets in one database. It is now
shopdb/utils/mysql_charset.py, imported by both, and preflight reports the
database's default charset.
BACKUP HONESTY. The dump was described as 'all of your asset data'. Uploaded
branding and floor-map images live in instance\ on disk, not in the database, so
a restore from the .sql alone comes back with no map. backup now archives
instance\ alongside it and says both are needed.
VERSIONING. AppVersion was hardcoded at 0.9.0 while the product, the frontend and
the newest tag said 0.7.0 - and 0.9.0 collides with a retired contract version.
Both builders now generate version.iss from shopdb/__init__.py.
Smaller: rollback overwrites .env before deleting it, as uninstall already did;
appcmd unlocks are scoped to this site's location rather than server-wide, with
the wide unlock as a fallback; DEVELOPMENT-SETUP says Python 3.14; the README
plugin list gains printedparts; prune-schema --force is documented as
first-provisioning-only; HTTPS is documented as not-the-default with the steps to
add it; the DBA SQL is on the wizard's database page; the features page says
unticking does not remove an installed feature; and the installer README states
that bundle-lock cannot vouch for the exe itself - that needs signing or an
out-of-band hash, neither of which is wired up.
Nine fixes from a review of the installer against its actual audience: DT leads
at sister sites who are not Windows, IIS or Python specialists and who will lean
on an AI assistant to get through it.
TRUTHFULNESS. The preflight was advisory - an operator read 'IIS is not
installed', pressed Next, answered five more pages and the install died partway
through with Python already on the box. The results page now blocks while
anything is failing, repaints on every run instead of latching after the first,
and offers 'Check again' so a fixed problem does not mean starting over. On
failure the wizard said 'Nothing was left running', which is false in every path
because the stages run with -OnFailure never: it now says the server is
part-configured, that re-running is safe, and how to remove it. The final page no
longer reads 'ShopDB-Flask is ready' after a failed install.
SECRETS. The generated MySQL root password went to Write-Host in a process the
wizard runs hidden - so nobody saw it - and stdout is forwarded into the setup
log operators are told to send to support, so it was permanently recorded for
everyone who did not need it. It now goes to an ACL'd file. Database dumps, which
contain every user password hash, landed in a ProgramData directory readable by
every user on the box; the directory is now locked at creation.
UPGRADES ON REMOTE-DATABASE SITES. mysqldump was looked for only under local
MySQL install paths, so a site whose database is on another host silently skipped
every pre-upgrade backup - after stage 2 had already stopped the pool and
replaced the tree. Find-MysqlTool now prefers a client shipped in the bundle,
stage 2 stages it onto the server, preflight reports when it is missing, and
mysqlclient\ is an optional locked payload.
UNINSTALL. A subpath install is an IIS Application, not a site; removing only the
site left the application pointing at a deleted directory, so the parent site -
at West Jefferson, the live classic ASP - served 503 on that path forever while
Add/Remove Programs reported success. Uninstall now reads MOUNT_PATH and removes
the application. The firewall rule was created as "$SiteName $SitePort" and
removed as the literal 'ShopDB-Flask 8090', which matches nothing.
DAY-2 TOOLING. Every shortcut now passes -AppRoot and -SitePort, and the console
forwards them through its own elevation and 32-bit relaunches instead of
discarding them - a non-default directory or port made it report a healthy site
as broken, from a shortcut the installer wrote. 'Open ShopDB-Flask' resolved to a
hardcoded localhost:8090 that was wrong for every subpath install; it now asks
the console, which reads the address the installer recorded, and no longer
demands administrator to open a browser.
SMOKE TEST. The parent-site port lookup filtered for an http binding and
defaulted to 80, so an https-only parent site failed a working install with a red
dialog.
DOCS AND /api/docs. The installer was invisible: nothing in docs/, README.md or
CLAUDE.md mentioned it, so a DT lead or their assistant landed on the manual IIS
runbook and hand-built the very server the installer then refuses to upgrade.
docs/INSTALL-WINDOWS.md and docs/OPERATE-WINDOWS.md are now the canonical route,
the two manual runbooks are bannered as reference-only, README and CLAUDE.md
route by target, and llms.txt tells an assistant which document to follow and to
ask for 'check -Json' before diagnosing. Both ship on the server, along with
openapi.json and llms.txt - without those the self-hosted /api/docs was broken on
every installed box, which matters most to the sites least able to debug it.
Stage 5 now checks it actually serves.
shopdb-admin.ps1 gains 'check -Json': one structured, secret-free block covering
version, publishing method, IIS state, HTTP reachability, database, Python
version, plugins and errors. That is the cheapest useful answer to 'the operator
will ask an LLM' - it works with no infrastructure, which a install-time MCP
server could not.
REQ-D: restore waitress and tzdata to requirements.in. They existed ONLY in the
generated requirements.txt (hand-added in bf9e60e), so the next
`uv pip compile` would have silently removed the WSGI server and the IANA
timezone database from every Windows install.
REQ-E: split production and development requirements. requirements.txt was
installing pytest, pytest-cov, pytest-flask, coverage, iniconfig and pluggy onto
production servers. Verified on a real Windows Server box before this change.
CI, scripts/test-external-plugin.sh and the dev docs now use requirements-dev.txt.
REQ-F: standardise on Python 3.14. The repo declared four different versions
(Dockerfile 3.12, DEPLOY-WINDOWS-IIS 3.12, INSTALL-WINDOWS-IIS 3.13, CI 3.13,
plus README, web.config and PLUGIN-EXTERNAL-REPO). 3.14 is in active bugfix
support until ~Apr 2027 and supported to Oct 2030; 3.13 entered security-only in
Apr 2026. All four compiled dependencies publish win_amd64 wheels for 3.14
(cryptography via an abi3 wheel), verified by building an offline wheelhouse and
installing it on Windows Server 2025.
REQ-G: state MySQL 8.0 as the standard for new installs; 5.7+/5.6 remain
supported on an existing server.
Lockfiles regenerated with uv pip compile. Production deps 44 -> 38.
Notification start/end times displayed and stored wrong by the tz offset
(a 2:34 PM entry showed 6:34 PM). Two stacked bugs: to_dict emitted stored
UTC as naive ISO (no offset) so the browser read it as local, and the form
filled the datetime-local input from toISOString() (UTC).
Fix and generalize to a configurable site timezone (multi-site):
- New setting site_timezone (default America/New_York), public, editable in
Settings > Site > Localization (common-zone dropdown).
- Backend tags datetimes UTC (_utc_iso); parse normalizes to naive UTC
(_parse_utc); daily-reset expiry uses the site zone (_next_site_time);
calendar allDay events key off the site-local day (_site_date).
- Shared frontend util datetime.js (Intl-based, DST-safe) converts between a
UTC instant and a site-zone wall clock. Notification form, list, and
calendar all render/enter in the site zone.
Generate docs/openapi.json (3.1, 362 operations) from the API inventory via
scripts/gen_openapi.py, and serve it with a self-hosted Redoc bundle at
/api/docs - no CDN, works on the air-gapped box. Also serve docs/llms.txt (a
concise LLM entrypoint) at /api/docs/llms.txt. New core 'docs' blueprint;
staticdocs/ excluded from the naming check (vendored minified JS).
Air-gapped sites cannot pip install / npm ci / docker pull, so a build-at-site
compose (build: .) fails and reports 'service api is not running'. Add a
build-once-ship-image path:
- scripts/build-offline-bundle.ps1: on a connected box, build shopdb-flask +
pull mysql:8.0, docker save both into one gzipped tarball with a sha256.
- docker-compose.airgap.yml: runs pre-loaded images (image:, never build:),
drops the ./plugins bind mount (which would mask the image's baked-in plugins
with an empty host dir and load zero plugins at an image-only site), and adds
a one-shot migrate service (db upgrade + plugin upgrade-all + seed) that api
waits on via service_completed_successfully, so 'up -d' brings a working site.
- docs/DEPLOY-AIRGAP.md: full runbook (build, transfer+verify, load+run, admin,
verify, upgrade, troubleshooting), incl the Zscaler in-build cert caveat.
- .env.example: IMAGE_TAG for the air-gap compose to pin the loaded image tag.
A geenforce.fetch token can now be pinned to specific manifest scopes so a
fleet-wide key (a display's, delivered by DSC or baked into the image) is not a
skeleton key for the whole content store. NULL binding = unrestricted, so every
existing service token keeps working.
Core:
- ApiToken.resourcescopes column + resourcescopelist property (migration
7d30_apitoken_resourcescopes; NULL = unrestricted).
- apitokens API create/update accept + persist an optional resourcescopes list
(a resource-name allowlist; not permission-catalog names).
- New contract helper authorized_service_token(scope): same check as
service_token_authorized but returns the ApiToken so a plugin can read its
binding. Contract 0.14.0 -> 0.15.0; also export SupportTeam.
GE-Enforce enforcement:
- get_manifest: a bound token requesting a scope outside its allowlist -> 403.
- get_payload: a bound token may only pull a blob its own scope(s) reference
(service.blob_referenced_by_scopes); anything else -> 404 (no hash probing).
- Decorator stashes the authorized token on g for the route to read.
Also fixes a pre-existing contract-surface violation: the printers/printedparts
alert helpers imported shopdb.core.models / shopdb.extensions directly; now
via shopdb.api (SupportTeam newly exported). Docs: GE-ENFORCE-DISPLAY.md
provisioning note, PLUGIN-HOOKS.md, CLAUDE.md.
9 new resource-binding tests; full suite 1131 passing.
Per decision: displays need none of the fleet-wide common scope's software, so
the gea-shopfloor-display scope carries everything it enforces and does not
inherit common. This avoids repackaging common's SMB-backed payloads for a
share-less display.
- Invert the client common-merge switch: -NoCommon (default-on) becomes
-IncludeCommon (default OFF). A scope now enforces alone unless opted in.
The capability stays for a future share-less non-display PC; displays omit it.
- Drop the common SMB-payload audit + inheritance sections from the display
seed comments and docs (GE-ENFORCE-DISPLAY.md); document self-sufficiency.
- GE-ENFORCE-CLIENT.md: common-scope inheritance is now opt-in.
Get GE-Enforce closer to running on credential-less Intune/Entra display PCs
that pull manifest + payloads over HTTPS instead of SMB.
Server (plugins/geenforce/api/routes.py):
- Rate-limit + 512MB served-size ceiling on GET /payload/<sha256> (reuses the
login limiter's cache pattern, config-overridable via GEENFORCE_PAYLOAD_*).
- New tests: payload hardening, manifestblobs model-vs-migration parity, and a
report-contract test locking the lowercase per-entry report keys.
PS client (plugins/geenforce/client/):
- Fix New-ShopdbReport per-entry key casing to lowercase (name/action/selfhealed/
exitcode/message) to match what the server reads; the engine emits PascalCase.
- Enforce TLS 1.2 in the network functions.
- Fetch + merge the fleet-wide common scope alongside the pctype scope
(pctype wins on conflict; -NoCommon opt-out).
- Normalize whatever the engine returns into a well-formed summary.
- Make the empty-cache fail-safe observable: event-log entry + report ping
instead of a silent exit 0.
Manifest (plugins/geenforce/seed_display_scope.py + docs/GE-ENFORCE-DISPLAY.md):
- Seed a gea-shopfloor-display scope: 4 Edge kiosk drift-heal registry entries
+ 1 data-driven dispatcher (Dashboard/Lobby/3DPrintRoom via display-type.txt).
Kiosk EXEs stay image-baked; the manifest heals policy/config drift only.
- Documents the common SMB-payload audit (entries needing http/inline before a
share-less display can inherit common).
Migration registry (shopdb/plugins/alembic_template.py + test):
- Register the pre-existing manifestblobs and the new printersupplyalerts tables
in PLUGIN_TABLE_OWNERS; update EXPECTED_HEAD_REVISION for geenforce (0002blobs),
printers (0002supplyalerts), and printedparts (0004txnrev) which had drifted.
send_webhook(title,text) posts alerts to an optional webhook (Teams Incoming
Webhook / Workflow, or generic JSON) via alert_webhook_url + alert_webhook_format
settings; send_alert fans out to it alongside email; exposed on shopdb.api
(0.13.0->0.14.0, PLUGIN-HOOKS synced); low-stock posts on its custom-recipient
path too. Also: recent-transactions table shows the consumed print-file revision.
A lean site still gets every plugin's tables from the shared core Alembic
baseline. prune-schema drops the tables of plugins not installed on this
site, leaving core + chosen-plugin tables, with no edit to any released
migration (the relocate-into-plugin-baselines alternative would mean
rewriting ~15 released core migrations for a cosmetic gain - see ADR-014).
- shopdb/plugins/cli.py: prune-schema command. Dry-run by default; --yes to
execute; refuses non-empty tables without --force. Drops by table name (no
plugin import) so it works on a lean image. MySQL: private AUTOCOMMIT engine
(db.engine's pooled connections sit idle-in-transaction in a CLI context and
would deadlock the DROP on a metadata lock). SQLite: db.engine, restoring the
prior foreign_keys pragma so the StaticPool connection is not left changed.
- tests/test_plugin_prune_schema.py: drop-only-not-installed, full no-op,
refuse-non-empty, force-drops-non-empty.
- docs/DEPLOY.md: lean provisioning step after upgrade-all.
- ADR-014 ACCEPTED; index updated.
Verified on MySQL: full install then prune = no-op (86 tables); lean install
(machines+printers) then prune drops the other 19 plugin tables; second run
no-op. Full suite 1077 passed.
Cross-plugin FK blocker ADR-013 cited is already resolved: the FKs into
machines were held only by dead legacy tables (machinerelationships,
printerdata, installedapps, communications.machineid) that existing
migrations 7a01/7c01 already drop. No live plugin table hard-FKs another
plugin. Schema-lean is unblocked.
Enabling change: create_plugin_tables now skips already-existing tables
(idempotent) so a plugin anchor can create its tables on a fresh lean
install and no-op on a database that has them from the pre-cutover
baseline. The load-bearing baseline lift is staged as ADR-014 Phase 2.
A re-review showed the previous "single import choke point" claim was wrong:
`plugins` is a normal importable package, so core request handlers that do
`from plugins.<name>.models import ...` never passed through the loader and ran
unverified - an attacker who dropped a file into plugins/<name>/ got arbitrary
in-process code execution on an ordinary HTTP request (and a planted .pyc ran
from cache). Gating load_plugin_class covered only plugin.py, one path of many.
Fix: importguard.py installs a sys.meta_path finder (under enforcement) that
intercepts EVERY plugins.<name>.* import, verifies the plugin's signed
provenance once, then verifies each module file against it and execs the exact
bytes it hashed - read once, compiled, exec'd, never a .pyc, never a re-opened
file. This closes the submodule bypass and the planted-bytecode read, and the
read-once exec closes the verify-vs-exec TOCTOU on the import path. The import
system, not one method, is the real choke point.
- init_app installs the guard when PLUGIN_REQUIRE_SIGNED, clears it otherwise.
- load_plugin_class now verifies plugin.py from a single read and execs that
buffer (finding #3 on that file); its submodule imports flow through the guard.
- docs: stamp-bundled must cover every plugin dir present (a disabled plugin's
module can be imported by core); recommend a read-only plugins/ owned by the
deploy user as defense in depth (closes the residual migrate-time race an
attacker with concurrent write could otherwise attempt).
Earlier review's fixes#3 (migrate code paths) and #4 (shelf content binding)
were confirmed sound and are unchanged. 7 import-guard tests (submodule verify,
tamper, unsigned refused, planted .pyc ignored, real import through the guard,
install/uninstall). 1061 pass, naming green.
Completes the marketplace security model. Verification stops being advisory:
a plugin only loads or migrates when its tree matches a trusted signature, and
plugins are pulled from a signed shelf with anti-rollback and revocation.
Enforcement (default OFF - existing deploys unchanged):
- verification.py PluginVerifier, shared by the loader (verify-at-load, before
plugin.py is imported) and the migration manager (verify-at-migrate, before
any DDL). Fail-closed: an unsigned/tampered/wrong-key plugin does not run.
- Gated by PLUGIN_REQUIRE_SIGNED. PLUGIN_DEV_TRUST_DIRS exempts named dirs but
only under DEBUG/TESTING; production ignores it.
- flask plugin stamp-bundled writes provenance into in-tree plugins so
verify-at-load applies to bundled plugins too (image build step).
- tier:core manifest guard: uninstall/disable refuse a core-tier plugin.
Shelf (shelf.py):
- Signed shelf-index.json (+ .sig): monotonic serial (a site refuses an older
index - anti-rollback), revoked list carried across builds, per-entry
version/tier/core_version for browse. Index is a browse layer only; adopt
reads security-bearing fields from the verified artifact.
- flask plugin shelf-build / shelf-list / adopt / audit. adopt verifies index +
artifact (signature + every file hash), unpacks to staging, re-verifies, then
atomically moves into place and installs+enables the closure. Refuses a
downgrade without --force-downgrade. Anti-rollback serial stored in
instance/shelf-state.json.
- config PLUGIN_SHELF_DIR; the app only reads the folder, never speaks a
network. .env.example + docs/PLUGIN-SIGNING.md document the flow.
22 tests: verifier policy (off / no-keys / signed / tampered / wrong-key /
dev-exempt), verify-at-load + verify-at-migrate integration, tier guard, index
sign/verify + tamper/wrong-key, serial state, revocation, version resolution,
verified atomic unpack + tamper refusal. Live-smoked keygen->pack->shelf-build
->list->adopt->audit + serial guard. 1050 pass, naming green.
Packaging + provenance for the plugin marketplace. No runtime behavior change
yet - verification is available on demand; enforcing it at plugin load/migrate
and pulling from a shelf are Phase 2.
- signing.py: ed25519 key pairs + provenance. Provenance is a sorted per-file
SHA-256 map plus metadata; the detached signature covers the exact
serialized provenance bytes, so verifying is re-hash files, re-serialize,
check signature. verify() accepts any of several trusted keys (rotation).
Uses cryptography (already a dependency).
- packaging.py: pack() builds a signed <name>-<version>.shopdbplugin (zip +
PROVENANCE.json + PROVENANCE.sig). verify_artifact()/verify_dir() re-hash
and check the signature, and flag a tampered file, an unexpected file, a
wrong/absent key - all fail closed.
- CLI: `flask plugin keygen` (publisher key pair), `flask plugin pack <name>
--key` (validates then signs), and `flask plugin validate` extended to a
signed artifact by path (--pubkey, else PLUGIN_TRUSTED_KEYS).
- config PLUGIN_TRUSTED_KEYS: os.pathsep-separated public-key PEM paths,
delivered with the site config, never read from the shelf. .env.example
documents it.
- docs/PLUGIN-SIGNING.md: curator flow (keygen offline, review, pack, publish,
pin keys, rotate).
The signature proves an artifact is exactly what a curator signed, not that the
code is safe - human review before signing is the control. 11 tests: sign/verify
round trip, wrong key, provenance excludes noise, serialize determinism, pack +
verify, tamper -> hash mismatch, extra file, no-key fail-closed, verify_dir.
1028 pass, naming green.
Design record for distributing optional plugins across GE sites: a small
mandatory core plus a catalog of optional plugins, packaged as signed
versioned artifacts, served from a transport-agnostic read-only shelf (a
SharePoint-synced or sneakernet folder - untrusted either way because every
decision-bearing byte is signed), verified at adopt AND at every load and
migrate. Lean per-site builds stage only chosen plugins into the backend
image and SPA bundle.
Status PROPOSED. Grounds the design in the real loader/contract/migration/
frontend code and records defects to fix along the way (upgrade-all
migrating unadopted folders, enable-without-install, reverse-dep checks
blind to unloaded plugins, missing cycle detection and dependency closure,
hardcoded plugin imports in the SPA router). Honest on scope: the frontend
re-org is the long pole (one core-router change plus per-plugin relocation),
not a mechanical move. Phased 0-5 with schema-lean and runtime-JS delivery
explicitly deferred.
New dev/eval seeder populates a small, broad dataset so a fresh site has
something on every screen: ~25 assets across machines, computers,
printers, network devices, and measuring tools, plus supporting
vendors/business-units/locations, six 3D-printed parts (two below their
low-stock threshold to exercise the alert), and a few relationships for
the map and relationship cards. Idempotent, keyed on a DEMO- assetnumber
prefix; skips the plugin sections that are not installed.
`flask seed demo-clear` removes exactly what it created: bulk-deletes the
DEMO- assets so the DB-level ON DELETE CASCADE drops each plugin subtype
row (per-object ORM delete would try to NULL the NOT NULL child assetid),
after clearing the demo relationships first. Leaves reference data,
settings, users, and any imported rows untouched.
Documented as an optional step in the dev setup guide.
Prod runs Python 3.13.7, not the originally planned 3.12. Align the
stack: CI both jobs 3.12->3.13, the IIS install runbook and the dev
setup guide to 3.13 (winget Python.Python.3.13). NOTE for whoever
maintains the offline kit: its wheels are still cp312 and must be
regenerated as cp313 before the next air-gapped deploy.
Add a winget block to the prerequisites (Git/Python/Node/VS Code/MySQL
or Docker) so a Windows dev provisions the whole toolchain from one
terminal, with a note that the LTS Node may be newer than CI's 20 and
it does not matter for this SPA (nvm-windows to pin if wanted).
From the Fable/Opus documentation audit (8 confirmed + verified
lab-drift the run's session limit had cut short):
- HIGH: the lab's kiosk _kiosk_find_item block showed the pre-stage-17
row-id resolver as current; replace with the shipped gagelabtag /
numeric-tail resolver, fix the stale 'resolved by row id' prose and
the 'stage-7 code is corrected' note.
- MED: the badge _external_lookup block used dict-only row access that
breaks on a tuple cursor; use the tuple-or-dict form shipped. Split
'&&' command chains (fail in PowerShell 5.1) in the lab.
- LOW/link: the Windows note's [DEVELOPMENT-SETUP] link dropped the .md
and 404'd in four docs; fix. Correct the stage-6a->16a comment and
the lab-stage tag range (..16 -> ..17).
- Leaks: drop /home/camp path from ADR-006, the internal gitea host
from PLUGINS.md.
- Windows: add an mklink junction note for the external-plugin symlink
dev loop.
- CI: prime root to mysql_native_password so pymysql connects to the
MySQL 8 service without the cryptography package (and its kit wheel).
The CI workflow comment named the internal server, and
PLUGIN-EXTERNAL-REPO carried internal gitea clone URLs (it becomes a
public wiki page). Point both at the GitHub home / a generic CI
mention so the publication scrub gate passes and the wiki does not
expose internal infrastructure.
GitHub had no CI, so naming/tests/build were unenforced on the public
mirror. Add .github/workflows/ci.yml mirroring the internal pipeline:
backend pytest, the naming gate, frontend vitest+build, and the
migrations-mysql job that proves a fresh flask db upgrade + every
plugin chain on utf8mb4 MySQL 8 is idempotent. Flip the dev-setup CI
note to reflect it. Add an identical Windows/VS Code convention note to
the four developer docs (venv\Scripts vs venv/bin, $env: vs export,
pointer to DEVELOPMENT-SETUP).
The naming check was documented as an auto-running pre-commit hook,
but .git/hooks is never cloned and no installer existed - a fresh
clone had nothing, and the real enforcement is CI. Say that plainly.
Ship .githooks/pre-commit (LF-pinned) so a dev who wants the local
check can opt in with 'git config core.hooksPath .githooks'; CI stays
the backstop that fails the build on a bad name.
PowerShell commands lead, bash equivalents in comments: venv
Activate.ps1 + execution-policy note, copy/$env:, a PowerShell
plugin-enable loop, and how the bash naming hook runs under Git Bash
(plus the pre-commit hook catching it automatically). The VS Code
Check task gets a Windows variant (venv\Scripts, bash for the .sh).
Pin shell scripts to LF in .gitattributes so a Windows checkout does
not CRLF-corrupt them into 'bad interpreter' failures.
New docs/DEVELOPMENT-SETUP.md: clone-to-first-change onboarding
(Docker fast path, manual venv+Node daily driver, VS Code, the dev
loop, first-change pointer at the plugin lab, troubleshooting). Ship
.vscode/ launch/tasks/extensions so F5 debugs the backend on 5001 and
a task runs both servers; personal settings.json stays ignored. Fix
the README manual path - it ran the backend on the default 5000, but
the frontend dev server proxies to 5001, so nothing loaded; also add
the plugin upgrade-all step and a VS Code pointer.
The gage lab assigns real WJRP asset numbers, so identity splits: the
internal itemcode stays auto-minted and a new optional unique
gagelabtag (migration 0003) carries the lab's number - settable on
create/edit, searchable, and resolved by the kiosk for scans and bare
keypad digits against the numeric tail of either identifier
(unique-match only). The print-files table becomes stacked revision
cards - filename with rev/current badges, one meta line, delete pinned
right - ending the horizontal scroll in that column.
The milestone workbook becomes a from-scratch guide with the actual
code inline for every core stage: models, the real migration baseline,
read routes and the list page, mutations and minting, the badge
resolver (final mode-aware form), the single-commit ledger invariant,
RBAC gating, both kiosk endpoints with the wedge-input and focus-guard
mechanics, the 1x0.5in label CSS, and the reconcile query. Field
extensions stay summarized against their tags. New section: how to
contribute a plugin through GitHub (branch, stage commits, the three
CI gates, PR expectations, review checklist, and how publication
folds PRs into release commits).
The resolver only read the self-hosted directory table, which is empty
at sites running the external HR directory - every kiosk badge fell to
the deny policy. It now branches on employee_directory_mode like the
usb plugin: selfhosted looks up DirectoryEmployee by SSO; external
queries the HR directory via employee_connection, resolving PayNo
badges by their real PayNo column and recovering the employee's SSO.
The plugin registry (instance/plugins.json), uploaded logos, floor
plans, item photos, and print files all write under instance/; with
the app pool at read-only, toggling a plugin in Settings surfaces as
an internal error and every upload fails. Grant Modify in step 7.3
and add the troubleshooting row.
Browsing the catalog (item list, detail, file listings) now requires
authentication plus the view permission, and the /printedparts pages
and the label print page require login. Still deliberately open: the
kiosk endpoints per the decision record, the image serve and file
download (img tags and anchor downloads cannot carry a JWT), and the
reports (product-wide jwt-optional convention). Grant
printedparts.view to the roles that should see the catalog.
