Enabling a plugin could not make its routes live. Flask refuses
register_blueprint once the app has served a request, and each gunicorn worker
holds its own app object, so registering in the worker that served the toggle
would leave the others answering 404 for the same URL.
Worse, the window before the restart was a window in which workers disagreed
about reality. Four defects, all of which answered 200 while going wrong:
- Enabling a not-yet-installed plugin ALWAYS 500'd on a running app:
install_plugin ran the migrations, updated the registry, and only then called
register_blueprint and hit Flask's assertion. The toggle half-applied and
reported failure.
- Enabling one plugin could hang the entire site. @jwt_required does SELECT
users, so the request's transaction sits open holding a shared metadata lock;
the plugin's migration runs on its own Alembic connection and needs an
EXCLUSIVE lock on users for a foreign key. Each waits on the other, and MySQL
grants metadata locks in request order, so every later login queued behind the
pending exclusive. Observed twice as a total outage from one plugin toggle.
- Each worker rewrote the WHOLE registry file from a snapshot taken at boot, so
concurrent enables erased each other: seven enables persisted three, with
HTTP 200 for all seven.
- Dependency checks read that same stale snapshot, refusing a plugin whose
dependency another worker had just enabled.
Routes still require the process to restart; what changes is who initiates it.
Each worker fingerprints instance/plugins.json at boot and exits gracefully when
it changes, and the supervisor replaces it. That is the primitive gunicorn's own
--reload uses. Recycles are staggered per worker so the pool rolls rather than
dropping at once, and the exit is scoped to gunicorn: on Windows sys.exit() in a
daemon thread ends the thread rather than the process, so IIS keeps the manual
restart until that path is proven on the Windows VM.
The registry now merges under an exclusive lock and replaces atomically, saves
only the entry being changed, and re-reads before any decision is made from it.
ADR-020 records the decision; ADR-013 is amended where it said the UI must tell
the operator to restart. The restart requirement and the honesty requirement
both stand - only the manual step is removed, and only where it is proven.
(cherry picked from commit 9d6342b3ab)
The device token assumes a handful of fixed screens an administrator
provisions once. That is not how these kiosks are used: most people open the
page on their own PC, and there is no login, so per-browser provisioning would
mean touching every PC on the floor and losing it again on the next reimage or
cleared profile. A screen in that state did not look refused either - the USB
kiosk rendered a working page whose every call answered 404.
Both kiosks now admit a caller on either ground, the same shape the GE-Enforce
client endpoints already use: the per-device token, or a source address inside
an allowlist. Wall screens keep the token and are not confined to the
allowlist; everyone else is admitted by where they are on the network. Still
gated on the site's enable flag, so nothing opens by itself, and both
allowlists ship empty - token-only until a site enters its ranges.
- shopdb.api gains ip_in_cidrs and request_ip_allowlisted (contract 0.35.0).
On the surface because both plugins need the identical gate, and a CIDR
parser copied into two of them is where a fail-open creeps in later. They
fail closed on an empty setting, an unparseable caller address and a
malformed entry, and they read remote_addr rather than raw X-Forwarded-For -
so the IIS rewrite rule is what makes them meaningful, which the module says
at length.
- Separate settings per kiosk, usb_kiosk_allowed_cidrs and
printedparts_kiosk_allowed_cidrs. One must not grant the other, for the same
reason the tokens are separate.
- Both settings pages get a picker listing the network plugin's subnets with
their VLANs. It stores resolved CIDR strings, not subnet ids: the setting has
to work at a site that does not run that plugin and must not break when an
inventory row is deleted. Entries matching no subnet are shown rather than
hidden, since a stale one left by an inventory edit looks exactly like a
hand-typed range.
- USBKiosk says it was not admitted instead of pretending to work, which
PartsKiosk already did.
- The USB settings page keyed text-vs-boolean off the token alone. Adding a
second text setting to the boolean path would have saved it as the word
'true' - the bug that replaced the kiosk token and 404'd every provisioned
screen. Both load and save now key off a list, with a spec on it.
The trade, stated plainly in the admin UI: anyone on an allowed network can
check a drive in or out against a badge number they guess, and the record
carries that person's name. Badges are attribution, not authorization. Keeping
the ranges narrow is what bounds it; login is the alternative and was ruled out
because these users do not sign in.
Findings F2, F4, F5, F6, F7, F8, F9, F10, F11 and F13 from
wj-architectural-review.md. F1/F1a (the non-sargable LOWER pattern) and F3
(the warranty sync holding a pooled connection) are being handled separately
and are untouched here.
- A collector check-in that raced another for the same PC lost everything:
the unique index on the application fingerprint raised, nothing caught it,
and the whole check-in rolled back, discarding state, events and hardware,
answering the agent with form copy meant for a person. The insert now runs
in a savepoint, reconciles what the other transaction already wrote, one row
per savepoint so a second collision skips a row rather than the batch, and
the check-in commits with a warning. This is the shape the delete side of
the same function already had.
- Enrollment is deliberately unauthenticated, so it now carries a per-address
rate limit and a cap on pending requests, and the cap is a setting an admin
can see and change. A flood can no longer bury the approval queue or grow
the table without bound.
- The fleet alert poll no longer formats debug strings on every request.
- BusinessUnit is on the contract surface (0.34.0), so the endpoints plugin
stops reaching into a core table with raw SQL.
- A test walks every asset type descriptor and proves the endpoint it names
really refuses a caller without the declared permission.
- The plugin settings routes now need a permission; they were telling anyone
which modules a site runs and where its components live.
- The connection pool is sized explicitly with a short timeout, and the
arithmetic against the server's connection ceiling is written next to the
worker count instead of being left to defaults nobody chose.
- Fleet alerts carry both the correct lowercase keys and the old camelCase
pair. Trays older than 1.7.0 match keys case-sensitively and would have gone
quiet on every PC not yet upgraded; the old pair goes when they are.
- The dev server proxy points at localhost again, with environment variables
for the remote-box workflow, and one developer's address is out of the
server certificate page.
Per-PC credentials (replaces the 09-14 client-certificate attempt):
- POST /api/endpoints/enroll issues a credential to a PC that matches exactly
one computer by hostname and BIOS serial and has none yet. Anything else
(unknown or duplicate hostname, serial mismatch, already enrolled) waits as
a request an admin approves or rejects. The agent's persisted nonce binds an
approved request to the PC that asked.
- One auth helper serves every agent route. With X-Endpoint-Credential the
asset comes from the credential and a request naming another PC gets 403
ASSET_MISMATCH. The shared key still works for PCs that have not enrolled,
is refused (401 SHARED_KEY_REFUSED) for any PC that has, and can be switched
off. A revoked credential gets 401 CREDENTIAL_INVALID so the agent
re-enrolls.
- The core collector accepts the credential for the computers, printers and
endpoints collectors through two new plugin hooks exposed on shopdb.api
(contract 0.33.0), and refuses the shared key for enrolled PCs.
- Admin API: enrollment list, approve, reject, revoke (un-enroll), settings.
Migration API: per-PC readiness with blockers, and upgrade waves that
package the agent once and queue the existing agent upgrade per PC.
- Migration 0023 creates endpointcredentials, endpointenrollrequests and
endpointmigrationwaves, drops the certificate columns from assets where
test databases have them, and renames app_fingerprint to appfingerprint.
0021 and 0022 keep their ids but no longer alter the core assets table.
- Removed: client_cert.py, certificate renewal, the unauthenticated
/test/client-cert route, nginx client certificate forwarding and the agent
certificate script. The client-cert header could be forged on IIS, the key
was readable by users, and the shared key remained a fallback.
- GET /settings/server-cert returns the SPKI SHA-256 pin for GE-Enforce.
Review fixes:
- report_command_result used g without importing it; every result was a 500.
- collect_log refuses setup answer files, appsettings, agent directories and
key files at enqueue.
- Fleet alerts: alerttimeout and alertfleet agree with the migration, the
scope is validated and a malformed scope no longer reaches the whole fleet,
and the ack race no longer raises NameError.
- A refused break-glass override leaves no hold and no audit row.
- Agent upgrade checks locks and permissions before repackaging, and writes
the package atomically.
- Server certificate upload validates the pair before replacing files;
generation no longer builds an openssl config from user input; audit rows
are actually written.
- Health evaluation runs in a savepoint so a caught error cannot lose the
check-in.
- Required-app policies are found by appid, survive renames, and deleting an
app deactivates policies that still name it. The UI and API now say such a
policy only takes effect inside a business policy.
- Backfill handles duplicate fingerprints and rolls back a failed batch.
- Plugin settings routes skip a plugin whose hook raises.
- The setup token is no longer printed to the log.
- The naming gate checks top-level files again and handles odd file names.
- Removed root scratch scripts, one of which carried a token.
- Rewrote the fleet alert scheduling and app delta tests on real fixtures.
Exports caller_is_role on the plugin authorization surface. Additive name, so a minor bump. Scoped personal tokens are always refused by it, matching require_role, which stops a plugin from accidentally letting a scoped token inherit role authority from its owner.
Registers the tables added by the endpoint lock and policy hold work (endpointlocks, policyactionholds) plus the endpoint health tables in the alembic template, so plugin migrations see them and the ownership guard does not report them as unclaimed.
wilm's backup implementation doesn't expose _backups_configured or
start_restore_finalizer - those were wj-specific. Since we took wilm's
complete backup system, remove the initialization code.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
The board was one card per source in a fixed grid: every source got the same
small box whatever it had to say (warranties showed 5 of 50), and a box per
source flattened priority, while the largest type on the page belonged to counts
that need nobody.
Now a severity rollup, one worst-first queue of everything needing a person, and
a row of context beneath. The queue is full width, capped per source so a noisy
check cannot bury a rare one, and names what did not fit with a link to it. Chips
filter the queue by check as well as by severity; both narrow what is already
loaded, so the counts and rows cannot disagree.
Dropped: an 'All assets' tile shown beside its own four addends, a printer strip
carrying the same count a third time, three cards whose job was to display 0, and
a Recent Devices table listing five machines all 'In Use'.
Contract 0.26.0 adds optional 'shortname' on dashboard cards - a title is written
as a heading and ran wider than the hostname it qualified as a row badge.
Also: the supplies card reported 'Low Supplies 0' when Zabbix was unreachable,
which reads as reassurance rather than the truth that nothing could see the
cartridges. It now distinguishes unreachable (says so) from unconfigured (stays
quiet - the site declined the feature) from a real zero.
Full suite 2182 passed, frontend 314 passed.
Default fuzzy reconcile threshold to 0 and refuse merges when digit runs differ so numbered labels like Bay 311 never bind to Bay 310. Keep edit forms from clearing saved model fields when category filtering hides untyped rows. Includes in-progress computers policy and endpoints command feature work.
PC records now carry Entra/Azure device identity so imported machines keep a durable link back to their directory object instead of matching on hostname alone. Adds azuredeviceid, azureobjectid, and azuredevicecategory to the Computer model with matching migrations, surfaces them read-only in PCDetail as an Entra Device Info card, and lets PCForm edit the device id directly. The computers plugin registers a machineassetnumber import hook that reconciles PC-to-machine links during bulk import and warns on unresolved or conflicting relationships.
Adds importsources adapters that normalize Azure device-group and Thor PC inventory exports into canonical ShopDB CSVs via hostname alignment, exposes register_import_hook and RowWarning on the shopdb.api contract surface (contract 0.21.0), and refreshes the Import Site Assets page with clearer source instructions. Includes reconcile_locations maintenance script and coverage for the new import paths.
Assign printers to a MACHINE and let whichever PC controls it install them, with
the observed half kept strictly apart from the assigned half: a collector report
can never write an assignment row. Adoption is explicit. Adds a singular
relationship type so defaultprinter can mean exactly one, client scripts that
converge a bay's queues and correct a drifted one in place, and a wave gate so a
fleet-wide driver rollout cannot stampede its own share.
On the container side: the compose stack never persisted the instance directory,
so a rebuild discarded plugins.json and every upload. It is a volume now, the
image ships instance/ owned by the app user so the mountpoint is writable, and
the database moves to MySQL 8.4 LTS, which the Windows runbook has recommended
since 8.0 went end of life. The image also could not build at all: the frontend
stage never copied the plugin staging script it runs.
Adds a topology migration guide covering IIS to Docker and back.
Catalog with photos and append-only print-file revisions, a stock
ledger attributing every take/restock/adjust to a badge-resolved
employee, a touch kiosk (scan bin, scan badge, keypad quantity) with a
deliberately open decrement-only take endpoint, 1x0.5in CODE128 bin
labels, low-stock email alerts to selected users, roles, or addresses,
and stock/consumption/by-person reports with a cache-vs-ledger
reconcile. Ships disabled; enable per site. Contract 0.13.0 adds the
mailer and the User/Role models to the plugin surface.
Flask + Vue rewrite of the classic-ASP shopdb, built as a plugin
framework so sister GE Aerospace sites can adopt it. Product version
0.7.0-pre-0.8.0, plugin contract 0.11.0, 966 backend tests.
Key entry points:
- docs/INSTALL-WINDOWS-IIS.md - tested Windows + IIS install runbook
(own-site and subpath deployment methods)
- docs/PILOT-DEPLOY.md - production pilot: stand up + import + verify
- docs/IMPORT-API.md - schema-agnostic legacy data import contract
- scripts/site_imports/wjf/ - West Jefferson reference import loader
- docs/adr/ - twelve accepted architecture decision records
