An admin generates a short-lived token in Settings > Endpoints > Endpoints and
copies the command it produces. A tech runs that one line in an elevated
PowerShell on a PC with no agent:
powershell -ExecutionPolicy Bypass -c "irm https://<server>/shopdb/api/
endpoints/siteagent/bootstrap.ps1?t=<token> | iex"
The script refuses to run unelevated, refuses a PC that already has the agent
(a managed bay gets it from GE-Enforce), downloads the package the existing
builder produces, checks both binaries against the hashes the server baked in,
installs with NO shared key, and reports what happened. The PC then asks to
enroll and waits for an admin to approve it, which the script watches for
locally for two minutes before exiting.
- New endpointbootstraptokens table and migration
endpoints0024bootstraptokens: hash only, expiry, use count, revoke.
- Admin routes to generate, list and revoke tokens; the secret is shown once.
- Two token-only routes: the script, and the package (which spends a use).
Anything wrong with the token answers 404 rather than confirming it exists,
both are rate limited per address and both are audited with the caller.
- The package builder is now shared rather than duplicated, and a bootstrap
package can never carry a collector key.
Found in an end-to-end test on a Windows 11 VM against a self-signed server.
- Rollback: a 1.20 agent re-enrolls within a second of an un-enroll, so an
older agent pushed afterwards arrived on an enrolled PC and was refused.
Revoke now takes {"blockreenroll": true, "reason": ...}; a blocked PC is
never issued a credential and keeps working on the shared key until
POST .../assets/<assetid>/allow-reenroll. The Enrollment page offers the
block when un-enrolling and shows blocked PCs with an allow button.
- First upgrade of old agents: agents before 1.20 check the server
certificate when downloading the upgrade, so a store wave failed on PCs that
do not trust a self-signed server while the Migration tab said ready. A
failed download now marks the PC blocked with "server certificate not
trusted by this PC". New Install-ShopDBServerCertificate.ps1 (GE-Enforce,
SYSTEM) adds a non-CA server certificate to the trusted roots after checking
its thumbprint; the server certificate page offers a .cer download and the
SHA-1 thumbprint.
- The generated installer writes ServerPins (with an optional next pin for a
planned rotation), can leave the shared key out so the PC waits for
approval, never touches an existing credential, and skips the trusted-root
import for a CA certificate.
- The agent re-reads server pins about every 60 seconds and recycles pooled
connections within 2 minutes, so a pin change needs no restart.
- An alert can only be acknowledged by a PC it targets.
- A wave no longer counts a PC as failed after a later wave upgraded it.
- The scripts report a pin mismatch only when the pin really is missing, and
otherwise log the real error.
- Runbook: trust step before the first wave, rollback order, pin re-read,
verifying the pin against the live URL; agent contract updated.
Per-PC credentials (replaces the 09-14 client-certificate attempt):
- POST /api/endpoints/enroll issues a credential to a PC that matches exactly
one computer by hostname and BIOS serial and has none yet. Anything else
(unknown or duplicate hostname, serial mismatch, already enrolled) waits as
a request an admin approves or rejects. The agent's persisted nonce binds an
approved request to the PC that asked.
- One auth helper serves every agent route. With X-Endpoint-Credential the
asset comes from the credential and a request naming another PC gets 403
ASSET_MISMATCH. The shared key still works for PCs that have not enrolled,
is refused (401 SHARED_KEY_REFUSED) for any PC that has, and can be switched
off. A revoked credential gets 401 CREDENTIAL_INVALID so the agent
re-enrolls.
- The core collector accepts the credential for the computers, printers and
endpoints collectors through two new plugin hooks exposed on shopdb.api
(contract 0.33.0), and refuses the shared key for enrolled PCs.
- Admin API: enrollment list, approve, reject, revoke (un-enroll), settings.
Migration API: per-PC readiness with blockers, and upgrade waves that
package the agent once and queue the existing agent upgrade per PC.
- Migration 0023 creates endpointcredentials, endpointenrollrequests and
endpointmigrationwaves, drops the certificate columns from assets where
test databases have them, and renames app_fingerprint to appfingerprint.
0021 and 0022 keep their ids but no longer alter the core assets table.
- Removed: client_cert.py, certificate renewal, the unauthenticated
/test/client-cert route, nginx client certificate forwarding and the agent
certificate script. The client-cert header could be forged on IIS, the key
was readable by users, and the shared key remained a fallback.
- GET /settings/server-cert returns the SPKI SHA-256 pin for GE-Enforce.
Review fixes:
- report_command_result used g without importing it; every result was a 500.
- collect_log refuses setup answer files, appsettings, agent directories and
key files at enqueue.
- Fleet alerts: alerttimeout and alertfleet agree with the migration, the
scope is validated and a malformed scope no longer reaches the whole fleet,
and the ack race no longer raises NameError.
- A refused break-glass override leaves no hold and no audit row.
- Agent upgrade checks locks and permissions before repackaging, and writes
the package atomically.
- Server certificate upload validates the pair before replacing files;
generation no longer builds an openssl config from user input; audit rows
are actually written.
- Health evaluation runs in a savepoint so a caught error cannot lose the
check-in.
- Required-app policies are found by appid, survive renames, and deleting an
app deactivates policies that still name it. The UI and API now say such a
policy only takes effect inside a business policy.
- Backfill handles duplicate fingerprints and rolls back a failed batch.
- Plugin settings routes skip a plugin whose hook raises.
- The setup token is no longer printed to the log.
- The naming gate checks top-level files again and handles odd file names.
- Removed root scratch scripts, one of which carried a token.
- Rewrote the fleet alert scheduling and app delta tests on real fixtures.
This commit adds:
- Database migrations for app_delta fingerprints and client certificates
- Scripts for certificate generation and fingerprint backfilling
- Comprehensive security audit reports and findings tracking
- Implementation documentation for certificate authentication
- Test utilities for certificate renewal and fingerprinting
- GE-ENFORCE collector fix documentation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
CRITICAL FIXES:
- Enforce cert authentication on collector telemetry endpoint
- Add authorization check in report_command_result (prefer cert over hostname)
- Implement server certificate pinning in C# agents (optional)
- Add constant-time fingerprint comparison to prevent timing attacks
- Validate fingerprint format (40 or 64 hex chars only)
AUTHENTICATION:
- Created shopdb/core/auth/client_cert.py with authenticate_by_client_cert()
- Extracts fingerprint from nginx header (SHA1), IIS environ, or ARR header
- Validates against assets.client_cert_fingerprint in database
- Checks certificate expiration (cert_expires_at column)
- Falls back to API key when cert not present (dual-auth during rollout)
AUTHORIZATION:
- shopdb/core/api/collector.py: Added cert auth to generic_collect() before
API key check. Sets g.cert_authenticated_asset_id when cert validates.
- plugins/endpoints/plugin.py: apply_collector_payload() compares
g.cert_authenticated_asset_id with payload's resolved assetid. Raises
PermissionError if mismatch (prevents agent A from submitting data for
asset B by changing hostname field).
- plugins/endpoints/api/routes.py: report_command_result() prefers
g.cert_authenticated_asset_id over hostname parameter. Requires either
cert or hostname for authorization.
AGENT CHANGES:
- AgentConfig.cs: Added ServerCertThumbprint config for certificate pinning
- CollectorClient.cs: Validates server cert thumbprint when configured
- CommandClient.cs: Validates server cert on WebSocket TLS
- FleetAlertClient.cs: Validates server cert on WebSocket TLS
- When ServerCertThumbprint is empty, accepts all certs (rollout compat)
REVOCATION SUPPORT:
- Added cert_revoked_at and cert_revocation_reason columns to assets table
- Migration 0022_client_cert_revocation.py adds columns
- Revocation check uses hasattr() for backward compatibility
- Revocation requires manual database UPDATE (instant kill switch)
SECURITY HARDENING:
- Constant-time fingerprint comparison via hmac.compare_digest()
- Format validation: must be 40 (SHA1) or 64 (SHA256) hex chars
- SHA1 documented as nginx limitation, acceptable for air-gapped self-signed
- Audit logging for failed authentication attempts (AuditLog.log)
DEPLOYMENT:
- Dual-auth during rollout: cert OR API key both work
- Agents without certs continue using API key (backward compatible)
- No impact on existing fleet until certs provisioned
- Rollout: server first, pilot certs, fleet rollout, retire API key
Addresses Fable security review findings: 1 critical + 5 high vulnerabilities
related to authentication bypass, authorization flaws, and timing attacks.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Rolling the containers failed on strict MySQL: the endpoints chain could not
stamp its new head with "Data too long for column version_num". Plugin version
tables declare version_num VARCHAR(32) and the new revision id was 33 chars.
Nothing in the suite caught it. The tests run on SQLite, which does not enforce
declared column widths, so an over-long id passes locally and fails only on a
real deploy - after the migration body has run and only the stamp fails. The
rollback was clean here, but the failure lands at the worst possible point.
Shorten the id and assert the limit in the test that already parses every
plugin revision, so the next over-long id fails on a laptop instead of during
a roll. Core is deliberately not covered: its alembic_version is VARCHAR(128),
and it already carries ids longer than 32 that are correct.
The product shipped one Configuration Policy carrying every shopfloor app
monitor, wrapped in a shipped Site Policy that applied it to every endpoint.
That bundled two decisions a site cannot separate: which monitors exist, and
which endpoints get them. A site that wanted eDNC on one PC type and MTConnect
on another had to delete the shipped policy and rebuild both halves.
Ship one Configuration Policy per monitor with no PC type attribution, and stop
shipping a Site Policy at all. Composition is a site decision; the product now
supplies parts, not an assembled opinion.
Shipped Configuration Policies become editable and deletable. Provenance stays
a source tag rather than a lock, which is safe because seeding is insert-if-
absent and no reconciling seeder exists, so a site edit is never stomped. The
stale comments claiming shipped rows were read-only and reconciled described
behavior the code never had.
Endpoint locks
--------------
Adds EndpointLock so an admin can hold an endpoint against automated and
operator action, with migration 0018 and enforcement in the enqueue path
rather than only in the UI, so a lock cannot be bypassed by calling the
API directly.
Split endpoints.deploy
----------------------
endpoints.deploy bundled three actions of very different risk behind a
single name:
scheduledtask_run -> starts a task already registered and allowlisted
update_app -> runs a pinned installer
run_script -> arbitrary code as SYSTEM unless the descriptor
pins a user context
Because they shared one grant there was no way to let an operator push an
application update without also handing them arbitrary SYSTEM code
execution on every endpoint they could reach. The permission name
described the lightest of the three while silently carrying the heaviest.
commands.ELEVATED_ACTION_PERMISSIONS now maps each action to its own
permission (endpoints.corptools, endpoints.updateapp,
endpoints.runscript). The enqueue route reads that table, so no route
logic changed. Migration 7d42 preserves existing access exactly: anyone
holding endpoints.deploy receives all three replacements.
Fleet ranked endpoints by raw critical-event counts, so a machine with a
noisy but harmless event log outranked one with a mapped drive down or a
blocking service stopped. Volume is not severity, and the aggregation ran
across the whole fleet on every page load.
Rank operational blockers above degraded conditions above event noise, and
cache the verdict per asset so Fleet reads a row instead of recomputing 48h
of history. Transitions are written only when a verdict changes, capped at
50 rows per asset.
Absence of a policy binding must never suppress a verdict: drives and events
are not policy-gated, and services fall back to a built-in default impact
when no mapping applies. Event 1001 drops to informational because the WER
bucket pairs with 1000 and double-counted every crash.
Parallelize the Fleet first-paint loaders that were awaiting serially.
Corporate Tools tab on PC Details, gated by endpoints.deploy and liveness.
Starts already-registered scheduled tasks; the agent's ScheduledTaskPolicy
allowlist remains the authority on what may actually run.
Dell Command | Update split into two actions (Scan, Apply Updates) because
they have very different consequences. Apply Updates carries a persistent
breakglass warning in the card naming Tanium as the primary update utility,
rendered between the row title and description so it is visible at all times
rather than only in the confirm dialog. Destructive actions keep the typed
RUN confirmation.
Fix scheduledtask_run being stripped by Configuration Policy: the
intersection in ALLOWED_ACTIONS_FOR dropped it on any endpoint whose
effective policy was non-empty, so every Corporate Tools click failed with
"not permitted by policy" regardless of the caller's role. Corporate tools
are role-gated only, via POLICY_EXEMPT_ACTIONS; policy keeps authority over
monitors and service/process primitives.
Fix blank PC Details page: watch(detailtabs) evaluates its source during
setup and read canDeploy before its const declaration, throwing a TDZ
ReferenceError that aborted setup with no API request made.
Mapped-drive refresh moves to the tray. SYSTEM cannot probe user-scoped
drives, so the service requests a re-probe over a named event and waits for
the spool collectedat stamp to advance before collecting. The probe task is
registered as a ShopFloor user principal; the Users-group principal remains
only as a degraded fallback since a group member cannot start it.
Adds rebootreasonsjson, bootinfojson and defenderjson with their migration in the same change, so the model never references a column the schema has not applied.
rebootreasons is not redundant with the existing pendingreboot column. That column is a single bool OR-ed from three registry signals and cannot say which fired, and the difference matters: a staged file rename means the box is mid-servicing, while a lone servicing key can sit set for weeks on a machine nobody rebooted.
Null means never reported and is deliberately distinct from an empty list, which means checked and nothing pending. Backfilling [] would assert every pre-existing endpoint was confirmed clean.
Adds diskhealthjson with its migration in the same change, so the model never references a column the schema has not applied.
Kept separate from disksjson rather than merged into it. That column is per volume and answers how full a drive is; this one is per physical device and answers whether the hardware is dying. They come from different WMI sources with different failure modes, so merging them would make a capacity collection failure indistinguishable from a health collection failure.
Nullable with no backfill: a drive whose health has never been reported is not a healthy drive, and defaulting to an empty list would read as no disks have problems on every endpoint predating the column.
Curated events were re-read from the newest 25 entries per channel on every
collect, and the server appended whatever arrived. One real event was
therefore stored again on every cadence tick for as long as it stayed in
that window, so event counts measured how long a box had been up rather
than what had happened to it.
The agent now keeps a per-channel watermark of the last Windows event
record id it reported, persisted under ProgramData so a service restart
does not replay history, and sends eventrecordid with each event. The
server dedupes on (assetid, eventchannel, eventrecordid) behind a unique
index, so a replay from an older agent cannot double-store either.
Six inventory keys the agent had been sending all along - bios, disks,
domain, memory, osinfo, processor - were parsed, logged as unpersisted,
and dropped. They now have columns and are written on check-in, along
with user context, pending reboot, and the cpupercent/memorypercent/
idleseconds fields that were declared in the contract but always null.
Session-scoped data cannot be read from the agent's LocalSystem session 0
at all: mapped drives and user-owned listeners do not exist there. A probe
runs in the interactive user's session and hands results back through a
spool file. It is not yet producing output on a live endpoint; the column
and panel are in place and read empty until that is fixed.
Also bounds the user-context script wait. It was WaitForSingleObject with
INFINITE, so a hung script pinned the command worker permanently.
The agent already reported agentversion but the interactive tray app, which ships and updates independently, had no reported version. The agent now reads the tray exe file version and reports it as trayversion; EndpointState caches it (migration 0011) and the Fleet card surfaces it beside Agent. The tray csproj gains an explicit Version so the file version is stamped. Groundwork for per-PC version surfacing and push-update targeting.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
Default fuzzy reconcile threshold to 0 and refuse merges when digit runs differ so numbered labels like Bay 311 never bind to Bay 310. Keep edit forms from clearing saved model fields when category filtering hides untyped rows. Includes in-progress computers policy and endpoints command feature work.
Migration tests knew no endpoints head revision, so the parametrized guards raised KeyError. Register it in EXPECTED_HEAD_REVISION.
EndpointState declared lastreportedat both index=True and an explicit db.Index over the same column, and the baseline migration named single-column indexes differently from what create_all builds. On a pre-existing table the name-keyed guard could not see the create_all index and added a duplicate, tripping the idempotency guard. Drop the redundant declaration and rename migration indexes to ix_<table>_<column> to match.
endpoints was absent from the universal site profile, so the installer would not offer it and prune-schema would drop its tables at provisioning. Add it.
The agent design record lived under docs/, which is stripped to the public wiki and normally absent from the tree; a lone file there un-skipped the wiki-publish guards. Move it into the plugin.
Server-side RMM telemetry plugin: current-snapshot endpointstates plus append-only curated endpointevents, resolved to existing Computer assets by hostname. Adds ADR-010 asset panels (status, services, recent events), dashboard quiet and stability cards, and a minimal Fleet list view that links into existing PC detail pages. Registers the activity sidebar icon and a dev-only sample seeder. Ingest arrives on the generic collector route; the future C# agent design record lives in docs/DESIGN-DECISIONS-agent.md.
