Findings F2, F4, F5, F6, F7, F8, F9, F10, F11 and F13 from
wj-architectural-review.md. F1/F1a (the non-sargable LOWER pattern) and F3
(the warranty sync holding a pooled connection) are being handled separately
and are untouched here.
- A collector check-in that raced another for the same PC lost everything:
the unique index on the application fingerprint raised, nothing caught it,
and the whole check-in rolled back, discarding state, events and hardware,
answering the agent with form copy meant for a person. The insert now runs
in a savepoint, reconciles what the other transaction already wrote, one row
per savepoint so a second collision skips a row rather than the batch, and
the check-in commits with a warning. This is the shape the delete side of
the same function already had.
- Enrollment is deliberately unauthenticated, so it now carries a per-address
rate limit and a cap on pending requests, and the cap is a setting an admin
can see and change. A flood can no longer bury the approval queue or grow
the table without bound.
- The fleet alert poll no longer formats debug strings on every request.
- BusinessUnit is on the contract surface (0.34.0), so the endpoints plugin
stops reaching into a core table with raw SQL.
- A test walks every asset type descriptor and proves the endpoint it names
really refuses a caller without the declared permission.
- The plugin settings routes now need a permission; they were telling anyone
which modules a site runs and where its components live.
- The connection pool is sized explicitly with a short timeout, and the
arithmetic against the server's connection ceiling is written next to the
worker count instead of being left to defaults nobody chose.
- Fleet alerts carry both the correct lowercase keys and the old camelCase
pair. Trays older than 1.7.0 match keys case-sensitively and would have gone
quiet on every PC not yet upgraded; the old pair goes when they are.
- The dev server proxy points at localhost again, with environment variables
for the remote-box workflow, and one developer's address is out of the
server certificate page.
All three found testing the bootstrap installer on Windows.
- The published command used the irm-pipe-iex form, which Microsoft Defender
detects as Trojan:Win32/Commando.A!ml and kills before it runs, every time,
on a default install. The command now downloads the script to a file, runs
it, and removes it in a finally block, which Defender does not object to.
The UI also offers the script as a plain download for a tech who would
rather save it.
- Approving a request only linked the asset, so a PC whose pending reason was
"unknown hostname" - which is every PC installed this way - kept getting 403
on every call afterwards, because its computer record had no hostname to
match the credential against. Approval now stamps the hostname when the
record has none, refuses with a 409 naming both when the record has a
different one, and overwrites only when the admin confirms. The Credentials
tab offers that confirmation.
- A hostname that resolves to no record now says so instead of reading as an
impostor, while still refusing the request.
- While waiting for approval the agent retried on the 15-minute telemetry
cadence, so a PC approved immediately enrolled a quarter of an hour later
and the installer always ended on "still pending". It now retries about
every minute for the first ten minutes, then every five to thirty minutes,
then every fifteen, with jitter. A rate limit or server error still uses the
slower backoff.
An admin generates a short-lived token in Settings > Endpoints > Endpoints and
copies the command it produces. A tech runs that one line in an elevated
PowerShell on a PC with no agent:
powershell -ExecutionPolicy Bypass -c "irm https://<server>/shopdb/api/
endpoints/siteagent/bootstrap.ps1?t=<token> | iex"
The script refuses to run unelevated, refuses a PC that already has the agent
(a managed bay gets it from GE-Enforce), downloads the package the existing
builder produces, checks both binaries against the hashes the server baked in,
installs with NO shared key, and reports what happened. The PC then asks to
enroll and waits for an admin to approve it, which the script watches for
locally for two minutes before exiting.
- New endpointbootstraptokens table and migration
endpoints0024bootstraptokens: hash only, expiry, use count, revoke.
- Admin routes to generate, list and revoke tokens; the secret is shown once.
- Two token-only routes: the script, and the package (which spends a use).
Anything wrong with the token answers 404 rather than confirming it exists,
both are rate limited per address and both are audited with the caller.
- The package builder is now shared rather than duplicated, and a bootstrap
package can never carry a collector key.
The page saved every field as 'true'/'false', the kiosk device token
included, and loaded the token as a boolean. One save of Settings > USB
Devices therefore stored the literal word "true" (or "false") as the token,
and every kiosk provisioned with the real token got 404 on each request -
reported as the kiosk no longer accepting an SSO. Generate + Save never
stored the generated value either.
The token now loads and saves as text. The server already keeps the stored
token when it receives the mask back unchanged. The two switches still save
as booleans. The parts kiosk page was not affected.
Found in the Windows VM check.
- The agent upgrade script wrapped Get-ShopDbServerPins in @(), which turned
several registry pins into one joined string. During a certificate rotation
the upgrade download would have refused the rotated server. It also merged
appsettings pins with the registry; it now follows the agent's rule: when the
ServerPins value exists, even empty, the registry owns the list. The shared
pin block in the upgrade script and both reporters gains
Test-ShopDbServerPinsValue.
- A key-less package installed over an enrolled PC printed both "existing
enrolled credential found; kept as is" and "This PC will request
enrollment". The enrollment notices now print only when no credential exists.
Found in the Windows VM re-test.
- A CA-flagged server certificate (common with a plain openssl self-signed
cert) produced an installer package with no pin at all, so agent 1.20
refused the server. The package now always carries the SPKI pin of the
certificate PCs receive; the CA flag only skips the trusted-root import.
When an uploaded CA-flagged certificate differs from the server's own, both
pins are included. The build response carries X-ShopDB-Build-Warning and the
installer logs what older agents need.
- Pins written into appsettings.json by the installer were merged with the
registry, so removing a pin from HKLM never took effect. When the ServerPins
value exists in the registry it is now authoritative, even when empty;
appsettings pins apply only when it is absent. Set-ShopDBServerPins.ps1
-Clear deletes the value; new -None writes an empty value (no pins).
- A script or package download sent with a credential revoked moments earlier
got 401 CREDENTIAL_INVALID and failed the command. The agent now retries that
download once with the shared key and records the fallback in the result.
Found in the Windows VM re-test.
- A PC installed without the shared key got 401 from POST
/api/endpoints/enroll while the shared key was allowed, so no request was
recorded and no admin could approve it. Enroll now never refuses: a valid
shared key only decides auto-issue on a clean match; no key or an invalid
key creates or updates the one pending request for that hostname and
serial. An approved request still issues only to its own nonce.
- A PC blocked with "server certificate not trusted by this PC" could never be
queued again after trust was fixed, because the blocker comes from its last
failed upgrade. Naming the PC in a wave now queues it when trust is its only
blocker, and the response lists it under "retried"; "next n" still skips
it. The Migration table lets such a PC be selected.
- counts.pending on GET /enrollment counts pending requests, including ones
for unknown hostnames; counts.pendingcomputers keeps the per-computer figure.
- Endpoints (shared key) page: the key is presented as what PCs use until
they enroll, with enrollment progress, a link to Enrollment, rewritten
guidance, and rotate/revoke confirmations that say how many PCs are not
enrolled yet.
- Settings cards describe the server certificate as what PCs actually
receive, on IIS or Docker; Enrollment has its own icon (the icon set lacked
the one the Server Certificate card already asked for).
- Server Certificate page checks the certificate the site presents at its
public address and compares its pin with the certificate file, warning when
they differ. The host comes from site settings only; the check is a TLS
handshake, never an HTTP request.
- Fleet list: Credential column and filter (enrolled, pending, re-enroll
blocked, shared key, not reporting), computed in the list query.
- PC detail: credential state, prefix and dates for fleet readers, state only
for others, with admin Revoke (optionally blocking re-enrollment) and Allow
re-enrollment.
- Dashboard: pending enrollment requests, and PCs blocked from re-enrolling
for more than seven days.
- Runbook: where to find each of these in the app.
Found in an end-to-end test on a Windows 11 VM against a self-signed server.
- Rollback: a 1.20 agent re-enrolls within a second of an un-enroll, so an
older agent pushed afterwards arrived on an enrolled PC and was refused.
Revoke now takes {"blockreenroll": true, "reason": ...}; a blocked PC is
never issued a credential and keeps working on the shared key until
POST .../assets/<assetid>/allow-reenroll. The Enrollment page offers the
block when un-enrolling and shows blocked PCs with an allow button.
- First upgrade of old agents: agents before 1.20 check the server
certificate when downloading the upgrade, so a store wave failed on PCs that
do not trust a self-signed server while the Migration tab said ready. A
failed download now marks the PC blocked with "server certificate not
trusted by this PC". New Install-ShopDBServerCertificate.ps1 (GE-Enforce,
SYSTEM) adds a non-CA server certificate to the trusted roots after checking
its thumbprint; the server certificate page offers a .cer download and the
SHA-1 thumbprint.
- The generated installer writes ServerPins (with an optional next pin for a
planned rotation), can leave the shared key out so the PC waits for
approval, never touches an existing credential, and skips the trusted-root
import for a CA certificate.
- The agent re-reads server pins about every 60 seconds and recycles pooled
connections within 2 minutes, so a pin change needs no restart.
- An alert can only be acknowledged by a PC it targets.
- A wave no longer counts a PC as failed after a later wave upgraded it.
- The scripts report a pin mismatch only when the pin really is missing, and
otherwise log the real error.
- Runbook: trust step before the first wave, rollback order, pin re-read,
verifying the pin against the live URL; agent contract updated.
The 2026-09-12 audit reports, the unvalidated findings and the Darkwing scan
output described weaknesses in detail on the production branch. They are
kept outside the repository with the site's other working records; git
history still holds them.
- Settings > Endpoints > Enrollment. Migration tab: readiness counts and
progress, each PC's state with the reasons it is blocked, "Upgrade
selected" and "Upgrade next 25", and live wave progress. Credentials tab:
enrolled PCs with revoke, pending requests with approve and reject, and the
shared key and auto-enroll switches. Turning the shared key off asks for
confirmation and shows how many PCs are not enrolled.
- The server certificate page shows the SPKI pin to put in the GE-Enforce
pin entry.
- The agent upgrade script and the asset and printer reporters send the PC's
credential when it has one (over https, only to the issuing server), fall
back to the shared key, check server pins, and log refusals clearly.
- New Set-ShopDBServerPins.ps1 (GE-Enforce entry) and Get-ShopDBServerPin.ps1.
- AGENT-CONTRACT.md documents per-PC credentials; the rollout runbook covers
phases, gates, rollback, pin and wave entries, and every place the shared
key lives and how to clear it at cutover.
- Removed the 09-14 certificate design notes and dated session reports, and
two VM notes that carried a live collector key and advised plain http.
- Corrected the GE-Enforce collector note, which claimed a plain registry
token is protected by ACL.
Agent 1.20.0, tray 1.7.0. Ship together.
Agent:
- Every HTTP client, including script and package downloads, is built by one
AgentHttpFactory. Server pins (SPKI SHA-256, current and next) come from
HKLM ServerPins or appsettings serverpins; with a pin set only a matching
key is trusted, otherwise the normal chain check applies.
- EnrollmentService enrolls once using the shared key the PC already has,
stores the credential DPAPI-protected in a SYSTEM-only directory, and swaps
to it without a restart. Pending requests retry every 15 minutes; an old
server without the enroll route leaves the agent on the shared key. A
revoked credential is deleted and the PC re-enrolls.
- A credential is only ever sent to the server that issued it.
- Telemetry posts to /api/endpoints/collect when using a credential.
- A named pipe lets the interactive user ask the agent to send status, send a
report, list alerts and acknowledge one. The report sidecar and screenshot
must sit in the caller's own profile and not be a link.
- --once and --report run as a user become pipe clients.
- appsettings.json is restricted to SYSTEM and Administrators once the pipe is
up.
- Removed client certificate renewal and expiry monitoring.
Tray:
- Holds no key and reads no agent configuration. Alerts, acknowledgements,
Send Status and Report an issue all go through the agent's pipe.
- An acknowledgement is remembered only when the agent confirms it.
- When the agent is not running, the menu says so and alert polling retries
quietly.
Per-PC credentials (replaces the 09-14 client-certificate attempt):
- POST /api/endpoints/enroll issues a credential to a PC that matches exactly
one computer by hostname and BIOS serial and has none yet. Anything else
(unknown or duplicate hostname, serial mismatch, already enrolled) waits as
a request an admin approves or rejects. The agent's persisted nonce binds an
approved request to the PC that asked.
- One auth helper serves every agent route. With X-Endpoint-Credential the
asset comes from the credential and a request naming another PC gets 403
ASSET_MISMATCH. The shared key still works for PCs that have not enrolled,
is refused (401 SHARED_KEY_REFUSED) for any PC that has, and can be switched
off. A revoked credential gets 401 CREDENTIAL_INVALID so the agent
re-enrolls.
- The core collector accepts the credential for the computers, printers and
endpoints collectors through two new plugin hooks exposed on shopdb.api
(contract 0.33.0), and refuses the shared key for enrolled PCs.
- Admin API: enrollment list, approve, reject, revoke (un-enroll), settings.
Migration API: per-PC readiness with blockers, and upgrade waves that
package the agent once and queue the existing agent upgrade per PC.
- Migration 0023 creates endpointcredentials, endpointenrollrequests and
endpointmigrationwaves, drops the certificate columns from assets where
test databases have them, and renames app_fingerprint to appfingerprint.
0021 and 0022 keep their ids but no longer alter the core assets table.
- Removed: client_cert.py, certificate renewal, the unauthenticated
/test/client-cert route, nginx client certificate forwarding and the agent
certificate script. The client-cert header could be forged on IIS, the key
was readable by users, and the shared key remained a fallback.
- GET /settings/server-cert returns the SPKI SHA-256 pin for GE-Enforce.
Review fixes:
- report_command_result used g without importing it; every result was a 500.
- collect_log refuses setup answer files, appsettings, agent directories and
key files at enqueue.
- Fleet alerts: alerttimeout and alertfleet agree with the migration, the
scope is validated and a malformed scope no longer reaches the whole fleet,
and the ack race no longer raises NameError.
- A refused break-glass override leaves no hold and no audit row.
- Agent upgrade checks locks and permissions before repackaging, and writes
the package atomically.
- Server certificate upload validates the pair before replacing files;
generation no longer builds an openssl config from user input; audit rows
are actually written.
- Health evaluation runs in a savepoint so a caught error cannot lose the
check-in.
- Required-app policies are found by appid, survive renames, and deleting an
app deactivates policies that still name it. The UI and API now say such a
policy only takes effect inside a business policy.
- Backfill handles duplicate fingerprints and rolls back a failed batch.
- Plugin settings routes skip a plugin whose hook raises.
- The setup token is no longer printed to the log.
- The naming gate checks top-level files again and handles odd file names.
- Removed root scratch scripts, one of which carried a token.
- Rewrote the fleet alert scheduling and app delta tests on real fixtures.
Tray 1.6.11.
- One alert with a null or malformed field no longer throws away the whole
alert list; nullable server fields are tolerated.
- Keystrokes can no longer acknowledge an alert the operator has not seen.
There is no default button, focus starts on the message, and acknowledging
takes a mouse click after the alert has been on screen for 1.5 seconds.
- Long messages scroll instead of being cut off, and an ampersand is shown as
typed.
- A timeout closes the alert without acknowledging it.
- Missing type names, titles and colours fall back to safe defaults.
- Shutdown waits on cancellation instead of sleeping, and closes an open alert
before the client is disposed.
- The acknowledged-alerts store moved from ProgramData, which the operator
cannot write, to the user's local app data, guarded by a mutex. Acked alerts
used to reappear at every logon.
Agent 1.19.0.
- With no pinned thumbprint the collector, command and fleet alert clients
accepted any server certificate. They now require a valid chain again.
Production uses a self-signed certificate, so this build must not reach bays
before server pinning ships with it.
- SYSTEM script runs read stdout and stderr while the script runs, cap what is
kept, and kill the whole process tree after five minutes. A chatty script
used to fill the pipe and hang the command channel until a service restart.
- Scripts and update_app packages are staged in a directory only SYSTEM and
Administrators can touch, under a random name, with the hash checked in
memory and again on disk. A standard user could previously swap the file
between the hash check and execution.
- The user-context transcript lives in a SYSTEM-created directory, and SYSTEM
refuses to follow a link when reading or deleting it.
- Strict mode is inserted after any param block, #Requires or using lines,
keeps line numbers and writes UTF-8 with a BOM.
- A user-context run that reports no exit code now fails.
- collect_log refuses setup answer files, appsettings files, the agent's own
directories, key files and links.
- The startup renewal check is capped at 15 seconds and exits non-zero when it
wants a restart, so service recovery restarts the agent.
Removed .with_for_update() from _replace_applications() which was
acquiring row-level locks on every collector check-in (800/hour).
These locks were holding during fingerprint comparison, blocking
concurrent reads and causing severe performance degradation.
The UNIQUE constraint on (assetid, app_fingerprint) already provides
sufficient concurrency protection without explicit row locks.
Impact: Eliminates 800 lock acquisitions per hour, unblocks reads
during check-in processing.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit adds:
- Database migrations for app_delta fingerprints and client certificates
- Scripts for certificate generation and fingerprint backfilling
- Comprehensive security audit reports and findings tracking
- Implementation documentation for certificate authentication
- Test utilities for certificate renewal and fingerprinting
- GE-ENFORCE collector fix documentation
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
UX Fix: Code block contrast in dark mode
Problem:
- Code block with JSON example still white-on-white in dark mode
- User reported: 'still white text on white bg'
- Previous dark mode fix didn't override parent styles
Root Cause:
- Parent .field-hint and .info-list classes override .code-block
- Need !important to force dark mode colors
- <pre> tag selector needed more specificity
Solution:
- Added !important to dark mode color rules
- Added explicit selectors for pre.code-block
- Added nested selectors: .info-list .code-block, .field-hint .code-block
- Ensures dark mode applies regardless of parent context
Dark Mode Colors (enforced):
- background: #2b2b2b (dark gray)
- color: #e0e0e0 (light gray)
- border: #444 (medium gray)
Testing:
- Run: node scripts/stage-frontend.mjs
- Check: JSON code block in 'Agent certificate pinning' section
- Verify: Dark background, light text, readable in dark mode
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
UX Fix: Dark mode readability
Problem:
- Warning banner: light yellow background with dark text unreadable in dark mode
- Code blocks: light gray background with dark text unreadable in dark mode
- User reported: 'warning on darkmode not visible, light yellow background on white text'
Solution:
- Added @media (prefers-color-scheme: dark) styles
- Warning banner: dark background (#664d03), light text (#ffecb5)
- Code blocks: dark background (#2b2b2b), light text (#e0e0e0)
- Maintains good contrast in both light and dark themes
Colors:
- Light mode warning: #fff3cd background, #856404 text (WCAG AA compliant)
- Dark mode warning: #664d03 background, #ffecb5 text (high contrast)
- Light mode code: #f8f9fa background, #212529 text
- Dark mode code: #2b2b2b background, #e0e0e0 text
Deployment Note:
After editing plugins/*/frontend/ files, run:
node scripts/stage-frontend.mjs
This copies plugin frontends to frontend/src/.plugins-staged/
where Vite reads them (per ADR-013 plugin pruning).
Context:
- Missed running staging script initially (my error)
- Manual copy to .plugins-staged worked but wrong workflow
- .plugins-staged/ is .gitignored (generated, not committed)
- Proper workflow: edit source -> stage -> Vite hot-reloads
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical Bug Fix: Server cert settings page blank screen
Problem:
- Navigating to /settings/endpoints/server-cert showed blank page
- Both localhost:5173 and localhost:5001 affected
- Browser console showed authentication errors
Root Cause:
- ServerCertSettings.vue used raw fetch() without authentication
- API endpoint requires admin role (@require_role('admin'))
- fetch() didn't include JWT bearer token from cookies
- API returned 401 Unauthorized, component failed silently
Solution:
- Converted from Options API to Composition API (Vue 3 standard)
- Changed from raw fetch() to authenticated api client
- Used 'import api from @/api' (same as AgentKeySettings.vue)
Changes:
Before (broken):
export default {
methods: {
async loadCertificate() {
const response = await fetch('/api/endpoints/settings/server-cert')
// No auth headers = 401 error
}
}
}
After (working):
import { ref, onMounted } from 'vue'
import api from '@/api'
async function loadCertificate() {
const response = await api.get('/endpoints/settings/server-cert')
// api.get() includes JWT from cookie automatically
}
API Client Features:
- Automatically includes Authorization header with JWT
- Handles response parsing
- Consistent error handling
- Used by all other admin settings pages
Result:
- Page loads successfully
- Certificate details display
- Generate/upload forms work
- All admin-only endpoints authenticated
Tested:
- Hard refresh clears browser cache
- Page renders certificate details
- Generate/upload buttons functional
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Build Fix: .NET 8 API changes
Problem:
- Agent failed to compile with .NET 8.0.424
- Error: 'File' does not contain a definition for 'GetAccessControl'
- Error: 'File' does not contain a definition for 'SetAccessControl'
Root Cause:
- File.GetAccessControl() and File.SetAccessControl() removed in .NET 5+
- Moved to System.IO.FileSystem.AccessControl package as extension methods
- Methods now on FileInfo/DirectoryInfo classes, not File static class
Solution:
1. Added package reference:
- System.IO.FileSystem.AccessControl v5.0.0 (ShopDbEndpointAgent.Core.csproj)
2. Added using directive:
- using System.Security.AccessControl; (CertificateRenewal.cs)
3. Changed API usage (CertificateRenewal.cs:290-298):
Before:
var acl = File.GetAccessControl(keyFile);
File.SetAccessControl(keyFile, acl);
After:
var fileInfo = new FileInfo(keyFile);
var acl = fileInfo.GetAccessControl();
fileInfo.SetAccessControl(acl);
Build Results:
- ShopDbEndpointAgent.Core: SUCCESS (1 warning, 0 errors)
- ShopDbEndpointAgent: SUCCESS (0 warnings, 0 errors)
- ShopDbEndpointAgent.Tray: SUCCESS (0 warnings, 0 errors)
- Published to agent/publish/agent/ and agent/publish/tray/
Context:
- Private key ACL grants required for tray app access (CertificateRenewal.cs:290)
- BUILTIN\Users group needs Read+ReadAndExecute on private key file
- Enables certificate renewal from both service and tray contexts
Related Changes:
- CertificateExpiryMonitor.cs (periodic monitoring, commit bb301c6)
- CertificateRenewal.cs (server cert validation, commit 702eeb3)
- All security fixes operational after this build
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Bug Fix: Fleet alert scheduling regression
Problem:
- User sets notification to run at 3:30 PM (future time)
- Fleet alert sent to agents immediately at 3:10 PM instead of waiting
- Agents see notification 20 minutes early
Root Cause:
- GET /api/endpoints/alerts query filtered for:
- alertfleet = True
- isactive = True
- endtime is NULL OR endtime > now (not expired)
- Not acknowledged
- MISSING CHECK: starttime is NULL OR starttime <= now (has started)
- Result: Any notification with future starttime appeared immediately
Solution:
- Added starttime filter to fleet alert query (routes.py:4151-4153)
- Query now checks:
db.or_(
Notification.starttime.is_(None),
Notification.starttime <= now
)
- Notifications with future starttime excluded until that time
Behavior After Fix:
- Notification with no starttime: appears immediately (existing behavior)
- Notification with past starttime: appears immediately (existing behavior)
- Notification with future starttime: waits until that time (FIXED)
- Notification with past endtime: never appears (existing behavior)
Testing:
- tests/test_plugins/test_endpoints_fleet_alert_scheduling.py
- test_fleet_alert_respects_future_starttime (regression test)
- test_fleet_alert_with_no_starttime_appears_immediately
- test_fleet_alert_past_starttime_appears
- test_fleet_alert_expired_does_not_appear
Timeline Example:
- 3:10 PM: User creates notification with starttime=3:30 PM, alertfleet=True
- 3:15 PM: Agent polls /api/endpoints/alerts -> empty (starttime not reached)
- 3:35 PM: Agent polls again -> alert appears (starttime has passed)
Impact:
- Agents poll every 60 seconds, so scheduled alerts appear within 1 minute of starttime
- No database migration needed (starttime column already exists)
- Backwards compatible (NULL starttime still means immediate)
Related:
- Notification.is_current property already checks starttime (models/notification.py:162-169)
- Shopfloor board respects starttime correctly
- Only fleet alert endpoint had this regression
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Documentation:
- certs/README.md: Complete guide for managing nginx TLS certificates
- Server cert generation (self-signed, CA-signed, Let's Encrypt)
- Certificate expiry monitoring and renewal procedures
- Agent certificate pinning configuration
- Air-gapped deployment notes
Tooling:
- certs/regenerate-server-cert.sh: Bash script for cert regeneration
- certs/regenerate-server-cert.ps1: PowerShell script for Windows
- Both scripts backup old certs and display new thumbprint
Config:
- Updated .gitignore to allow cert management files (docs/scripts/config)
- Private keys and actual certificates still excluded (security)
Context:
- User asked where to manage server-side certificate
- Current cert: self-signed, CN=localhost, valid 1 year
- Agents validate server cert via pinning or Windows trusted roots
- Critical for mTLS client cert authentication (ADR-006)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
CRITICAL OPERATIONAL FIX: Agents running 90+ days without restart
Problem:
- Certificate renewal check only ran at service startup
- Agents running continuously for 30+ days never renewed certificates
- Certificates would expire at day 730 causing authentication failures
- User confirmed agents routinely run 90+ days without restart
Solution:
- Added CertificateExpiryMonitor background service
- Checks certificate expiry every 24 hours during runtime
- Triggers renewal when < 30 days remaining
- Gracefully stops service (Windows Service Manager restarts it)
- New process picks up renewed certificate
Implementation:
- CertificateExpiryMonitor.cs: Background service (IHostedService)
- Registered in Program.cs as hosted service
- First check at 1 hour after startup, then every 24 hours
- Uses same CertificateRenewal logic as startup check
- Calls IHostApplicationLifetime.StopApplication() after renewal
Timeline Example:
- Day 0: Agent starts with 730-day cert
- Day 1-89: Agent runs continuously (no restart)
- Day 90: Periodic check runs, cert has 640 days left, continue
- Day 700: Periodic check runs, cert has 30 days left, renews
- Agent stops gracefully, service manager restarts
- Day 701: Agent continues with new 730-day cert
Without this fix:
- Agent starts day 0, runs continuously
- Day 730: Cert expires, no renewal check ever ran
- Authentication fails at TLS handshake
- Agent dead until manual intervention
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Addresses all critical and high-severity findings from Fable security review.
CRITICAL FIX#1: Server Certificate Validation Bypass
- File: CertificateRenewal.cs
- Issue: Agent accepted ANY server certificate when ServerCertThumbprint not configured
- Fix: Now validates against Windows trusted roots when pinning not configured
- Impact: Prevents MITM attacks during renewal requests
CRITICAL FIX#2: Certificate Revocation Check Incomplete
- Files: asset.py, client_cert.py, migration 0022
- Issue: Revocation check used hasattr() and silently skipped when columns missing
- Fix: Applied migration 0022, uncommented revocation columns, removed hasattr()
- Impact: Revoked certificates can no longer authenticate or renew
CRITICAL FIX#3: Expired Certificates Cannot Renew
- Files: routes.py (renewal endpoint), CertificateRenewal.cs
- Issue: Expired certs fail at TLS handshake before reaching renewal endpoint
- Fix: Added API key + hostname fallback for renewal when cert expired
- Agent now sends hostname in request body for API key authentication
- Impact: Agents can renew expired certificates, 30-day window becomes effective
HIGH FIX#4: Hostname Injection Vulnerability
- File: routes.py
- Issue: No validation before passing hostname to certificate generation
- Fix: Added regex validation - 1-64 chars, alphanumeric plus dot/hyphen/underscore
- Impact: Prevents injection attacks via malicious hostnames
HIGH FIX#5: Rate Limit Bypass via Clock Manipulation
- File: routes.py
- Issue: Naive UTC datetime arithmetic vulnerable to timezone changes
- Fix: Use timezone-aware datetime with explicit UTC conversion
- Impact: Rate limit cannot be bypassed via clock manipulation
HIGH FIX#6: Hard Process Termination Loses In-Flight Work
- File: Program.cs
- Issue: Environment.Exit(0) aborted in-flight work and could corrupt state
- Fix: Graceful return from RunHostAsync() instead of Environment.Exit()
- Impact: Service exits cleanly, no lost telemetry or corrupted local state
HIGH FIX#7: Missing Database Transaction Isolation
- File: routes.py
- Issue: Race condition in concurrent renewal requests
- Fix: Added with_for_update() row-level locking for both auth paths
- Impact: Prevents concurrent renewals from bypassing rate limit
MEDIUM FIX#8: Missing Audit Logging for Failures
- File: routes.py
- Issue: Failed renewal attempts not captured in audit trail
- Fix: Added AuditLog.log() for cert generation and database update failures
- Impact: Security monitoring can detect repeated renewal failures
SECURITY HARDENING:
- Server cert validation: Never accepts certificates blindly
- Revocation enforcement: Fail closed if columns missing
- Expired cert handling: Dual-auth allows renewal while maintaining security
- Input validation: Hostname must match safe character set
- Transaction safety: Row locks prevent race conditions
- Audit trail: All failures logged for security monitoring
OPERATIONAL IMPROVEMENTS:
- Agents can now renew expired certificates (no more manual intervention)
- Graceful shutdown prevents data loss during renewal
- Rate limiting immune to timezone issues
- Comprehensive audit trail for troubleshooting
ADR COMPLIANCE:
- Maintains ADR-001 (Asset as platform contract)
- Maintains ADR-006 (Collector contract - backward compatible)
- Database naming (snake_case columns)
- No unicode/emojis
- Security best practices: ALL CRITICAL/HIGH issues resolved
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
AGENT SELF-RENEWAL:
- Agent checks certificate expiry on startup
- If < 30 days remaining, requests renewal from server
- Server generates new certificate and updates database
- Agent installs new cert, updates config, and restarts
SERVER-SIDE (/api/endpoints/certificate/renew):
- Requires authentication via existing (expiring) client certificate
- Rate limited: 1 renewal per asset per 24 hours
- Generates 2-year self-signed certificate with cryptography library
- Computes SHA1 fingerprint (matches nginx $ssl_client_fingerprint)
- Updates assets table with new fingerprint and expiration
- Returns PFX/P12 certificate with random password to agent
- Audit logs all renewal operations
AGENT-SIDE (CertificateRenewal.cs):
- CheckAndRenewIfNeeded() called on service startup before host starts
- Finds current certificate by thumbprint in LocalMachine\My store
- Checks expiry: if < 30 days, POST to /api/endpoints/certificate/renew
- Authenticates renewal request with expiring certificate (mTLS)
- Receives PFX + password in JSON response
- Imports new certificate to LocalMachine\My store
- Grants BUILTIN\Users read access to private key (for tray app)
- Updates appsettings.json with new thumbprint
- Exits with code 0 so service manager restarts agent with new cert
CERTIFICATE GENERATION (certificate_renewal.py):
- Uses cryptography library to generate RSA 2048-bit key pairs
- Self-signed certificates with CN=hostname, 2-year validity
- Extended key usage: CLIENT_AUTH
- Subject Alternative Name: hostname as DNS name
- Exports as password-protected PFX/P12 for Windows import
- SHA1 fingerprint computed from DER encoding (nginx compatible)
STARTUP FLOW:
1. Agent service starts (Program.cs RunHostAsync)
2. CertificateRenewal.CheckAndRenewIfNeeded() runs BEFORE host starts
3. If cert expires in < 30 days, requests renewal from server
4. If renewal succeeds, agent exits (Environment.Exit(0))
5. Windows Service Manager automatically restarts the service
6. New process picks up updated thumbprint from appsettings.json
7. Agent continues with renewed certificate
SECURITY:
- Renewal requires valid (not expired) existing certificate
- Rate limit prevents renewal loop abuse
- Server validates cert not revoked before renewal
- Audit logging tracks all renewal operations
- PFX password is random, used once during import, never stored
PERFORMANCE:
- Renewal check adds ~100-200ms to service startup
- Only triggers when expiry < 30 days (rare operation)
- No impact on steady-state agent performance
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
CRITICAL FIXES:
- Enforce cert authentication on collector telemetry endpoint
- Add authorization check in report_command_result (prefer cert over hostname)
- Implement server certificate pinning in C# agents (optional)
- Add constant-time fingerprint comparison to prevent timing attacks
- Validate fingerprint format (40 or 64 hex chars only)
AUTHENTICATION:
- Created shopdb/core/auth/client_cert.py with authenticate_by_client_cert()
- Extracts fingerprint from nginx header (SHA1), IIS environ, or ARR header
- Validates against assets.client_cert_fingerprint in database
- Checks certificate expiration (cert_expires_at column)
- Falls back to API key when cert not present (dual-auth during rollout)
AUTHORIZATION:
- shopdb/core/api/collector.py: Added cert auth to generic_collect() before
API key check. Sets g.cert_authenticated_asset_id when cert validates.
- plugins/endpoints/plugin.py: apply_collector_payload() compares
g.cert_authenticated_asset_id with payload's resolved assetid. Raises
PermissionError if mismatch (prevents agent A from submitting data for
asset B by changing hostname field).
- plugins/endpoints/api/routes.py: report_command_result() prefers
g.cert_authenticated_asset_id over hostname parameter. Requires either
cert or hostname for authorization.
AGENT CHANGES:
- AgentConfig.cs: Added ServerCertThumbprint config for certificate pinning
- CollectorClient.cs: Validates server cert thumbprint when configured
- CommandClient.cs: Validates server cert on WebSocket TLS
- FleetAlertClient.cs: Validates server cert on WebSocket TLS
- When ServerCertThumbprint is empty, accepts all certs (rollout compat)
REVOCATION SUPPORT:
- Added cert_revoked_at and cert_revocation_reason columns to assets table
- Migration 0022_client_cert_revocation.py adds columns
- Revocation check uses hasattr() for backward compatibility
- Revocation requires manual database UPDATE (instant kill switch)
SECURITY HARDENING:
- Constant-time fingerprint comparison via hmac.compare_digest()
- Format validation: must be 40 (SHA1) or 64 (SHA256) hex chars
- SHA1 documented as nginx limitation, acceptable for air-gapped self-signed
- Audit logging for failed authentication attempts (AuditLog.log)
DEPLOYMENT:
- Dual-auth during rollout: cert OR API key both work
- Agents without certs continue using API key (backward compatible)
- No impact on existing fleet until certs provisioned
- Rollout: server first, pilot certs, fleet rollout, retire API key
Addresses Fable security review findings: 1 critical + 5 high vulnerabilities
related to authentication bypass, authorization flaws, and timing attacks.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Problem: Users experiencing frequent forced re-logins after backend restarts,
especially with 8-hour token expiration in production.
Root cause analysis:
1. JWT tokens stored in localStorage on browser
2. Backend restarts with potentially different JWT secret key
3. Frontend doesn't validate stored token on app startup
4. User navigates normally, then first API request returns 401
5. User surprised with sudden logout
This fix adds token validation on app startup. If a stored token exists,
immediately verify it with /api/auth/me before mounting the app. If the
token is invalid (expired or stale secret), user is logged out at startup
rather than mid-session.
Additional issue found (not committed - .env files in .gitignore):
- .env and .env.docker had DIFFERENT JWT_SECRET_KEY values
- Fixed locally: standardized to same key across all .env files
- Action required: Verify production JWT_SECRET_KEY is consistent and persisted
Changes:
- frontend/src/main.js: Call authStore.fetchUser() on app load if token exists
- Logout happens immediately at page load, not on first interaction
- Better UX: "please log in" vs "surprise logout while working"
Future improvement (not in this commit):
- Implement automatic token refresh before expiration
- Add proactive refresh timer (e.g., every 7 hours for 8-hour tokens)
- Would eliminate re-login entirely for active sessions
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Feature: Support Code 39 symbology alongside existing QR and CODE128
- Added CODE39 option to code type dropdown (front and back sides)
- Updated renderOne() to handle code39 type with format parameter
- Code 39 shares CSS styling with CODE128 (linear barcode layout)
- Updated error message to be format-agnostic
Use case: Legacy systems and equipment that require Code 39 format
for compatibility (common in manufacturing and inventory systems)
Technical notes:
- JsBarcode library already supports CODE39 format
- Passed format parameter to barcodeSvgDataUri() function
- Both CODE128 and CODE39 use .type-barcode CSS class
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix 1: Remove duplicate Setting import violating plugin contract
- plugins/endpoints/api/routes.py:3551 imported shopdb.core.models.Setting
- Setting already imported from shopdb.api at line 21
- Removed local import to comply with plugin contract (ADR-001)
- Test: test_plugins_only_import_contract_surface
Fix 2: Align installer defaults with measuringtools manifest
- measuringtools manifest declares default_enabled: true
- Installer (ShopDBFlask.iss) excluded it from PluginDefault()
- Removed measuringtools from exclusion list (now 4 excluded, not 5)
- Exclusions now match manifests: usb, employees, geenforce, printedparts
- Test: test_wizard_defaults_match_the_manifests
Historical note: measuringtools was promoted from default_enabled false to
true but the installer config was never updated to reflect this change.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Bugfix: Fix AttributeError in /api/plugins/settings-routes endpoint
- Changed get_enabled_plugin_instances() to get_all_plugins()
- Added explicit enabled check with pm.registry.is_enabled()
- Test failure: test_no_route_answers_a_nonsense_filter_with_a_server_error
Optimization: Enable parallel test execution (Phase 1 of pytest speedup)
- Added pytest-xdist configuration to pytest.ini
- Enabled -n auto (auto-detect CPU cores, typically 8 workers)
- Added slow marker for future test categorization
- Expected speedup: 4-8x (20 min -> 2.5-5 min)
Root cause: 2857 tests recreate entire schema per test (0.4-0.6s overhead each)
- Phase 2 (savepoint isolation) will achieve 80x total speedup
- See docs/PYTEST-OPTIMIZATION-PLAN.md for full analysis
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
New endpoint must answer unauthenticated pages (kiosks, TV displays)
so they can load plugin routes before login. Gate test now passes.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Document new endpoint that exposes plugin-contributed settings routes
for dynamic frontend UI extension.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Plugins can now register their settings pages dynamically via get_settings_routes()
hook, eliminating hardcoded imports that break lean builds.
Backend:
- Add get_settings_routes() hook to BasePlugin (shopdb/plugins/base.py)
- Implement in ComputersPlugin for Operating Systems management
- Add /api/plugins/settings-routes endpoint to expose plugin routes
Frontend:
- ModelCatalogHub fetches plugin routes and dynamically adds them via router.addRoute()
- Tabs render dynamically from pluginTabs ref
- Components lazy-load when clicked
Benefits:
- Lean builds work without manual route exclusions
- Plugins self-contain their UI extensions
- No core code changes needed when adding plugin settings pages
- Follows plugin architecture best practices
Fixes TODO from commit a03fd3de4f
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Use git ls-files instead of grep -r for ~5x speedup
- Only scans tracked files, respects .gitignore automatically
- Add USE_GIT_LS_FILES=1 flag (default enabled)
- Filter out *.standalone.js and *.min.js from checks
- Fallback to grep -r for non-git directories
Before: ~5-10 seconds scanning all files on disk
After: ~1-2 seconds scanning only tracked files
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Replace all unicode (emojis, checkmarks, arrows, em-dashes) with ASCII in .claude/ docs
- Comment out OperatingSystemsList route in core.js - breaks lean builds without computers plugin
- Comment out Operating Systems tab in ModelCatalogHub
- TODO: Plugins should dynamically register their settings routes
Fixes#2 CI check failures on wj branch
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Security fix: .claude/mcp-servers.json contained plain-text passwords:
- MySQL dev password: shopdb-dev-password
- VM SSH password: vos313 (SupportUser and shopfloor accounts)
These credentials are now in GitHub history and should be rotated.
Changes:
- Add .claude/settings*.json and mcp-servers.json to .gitignore
- Remove these files from git tracking (files remain locally)
- Keep shared team resources: skills/, plans/, setup/, maintenance/
These are personal/machine-specific configuration:
- settings.json: Personal hooks (stop-dev.cmd), preferences
- mcp-servers.json: Local paths, database credentials, SSH passwords
Each developer should configure their own .claude/ settings locally.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Brings docker-compose.prebuilt.yml in line with the migrate service
refactor from wilm (commit 58a3670529). Migrations, plugin bootstrap,
and seeds now run in a dedicated service with restart: "no" so failures
block deployment instead of crash-looping.
Changes:
- Add dedicated migrate service (runs db upgrade, bootstrap, seeds)
- API now depends on migrate: service_completed_successfully
- Database bound to 127.0.0.1:3306 (not 0.0.0.0) for security
- UTF-8 charset enforced server-wide
- Healthcheck uses TCP not socket (prevents false-healthy during init)
Testing:
- Verified migrations are idempotent (safe to re-run)
- Verified failed migration blocks API startup (fail-closed)
- Verified startup sequence: db -> migrate -> api (44s)
- Database at schema version 7d51_endpoint_logcollections (head)
Aligns dev environment with pull/airgap deployment pattern.
Endpoints plugin now respects the site_timezone setting from /settings.
All timestamps in endpoint details, script runs, and log collections are
now converted from UTC to the configured site timezone for display.
Changes:
- Added _get_site_timezone() helper to read site_timezone setting
- Added _localize_timestamp() to convert UTC datetimes to site timezone
- Applied localization to GET /api/endpoints/<id> (last_seen, created_at)
- Applied localization to GET /api/endpoints/<id>/scriptruns (execution times)
- Applied localization to GET /api/endpoints/<id>/logcollections (collection times)
This aligns endpoints plugin with notifications and backups plugins which
already use site_timezone for their time-sensitive operations.
Changed FleetAlertStore to save acknowledgements in:
ProgramData/ShopDB/FleetAlerts/acknowledged-alerts.json
Instead of:
ProgramData/ShopDbAgent/acknowledged-alerts.json
Now matches the standard folder structure used by other agent components.
The dev stack ran schema work from the api entrypoint while the pull and
airgap stacks ran it from a dedicated migrate service. Same repo, two
different init paths, and the dev one was missing a step.
That gap was real, not theoretical: the first run of this service on a
stack that had been up for days created 13 default settings. The entrypoint
only ran `db upgrade` and `plugin upgrade-all`, never the seeds, so any
patch adding a permission or a setting reached pull/airgap sites and
silently skipped dev. endpoints.agentupdate was exactly that shape of
change.
Running init from the api entrypoint has two other problems. Under
`restart: unless-stopped` a failing migration crash-loops, re-attempting
partially-applied schema work and burying the original error under retry
noise; the migrate service uses `restart: "no"` so a failure exits once and
stays readable. And `plugin upgrade-all` was tolerated as a warning, so the
app would start with plugin tables missing - the shape of the
endpointlogcollections bug. Chained with && it now stops the rollout
instead.
No fallback path is kept in the entrypoint. Compose honours
service_completed_successfully even when a single service is named, so
`up -d api` still runs migrate to completion first - verified, api stays in
Created until migrate exits 0. A skip-flag would have meant the safe
behaviour depended on remembering to set a variable, and deployment here
goes through an install script that invokes compose, so there is no second
caller to protect. The entrypoint keeps only the database wait, which is
about this container's readiness to serve rather than schema state.
The seeds are all create-if-missing, so running them on every start is a
no-op once current: the second run reported 0 permissions, 0 settings and
scopes already up to date. Ordering in the chain is load-bearing - plugin
tables cannot be created before the plugins are installed, and seedscopes
only links permissions that already exist as rows, so seeds run last.
migrate carries the full app environment including CORS_ORIGINS even though
it never serves a request, because create_app() validates config before any
CLI command dispatches and the production config refuses to build without an
explicit allowlist. It mounts instance_data because bootstrap-plugins.sh
writes plugins.json there; it and api never run concurrently, so the shared
mount is not contended.
The Dockerfile's standalone `docker run` note now states that the image does
not migrate itself and lists the sequence to run first.
