ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make main the lean product a site receives

Browse Source 
main was 16 days and 259 commits behind the line the product is actually built
on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It
was neither current nor curated - just stale. This brings it forward and, at the
same time, makes it mean something: main is what a receiving site installs, and
nothing else.

WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides,
knowledgebase, machines, network, notifications, printers, slides, tools,
warranty - plus applications, which is manifest-less core and always ships. The
only manifest dependency inside that set is endpoints on computers, and both are
in it, so the closure is self-contained.

WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb
stay on the site line. So do the Windows installer tree, the VM rehearsal
helpers, the agent tooling config, one site's data loader, the internal audit
and findings records, and the planner spreadsheet. 1,135 tracked files against
1,438.

A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped:

deploy/site-profile-universal.json still named all seventeen plugins. It is the
file docs/DEPLOY.md tells a site to provision from, so a fresh install would
have failed at `flask plugin apply-profile` with PluginError - the first thing a
receiving site does, broken by the branch meant to serve them. It now names the
twelve that exist, and its comment says the two must agree.

agent/ was never refreshed. Whoever populated this branch copied shopdb/,
plugins/, frontend/ and tests/ from the site line and left the agent tree at its
September state: 22 C# and PowerShell sources missing, including AgentCredentials,
EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the
site line had deleted still present. The old tree is internally consistent, so it
compiles, and nothing in CI builds it - every gate was green over a server that
speaks enrollment and per-PC credentials to a bundled agent source that cannot.
Resynced whole.

DECISIONS THIS ENCODES, rather than defers:

- The plugin-authoring lab builds printedparts, whose manifest depends on
  employees, so it could not stay. The lab remains on the site line and main
  ships no build-along tutorial. That is a real loss for an adopting developer
  and is recorded here rather than discovered later.
- Recognition, Training and Recertification notifications are no longer seeded.
  Each splits one notification into a card per employee and renders a photo,
  which needs a plugin this branch does not bundle. Seeding them would offer an
  author a type that cannot resolve anybody. A site that installs employees gets
  them from that plugin's own seed.
- Windows/IIS is documented as the legacy path throughout, and the README leads
  with Docker, because that is what a receiving site gets.

The API description is regenerated for THIS branch - 347 paths and 483
operations, down from 415 and 573 - so the published contract describes what is
here rather than what the site line serves. docs/CONFIG.md lost the two
measuring-tool label keys the code no longer declares, which two lanes had
decided opposite ways.

Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run
rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
cproudlock
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.dockerignore
View File

@@ -17,7 +17,3 @@ venv
.gitignore
*.log
.env.local
# Installer build artifacts (not part of the app image)
installer-arch-dev/build
installer-arch-dev/dist

22
.dockerignore.prebuilt Normal file
View File

@@ -0,0 +1,22 @@
# Dockerignore for pre-built frontend approach
# frontend/dist is INCLUDED (not ignored) for this build
**/node_modules
frontend/src/.plugins-staged
frontend/src/router/routes.gen.js
# Python caches
**/__pycache__
**/*.pyc
.venv
venv
# Local dev and VCS
.git
.gitignore
*.log
.env.local
# Installer artifacts
installer-arch-dev/build
installer-arch-dev/dist

61
.env.example
View File

@@ -4,7 +4,11 @@
# refuses to boot when SECRET_KEY, JWT_SECRET_KEY, DATABASE_URL, or
# CORS_ORIGINS are missing or use the dev defaults.
#
# See the DEPLOY wiki page for the full per-site deployment runbook.
# This template covers what a container deploy sets. Docker is the shipping
# path for a receiving site: docker-compose.yml builds from source,
# docker-compose.pull.yml runs a published image, and
# docker-compose.airgap.yml runs an image loaded from a tarball. See the
# DEPLOY wiki page for the full per-site runbook.
# ---- Flask ----
FLASK_APP=wsgi.py
@@ -37,11 +41,12 @@ DATABASE_URL=mysql+pymysql://shopdb:CHANGE_ME@db:3306/shopdb_flask?charset=utf8m
CORS_ORIGINS=http://localhost:5173
# ---- Reverse proxy (optional, default 0 = no proxy) ----
# Set to the number of proxies actually in front of this instance - 1 when IIS
# or nginx serves it under a subpath such as /shopdb. Without it the app sees
# the proxy's connection instead of the browser's (scheme http, the proxy's
# host and IP), so any absolute URL it builds is wrong at the browser. That is
# what made a redirect on the endpoints list unfollowable behind IIS.
# Set to the number of proxies actually in front of this instance - 1 when
# nginx or another front end terminates TLS for the container. Without it the
# app sees the proxy's connection instead of the browser's (scheme http, the
# proxy's host and IP), so any absolute URL it builds is wrong at the browser.
# That is what made a redirect on the endpoints list unfollowable behind a
# proxy.
#
# Leave it at 0 when nothing fronts the app. It makes X-Forwarded-* trusted,
# and a client can send those headers itself - the login rate limit and the
@@ -56,15 +61,22 @@ JWT_REFRESH_TOKEN_EXPIRES=2592000
LOG_LEVEL=INFO
# ---- docker-compose only ----
# These are read by docker-compose.yml; not used by the Flask app directly.
# These are read by the compose files; not used by the Flask app directly.
MYSQL_ROOT_PASSWORD=CHANGE_ME_ROOT_PASSWORD
MYSQL_PASSWORD=CHANGE_ME_APP_PASSWORD
MYSQL_PORT=3306
API_PORT=5001
# Air-gapped deploy only (docker-compose.airgap.yml): the loaded image tag,
# which MUST match what build-offline-bundle.ps1 -Version produced. Ignored by
# the connected build template (docker-compose.yml builds from source).
# the connected build template (docker-compose.yml builds from source) and by
# the pull template (which uses SHOPDB_IMAGE instead).
IMAGE_TAG=0.7.0
# Pull deploy only (docker-compose.pull.yml): the full image reference to pull.
# SHOPDB_IMAGE=ghcr.io/<org>/shopdb-flask:stable
#
# Non-empty lights a channel badge in the sidebar. Leave unset on a real site;
# the dev stacks set it to DEV.
# SHOPDB_ENV_LABEL=
# ---- Zabbix integration (optional, for printer supply monitoring) ----
ZABBIX_URL=
@@ -89,36 +101,25 @@ ZABBIX_TOKEN=
# (flask plugin stamp-bundled) and pinning keys above.
# PLUGIN_REQUIRE_SIGNED=false
#
# Dev-only: directories whose UNSIGNED plugins are trusted, honored ONLY under
# DEBUG/TESTING (external-repo/symlink dev). Production ignores this.
# PLUGIN_DEV_TRUST_DIRS=/home/dev/my-plugin-repo
#
# Read-only folder the app pulls plugin artifacts from (a SharePoint-synced or
# copied shelf). The app reads this folder; it never speaks any network.
# PLUGIN_SHELF_DIR=/srv/shopdb/plugin-shelf
# ---- Employee directory database (optional, read-only) ----
# Separate HR/employee lookup DB consumed by the notifications plugin and the
# public shopfloor kiosks. Leave unset if the feature is not used; there is no
# safe default for the password, so an unset password fails loud.
# Separate HR/employee lookup DB. Core reads it in two places: the employee
# domain of global search (shopdb/core/api/search.py) and audit-log name
# resolution (shopdb/core/api/auditlogs.py). Leave unset if the site has no
# such database; there is no safe default for the password, so an unset
# password fails loud rather than connecting as someone else.
# EMPLOYEE_DB_HOST=
# EMPLOYEE_DB_USER=
# EMPLOYEE_DB_PASSWORD=
# EMPLOYEE_DB_NAME=wjf_employees
# ---- cmmc_usb database (USB check-in/out) ----
# Separate read-write MySQL DB used by the USB plugin to track device
# check-in/out, lockers, and the check-in/out log. Leave unset if the feature
# is not used; there is no safe default for the password, so an unset password
# fails loud.
# CMMC_USB_DB_HOST=
# CMMC_USB_DB_USER=
# CMMC_USB_DB_PASSWORD=
# CMMC_USB_DB_NAME=cmmc_usb
# EMPLOYEE_DB_NAME=employees
# ---- Subpath deployment (optional) ----
# Serve the app under a URL prefix instead of the server root, e.g. as an IIS
# Application at /ops under an existing site. The frontend must be rebuilt with
# the matching base: VITE_BASE_PATH=/ops/ npm run build. Leave unset when the
# app owns its own site/port (the default). See the INSTALL-WINDOWS-IIS wiki page.
# Serve the app under a URL prefix instead of the server root, e.g. behind an
# nginx location at /ops. wsgi.py reads this to mount the WSGI app under the
# prefix. The frontend must be rebuilt with the matching base:
# VITE_BASE_PATH=/ops/ npm run build. Leave unset when the app owns its own
# host/port, which is the container default.
# MOUNT_PATH=/ops

6
.gitattributes vendored
View File

@@ -4,9 +4,3 @@
*.sh text eol=lf
scripts/check-naming-and-style.sh text eol=lf
.githooks/pre-commit text eol=lf
# Batch files must be CRLF. cmd.exe parses a lone LF inconsistently around
# labels and goto, so an LF-only .cmd can run its happy path and then jump
# nowhere on the error path - which is the path that matters in a promote
# script.
*.cmd text eol=crlf

57
.github/workflows/ci.yml vendored
View File

@@ -88,9 +88,31 @@ jobs:
- run: npm run build
lean-build:
# ADR-013 Phase 5: prove a per-site build carries only its chosen plugins.
# Builds a lean site (machines + printers) and asserts an omitted plugin's
# code is absent from the bundle - the delete-a-plugin guarantee in CI.
# ADR-013: prove a per-site build carries only its chosen plugins.
#
# The split below is SYNTHETIC, and deliberately so. Every plugin in this
# repository ships on this line, so there is no "omitted" set to borrow from
# a real site - and a job that iterated an empty omitted list would assert
# nothing while staying green, which is the exact failure the omission
# guarantee exists to catch. So the profile names a genuine SUBSET of the
# bundled plugins and the assertions prove the SUBSETTING MECHANISM: what
# the profile chooses is staged, and what it leaves out is not.
#
# Chosen (7): backups computers endpoints machines network notifications
# printers
# Omitted (5): guides knowledgebase slides tools warranty
#
# The omitted five are picked because nothing depends on them, so the
# closure stays exactly the chosen set. 'applications' is manifest-less core
# and ships in every build (ADR-013). endpoints hard-depends on computers,
# which is already chosen, so the closure adds nothing - if that changes,
# resolve_plugin_closure.py pulls the dependency in and the absence
# assertions below start failing, which is the signal to update this list
# rather than to loosen the check.
#
# Verified by planting a leak, per the standing rule that a green gate may
# be vacuous: adding 'guides' to the profile and rebuilding makes both the
# GuidesList assertion and the plugins/guides directory assertion fail.
runs-on: arc-runner-set
steps:
- uses: actions/checkout@v5
@@ -100,21 +122,29 @@ jobs:
cache: npm
cache-dependency-path: frontend/package-lock.json
- run: cd frontend && npm ci
- name: Build lean site (machines + printers)
- name: Build lean site (seven-plugin subset)
run: |
printf '{ "site": "ci-lean", "plugins": ["machines", "printers"] }' \
printf '%s' '{ "site": "ci-lean", "plugins": ["backups", "computers", "endpoints", "machines", "network", "notifications", "printers"] }' \
> /tmp/lean-profile.json
bash scripts/build-site.sh /tmp/lean-profile.json /tmp/leansite
- name: Assert omitted plugin code is absent, chosen present
run: |
assets=/tmp/leansite/frontend-dist/assets
for code in PartsKiosk ManifestEditor USBLabelBatch KnowledgeBaseDetail; do
# One component per OMITTED plugin: guides, knowledgebase, slides,
# tools, warranty. Each name is proven to exist by the chosen-plugin
# loop on a full build, so these are known-good grep targets rather
# than strings that would pass by being misspelled.
for code in GuidesList KnowledgeBaseDetail SlideManager ToolsIndex \
WarrantiesList; do
if grep -rqoh "$code" "$assets"/*.js; then
echo "FAIL: omitted-plugin code '$code' leaked into the lean bundle"
exit 1
fi
done
for code in MachineDetail PrinterDetail; do
# One component per CHOSEN plugin, so a plugin silently dropped from
# the bundle fails here instead of on the floor.
for code in BackupHistory PCsList EndpointsList MachineDetail \
NetworkDevicesList NotificationsList PrinterDetail; do
grep -rqoh "$code" "$assets"/*.js || {
echo "FAIL: chosen-plugin code '$code' missing from the lean bundle"
exit 1; }
@@ -124,8 +154,17 @@ jobs:
grep -rqoh "ApplicationsList" "$assets"/*.js || {
echo "FAIL: core page 'ApplicationsList' missing from the lean bundle"
exit 1; }
test -d /tmp/leansite/plugins/machines
test ! -d /tmp/leansite/plugins/printedparts
for plugin in backups computers endpoints machines network \
notifications printers; do
test -d "/tmp/leansite/plugins/$plugin" || {
echo "FAIL: chosen plugin '$plugin' missing from the backend tree"
exit 1; }
done
for plugin in guides knowledgebase slides tools warranty; do
test ! -d "/tmp/leansite/plugins/$plugin" || {
echo "FAIL: omitted plugin '$plugin' staged into the backend tree"
exit 1; }
done
echo "lean build verified: only chosen plugins present"
# NOTE: the MySQL-8 migration/seed job (fresh `flask db upgrade` + every

32
.github/workflows/scan-darkwing-portable.yml vendored
View File

@@ -1,32 +0,0 @@
# Darkwing vulnerability scan, manual dispatch.
#
# Copied verbatim from the ge-aero/darkwing README so it stays byte-identical
# to the sanctioned integration. Do not edit this file to add triggers or
# gating - see scan-darkwing-pr.yml for that, which is ours to change.
#
# The file must sit on the default branch before a scan can run on ANY branch.
name: Scan (darkwing, portable)
on:
workflow_dispatch:
inputs:
target_ref:
description: Ref, branch, or sha of this repo to scan
required: false
type: string
default: main
permissions:
contents: read # checkout this repo via the ambient GITHUB_TOKEN
security-events: write # SARIF upload to the Security tab
actions: read # required by upload-sarif on private/internal repos
jobs:
scan:
runs-on: darkwing-scan
steps:
# @v1 is a moving tag pointing to the latest release that is compatible
# with this workflow file.
- uses: ge-aero/darkwing@v1
with:
target_ref: ${{ inputs.target_ref || 'main' }}

62
.github/workflows/scan-darkwing-pr.yml vendored
View File

@@ -1,62 +0,0 @@
# Darkwing scan on pull requests, so the "check against darkwing" is a check
# and not a thing somebody remembers to click.
#
# WHY THIS IS SEPARATE from scan-darkwing-portable.yml: that file is copied
# byte-for-byte from the ge-aero/darkwing README and is left alone so it keeps
# matching the sanctioned integration. Everything below is ours.
#
# THE TRAP THIS AVOIDS. The README workflow resolves its scan target as
# `inputs.target_ref || 'main'`. On a pull_request event there are no inputs,
# so that expression falls through to 'main' and the job scans the default
# branch instead of the code under review - a green Darkwing check that never
# looked at the change. Resolve the head SHA explicitly.
#
# Darkwing exits 0 whatever it finds (its own FAQ: "Does a finding fail my
# build? No."), so a passing job means the scan RAN, not that the code is
# clean. Findings land in Security -> Code scanning. Turning a finding into a
# hard failure means reading findings.sarif ourselves; the commented step at
# the bottom is the shape of that, deliberately left off until we have seen the
# false-positive rate on this repo.
name: Scan (darkwing, PR)
on:
pull_request:
permissions:
contents: read
security-events: write
actions: read
# One scan per PR. A new push supersedes the old one rather than queueing a
# second Bedrock spend against a diff nobody is reviewing any more.
concurrency:
group: darkwing-pr-${{ github.ref }}
cancel-in-progress: true
jobs:
scan:
runs-on: darkwing-scan
steps:
- uses: ge-aero/darkwing@v1
with:
# The PR's head, NOT the default branch. See the note above.
target_ref: ${{ github.event.pull_request.head.sha }}
# Gating, once the signal is trusted. Darkwing writes findings.sarif into
# its work dir and attaches it as the `findings` artifact; this reads the
# severities out and fails on error-level results only.
#
# - name: Fail on error-level findings
# run: |
# python - <<'PY'
# import json, pathlib, sys
# sarif = pathlib.Path('darkwing/findings.sarif')
# if not sarif.is_file():
# sys.exit('no findings.sarif: the scan did not produce results')
# doc = json.loads(sarif.read_text())
# bad = [r for run in doc.get('runs', [])
# for r in run.get('results', [])
# if r.get('level') == 'error']
# if bad:
# sys.exit(f'{len(bad)} error-level finding(s); see the Security tab')
# PY

1
.gitignore vendored
View File

@@ -83,7 +83,6 @@ secrets.yml
*_secret
*_secrets
credentials.json
scripts/site_imports/wjf/idmap.json
# work-PC publication clone: bundle drop folder for the transfer pipeline
_transfer/

20
Dockerfile
View File

@@ -1,13 +1,19 @@
# shopdb-flask single-site container.
#
# One image, one site. Per ADR-004, each adopting facility runs its own
# stack with its own DB, secrets, and enabled-plugin list. This image
# bundles all 13 catalog plugins (computers, employees, geenforce,
# knowledgebase, machines, measuringtools, network, notifications,
# printedparts, printers, slides, usb, warranty); a site installs + enables
# the ones it wants with `flask plugin install <name>` (or, declaratively,
# `flask plugin apply-profile <profile.json>`). Per ADR-013 a future lean
# build stages only the chosen plugin directories into this image.
# stack with its own DB, secrets, and enabled-plugin list. This image bundles
# the whole catalog - backups, computers, endpoints, guides, knowledgebase,
# machines, network, notifications, printers, slides, tools, warranty - and a
# site installs + enables the ones it wants with `flask plugin install <name>`
# (or, declaratively, `flask plugin apply-profile <profile.json>`).
#
# WHY no count here: a hand-typed number goes stale the moment a plugin is
# added or dropped, and nothing in the test suite polices a count outside
# docs/. The generated PROJECT-MAP wiki page is the authority on what ships.
#
# Per ADR-013 a lean build stages only the chosen plugin directories into this
# image; scripts/build-site.sh plus scripts/resolve_plugin_closure.py are that
# mechanism.
#
# The frontend is built in a first stage and its dist output is copied into
# the final image so Flask can serve the SPA (register_frontend_routes in

47
Dockerfile.prebuilt Normal file
View File

@@ -0,0 +1,47 @@
# Dockerfile for pre-built frontend
# Use this when frontend/dist is already built on the host
# Avoids npm issues with corporate proxy
FROM python:3.14-slim AS base
WORKDIR /app
# Install system dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential \
default-libmysqlclient-dev \
pkg-config \
curl \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir gunicorn gevent
# Copy application code
COPY shopdb/ ./shopdb/
COPY plugins/ ./plugins/
COPY migrations/ ./migrations/
COPY scripts/ ./scripts/
COPY wsgi.py ./
# Copy pre-built frontend
COPY frontend/dist ./frontend/dist
# Make scripts executable
RUN chmod +x /app/scripts/docker-entrypoint.sh /app/scripts/bootstrap-plugins.sh
# Create instance directory
RUN mkdir -p /app/instance
# Expose port
EXPOSE 5001
# Set environment
ENV PYTHONUNBUFFERED=1
ENV FLASK_APP=shopdb
# Entrypoint runs migrations and starts gunicorn
ENTRYPOINT ["/app/scripts/docker-entrypoint.sh"]

78
README.md
View File

@@ -9,13 +9,13 @@ ShopDB tracks and manages:
- **PCs** - Shopfloor computers, engineering workstations
- **Printers** - Network printers with Zabbix supply integration
- **Network devices** - Switches, routers, and the subnet browser
- **Measuring tools** - Gage-lab instruments with calibration tracking
- **Applications** - Software deployed across the shop floor, with per-PC install tracking
- **Employees** - Directory, recognition and training notifications
- **Warranties** - Coverage records with Dell warranty lookups
- **USB devices** - CMMC check-in/out tracking
- **Knowledge Base** - Documentation and troubleshooting guides
- **GE-Enforce manifests** - Imaging/software manifest editing and fleet compliance
- **Endpoints** - RMM telemetry reported by the ShopDB agent: metrics, services, inventory, Windows events
- **Backups** - Per-asset configuration backups with revision history and diffs
- **Warranties** - Coverage records with vendor warranty lookups
- **Knowledge Base and Guides** - Reference articles and step-by-step procedures with working checklists
- **Notifications** - Shopfloor announcements and recognition feed
- **Slides** - Lobby display and shopfloor screensaver slideshows
## Tech Stack
@@ -108,10 +108,11 @@ SQLite). Do not run dev or production against SQLite.
### Distribution
Source lives here. The Windows installer ships as a release asset rather than a
file in the repository - it is around 240 MB, well past what a repository takes
and well inside what a release asset does. There is no package or image
registry; a site installs from that `.exe` or builds from source.
Source lives here, and a site stands up ShopDB with Docker. Three compose
files cover the three situations: `docker-compose.yml` builds the image from
this checkout, `docker-compose.pull.yml` runs an image published to a
registry, and `docker-compose.airgap.yml` runs an image loaded from a tarball
at a site with no internet.
### Fast path (Docker)
@@ -174,42 +175,19 @@ project wiki.
To import a site's legacy data, use the HTTP import surface: an admin API
token plus the IMPORT-API page in the project wiki drive the whole migration
through documented endpoints (`X-Import-Mode` preserves original timestamps).
`scripts/site_imports/wjf/` is the reference site's loader, kept as a worked
example of the whole sequence.
`scripts/site_imports/` holds the shared import machinery - the resolver, the
idempotent journal, the approval packet builder and the WINC machine-adoption
writer that the core import routes call.
### Which deployment route
| Target | Use |
|---|---|
| **Windows Server + IIS** (how sister sites run) | **the INSTALL-WINDOWS page in the project wiki** - one installer `.exe`, offline, no manual IIS work. Day 2: the OPERATE-WINDOWS page in the project wiki |
| Linux / Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki |
| Linux / Docker, connected | the DEPLOY page in the project wiki |
| Docker, connected | the DEPLOY page in the project wiki (`docker-compose.yml` or `docker-compose.pull.yml`) |
| Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki (`docker-compose.airgap.yml`) |
For every environment variable and Setting key see the CONFIG page in the project wiki.
### What the Windows installer requires
The installer checks all of this before it changes anything and refuses rather
than half-installing.
| | Minimum | Notes |
|---|---|---|
| Windows Server | **2019** (build 10.0.17763) | Ships **IIS 10.0**, which is therefore the earliest IIS supported. Server 2016 is also IIS 10.0 but falls below the build floor and is refused. |
| Windows client (test boxes) | **10 22H2** (build 10.0.19045) or 11 | **Pro or higher** - Home has no IIS at all. |
| Architecture | 64-bit | The payload is cp314 win_amd64. |
| IIS Web Server role | installed beforehand | `Install-WindowsFeature -Name Web-Server -IncludeManagementTools`. Needs no internet. |
| Free disk | 5 GB | Refused below this. 40 GB is comfortable once uploads and config backups accumulate. |
| Database | bundled MySQL 8.4 LTS, or your own | Bring credentials if you use an existing server. |
The binding constraint is the operating system rather than IIS: the payload is
Python 3.14 plus HttpPlatformHandler and URL Rewrite, and the handler itself runs
on older IIS. An older host is untested rather than known-broken.
If you must deploy onto something older, the INSTALL-WINDOWS-IIS page in the project wiki
is the manual procedure and has no OS gate - it even documents MySQL 5.6, which is
that era of machine. The trade is stated there: it produces a server the
installer will not subsequently upgrade.
## Configuration
Environment variables (`.env`):
@@ -235,7 +213,7 @@ The REST API follows standard conventions:
| DELETE | `/api/machines/:id` | Soft delete machine |
Each asset plugin exposes the same CRUD pattern on its own prefix
(`/api/computers`, `/api/printers`, `/api/network`, `/api/measuringtools`),
(`/api/computers`, `/api/printers`, `/api/network`, `/api/machines`),
and cross-cutting asset endpoints live under `/api/assets`.
Query parameters for list endpoints:
@@ -244,7 +222,7 @@ Query parameters for list endpoints:
- `sort` - Sort field
- `dir` - Sort direction (asc/desc)
- `search` - Search term
- `assettype` - Filter by asset type (computer, printer, machine, networkdevice, measuringtool)
- `assettype` - Filter by asset type (computer, printer, machine, networkdevice)
## Where to start
@@ -261,21 +239,21 @@ The image bundles these plugins; only the ones a site installs are loaded. The
current count and each plugin's version and migration head are in
the PROJECT-MAP page in the project wiki, which is generated:
- **backups** - Machine configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
- **backups** - Per-asset configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
- **computers** - Shopfloor PCs and workstations, collector fleet ingest
- **employees** - Employee directory
- **geenforce** - GE-Enforce imaging/software manifests and fleet compliance
- **endpoints** - RMM telemetry from the ShopDB agent: metrics, services, inventory, Windows events
- **guides** - Step-by-step procedures with images, working checklists and printable output
- **knowledgebase** - Documentation and troubleshooting articles
- **machines** - CNC, CMM, and other shop-floor machines
- **measuringtools** - Gage-lab instruments with calibration tracking
- **knowledgebase** - Documentation and troubleshooting guides
- **network** - Network devices and subnets
- **notifications** - Shopfloor notifications and recognition feed
- **printedparts** - 3D-printed part catalogue, kiosk issue tracking and stock alerts
- **printers** - Extended printer management with Zabbix integration
- **slides** - TV/kiosk slideshows
- **tools** - Tech Tools: barcode and QR generation laid out for real label stock
- **usb** - CMMC USB check-in/out tracking
- **warranty** - Warranty records with Dell lookups
- **slides** - Lobby display and shopfloor screensaver slideshows
- **tools** - Tech Tools: barcode/QR label generation and one-view secure send
- **warranty** - Warranty records with derived coverage status
Applications is not in that list because it is not a plugin: it is core, has no
manifest, and always ships.
## Legacy Migration

25
agent/README.md
View File

@@ -84,12 +84,23 @@ event channel and system WMI.
## Tray app (system-tray icon)
The tray is a per-user process, not a service, so it starts at sign-in rather
than under `sc.exe`. `install-shopdbagent.ps1` handles it: it copies
`shopdb-endpoint-tray.exe` next to the agent, registers an HKCU logon autostart
(`Software\Microsoft\Windows\CurrentVersion\Run\ShopDbEndpointTray`), and
launches it for the current user. Run the installer from the operator's own
session so the autostart lands in the right user hive. Pass `-NoTray` for a
headless, service-only install.
than under `sc.exe`. The generated `install.ps1` handles it: it copies
`shopdb-endpoint-tray.exe` next to the agent and registers a machine-wide logon
autostart (`HKLM\Software\Microsoft\Windows\CurrentVersion\Run\ShopDbEndpointTray`,
64-bit view). The installer deliberately does NOT start the tray itself: it runs
elevated, so any process it starts inherits an admin token, and mapped drives are
per logon token - an elevated tray probes the wrong drive table. The Run key only
fires at logon, which would leave the tray dead until sign-out on a machine
someone is already using, so the installer also registers a transient
`ShopDB tray first launch` scheduled task under `S-1-5-32-545` at `RunLevel
Limited`, runs it once, and unregisters it. That task is the only mechanism that
can reach into an existing interactive session without carrying the installer's
token. Pass `-NoTray` for a headless, service-only install.
User-context collection does not depend on the tray. The installer registers a
`ShopDB user-context probe` scheduled task against the Users group at
`RunLevel Limited`, which runs `shopdbagent.exe --collect-usercontext` in the
operator's own session at logon and every 30 minutes.
Right-click the tray icon for:
@@ -143,7 +154,7 @@ logic itself changes.
## Diagnostics
- Rolling file log: `%ProgramData%\ShopDB\EndpointAgent\logs\agent-YYYYMMDD.log`
- Rolling file log: `%ProgramData%\ShopDB\EndpointAgent\logs\YYYYMMDD-agent.log`
(14-day retention).
- Windows Event Log: agent runtime failures and unhandled exceptions are written
to the `Application` log under source `ShopDB Endpoint Agent`. The source is

106
agent/ShopDbEndpointAgent.Core/AgentConfig.cs
View File

@@ -13,10 +13,40 @@ public sealed class AgentConfig
[JsonPropertyName("apikey")] public string ApiKey { get; set; } = "";
[JsonPropertyName("cadenceseconds")] public int CadenceSeconds { get; set; } = 900;
// Legacy pin: SHA-1 thumbprint of the server leaf certificate (hex, no
// separators). Honoured with serverpins only while the registry ServerPins
// value is absent; breaks on every cert renewal, so prefer an SPKI pin.
// See AgentHttpFactory.
[JsonPropertyName("servercertthumbprint")] public string? ServerCertThumbprint { get; set; }
// Server public key pins: lowercase hex SHA-256 of the leaf certificate's
// SubjectPublicKeyInfo. Fallback only: HKLM\SOFTWARE\GE\ShopDB ServerPins
// (REG_MULTI_SZ), when it exists in either view, replaces these entirely,
// even when empty. Any pin in effect = the leaf key must match one; chain
// and name errors are then ignored. None = normal Windows chain validation.
[JsonPropertyName("serverpins")] public List<string> ServerPins { get; set; } = new();
// Jitter spreads a fleet's reports so they do not arrive in lockstep
// (thundering herd). Actual delay = cadence +/- up to jitterseconds.
[JsonPropertyName("jitterseconds")] public int JitterSeconds { get; set; } = 60;
// One-time random delay before FIRST contact, applied by both the telemetry
// and command loops at startup. JitterSeconds only spreads the gap BETWEEN
// runs, so without this every agent in the fleet still makes its first
// request the instant its service starts - and the service is installed
// start=auto, so a patch reboot starts them all together.
//
// Measured on a 700-endpoint burst (devnotes/gevent-sizing-simulation.md):
// simultaneous arrival drove TLS handshakes to 1482ms worst case and pinned
// CPU at 305% while the database sat idle at 3 active queries. Spreading the
// same arrivals over 30s cut worst-case handshake to 63.7ms. The server is
// not the constraint; simultaneity is.
//
// Costs up to this many seconds before a rebooted endpoint picks up a queued
// command. That window is on top of a reboot the machine was already
// unreachable for. 0 disables the delay.
[JsonPropertyName("startupjitterseconds")] public int StartupJitterSeconds { get; set; } = 30;
[JsonPropertyName("adoptonfirstrun")] public bool AdoptOnFirstRun { get; set; } = true;
// Directory holding site-provided PowerShell "agent utilities" the runtime
@@ -89,42 +119,52 @@ public sealed class AgentConfig
return bare;
}
// Per-name allowlist of pre-staged scripts for the generic run_script
// action. Exactly like LaunchPaths, the wire never carries a script body or
// path - only a script NAME the agent already holds here. A name absent
// from this map fails closed. There are no shipped defaults: a script is a
// site-specific artifact, so an empty map means run_script does nothing
// until a site stages scripts and lists them in appsettings.json. Keyed by
// script name (case-insensitive, .ps1 optional).
[JsonPropertyName("scriptpaths")]
public Dictionary<string, string> ScriptPaths { get; set; } = new();
// REMOVED DELIBERATELY - DO NOT REINTRODUCE: 'scriptpaths' / ScriptPaths.
//
// This used to be a per-name map of PRE-STAGED scripts on the endpoint
// (typically under C:\ProgramData\ShopDB\EndpointAgent\scripts), which
// run_script would look up by name and execute. It was superseded by
// server-hosted delivery and left behind as dead config: nothing read the
// map, so it was an execution path that appeared supported but was not.
//
// It is gone rather than merely unused because reviving it is a security
// regression, and dead code invites revival. That model had NO integrity
// check - no digest, no provenance - so whoever could write into the staging
// directory chose what the agent ran as SYSTEM. ProgramData is world-
// writable by default unless the tree is explicitly tightened, which made
// the gate "is this name in appsettings.json", not "is this content
// trusted".
//
// The supported path is run_script with a server-issued delivery descriptor:
// the body is fetched from ShopDB over the collector credential and verified
// against a server-pinned SHA-256 before it executes, so the content is
// authenticated and the action is attributable in the command log. If you
// need a script to run on an endpoint, publish it server-side and send it
// through that path. If you need to trigger something already installed on
// the box, use scheduledtask_run, which is far narrower.
// The effective script map: appsettings.json entries only (no defaults).
// Keys are normalized to bare names so a lookup tolerates 'reset' or
// 'reset.ps1'.
public IReadOnlyDictionary<string, string> ResolveScriptPaths()
{
var resolved = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
foreach (var pair in ScriptPaths)
{
if (string.IsNullOrWhiteSpace(pair.Key) || string.IsNullOrWhiteSpace(pair.Value))
continue;
resolved[NormalizeScript(pair.Key)] = pair.Value;
}
return resolved;
}
private static string NormalizeScript(string name)
{
var bare = (name ?? "").Trim();
if (bare.EndsWith(".ps1", StringComparison.OrdinalIgnoreCase))
bare = bare[..^4];
return bare;
}
// Approved scheduled task paths for the scheduledtask_run action. The wire
// carries a free-text task path so the WebUI can expose it as a single
// field; ScheduledTaskPolicy validates it against this list and fails closed
// when it does not match. An entry matches that exact task path, or acts as
// a folder prefix for anything beneath it.
//
// Empty means "use the shipped Corporate Tools defaults"
// (ScheduledTaskPolicy.DefaultApprovedTaskPaths). Setting it REPLACES those
// defaults, so a site can narrow the set as well as widen it without an
// agent rebuild. Keep it tight: every entry here is something an operator
// holding endpoints.deploy can start, in whatever context the task itself
// registered.
[JsonPropertyName("approvedtaskpaths")]
public List<string> ApprovedTaskPaths { get; set; } = new();
public bool IsUsable()
=> !string.IsNullOrWhiteSpace(BaseUrl)
&& !string.IsNullOrWhiteSpace(ApiKey)
=> !string.IsNullOrWhiteSpace(BaseUrl) && HasSharedKey();
// True when a real shared collector key is provisioned (not the shipped
// placeholder). An enrolled agent runs without one.
public bool HasSharedKey()
=> !string.IsNullOrWhiteSpace(ApiKey)
&& ApiKey != "REPLACE_WITH_COLLECTOR_KEY";
// Validates the BaseUrl scheme before any request is made. A plaintext

269
agent/ShopDbEndpointAgent.Core/AgentCredentials.cs Normal file
View File

@@ -0,0 +1,269 @@
using System.Security.Cryptography;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// This PC's own endpoint credential, issued once by POST /api/endpoints/enroll.
// Readable only by SYSTEM and Administrators (DPAPI LocalMachine inside a
// protected directory). The JSON key names are a contract: SYSTEM PowerShell
// (upgrade-shopdbagent.ps1, the reporters) reads the same file with
// [System.Security.Cryptography.ProtectedData]::Unprotect(bytes, $null, 'LocalMachine').
public sealed class AgentCredential
{
[JsonPropertyName("credential")] public string Credential { get; set; } = "";
[JsonPropertyName("assetid")] public int AssetId { get; set; }
[JsonPropertyName("issuedat")] public string IssuedAt { get; set; } = "";
[JsonPropertyName("baseurl")] public string BaseUrl { get; set; } = "";
// First 12 chars, the same prefix the server shows. Never log more.
[JsonIgnore]
public string Prefix => Credential.Length <= 12 ? Credential : Credential[..12];
}
// Why the server refused an agent request. Parsed from the error envelope.
public static class AgentAuthErrorCodes
{
public const string CredentialInvalid = "CREDENTIAL_INVALID";
public const string SharedKeyRefused = "SHARED_KEY_REFUSED";
public const string AssetMismatch = "ASSET_MISMATCH";
}
// Current auth material for outbound requests. Swaps at runtime: enrollment
// sets a credential, a CREDENTIAL_INVALID clears it, neither needs a restart.
public interface ICredentialProvider
{
// Null = not enrolled (or the stored credential belongs to another server).
AgentCredential? Current { get; }
// Shared fleet collector key, null when not provisioned.
string? SharedKey { get; }
// Raised after Current changes (set or cleared).
event EventHandler? Changed;
// Raised when the enrollment service should run now.
event EventHandler? EnrollmentNeeded;
// Called by the auth handler for every 401/403 carrying an auth code.
// sentCredential is the credential the failed request carried, or null
// when it went out on the shared key.
void ReportRejected(string code, string? sentCredential);
}
public sealed class AgentCredentialProvider : ICredentialProvider
{
private readonly object gate = new();
private readonly CredentialStore? store;
private readonly ILogger logger;
private AgentCredential? current;
public AgentCredentialProvider(AgentConfig config, CredentialStore? store, ILogger logger)
{
this.store = store;
this.logger = logger;
SharedKey = config.HasSharedKey() ? config.ApiKey : null;
var loaded = store?.TryLoad();
if (loaded is not null && !SameServer(loaded.BaseUrl, config.BaseUrl))
{
// Never send a credential to a server that did not issue it. Kept on
// disk so pointing BaseUrl back restores it; a new enroll overwrites.
logger.LogWarning(
"Stored credential {Prefix} was issued by {Issuer}, not {BaseUrl}; not using it.",
loaded.Prefix, loaded.BaseUrl, config.BaseUrl);
loaded = null;
}
current = loaded;
if (current is not null)
logger.LogInformation("Using endpoint credential {Prefix} (asset {AssetId}).",
current.Prefix, current.AssetId);
}
public AgentCredential? Current
{
get { lock (gate) return current; }
}
public string? SharedKey { get; }
public event EventHandler? Changed;
public event EventHandler? EnrollmentNeeded;
// Persist then swap. Throws when the store cannot be written: a credential
// held only in memory is lost at restart and the server will not reissue.
public void Set(AgentCredential credential)
{
store?.Save(credential);
lock (gate) current = credential;
logger.LogInformation("Endpoint credential {Prefix} stored (asset {AssetId}).",
credential.Prefix, credential.AssetId);
Changed?.Invoke(this, EventArgs.Empty);
}
public void ReportRejected(string code, string? sentCredential)
{
if (code == AgentAuthErrorCodes.CredentialInvalid && sentCredential is not null)
{
bool cleared = false;
lock (gate)
{
// Only the credential that was refused. A late response for an
// old credential must not wipe a freshly issued one.
if (current is not null && current.Credential == sentCredential)
{
logger.LogWarning(
"Server refused credential {Prefix} (revoked or unknown); deleting it and re-enrolling.",
current.Prefix);
current = null;
cleared = true;
}
}
if (cleared)
{
try
{
store?.Delete();
}
catch (Exception error)
{
logger.LogError(error, "Could not delete the refused credential file.");
}
Changed?.Invoke(this, EventArgs.Empty);
EnrollmentNeeded?.Invoke(this, EventArgs.Empty);
}
}
else if (code == AgentAuthErrorCodes.SharedKeyRefused && sentCredential is null && Current is null)
{
logger.LogWarning("Server refused the shared key for this PC; requesting enrollment.");
EnrollmentNeeded?.Invoke(this, EventArgs.Empty);
}
else if (code == AgentAuthErrorCodes.AssetMismatch)
{
logger.LogError(
"Server reported ASSET_MISMATCH: this PC's credential belongs to a different asset than the request named.");
}
}
private static bool SameServer(string issued, string configured)
=> string.Equals(Normalize(issued), Normalize(configured), StringComparison.OrdinalIgnoreCase);
private static string Normalize(string url) => (url ?? "").Trim().TrimEnd('/');
}
// credential.bin and enroll.json under %ProgramData%\ShopDB\EndpointAgent\credential.
public sealed class CredentialStore
{
public const string DirectoryName = "credential";
public const string CredentialFileName = "credential.bin";
public const string EnrollFileName = "enroll.json";
private const int MaxFileBytes = 64 * 1024;
private static readonly JsonSerializerOptions JsonOptions = new()
{
PropertyNameCaseInsensitive = true,
};
private readonly ILogger logger;
public CredentialStore(ILogger logger)
{
this.logger = logger;
}
private static string StoreDirectory => ProtectedFileSystem.EnsureDirectory(DirectoryName);
public AgentCredential? TryLoad()
{
try
{
var path = Path.Combine(StoreDirectory, CredentialFileName);
if (ProtectedFileSystem.TryGetAttributes(path) is null)
return null;
var blob = ReadAllBytesNoFollow(path);
var json = ProtectedData.Unprotect(blob, null, DataProtectionScope.LocalMachine);
var credential = JsonSerializer.Deserialize<AgentCredential>(json, JsonOptions);
if (credential is null || !credential.Credential.StartsWith("sdbe_", StringComparison.Ordinal))
{
logger.LogWarning("Stored credential file is not a usable credential; ignoring it.");
return null;
}
return credential;
}
catch (Exception error)
{
logger.LogError(error, "Could not read the stored endpoint credential; running unenrolled.");
return null;
}
}
public void Save(AgentCredential credential)
{
var dir = StoreDirectory;
var json = JsonSerializer.SerializeToUtf8Bytes(credential);
var blob = ProtectedData.Protect(json, null, DataProtectionScope.LocalMachine);
var target = Path.Combine(dir, CredentialFileName);
var temp = Path.Combine(dir, CredentialFileName + ".new");
using (var file = ProtectedFileSystem.CreateFile(temp))
{
file.Write(blob);
file.Flush(flushToDisk: true);
}
// Moved file keeps its protected DACL.
File.Move(temp, target, overwrite: true);
}
public void Delete()
=> ProtectedFileSystem.DeleteFileNoFollow(Path.Combine(StoreDirectory, CredentialFileName));
// The enroll nonce is generated once and kept for the life of the PC, so
// an approved request can only be claimed by the agent that made it.
public string GetOrCreateEnrollNonce()
{
var path = Path.Combine(StoreDirectory, EnrollFileName);
if (ProtectedFileSystem.TryGetAttributes(path) is not null)
{
try
{
using var doc = JsonDocument.Parse(ReadAllBytesNoFollow(path));
if (doc.RootElement.TryGetProperty("enrollnonce", out var value)
&& value.GetString() is string nonce && nonce.Length >= 32)
return nonce;
logger.LogWarning("enroll.json has no usable nonce; generating a new one.");
}
catch (Exception error) when (error is JsonException or IOException or UnauthorizedAccessException)
{
logger.LogWarning(error, "enroll.json unreadable; generating a new nonce.");
}
}
var fresh = Base64Url(RandomNumberGenerator.GetBytes(32));
var body = JsonSerializer.SerializeToUtf8Bytes(new
{
enrollnonce = fresh,
createdat = DateTime.UtcNow.ToString("o"),
});
using (var file = ProtectedFileSystem.CreateFile(path))
{
file.Write(body);
file.Flush(flushToDisk: true);
}
return fresh;
}
internal static string Base64Url(byte[] bytes)
=> Convert.ToBase64String(bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_');
private static byte[] ReadAllBytesNoFollow(string path)
{
using var stream = ProtectedFileSystem.OpenReadNoFollow(path);
if (stream.Length > MaxFileBytes)
throw new InvalidDataException($"{path} is larger than expected");
using var buffer = new MemoryStream();
stream.CopyTo(buffer);
return buffer.ToArray();
}
}

387
agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs Normal file
View File

@@ -0,0 +1,387 @@
using System.Net;
using System.Net.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Microsoft.Win32;
namespace ShopDbEndpointAgent.Core;
// What an HttpClient is for. Decides pooling, timeout and auth.
public enum AgentHttpChannel
{
Collector,
Command,
FleetAlert,
Download,
Enroll,
}
// The ONLY place the agent builds an HttpClient or a handler. Every channel gets
// the same server certificate check (ServerPinSet) and the same per-request auth
// (AgentAuthHandler), so a new channel cannot forget either.
public sealed class AgentHttpFactory
{
private readonly AgentConfig config;
private readonly ICredentialProvider credentials;
private readonly ILogger logger;
private readonly ServerPinSet pins;
public AgentHttpFactory(AgentConfig config, ICredentialProvider credentials, ILogger logger)
{
this.config = config;
this.credentials = credentials;
this.logger = logger;
pins = ServerPinSet.Load(config, logger);
}
public ICredentialProvider Credentials => credentials;
// Caller owns and disposes the client (it disposes the handler chain).
public HttpClient CreateClient(AgentHttpChannel channel, TimeSpan? timeout = null)
{
// Lifetime cap: pooled TLS sessions were validated against the pins of
// their handshake, so recycle them for a pin change to reach new ones.
var sockets = new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(2),
SslOptions = new SslClientAuthenticationOptions
{
RemoteCertificateValidationCallback = (_, cert, _, errors)
=> pins.Validate(cert, errors, channel),
},
};
switch (channel)
{
case AgentHttpChannel.Command:
// Long-poll: retire pooled TLS connections faster than the
// server keep-alive, or a reused dead socket yields "unexpected
// EOF or 0 bytes from the transport stream" (1.6.2).
sockets.PooledConnectionIdleTimeout = TimeSpan.FromSeconds(1);
break;
case AgentHttpChannel.FleetAlert:
sockets.PooledConnectionIdleTimeout = TimeSpan.FromSeconds(90);
break;
}
var auth = new AgentAuthHandler(credentials, logger,
sharedKeyOnly: channel == AgentHttpChannel.Enroll)
{
InnerHandler = sockets,
};
return new HttpClient(auth, disposeHandler: true)
{
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
Timeout = timeout ?? DefaultTimeout(channel),
};
}
private static TimeSpan DefaultTimeout(AgentHttpChannel channel) => channel switch
{
// Claim is held open server-side (~25s); clear that plus slack.
AgentHttpChannel.Command => TimeSpan.FromSeconds(60),
AgentHttpChannel.Download => TimeSpan.FromMinutes(10),
_ => TimeSpan.FromSeconds(30),
};
}
// Adds exactly one auth header per request, read at send time so a credential
// issued or revoked mid-run takes effect on the next request. Watches 401/403
// replies for the auth error codes and reports them to the provider.
public sealed class AgentAuthHandler : DelegatingHandler
{
public const string CredentialHeader = "X-Endpoint-Credential";
public const string SharedKeyHeader = "X-API-Key";
private readonly ICredentialProvider credentials;
private readonly ILogger logger;
private readonly bool sharedKeyOnly;
public AgentAuthHandler(ICredentialProvider credentials, ILogger logger, bool sharedKeyOnly = false)
{
this.credentials = credentials;
this.logger = logger;
this.sharedKeyOnly = sharedKeyOnly;
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Headers.Remove(CredentialHeader);
request.Headers.Remove(SharedKeyHeader);
var credential = sharedKeyOnly ? null : credentials.Current?.Credential;
if (credential is not null)
request.Headers.TryAddWithoutValidation(CredentialHeader, credential);
else if (credentials.SharedKey is string sharedKey)
request.Headers.TryAddWithoutValidation(SharedKeyHeader, sharedKey);
var response = await base.SendAsync(request, cancellationToken);
if (response.StatusCode is HttpStatusCode.Unauthorized or HttpStatusCode.Forbidden)
{
try
{
// Buffered so the caller can still read the body.
await response.Content.LoadIntoBufferAsync();
var body = await response.Content.ReadAsStringAsync(cancellationToken);
var code = AgentServerError.TryParse(body)?.Code;
if (!string.IsNullOrEmpty(code))
credentials.ReportRejected(code, credential);
}
catch (Exception error) when (error is not OperationCanceledException)
{
logger.LogDebug(error, "Could not read an auth failure body");
}
}
return response;
}
}
// The server's error envelope: {"status":"error","data":{"error":{"code","message"}}}.
public sealed record AgentServerError(string Code, string Message)
{
public static AgentServerError? TryParse(string? body)
{
if (string.IsNullOrWhiteSpace(body))
return null;
try
{
using var doc = JsonDocument.Parse(body);
var root = doc.RootElement;
if (root.ValueKind != JsonValueKind.Object)
return null;
if (root.TryGetProperty("data", out var data) && data.ValueKind == JsonValueKind.Object)
root = data;
if (!root.TryGetProperty("error", out var error) || error.ValueKind != JsonValueKind.Object)
return null;
var code = error.TryGetProperty("code", out var c) && c.ValueKind == JsonValueKind.String
? c.GetString() ?? "" : "";
var message = error.TryGetProperty("message", out var m) && m.ValueKind == JsonValueKind.String
? m.GetString() ?? "" : "";
return new AgentServerError(code, message);
}
catch (JsonException)
{
return null;
}
}
}
// Server certificate trust. Pins are hex SHA-256 of the leaf SubjectPublicKeyInfo.
// HKLM\SOFTWARE\GE\ShopDB ServerPins is AUTHORITATIVE whenever the value exists
// in either view, even empty (empty = no pins, Windows chain validation), so a
// removed or rotated-out pin really goes away. Only when the value is absent in
// both views do appsettings serverpins and the legacy SHA-1 servercertthumbprint
// apply. With any pin in effect only a matching leaf is accepted and chain/name
// errors are ignored (self-signed servers on the air gap). Without pins:
// SslPolicyErrors.None. Registry re-read at most every RefreshInterval on the
// handshake path, so a GE-Enforce fix or a rotation lands without a restart.
public sealed class ServerPinSet
{
public const string RegistryValue = "ServerPins";
public static readonly TimeSpan RefreshInterval = TimeSpan.FromSeconds(60);
// Present = the value exists in at least one view.
private sealed record RegistryRead(bool Present, List<string> Values)
{
public string Raw => Present ? "registry:" + string.Join("|", Values) : "absent";
}
private sealed record Snapshot(HashSet<string> SpkiPins, string? LegacyThumbprint, bool FromRegistry, string RawRegistry)
{
public string Signature => (FromRegistry ? "registry:" : "appsettings:")
+ string.Join(",", SpkiPins.OrderBy(p => p, StringComparer.Ordinal))
+ (LegacyThumbprint is null ? "" : "+legacy");
public bool Pinned => SpkiPins.Count > 0 || LegacyThumbprint is not null;
public string Source => FromRegistry ? "registry" : "appsettings.json";
}
private readonly HashSet<string> configPins;
private readonly string? legacyThumbprint;
private readonly ILogger logger;
private readonly Func<bool, RegistryRead?> readRegistry;
private readonly object refreshLock = new();
private volatile Snapshot current;
private long nextRefreshTicks;
private ServerPinSet(HashSet<string> configPins, string? legacyThumbprint, ILogger logger,
Func<bool, RegistryRead?> readRegistry)
{
this.configPins = configPins;
this.legacyThumbprint = legacyThumbprint;
this.logger = logger;
this.readRegistry = readRegistry;
current = Build(readRegistry(false) ?? new RegistryRead(false, new List<string>()));
nextRefreshTicks = Environment.TickCount64 + (long)RefreshInterval.TotalMilliseconds;
}
public bool IsPinned => Current().Pinned;
public static ServerPinSet Load(AgentConfig config, ILogger logger)
{
var pins = new HashSet<string>(StringComparer.Ordinal);
foreach (var pin in config.ServerPins ?? new List<string>())
AddPin(pins, pin, "appsettings.json", logger);
string? thumbprint = null;
if (!string.IsNullOrWhiteSpace(config.ServerCertThumbprint))
thumbprint = config.ServerCertThumbprint.Replace(":", "").Replace(" ", "").Trim().ToUpperInvariant();
var set = new ServerPinSet(pins, thumbprint, logger, refresh => ReadRegistryPins(logger, refresh));
var snapshot = set.current;
logger.LogInformation(
"Server certificate trust from {Source}: {Pins} SPKI pin(s){Legacy}; {Mode}.",
snapshot.Source, snapshot.SpkiPins.Count,
snapshot.LegacyThumbprint is null ? "" : " + legacy thumbprint",
snapshot.Pinned ? "pinned" : "Windows chain validation");
if (snapshot.FromRegistry && (pins.Count > 0 || thumbprint is not null))
logger.LogInformation(
"Registry {Value} is set; ignoring appsettings.json serverpins/servercertthumbprint.", RegistryValue);
return set;
}
// Cheap tick compare per handshake; registry read only when the window lapsed.
private Snapshot Current()
{
if (Environment.TickCount64 < Interlocked.Read(ref nextRefreshTicks))
return current;
lock (refreshLock)
{
if (Environment.TickCount64 < nextRefreshTicks)
return current;
var read = readRegistry(true);
Interlocked.Exchange(ref nextRefreshTicks,
Environment.TickCount64 + (long)RefreshInterval.TotalMilliseconds);
var previous = current;
if (read is null || read.Raw == previous.RawRegistry)
return previous;
var next = Build(read);
current = next;
if (next.Signature != previous.Signature)
logger.LogInformation(
"Server pins changed, source {Source}: {Count} SPKI pin(s) [{Pins}]{Legacy}; {Mode}.",
next.Source,
next.SpkiPins.Count,
string.Join(", ", next.SpkiPins.OrderBy(p => p, StringComparer.Ordinal).Select(p => p[..12])),
next.LegacyThumbprint is null ? "" : " + legacy thumbprint",
next.Pinned ? "pinned" : "Windows chain validation");
return next;
}
}
// Malformed-pin warnings only fire when the raw registry text changes.
private Snapshot Build(RegistryRead read)
{
if (!read.Present)
return new Snapshot(new HashSet<string>(configPins, StringComparer.Ordinal),
legacyThumbprint, FromRegistry: false, read.Raw);
var pins = new HashSet<string>(StringComparer.Ordinal);
foreach (var pin in read.Values)
AddPin(pins, pin, "registry", logger);
return new Snapshot(pins, LegacyThumbprint: null, FromRegistry: true, read.Raw);
}
public bool Validate(X509Certificate? certificate, SslPolicyErrors errors, AgentHttpChannel channel)
{
if (certificate is null)
{
logger.LogWarning("Server provided no certificate ({Channel})", channel);
return false;
}
var snapshot = Current();
if (!snapshot.Pinned)
{
if (errors == SslPolicyErrors.None)
return true;
logger.LogWarning(
"Server certificate rejected on {Channel} channel: {Errors} ({Subject})",
channel, errors, certificate.Subject);
return false;
}
var owned = certificate as X509Certificate2;
var leaf = owned ?? new X509Certificate2(certificate);
try
{
var spki = SpkiSha256(leaf);
if (snapshot.SpkiPins.Contains(spki))
return true;
if (snapshot.LegacyThumbprint is not null
&& string.Equals(leaf.GetCertHashString(), snapshot.LegacyThumbprint, StringComparison.OrdinalIgnoreCase))
return true;
logger.LogError(
"Server certificate does not match any pin on {Channel} channel: spki {Spki} ({Subject})",
channel, spki, leaf.Subject);
return false;
}
finally
{
if (owned is null)
leaf.Dispose();
}
}
public static string SpkiSha256(X509Certificate2 certificate)
=> Convert.ToHexString(SHA256.HashData(certificate.PublicKey.ExportSubjectPublicKeyInfo()))
.ToLowerInvariant();
private static void AddPin(HashSet<string> pins, string? raw, string source, ILogger logger)
{
var pin = (raw ?? "").Replace(":", "").Replace(" ", "").Trim().ToLowerInvariant();
if (pin.Length == 0)
return;
if (pin.Length != 64 || !pin.All(Uri.IsHexDigit))
{
logger.LogWarning("Ignoring malformed server pin from {Source}: not 64 hex chars", source);
return;
}
pins.Add(pin);
}
// Startup keeps whatever views read. A refresh returns null on any failed view
// so a transient denial cannot silently swap the pin source (and the server with it).
private static RegistryRead? ReadRegistryPins(ILogger logger, bool refresh)
{
var present = false;
var found = new List<string>();
foreach (var view in new[] { RegistryView.Registry64, RegistryView.Registry32 })
{
try
{
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, view);
using var key = machine.OpenSubKey(AgentConfigProvider.RegistryPath);
var value = key?.GetValue(RegistryValue);
if (value is not null)
present = true;
switch (value)
{
case string[] many:
found.AddRange(many);
break;
case string one:
found.AddRange(one.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries));
break;
}
}
catch (Exception error)
{
if (refresh)
{
logger.LogDebug(error, "Registry re-read of {Value} in {View} view failed; keeping pins", RegistryValue, view);
return null;
}
logger.LogWarning(error, "Registry read of {Value} in {View} view failed", RegistryValue, view);
}
}
return new RegistryRead(present, found);
}
}

180
agent/ShopDbEndpointAgent.Core/AgentPipeClient.cs Normal file
View File

@@ -0,0 +1,180 @@
using System.IO.Pipes;
using System.Security.Principal;
using System.Text;
using System.Text.Json;
namespace ShopDbEndpointAgent.Core;
// Wire constants for the local agent pipe. One UTF-8 JSON object per line:
// request, response, close.
public static class AgentPipeProtocol
{
public const string PipeName = "ShopDB.EndpointAgent.v1";
public const int MaxRequestBytes = 64 * 1024;
public const int MaxResponseBytes = 4 * 1024 * 1024;
public static readonly TimeSpan ConnectTimeout = TimeSpan.FromSeconds(3);
public static readonly TimeSpan RequestReadTimeout = TimeSpan.FromSeconds(10);
}
// Ok/Error as the agent answered. Data is the whole response object (ping
// fields such as agentversion/enrolled/assetid live there). Data is null when
// the agent could not be reached or answered garbage; Error then says why.
public sealed record PipeResponse(bool Ok, string? Error, JsonElement? Data);
public sealed class AgentPipeException : Exception
{
public AgentPipeException(string message) : base(message)
{
}
public AgentPipeException(string message, Exception inner) : base(message, inner)
{
}
}
// Client for the SYSTEM agent's local pipe. Used by the tray (which holds no
// secret) and by shopdbagent.exe --once/--report when not running as SYSTEM.
// Only GetAlertsAsync throws; the others report failure through PipeResponse.
public sealed class AgentPipeClient
{
// Send Status waits on the user-context refresh plus the upload retries.
private static readonly TimeSpan LongOperationTimeout = TimeSpan.FromMinutes(3);
private static readonly TimeSpan ShortOperationTimeout = TimeSpan.FromSeconds(45);
private static readonly JsonSerializerOptions AlertOptions = new()
{
Converters = { new TolerantFleetAlertListConverter() },
};
public Task<PipeResponse> PingAsync(CancellationToken cancellationToken)
=> SendSafeAsync(new { op = "ping" }, ShortOperationTimeout, cancellationToken);
public Task<PipeResponse> SendStatusAsync(CancellationToken cancellationToken)
=> SendSafeAsync(new { op = "sendstatus" }, LongOperationTimeout, cancellationToken);
public Task<PipeResponse> SendReportAsync(string spoolPath, CancellationToken cancellationToken)
=> SendSafeAsync(new { op = "sendreport", spoolpath = spoolPath }, LongOperationTimeout,
cancellationToken);
public async Task<IReadOnlyList<FleetAlert>> GetAlertsAsync(CancellationToken cancellationToken)
{
var response = await SendAsync(new { op = "alerts" }, ShortOperationTimeout, cancellationToken);
if (!response.Ok)
throw new AgentPipeException(response.Error ?? "agent refused the alerts request");
if (response.Data is not JsonElement data
|| !data.TryGetProperty("alerts", out var alerts)
|| alerts.ValueKind != JsonValueKind.Array)
throw new AgentPipeException("agent response has no alerts array");
try
{
return alerts.Deserialize<List<FleetAlert>>(AlertOptions) ?? new List<FleetAlert>();
}
catch (JsonException error)
{
throw new AgentPipeException("agent alerts could not be parsed", error);
}
}
public Task<PipeResponse> AckAlertAsync(int notificationId, CancellationToken cancellationToken)
=> SendSafeAsync(new { op = "ackalert", notificationid = notificationId },
ShortOperationTimeout, cancellationToken);
private static async Task<PipeResponse> SendSafeAsync(
object request, TimeSpan timeout, CancellationToken cancellationToken)
{
try
{
return await SendAsync(request, timeout, cancellationToken);
}
catch (AgentPipeException error)
{
return new PipeResponse(false, error.Message, null);
}
}
// Throws AgentPipeException on any transport or framing failure, and
// OperationCanceledException only when the caller cancelled.
private static async Task<PipeResponse> SendAsync(
object request, TimeSpan timeout, CancellationToken cancellationToken)
{
using var timeoutSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
timeoutSource.CancelAfter(timeout);
var token = timeoutSource.Token;
try
{
// Identification: the agent may learn who we are, never act as us.
await using var pipe = new NamedPipeClientStream(".", AgentPipeProtocol.PipeName,
PipeDirection.InOut, PipeOptions.Asynchronous, TokenImpersonationLevel.Identification);
try
{
await pipe.ConnectAsync((int)AgentPipeProtocol.ConnectTimeout.TotalMilliseconds, token);
}
catch (TimeoutException error)
{
throw new AgentPipeException("agent not reachable (is the ShopDB Endpoint Agent service running?)", error);
}
var line = JsonSerializer.SerializeToUtf8Bytes(request);
await pipe.WriteAsync(line, token);
await pipe.WriteAsync(new[] { (byte)'\n' }, token);
await pipe.FlushAsync(token);
var body = await ReadLineAsync(pipe, AgentPipeProtocol.MaxResponseBytes, token)
?? throw new AgentPipeException("agent closed the pipe without answering");
using var doc = JsonDocument.Parse(body);
var root = doc.RootElement;
if (root.ValueKind != JsonValueKind.Object)
throw new AgentPipeException("agent response is not a JSON object");
var ok = root.TryGetProperty("ok", out var okValue) && okValue.ValueKind == JsonValueKind.True;
var errorText = root.TryGetProperty("error", out var errorValue)
&& errorValue.ValueKind == JsonValueKind.String
? errorValue.GetString() : null;
return new PipeResponse(ok, ok ? null : errorText ?? "agent reported failure", root.Clone());
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
throw;
}
catch (OperationCanceledException error)
{
throw new AgentPipeException("agent did not answer in time", error);
}
catch (AgentPipeException)
{
throw;
}
catch (Exception error) when (error is IOException or UnauthorizedAccessException or JsonException
or InvalidDataException)
{
throw new AgentPipeException("agent pipe failed: " + error.Message, error);
}
}
// One newline-terminated UTF-8 line, or null on EOF before any byte.
// Throws InvalidDataException past maxBytes. Shared with the server.
internal static async Task<string?> ReadLineAsync(Stream stream, int maxBytes, CancellationToken token)
{
var buffer = new MemoryStream();
var chunk = new byte[4096];
while (true)
{
var read = await stream.ReadAsync(chunk, token);
if (read == 0)
return buffer.Length == 0 ? null : Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
var newline = Array.IndexOf(chunk, (byte)'\n', 0, read);
var take = newline >= 0 ? newline : read;
if (buffer.Length + take > maxBytes)
throw new InvalidDataException($"pipe message exceeds {maxBytes} bytes");
buffer.Write(chunk, 0, take);
if (newline >= 0)
return Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
}
}
}

562
agent/ShopDbEndpointAgent.Core/AgentPipeServer.cs Normal file
View File

@@ -0,0 +1,562 @@
using System.IO.Pipes;
using System.Security.AccessControl;
using System.Security.Principal;
using System.Text.Json;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Win32;
namespace ShopDbEndpointAgent.Core;
// \\.\pipe\ShopDB.EndpointAgent.v1: the tray's only way to reach the server.
// The tray holds no secret; it asks the SYSTEM agent to send status, upload a
// report, list alerts or acknowledge one, and the agent does it with its own
// credential. Protocol in AgentPipeProtocol / AgentPipeClient.
//
// ACL: SYSTEM and Administrators full control, INTERACTIVE read/write (no
// create-instance, so a user cannot add instances to our pipe). The first
// instance is created with FirstPipeInstance so a process that squatted the
// name before the service started is refused rather than joined.
public sealed class AgentPipeServer : BackgroundService
{
public const int MaxInstances = 8;
private const int MaxScreenshotBytes = 25 * 1024 * 1024;
private static readonly TimeSpan LongOperationTimeout = TimeSpan.FromMinutes(2);
private static readonly TimeSpan ShortOperationTimeout = TimeSpan.FromSeconds(40);
private static readonly JsonSerializerOptions PayloadOptions = new()
{
PropertyNameCaseInsensitive = true,
};
private readonly AgentConfig config;
private readonly TelemetryWorker telemetry;
private readonly CollectorClient collector;
private readonly FleetAlertClient alerts;
private readonly ICredentialProvider credentials;
private readonly string agentVersion;
private readonly ILogger<AgentPipeServer> logger;
private readonly SemaphoreSlim statusGate = new(1, 1);
private readonly SemaphoreSlim reportGate = new(1, 1);
public AgentPipeServer(AgentConfig config, TelemetryWorker telemetry, CollectorClient collector,
FleetAlertClient alerts, ICredentialProvider credentials, string agentVersion,
ILogger<AgentPipeServer> logger)
{
this.config = config;
this.telemetry = telemetry;
this.collector = collector;
this.alerts = alerts;
this.credentials = credentials;
this.agentVersion = agentVersion;
this.logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var slots = new SemaphoreSlim(MaxInstances, MaxInstances);
bool first = true;
while (!stoppingToken.IsCancellationRequested)
{
try
{
await slots.WaitAsync(stoppingToken);
}
catch (OperationCanceledException)
{
break;
}
NamedPipeServerStream pipe;
try
{
pipe = CreateInstance(first);
}
catch (Exception error)
{
slots.Release();
logger.LogError(error,
first
? "Could not create the agent pipe; another process may own the name. Retrying in 30s."
: "Could not create an agent pipe instance. Retrying in 30s.");
if (!await SafeDelay(TimeSpan.FromSeconds(30), stoppingToken))
break;
continue;
}
if (first)
{
first = false;
logger.LogInformation("Agent pipe listening on {Pipe}", AgentPipeProtocol.PipeName);
RestrictAppSettingsAcl();
}
try
{
await pipe.WaitForConnectionAsync(stoppingToken);
}
catch (Exception error)
{
pipe.Dispose();
slots.Release();
if (stoppingToken.IsCancellationRequested)
break;
logger.LogWarning(error, "Agent pipe connection failed");
if (!await SafeDelay(TimeSpan.FromSeconds(1), stoppingToken))
break;
continue;
}
_ = Task.Run(async () =>
{
try
{
await HandleConnectionAsync(pipe, stoppingToken);
}
catch (Exception error)
{
logger.LogDebug(error, "Agent pipe request ended with an error");
}
finally
{
pipe.Dispose();
slots.Release();
}
}, CancellationToken.None);
}
}
private static NamedPipeServerStream CreateInstance(bool first)
{
var security = new PipeSecurity();
security.AddAccessRule(new PipeAccessRule(
new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null),
PipeAccessRights.FullControl, AccessControlType.Allow));
security.AddAccessRule(new PipeAccessRule(
new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null),
PipeAccessRights.FullControl, AccessControlType.Allow));
security.AddAccessRule(new PipeAccessRule(
new SecurityIdentifier(WellKnownSidType.InteractiveSid, null),
PipeAccessRights.ReadWrite, AccessControlType.Allow));
var options = PipeOptions.Asynchronous;
if (first)
options |= PipeOptions.FirstPipeInstance;
return NamedPipeServerStreamAcl.Create(AgentPipeProtocol.PipeName, PipeDirection.InOut,
MaxInstances, PipeTransmissionMode.Byte, options, 4096, 4096, security);
}
private async Task HandleConnectionAsync(NamedPipeServerStream pipe, CancellationToken stoppingToken)
{
string? line;
using (var readTimeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken))
{
readTimeout.CancelAfter(AgentPipeProtocol.RequestReadTimeout);
try
{
line = await AgentPipeClient.ReadLineAsync(pipe, AgentPipeProtocol.MaxRequestBytes,
readTimeout.Token);
}
catch (InvalidDataException)
{
await WriteAsync(pipe, Failure("request too large"), stoppingToken);
return;
}
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
{
logger.LogDebug("Agent pipe client sent no request within {Timeout}",
AgentPipeProtocol.RequestReadTimeout);
return;
}
}
if (line is null)
return;
// Impersonation needs a completed read first, which we just did.
var (callerName, callerSid) = IdentifyCaller(pipe);
string op;
JsonElement request;
try
{
using var doc = JsonDocument.Parse(line);
request = doc.RootElement.Clone();
op = request.ValueKind == JsonValueKind.Object
&& request.TryGetProperty("op", out var opValue) && opValue.ValueKind == JsonValueKind.String
? opValue.GetString() ?? ""
: "";
}
catch (JsonException)
{
await WriteAsync(pipe, Failure("request is not valid JSON"), stoppingToken);
return;
}
object response;
try
{
response = op switch
{
"ping" => Ping(),
"sendstatus" => await SendStatusAsync(callerName, stoppingToken),
"sendreport" => await SendReportAsync(request, callerName, callerSid, stoppingToken),
"alerts" => await ListAlertsAsync(stoppingToken),
"ackalert" => await AckAlertAsync(request, callerName, stoppingToken),
_ => Failure("unknown op"),
};
}
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
{
response = Failure("the agent timed out talking to ShopDB");
}
await WriteAsync(pipe, response, stoppingToken);
}
private (string? Name, SecurityIdentifier? Sid) IdentifyCaller(NamedPipeServerStream pipe)
{
string? name = null;
SecurityIdentifier? sid = null;
try
{
name = pipe.GetImpersonationUserName();
}
catch (Exception error)
{
logger.LogDebug(error, "GetImpersonationUserName failed");
}
try
{
pipe.RunAsClient(() =>
{
using var identity = WindowsIdentity.GetCurrent(TokenAccessLevels.Query);
sid = identity.User;
});
}
catch (Exception error)
{
logger.LogDebug(error, "Could not read the pipe caller's SID");
}
return (name, sid);
}
private object Ping()
{
var current = credentials.Current;
return new
{
ok = true,
agentversion = agentVersion,
enrolled = current is not null,
assetid = current?.AssetId,
};
}
private async Task<object> SendStatusAsync(string? caller, CancellationToken stoppingToken)
{
if (!await statusGate.WaitAsync(0, stoppingToken))
return Failure("a status send is already running");
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
timeout.CancelAfter(LongOperationTimeout);
logger.LogInformation("Status send requested over the pipe by {Caller}", caller);
var result = await telemetry.RunOnceAsync(config.AdoptOnFirstRun, timeout.Token);
logger.LogInformation("Pipe status send complete: action={Action}", result.Action);
return Success();
}
catch (CollectorException error)
{
logger.LogWarning("Pipe status send failed: HTTP {Status}", error.StatusCode);
return Failure(ServerFailure(error));
}
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
&& !stoppingToken.IsCancellationRequested)
{
logger.LogWarning(error, "Pipe status send failed");
return Failure("could not reach ShopDB");
}
finally
{
statusGate.Release();
}
}
// Does what --report does, but only with files inside the CALLER's own
// profile: the pipe is open to every interactive user and this process is
// SYSTEM, so an unchecked path would upload any file SYSTEM can read. The
// sidecar is not deleted here (the tray removes it); deleting in a user-
// writable tree as SYSTEM is a redirection target.
private async Task<object> SendReportAsync(JsonElement request, string? caller,
SecurityIdentifier? callerSid, CancellationToken stoppingToken)
{
if (callerSid is null)
return Failure("could not identify the calling user");
var spoolPath = request.TryGetProperty("spoolpath", out var pathValue)
&& pathValue.ValueKind == JsonValueKind.String
? pathValue.GetString()
: null;
if (string.IsNullOrWhiteSpace(spoolPath))
return Failure("spoolpath is required");
var profile = ProfilePathFor(callerSid);
if (profile is null)
return Failure("no local profile for the calling user");
if (!await reportGate.WaitAsync(0, stoppingToken))
return Failure("a report upload is already running");
try
{
OperatorReportPayload? payload;
try
{
using var sidecar = OpenCallerFile(spoolPath, profile, AgentPipeProtocol.MaxRequestBytes);
payload = await JsonSerializer.DeserializeAsync<OperatorReportPayload>(
sidecar, PayloadOptions, stoppingToken);
}
catch (Exception error) when (error is IOException or UnauthorizedAccessException
or JsonException or ArgumentException
or NotSupportedException)
{
logger.LogWarning("Refused report sidecar from {Caller}: {Reason}", caller, error.Message);
return Failure("report file refused: " + error.Message);
}
if (payload is null)
return Failure("report file is empty");
FileStream? screenshot = null;
try
{
if (!string.IsNullOrWhiteSpace(payload.ScreenshotPath))
{
if (!payload.ScreenshotPath.EndsWith(".png", StringComparison.OrdinalIgnoreCase))
return Failure("screenshot must be a .png file");
try
{
screenshot = OpenCallerFile(payload.ScreenshotPath, profile, MaxScreenshotBytes);
}
catch (Exception error) when (error is IOException or UnauthorizedAccessException
or ArgumentException or NotSupportedException)
{
logger.LogWarning("Refused report screenshot from {Caller}: {Reason}",
caller, error.Message);
return Failure("screenshot refused: " + error.Message);
}
}
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
timeout.CancelAfter(LongOperationTimeout);
var uploaded = await collector.UploadOperatorReportAsync(
payload.Reason, payload.Details, screenshot,
payload.ScreenshotPath is null ? null : Path.GetFileName(payload.ScreenshotPath),
Environment.MachineName, timeout.Token);
logger.LogInformation("Operator report from {Caller} uploaded: id={ReportId}",
caller, uploaded?.OperatorReportId);
return Success();
}
catch (CollectorException error)
{
logger.LogWarning("Operator report upload failed: HTTP {Status}", error.StatusCode);
return Failure(ServerFailure(error));
}
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
&& !stoppingToken.IsCancellationRequested)
{
logger.LogWarning(error, "Operator report upload failed");
return Failure("could not reach ShopDB");
}
finally
{
screenshot?.Dispose();
}
}
finally
{
reportGate.Release();
}
}
private async Task<object> ListAlertsAsync(CancellationToken stoppingToken)
{
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
timeout.CancelAfter(ShortOperationTimeout);
var list = await alerts.FetchAlertsAsync(Environment.MachineName, timeout.Token);
return new { ok = true, alerts = list.ToArray() };
}
catch (CollectorException error)
{
logger.LogDebug("Alert fetch for the tray failed: HTTP {Status}", error.StatusCode);
return Failure(ServerFailure(error));
}
catch (Exception error) when (error is HttpRequestException or TaskCanceledException or JsonException
&& !stoppingToken.IsCancellationRequested)
{
logger.LogDebug(error, "Alert fetch for the tray failed");
return Failure("could not reach ShopDB");
}
}
private async Task<object> AckAlertAsync(JsonElement request, string? caller, CancellationToken stoppingToken)
{
if (!request.TryGetProperty("notificationid", out var idValue)
|| idValue.ValueKind != JsonValueKind.Number
|| !idValue.TryGetInt32(out var notificationId)
|| notificationId <= 0)
return Failure("notificationid must be a positive integer");
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
timeout.CancelAfter(ShortOperationTimeout);
// 0 = "the caller": the server resolves it from credential or hostname.
var assetId = credentials.Current?.AssetId ?? 0;
await alerts.AcknowledgeAlertAsync(notificationId, assetId, Environment.MachineName, timeout.Token);
logger.LogInformation("Alert {NotificationId} acknowledged by {Caller}", notificationId, caller);
return Success();
}
catch (CollectorException error)
{
logger.LogWarning("Alert {NotificationId} acknowledge failed: HTTP {Status}",
notificationId, error.StatusCode);
return Failure(ServerFailure(error));
}
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
&& !stoppingToken.IsCancellationRequested)
{
logger.LogWarning(error, "Alert {NotificationId} acknowledge failed", notificationId);
return Failure("could not reach ShopDB");
}
}
// Profile root for a SID from ProfileList. Null when absent.
private static string? ProfilePathFor(SecurityIdentifier sid)
{
try
{
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64);
using var key = machine.OpenSubKey(
@"SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\" + sid.Value);
if (key?.GetValue("ProfileImagePath") is not string raw || string.IsNullOrWhiteSpace(raw))
return null;
return Path.GetFullPath(Environment.ExpandEnvironmentVariables(raw)).TrimEnd('\\');
}
catch
{
return null;
}
}
// Open a caller-named file only if it sits inside the caller's profile, no
// directory from the profile down is a reparse point, the file itself is
// not one, and the opened handle's final path is still inside the profile.
private static FileStream OpenCallerFile(string path, string profile, int maxBytes)
{
if (!Path.IsPathFullyQualified(path) || path.StartsWith(@"\\", StringComparison.Ordinal))
throw new UnauthorizedAccessException("path must be a local absolute path");
var full = Path.GetFullPath(path);
var root = profile + @"\";
if (!full.StartsWith(root, StringComparison.OrdinalIgnoreCase))
throw new UnauthorizedAccessException("path is outside the caller's profile");
var directory = Path.GetDirectoryName(full);
while (directory is not null && directory.Length >= profile.Length)
{
if (ProtectedFileSystem.IsReparsePoint(directory))
throw new UnauthorizedAccessException("path crosses a reparse point");
directory = Path.GetDirectoryName(directory);
}
var stream = ProtectedFileSystem.OpenReadNoFollow(full);
try
{
var final = ProtectedFileSystem.FinalPath(stream.SafeFileHandle);
if (final is null || !final.StartsWith(root, StringComparison.OrdinalIgnoreCase))
throw new UnauthorizedAccessException("path resolves outside the caller's profile");
if (stream.Length > maxBytes)
throw new IOException($"file is larger than {maxBytes} bytes");
return stream;
}
catch
{
stream.Dispose();
throw;
}
}
// Only SYSTEM (the service) and Administrators read appsettings.json now:
// no user-context process needs it once the tray talks through this pipe.
private void RestrictAppSettingsAcl()
{
try
{
using var identity = WindowsIdentity.GetCurrent();
if (!identity.IsSystem)
return;
var path = Path.Combine(AppContext.BaseDirectory, "appsettings.json");
if (ProtectedFileSystem.TryGetAttributes(path) is not FileAttributes attributes
|| attributes.HasFlag(FileAttributes.ReparsePoint))
return;
var security = new FileSecurity();
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
security.AddAccessRule(new FileSystemAccessRule(
new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null),
FileSystemRights.FullControl, AccessControlType.Allow));
security.AddAccessRule(new FileSystemAccessRule(
new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null),
FileSystemRights.FullControl, AccessControlType.Allow));
new FileInfo(path).SetAccessControl(security);
logger.LogInformation("appsettings.json restricted to SYSTEM and Administrators.");
}
catch (Exception error)
{
logger.LogWarning(error, "Could not restrict the appsettings.json ACL");
}
}
private static string ServerFailure(CollectorException error)
=> string.IsNullOrEmpty(error.ServerCode)
? $"ShopDB returned HTTP {error.StatusCode}"
: $"ShopDB returned HTTP {error.StatusCode} {error.ServerCode}";
private static object Success() => new { ok = true };
private static object Failure(string error) => new { ok = false, error };
private static async Task WriteAsync(Stream pipe, object response, CancellationToken token)
{
try
{
var bytes = JsonSerializer.SerializeToUtf8Bytes(response);
await pipe.WriteAsync(bytes, token);
await pipe.WriteAsync(new[] { (byte)'\n' }, token);
await pipe.FlushAsync(token);
}
catch (IOException)
{
// Client went away.
}
}
private static async Task<bool> SafeDelay(TimeSpan delay, CancellationToken token)
{
try
{
await Task.Delay(delay, token);
return true;
}
catch (OperationCanceledException)
{
return false;
}
}
}

119
agent/ShopDbEndpointAgent.Core/CollectorClient.cs
View File

@@ -5,8 +5,9 @@ using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Posts one EndpointReport to POST {baseurl}/api/collector/endpoints with the
// X-API-Key header (plugins/endpoints/AGENT-CONTRACT.md section 1), unwrapping
// Posts one EndpointReport to POST {baseurl}/api/endpoints/collect (per-PC
// credential) or /api/collector/endpoints (shared key), auth added by
// AgentHttpFactory (plugins/endpoints/AGENT-CONTRACT.md section 1), unwrapping
// the ShopDB envelope { status, data, meta } to return the CollectorResponse.
//
// Resilience: transient failures (network errors, timeouts, 5xx, 429) are
@@ -27,17 +28,15 @@ public sealed class CollectorClient : IDisposable
private readonly int maxAttempts;
private readonly Random jitter = new();
public CollectorClient(AgentConfig config, ILogger<CollectorClient> logger,
private readonly ICredentialProvider credentials;
public CollectorClient(AgentHttpFactory httpFactory, ILogger<CollectorClient> logger,
int maxAttempts = 4)
{
this.logger = logger;
this.maxAttempts = Math.Max(1, maxAttempts);
http = new HttpClient
{
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
Timeout = TimeSpan.FromSeconds(30),
};
http.DefaultRequestHeaders.Add("X-API-Key", config.ApiKey);
credentials = httpFactory.Credentials;
http = httpFactory.CreateClient(AgentHttpChannel.Collector);
}
public async Task<CollectorResponse> SendAsync(
@@ -76,8 +75,12 @@ public sealed class CollectorClient : IDisposable
private async Task<CollectorResponse> SendOnceAsync(
EndpointReport report, CancellationToken token)
{
using var response = await http.PostAsJsonAsync(
"api/collector/endpoints", report, JsonOptions, token);
// Enrolled agents post to the credential route; the shared key keeps
// the core collector route so this build still reports to an older server.
var route = credentials.Current is not null
? "api/endpoints/collect"
: "api/collector/endpoints";
using var response = await http.PostAsJsonAsync(route, report, JsonOptions, token);
var body = await response.Content.ReadAsStringAsync(token);
if (!response.IsSuccessStatusCode)
@@ -91,7 +94,7 @@ public sealed class CollectorClient : IDisposable
// Fetch the server-resolved watch list (policy-attached processes/services)
// for this hostname, GET /api/endpoints/monitors/resolved. Shares the same
// X-API-Key credential as telemetry. Best-effort: returns null on any
// auth as telemetry. Best-effort: returns null on any
// failure so the caller falls back to the last-known/local list rather than
// blanking monitoring on a transient blip. A single attempt, no retry - the
// next telemetry cycle retries naturally, and a stale-but-usable watch list
@@ -134,49 +137,77 @@ public sealed class CollectorClient : IDisposable
// Uploads one operator issue report (free-text reason/details plus an
// optional PNG screenshot) to POST /api/endpoints/operatorreports as
// multipart/form-data, authorized by the same X-API-Key collector key as
// telemetry. The server resolves this hostname to its asset. A single
// multipart/form-data, authorized the same way as telemetry. The server
// resolves this hostname (or the credential) to its asset. A single
// attempt: the tray shows the operator the outcome and can retry by hand,
// and a stuck retry loop holding a screenshot in memory is worse than a
// clear failure. Returns the parsed upload response (new row id) or throws.
public async Task<OperatorReportUploadResponse?> UploadOperatorReportAsync(
string reason, string details, string? screenshotPath, string hostname,
CancellationToken token)
{
FileStream? imageStream = null;
try
{
if (!string.IsNullOrWhiteSpace(screenshotPath) && File.Exists(screenshotPath))
imageStream = File.OpenRead(screenshotPath);
return await UploadOperatorReportAsync(reason, details, imageStream,
screenshotPath is null ? null : Path.GetFileName(screenshotPath),
hostname, token);
}
finally
{
imageStream?.Dispose();
}
}
// Same upload from an already opened screenshot stream. The pipe path opens
// the file itself (no reparse follow, final path checked) and hands it here.
public async Task<OperatorReportUploadResponse?> UploadOperatorReportAsync(
string reason, string details, Stream? screenshot, string? screenshotName,
string hostname, CancellationToken token)
{
using var form = new MultipartFormDataContent();
form.Add(new StringContent(reason ?? string.Empty), "reason");
form.Add(new StringContent(details ?? string.Empty), "details");
form.Add(new StringContent(hostname ?? string.Empty), "hostname");
StreamContent? imageContent = null;
FileStream? imageStream = null;
try
if (screenshot is not null)
{
if (!string.IsNullOrWhiteSpace(screenshotPath) && File.Exists(screenshotPath))
{
imageStream = File.OpenRead(screenshotPath);
imageContent = new StreamContent(imageStream);
imageContent.Headers.ContentType =
new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
form.Add(imageContent, "screenshot",
Path.GetFileName(screenshotPath));
}
using var response = await http.PostAsync(
"api/endpoints/operatorreports", form, token);
var body = await response.Content.ReadAsStringAsync(token);
if (!response.IsSuccessStatusCode)
throw new CollectorException((int)response.StatusCode, body);
var envelope = JsonSerializer.Deserialize<OperatorReportUploadEnvelope>(
body, JsonOptions);
return envelope?.Data;
}
finally
{
imageContent?.Dispose();
imageStream?.Dispose();
// Not disposed with the form: the caller owns the stream.
var imageContent = new StreamContent(new NonClosingStream(screenshot));
imageContent.Headers.ContentType =
new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
form.Add(imageContent, "screenshot", screenshotName ?? "screenshot.png");
}
using var response = await http.PostAsync(
"api/endpoints/operatorreports", form, token);
var body = await response.Content.ReadAsStringAsync(token);
if (!response.IsSuccessStatusCode)
throw new CollectorException((int)response.StatusCode, body);
var envelope = JsonSerializer.Deserialize<OperatorReportUploadEnvelope>(
body, JsonOptions);
return envelope?.Data;
}
private sealed class NonClosingStream : Stream
{
private readonly Stream inner;
public NonClosingStream(Stream inner) => this.inner = inner;
public override bool CanRead => inner.CanRead;
public override bool CanSeek => inner.CanSeek;
public override bool CanWrite => false;
public override long Length => inner.Length;
public override long Position { get => inner.Position; set => inner.Position = value; }
public override void Flush() { }
public override int Read(byte[] buffer, int offset, int count) => inner.Read(buffer, offset, count);
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
=> inner.ReadAsync(buffer, cancellationToken);
public override long Seek(long offset, SeekOrigin origin) => inner.Seek(offset, origin);
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
}
public void Dispose() => http.Dispose();
@@ -193,9 +224,15 @@ public sealed class CollectorException : Exception
StatusCode >= 400 && StatusCode < 500
&& StatusCode != (int)HttpStatusCode.TooManyRequests;
public string Body { get; }
// Error code from the server's JSON envelope (e.g. CREDENTIAL_INVALID), or null.
public string? ServerCode => AgentServerError.TryParse(Body)?.Code;
public CollectorException(int statusCode, string body)
: base($"collector returned HTTP {statusCode}: {body}")
{
StatusCode = statusCode;
Body = body;
}
}

31
agent/ShopDbEndpointAgent.Core/CommandClient.cs
View File

@@ -10,9 +10,8 @@ namespace ShopDbEndpointAgent.Core;
// GET /api/endpoints/commands/next, which the server holds open until a command
// is queued (HTTP 200) or the hold window elapses (HTTP 204). After running the
// claimed command locally the agent posts the outcome to
// POST /api/endpoints/commands/{id}/result. Both share the endpoints collector
// key via X-API-Key, so the command channel rides the same kill switch as
// telemetry and adds no second secret.
// POST /api/endpoints/commands/{id}/result. Both use the same auth as telemetry
// (per-PC credential, else the shared collector key), added by AgentHttpFactory.
public sealed class CommandClient : IDisposable
{
private static readonly JsonSerializerOptions JsonOptions = new()
@@ -22,33 +21,13 @@ public sealed class CommandClient : IDisposable
};
private readonly HttpClient http;
private readonly SocketsHttpHandler handler;
private readonly ILogger<CommandClient> logger;
public CommandClient(AgentConfig config, ILogger<CommandClient> logger)
public CommandClient(AgentHttpFactory httpFactory, ILogger<CommandClient> logger)
{
this.logger = logger;
// The command channel long-polls: the server holds each claim open for a
// hold window and then, between windows, the pooled TLS connection sits
// idle. Gunicorn's keep-alive closes an idle socket after a couple of
// seconds, so a connection reused after that yields "unexpected EOF or 0
// bytes from the transport stream" on the next TLS handshake. Retire
// pooled connections faster than the server's keep-alive so a socket the
// server has already dropped is never handed back out; the lifetime cap
// is a hard backstop so no connection lingers indefinitely.
handler = new SocketsHttpHandler
{
PooledConnectionIdleTimeout = TimeSpan.FromSeconds(1),
PooledConnectionLifetime = TimeSpan.FromMinutes(2),
};
http = new HttpClient(handler, disposeHandler: true)
{
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
// Longer than the collector client: the claim request is held open
// server-side (~25s), so the read timeout must clear that plus slack.
Timeout = TimeSpan.FromSeconds(60),
};
http.DefaultRequestHeaders.Add("X-API-Key", config.ApiKey);
// Pooling tuned for the long-poll lives in AgentHttpFactory (Command).
http = httpFactory.CreateClient(AgentHttpChannel.Command);
}
// Claim the next pending command for this hostname, or null when the server

1289
agent/ShopDbEndpointAgent.Core/CommandExecutor.cs
View File

File diff suppressed because it is too large Load Diff

31
agent/ShopDbEndpointAgent.Core/CommandWorker.cs
View File

@@ -41,15 +41,20 @@ public sealed class CommandWorker : BackgroundService
private readonly CommandExecutor executor;
private readonly ILogger<CommandWorker> logger;
private readonly CollectSignal? collectSignal;
private readonly AgentConfig? config;
private readonly string hostname;
private readonly EnrollmentService? enrollment;
public CommandWorker(CommandClient client, CommandExecutor executor,
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null)
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null,
AgentConfig? config = null, EnrollmentService? enrollment = null)
{
this.enrollment = enrollment;
this.client = client;
this.executor = executor;
this.logger = logger;
this.collectSignal = collectSignal;
this.config = config;
hostname = Environment.MachineName;
}
@@ -57,6 +62,30 @@ public sealed class CommandWorker : BackgroundService
{
logger.LogInformation("Command channel started for host {Host}", hostname);
// Spread first contact across the fleet. Every agent otherwise polls the
// instant its service starts, and the service is start=auto, so a patch
// reboot puts the whole fleet on the wire together. See StartupJitter.
var startupDelay = StartupJitter.Next(config?.StartupJitterSeconds ?? 0, logger);
if (startupDelay > TimeSpan.Zero)
{
logger.LogInformation(
"Delaying first command poll by {Delay}ms (startup jitter)",
(int)startupDelay.TotalMilliseconds);
await SafeDelay(startupDelay, stoppingToken);
}
if (enrollment is not null)
{
try
{
await enrollment.WaitForFirstAttemptAsync(TimeSpan.FromSeconds(20), stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
while (!stoppingToken.IsCancellationRequested)
{
var pollStarted = System.Diagnostics.Stopwatch.StartNew();

51
agent/ShopDbEndpointAgent.Core/ConsoleWindow.cs Normal file
View File

@@ -0,0 +1,51 @@
using System.Runtime.InteropServices;
namespace ShopDbEndpointAgent.Core;
// Hides the console window this process was given.
//
// shopdbagent.exe is a console executable, so anything that starts it in an
// interactive session gets a visible black window. That was invisible while the
// tray owned the probe - it launched with CreateNoWindow - but the user-context
// scheduled task runs the exe directly in the operator's session, where the
// window is very much visible and pops up on every collection cycle.
//
// The window is hidden rather than the exe rebuilt as a WinExe: the agent is
// also a service host and a command-line tool, and both of those want a real
// console for --once, --report and service debugging.
public static class ConsoleWindow
{
private const int SW_HIDE = 0;
[DllImport("kernel32.dll")]
private static extern nint GetConsoleWindow();
[DllImport("user32.dll")]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool ShowWindow(nint window, int command);
// Best-effort: a process with no console (already detached, or redirected by
// a caller) simply has nothing to hide, and failing to hide a window must
// never stop the collection that window belongs to.
public static void Hide()
{
if (!OperatingSystem.IsWindows())
{
return;
}
try
{
var window = GetConsoleWindow();
if (window != nint.Zero)
{
ShowWindow(window, SW_HIDE);
}
}
catch
{
// Nothing useful to do or say: there is no log sink yet at the point
// this runs, and a visible window is cosmetic next to losing data.
}
}
}

71
agent/ShopDbEndpointAgent.Core/Contracts.cs
View File

@@ -44,11 +44,57 @@ public sealed class EndpointReport
// collection: both Uninstall registry hives, {displayname, displayversion,
// publisher}. Server logs it as an unpersisted key until the SBOM store lands.
[JsonPropertyName("applications")] public object[]? Applications { get; set; }
// True when Windows is holding a reboot. A machine that has been waiting to
// reboot for weeks explains stalled patching and "the fix did not take"
// reports, and on a shop floor PC nobody reboots voluntarily.
[JsonPropertyName("pendingreboot")] public bool? PendingReboot { get; set; }
// Which signal(s) drove pendingreboot, so the server can tell a box staged
// mid-servicing (pendingfilerename) from one with a stale servicing key.
// Empty array = checked, nothing pending. Omitted = the check failed.
[JsonPropertyName("rebootreasons")] public string[]? RebootReasons { get; set; }
// hiberbootenabled (fast startup configured) and lastboottype (what the
// last boot actually did). Uptime is not trustworthy when these disagree.
[JsonPropertyName("bootinfo")] public object? BootInfo { get; set; }
// Defender health. Persisted but deliberately not surfaced: known false
// positives in this environment.
[JsonPropertyName("defender")] public object? Defender { get; set; }
// Physical drive health from two sources: MSFT_PhysicalDisk (the only one
// that reports NVMe) and Win32_DiskDrive.Status (ATA/SATA and fallback).
// Each entry carries a precomputed predictfailure so an alert is a boolean
// test rather than a string match against an enum. Null predictfailure
// means the drive did not answer - unknown, not healthy.
[JsonPropertyName("diskhealth")] public object[]? DiskHealth { get; set; }
// Session-scoped data gathered by a probe running as the signed-in user:
// mapped drives and user-owned listeners. Null when nobody is signed in or
// the probe did not produce a result this cycle - absent means "not
// observed", which is deliberately distinct from an empty list.
[JsonPropertyName("usercontext")] public object? UserContext { get; set; }
// Pending Dell Command | Update results, produced by the shopdb-dellupdates
// agent utility rather than collected in C#: the data comes from dcu-cli's
// own scan output, so the utility that parses it belongs next to the task
// that writes it. Null means no scan has ever run (or the utility is not
// installed), which is deliberately distinct from a scan that found nothing
// pending - the first is "unknown", the second is "known clean".
[JsonPropertyName("dellupdates")] public object? DellUpdates { get; set; }
}
public sealed class EndpointEventRecord
{
[JsonPropertyName("eventid")] public int EventId { get; set; }
// The Windows event record id: monotonic and unique within its channel on a
// given machine. It is both the agent's harvest watermark and the server's
// dedupe key, so the same record can never be stored twice. Null only when
// the source record did not expose one.
[JsonPropertyName("eventrecordid")] public long? EventRecordId { get; set; }
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
@@ -136,6 +182,28 @@ public sealed class AppDeliveryClaim
// downloads with its collector credential; Sha256 pins the exact bytes;
// RunContext is 'system' (the agent's own service account) or 'user' (the
// interactive logged-on session), decided per script in the ShopDB library.
// WorkingDirectory is the optional folder to run the script IN. Empty means the
// agent's staging directory, which is what every script did before the field
// existed. It is there so a script can sit beside a corporate tool already
// installed on the endpoint and invoke it by relative path; it confers no
// rights, since the body is already arbitrary PowerShell that could change its
// own directory.
// ErrorMode is 'default' or 'strict'. Default is what PowerShell does unaided:
// a non-terminating error is written and the script continues, so powershell.exe
// still exits 0 and the command reports SUCCEEDED even when the body did none of
// its work. Strict makes the agent prepend $ErrorActionPreference = 'Stop' so the
// first error terminates and the exit code means what an operator reads it to
// mean. Per script, never global: strict changes execution semantics, and a body
// written to tolerate a failing cleanup or to probe with a cmdlet that errors on
// absence would start aborting halfway. An absent or unrecognised value is
// treated as default, so an old server that never sends the field is safe.
// CaptureOutput decides whether a SUCCESSFUL run returns its output to the
// server. Failures always return it, because a failure has to be diagnosable
// without walking to the machine. Success is opt-in per script: output carries
// whatever the body printed, and that is not worth persisting server-side for a
// routine green run. The endpoint log keeps the full copy regardless, so this
// only governs what crosses the wire. Absent means false, which is the same
// behaviour as before the field existed.
public sealed class ScriptDeliveryClaim
{
[JsonPropertyName("scriptid")] public int ScriptId { get; set; }
@@ -143,6 +211,9 @@ public sealed class ScriptDeliveryClaim
[JsonPropertyName("installpath")] public string? InstallPath { get; set; }
[JsonPropertyName("sha256")] public string? Sha256 { get; set; }
[JsonPropertyName("runcontext")] public string? RunContext { get; set; }
[JsonPropertyName("workingdirectory")] public string? WorkingDirectory { get; set; }
[JsonPropertyName("errormode")] public string? ErrorMode { get; set; }
[JsonPropertyName("captureoutput")] public bool CaptureOutput { get; set; }
}
// Envelope for the claim long-poll: { status, data: { endpointcommandid, action } }.

8
agent/ShopDbEndpointAgent.Core/CuratedEventCatalog.cs
View File

@@ -27,7 +27,15 @@ public static class CuratedEventCatalog
new(55, "System", "NTFS file system corruption"),
new(7011, "System", "Service Control Manager timeout"),
new(10016, "System", "DCOM permission error"),
// Windows Error Reporting on the Application channel: 1000 is the
// faulting-application record (app, version, faulting module, exception
// code), 1001 is the WER bucket record Windows pairs with it, and 1002
// is an unresponsive app rather than a crash. The structured detail for
// all three arrives in the record's EventData, which the collector lifts
// into `fields`.
new(1000, "Application", "Application error / faulting module"),
new(1001, "Application", "Application error / bugcheck report"),
new(1002, "Application", "Application hang"),
};
// The distinct channels the collector must read.

836
agent/ShopDbEndpointAgent.Core/EndpointCollector.cs
View File

@@ -1,5 +1,8 @@
using System.Diagnostics;
using System.Diagnostics.Eventing.Reader;
using System.Management;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Logging;
using Microsoft.Win32;
@@ -19,16 +22,28 @@ public sealed class EndpointCollector : ICollector
// keep the payload bounded on a chatty Security log.
private const int MaxEventsPerChannel = 25;
// The spool file is read every cycle. Reading it is a small file read, and
// the tray - not this cadence - decides how often the data is refreshed, so
// throttling the read here would only delay already-collected data reaching
// the server without saving any real work.
private readonly ILogger<EndpointCollector> logger;
private readonly AgentConfig config;
private readonly IAgentUtilityRunner? utilities;
private readonly EventWatermarkStore? watermarks;
private readonly UserContextSpool? userContextSpool;
private object? lastUserContext;
public EndpointCollector(ILogger<EndpointCollector> logger, AgentConfig config,
IAgentUtilityRunner? utilities = null)
IAgentUtilityRunner? utilities = null, EventWatermarkStore? watermarks = null,
UserContextSpool? userContextSpool = null)
{
this.logger = logger;
this.config = config;
this.utilities = utilities;
this.watermarks = watermarks;
this.userContextSpool = userContextSpool;
}
public EndpointReport Collect(string agentVersion, bool adopt)
@@ -49,23 +64,232 @@ public sealed class EndpointCollector : ICollector
MachineNumber = GetMachineNumber(),
LastBootTime = GetLastBootTime(),
Services = GetMonitoredTargets(),
Ports = GetListeningPorts(),
Interfaces = GetInterfaces(),
Events = GetCuratedEvents(),
Disks = GetDisks(),
DiskHealth = GetDiskHealth(),
Bios = GetBios(),
Memory = GetMemory(),
Processor = GetProcessor(),
OsInfo = GetOsInfo(),
Domain = GetDomain(),
Applications = GetApplications(),
PendingReboot = GetPendingReboot(),
RebootReasons = GetRebootReasons(),
BootInfo = GetBootInfo(),
Defender = GetDefender(),
UserContext = GetUserContext(),
};
if (report.LastBootTime is DateTimeOffset boot)
report.UptimeSeconds = (long)(DateTimeOffset.UtcNow - boot).TotalSeconds;
report.CpuPercent = GetCpuPercent();
report.MemoryPercent = GetMemoryPercent();
report.IdleSeconds = GetIdleSecondsFromUserContext();
return report;
}
// Instantaneous CPU load from the _Total performance instance.
//
// Uses the WMI formatted-data class rather than PerformanceCounter because a
// PerformanceCounter reads 0 on its first sample and needs a second sample a
// second later to mean anything. Sleeping a second inside the collect path
// would delay the heartbeat on every tick for one number. The formatted-data
// class is already averaged by the OS, so a single read is usable.
private double? GetCpuPercent()
{
try
{
using var searcher = new ManagementObjectSearcher(
"SELECT PercentProcessorTime FROM " +
"Win32_PerfFormattedData_PerfOS_Processor WHERE Name='_Total'");
foreach (var mo in searcher.Get())
{
var value = ToLong(mo["PercentProcessorTime"]);
if (value is long percent)
return Math.Clamp(percent, 0, 100);
}
}
catch (Exception error)
{
logger.LogDebug(error, "CPU percent query failed");
}
return null;
}
// Percentage of physical memory in use. Reported alongside the absolute
// byte counts in the memory object because the percentage is what an alert
// threshold keys off, and deriving it server-side would break whenever the
// byte fields are absent.
private double? GetMemoryPercent()
{
try
{
using var searcher = new ManagementObjectSearcher(
"SELECT TotalVisibleMemorySize, FreePhysicalMemory " +
"FROM Win32_OperatingSystem");
foreach (var mo in searcher.Get())
{
var totalKb = ToLong(mo["TotalVisibleMemorySize"]);
var freeKb = ToLong(mo["FreePhysicalMemory"]);
if (totalKb is long total && total > 0 && freeKb is long free)
{
var used = (double)(total - free) / total * 100.0;
return Math.Round(Math.Clamp(used, 0, 100), 1);
}
}
}
catch (Exception error)
{
logger.LogDebug(error, "Memory percent query failed");
}
return null;
}
// Idle time is measured by the user-context probe, because GetLastInputInfo
// is per session and this service runs in session 0. Null when no probe
// result is available, which correctly reads as "not observed" rather than
// "the machine has been idle for zero seconds".
private long? GetIdleSecondsFromUserContext()
{
if (lastUserContext is not Dictionary<string, object?> context)
return null;
if (!context.TryGetValue("idleseconds", out var raw) || raw is null)
return null;
try
{
if (raw is JsonElement element)
{
return element.ValueKind == JsonValueKind.Number
&& element.TryGetInt64(out var parsed) ? parsed : null;
}
return Convert.ToInt64(raw);
}
catch (Exception error)
{
logger.LogDebug(error, "Could not read idleseconds from the user context");
return null;
}
}
// Listening TCP ports with the owning process name.
//
// Only listeners, never established connections: listeners answer "what
// service is this machine offering" (the DNC port, the MTConnect adapter),
// which is stable and small. Established connections churn constantly and
// would make every report differ from the last for no diagnostic gain, so
// the full table is exposed as an on-demand command instead.
private const int MaxListeners = 100;
private object[] GetListeningPorts()
{
try
{
var listeners = TcpTableReader.GetListeners();
if (listeners.Count == 0)
return Array.Empty<object>();
var names = GetProcessNames();
var result = new List<object>();
foreach (var listener in listeners)
{
if (result.Count >= MaxListeners)
{
logger.LogDebug(
"Listener list truncated at {Limit} entries.", MaxListeners);
break;
}
names.TryGetValue(listener.ProcessId, out var processName);
result.Add(new
{
localaddress = listener.LocalAddress,
localport = listener.LocalPort,
processid = listener.ProcessId,
processname = processName,
protocol = "tcp",
});
}
return result.ToArray();
}
catch (Exception error)
{
logger.LogDebug(error, "Listening port enumeration failed");
return Array.Empty<object>();
}
}
// Snapshot the process table once and index by id. Resolving each listener
// individually would be an O(n) call per row and can race with process exit.
private Dictionary<int, string> GetProcessNames()
{
var names = new Dictionary<int, string>();
try
{
foreach (var process in Process.GetProcesses())
{
try
{
names[process.Id] = process.ProcessName;
}
catch
{
// The process exited between enumeration and read.
}
finally
{
process.Dispose();
}
}
}
catch (Exception error)
{
logger.LogDebug(error, "Process name snapshot failed");
}
return names;
}
// Session-scoped data, read from the spool file the tray writes. The service
// only reads: it cannot collect this itself, because mapped drives and idle
// time do not exist in session 0. Between reads the previous result is
// repeated, because a mapped drive stays mapped and dropping the key would
// read on the server as "the drive went away". The spool's own age limit is
// what eventually retires a result, so a logged-off machine goes quiet
// instead of reporting the last operator forever.
private object? GetUserContext()
{
if (userContextSpool is null)
return null;
var resultPath = userContextSpool.TryGetResultPath(out var reason);
if (resultPath is null)
{
logger.LogDebug("Skipping the user-context read: {Reason}", reason);
return lastUserContext;
}
// Peek rather than consume: the probe overwrites the spool on every run,
// so the newest result is always the one on disk. The file deliberately
// survives a service restart, which the previous delete-on-read did not -
// lastUserContext is in-memory only, so a restart used to leave the agent
// with no drives to report until the tray's next probe up to 30 minutes
// later. lastUserContext still repeats the previous value between reads,
// because the tray collects on a slower cadence than telemetry and
// dropping the key in between would read on the server as "the drives
// went away".
var collected = userContextSpool.ConsumeResult(resultPath);
if (collected is not null)
lastUserContext = collected;
return lastUserContext;
}
// The interactive tray app ships and updates independently of this service,
// so the agent reports the tray's file version as its own field. Resolve the
// tray exe beside the agent first (the normal install layout), then the
@@ -296,6 +520,7 @@ public sealed class EndpointCollector : ICollector
private object[] GetInterfaces()
{
var result = new List<object>();
var linkstate = GetLinkState();
try
{
using var searcher = new ManagementObjectSearcher(
@@ -321,6 +546,10 @@ public sealed class EndpointCollector : ICollector
var gateway = gateways is null || gateways.Length == 0
? null : gateways[0];
// Paired by MAC from MSFT_NetAdapter. Absent when that class is
// unavailable, which must leave the rest of the entry intact.
linkstate.TryGetValue(NormalizeMac(mac) ?? "", out var link);
result.Add(new
{
alias = mo["Description"]?.ToString()?.Trim() ?? "",
@@ -332,6 +561,10 @@ public sealed class EndpointCollector : ICollector
gateway = gateway ?? "",
subnet = subnet ?? "",
dhcp = mo["DHCPEnabled"] as bool? ?? false,
// Null rather than 0/false when link state was not
// readable: an unknown duplex must not read as half.
linkspeedbps = link.Speed,
fullduplex = link.FullDuplex,
});
}
}
@@ -451,40 +684,33 @@ public sealed class EndpointCollector : ICollector
}
}
// Walk each curated channel newest-first, keep only the curated IDs for
// that channel, cap per channel. Reads Security/System/Application - the
// old scaffold read System only and filtered by the wrong IDs.
// Walk each curated channel newest-first and report ONLY records newer than
// the stored watermark, so a record crosses the wire exactly once.
//
// Two things changed here versus the original scaffold. First, filtering by
// event id now happens inside the event log service through an XPath query
// instead of walking every entry in the channel from C#; on a busy Security
// log that is the difference between touching a handful of records and
// hundreds of thousands. Second, EventLogReader (not the legacy EventLog
// API) exposes RecordId, which is what makes exactly-once possible.
//
// First contact on a channel has no watermark, so it sends a bounded tail
// (MaxEventsPerChannel) for initial context rather than the whole backlog.
// The same cap also bounds a burst: if more than the cap arrived since the
// last cycle, the NEWEST are taken and the watermark still jumps to the
// newest, so the payload can never balloon.
private EndpointEventRecord[] GetCuratedEvents()
{
var records = new List<EndpointEventRecord>();
foreach (var channel in CuratedEventCatalog.Channels)
{
var wanted = CuratedEventCatalog.IdsForChannel(channel);
if (wanted.Count == 0)
continue;
try
{
using var log = new EventLog(channel);
int kept = 0;
for (int i = log.Entries.Count - 1;
i >= 0 && kept < MaxEventsPerChannel; i--)
{
EventLogEntry entry;
try { entry = log.Entries[i]; } catch { continue; }
int id = (int)(entry.InstanceId & 0xFFFF);
if (!wanted.Contains(id))
continue;
records.Add(new EndpointEventRecord
{
EventId = id,
EventChannel = channel,
EventSource = entry.Source,
EventTime = new DateTimeOffset(
entry.TimeGenerated.ToUniversalTime(), TimeSpan.Zero),
EventMessage = Truncate(entry.Message, 1000),
});
kept++;
}
records.AddRange(ReadChannel(channel, wanted));
}
catch (Exception error)
{
@@ -493,9 +719,172 @@ public sealed class EndpointCollector : ICollector
logger.LogDebug(error, "Event channel {Channel} unreadable", channel);
}
}
watermarks?.Save();
return records.ToArray();
}
private IEnumerable<EndpointEventRecord> ReadChannel(
string channel, IReadOnlySet<int> wanted)
{
long watermark = watermarks?.GetWatermark(channel) ?? 0;
bool seeding = watermarks is null || !watermarks.HasWatermark(channel);
var query = new EventLogQuery(channel, PathType.LogName, BuildIdFilter(wanted))
{
// Newest-first so the cap keeps the most recent records and we can
// stop as soon as we reach the watermark.
ReverseDirection = true,
};
var found = new List<EndpointEventRecord>();
long newestSeen = 0;
using var reader = new EventLogReader(query);
while (found.Count < MaxEventsPerChannel)
{
EventRecord? entry;
try { entry = reader.ReadEvent(); }
catch (Exception error)
{
logger.LogDebug(error, "Event read failed on {Channel}", channel);
break;
}
if (entry is null)
break;
using (entry)
{
long recordId = entry.RecordId ?? 0;
if (recordId > newestSeen)
newestSeen = recordId;
// Reverse order means everything from here down is older than
// the watermark and has already been reported.
if (!seeding && recordId != 0 && recordId <= watermark)
break;
found.Add(ToEventRecord(entry, channel, recordId));
}
}
// A cleared log restarts record ids low; reseed rather than go silent.
if (!seeding && newestSeen > 0
&& watermarks is not null && watermarks.ResetIfStale(channel, newestSeen))
{
return found;
}
if (newestSeen > 0)
watermarks?.Advance(channel, newestSeen);
return found;
}
private EndpointEventRecord ToEventRecord(
EventRecord entry, string channel, long recordId)
{
string? message = null;
try { message = entry.FormatDescription(); }
catch (Exception error)
{
// A provider whose message DLL is missing cannot render text; the
// id, time, and source are still worth reporting.
logger.LogDebug(error, "Event message not renderable on {Channel}", channel);
}
return new EndpointEventRecord
{
EventId = entry.Id,
EventRecordId = recordId == 0 ? null : recordId,
EventChannel = channel,
EventSource = entry.ProviderName,
EventTime = entry.TimeCreated is DateTime created
? new DateTimeOffset(created.ToUniversalTime(), TimeSpan.Zero)
: null,
EventMessage = Truncate(message, 1000),
Fields = GetEventFields(entry, channel),
};
}
// Bounds on the structured field extraction. The rendered message is often
// a paragraph of prose while the useful specifics (which account, which
// logon type, which faulting module) live in EventData, so it is worth
// carrying - but a single record can declare a lot of elements and some
// providers emit large blobs, so both the count and each value are capped.
private const int MaxEventFields = 30;
private const int MaxEventFieldLength = 512;
// Pull the provider's structured payload out of the event XML into the
// existing `fields` dictionary, which the server already stores as
// payloadjson. Most providers use <EventData><Data Name="x">v</Data></>;
// some use <UserData> with an arbitrary provider-defined subtree, so both
// are walked. Unnamed <Data> elements keep their positional index as the
// key, because for several catalog IDs position IS the meaning.
//
// Never throws: a provider with malformed or unreadable XML degrades to no
// fields rather than losing the whole record.
private Dictionary<string, object>? GetEventFields(EventRecord entry, string channel)
{
try
{
var xml = entry.ToXml();
if (string.IsNullOrWhiteSpace(xml))
return null;
var document = System.Xml.Linq.XDocument.Parse(xml);
var root = document.Root;
if (root is null)
return null;
var container =
root.Elements().FirstOrDefault(e => e.Name.LocalName == "EventData")
?? root.Elements().FirstOrDefault(e => e.Name.LocalName == "UserData");
if (container is null)
return null;
var fields = new Dictionary<string, object>();
int unnamed = 0;
foreach (var element in container.Descendants())
{
if (fields.Count >= MaxEventFields)
break;
if (element.HasElements)
continue;
var value = element.Value;
if (string.IsNullOrWhiteSpace(value))
continue;
var name = element.Attribute("Name")?.Value;
if (string.IsNullOrWhiteSpace(name))
{
name = element.Name.LocalName == "Data"
? "data" + unnamed++
: element.Name.LocalName;
}
fields[name!] = Truncate(value, MaxEventFieldLength) ?? string.Empty;
}
return fields.Count > 0 ? fields : null;
}
catch (Exception error)
{
logger.LogDebug(error, "Event XML not parseable on {Channel}", channel);
return null;
}
}
// An XPath the event log service evaluates itself, e.g.
// *[System[(EventID=4624 or EventID=4625)]]
private static string BuildIdFilter(IReadOnlySet<int> wanted)
{
var clauses = string.Join(" or ", wanted.Select(id => "EventID=" + id));
return "*[System[(" + clauses + ")]]";
}
// Installed-software SBOM. Mirrors the Assess-Orchestrator applications
// collection exactly: enumerate both Uninstall registry hives (64-bit and
// WOW6432Node), keep only entries with a DisplayName, and emit
@@ -547,11 +936,15 @@ public sealed class EndpointCollector : ICollector
if (!seen.Add(displayName + "\u0000" + displayVersion))
continue;
var fingerprint = ComputeApplicationFingerprint(
displayName, displayVersion, publisher);
result.Add(new
{
displayname = displayName,
displayversion = displayVersion,
publisher,
fingerprint,
});
}
catch (Exception error)
@@ -566,6 +959,256 @@ public sealed class EndpointCollector : ICollector
}
}
// Compute SHA256 fingerprint for application inventory delta optimization.
// Format: SHA256(displayname|displayversion|publisher) as lowercase hex string.
// Matches backend algorithm in plugins/endpoints/plugin.py and backfill script.
// Empty strings are preserved (not converted to null) to match backend behavior.
private static string ComputeApplicationFingerprint(
string displayName, string displayVersion, string publisher)
{
var normalized = $"{displayName}|{displayVersion}|{publisher}";
var bytes = Encoding.UTF8.GetBytes(normalized);
var hash = SHA256.HashData(bytes);
return Convert.ToHexString(hash).ToLowerInvariant();
}
// Whether Windows is waiting on a reboot.
//
// There is no single API for this, so we check the three signals Windows
// itself uses: the CBS servicing flag, the Windows Update auto-update flag,
// and a queued file-rename operation. Any one of them is sufficient. Checked
// by key EXISTENCE rather than value, which is how these flags work - the
// key is created when a reboot is needed and deleted once it happens.
// Link speed and duplex, keyed by MAC so the caller can pair them onto the
// matching GetInterfaces() entry. Deliberately a separate query: the
// IPEnabled config class above does not carry link state, and a NIC that
// negotiated 100Mb half-duplex on a gigabit switch port is a cable/port
// fault that otherwise presents as "the machine is slow sometimes".
private Dictionary<string, (long? Speed, bool? FullDuplex)> GetLinkState()
{
var byMac = new Dictionary<string, (long?, bool?)>(
StringComparer.OrdinalIgnoreCase);
// MSFT_NetAdapter carries duplex; Win32_NetworkAdapter does not. It
// lives in the same Storage-style optional namespace family, so a
// failure here must not take the interface list with it.
try
{
using var searcher = new ManagementObjectSearcher(
@"root\StandardCimv2",
"SELECT PermanentAddress, Speed, FullDuplex, MediaConnectState " +
"FROM MSFT_NetAdapter");
foreach (var mo in searcher.Get())
{
var mac = NormalizeMac(mo["PermanentAddress"]?.ToString());
if (mac is null)
continue;
byMac[mac] = (ToLong(mo["Speed"]), mo["FullDuplex"] as bool?);
}
}
catch (Exception error)
{
logger.LogDebug(error, "Net adapter link state query failed");
}
return byMac;
}
// WMI reports MACs in several shapes (00:11:22, 001122AABBCC). Strip to hex
// so the two sources can be matched at all.
private static string? NormalizeMac(string? raw)
{
if (string.IsNullOrWhiteSpace(raw))
return null;
var hex = new string(Array.FindAll(raw.ToCharArray(), Uri.IsHexDigit));
return hex.Length == 12 ? hex : null;
}
// Whether the last start was a true cold boot or a fast-startup resume.
//
// This matters because uptime LIES under fast startup: the box reports days
// of uptime after what the operator experienced as a shutdown, and a
// "reboot" that was really a hybrid resume does not clear the driver or
// kernel state that a support call is usually trying to clear.
//
// Two distinct facts, deliberately not collapsed:
// hiberbootenabled - fast startup is CONFIGURED (registry, always readable)
// lastboottype - what the last boot ACTUALLY did (Kernel-Boot event 27)
// The registry alone cannot tell you what the last boot did, and the event
// may have aged out of the log, so neither substitutes for the other.
private object? GetBootInfo()
{
bool? hiberboot = null;
try
{
using var power = Registry.LocalMachine.OpenSubKey(
@"SYSTEM\CurrentControlSet\Control\Session Manager\Power");
if (power?.GetValue("HiberbootEnabled") is int raw)
hiberboot = raw != 0;
}
catch (Exception error)
{
logger.LogDebug(error, "Hiberboot registry read failed");
}
string? lastboottype = null;
try
{
// Kernel-Boot 27 BootType: 0 cold, 1 hybrid (fast startup),
// 2 resume from hibernate.
var query = new EventLogQuery(
"System", PathType.LogName,
"*[System[Provider[@Name='Microsoft-Windows-Kernel-Boot'] " +
"and (EventID=27)]]")
{ ReverseDirection = true };
using var reader = new EventLogReader(query);
using var record = reader.ReadEvent();
if (record?.Properties.Count > 0)
{
lastboottype = ToInt(record.Properties[0].Value) switch
{
0 => "Cold",
1 => "Fast startup",
2 => "Resume",
_ => null,
};
}
}
catch (Exception error)
{
logger.LogDebug(error, "Boot type event read failed");
}
if (hiberboot is null && lastboottype is null)
return null;
return new
{
hiberbootenabled = hiberboot,
lastboottype = lastboottype ?? "Unknown",
};
}
// Defender status. Collected and persisted but NOT surfaced: this
// environment is known to produce false positives, so wiring it to a card
// or an alert now would train operators to ignore the card. Stored so the
// signal is already there when someone is ready to interpret it.
private object? GetDefender()
{
try
{
using var searcher = new ManagementObjectSearcher(
@"root\Microsoft\Windows\Defender",
"SELECT AMServiceEnabled, RealTimeProtectionEnabled, " +
"AntivirusSignatureAge, AntispywareSignatureAge, " +
"QuickScanAge FROM MSFT_MpComputerStatus");
foreach (var mo in searcher.Get())
{
return new
{
serviceenabled = mo["AMServiceEnabled"] as bool?,
realtimeenabled = mo["RealTimeProtectionEnabled"] as bool?,
// Ages in days. Null when Defender did not report rather
// than 0, which would read as "updated today".
signatureagedays = ToLong(mo["AntivirusSignatureAge"]),
antispywareagedays = ToLong(mo["AntispywareSignatureAge"]),
quickscanagedays = ToLong(mo["QuickScanAge"]),
};
}
}
catch (Exception error)
{
// A third-party AV box or a stripped image has no Defender WMI
// provider at all. That is normal here, not a fault.
logger.LogDebug(error, "Defender status query failed");
}
return null;
}
// Which signal(s) drove pendingreboot=true, in report order.
//
// pendingreboot is a single bool OR-ed from three sources, so it cannot say
// WHY. That distinction is the point: PendingFileRenameOperations is the
// stronger signal - files are staged to be replaced on next boot, so the
// box is mid-servicing - whereas a lone CBS or Windows Update key can sit
// set for weeks on a machine nobody has rebooted. Same boolean, very
// different urgency.
//
// Empty array = checked, nothing pending. Null = the check failed, matching
// pendingreboot's own null-is-unknown rule.
private string[]? GetRebootReasons()
{
try
{
var reasons = new List<string>();
if (LocalMachineKeyExists(
@"SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending"))
{
reasons.Add("componentbasedservicing");
}
if (LocalMachineKeyExists(
@"SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired"))
{
reasons.Add("windowsupdate");
}
using var sessionManager = Registry.LocalMachine.OpenSubKey(
@"SYSTEM\CurrentControlSet\Control\Session Manager");
if (sessionManager?.GetValue("PendingFileRenameOperations") is string[] pending
&& pending.Length > 0)
{
reasons.Add("pendingfilerename");
}
return reasons.ToArray();
}
catch (Exception error)
{
logger.LogDebug(error, "Reboot reason check failed");
return null;
}
}
private bool? GetPendingReboot()
{
try
{
const string cbs =
@"SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending";
const string windowsUpdate =
@"SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired";
if (LocalMachineKeyExists(cbs) || LocalMachineKeyExists(windowsUpdate))
return true;
using var sessionManager = Registry.LocalMachine.OpenSubKey(
@"SYSTEM\CurrentControlSet\Control\Session Manager");
if (sessionManager?.GetValue("PendingFileRenameOperations") is string[] pending
&& pending.Length > 0)
{
return true;
}
return false;
}
catch (Exception error)
{
// Null, not false: an unreadable registry means we do not know, and
// reporting a confident "no reboot pending" would be worse than
// reporting nothing.
logger.LogDebug(error, "Pending reboot check failed");
return null;
}
}
private static bool LocalMachineKeyExists(string path)
{
using var key = Registry.LocalMachine.OpenSubKey(path);
return key is not null;
}
// Fixed local disks with capacity/free so the server can flag a PC that is
// running out of space. Win32_LogicalDisk DriveType 3 = local fixed disk;
// network/removable drives are skipped. Bytes stay as longs; the server can
@@ -591,6 +1234,14 @@ public sealed class EndpointCollector : ICollector
freebytes = free,
usedbytes = size is long total && free is long available
? total - available : (long?)null,
// Precomputed so an alert threshold is a direct comparison.
// A 40GB free reading means nothing without the capacity it
// came from, and the server would have to guard against a
// null or zero size on every evaluation to derive it.
freepercent = size is long capacity && capacity > 0
&& free is long remaining
? Math.Round((double)remaining / capacity * 100.0, 1)
: (double?)null,
});
}
}
@@ -601,6 +1252,129 @@ public sealed class EndpointCollector : ICollector
return result.ToArray();
}
// Physical drive health, so a failing disk is caught before it takes the
// machine and whatever was local on it.
//
// Two sources, because neither covers the fleet alone:
//
// MSFT_PhysicalDisk (Storage namespace) is the only one that reports NVMe.
// The classic SMART class, MSStorageDriver_FailurePredictStatus, is ATA/SATA
// only and returns NOTHING on an NVMe box - verified on a two-NVMe machine
// here, where it yielded zero instances while the drives were present and
// healthy. Collecting only that class would have reported an empty disk
// health list on every modern endpoint and looked like "no problems".
//
// Win32_DiskDrive.Status is the old reliable string ("OK"/"Pred Fail") and
// covers anything the Storage namespace does not, including machines where
// that namespace is missing or blocked.
//
// Keyed by DeviceId so the server can pair the two views per drive the same
// way mapped drives are paired.
private object[] GetDiskHealth()
{
var result = new List<object>();
// MSFT_PhysicalDisk returns raw enum INTEGERS through
// ManagementObjectSearcher, not the friendly strings Get-PhysicalDisk
// prints. Verified on a live box: Health=0 Op=2 for a healthy NVMe.
// Mapping them here keeps the wire self-describing; shipping bare
// integers would make the server guess at an undocumented enum.
try
{
using var searcher = new ManagementObjectSearcher(
@"root\Microsoft\Windows\Storage",
"SELECT DeviceId, FriendlyName, MediaType, BusType, " +
"HealthStatus, OperationalStatus FROM MSFT_PhysicalDisk");
foreach (var mo in searcher.Get())
{
var health = ToInt(mo["HealthStatus"]);
result.Add(new
{
deviceid = mo["DeviceId"]?.ToString()?.Trim() ?? "",
model = mo["FriendlyName"]?.ToString()?.Trim() ?? "",
mediatype = PhysicalDiskMediaType(ToInt(mo["MediaType"])),
bustype = PhysicalDiskBusType(ToInt(mo["BusType"])),
healthstatus = PhysicalDiskHealth(health),
// The actionable bit, precomputed so an alert is a direct
// boolean test rather than a string comparison against an
// enum the server would have to keep in sync with this file.
// Null when health is unreadable: unknown is not healthy,
// but it is not a confident failure either.
predictfailure = health is null ? (bool?)null : health != 0,
source = "physicaldisk",
});
}
}
catch (Exception error)
{
logger.LogDebug(error, "Physical disk health query failed");
}
// Win32_DiskDrive as the ATA/SATA and fallback view. Status is a plain
// string here, no enum mapping needed.
try
{
using var searcher = new ManagementObjectSearcher(
"SELECT DeviceID, Model, Status FROM Win32_DiskDrive");
foreach (var mo in searcher.Get())
{
var status = mo["Status"]?.ToString()?.Trim() ?? "";
result.Add(new
{
deviceid = mo["DeviceID"]?.ToString()?.Trim() ?? "",
model = mo["Model"]?.ToString()?.Trim() ?? "",
mediatype = "",
bustype = "",
healthstatus = status,
// "OK" is the only healthy value this class reports;
// "Pred Fail" is the SMART trip. An empty status means the
// drive did not answer, which is unknown rather than failed.
predictfailure = status.Length == 0
? (bool?)null
: !status.Equals("OK", StringComparison.OrdinalIgnoreCase),
source = "diskdrive",
});
}
}
catch (Exception error)
{
logger.LogDebug(error, "Disk drive status query failed");
}
return result.ToArray();
}
// MSFT_PhysicalDisk HealthStatus: 0 Healthy, 1 Warning, 2 Unhealthy.
private static string PhysicalDiskHealth(int? value) => value switch
{
0 => "Healthy",
1 => "Warning",
2 => "Unhealthy",
_ => "Unknown",
};
// MSFT_PhysicalDisk MediaType: 3 HDD, 4 SSD, 5 SCM.
private static string PhysicalDiskMediaType(int? value) => value switch
{
3 => "HDD",
4 => "SSD",
5 => "SCM",
_ => "Unknown",
};
// MSFT_PhysicalDisk BusType. Only the values that actually appear on an
// endpoint are named; anything else stays numeric rather than guessed at.
private static string PhysicalDiskBusType(int? value) => value switch
{
3 => "ATA",
7 => "USB",
8 => "RAID",
9 => "iSCSI",
11 => "SATA",
17 => "NVMe",
_ => value is null ? "Unknown" : value.Value.ToString(),
};
// Firmware identity from Win32_BIOS: vendor, version string, and release
// date. Lets the server track BIOS/UEFI levels for a fleet update campaign.
private object? GetBios()
@@ -737,6 +1511,14 @@ public sealed class EndpointCollector : ICollector
catch { return null; }
}
private static int? ToInt(object? value)
{
if (value is null)
return null;
try { return Convert.ToInt32(value); }
catch { return null; }
}
// Win32 CIM datetimes (yyyymmddHHMMSS.ffffff+zzz) render as an ISO-8601 UTC
// string; anything unparseable passes through untouched so nothing is lost.
private string? ToBiosDate(string? raw)

357
agent/ShopDbEndpointAgent.Core/EnrollmentService.cs Normal file
View File

@@ -0,0 +1,357 @@
using System.Management;
using System.Net;
using System.Net.Http.Json;
using System.Text.Json;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Win32;
namespace ShopDbEndpointAgent.Core;
// Gets this PC its own endpoint credential (POST /api/endpoints/enroll) and
// keeps trying until it has one. Workers run on the shared key meanwhile, so an
// unenrolled or pending PC keeps reporting.
//
// 201 enrolled store credential, swap provider, idle until woken
// 202 pending PendingSchedule: 1 min, then 5 min, then 15 min
// 404 server predates enrollment: idle, retry every 6 h
// other back off 1, 5, 15 min (cap); a 429 or 5xx lands here, so a
// rate limit drops out of the fast pending phase for that try
//
// Woken early by the provider on CREDENTIAL_INVALID (credential deleted) and on
// SHARED_KEY_REFUSED while unenrolled, floored at one attempt a minute.
public sealed class EnrollmentService : BackgroundService
{
public const string DeviceIdValue = "DeviceId";
// Retry while pending, by how long this request has been pending. An admin
// usually approves within a couple of minutes and a tech is standing there,
// so ask often at first. Jitter is added, never subtracted, so MinWakeInterval
// cannot clip it back into lockstep across a room of PCs.
private static readonly (TimeSpan Waited, TimeSpan Retry, TimeSpan Jitter)[] PendingSchedule =
{
(TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(8)),
(TimeSpan.FromMinutes(30), TimeSpan.FromMinutes(5), TimeSpan.FromSeconds(20)),
(TimeSpan.MaxValue, TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(2)),
};
private static readonly TimeSpan UnsupportedRetry = TimeSpan.FromHours(6);
private static readonly TimeSpan[] ErrorBackoff =
{
TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(15),
};
private static readonly TimeSpan MinWakeInterval = TimeSpan.FromMinutes(1);
private readonly AgentConfig config;
private readonly AgentHttpFactory httpFactory;
private readonly AgentCredentialProvider credentials;
private readonly CredentialStore store;
private readonly string agentVersion;
private readonly ILogger<EnrollmentService> logger;
private readonly SemaphoreSlim wake = new(0, 1);
private readonly TaskCompletionSource firstAttempt =
new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly Random jitter = new();
public EnrollmentService(AgentConfig config, AgentHttpFactory httpFactory,
AgentCredentialProvider credentials, CredentialStore store, string agentVersion,
ILogger<EnrollmentService> logger)
{
this.config = config;
this.httpFactory = httpFactory;
this.credentials = credentials;
this.store = store;
this.agentVersion = agentVersion;
this.logger = logger;
credentials.EnrollmentNeeded += (_, _) => Wake();
}
// Workers await this (bounded) before first contact so a PC that can enroll
// right away does its first report on its own credential.
public async Task WaitForFirstAttemptAsync(TimeSpan limit, CancellationToken token)
{
try
{
await firstAttempt.Task.WaitAsync(limit, token);
}
catch (TimeoutException)
{
}
}
public void Wake()
{
try
{
if (wake.CurrentCount == 0)
wake.Release();
}
catch (SemaphoreFullException)
{
}
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
int failures = 0;
var lastAttempt = DateTime.MinValue;
// Start of the current pending run; reset on a new request id so a
// reject-and-ask-again (or a revoke) gets the fast phase over again.
DateTime? pendingSince = null;
int? pendingRequest = null;
using var client = httpFactory.CreateClient(AgentHttpChannel.Enroll);
try
{
while (!stoppingToken.IsCancellationRequested)
{
if (credentials.Current is not null)
{
pendingSince = null;
pendingRequest = null;
firstAttempt.TrySetResult();
await wake.WaitAsync(stoppingToken);
continue;
}
var sinceLast = DateTime.UtcNow - lastAttempt;
if (sinceLast < MinWakeInterval)
await Task.Delay(MinWakeInterval - sinceLast, stoppingToken);
lastAttempt = DateTime.UtcNow;
var (outcome, requestId) = await TryEnrollAsync(client, stoppingToken);
firstAttempt.TrySetResult();
TimeSpan delay;
switch (outcome)
{
case EnrollOutcome.Enrolled:
failures = 0;
pendingSince = null;
pendingRequest = null;
continue;
case EnrollOutcome.Pending:
failures = 0;
if (pendingSince is null
|| (requestId is not null && requestId != pendingRequest))
{
pendingSince = DateTime.UtcNow;
pendingRequest = requestId;
}
delay = PendingDelay(DateTime.UtcNow - pendingSince.Value);
logger.LogDebug("Next enrollment attempt in {Delay}.", delay);
break;
case EnrollOutcome.Unsupported:
failures = 0;
pendingSince = null;
pendingRequest = null;
delay = UnsupportedRetry;
break;
default:
delay = ErrorBackoff[Math.Min(failures, ErrorBackoff.Length - 1)];
failures++;
break;
}
// A wake cuts the wait short (credential refused, shared key refused).
await wake.WaitAsync(delay, stoppingToken);
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
}
finally
{
firstAttempt.TrySetResult();
}
}
private enum EnrollOutcome { Enrolled, Pending, Unsupported, Failed }
private TimeSpan PendingDelay(TimeSpan waited)
{
foreach (var step in PendingSchedule)
{
if (waited < step.Waited)
return step.Retry
+ TimeSpan.FromSeconds(jitter.NextDouble() * step.Jitter.TotalSeconds);
}
return PendingSchedule[^1].Retry;
}
private async Task<(EnrollOutcome Outcome, int? RequestId)> TryEnrollAsync(
HttpClient client, CancellationToken token)
{
var hostname = Environment.MachineName;
var serial = ReadBiosSerial();
if (string.IsNullOrWhiteSpace(serial))
{
logger.LogError("Enrollment skipped: no BIOS serial number from WMI Win32_BIOS.");
return (EnrollOutcome.Failed, null);
}
string nonce;
try
{
nonce = store.GetOrCreateEnrollNonce();
}
catch (Exception error)
{
logger.LogError(error, "Enrollment skipped: could not read or create enroll.json.");
return (EnrollOutcome.Failed, null);
}
var request = new
{
hostname,
serialnumber = serial,
deviceid = ReadDeviceId(),
enrollnonce = nonce,
agentversion = agentVersion,
};
try
{
using var response = await client.PostAsJsonAsync("api/endpoints/enroll", request, token);
var body = await response.Content.ReadAsStringAsync(token);
if (response.StatusCode == HttpStatusCode.NotFound
&& AgentServerError.TryParse(body)?.Code is null or "" or "NOT_FOUND")
{
logger.LogInformation(
"Server has no enrollment endpoint (HTTP 404); staying on the shared key, retry in {Delay}.",
UnsupportedRetry);
return (EnrollOutcome.Unsupported, null);
}
if (response.StatusCode == HttpStatusCode.Created)
{
var result = ParseResult(body);
var secret = result.GetString("credential");
if (secret is null || !secret.StartsWith("sdbe_", StringComparison.Ordinal))
{
logger.LogError("Enrollment returned 201 without a usable credential.");
return (EnrollOutcome.Failed, null);
}
var credential = new AgentCredential
{
Credential = secret,
AssetId = result.GetInt("assetid") ?? 0,
IssuedAt = DateTime.UtcNow.ToString("o"),
BaseUrl = config.BaseUrl.Trim().TrimEnd('/'),
};
try
{
credentials.Set(credential);
}
catch (Exception error)
{
// Shown once by the server; an admin must revoke and let it re-enroll.
logger.LogError(error,
"Enrolled as asset {AssetId} but could not store credential {Prefix}. " +
"Revoke it in ShopDB so this PC can enroll again.",
credential.AssetId, credential.Prefix);
return (EnrollOutcome.Failed, null);
}
return (EnrollOutcome.Enrolled, null);
}
if (response.StatusCode == HttpStatusCode.Accepted)
{
var result = ParseResult(body);
var requestId = result.GetInt("requestid");
logger.LogInformation(
"Enrollment pending (request {RequestId}): {Reason}.",
requestId, result.GetString("reason") ?? "no reason given");
return (EnrollOutcome.Pending, requestId);
}
var refusal = AgentServerError.TryParse(body);
if (refusal?.Code == "ENROLL_NONCE_MISMATCH")
{
logger.LogError(
"Enrollment refused: the approved request belongs to a different enroll nonce. " +
"An admin must reject it so this PC can request again.");
// No request id: keeps the current pending run and its schedule.
return (EnrollOutcome.Pending, null);
}
logger.LogWarning("Enrollment failed: HTTP {Status} {Code} {Message}",
(int)response.StatusCode, refusal?.Code, refusal?.Message);
return (EnrollOutcome.Failed, null);
}
catch (Exception error) when (
error is HttpRequestException or TaskCanceledException or JsonException
&& !token.IsCancellationRequested)
{
logger.LogWarning("Enrollment request failed: {Message}", error.Message);
return (EnrollOutcome.Failed, null);
}
}
// Accepts the bare body or the success envelope's data object.
private static ResultReader ParseResult(string body)
{
using var doc = JsonDocument.Parse(body);
var root = doc.RootElement;
if (root.ValueKind == JsonValueKind.Object
&& root.TryGetProperty("data", out var data) && data.ValueKind == JsonValueKind.Object)
root = data;
return new ResultReader(root.Clone());
}
private readonly struct ResultReader
{
private readonly JsonElement root;
public ResultReader(JsonElement root) => this.root = root;
public string? GetString(string name)
=> root.ValueKind == JsonValueKind.Object
&& root.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String
? value.GetString() : null;
public int? GetInt(string name)
=> root.ValueKind == JsonValueKind.Object
&& root.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.Number
&& value.TryGetInt32(out var number)
? number : null;
}
private string? ReadBiosSerial()
{
try
{
using var searcher = new ManagementObjectSearcher("SELECT SerialNumber FROM Win32_BIOS");
foreach (var item in searcher.Get())
{
var value = item["SerialNumber"]?.ToString()?.Trim();
if (!string.IsNullOrWhiteSpace(value))
return value.Length > 128 ? value[..128] : value;
}
}
catch (Exception error)
{
logger.LogWarning(error, "WMI Win32_BIOS query failed");
}
return null;
}
private string? ReadDeviceId()
{
foreach (var view in new[] { RegistryView.Registry64, RegistryView.Registry32 })
{
try
{
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, view);
using var key = machine.OpenSubKey(AgentConfigProvider.RegistryPath);
if (key?.GetValue(DeviceIdValue) is string value && !string.IsNullOrWhiteSpace(value))
return value.Trim();
}
catch (Exception error)
{
logger.LogDebug(error, "Registry read of {Value} in {View} view failed", DeviceIdValue, view);
}
}
return null;
}
}

132
agent/ShopDbEndpointAgent.Core/EventWatermarkStore.cs Normal file
View File

@@ -0,0 +1,132 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Remembers the newest Windows event record id already reported for each
// curated channel, so a collect cycle harvests ONLY records the server has not
// seen.
//
// Why this exists: the collector used to re-read the newest N records of every
// channel on every cycle, and the server ingest appends event rows without a
// dedupe check. A single crash therefore landed in the database once per
// cadence tick for as long as it stayed inside that window - up to 96 identical
// rows a day per event. Carrying a watermark makes the steady state "only what
// is new", which is also what makes the full event XML affordable to send.
//
// The watermark is persisted under %ProgramData% (not in memory) because a
// service restart would otherwise re-report the whole window again, which is
// exactly the crash-loop case where duplicate events are most likely.
//
// Record ids are per channel and monotonic on a given machine. A cleared or
// rebuilt event log restarts them at 1; that is handled by treating a stored
// watermark HIGHER than the newest available record as stale and resetting it,
// otherwise the channel would go permanently silent.
public sealed class EventWatermarkStore
{
private const string StoreFileName = "eventwatermark.json";
// %ProgramData%\ShopDB\EndpointAgent - the same root the rolling agent log
// lives under, so all agent-owned state sits in one place.
public static string DefaultStateDirectory => Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"ShopDB", "EndpointAgent");
private readonly string storePath;
private readonly ILogger<EventWatermarkStore> logger;
private readonly Dictionary<string, long> watermarks;
public EventWatermarkStore(ILogger<EventWatermarkStore> logger,
string? stateDirectory = null)
{
this.logger = logger;
var directory = string.IsNullOrWhiteSpace(stateDirectory)
? DefaultStateDirectory
: stateDirectory;
storePath = Path.Combine(directory, StoreFileName);
watermarks = Load();
}
// The last record id reported for this channel, or 0 when the channel has
// never been harvested. 0 means "first contact": the caller sends a bounded
// tail for initial context rather than the entire backlog.
public long GetWatermark(string channel)
=> watermarks.TryGetValue(channel, out var value) ? value : 0;
public bool HasWatermark(string channel)
=> watermarks.ContainsKey(channel);
// Advance the channel forward only. Never moves backwards, so an
// out-of-order read cannot cause the same record to be reported twice.
public void Advance(string channel, long recordId)
{
if (recordId <= 0)
return;
if (watermarks.TryGetValue(channel, out var current) && current >= recordId)
return;
watermarks[channel] = recordId;
}
// A cleared or rebuilt log restarts record ids near 1. When the newest
// record available is below the stored watermark the store is stale, so
// drop it and let the channel re-seed instead of going silent forever.
public bool ResetIfStale(string channel, long newestAvailableRecordId)
{
if (newestAvailableRecordId <= 0)
return false;
if (!watermarks.TryGetValue(channel, out var current) || current <= newestAvailableRecordId)
return false;
logger.LogInformation(
"Event channel {Channel} record id went backwards ({Stored} -> {Newest}); log was cleared, reseeding",
channel, current, newestAvailableRecordId);
watermarks.Remove(channel);
return true;
}
// Best-effort persist. A failure here costs at most a repeated report of
// records already sent, so it must never abort a collection.
public void Save()
{
try
{
var directory = Path.GetDirectoryName(storePath);
if (!string.IsNullOrWhiteSpace(directory))
Directory.CreateDirectory(directory);
// Write-then-rename so a crash mid-write cannot leave a truncated
// file that would reseed every channel on the next start.
var temporaryPath = storePath + ".tmp";
File.WriteAllText(temporaryPath,
JsonSerializer.Serialize(watermarks));
File.Move(temporaryPath, storePath, overwrite: true);
}
catch (Exception error)
{
logger.LogDebug(error, "Could not persist event watermarks to {Path}", storePath);
}
}
private Dictionary<string, long> Load()
{
try
{
if (!File.Exists(storePath))
return new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase);
var loaded = JsonSerializer.Deserialize<Dictionary<string, long>>(
File.ReadAllText(storePath));
return loaded is null
? new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase)
: new Dictionary<string, long>(loaded, StringComparer.OrdinalIgnoreCase);
}
catch (Exception error)
{
// A corrupt store reseeds every channel, which costs one bounded
// tail per channel - acceptable versus failing the collection.
logger.LogWarning(error,
"Event watermark store unreadable at {Path}; reseeding channels", storePath);
return new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase);
}
}
}

62
agent/ShopDbEndpointAgent.Core/FileLoggerProvider.cs
View File

@@ -14,32 +14,74 @@ public sealed class FileLoggerProvider : ILoggerProvider
private readonly string directory;
private readonly LogLevel minLevel;
private readonly int retentionDays;
private readonly string filePrefix;
private readonly object gate = new();
private readonly ConcurrentDictionary<string, FileLogger> loggers = new();
private string currentDateStamp = "";
private StreamWriter? writer;
// Why the sink can be unusable, or null when it opened normally. The
// constructor cannot throw (see below), but a caller that has a second,
// known-writable place to complain to needs to be able to ask.
public string? SinkProblem { get; private set; }
public FileLoggerProvider(string directory, LogLevel minLevel = LogLevel.Information,
int retentionDays = 14)
int retentionDays = 14, string filePrefix = "agent")
{
this.directory = directory;
this.minLevel = minLevel;
this.retentionDays = retentionDays;
this.filePrefix = filePrefix;
// Best-effort: creating the log directory can fail on a locked-down host
// or when a non-privileged user targets a SYSTEM-owned path. Never let
// that abort construction; Write() degrades to a no-op if the sink is
// unusable.
//
// The reason is KEPT rather than discarded. A silently dead sink is what
// made the user-context probe undiagnosable: running as the operator it
// could not write into the SYSTEM-owned ProgramData log directory, so a
// probe that failed and a probe that never ran produced identical
// silence. Callers that have a writable fallback can now surface this.
try { Directory.CreateDirectory(directory); }
catch (Exception error) when (error is IOException or UnauthorizedAccessException) { }
catch (Exception error) when (error is IOException or UnauthorizedAccessException)
{
SinkProblem = $"{directory}: {error.Message}";
}
}
// The default log directory when config leaves LogPath empty:
// %ProgramData%\ShopDB\EndpointAgent\logs
//
// SERVICE-ONLY. This path is SYSTEM-owned and a signed-in operator cannot
// write to it, so anything running in the user's session must use
// UserProfileLogDirectory() instead.
public static string DefaultDirectory()
=> Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"ShopDB", "EndpointAgent", "logs");
// The shop-floor operator account. Every shop-floor endpoint uses this same
// local account name, so the path is fixed rather than resolved through the
// interactive token: the token lookup is the part that fails on a hardened
// endpoint, and when it fails the writer and the reader silently disagree
// about where the spool lives. If a site ever uses a different account only
// user-context is lost - telemetry, commands and inventory are unaffected.
public const string ShopFloorUserName = "ShopFloor";
public static string ShopFloorProfileDirectory()
=> Path.Combine(DrivesRoot, "Users", ShopFloorUserName);
private static string DrivesRoot =>
Path.GetPathRoot(Environment.GetFolderPath(Environment.SpecialFolder.Windows))
?? @"C:\";
// Where the TRAY and the user-context probe log:
// C:\Users\ShopFloor\ShopDB\Logs
// The operator owns this directory, so it is writable from the user session
// where ProgramData is not.
public static string UserProfileLogDirectory()
=> Path.Combine(ShopFloorProfileDirectory(), "ShopDB", "Logs");
public ILogger CreateLogger(string categoryName)
=> loggers.GetOrAdd(categoryName, name => new FileLogger(this, name));
@@ -85,7 +127,14 @@ public sealed class FileLoggerProvider : ILoggerProvider
writer?.Flush();
writer?.Dispose();
currentDateStamp = stamp;
var path = Path.Combine(directory, $"agent-{stamp}.log");
// One convention for every log this codebase writes:
// "<yyyyMMdd>-<prefix>.log", so agent, tray and installer logs sort
// chronologically when they share a directory and can be collected with
// a single date-leading glob. The service previously used the opposite
// order, which meant a support request for "the logs from the 5th" had to
// know which component wrote which shape.
var name = $"{stamp}-{filePrefix}.log";
var path = Path.Combine(directory, name);
writer = new StreamWriter(new FileStream(
path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
{
@@ -99,7 +148,12 @@ public sealed class FileLoggerProvider : ILoggerProvider
try
{
var cutoff = DateTime.Now.AddDays(-retentionDays);
foreach (var file in Directory.EnumerateFiles(directory, "agent-*.log"))
// Both shapes: retention has to keep covering the "<prefix>-<date>"
// files written before the convention was unified, or an upgraded
// endpoint would keep its pre-upgrade logs forever.
string[] patterns = [$"*-{filePrefix}.log", $"{filePrefix}-*.log"];
foreach (var pattern in patterns)
foreach (var file in Directory.EnumerateFiles(directory, pattern))
{
if (File.GetLastWriteTime(file) < cutoff)
File.Delete(file);

167
agent/ShopDbEndpointAgent.Core/FleetAlertClient.cs Normal file
View File

@@ -0,0 +1,167 @@
using System.Net;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Polls the fleet alert channel to fetch active notifications for this endpoint.
// The server may publish site-wide alerts (maintenance windows, policy changes)
// or endpoint-specific warnings (outdated agent version, failed telemetry push).
// The agent polls GET /api/endpoints/alerts?hostname=X every 60 seconds when idle
// and deserializes the JSON response into FleetAlert DTOs. Same auth as
// telemetry and commands (AgentHttpFactory). In 1.20 the service calls this and
// serves the tray over the local pipe (AgentPipeServer).
public sealed class FleetAlertClient : IDisposable
{
private static readonly JsonSerializerOptions JsonOptions = new()
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
};
private readonly HttpClient http;
private readonly ILogger logger;
public FleetAlertClient(AgentHttpFactory httpFactory, ILogger logger)
{
this.logger = logger;
http = httpFactory.CreateClient(AgentHttpChannel.FleetAlert);
}
// Tray compatibility until tray 1.7.0 moves to AgentPipeClient: shared key
// only, no credential. Remove with the last tray that calls it.
public FleetAlertClient(AgentConfig config, ILogger logger)
: this(new AgentHttpFactory(config, new SharedKeyOnlyProvider(config), logger), logger)
{
}
private sealed class SharedKeyOnlyProvider : ICredentialProvider
{
public SharedKeyOnlyProvider(AgentConfig config)
=> SharedKey = config.HasSharedKey() ? config.ApiKey : null;
public AgentCredential? Current => null;
public string? SharedKey { get; }
public event EventHandler? Changed { add { } remove { } }
public event EventHandler? EnrollmentNeeded { add { } remove { } }
public void ReportRejected(string code, string? sentCredential) { }
}
public async Task<List<FleetAlert>> FetchAlertsAsync(
string hostname, CancellationToken token)
{
var url = "api/endpoints/alerts?hostname="
+ Uri.EscapeDataString(hostname);
using var response = await http.GetAsync(url, token);
if (response.StatusCode == HttpStatusCode.NoContent)
return new List<FleetAlert>();
var body = await response.Content.ReadAsStringAsync(token);
if (!response.IsSuccessStatusCode)
throw new CollectorException((int)response.StatusCode, body);
var envelope = JsonSerializer.Deserialize<FleetAlertEnvelope>(
body, JsonOptions);
if (envelope?.Data is FleetAlertList parsed)
foreach (var skipped in parsed.Skipped)
logger.LogWarning("Skipped malformed fleet alert: {Reason}", skipped);
return envelope?.Data ?? new List<FleetAlert>();
}
public async Task AcknowledgeAlertAsync(
int notificationId, int assetId, string hostname, CancellationToken token)
{
var url = $"api/endpoints/alerts/{notificationId}/acknowledge";
var payload = new
{
assetid = assetId,
hostname
};
var json = JsonSerializer.Serialize(payload, JsonOptions);
var content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");
using var response = await http.PostAsync(url, content, token);
var body = await response.Content.ReadAsStringAsync(token);
if (!response.IsSuccessStatusCode)
throw new CollectorException((int)response.StatusCode, body);
}
public void Dispose() => http.Dispose();
}
// Server-sent strings may be JSON null; setters coerce to "" so consumers
// never see null. Timeout null or <= 0 means no timeout.
public sealed class FleetAlert
{
private string typeName = "";
private string typeColor = "";
private string title = "";
private string message = "";
[JsonPropertyName("notificationid")] public int NotificationId { get; set; }
[JsonPropertyName("type")] public string TypeName { get => typeName; set => typeName = value ?? ""; }
[JsonPropertyName("typecolor")] public string TypeColor { get => typeColor; set => typeColor = value ?? ""; }
[JsonPropertyName("title")] public string Title { get => title; set => title = value ?? ""; }
[JsonPropertyName("message")] public string Message { get => message; set => message = value ?? ""; }
[JsonPropertyName("timeout")] public int? Timeout { get; set; }
[JsonIgnore] public int TimeoutMinutes => Timeout is > 0 ? Timeout.Value : 0;
}
public sealed class FleetAlertEnvelope
{
[JsonPropertyName("status")] public string Status { get; set; } = "";
[JsonPropertyName("data")]
[JsonConverter(typeof(TolerantFleetAlertListConverter))]
public List<FleetAlert>? Data { get; set; }
}
// List of alerts that parsed, plus a note per element that did not. The
// caller logs Skipped; one malformed alert must not drop the rest.
public sealed class FleetAlertList : List<FleetAlert>
{
public List<string> Skipped { get; } = new();
}
// Deserializes the data array one element at a time. An element that fails
// to bind, or has no usable notificationid, is recorded and skipped.
public sealed class TolerantFleetAlertListConverter : JsonConverter<List<FleetAlert>>
{
public override List<FleetAlert>? Read(
ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
if (reader.TokenType == JsonTokenType.Null)
return null;
if (reader.TokenType != JsonTokenType.StartArray)
throw new JsonException("Fleet alert data is not an array.");
var list = new FleetAlertList();
var index = 0;
while (reader.Read() && reader.TokenType != JsonTokenType.EndArray)
{
using var element = JsonDocument.ParseValue(ref reader);
try
{
var alert = element.RootElement.Deserialize<FleetAlert>(options);
if (alert is null || alert.NotificationId <= 0)
list.Skipped.Add($"element {index}: missing notificationid");
else
list.Add(alert);
}
catch (Exception exception) when (exception is JsonException or InvalidOperationException)
{
list.Skipped.Add($"element {index}: {exception.Message}");
}
index++;
}
return list;
}
public override void Write(
Utf8JsonWriter writer, List<FleetAlert> value, JsonSerializerOptions options)
{
JsonSerializer.Serialize(writer, value.ToArray(), options);
}
}

190
agent/ShopDbEndpointAgent.Core/FleetAlertStore.cs Normal file
View File

@@ -0,0 +1,190 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Local JSON file store for tracking acknowledged fleet alerts.
// Stores notification IDs that have been acknowledged so they don't
// reappear in future polls. Lives under %LOCALAPPDATA% because the tray runs
// as the standard operator, who cannot write to %ProgramData%. Read-modify-write
// is serialized across processes by a named mutex.
public sealed class FleetAlertStore
{
private const int MaxIds = 1000;
private const string MutexName = @"Local\ShopDB-FleetAlertStore";
private static readonly TimeSpan MutexWait = TimeSpan.FromSeconds(5);
private static readonly string DataDir = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),
"ShopDB",
"FleetAlerts");
private static readonly string StoreFile = Path.Combine(DataDir, "acknowledged-alerts.json");
// Pre-1.6.11 location. Standard users could not write it, but an elevated
// run may have left IDs there; merged once.
private static readonly string LegacyStoreFile = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"ShopDB",
"FleetAlerts",
"acknowledged-alerts.json");
private readonly object fileLock = new();
private readonly ILogger logger;
private HashSet<int> acknowledged = new();
public FleetAlertStore(ILogger logger)
{
this.logger = logger;
Load();
}
public bool IsAcknowledged(int notificationId)
{
lock (fileLock)
{
return acknowledged.Contains(notificationId);
}
}
public void MarkAcknowledged(int notificationId)
{
lock (fileLock)
{
WithFileMutex(() =>
{
// Merge what another instance may have written since our load.
var onDisk = ReadFile(StoreFile);
if (onDisk?.AcknowledgedIds is not null)
acknowledged.UnionWith(onDisk.AcknowledgedIds);
acknowledged.Add(notificationId);
Prune();
Save(onDisk?.MigratedLegacy ?? true);
});
}
}
private void Load()
{
lock (fileLock)
{
WithFileMutex(() =>
{
var stored = ReadFile(StoreFile);
acknowledged = stored?.AcknowledgedIds?.ToHashSet() ?? new HashSet<int>();
if (stored?.MigratedLegacy == true)
return;
var legacy = ReadFile(LegacyStoreFile);
var merged = legacy?.AcknowledgedIds?.Count(id => acknowledged.Add(id)) ?? 0;
if (merged > 0)
{
logger.LogInformation(
"Merged {Count} acknowledged alert ID(s) from legacy store {Path}.",
merged, LegacyStoreFile);
Prune();
}
if (merged > 0 || stored is not null)
Save(migratedLegacy: true);
});
}
}
private StoredData? ReadFile(string path)
{
try
{
if (!File.Exists(path))
return null;
return JsonSerializer.Deserialize<StoredData>(File.ReadAllText(path));
}
catch (Exception ex)
{
logger.LogWarning(ex, "Could not read acknowledged alerts store {Path}.", path);
return null;
}
}
private void Save(bool migratedLegacy)
{
try
{
Directory.CreateDirectory(DataDir);
var data = new StoredData
{
AcknowledgedIds = acknowledged.ToList(),
LastUpdated = DateTime.UtcNow,
MigratedLegacy = migratedLegacy,
};
var json = JsonSerializer.Serialize(data, new JsonSerializerOptions
{
WriteIndented = true
});
var temp = StoreFile + ".tmp";
File.WriteAllText(temp, json);
File.Move(temp, StoreFile, overwrite: true);
}
catch (Exception ex)
{
logger.LogError(ex,
"Could not save acknowledged alerts store {Path}; acknowledged alerts may reappear.",
StoreFile);
}
}
// Runs action under the cross-process mutex. If the mutex cannot be had,
// logs and runs anyway: a rare clobber beats never recording an ack.
private void WithFileMutex(Action action)
{
Mutex? mutex = null;
var owned = false;
try
{
mutex = new Mutex(false, MutexName);
try
{
owned = mutex.WaitOne(MutexWait);
}
catch (AbandonedMutexException)
{
owned = true;
}
if (!owned)
logger.LogWarning("Timed out waiting for fleet alert store mutex; proceeding without it.");
}
catch (Exception ex)
{
logger.LogWarning(ex, "Could not open fleet alert store mutex; proceeding without it.");
}
try
{
action();
}
finally
{
if (owned)
mutex!.ReleaseMutex();
mutex?.Dispose();
}
}
private void Prune()
{
// Keep only newest IDs to prevent unbounded growth
if (acknowledged.Count > MaxIds)
acknowledged = acknowledged.OrderByDescending(id => id).Take(MaxIds).ToHashSet();
}
private sealed class StoredData
{
public List<int> AcknowledgedIds { get; set; } = new();
public DateTime LastUpdated { get; set; }
public bool MigratedLegacy { get; set; }
}
}

133
agent/ShopDbEndpointAgent.Core/InteractiveProcessLauncher.cs
View File

@@ -1,5 +1,7 @@
using System.ComponentModel;
using System.Runtime.InteropServices;
using System.Security.Principal;
using System.Text;
namespace ShopDbEndpointAgent.Core;
@@ -18,7 +20,11 @@ namespace ShopDbEndpointAgent.Core;
// Only process_start uses this. Stop/status/service actions stay as SYSTEM.
public static class InteractiveProcessLauncher
{
public readonly record struct LaunchOutcome(bool Launched, string? Error, int? ExitCode = null);
// TimedOut=true means the process launched fine but was still running when
// the caller's wait budget expired. It is deliberately NOT killed, so the
// work still completes; the caller just stops blocking on it.
public readonly record struct LaunchOutcome(
bool Launched, string? Error, int? ExitCode = null, bool TimedOut = false);
private const int TokenDuplicate = 0x0002;
private const int TokenQuery = 0x0008;
@@ -34,6 +40,10 @@ public static class InteractiveProcessLauncher
private const uint Infinite = 0xFFFFFFFF;
// WaitForSingleObject returns this when the budget expires with the handle
// still unsignalled.
private const uint WaitTimeout = 0x00000102;
// Returns Launched=true when CreateProcessAsUser succeeds. Never throws;
// failures come back as Launched=false with a descriptive Error so the
// caller can surface a clean CommandResult.
@@ -51,9 +61,17 @@ public static class InteractiveProcessLauncher
// needs the actual outcome (run_script in the user session) can report
// completed/failed instead of fire-and-forget "launched". GUI callers
// (process_start) leave it false and return as soon as the window is up.
//
// waitTimeout bounds that wait. This matters because the caller may be the
// telemetry loop, and telemetry IS the heartbeat: a script that hangs in the
// user session used to block here forever (the wait was hardcoded to
// Infinite), which would stall reporting and make a perfectly healthy PC
// read as offline in the WebUI. On expiry the child is left running and the
// outcome comes back with TimedOut=true. Null means wait forever and should
// only be used by a caller that is not on a timed path.
public static LaunchOutcome LaunchInActiveSession(
string applicationPath, string? arguments, string? workingDirectory,
bool waitForExit = false)
bool waitForExit = false, TimeSpan? waitTimeout = null)
{
var sessionId = WTSGetActiveConsoleSessionId();
if (sessionId == InvalidSessionId)
@@ -129,11 +147,25 @@ public static class InteractiveProcessLauncher
}
int? exitCode = null;
bool timedOut = false;
if (waitForExit && processInformation.hProcess != IntPtr.Zero)
{
WaitForSingleObject(processInformation.hProcess, Infinite);
if (GetExitCodeProcess(processInformation.hProcess, out var code))
uint milliseconds = waitTimeout is TimeSpan budget
? (uint)Math.Clamp(budget.TotalMilliseconds, 0, uint.MaxValue - 1)
: Infinite;
if (WaitForSingleObject(processInformation.hProcess, milliseconds)
== WaitTimeout)
{
// Still running. Leave it alone - killing it would abandon
// half-done work, and a file-spooling probe will still write
// its result for the next cycle to pick up.
timedOut = true;
}
else if (GetExitCodeProcess(processInformation.hProcess, out var code))
{
exitCode = unchecked((int)code);
}
}
if (processInformation.hProcess != IntPtr.Zero)
@@ -141,7 +173,7 @@ public static class InteractiveProcessLauncher
if (processInformation.hThread != IntPtr.Zero)
CloseHandle(processInformation.hThread);
return new LaunchOutcome(true, null, exitCode);
return new LaunchOutcome(true, null, exitCode, timedOut);
}
finally
{
@@ -154,6 +186,93 @@ public static class InteractiveProcessLauncher
}
}
// Profile directory of the active console user, for example
// C:\Users\ShopFloor. The agent runs as LocalSystem, so expanding
// %USERPROFILE% here would return SYSTEM's own profile under
// config\systemprofile rather than the operator's. The interactive user's
// token is the only reliable source, and this class already knows how to
// obtain it. Returns null with a reason when nobody is signed in.
public static string? TryGetActiveUserProfileDirectory(out string? error)
{
var sessionId = WTSGetActiveConsoleSessionId();
if (sessionId == InvalidSessionId)
{
error = "No active console session (no interactive user is signed in).";
return null;
}
if (!WTSQueryUserToken(sessionId, out var userToken))
{
error = "Could not obtain the interactive user token: "
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
return null;
}
try
{
uint size = 0;
GetUserProfileDirectory(userToken, null, ref size);
if (size == 0)
{
error = "Could not size the user profile directory: "
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
return null;
}
var buffer = new StringBuilder((int)size);
if (!GetUserProfileDirectory(userToken, buffer, ref size))
{
error = "Could not resolve the user profile directory: "
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
return null;
}
error = null;
return buffer.ToString();
}
finally
{
if (userToken != IntPtr.Zero)
CloseHandle(userToken);
}
}
// SID of the active console user, so SYSTEM can grant that one account
// access to a file it stages. Null with a reason when nobody is signed in.
public static SecurityIdentifier? TryGetActiveUserSid(out string? error)
{
var sessionId = WTSGetActiveConsoleSessionId();
if (sessionId == InvalidSessionId)
{
error = "No active console session (no interactive user is signed in).";
return null;
}
if (!WTSQueryUserToken(sessionId, out var userToken))
{
error = "Could not obtain the interactive user token (WTSQueryUserToken failed): "
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
return null;
}
try
{
using var identity = new WindowsIdentity(userToken);
error = identity.User is null ? "Interactive user token carries no SID." : null;
return identity.User;
}
catch (Exception exception)
{
error = "Could not read the interactive user SID: " + exception.Message;
return null;
}
finally
{
if (userToken != IntPtr.Zero)
CloseHandle(userToken);
}
}
private enum SecurityImpersonationLevel
{
SecurityAnonymous,
@@ -222,6 +341,10 @@ public static class InteractiveProcessLauncher
[DllImport("userenv.dll", SetLastError = true)]
private static extern bool DestroyEnvironmentBlock(IntPtr environment);
[DllImport("userenv.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern bool GetUserProfileDirectory(
IntPtr token, StringBuilder? profileDirectory, ref uint size);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern bool CreateProcessAsUser(
IntPtr token,

285
agent/ShopDbEndpointAgent.Core/ProtectedFileSystem.cs Normal file
View File

@@ -0,0 +1,285 @@
using System.Runtime.InteropServices;
using System.Security.AccessControl;
using System.Security.Principal;
using System.Text;
using Microsoft.Win32.SafeHandles;
namespace ShopDbEndpointAgent.Core;
// File access for paths the agent touches AS SYSTEM where a standard user could
// otherwise interfere.
//
// %ProgramData% lets any user create files and folders beneath it, so a staging
// path the agent does not lock down can be pre-created, swapped between the hash
// check and execution, or replaced with a junction that redirects a SYSTEM read
// or delete to an arbitrary file. Directories made here carry a protected DACL:
// SYSTEM and BUILTIN\Administrators only, inheritance off. SIDs are well-known
// strings so a localized OS resolves them the same way.
internal static class ProtectedFileSystem
{
private static readonly SecurityIdentifier LocalSystemSid = new("S-1-5-18");
private static readonly SecurityIdentifier AdministratorsSid = new("S-1-5-32-544");
private const uint GenericRead = 0x80000000;
private const uint FileShareRead = 0x00000001;
private const uint FileShareWrite = 0x00000002;
private const uint FileShareDelete = 0x00000004;
private const uint OpenExisting = 3;
private const uint FileFlagOpenReparsePoint = 0x00200000;
// %ProgramData%\ShopDB\EndpointAgent. The Defender-excluded agent root.
public static string AgentDataRoot => Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"ShopDB", "EndpointAgent");
// Create or re-secure <AgentDataRoot>\<name>. readerSid, when set, gets
// list/traverse on the directory itself (not inherited) so a process in that
// user's session can run from it and open files it was granted explicitly.
// Throws when the directory cannot be proven safe; callers fail closed.
public static string EnsureDirectory(string name, SecurityIdentifier? readerSid = null)
{
var root = AgentDataRoot;
Directory.CreateDirectory(root);
RefuseReparseChain(root);
var path = Path.Combine(root, name);
var attributes = TryGetAttributes(path);
if (attributes is FileAttributes existing
&& (existing.HasFlag(FileAttributes.ReparsePoint)
|| !existing.HasFlag(FileAttributes.Directory)))
{
// Link or planted file at our name. Removing a link never touches
// its target.
if (existing.HasFlag(FileAttributes.Directory))
Directory.Delete(path);
else
File.Delete(path);
attributes = null;
}
if (attributes is null)
new DirectoryInfo(path).Create(DirectorySecurityFor(readerSid));
var info = new DirectoryInfo(path);
if (!IsTrustedOwner(info))
{
// Pre-created by someone else. The owner can always rewrite the
// DACL, so re-securing is not enough: replace the directory.
Directory.Delete(path, recursive: true);
new DirectoryInfo(path).Create(DirectorySecurityFor(readerSid));
info = new DirectoryInfo(path);
if (!IsTrustedOwner(info))
throw new IOException($"Staging directory {path} has an untrusted owner");
}
if (IsReparsePoint(path))
throw new IOException($"Staging directory {path} is a reparse point");
info.SetAccessControl(DirectorySecurityFor(readerSid));
return path;
}
// New file inside a protected directory. Any existing entry at the name is
// removed first, then CreateNew so nothing can be slipped in between. The
// DACL is protected; userSid, when set, is granted userRights explicitly.
public static FileStream CreateFile(
string path, SecurityIdentifier? userSid = null,
FileSystemRights userRights = FileSystemRights.ReadAndExecute)
{
if (TryGetAttributes(path) is not null)
File.Delete(path);
var security = new FileSecurity();
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
security.AddAccessRule(new FileSystemAccessRule(
LocalSystemSid, FileSystemRights.FullControl, AccessControlType.Allow));
security.AddAccessRule(new FileSystemAccessRule(
AdministratorsSid, FileSystemRights.FullControl, AccessControlType.Allow));
if (userSid is not null)
security.AddAccessRule(new FileSystemAccessRule(
userSid, userRights, AccessControlType.Allow));
return new FileInfo(path).Create(
FileMode.CreateNew, FileSystemRights.Read | FileSystemRights.Write,
FileShare.Read, 4096, FileOptions.None, security);
}
// Attributes of the entry itself (a link reports its own, not its target's).
public static FileAttributes? TryGetAttributes(string path)
{
try
{
return File.GetAttributes(path);
}
catch (FileNotFoundException)
{
return null;
}
catch (DirectoryNotFoundException)
{
return null;
}
}
public static bool IsReparsePoint(string path)
=> TryGetAttributes(path) is FileAttributes attributes
&& attributes.HasFlag(FileAttributes.ReparsePoint);
// Open for reading without following a reparse point at the final
// component, and refuse if the opened object is one. Share mode is wide so
// files a running application holds open still read.
public static FileStream OpenReadNoFollow(string path)
{
var handle = CreateFileW(path, GenericRead,
FileShareRead | FileShareWrite | FileShareDelete, IntPtr.Zero,
OpenExisting, FileFlagOpenReparsePoint, IntPtr.Zero);
if (handle.IsInvalid)
{
var hr = Marshal.GetHRForLastWin32Error();
handle.Dispose();
throw Marshal.GetExceptionForHR(hr) ?? new IOException($"Could not open {path}");
}
try
{
if (!GetFileInformationByHandle(handle, out var information))
throw Marshal.GetExceptionForHR(Marshal.GetHRForLastWin32Error())
?? new IOException($"Could not query {path}");
var attributes = (FileAttributes)information.FileAttributes;
if (attributes.HasFlag(FileAttributes.ReparsePoint))
throw new UnauthorizedAccessException($"{path} is a reparse point");
if (attributes.HasFlag(FileAttributes.Directory))
throw new UnauthorizedAccessException($"{path} is a directory");
return new FileStream(handle, FileAccess.Read);
}
catch
{
handle.Dispose();
throw;
}
}
// The path the handle really refers to after every junction and link in
// its ancestry is resolved. Null when it cannot be read.
public static string? FinalPath(SafeFileHandle handle)
{
var buffer = new StringBuilder(1024);
var length = GetFinalPathNameByHandleW(handle, buffer, (uint)buffer.Capacity, 0);
if (length == 0)
return null;
if (length >= buffer.Capacity)
{
buffer = new StringBuilder((int)length + 1);
length = GetFinalPathNameByHandleW(handle, buffer, (uint)buffer.Capacity, 0);
if (length == 0 || length >= buffer.Capacity)
return null;
}
var final = buffer.ToString();
if (final.StartsWith(@"\\?\UNC\", StringComparison.OrdinalIgnoreCase))
return @"\\" + final[8..];
if (final.StartsWith(@"\\?\", StringComparison.Ordinal))
return final[4..];
return final;
}
// Read at most maxBytes of text from the head of a file, BOM-aware.
public static string ReadTextHeadNoFollow(string path, int maxBytes)
{
using var stream = OpenReadNoFollow(path);
var buffer = new byte[(int)Math.Min(maxBytes, Math.Max(stream.Length, 0))];
var total = 0;
int read;
while (total < buffer.Length
&& (read = stream.Read(buffer, total, buffer.Length - total)) > 0)
total += read;
using var reader = new StreamReader(
new MemoryStream(buffer, 0, total), Encoding.UTF8, true);
return reader.ReadToEnd();
}
// Delete the entry itself. DeleteFile removes a link, never its target;
// a directory at a file path is left alone.
public static void DeleteFileNoFollow(string path)
{
if (TryGetAttributes(path) is not FileAttributes attributes)
return;
if (attributes.HasFlag(FileAttributes.Directory))
throw new IOException($"{path} is a directory, not a file");
File.Delete(path);
}
private static DirectorySecurity DirectorySecurityFor(SecurityIdentifier? readerSid)
{
var security = new DirectorySecurity();
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
const InheritanceFlags inherit =
InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit;
security.AddAccessRule(new FileSystemAccessRule(LocalSystemSid,
FileSystemRights.FullControl, inherit, PropagationFlags.None,
AccessControlType.Allow));
security.AddAccessRule(new FileSystemAccessRule(AdministratorsSid,
FileSystemRights.FullControl, inherit, PropagationFlags.None,
AccessControlType.Allow));
if (readerSid is not null)
security.AddAccessRule(new FileSystemAccessRule(readerSid,
FileSystemRights.ReadAndExecute, InheritanceFlags.None,
PropagationFlags.None, AccessControlType.Allow));
return security;
}
private static bool IsTrustedOwner(DirectoryInfo info)
{
var owner = info.GetAccessControl(AccessControlSections.Owner)
.GetOwner(typeof(SecurityIdentifier)) as SecurityIdentifier;
return owner is not null
&& (owner.Equals(LocalSystemSid) || owner.Equals(AdministratorsSid));
}
// A junction anywhere from ProgramData down to the agent root would move
// every "protected" directory somewhere else.
private static void RefuseReparseChain(string root)
{
var programData = Environment.GetFolderPath(
Environment.SpecialFolder.CommonApplicationData).TrimEnd('\\');
var current = Path.GetFullPath(root).TrimEnd('\\');
while (current.Length > programData.Length
&& current.StartsWith(programData, StringComparison.OrdinalIgnoreCase))
{
if (IsReparsePoint(current))
throw new IOException($"{current} is a reparse point; refusing to stage under it");
current = Path.GetDirectoryName(current) ?? "";
}
}
[StructLayout(LayoutKind.Sequential)]
private struct ByHandleFileInformation
{
public uint FileAttributes;
public System.Runtime.InteropServices.ComTypes.FILETIME CreationTime;
public System.Runtime.InteropServices.ComTypes.FILETIME LastAccessTime;
public System.Runtime.InteropServices.ComTypes.FILETIME LastWriteTime;
public uint VolumeSerialNumber;
public uint FileSizeHigh;
public uint FileSizeLow;
public uint NumberOfLinks;
public uint FileIndexHigh;
public uint FileIndexLow;
}
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern SafeFileHandle CreateFileW(
string fileName, uint desiredAccess, uint shareMode, IntPtr securityAttributes,
uint creationDisposition, uint flagsAndAttributes, IntPtr templateFile);
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool GetFileInformationByHandle(
SafeFileHandle file, out ByHandleFileInformation information);
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern uint GetFinalPathNameByHandleW(
SafeFileHandle file, StringBuilder filePath, uint filePathLength, uint flags);
}

159
agent/ShopDbEndpointAgent.Core/ScheduledTaskPolicy.cs Normal file
View File

@@ -0,0 +1,159 @@
namespace ShopDbEndpointAgent.Core;
// Guards which registered scheduled tasks the scheduledtask_run action may
// start. The wire carries a free-text task path (so the WebUI can expose it as
// one field, and a site can widen the set without an agent rebuild); this
// policy is the safeguard that makes that free text safe to honor.
//
// WHY THIS EXISTS AT ALL. The agent runs as SYSTEM. A scheduled task is an
// INDIRECTION: the task path names a definition, and that definition chooses
// what actually runs. So "start the task named X" is only as safe as the set of
// X the agent will accept - an unconstrained task name would be a general-
// purpose "run anything already registered on this box, as whatever context it
// asked for" primitive. That is a materially bigger grant than any other
// allowlisted action, and it would be invisible in the audit trail (the command
// log would read "scheduledtask_run \Foo" no matter what \Foo does).
//
// WHAT THIS DOES AND DOES NOT BUY. A prefix allowlist stops a compromised or
// misconfigured SERVER from starting arbitrary tasks on the fleet - that is the
// threat it actually addresses, and it is the realistic one. It does NOT stop an
// attacker who can already register or rewrite task definitions on the endpoint:
// such an attacker can define a task under an approved prefix and have the agent
// start it. That is accepted, because anyone who can write task definitions on a
// box already has the privilege the task would grant them. Local task definition
// ACLs, not this policy, are the control for that.
//
// The approved-prefix list ships with the Corporate Tools set and is overridable
// per site via AgentConfig.ApprovedTaskPaths, matching the LaunchPathPolicy
// pattern so the future Site Agent Configuration tool can narrow or widen it
// without a rebuild.
public sealed class ScheduledTaskPolicy
{
// Shipped defaults: the Corporate Tools task set. Prefix entries end with a
// backslash and match a whole folder; entries without one match that exact
// task path only. Kept deliberately tight - widen per site, not here.
//
// The ShopDB-* entries are tasks ShopDB itself owns. They are separate task
// definitions from any corporate-initiated equivalent ON PURPOSE: a
// ShopDB-initiated Dell Command scan must write its results somewhere the
// agent's collector can attribute to ShopDB, rather than into whatever
// location a corporate-managed task already writes. Keeping the definitions
// distinct is what makes "who asked for this scan" answerable at all - see
// the note in commands.py about the DCU output path convention.
public static IReadOnlyList<string> DefaultApprovedTaskPaths { get; } = new[]
{
@"\ReportIP",
@"\SFLD-ApplyDSCConfig",
@"\SFLD - Consume Credentials",
@"\ShopDB-DellCommandScan",
@"\ShopDB-DellCommandUpdate",
};
private readonly IReadOnlyList<string> approvedPaths;
public ScheduledTaskPolicy(IEnumerable<string>? approvedPathsOverride = null)
{
var paths = (approvedPathsOverride ?? Enumerable.Empty<string>())
.Where(path => !string.IsNullOrWhiteSpace(path))
.Select(NormalizeTaskPath)
.ToList();
approvedPaths = paths.Count > 0 ? paths : DefaultApprovedTaskPaths;
}
// Validates a candidate task path. On success taskPath is the normalized
// path to hand to Task Scheduler; on failure reason explains the rejection
// so the command result tells the operator WHY rather than just "denied".
public bool TryValidate(string? candidate, out string taskPath, out string reason)
{
taskPath = "";
reason = "";
if (string.IsNullOrWhiteSpace(candidate))
{
reason = "empty task path";
return false;
}
var normalized = NormalizeTaskPath(candidate);
// Traversal in a task path is meaningless to Task Scheduler but very
// meaningful to a prefix check: "\Approved\..\Evil" starts with an
// approved prefix as a STRING while naming something else entirely.
// Reject rather than try to canonicalize it away.
if (normalized.Contains(".."))
{
reason = "task path contains a traversal segment";
return false;
}
// Control characters and quotes have no place in a task path and are the
// shape of an attempt to break out of the argument. The invocation uses
// ArgumentList (no shell, no string concatenation) so this is defense in
// depth rather than the primary control, but a task path that needs
// these is not a task path we should be starting.
foreach (var ch in normalized)
{
if (char.IsControl(ch) || ch == '"' || ch == '&' || ch == '|')
{
reason = "task path contains an illegal character";
return false;
}
}
if (!IsApproved(normalized))
{
reason = "task path is not in the approved task list";
return false;
}
taskPath = normalized;
return true;
}
// Task Scheduler paths are rooted and backslash-separated. Accept the
// operator writing "ReportIP", "\ReportIP", or "/ReportIP" and settle on the
// rooted form, so an allowlist entry does not hinge on how it was typed.
private static string NormalizeTaskPath(string path)
{
var trimmed = (path ?? "").Trim().Replace('/', '\\');
if (trimmed.Length == 0)
return "";
if (!trimmed.StartsWith('\\'))
trimmed = "\\" + trimmed;
// Collapse a trailing separator so "\Folder\" and "\Folder" compare
// equal as exact entries; prefix semantics are handled in IsApproved.
while (trimmed.Length > 1 && trimmed.EndsWith('\\'))
trimmed = trimmed[..^1];
return trimmed;
}
// An entry ending in a separator is a FOLDER prefix and matches anything
// beneath it; any other entry must match the whole task path exactly. The
// exact-match case is why "\Approved" does not silently authorize
// "\ApprovedEvil".
private bool IsApproved(string candidate)
{
foreach (var entry in approvedPaths)
{
if (entry.Length == 0)
continue;
var isFolderPrefix = entry.EndsWith('\\');
if (isFolderPrefix)
{
if (candidate.StartsWith(entry, StringComparison.OrdinalIgnoreCase))
return true;
continue;
}
if (string.Equals(candidate, entry, StringComparison.OrdinalIgnoreCase))
return true;
// Allow the folder form without its trailing separator to still act
// as a folder: "\SFLD" approves "\SFLD\Thing" but never "\SFLDEvil".
if (candidate.StartsWith(entry + "\\", StringComparison.OrdinalIgnoreCase))
return true;
}
return false;
}
}

9
agent/ShopDbEndpointAgent.Core/ShopDbEndpointAgent.Core.csproj
View File

@@ -15,8 +15,17 @@
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
<PackageReference Include="Microsoft.Win32.Registry" Version="5.0.0" />
<PackageReference Include="System.Management" Version="8.0.0" />
<!-- DPAPI LocalMachine for the per-PC credential (CredentialStore). -->
<PackageReference Include="System.Security.Cryptography.ProtectedData" Version="8.0.0" />
<PackageReference Include="System.Diagnostics.EventLog" Version="8.0.0" />
<PackageReference Include="System.ServiceProcess.ServiceController" Version="8.0.0" />
<!-- EventWaitHandleAcl: creating the user-context refresh event with an
explicit DACL. The service creates it as LocalSystem, and without a
granted ACE the unelevated tray cannot open it. -->
<PackageReference Include="System.Threading.AccessControl" Version="8.0.0" />
<!-- FileSystem.AccessControl: protected DACLs on staging, credential and
appsettings.json files (ProtectedFileSystem, AgentPipeServer). -->
<PackageReference Include="System.IO.FileSystem.AccessControl" Version="5.0.0" />
</ItemGroup>
</Project>

51
agent/ShopDbEndpointAgent.Core/StartupJitter.cs Normal file
View File

@@ -0,0 +1,51 @@
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// One-time startup delay, shared by TelemetryWorker and CommandWorker.
//
// Both loops otherwise make their first request the instant the service starts,
// and the service is installed start=auto - so a patch-day reboot has an entire
// fleet arriving together, twice per machine (a telemetry report AND a command
// poll). AgentConfig.JitterSeconds does not help: it spreads the gap BETWEEN
// runs, not the first contact.
//
// Lives here rather than in either worker so the two cannot drift apart, and so
// the clamp is enforced in one place.
internal static class StartupJitter
{
// Ceiling on the configured value. A fat-fingered appsettings.json must not
// be able to park an endpoint for hours before it first checks in: the
// command channel is how an operator would correct it, so an unbounded
// delay would take away the means of recovery.
private const int MaxStartupJitterSeconds = 300;
public static TimeSpan Next(int configuredSeconds, ILogger logger)
{
// Warn rather than clamp silently: an operator who deliberately set a
// large value and got a smaller one has config and behaviour disagreeing
// with no way to tell why. A negative value is a mistake too, so it is
// worth the same line.
if (configuredSeconds > MaxStartupJitterSeconds)
{
logger.LogWarning(
"startupjitterseconds {Configured} exceeds the {Max}s ceiling; using {Max}s",
configuredSeconds, MaxStartupJitterSeconds, MaxStartupJitterSeconds);
}
else if (configuredSeconds < 0)
{
logger.LogWarning(
"startupjitterseconds {Configured} is negative; startup jitter disabled",
configuredSeconds);
}
int spread = Math.Clamp(configuredSeconds, 0, MaxStartupJitterSeconds);
if (spread == 0)
return TimeSpan.Zero;
// Milliseconds, not whole seconds: 700 endpoints over a 30s window would
// otherwise land in 30 clumps of ~23 simultaneous arrivals, which is the
// same pile-up in miniature.
return TimeSpan.FromMilliseconds(Random.Shared.Next(0, spread * 1000));
}
}

293
agent/ShopDbEndpointAgent.Core/StrictModePreamble.cs Normal file
View File

@@ -0,0 +1,293 @@
using System.Text;
namespace ShopDbEndpointAgent.Core;
// Inserts the strict error-mode statement into a PowerShell script body.
//
// The statement cannot simply go first: using statements, [CmdletBinding()] and
// param(...) must precede every other statement or the script stops parsing.
// So the scanner walks the leading header (whitespace, # and <# #> comments
// including #Requires and comment help, using lines, attributes, the param
// block) and inserts on the SAME line as the first real statement, followed by
// '; ', so every line number PowerShell reports still matches the stored body.
// Anything it cannot read with confidence (named begin/process/end blocks,
// subexpressions or here-strings inside the header, unbalanced brackets, bytes
// that are not valid text) returns null and the caller skips injection.
internal static class StrictModePreamble
{
private static readonly UTF8Encoding StrictUtf8 = new(false, true);
private static readonly UTF8Encoding Utf8WithBom = new(true);
private static readonly string[] NamedBlocks =
{ "begin", "process", "end", "dynamicparam", "clean" };
// Returns the rewritten body as UTF-8 WITH BOM (Windows PowerShell 5.1
// reads a BOM-less file as the ANSI code page), or null with a reason.
public static byte[]? TryApply(byte[] body, string statement, out string? reason)
{
if (!TryDecode(body, out var text))
{
reason = "script is not valid UTF-8 or UTF-16 text";
return null;
}
var index = FindInsertionPoint(text, out reason);
if (index < 0)
return null;
var rewritten = text.Insert(index, statement + "; ");
var preamble = Utf8WithBom.GetPreamble();
var encoded = Utf8WithBom.GetBytes(rewritten);
var result = new byte[preamble.Length + encoded.Length];
preamble.CopyTo(result, 0);
encoded.CopyTo(result, preamble.Length);
return result;
}
private static bool TryDecode(byte[] body, out string text)
{
text = "";
try
{
if (body.Length >= 3 && body[0] == 0xEF && body[1] == 0xBB && body[2] == 0xBF)
text = StrictUtf8.GetString(body, 3, body.Length - 3);
else if (body.Length >= 2 && body[0] == 0xFF && body[1] == 0xFE)
text = new UnicodeEncoding(false, false, true).GetString(body, 2, body.Length - 2);
else if (body.Length >= 2 && body[0] == 0xFE && body[1] == 0xFF)
text = new UnicodeEncoding(true, false, true).GetString(body, 2, body.Length - 2);
else
text = StrictUtf8.GetString(body);
return true;
}
catch (DecoderFallbackException)
{
return false;
}
}
// Index of the first statement after the script header, or -1.
internal static int FindInsertionPoint(string text, out string? reason)
{
reason = null;
var index = SkipTrivia(text, 0);
while (index >= 0 && MatchWord(text, index, "using"))
{
var lineEnd = text.IndexOf('\n', index);
index = SkipTrivia(text, lineEnd < 0 ? text.Length : lineEnd);
}
if (index < 0)
return Uncertain("unterminated block comment in header", out reason);
var cursor = index;
while (cursor < text.Length && text[cursor] == '[')
{
cursor = SkipBalanced(text, cursor);
if (cursor < 0)
return Uncertain("could not read attribute in header", out reason);
cursor = SkipTrivia(text, cursor);
if (cursor < 0)
return Uncertain("unterminated block comment in header", out reason);
}
if (MatchWord(text, cursor, "param"))
{
var open = SkipTrivia(text, cursor + "param".Length);
if (open < 0 || open >= text.Length || text[open] != '(')
return Uncertain("could not read param block", out reason);
var close = SkipBalanced(text, open);
if (close < 0)
return Uncertain("could not read param block", out reason);
index = SkipTrivia(text, close);
if (index < 0)
return Uncertain("unterminated block comment after param block", out reason);
}
// Brackets not followed by param are a type literal starting the first
// statement, so index stays in front of them.
if (index >= text.Length)
return Uncertain("no statements after the script header", out reason);
foreach (var block in NamedBlocks)
{
if (MatchWord(text, index, block))
return Uncertain($"script uses a named '{block}' block", out reason);
}
return index;
}
private static int Uncertain(string why, out string? reason)
{
reason = why;
return -1;
}
// Skip whitespace, line comments and block comments. -1 on an unterminated
// block comment.
private static int SkipTrivia(string text, int index)
{
while (index < text.Length)
{
var character = text[index];
if (char.IsWhiteSpace(character) || character == '\uFEFF')
{
index++;
}
else if (character == '<' && index + 1 < text.Length && text[index + 1] == '#')
{
var end = text.IndexOf("#>", index + 2, StringComparison.Ordinal);
if (end < 0)
return -1;
index = end + 2;
}
else if (character == '#')
{
var lineEnd = text.IndexOf('\n', index);
index = lineEnd < 0 ? text.Length : lineEnd + 1;
}
else
{
break;
}
}
return index;
}
private static bool MatchWord(string text, int index, string word)
{
if (index < 0 || index + word.Length > text.Length)
return false;
if (string.Compare(text, index, word, 0, word.Length, StringComparison.OrdinalIgnoreCase) != 0)
return false;
var after = index + word.Length;
if (after == text.Length)
return true;
var next = text[after];
return !(char.IsLetterOrDigit(next) || next == '_' || next == '-');
}
private static bool IsSingleQuote(char character)
=> character is '\'' or '\u2018' or '\u2019' or '\u201A' or '\u201B';
private static bool IsDoubleQuote(char character)
=> character is '"' or '\u201C' or '\u201D' or '\u201E';
// From an opening ( [ or { return the index just past its matching closer,
// honoring quotes and comments. -1 when unsure.
private static int SkipBalanced(string text, int index)
{
var closers = new Stack<char>();
while (index < text.Length)
{
var character = text[index];
switch (character)
{
case '(':
closers.Push(')');
index++;
break;
case '[':
closers.Push(']');
index++;
break;
case '{':
closers.Push('}');
index++;
break;
case ')':
case ']':
case '}':
if (closers.Count == 0 || closers.Pop() != character)
return -1;
index++;
if (closers.Count == 0)
return index;
break;
case '`':
index += 2;
break;
case '@' when index + 1 < text.Length
&& (IsSingleQuote(text[index + 1]) || IsDoubleQuote(text[index + 1])):
// Here-string; not worth reading in a header.
return -1;
case '<' when index + 1 < text.Length && text[index + 1] == '#':
var end = text.IndexOf("#>", index + 2, StringComparison.Ordinal);
if (end < 0)
return -1;
index = end + 2;
break;
case '#':
if (index > 0 && (char.IsLetterOrDigit(text[index - 1]) || text[index - 1] == '_'))
return -1;
var lineEnd = text.IndexOf('\n', index);
if (lineEnd < 0)
return -1;
index = lineEnd + 1;
break;
default:
if (IsSingleQuote(character))
{
index = SkipSingleQuoted(text, index + 1);
if (index < 0)
return -1;
}
else if (IsDoubleQuote(character))
{
index = SkipDoubleQuoted(text, index + 1);
if (index < 0)
return -1;
}
else
{
index++;
}
break;
}
}
return -1;
}
private static int SkipSingleQuoted(string text, int index)
{
while (index < text.Length)
{
if (IsSingleQuote(text[index]))
{
if (index + 1 < text.Length && IsSingleQuote(text[index + 1]))
{
index += 2;
continue;
}
return index + 1;
}
index++;
}
return -1;
}
private static int SkipDoubleQuoted(string text, int index)
{
while (index < text.Length)
{
var character = text[index];
if (character == '`')
{
index += 2;
continue;
}
if (character == '$' && index + 1 < text.Length && text[index + 1] == '(')
return -1;
if (IsDoubleQuote(character))
{
if (index + 1 < text.Length && IsDoubleQuote(text[index + 1]))
{
index += 2;
continue;
}
return index + 1;
}
index++;
}
return -1;
}
}

128
agent/ShopDbEndpointAgent.Core/TcpTableReader.cs Normal file
View File

@@ -0,0 +1,128 @@
using System.Net;
using System.Runtime.InteropServices;
namespace ShopDbEndpointAgent.Core;
// TCP connection state with owning process ids, via GetExtendedTcpTable.
//
// Why P/Invoke instead of shelling out to netstat.exe: parsing netstat text
// output means spawning a process on every telemetry tick and depending on the
// column layout of a localized command. GetExtendedTcpTable is the API netstat
// itself calls, returns process ids directly, and costs a single allocation.
//
// Listeners are the cheap, bounded half of this (a handful of rows, stable over
// time) and ride the normal cadence. The full connection table is unbounded and
// churns constantly, so it stays on-demand.
public static class TcpTableReader
{
private const int AF_INET = 2;
private const uint ERROR_INSUFFICIENT_BUFFER = 122;
private const int TCP_TABLE_OWNER_PID_ALL = 5;
public enum TcpState
{
Closed = 1,
Listen = 2,
SynSent = 3,
SynReceived = 4,
Established = 5,
FinWait1 = 6,
FinWait2 = 7,
CloseWait = 8,
Closing = 9,
LastAck = 10,
TimeWait = 11,
DeleteTcb = 12,
}
public readonly record struct TcpRow(
string LocalAddress,
int LocalPort,
string RemoteAddress,
int RemotePort,
TcpState State,
int ProcessId);
// Full IPv4 TCP table. Throws only on catastrophic interop failure; callers
// treat an empty list as "not observed".
public static List<TcpRow> GetTcpTable()
{
var rows = new List<TcpRow>();
var buffer = IntPtr.Zero;
var size = 0;
try
{
// First call sizes the buffer. It is expected to fail with
// INSUFFICIENT_BUFFER; anything else is a real error.
var result = GetExtendedTcpTable(
IntPtr.Zero, ref size, false, AF_INET, TCP_TABLE_OWNER_PID_ALL, 0);
if (result != ERROR_INSUFFICIENT_BUFFER && result != 0)
return rows;
buffer = Marshal.AllocHGlobal(size);
result = GetExtendedTcpTable(
buffer, ref size, false, AF_INET, TCP_TABLE_OWNER_PID_ALL, 0);
if (result != 0)
return rows;
var entryCount = Marshal.ReadInt32(buffer);
var rowSize = Marshal.SizeOf<MIB_TCPROW_OWNER_PID>();
var cursor = IntPtr.Add(buffer, 4);
for (var index = 0; index < entryCount; index++)
{
var row = Marshal.PtrToStructure<MIB_TCPROW_OWNER_PID>(cursor);
rows.Add(new TcpRow(
FormatAddress(row.localAddress),
FormatPort(row.localPort),
FormatAddress(row.remoteAddress),
FormatPort(row.remotePort),
(TcpState)row.state,
(int)row.owningProcessId));
cursor = IntPtr.Add(cursor, rowSize);
}
}
finally
{
if (buffer != IntPtr.Zero)
Marshal.FreeHGlobal(buffer);
}
return rows;
}
public static List<TcpRow> GetListeners()
=> GetTcpTable().FindAll(row => row.State == TcpState.Listen);
private static string FormatAddress(uint address)
=> new IPAddress(BitConverter.GetBytes(address)).ToString();
// The port arrives as a big-endian value packed into the low two bytes of a
// host-order uint, so the bytes must be swapped rather than simply cast.
private static int FormatPort(uint port)
{
var bytes = BitConverter.GetBytes(port);
return (bytes[0] << 8) | bytes[1];
}
[StructLayout(LayoutKind.Sequential)]
private struct MIB_TCPROW_OWNER_PID
{
public uint state;
public uint localAddress;
public uint localPort;
public uint remoteAddress;
public uint remotePort;
public uint owningProcessId;
}
[DllImport("iphlpapi.dll", SetLastError = true)]
private static extern uint GetExtendedTcpTable(
IntPtr tcpTable,
ref int size,
bool order,
int addressFamily,
int tableClass,
int reserved);
}

155
agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs
View File

@@ -15,6 +15,11 @@ namespace ShopDbEndpointAgent.Core;
// cadence tick rather than hammering the server.
public sealed class TelemetryWorker : BackgroundService
{
// The one agent utility whose output is forwarded to the server today. The
// name is the .ps1 filename without its extension, which is how
// PowerShellUtilityRunner keys its results.
public const string DellUpdatesUtilityName = "shopdb-dellupdates";
private readonly ICollector collector;
private readonly CollectorClient client;
private readonly IAgentUtilityRunner? utilities;
@@ -22,12 +27,38 @@ public sealed class TelemetryWorker : BackgroundService
private readonly string agentVersion;
private readonly ILogger<TelemetryWorker> logger;
private readonly CollectSignal? collectSignal;
private readonly UserContextSpool? userContextSpool;
private readonly UserContextRefreshChannel? userContextRefresh;
private readonly EnrollmentService? enrollment;
private readonly Random jitter = new();
// Upper bound on holding first contact for the enrollment attempt.
private static readonly TimeSpan EnrollmentWaitLimit = TimeSpan.FromSeconds(20);
// How long to wait for the tray to answer a refresh request.
//
// Bounded, and bounded SHORT. Telemetry is this agent's liveness signal: a
// report that arrives with slightly stale mapped drives is a working
// endpoint, while a report that never arrives because we blocked on an
// absent tray is an endpoint that looks dead. The probe itself is a single
// short PowerShell run, so a tray that is alive answers well inside this;
// anything longer means nobody is home.
private static readonly TimeSpan UserContextRefreshTimeout = TimeSpan.FromSeconds(12);
// Poll interval while waiting for the stamp to advance. There is no
// completion signal coming back the other way - the tray's answer IS the
// rewritten spool file - so this is a poll by design, kept coarse enough to
// be free and fine enough not to add visible latency.
private static readonly TimeSpan UserContextPollInterval = TimeSpan.FromMilliseconds(400);
public TelemetryWorker(ICollector collector, CollectorClient client,
AgentConfig config, string agentVersion, ILogger<TelemetryWorker> logger,
IAgentUtilityRunner? utilities = null, CollectSignal? collectSignal = null)
IAgentUtilityRunner? utilities = null, CollectSignal? collectSignal = null,
UserContextSpool? userContextSpool = null,
UserContextRefreshChannel? userContextRefresh = null,
EnrollmentService? enrollment = null)
{
this.enrollment = enrollment;
this.collector = collector;
this.client = client;
this.config = config;
@@ -35,6 +66,77 @@ public sealed class TelemetryWorker : BackgroundService
this.logger = logger;
this.utilities = utilities;
this.collectSignal = collectSignal;
this.userContextSpool = userContextSpool;
this.userContextRefresh = userContextRefresh;
}
// Request a tray-side re-probe and wait, briefly, for the spool to be
// rewritten. Never throws and never fails the run.
//
// The service does NOT collect session data itself and must not start doing
// so: mapped drives belong to the operator's logon token, and a LocalSystem
// probe reads a different, near-empty drive table. See the tombstone in
// Program.cs. This method is the entire service-side involvement - ask, then
// wait for the answer to appear on disk.
private async Task RefreshUserContextAsync(CancellationToken token)
{
if (userContextSpool is null || userContextRefresh is null
|| !OperatingSystem.IsWindows())
{
return;
}
var resultPath = userContextSpool.TryGetResultPath(out _);
if (resultPath is null)
return;
// Captured BEFORE the request so "advanced" is measured against what we
// would otherwise have reported. A missing spool reads as null, and any
// stamp at all then counts as progress - that is the first-probe case.
var before = userContextSpool.TryReadCollectedAt(resultPath);
if (!userContextRefresh.RequestRefresh())
{
// Could not signal at all. Note that the inverse does NOT mean a tray
// is listening: the channel opens-or-creates and does not hold the
// handle, so signalling into an empty machine succeeds by creating
// the event and finding nobody waiting on it. Detecting an absent
// tray is the timeout's job below, not this return value's.
logger.LogDebug(
"Could not request a user-context refresh; reporting the existing spool.");
return;
}
var deadline = DateTime.UtcNow + UserContextRefreshTimeout;
while (DateTime.UtcNow < deadline)
{
try
{
await Task.Delay(UserContextPollInterval, token);
}
catch (OperationCanceledException)
{
return;
}
var after = userContextSpool.TryReadCollectedAt(resultPath);
if (after is not null && (before is null || after > before))
{
logger.LogDebug(
"User-context spool refreshed by the tray (collectedat={Stamp}).",
after);
return;
}
}
// Timed out. Proceed anyway - deliberately. Session data is one card on
// one page; telemetry carries heartbeat, inventory, events and command
// results, and holding all of that hostage to a tray that did not answer
// in twelve seconds trades a small inaccuracy for a total outage.
logger.LogDebug(
"The tray did not refresh the user-context spool within {Timeout}; "
+ "reporting with the existing session data.",
UserContextRefreshTimeout);
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
@@ -43,6 +145,37 @@ public sealed class TelemetryWorker : BackgroundService
"Endpoint agent {Version} -> {BaseUrl}, cadence {Cadence}s +/-{Jitter}s",
agentVersion, config.BaseUrl, config.CadenceSeconds, config.JitterSeconds);
// Spread first contact across the fleet. JitterSeconds only spreads the
// gap BETWEEN runs, so without this the whole fleet reports the instant
// its service starts. See StartupJitter.
var startupDelay = StartupJitter.Next(config.StartupJitterSeconds, logger);
if (startupDelay > TimeSpan.Zero)
{
logger.LogInformation(
"Delaying first telemetry report by {Delay}ms (startup jitter)",
(int)startupDelay.TotalMilliseconds);
try
{
await Task.Delay(startupDelay, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
if (enrollment is not null)
{
try
{
await enrollment.WaitForFirstAttemptAsync(EnrollmentWaitLimit, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
bool firstRun = true;
while (!stoppingToken.IsCancellationRequested)
{
@@ -111,6 +244,15 @@ public sealed class TelemetryWorker : BackgroundService
config.Monitors.Processes.Length, config.Monitors.Services.Length);
}
// Ask the tray to re-probe session data and WAIT for it, before
// collecting. Ordering is the whole point: the collector reads the spool
// file, so a refresh that lands after Collect() is a refresh the server
// does not see until the NEXT cycle. That was the bug - "Send Status"
// and "Refresh Status" both re-sent a spool nobody had rewritten, so a
// drive the operator had just unmapped kept reporting OK for up to half
// an hour and no amount of clicking changed it.
await RefreshUserContextAsync(token);
var report = collector.Collect(agentVersion, adopt);
if (utilities is not null)
@@ -118,6 +260,17 @@ public sealed class TelemetryWorker : BackgroundService
var extra = utilities.RunAll(token);
if (extra.Count > 0)
logger.LogDebug("Loaded {Count} agent utility result(s)", extra.Count);
// NARROW ON PURPOSE. The runner executes every .ps1 in the utilities
// directory, but only this one known key is attached to the report.
// Forwarding the whole dictionary would mean any script dropped into
// that directory starts uploading its output to the server on every
// cadence - see the signing caveat in PowerShellUtilityRunner's
// header. Widening this to the full map is the intended end state of
// the IAgentUtilityRunner seam, but it needs script signing or an
// allowlist first. Until then, add a key here per utility.
if (extra.TryGetValue(DellUpdatesUtilityName, out var dellUpdates))
report.DellUpdates = dellUpdates;
}
logger.LogInformation(

362
agent/ShopDbEndpointAgent.Core/UserContextProbe.cs Normal file
View File

@@ -0,0 +1,362 @@
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Collects the session-scoped data that does not exist in the agent's own
// context, by running a short PowerShell probe IN THE CALLING PROCESS'S
// SESSION and writing the result to the user's spool file.
//
// Collects only data that genuinely does not exist in the agent's own context:
// netuse - mapped drives are per logon session; "net use" from
// LocalSystem returns SYSTEM's (empty) mapping table, never the
// operator's Z: to the file server.
// userlisteners- TCP listeners owned by the signed-in user's processes, which
// is what identifies an operator-launched DNC or CMM listener.
// idleseconds - GetLastInputInfo is per session, so session 0 cannot answer
// it. Sampled at probe time only, so treat it as a coarse
// "was anyone at the machine" signal, not live presence.
//
// This runs from `shopdbagent.exe --collect-usercontext`, which the tray - and
// only the tray - invokes, because the tray is already running as the signed-in
// operator. The service does NOT call this: it previously tried to descend into
// the session with CreateProcessAsUser, which is the direction that has to
// duplicate a token, is what endpoint protection watches for, and produced
// nothing on a live endpoint. Nothing here crosses a privilege boundary now;
// the probe simply runs as whoever the tray is.
//
// The script is embedded rather than shipped as a .ps1 so the probe cannot drift
// from the agent binary or be tampered with on disk between cycles.
public sealed class UserContextProbe
{
// The spool directory is injected by BuildProbeScript rather than derived in
// the script, so the writer cannot disagree with the reader. See
// UserContextSpool.PinnedSpoolDirectory for why $env:USERPROFILE was wrong.
private const string SpoolDirectoryToken = "__SHOPDB_SPOOL_DIR__";
private const string ProbeScriptTemplate = """
$ErrorActionPreference = 'Stop'
$result = [ordered]@{}
# Injected by the agent from UserContextSpool.PinnedSpoolDirectory() so the
# writer and the reader resolve the same directory. Deliberately NOT
# $env:USERPROFILE: the tray can inherit another account's environment on a
# locked-down image, which silently wrote this file into the wrong profile.
$spool = '__SHOPDB_SPOOL_DIR__'
$target = Join-Path $spool 'usercontext.json'
$temporary = "$target.tmp"
try {
New-Item -ItemType Directory -Path $spool -Force | Out-Null
$netuse = @()
try {
Get-CimInstance -ClassName Win32_NetworkConnection -ErrorAction Stop |
ForEach-Object {
$netuse += [ordered]@{
localname = $_.LocalName
remotename = $_.RemoteName
status = $_.Status
username = $_.UserName
persistent = [bool]$_.Persistent
}
}
} catch {
$result['netuseerror'] = $_.Exception.Message
}
$result['netuse'] = $netuse
# Second, independent view of the same drives. Win32_NetworkConnection
# is the "net use" table and Get-SmbMapping is the SMB client's own,
# and they disagree: a share can be Unavailable in one and absent or
# OK in the other. Neither is authoritative, so both are reported as
# collected and the server decides what to show. Normalising them
# here would discard exactly the disagreement that is diagnostic.
$smbmapping = @()
try {
Get-SmbMapping -ErrorAction Stop |
ForEach-Object {
$smbmapping += [ordered]@{
localpath = $_.LocalPath
remotepath = $_.RemotePath
status = [string]$_.Status
}
}
} catch {
# Absent on older builds and in some session types. A missing
# source is not a drive fault, so it is recorded and left out
# rather than reported as a failed mapping.
$result['smbmappingerror'] = $_.Exception.Message
}
$result['smbmapping'] = $smbmapping
$listeners = @()
try {
$ownprocesses = @{}
Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
$ownprocesses[$_.Id] = $_.ProcessName
}
Get-NetTCPConnection -State Listen -ErrorAction Stop |
Where-Object { $ownprocesses.ContainsKey($_.OwningProcess) } |
ForEach-Object {
$listeners += [ordered]@{
localaddress = $_.LocalAddress
localport = $_.LocalPort
processid = $_.OwningProcess
processname = $ownprocesses[$_.OwningProcess]
}
}
} catch {
$result['userlistenerserror'] = $_.Exception.Message
}
$result['userlisteners'] = $listeners
# Idle time must be measured inside the user's session:
# GetLastInputInfo is per session, so asking from the agent's
# session 0 would report the service's own idle time, which is
# meaningless. This is what distinguishes "powered on" from
# "someone is actually working at this machine".
try {
# Built as a single-quoted string rather than a here-string: a
# here-string terminator must sit at column 0, which cannot be
# expressed inside an indented C# raw string literal.
$signature = 'using System; using System.Runtime.InteropServices; ' +
'public class ShopDbIdle { ' +
'[StructLayout(LayoutKind.Sequential)] ' +
'struct LASTINPUTINFO { public uint cbSize; public uint dwTime; } ' +
'[DllImport("user32.dll")] ' +
'static extern bool GetLastInputInfo(ref LASTINPUTINFO info); ' +
'public static long IdleSeconds() { ' +
'LASTINPUTINFO info = new LASTINPUTINFO(); ' +
'info.cbSize = (uint)Marshal.SizeOf(info); ' +
'if (!GetLastInputInfo(ref info)) return -1; ' +
'return (long)((uint)Environment.TickCount - info.dwTime) / 1000; } }'
Add-Type -TypeDefinition $signature -ErrorAction Stop
$idle = [ShopDbIdle]::IdleSeconds()
if ($idle -ge 0) { $result['idleseconds'] = $idle }
} catch {
$result['idlesecondserror'] = $_.Exception.Message
}
$result['collectedat'] = (Get-Date).ToUniversalTime().ToString('o')
$result['username'] = $env:USERNAME
# Stamp the token's elevation state. The username alone does not
# settle it: the SAME account elevated and unelevated gets two
# different logon tokens and therefore two different drive tables,
# so a spool from an elevated ShopFloor session passes the account
# check while still reporting every drive as broken.
try {
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
$result['elevated'] = [bool] $principal.IsInRole(
[Security.Principal.WindowsBuiltInRole]::Administrator)
} catch {
$result['elevatederror'] = $_.Exception.Message
}
# Write to a temporary name and rename into place, so the reader can
# never observe a half-written file and discard a good result.
$result | ConvertTo-Json -Depth 4 -Compress |
Set-Content -Path $temporary -Encoding UTF8 -NoNewline
Move-Item -Path $temporary -Destination $target -Force
} catch {
exit 1
}
exit 0
""";
// Substitute the resolved spool directory into the template. The path goes in
// as a PowerShell single-quoted literal, so an embedded quote is doubled and
// nothing else in the path is interpreted - backslashes and $ are literal
// inside single quotes, which is exactly what a Windows path needs.
private static string BuildProbeScript(string spoolDirectory)
=> ProbeScriptTemplate.Replace(
SpoolDirectoryToken, spoolDirectory.Replace("'", "''"));
// How long the probe is allowed to take. Generous: this no longer runs on
// the telemetry path, so an unresponsive WMI call delays nothing but itself.
private static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(60);
// powershell.exe's exit code when it rejects its command line and prints
// usage instead of running anything. Named so the failure is recognisable.
private const int PowerShellStartupFailure = -196608;
private readonly ILogger<UserContextProbe> logger;
public UserContextProbe(ILogger<UserContextProbe> logger)
{
this.logger = logger;
}
// Run the probe in THIS process's session and write the spool file.
// Returns true only when the spool file is present on disk afterwards.
//
// Called from the --collect-usercontext entry point, which the tray runs as
// the signed-in operator. Never throws: this is background collection and a
// failure must not take the tray or the agent process down with it.
public bool Collect()
{
var spoolDirectory = UserContextSpool.PinnedSpoolDirectory();
var resultPath = UserContextSpool.PinnedResultPath();
try
{
// Logged before the launch, not after. When this goes wrong the
// interesting question is always "which directory did it target",
// and a failure later in this method must not take the answer with
// it.
logger.LogInformation(
"Running the user-context probe as {User}; spool target {Path}",
Environment.UserName, resultPath);
// The probe writes a .tmp beside the target and moves it into place,
// so the directory has to exist before powershell.exe starts. The
// script creates it too, but doing it here means a permission problem
// is reported by the agent with a real exception rather than showing
// up as an opaque non-zero exit code.
try
{
Directory.CreateDirectory(spoolDirectory);
}
catch (Exception error) when (error is IOException or UnauthorizedAccessException)
{
logger.LogWarning(error,
"Could not create the user-context spool directory at {Directory}",
spoolDirectory);
return false;
}
// Passed as -EncodedCommand rather than staged as a .ps1 and run with
// -File. The script used to be written into %TEMP% and executed from
// there, which put a dependency on that file still existing, and still
// being readable, at the moment powershell.exe started. On a hardened
// endpoint that window is not safe: endpoint protection quarantines a
// freshly written script in the temp directory between the write and
// the launch, and roaming/redirected profiles can make the path
// unusable outright.
//
// When powershell.exe cannot use its -File argument it prints its usage
// banner and exits -196608 (0xFFFD0000) WITHOUT running a single
// statement. That is a startup failure, not a script failure, so the
// script's own try/catch never runs, nothing is logged about drives,
// and the operator sees a silent "no mapped drives reported" while the
// log only shows the bare exit code. -EncodedCommand carries the script
// in the command line itself, so there is no file to lose and no
// execution-policy or quarantine step between us and the probe.
var encoded = Convert.ToBase64String(
System.Text.Encoding.Unicode.GetBytes(BuildProbeScript(spoolDirectory)));
var startInfo = new System.Diagnostics.ProcessStartInfo
{
FileName = "powershell.exe",
Arguments =
"-NoProfile -NonInteractive -WindowStyle Hidden -EncodedCommand "
+ encoded,
WorkingDirectory = Path.GetTempPath(),
UseShellExecute = false,
CreateNoWindow = true,
};
using var process = System.Diagnostics.Process.Start(startInfo);
if (process is null)
{
logger.LogWarning("The user-context probe could not be started.");
return false;
}
if (!process.WaitForExit((int)ProbeTimeout.TotalMilliseconds))
{
logger.LogWarning(
"The user-context probe did not finish within {Seconds} seconds.",
ProbeTimeout.TotalSeconds);
TryKill(process);
return false;
}
if (process.ExitCode != 0)
{
// 0xFFFD0000 is powershell.exe refusing its own command line and
// printing usage: the probe never ran. Called out by name because
// as a bare number it reads like a script error and sends the
// reader looking for a WMI or drive fault that is not there.
var detail = process.ExitCode == PowerShellStartupFailure
? " (powershell.exe rejected its command line; the probe did not run)"
: string.Empty;
logger.LogWarning(
"The user-context probe exited with code {ExitCode}.{Detail}",
process.ExitCode, detail);
return false;
}
// Exit code 0 is necessary but NOT sufficient. The probe used to be
// trusted on its exit code alone, which meant a run that wrote its
// file into a different profile entirely still reported success - the
// log claimed a spool had been written for weeks while nothing was
// ever there. Confirm the file exists where the reader will look.
if (!File.Exists(resultPath))
{
logger.LogWarning(
"The user-context probe reported success but no spool file is "
+ "present at {Path}.", resultPath);
return false;
}
// Byte count alone does not distinguish a healthy spool from a
// well-formed empty one, and "0 drives" is the exact symptom this
// whole path exists to diagnose. Summarising the contents means a
// single log line from any given task run answers whether the probe
// saw the operator's drives, without needing the spool itself.
var summary = SummariseSpool(resultPath);
logger.LogInformation(
"The user-context probe wrote {Bytes} bytes to {Path} as {User}: {Summary}",
new FileInfo(resultPath).Length, resultPath, Environment.UserName, summary);
return true;
}
catch (Exception exception)
{
logger.LogWarning(exception, "The user-context probe failed.");
return false;
}
}
// One-line description of what the probe actually captured, for the run log.
// Best-effort by design: this is diagnostic colour on an already-successful
// run, so a parse problem here must never turn that run into a failure.
private static string SummariseSpool(string resultPath)
{
try
{
using var document = System.Text.Json.JsonDocument.Parse(
File.ReadAllText(resultPath));
var root = document.RootElement;
static int CountOf(System.Text.Json.JsonElement root, string name) =>
root.TryGetProperty(name, out var value)
&& value.ValueKind == System.Text.Json.JsonValueKind.Array
? value.GetArrayLength()
: 0;
var user = root.TryGetProperty("username", out var stamped)
? stamped.GetString()
: "(unstamped)";
return $"user={user} netuse={CountOf(root, "netuse")} "
+ $"smbmapping={CountOf(root, "smbmapping")} "
+ $"listeners={CountOf(root, "userlisteners")}";
}
catch (Exception)
{
return "(could not summarise)";
}
}
private void TryKill(System.Diagnostics.Process process)
{
try
{
process.Kill(entireProcessTree: true);
}
catch (Exception exception)
{
logger.LogDebug(exception, "Could not stop the timed-out user-context probe.");
}
}
}

179
agent/ShopDbEndpointAgent.Core/UserContextRefreshChannel.cs Normal file
View File

@@ -0,0 +1,179 @@
using System.Runtime.Versioning;
using System.Security.AccessControl;
using System.Security.Principal;
using System.Threading;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// The service-to-tray request channel for an on-demand user-context refresh.
//
// WHY THIS EXISTS AT ALL. Mapped drives are a property of the operator's logon
// token, not of the machine. The agent is a session-0 LocalSystem service, so it
// CANNOT enumerate them - not by impersonation, not by CreateProcessAsUser, not
// by starting the probe task itself. That approach was built once and reverted;
// see the tombstone in Program.cs where UserContextProbe is deliberately kept
// out of the service's DI container. The only process on the box that can see
// the right drive table is one already running inside the operator's session:
// the tray.
//
// So the service does not probe. It ASKS. This channel is the ask, and the
// spool file remains the answer.
//
// WHY A NAMED EVENT rather than a pipe or a socket. The request carries no data
// - it is a single bit, "please re-probe now" - and the reply already has a
// transport (the spool file, whose collectedat stamp the service watches
// advance). A kernel event is the smallest thing that crosses the session-0
// boundary, needs no serialization, no port, no firewall rule, and no cleanup if
// either side dies. A file-watcher trigger would also work but leaves litter on
// disk when the service is killed mid-request.
//
// THE Global\ PREFIX IS LOAD-BEARING. Without it the name is scoped to the
// caller's session, so the service (session 0) and the tray (session 1+) would
// create two unrelated events with the same name and never see each other. That
// failure is silent: both sides "work", nothing is ever refreshed.
//
// THE DACL IS ALSO LOAD-BEARING. The service creates the event as LocalSystem,
// which by default produces an object a non-elevated user cannot even open. The
// tray runs unelevated as ShopFloor by design (an ELEVATED tray is refused by
// the probe, because it would read the wrong drive table). So the creator grants
// Users explicit Synchronize + Modify rights, or the tray gets access-denied on
// open and the refresh silently never happens.
public sealed class UserContextRefreshChannel
{
// Session-global so session 0 and the interactive session share one object.
public const string EventName = @"Global\ShopDbEndpointAgent.UserContextRefresh";
private readonly ILogger logger;
// Takes the base ILogger rather than ILogger<T>: both the service (generic
// host) and the tray (its own file-backed logger named "Tray") construct
// this, and the tray has no DI container to resolve a typed logger from.
public UserContextRefreshChannel(ILogger logger)
{
this.logger = logger;
}
// Signal the tray that a fresh probe is wanted.
//
// Returns false when the event could not be opened or created, which is the
// normal state on a machine with nobody signed in and no tray running. The
// caller treats that as "no refresh available" and reports what it already
// has - a stale mapped-drive list is worth strictly more than a missed
// heartbeat, because telemetry IS the liveness signal.
//
// AutoReset, so exactly one waiting tray is released per request and the
// event does not latch on for every subsequent listener iteration.
[SupportedOSPlatform("windows")]
public bool RequestRefresh()
{
try
{
using var handle = OpenOrCreate(out var created);
if (handle is null)
return false;
handle.Set();
logger.LogDebug(
"Requested a user-context refresh on {Event} (created={Created})",
EventName, created);
return true;
}
catch (Exception error)
{
// Never throws to the caller. This sits directly on the telemetry
// path, and a refresh is an enhancement to the report, not a
// precondition for sending it.
logger.LogDebug(error,
"Could not request a user-context refresh on {Event}", EventName);
return false;
}
}
// Tray side: block until the service asks for a refresh, the timeout expires,
// or the token is cancelled. Returns true only for an actual request.
[SupportedOSPlatform("windows")]
public bool WaitForRequest(TimeSpan timeout, CancellationToken token)
{
try
{
using var handle = OpenOrCreate(out _);
if (handle is null)
return false;
var index = WaitHandle.WaitAny(
new[] { handle, token.WaitHandle }, timeout);
// Index 1 is cancellation, WaitTimeout is the idle path; only index 0
// is a genuine request.
return index == 0;
}
catch (Exception error)
{
logger.LogDebug(error,
"Could not wait on the user-context refresh channel {Event}", EventName);
return false;
}
}
// Open the shared event, creating it if this process is first.
//
// Either side may start first - the service can boot before anyone signs in,
// and the tray can start before the service finishes initializing - so both
// sides use the same create-or-open call rather than designating an owner.
// Whoever wins applies the DACL; the loser opens the existing object.
[SupportedOSPlatform("windows")]
private EventWaitHandle? OpenOrCreate(out bool created)
{
created = false;
try
{
var security = new EventWaitHandleSecurity();
// BUILTIN\Users by SID, not by name: on a non-English Windows the
// group is not called "Users", and this agent ships to whatever
// locale the site runs. Same reasoning as the task registration.
var users = new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null);
security.AddAccessRule(new EventWaitHandleAccessRule(
users,
EventWaitHandleRights.Synchronize | EventWaitHandleRights.Modify,
AccessControlType.Allow));
// FullControl for SYSTEM and Administrators so the service retains
// ownership semantics regardless of which side created the object.
var system = new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null);
security.AddAccessRule(new EventWaitHandleAccessRule(
system, EventWaitHandleRights.FullControl, AccessControlType.Allow));
var admins = new SecurityIdentifier(
WellKnownSidType.BuiltinAdministratorsSid, null);
security.AddAccessRule(new EventWaitHandleAccessRule(
admins, EventWaitHandleRights.FullControl, AccessControlType.Allow));
return EventWaitHandleAcl.Create(
initialState: false,
mode: EventResetMode.AutoReset,
name: EventName,
createdNew: out created,
eventSecurity: security);
}
catch (UnauthorizedAccessException)
{
// The object exists with a DACL that does not admit this caller.
// Retry read-side-only rights: the tray only ever needs to wait, and
// an event created by an older agent build may not grant Modify.
try
{
return EventWaitHandle.OpenExisting(EventName);
}
catch (Exception error)
{
logger.LogDebug(error,
"The user-context refresh event {Event} exists but is not accessible",
EventName);
return null;
}
}
}
}

389
agent/ShopDbEndpointAgent.Core/UserContextSpool.cs Normal file
View File

@@ -0,0 +1,389 @@
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// Reads the result file that the tray-run user-context probe writes into the
// signed-in user's ShopDB spool folder.
//
// Why a file and not a return value: the agent is a session-0 LocalSystem
// service, so anything session-scoped (mapped drives from "net use", per-user
// network state) simply does not exist in its own context. The probe runs as the
// operator under the tray, in a completely separate process with no pipe back to
// the service. A file in the user's own profile is the handoff, and the user can
// always write there without elevation.
//
// Freshness is judged from the 'collectedat' stamp inside the file, NOT by
// deleting the file around each read. Writer and reader are now independent
// processes on unrelated schedules, so a delete-then-read would race the tray
// and usually destroy a perfectly good result before it was ever reported. The
// file is therefore left alone and simply ignored once it ages out, which also
// means a logged-off machine stops reporting session data instead of pinning the
// last operator's mapped drives forever.
public sealed class UserContextSpool
{
public const string SpoolFolderName = "ShopDB";
public const string ResultFileName = "usercontext.json";
// The one definition of where the spool lives. Both the probe (writer) and
// this class (reader) call it, so the two can no longer disagree.
//
// The writer used to build its own path from $env:USERPROFILE while the
// reader used the pinned ShopFloor profile. On a locked-down image those are
// not the same directory - the tray is launched through explorer.exe by an
// elevated installer and can inherit another account's environment block, as
// the tray already documents for %TEMP%. The probe then wrote a perfectly
// good file into the wrong profile, exited 0, and the service found nothing:
// no spool ever appeared and no mapped drive was ever reported.
public static string PinnedSpoolDirectory()
=> Path.Combine(FileLoggerProvider.ShopFloorProfileDirectory(), SpoolFolderName);
public static string PinnedResultPath()
=> Path.Combine(PinnedSpoolDirectory(), ResultFileName);
// A probe result is small - a handful of mapped drives and listeners. A file
// larger than this is either corrupt or hostile, and we will not parse it.
private const long MaxResultBytes = 512 * 1024;
// A result older than this is treated as absent. Sized at roughly three tray
// probe intervals (30 min each), so a single missed run never blanks the card
// but genuinely stale session data retires quickly.
//
// This was 24h when the reader deleted the file after each read; the delete
// was doing the real retiring and the age check was only a backstop. Now that
// the spool persists, this value IS the staleness bound, so it is sized
// against the writer's cadence rather than the server's quiet-report window.
//
// Not read from the server: this is an offline-safe local bound, and the
// agent must retire stale session data even when it cannot reach ShopDB.
private static readonly TimeSpan MaxResultAge = TimeSpan.FromMinutes(90);
private readonly ILogger<UserContextSpool> logger;
public UserContextSpool(ILogger<UserContextSpool> logger)
{
this.logger = logger;
}
// Full path to the spool file. Fixed at C:\Users\ShopFloor\ShopDB, because
// every shop-floor endpoint signs in as the same local ShopFloor account.
//
// This used to be resolved through WTSQueryUserToken/GetUserProfileDirectory,
// and that lookup is precisely what fails on a hardened endpoint: querying
// another session's token is a privileged operation that endpoint protection
// blocks. When it failed the service simply reported "no session data" while
// the tray was writing a perfectly good file, so writer and reader silently
// disagreed about the location and the operator saw an empty card.
//
// The token lookup is kept ONLY as a fallback for a machine whose operator
// account is named something else. If both are unavailable there is no
// session data, which costs mapped drives and idle time but never affects
// telemetry, commands or inventory.
public string? TryGetResultPath(out string? reason)
{
var pinned = FileLoggerProvider.ShopFloorProfileDirectory();
if (Directory.Exists(pinned))
{
reason = null;
return PinnedResultPath();
}
var profileDirectory =
InteractiveProcessLauncher.TryGetActiveUserProfileDirectory(out var error);
if (string.IsNullOrWhiteSpace(profileDirectory))
{
reason = $"No {FileLoggerProvider.ShopFloorUserName} profile at {pinned}, "
+ "and the interactive profile could not be resolved either: "
+ (error ?? "unknown error");
return null;
}
reason = null;
return Path.Combine(profileDirectory, SpoolFolderName, ResultFileName);
}
// Remove a result file. Only used to discard one that is corrupt or has aged
// out; a valid result is left in place for the next read.
public void ClearStaleResult(string resultPath)
{
try
{
if (File.Exists(resultPath))
{
File.Delete(resultPath);
logger.LogDebug(
"Removed a stale user-context spool file at {Path}", resultPath);
}
}
catch (Exception exception)
{
// Not fatal on its own, but it does mean we cannot trust whatever we
// read next, so the caller must know.
logger.LogWarning(exception,
"Could not remove the stale user-context spool file at {Path}", resultPath);
}
}
public void EnsureSpoolDirectory(string resultPath)
{
var directory = Path.GetDirectoryName(resultPath);
if (string.IsNullOrWhiteSpace(directory))
return;
try
{
Directory.CreateDirectory(directory);
}
catch (Exception exception)
{
logger.LogWarning(exception,
"Could not create the user-context spool directory at {Directory}", directory);
}
}
// Read the probe result without consuming it. Returns null when there is no
// usable, sufficiently recent result, which the caller reports as "no session
// data this cycle" rather than as empty data - absent and empty mean
// different things here.
public Dictionary<string, object?>? TryReadResult(string resultPath)
{
try
{
if (!File.Exists(resultPath))
return null;
var info = new FileInfo(resultPath);
if (info.Length == 0)
{
logger.LogDebug("The user-context spool file was empty.");
return null;
}
if (info.Length > MaxResultBytes)
{
logger.LogWarning(
"The user-context spool file was {Bytes} bytes, above the {Limit} byte "
+ "limit; discarding it.",
info.Length, MaxResultBytes);
return null;
}
var json = File.ReadAllText(resultPath);
if (string.IsNullOrWhiteSpace(json))
return null;
var parsed = JsonSerializer.Deserialize<Dictionary<string, object?>>(json);
if (parsed is null || parsed.Count == 0)
return null;
if (IsExpired(parsed, resultPath))
return null;
if (IsWrongAccount(parsed))
return null;
if (IsElevatedCollection(parsed))
return null;
return parsed;
}
catch (JsonException exception)
{
// A truncated file is the expected shape of this failure: the probe
// was still writing when we looked. Next cycle gets a clean one.
logger.LogWarning(exception,
"The user-context spool file was not valid JSON; discarding it.");
return null;
}
catch (Exception exception)
{
logger.LogWarning(exception,
"Could not read the user-context spool file at {Path}", resultPath);
return null;
}
}
// Read the spool WITHOUT removing it. The probe overwrites the file on every
// run (write .tmp, move into place), so the newest result always replaces the
// previous one and there is nothing for the reader to clean up.
//
// This used to delete the file after a successful read, to stop a dead tray
// re-reporting the same drives forever. The collectedat age check already
// does that job, and deleting created a worse problem: lastUserContext in the
// collector is in-memory only, so a service restart wiped the cache while the
// spool had already been consumed. The agent then reported NO mapped drives
// until the tray's next probe - and the installer restarts the service and
// the tray on every install, so a fresh install was guaranteed to show an
// empty card at exactly the moment someone looked at it.
//
// The contents are the operator's own session data, visible to them in their
// own context, so leaving the file in place costs nothing.
public Dictionary<string, object?>? ConsumeResult(string resultPath)
=> TryReadResult(resultPath);
// The collectedat stamp on disk, or null when there is no readable stamp.
//
// This is how the service PROVES a refresh actually happened rather than
// assuming it. Signalling the tray is asynchronous and the probe takes a
// second or two, so a caller that simply slept would either send the old
// spool anyway or pad the delay and hope. Watching this value advance past
// the pre-request stamp is the only honest way to know the file in front of
// us is the answer to OUR request and not the previous one.
//
// Deliberately NOT routed through TryReadResult: this must report the stamp
// of whatever is on disk, including a spool that is expired, elevated, or
// from the wrong account. Those are all rejected later for reporting, but
// for ordering we only care whether the file was rewritten - conflating "not
// usable" with "not fresh" would make the service wait out the full timeout
// on every cycle for a machine whose spool is being rejected for some other
// reason.
//
// Also deliberately NOT the file's LastWriteTime, for the reason IsExpired
// gives: a spool can be copied or restored with a roaming profile and carry
// a misleading filesystem timestamp. The stamp inside the file is written by
// the probe itself.
public DateTimeOffset? TryReadCollectedAt(string resultPath)
{
try
{
if (!File.Exists(resultPath))
return null;
var info = new FileInfo(resultPath);
if (info.Length == 0 || info.Length > MaxResultBytes)
return null;
var json = File.ReadAllText(resultPath);
if (string.IsNullOrWhiteSpace(json))
return null;
var parsed = JsonSerializer.Deserialize<Dictionary<string, object?>>(json);
if (parsed is null
|| !parsed.TryGetValue("collectedat", out var raw)
|| raw is null)
{
return null;
}
if (DateTimeOffset.TryParse(
raw.ToString(),
System.Globalization.CultureInfo.InvariantCulture,
System.Globalization.DateTimeStyles.RoundtripKind,
out var collectedat))
{
return collectedat;
}
return null;
}
catch (Exception exception)
{
// A torn read while the probe is mid-write is the expected failure
// here and is not worth a warning: the caller polls, so the next
// attempt a moment later sees the completed file.
logger.LogDebug(exception,
"Could not read the user-context collectedat stamp at {Path}", resultPath);
return null;
}
}
// Reject a spool that some other account produced.
//
// Pinning the path guarantees the file lands where the reader looks, but it
// cannot guarantee WHO ran the probe. Mapped drives and idle time are per
// logon session, so a probe that ran as an administrator (for example a tray
// that inherited an elevated installer's environment) would enumerate that
// account's drive table and write it into the operator's spool. That is worse
// than reporting nothing, because the card would look populated and correct
// while showing drives the operator does not have.
//
// The probe stamps $env:USERNAME into the file, so the mismatch is checkable.
// Logged at warning, not debug: this is a misconfigured endpoint, not a quiet
// absence of data.
private bool IsWrongAccount(Dictionary<string, object?> parsed)
{
if (!parsed.TryGetValue("username", out var raw) || raw is null)
{
logger.LogWarning(
"The user-context spool has no username stamp; ignoring it because "
+ "the session it describes cannot be confirmed.");
return true;
}
var username = raw.ToString();
if (string.Equals(username, FileLoggerProvider.ShopFloorUserName,
StringComparison.OrdinalIgnoreCase))
{
return false;
}
logger.LogWarning(
"The user-context spool was written by {Actual}, not {Expected}; "
+ "ignoring it. Session data is per logon session, so this would "
+ "report the wrong account's mapped drives.",
username, FileLoggerProvider.ShopFloorUserName);
return true;
}
// An elevated probe runs under a different logon token to the operator's
// interactive one and enumerates a different, usually empty, drive table.
// The agent refuses to collect while elevated, so a spool stamped elevated
// should not exist; this rejects any that predates that guard or arrives
// from a hand-run --collect-usercontext in an admin shell.
//
// Note the asymmetry with IsWrongAccount: a MISSING stamp is accepted here
// rather than rejected. An unstamped spool is simply one written by an older
// agent, and treating that as elevated would blank out session data across a
// fleet mid-upgrade. An unstamped spool has already had to pass the username
// check to reach this point.
private bool IsElevatedCollection(Dictionary<string, object?> parsed)
{
if (!parsed.TryGetValue("elevated", out var raw) || raw is null)
return false;
var elevated = raw is JsonElement element
? element.ValueKind == JsonValueKind.True
: string.Equals(raw.ToString(), "true", StringComparison.OrdinalIgnoreCase);
if (!elevated)
return false;
logger.LogWarning(
"The user-context spool was collected by an ELEVATED process; ignoring "
+ "it. Mapped drives are per logon token, so an elevated probe reports "
+ "the wrong drive table even under the right account. Collection should "
+ "come from the 'ShopDB user-context probe' scheduled task, which runs "
+ "at RunLevel Limited.");
return true;
}
// Age is taken from the stamp the probe wrote, not from the file's last
// write time, because the file can be copied or restored with a roaming
// profile and carry a misleading timestamp. A result with no readable stamp
// is treated as expired: we cannot prove it is current, so we will not claim
// it is.
private bool IsExpired(Dictionary<string, object?> parsed, string resultPath)
{
if (!parsed.TryGetValue("collectedat", out var raw) || raw is null
|| !DateTimeOffset.TryParse(
raw.ToString(),
System.Globalization.CultureInfo.InvariantCulture,
System.Globalization.DateTimeStyles.RoundtripKind,
out var collectedat))
{
logger.LogDebug(
"The user-context spool file has no usable collectedat stamp; ignoring it.");
return true;
}
var age = DateTimeOffset.UtcNow - collectedat;
if (age <= MaxResultAge)
return false;
logger.LogDebug(
"The user-context spool file is {Hours} hours old, past the {Limit} hour limit.",
Math.Round(age.TotalHours, 1), MaxResultAge.TotalHours);
ClearStaleResult(resultPath);
return true;
}
}

51
agent/ShopDbEndpointAgent.PolicyProbe/Program.cs
View File

@@ -92,6 +92,57 @@ else
var shortRoot = Path.Combine(@"C:\PROGRA~1", "LppFixture", "good.exe");
Check("8.3 short name resolving into the approved root", true, shortRoot, policy);
// --- ScheduledTaskPolicy ----------------------------------------------------
// Pure string logic, unlike LaunchPathPolicy: a task path names an entry in the
// Task Scheduler namespace, not a file, so there is no disk state to plant. The
// cases that matter are the ones where a candidate LOOKS like an approved entry
// as a string but is not the same task.
void CheckTask(string label, bool expectAccept, string candidate, ScheduledTaskPolicy policy)
{
var ok = policy.TryValidate(candidate, out var full, out var reason);
var pass = ok == expectAccept;
if (!pass) failures++;
var verdict = pass ? "PASS" : "FAIL";
var got = ok ? $"accepted -> {full}" : $"rejected ({reason})";
Console.WriteLine($"{verdict} {label}");
Console.WriteLine($" want={(expectAccept ? "accept" : "reject")} got={got}");
}
Console.WriteLine();
Console.WriteLine("--- ScheduledTaskPolicy ---");
var taskPolicy = new ScheduledTaskPolicy(new[] { @"\ReportIP", @"\SFLD\" });
CheckTask("exact approved task", true, @"\ReportIP", taskPolicy);
// Task Scheduler is case-insensitive, so the policy must be too, or the same
// task typed differently would be refused.
CheckTask("approved task, different case", true, @"\reportip", taskPolicy);
// The operator should not have to know the rooted form.
CheckTask("approved task without leading separator", true, "ReportIP", taskPolicy);
CheckTask("approved task with forward slashes", true, "/ReportIP", taskPolicy);
// The prefix-suffix confusion this policy exists to prevent: "\ReportIP" must
// NOT authorize "\ReportIPEvil".
CheckTask("sibling task sharing an approved prefix", false, @"\ReportIPEvil", taskPolicy);
CheckTask("unapproved task", false, @"\Evil", taskPolicy);
CheckTask("empty task path", false, "", taskPolicy);
// Folder entries authorize everything beneath them, but not a sibling folder
// whose name merely starts the same way.
CheckTask("task inside an approved folder", true, @"\SFLD\ApplyDSCConfig", taskPolicy);
CheckTask("folder itself", true, @"\SFLD", taskPolicy);
CheckTask("sibling folder sharing the prefix", false, @"\SFLDEvil\Thing", taskPolicy);
// Traversal would let an approved prefix string-match something else entirely.
CheckTask("traversal out of an approved folder", false, @"\SFLD\..\Evil", taskPolicy);
// Defense in depth: the invocation uses ArgumentList, but a task path with
// these characters is not one we should be starting.
CheckTask("argument-injection characters", false, "\\ReportIP\" & calc", taskPolicy);
CheckTask("control character", false, "\\Report\u0000IP", taskPolicy);
// An empty override must fall back to the shipped defaults, never to "allow
// everything".
var defaultPolicy = new ScheduledTaskPolicy(Array.Empty<string>());
CheckTask("shipped default is honored when override is empty", true, @"\ReportIP", defaultPolicy);
CheckTask("empty override does not mean allow-all", false, @"\Evil", defaultPolicy);
Console.WriteLine();
Console.WriteLine(failures == 0 ? "ALL EXPECTATIONS MET" : $"{failures} EXPECTATION(S) NOT MET");
return failures == 0 ? 0 : 1;

4
agent/ShopDbEndpointAgent.PolicyProbe/ShopDbEndpointAgent.PolicyProbe.csproj
View File

@@ -16,5 +16,9 @@
</PropertyGroup>
<ItemGroup>
<Compile Include="../ShopDbEndpointAgent.Core/LaunchPathPolicy.cs" Link="LaunchPathPolicy.cs" />
<!-- ScheduledTaskPolicy is pure string logic with no filesystem state to
plant, so unlike LaunchPathPolicy it would run anywhere. It is linked
here anyway to keep both allowlist policies provable from one harness. -->
<Compile Include="../ShopDbEndpointAgent.Core/ScheduledTaskPolicy.cs" Link="ScheduledTaskPolicy.cs" />
</ItemGroup>
</Project>

457
agent/ShopDbEndpointAgent.Tray/FleetAlertForm.cs Normal file
View File

@@ -0,0 +1,457 @@
using System.Drawing;
using System.Drawing.Drawing2D;
using Microsoft.Extensions.Logging;
using ShopDbEndpointAgent.Core;
namespace ShopDbEndpointAgent.Tray;
// Displays fleet-wide alert notifications, relayed by the local agent, in a toast-style
// window. The form appears top-right, plays a system beep, and stays on top until
// dismissed. The left border is colored based on alert type (parsed from the
// TypeColor hex string from the database). The operator acknowledges the alert by
// clicking the button or the X with the mouse. The form steals focus when shown,
// so neither button is a tab stop, there is no Accept/Cancel button, keyboard
// clicks are ignored, and mouse clicks are ignored until ArmDelay has passed:
// keystrokes meant for another app must not acknowledge an unseen alert.
// Closing any other way (server withdrew it, timeout, shutdown) is NOT an
// acknowledgement.
internal sealed class FleetAlertForm : Form
{
private const int MaxMessageLength = 8000;
private const string DefaultTypeName = "Alert";
private const string DefaultTitle = "(no title)";
private static readonly TimeSpan ArmDelay = TimeSpan.FromMilliseconds(1500);
private static readonly Color DefaultTypeColor = Color.Gray;
private readonly FleetAlert alert;
private readonly AgentPipeClient? client;
private readonly ILogger? logger;
private System.Windows.Forms.Timer? countdownTimer;
private System.Windows.Forms.Timer? statusCheckTimer;
private readonly CancellationTokenSource statusCheckCancellation = new();
private int remainingSeconds;
private Label? countdownLabel;
private TextBox? messageBox;
private readonly List<Font> fonts = new();
private DateTime armedAtUtc = DateTime.MaxValue;
private bool acknowledged;
public event EventHandler? OnAcknowledgeClicked;
public FleetAlertForm(
FleetAlert alert, AgentPipeClient? client = null, ILogger? logger = null)
{
this.alert = alert ?? throw new ArgumentNullException(nameof(alert));
this.client = client;
this.logger = logger;
SuspendLayout();
AutoScaleDimensions = new SizeF(96F, 96F);
AutoScaleMode = AutoScaleMode.Dpi;
Text = string.Empty;
FormBorderStyle = FormBorderStyle.None;
TopMost = true;
ShowInTaskbar = false;
StartPosition = FormStartPosition.Manual;
ClientSize = new Size(400, 320);
BackColor = Color.White;
KeyPreview = false;
BuildLayout();
ResumeLayout(false);
Load += FleetAlertForm_Load;
Shown += FleetAlertForm_Shown;
Resize += (s, e) => ApplyRoundedCorners();
FormClosing += FleetAlertForm_FormClosing;
}
private void FleetAlertForm_FormClosing(object? sender, FormClosingEventArgs e)
{
countdownTimer?.Stop();
statusCheckTimer?.Stop();
statusCheckCancellation.Cancel();
}
private void FleetAlertForm_Load(object? sender, EventArgs e)
{
// Size is final (DPI scaled) by Load; place and shape here.
var screen = Screen.PrimaryScreen?.WorkingArea ?? Screen.PrimaryScreen?.Bounds ?? new Rectangle(0, 0, 1920, 1080);
Location = new Point(screen.Right - Width - 20, screen.Top + 20);
ApplyRoundedCorners();
System.Media.SystemSounds.Beep.Play();
if (client != null)
{
StartStatusCheck();
}
}
private void FleetAlertForm_Shown(object? sender, EventArgs e)
{
armedAtUtc = DateTime.UtcNow + ArmDelay;
if (messageBox != null)
{
ActiveControl = messageBox;
messageBox.Select(0, 0);
}
}
private void StartStatusCheck()
{
statusCheckTimer = new System.Windows.Forms.Timer { Interval = 30000 };
statusCheckTimer.Tick += StatusCheckTimer_Tick;
statusCheckTimer.Start();
}
private void StatusCheckTimer_Tick(object? sender, EventArgs e)
{
if (client == null)
return;
var token = statusCheckCancellation.Token;
if (token.IsCancellationRequested)
return;
Task.Run(async () =>
{
try
{
var alerts = await client.GetAlertsAsync(token)
.ConfigureAwait(false);
var stillActive = alerts.Any(a => a.NotificationId == alert.NotificationId);
if (!stillActive && !token.IsCancellationRequested && IsHandleCreated && !IsDisposed)
{
BeginInvoke(new Action(() =>
{
statusCheckTimer?.Stop();
Close();
}));
}
}
catch (OperationCanceledException) when (token.IsCancellationRequested)
{
// Form is closing
}
catch (Exception exception)
{
// Agent unreachable or refused: keep the form open
logger?.LogDebug(exception,
"Status check for fleet alert {NotificationId} failed.", alert.NotificationId);
}
});
}
private void ApplyRoundedCorners()
{
using var path = new GraphicsPath();
int radius = LogicalToDeviceUnits(8);
var rect = new Rectangle(0, 0, Width, Height);
path.AddArc(rect.X, rect.Y, radius, radius, 180, 90);
path.AddArc(rect.Right - radius, rect.Y, radius, radius, 270, 90);
path.AddArc(rect.Right - radius, rect.Bottom - radius, radius, radius, 0, 90);
path.AddArc(rect.X, rect.Bottom - radius, radius, radius, 90, 90);
path.CloseFigure();
var old = Region;
Region = new Region(path);
old?.Dispose();
}
// Not wired (1.6.2 made alerts manual-only). If wired, a timeout closes the
// form WITHOUT acknowledging, so the alert can be surfaced again.
private void StartCountdown()
{
var minutes = alert.TimeoutMinutes;
if (minutes <= 0)
return;
remainingSeconds = minutes * 60;
countdownTimer = new System.Windows.Forms.Timer { Interval = 1000 };
countdownTimer.Tick += CountdownTimer_Tick;
countdownTimer.Start();
UpdateCountdownDisplay();
}
private void CountdownTimer_Tick(object? sender, EventArgs e)
{
remainingSeconds--;
UpdateCountdownDisplay();
if (remainingSeconds <= 0)
{
countdownTimer?.Stop();
Close();
}
}
private void UpdateCountdownDisplay()
{
if (countdownLabel != null)
{
int minutes = remainingSeconds / 60;
int seconds = remainingSeconds % 60;
countdownLabel.Text = $"Closes in {minutes}:{seconds:D2}";
}
}
private Font CreateFont(string family, float size, FontStyle style = FontStyle.Regular)
{
var font = new Font(family, size, style);
fonts.Add(font);
return font;
}
private static string SafeText(string? value, string fallback)
{
return string.IsNullOrWhiteSpace(value) ? fallback : value.Trim();
}
private static string PrepareMessage(string? value)
{
var text = value ?? string.Empty;
if (text.Length > MaxMessageLength)
text = text.Substring(0, MaxMessageLength) + "...";
return text.Replace("\r\n", "\n").Replace('\r', '\n').Replace("\n", "\r\n");
}
private void BuildLayout()
{
// Dock order: Fill added first, outermost edges added last.
var contentPanel = new Panel
{
Dock = DockStyle.Fill,
BackColor = Color.White,
};
Controls.Add(contentPanel);
var borderPanel = new Panel
{
BackColor = ParseTypeColor(alert.TypeColor),
Dock = DockStyle.Left,
Width = 5,
};
Controls.Add(borderPanel);
var messagePanel = new Panel
{
Dock = DockStyle.Fill,
Padding = new Padding(20, 4, 12, 4),
BackColor = Color.White,
};
contentPanel.Controls.Add(messagePanel);
messageBox = new TextBox
{
Text = PrepareMessage(alert.Message),
Dock = DockStyle.Fill,
Multiline = true,
ReadOnly = true,
WordWrap = true,
ScrollBars = ScrollBars.Vertical,
BorderStyle = BorderStyle.None,
BackColor = Color.White,
Font = CreateFont("Segoe UI", 9.75f),
ForeColor = Color.FromArgb(85, 85, 85),
TabIndex = 0,
};
messagePanel.Controls.Add(messageBox);
var titleLabel = new Label
{
Text = SafeText(alert.Title, DefaultTitle),
Dock = DockStyle.Top,
Height = 50,
Padding = new Padding(16, 8, 16, 0),
Font = CreateFont("Segoe UI", 11, FontStyle.Bold),
ForeColor = Color.FromArgb(51, 51, 51),
AutoSize = false,
AutoEllipsis = true,
UseMnemonic = false,
};
contentPanel.Controls.Add(titleLabel);
var buttonPanel = new Panel
{
Dock = DockStyle.Bottom,
Height = 46,
BackColor = Color.White,
};
contentPanel.Controls.Add(buttonPanel);
var acknowledgeButton = new MouseOnlyButton(IsArmed)
{
Text = "Acknowledge",
Width = 140,
Height = 32,
Left = (395 - 140) / 2,
Top = 7,
Anchor = AnchorStyles.None,
BackColor = Color.FromArgb(76, 175, 80),
ForeColor = Color.White,
FlatStyle = FlatStyle.Flat,
Font = CreateFont("Segoe UI", 9, FontStyle.Bold),
Cursor = Cursors.Hand,
TabStop = false,
UseMnemonic = false,
};
acknowledgeButton.FlatAppearance.BorderSize = 0;
acknowledgeButton.FlatAppearance.MouseOverBackColor = Color.FromArgb(69, 160, 73);
acknowledgeButton.Click += AcknowledgeButton_Click;
buttonPanel.Controls.Add(acknowledgeButton);
var footerPanel = new Panel
{
Dock = DockStyle.Bottom,
Height = 40,
BackColor = Color.White,
};
contentPanel.Controls.Add(footerPanel);
countdownLabel = new Label
{
Text = "Manual acknowledgment required",
Dock = DockStyle.Fill,
TextAlign = ContentAlignment.MiddleCenter,
Font = CreateFont("Segoe UI", 8.25f),
ForeColor = Color.FromArgb(153, 153, 153),
UseMnemonic = false,
};
footerPanel.Controls.Add(countdownLabel);
var footerBorder = new Panel
{
Dock = DockStyle.Top,
Height = 1,
BackColor = Color.FromArgb(238, 238, 238),
};
footerPanel.Controls.Add(footerBorder);
var headerPanel = new Panel
{
Dock = DockStyle.Top,
Height = 45,
BackColor = Color.FromArgb(245, 245, 245),
};
contentPanel.Controls.Add(headerPanel);
var typeLabel = new Label
{
Text = SafeText(alert.TypeName, DefaultTypeName).ToUpperInvariant(),
Dock = DockStyle.Fill,
Padding = new Padding(12, 0, 0, 0),
TextAlign = ContentAlignment.MiddleLeft,
AutoEllipsis = true,
Font = CreateFont("Segoe UI", 10, FontStyle.Bold),
ForeColor = Color.FromArgb(51, 51, 51),
UseMnemonic = false,
};
headerPanel.Controls.Add(typeLabel);
var closeButton = new MouseOnlyButton(IsArmed)
{
Text = "X",
Dock = DockStyle.Right,
Width = 40,
FlatStyle = FlatStyle.Flat,
Font = CreateFont("Segoe UI", 11),
ForeColor = Color.FromArgb(102, 102, 102),
BackColor = Color.Transparent,
Cursor = Cursors.Hand,
TabStop = false,
UseMnemonic = false,
};
closeButton.FlatAppearance.BorderSize = 0;
closeButton.FlatAppearance.MouseOverBackColor = Color.FromArgb(224, 224, 224);
closeButton.Click += CloseButton_Click;
headerPanel.Controls.Add(closeButton);
}
private bool IsArmed() => DateTime.UtcNow >= armedAtUtc;
private static Color ParseTypeColor(string? hexColor)
{
if (string.IsNullOrWhiteSpace(hexColor))
return DefaultTypeColor;
var hex = hexColor.Trim().TrimStart('#');
if (hex.Length != 6 || !int.TryParse(hex, System.Globalization.NumberStyles.AllowHexSpecifier,
System.Globalization.CultureInfo.InvariantCulture, out var rgb))
return DefaultTypeColor;
return Color.FromArgb((rgb >> 16) & 0xFF, (rgb >> 8) & 0xFF, rgb & 0xFF);
}
private void Acknowledge()
{
if (acknowledged)
return;
acknowledged = true;
countdownTimer?.Stop();
OnAcknowledgeClicked?.Invoke(this, EventArgs.Empty);
DialogResult = DialogResult.OK;
Close();
}
private void AcknowledgeButton_Click(object? sender, EventArgs e) => Acknowledge();
private void CloseButton_Click(object? sender, EventArgs e) => Acknowledge();
protected override void Dispose(bool disposing)
{
if (disposing)
{
countdownTimer?.Stop();
countdownTimer?.Dispose();
statusCheckTimer?.Stop();
statusCheckTimer?.Dispose();
statusCheckCancellation.Cancel();
statusCheckCancellation.Dispose();
foreach (var font in fonts)
{
font.Dispose();
}
fonts.Clear();
}
base.Dispose(disposing);
}
// Raises Click only for a left mouse release on the button once armed.
// Space, Enter, mnemonics and PerformClick all route through OnClick
// without passing OnMouseUp, so they are dropped.
private sealed class MouseOnlyButton : Button
{
private readonly Func<bool> isArmed;
private bool inMouseUp;
public MouseOnlyButton(Func<bool> isArmed)
{
this.isArmed = isArmed;
}
protected override void OnMouseUp(MouseEventArgs mevent)
{
inMouseUp = true;
try
{
base.OnMouseUp(mevent);
}
finally
{
inMouseUp = false;
}
}
protected override void OnClick(EventArgs e)
{
if (inMouseUp && isArmed())
base.OnClick(e);
}
}
}

269
agent/ShopDbEndpointAgent.Tray/FleetAlertWorker.cs Normal file
View File

@@ -0,0 +1,269 @@
using Microsoft.Extensions.Logging;
using ShopDbEndpointAgent.Core;
namespace ShopDbEndpointAgent.Tray;
// Asks the local agent for fleet-wide alerts every 60 seconds and displays new
// alerts to the operator via FleetAlertForm. The agent holds the credential and
// talks to the server; the tray only speaks the pipe. An alert is recorded in
// FleetAlertStore only once the agent confirms the acknowledgement, so a failed
// ack lets it reappear. Runs for the tray's lifetime on a background thread.
internal sealed class FleetAlertWorker : IDisposable
{
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
private static readonly TimeSpan StartupDelay = TimeSpan.FromSeconds(15);
// Agent down polls every cycle; warn once, then at most this often.
private static readonly TimeSpan UnreachableLogInterval = TimeSpan.FromMinutes(15);
private readonly AgentPipeClient client;
private readonly FleetAlertStore store;
private readonly ILogger logger;
private readonly SynchronizationContext? uiContext;
private readonly CancellationTokenSource cancellation;
private readonly Thread workerThread;
// Set and cleared on the UI thread only.
private FleetAlertForm? openForm;
// Poller thread only.
private DateTime lastUnreachableLogUtc = DateTime.MinValue;
private int unreachableCount;
public FleetAlertWorker(
AgentPipeClient client,
FleetAlertStore store,
ILogger logger,
SynchronizationContext? uiContext = null)
{
this.client = client ?? throw new ArgumentNullException(nameof(client));
this.store = store ?? throw new ArgumentNullException(nameof(store));
this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
this.uiContext = uiContext ?? SynchronizationContext.Current;
cancellation = new CancellationTokenSource();
workerThread = new Thread(Run)
{
IsBackground = true,
Name = "ShopDB fleet alert poller",
};
}
public void Start()
{
logger.LogInformation(
"Fleet alert worker starting; first poll in {Delay}, then every {Interval}.",
StartupDelay, PollInterval);
workerThread.Start();
}
private void Run()
{
var token = cancellation.Token;
if (token.WaitHandle.WaitOne(StartupDelay))
return;
while (!token.IsCancellationRequested)
{
try
{
PollAndDisplayAlerts(token);
}
catch (OperationCanceledException) when (token.IsCancellationRequested)
{
return;
}
catch (AgentPipeException exception)
{
LogUnreachable(exception);
}
catch (Exception exception)
{
logger.LogWarning(exception, "Fleet alert poll failed; retrying on next cycle.");
}
if (token.WaitHandle.WaitOne(PollInterval))
return;
}
}
private void LogUnreachable(AgentPipeException exception)
{
unreachableCount++;
var now = DateTime.UtcNow;
if (now - lastUnreachableLogUtc < UnreachableLogInterval)
return;
lastUnreachableLogUtc = now;
logger.LogWarning(
"Could not get fleet alerts from the agent ({Count} failed poll(s)): {Error}. "
+ "Retrying on next cycle.", unreachableCount, exception.Message);
}
private void PollAndDisplayAlerts(CancellationToken token)
{
var alerts = client.GetAlertsAsync(token).GetAwaiter().GetResult();
if (unreachableCount > 0)
{
logger.LogInformation(
"Agent answered the alert poll again after {Count} failed poll(s).", unreachableCount);
unreachableCount = 0;
lastUnreachableLogUtc = DateTime.MinValue;
}
if (alerts is FleetAlertList { Skipped.Count: > 0 } parsed)
{
logger.LogWarning(
"Skipped {Count} malformed fleet alert(s): {Details}",
parsed.Skipped.Count, string.Join("; ", parsed.Skipped));
}
if (alerts.Count == 0)
{
logger.LogDebug("No active fleet alerts.");
return;
}
var newAlerts = alerts
.Where(alert => !store.IsAcknowledged(alert.NotificationId))
.ToList();
if (newAlerts.Count == 0)
{
logger.LogDebug("All {Count} fleet alerts already acknowledged.", alerts.Count);
return;
}
logger.LogInformation("Found {Count} new fleet alert(s).", newAlerts.Count);
foreach (var alert in newAlerts)
{
if (token.IsCancellationRequested)
return;
ShowAlertOnUIThread(alert, token);
}
}
private void ShowAlertOnUIThread(FleetAlert alert, CancellationToken token)
{
var shown = new ManualResetEventSlim(false);
Exception? formException = null;
var showAction = new SendOrPostCallback(_ =>
{
try
{
if (token.IsCancellationRequested)
return;
using var form = new FleetAlertForm(alert, client, logger);
form.OnAcknowledgeClicked += (s, e) => OnAlertAcknowledged(alert, token);
openForm = form;
try
{
form.ShowDialog();
}
finally
{
openForm = null;
}
}
catch (Exception exception)
{
formException = exception;
}
finally
{
shown.Set();
}
});
if (uiContext is not null)
uiContext.Post(showAction, null);
else
showAction(null);
shown.Wait(token);
if (formException is not null)
{
logger.LogWarning(formException,
"Could not display fleet alert {NotificationId}.", alert.NotificationId);
}
}
private void OnAlertAcknowledged(FleetAlert alert, CancellationToken token)
{
logger.LogInformation(
"Fleet alert {NotificationId} acknowledged; sending to the agent.", alert.NotificationId);
// Off the UI thread. Recorded locally only on Ok, so a failed ack
// shows the alert again on a later poll instead of losing it.
Task.Run(async () =>
{
try
{
var response = await client.AckAlertAsync(alert.NotificationId, token);
if (response.Ok)
{
store.MarkAcknowledged(alert.NotificationId);
logger.LogInformation("Fleet alert {NotificationId} acknowledged by the agent.",
alert.NotificationId);
}
else
{
logger.LogWarning(
"Agent did not acknowledge fleet alert {NotificationId}: {Error}. It may be shown again.",
alert.NotificationId, response.Error);
}
}
catch (OperationCanceledException) when (token.IsCancellationRequested)
{
logger.LogInformation(
"Acknowledgement of fleet alert {NotificationId} cancelled by shutdown.",
alert.NotificationId);
}
catch (Exception exception)
{
logger.LogWarning(exception,
"Could not acknowledge fleet alert {NotificationId}; it may be shown again.",
alert.NotificationId);
}
});
}
// Closes a displayed alert without acknowledging it, so the form's status
// check stops polling the agent on shutdown.
public void CloseOpenAlert()
{
var form = openForm;
if (form is null || form.IsDisposed)
return;
try
{
if (form.InvokeRequired)
form.BeginInvoke(new Action(form.Close));
else
form.Close();
}
catch (Exception exception)
{
logger.LogWarning(exception, "Could not close open fleet alert on shutdown.");
}
}
public void Dispose()
{
cancellation.Cancel();
CloseOpenAlert();
if (workerThread.IsAlive)
{
if (!workerThread.Join(TimeSpan.FromSeconds(5)))
{
logger.LogWarning("Fleet alert worker thread did not exit within timeout.");
}
}
cancellation.Dispose();
}
}

113
agent/ShopDbEndpointAgent.Tray/ShopDbEndpointAgent.Tray.csproj
View File

@@ -8,11 +8,122 @@
<UseWindowsForms>true</UseWindowsForms>
<RootNamespace>ShopDbEndpointAgent.Tray</RootNamespace>
<AssemblyName>shopdb-endpoint-tray</AssemblyName>
<!-- 1.1.0: the tray now collects session-scoped user context. It invokes
shopdbagent.exe with the collect-usercontext verb on a timer (once
shortly after logon, then periodically) and the agent service reads the
resulting spool file. The tray owns this because it is the only ShopDB
process already running as the operator: mapped drives, user-owned
listeners, and input idle time do not exist in the service's session 0.
Requires agent 1.8.0 or newer: an older agent does not recognize the
verb and falls through to starting a full host, so the tray and agent
must be deployed together. -->
<!-- 1.0.0: first explicitly stamped tray version. The tray and the agent
service version independently (they are separate exes and can be pushed
apart), so the tray carries its own <Version> and the agent reports it
as trayversion alongside its own agentversion (AGENT-CONTRACT.md 2). -->
<Version>1.0.0</Version>
<!-- 1.1.1: the tray now logs. Every user-context failure path was silent -
a missing shopdbagent.exe returned bare, and the probe launch was
wrapped in an empty catch - so "no mapped drives" produced no evidence
anywhere on the machine. The tray writes
C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log (the operator's
own profile; it cannot write to ProgramData or Program Files) covering
tray startup, agent-exe resolution, probe launch and probe exit code. -->
<!-- 1.2.0: the tray stops claiming success it did not verify. The probe
completion message printed a hardcoded spool path regardless of what
happened, so a probe that exited 0 without writing anything read as a
clean collection; it now reports the real pinned path and distinguishes
spool-present, exit-0-but-no-file, and non-zero exit. Startup probe
delay is chosen from session uptime: the 2-minute delay still applies at
logon (drive mappings are not established yet), but a tray started into
an already-running session - which is what an installer does - probes
after 15 seconds instead of leaving a fresh install blind for the full
30-minute cycle. A log sink that cannot open its directory raises a
one-time balloon, since it cannot report its own failure through the
log. Requires agent 1.9.0: the two carry the shared pinned spool path
and must be deployed together. -->
<!-- 1.3.0: the tray is no longer the authoritative user-context collector.
The 'ShopDB user-context probe' scheduled task now owns collection,
because it is the only mechanism that runs in the operator's own logon
session at RunLevel Limited; the tray keeps its own probe cadence as
belt-and-braces and writes the same spool. The installer no longer
starts the tray directly (explorer.exe de-elevated it but did not
change ACCOUNT, so an admin install produced an admin tray whose
probe saw the wrong drive table). Autostart at logon is the Run key;
install-time startup on a machine someone is already signed in to is
a transient 'ShopDB tray first launch' scheduled task, which is the
only way to enter the operator's existing session unelevated. Probe
exit code 2 - the agent refusing to collect while elevated - is
reported by name instead of as a bare code. Requires agent 1.10.1. -->
<!-- 1.4.0: the tray answers on-demand refresh requests. A background
listener waits on Global\ShopDbEndpointAgent.UserContextRefresh and
runs the probe when the service asks, which is what makes a WebUI
"Refresh Status" click produce current mapped drives instead of the
last cadence's answer - the service cannot collect this itself at any
price, since drive letters live on the operator's logon token, so the
request has to be serviced HERE. Each probe is marshalled onto the UI
thread: the cadence timer and a service request could otherwise both
pass the in-flight guard and race the same spool file. "Send Status"
now awaits a probe before invoking the once verb, because that verb
reads the spool and would otherwise publish the previous probe's
answer to an operator who just changed something and is watching
for it. The probe
body is one shared method used by all three callers rather than three
copies of the exit-code handling. A tray that cannot open the channel
logs and keeps its timer, so the failure mode is "refresh is not
immediate", not "session data stops working". Requires agent 1.11.0. -->
<!-- 1.6.1: fix crash when clicking acknowledge button. The countdown timer
is now properly stopped before the form closes to prevent disposal issues. -->
<!-- 1.6.2: disable auto-dismiss timer. Fleet alerts now require manual
acknowledgment only. Footer shows "Manual acknowledgment required" instead
of countdown timer. -->
<!-- 1.6.3: auto-close when notification becomes inactive. Fleet alert forms now
check every 30 seconds if the notification is still active on the server and
automatically close if it has ended or been deactivated. -->
<!-- 1.6.4: fix UI thread deadlock. Acknowledgment and status checks now run async
properly without blocking the UI thread. Acknowledgment is marked locally first,
then posted to server in background to prevent lockups. -->
<!-- 1.6.5: fix resource leaks and disposal issues. GraphicsPath properly disposed,
Font objects tracked and disposed, worker thread joined before client disposal,
Screen.PrimaryScreen null dereference prevented, status check cancellation token
propagated to prevent orphaned HTTP requests. -->
<!-- 1.6.6: coordinated release with agent 1.13.5 (no tray changes).
No tray code changes, but deployed together for version consistency. -->
<!-- 1.6.7: coordinated release with agent 1.14.1 (wilm merge).
No tray code changes, but deployed together after wilm merge. -->
<!-- 1.6.8: coordinated release with agent 1.14.2 (fleet alert scope fixes).
No tray code changes, but deployed together for version consistency. -->
<!-- 1.6.9: coordinated release with agent 1.17.0 (app delta optimization).
No tray code changes, but deployed together for version consistency. -->
<!-- 1.6.10: coordinated release with agent 1.18.0 (fingerprint computation).
No tray code changes, but deployed together for version consistency. -->
<!-- 1.6.11: fleet alert hardening. The toast steals focus, so its buttons are
no longer tab stops, there is no Accept/Cancel button, and acknowledge
needs a mouse click at least 1.5s after the form is shown: keystrokes
meant for another app cannot acknowledge an unseen alert. Long messages
scroll instead of being cut off and '&' is shown literally. A null or
malformed alert field no longer throws away the whole poll. A timeout
closes without acknowledging. The acknowledged store moves to
%LOCALAPPDATA%\ShopDB\FleetAlerts (the operator could not write the
ProgramData copy, so acks were lost every logon), merging the old file
once. Shutdown no longer blocks on the poller's sleeps. -->
<!-- 1.7.0: the tray holds no credential. Fleet alerts, alert acknowledgement,
Send Status and Report an issue all go through the SYSTEM agent's local
pipe (ShopDB.EndpointAgent.v1) instead of reading the collector key and
config or shelling the agent exe with the once and report verbs, so a
standard user on the bay can no longer lift the fleet key through the
tray's access path. The state.json asset-id reader is gone (the agent
knows its own identity). An unreachable agent shows the agent-not-running
balloon, and alert polling logs at a throttled rate and retries. An alert
is recorded as acknowledged only when the agent confirms it, so a failed
ack lets it reappear. The collect-usercontext probe is unchanged.
Requires the agent release that hosts the pipe.
1.7.1: no behaviour of its own. The fleet alert key rename landed
after the 1.7.0 binary was built and shipped to the display scope,
so the rebuild carries different bytes under the same version. The
version moves so the two can be told apart. Pairs with agent
1.20.1. -->
<Version>1.7.1</Version>
<InvariantGlobalization>true</InvariantGlobalization>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
<SelfContained>true</SelfContained>

444
agent/ShopDbEndpointAgent.Tray/TrayContext.cs
View File

@@ -1,6 +1,8 @@
using System.Diagnostics;
using System.Reflection;
using Microsoft.Extensions.Logging;
using ShopDbEndpointAgent.Core;
namespace ShopDbEndpointAgent.Tray;
@@ -9,11 +11,11 @@ namespace ShopDbEndpointAgent.Tray;
// and headless-friendly: no main window, the icon is the whole UI surface until
// the operator opens the ticket dialog.
//
// "Send Status now" reuses the exact production collection path by shelling out
// to the installed shopdbagent.exe with --once. That keeps the collector key and
// all collection logic in the service binary: the tray never holds a secret and
// never duplicates wiring, it just asks the agent to report on demand so the
// ShopDB monitor page reflects the box immediately instead of at next cadence.
// "Send Status now", "Report an issue" and fleet alerts all go through the
// SYSTEM agent's local pipe (AgentPipeClient). The agent holds the credential
// and all collection logic: the tray never reads a key or config and never
// duplicates wiring, it just asks the agent to act on demand so the ShopDB
// monitor page reflects the box immediately instead of at next cadence.
internal sealed class TrayContext : ApplicationContext
{
// The headless agent/service executable. The tray ships next to it in the
@@ -25,8 +27,98 @@ internal sealed class TrayContext : ApplicationContext
private readonly Icon brandIcon;
private bool sending;
private const string AgentUnreachableText =
"Could not reach the ShopDB Endpoint Agent. Is the ShopDB Endpoint Agent service running?";
private readonly AgentPipeClient pipeClient = new();
// Cancels in-flight pipe calls from menu handlers when the tray exits.
private readonly CancellationTokenSource shutdownCancellation = new();
// Session-scoped collection. The service is LocalSystem and cannot see the
// operator's mapped drives, user-owned listeners, or input idle time, so the
// tray - which is already the operator - collects them and leaves the result
// in a spool file the service reads on its own cadence.
//
// Deliberately longer than the telemetry cadence: this data changes rarely
// (a mapped drive is mapped at logon and stays mapped), and each run spawns
// a short PowerShell process on a machine an operator is actively using.
private static readonly TimeSpan UserContextInterval = TimeSpan.FromMinutes(30);
// Logon is the worst moment to add work: drive mappings and profile setup
// are still settling, and a probe fired now would capture an incomplete
// picture while competing with everything else starting up.
private static readonly TimeSpan UserContextStartupDelay = TimeSpan.FromMinutes(2);
// ...but that reasoning only applies at logon. When the tray starts into a
// session that has been up for a while - which is what an installer restart
// or a manual relaunch does - nothing is settling and the full delay just
// extends the window where the server has no session data. The installer
// kills and restarts the tray on every install, so the 2-minute delay was
// being paid at exactly the moment someone was watching the card to see
// whether the install had worked.
private static readonly TimeSpan UserContextEstablishedDelay = TimeSpan.FromSeconds(15);
// How long the session must have been up to count as established. Comfortably
// past the logon storm, well short of any realistic install-to-check gap.
private static readonly TimeSpan EstablishedSessionUptime = TimeSpan.FromMinutes(5);
private readonly System.Windows.Forms.Timer userContextTimer;
private bool userContextStarted;
private bool collectingUserContext;
// Lets the service ask this tray to re-probe on demand, so a "Refresh
// Status" click in the WebUI produces genuinely current session data instead
// of whatever the last 30-minute tick happened to capture. The service
// cannot collect this itself at any price - mapped drives live on the
// operator's logon token - so the only way for an on-demand refresh to be
// accurate is for the request to end up here, in the operator's session.
private readonly UserContextRefreshChannel? refreshChannel;
private readonly CancellationTokenSource refreshCancellation = new();
// How long each wait blocks before looping. The wait returns false on
// timeout and the loop simply waits again, so this is not a deadline for
// anything - it just bounds how long the handle stays open, which lets a
// channel that was somehow broken get re-opened on the next pass instead of
// leaving the tray deaf until it restarts.
private static readonly TimeSpan RefreshWaitTimeout = TimeSpan.FromMinutes(5);
// The UI thread's context, captured in the constructor. The listener runs on
// a background thread, but every probe attempt is marshalled back here so
// that collectingUserContext is only ever read or written from one thread.
// Without this the timer tick and a service request could both pass the
// guard and launch two probes at the same spool file.
private readonly SynchronizationContext? uiContext;
// Fleet alerts are fetched through the agent pipe and displayed to the
// operator as toast notifications in the top-right corner. The worker polls
// every 60 seconds, filters out already-acknowledged alerts via the store,
// and sends acknowledgements to the agent when the operator dismisses the form.
private readonly FleetAlertStore? fleetAlertStore;
private readonly FleetAlertWorker? fleetAlertWorker;
// The tray logs into the operator's own profile
// (C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log), NOT into the
// service's ProgramData directory. The operator cannot write there, and the
// file sink fails silently when it cannot create its directory, so tray-side
// failures previously left no trace anywhere. Keeping the two apart means a
// tray problem is diagnosable from a file the operator can actually reach.
private readonly FileLoggerProvider logProvider;
private readonly ILogger logger;
public TrayContext()
{
logProvider = new FileLoggerProvider(
FileLoggerProvider.UserProfileLogDirectory(),
filePrefix: "trayagent");
logger = logProvider.CreateLogger("Tray");
var startupDelay = ResolveStartupDelay();
logger.LogInformation(
"Tray {Version} started; user-context probe in {Delay}, then every {Interval}.",
typeof(TrayContext).Assembly.GetName().Version?.ToString() ?? "unknown",
startupDelay, UserContextInterval);
var menu = new ContextMenuStrip();
menu.Items.Add("Send Status now", null, OnSendStatus);
menu.Items.Add("Report an issue...", null, OnReportIssue);
@@ -45,6 +137,239 @@ internal sealed class TrayContext : ApplicationContext
ContextMenuStrip = menu,
};
trayIcon.DoubleClick += OnSendStatus;
// A dead log sink cannot report itself through the log. Complaining via
// the logger would write the one message that explains the silence into
// the sink that is swallowing messages, so the balloon is the only
// channel that survives. Shown once at startup and never repeated: this
// is a "your diagnostics are blind" notice for whoever is standing at the
// machine, not a recurring nag about something the operator cannot fix.
if (logProvider.SinkProblem is not null)
{
trayIcon.ShowBalloonTip(
10000,
"ShopDB Endpoint Agent",
"Tray logging is unavailable, so tray diagnostics will not be "
+ $"recorded. {logProvider.SinkProblem}",
ToolTipIcon.Warning);
}
userContextTimer = new System.Windows.Forms.Timer
{
Interval = (int)startupDelay.TotalMilliseconds,
};
userContextTimer.Tick += OnUserContextTimer;
userContextTimer.Start();
uiContext = SynchronizationContext.Current;
// Best-effort. A tray that cannot open the channel still probes on its
// timer, so the failure mode is "refresh is not immediate", not "session
// data stops working" - which is why this is caught and logged rather
// than allowed to take the tray down at startup.
try
{
refreshChannel = new UserContextRefreshChannel(
logProvider.CreateLogger(nameof(UserContextRefreshChannel)));
StartRefreshListener();
}
catch (Exception exception)
{
logger.LogWarning(exception,
"Could not open the user-context refresh channel; on-demand "
+ "refresh requests from the service will be ignored and session "
+ "data will only update on the {Interval} timer.",
UserContextInterval);
}
// Fleet alert worker. Best-effort: if the worker fails to start, the tray
// still runs and the other features work. An agent that is down is not a
// startup failure; the worker retries the pipe every cycle.
try
{
fleetAlertStore = new FleetAlertStore(
logProvider.CreateLogger(nameof(FleetAlertStore)));
fleetAlertWorker = new FleetAlertWorker(
pipeClient,
fleetAlertStore,
logProvider.CreateLogger(nameof(FleetAlertWorker)),
uiContext);
fleetAlertWorker.Start();
}
catch (Exception exception)
{
logger.LogWarning(exception,
"Could not start the fleet alert worker; fleet-wide alerts will not "
+ "be displayed. Other tray features are unaffected.");
}
}
// Waits for service refresh requests for the life of the tray.
//
// A dedicated background thread rather than a Task: this blocks on a kernel
// wait handle essentially forever, which is exactly the workload that should
// not sit on a thread-pool thread. IsBackground keeps it from holding the
// process open at exit.
private void StartRefreshListener()
{
var thread = new Thread(() =>
{
var token = refreshCancellation.Token;
while (!token.IsCancellationRequested)
{
try
{
if (!refreshChannel!.WaitForRequest(RefreshWaitTimeout, token))
continue;
logger.LogInformation(
"The service requested a user-context refresh.");
// Marshalled, not awaited: the service is not waiting on a
// reply through this channel, it is watching the spool file's
// collectedat stamp. Our only job is to get the probe started
// on the UI thread and go back to waiting.
var post = uiContext;
if (post is not null)
post.Post(_ => _ = CollectUserContextAsync(), null);
else
_ = CollectUserContextAsync();
}
catch (OperationCanceledException)
{
return;
}
catch (Exception exception)
{
// Never let a bad wait kill the listener - if this thread
// exits, on-demand refresh is silently dead for the rest of
// the session with no other symptom.
logger.LogWarning(exception,
"The user-context refresh listener failed; retrying.");
try
{
Thread.Sleep(TimeSpan.FromSeconds(5));
}
catch (Exception)
{
return;
}
}
}
})
{
IsBackground = true,
Name = "ShopDB user-context refresh listener",
};
thread.Start();
}
// Fires once after the startup delay, then re-arms at the steady-state
// interval. Failures are swallowed on purpose: session data is a
// nice-to-have, and an operator must never see a dialog or a crashed tray
// because a probe could not read a drive mapping.
private async void OnUserContextTimer(object? sender, EventArgs e)
{
if (!userContextStarted)
{
userContextStarted = true;
userContextTimer.Interval = (int)UserContextInterval.TotalMilliseconds;
}
await CollectUserContextAsync();
}
// Runs the probe once and returns when the spool has been rewritten.
//
// Shared by all three callers - the cadence timer, an on-demand service
// request, and "Send Status now" - so there is exactly one place that knows
// how to invoke the probe and how to interpret its exit codes. Returns true
// only when a spool file is actually on disk afterwards, which is what lets
// Send Status decide whether it is worth waiting on.
private async Task<bool> CollectUserContextAsync()
{
if (collectingUserContext)
{
// Another probe is already in flight against the same spool path.
// Its result is at most a second or two old and satisfies whatever
// this caller wanted, so coalescing is correct here, not a shortcut.
logger.LogDebug(
"A user-context probe is already running; skipping this request.");
return false;
}
var agentExe = ResolveAgentExe();
if (agentExe is null)
{
// Previously a bare return. This is the single most likely reason for
// "no mapped drives" with nothing at all in any log, so it must say so.
logger.LogWarning(
"Skipping the user-context probe: {Agent} was not found beside the tray "
+ "({Beside}) or at {Installed}.",
AgentExeName, AppContext.BaseDirectory, DefaultInstallDir);
return false;
}
collectingUserContext = true;
try
{
logger.LogInformation("Running the user-context probe: {Agent}", agentExe);
var exitCode = await RunAgentAsync(agentExe, "--collect-usercontext");
// Check the disk rather than announcing what the exit code implies.
// This line used to print a hardcoded path whenever the child exited
// 0, so it reported "spool written to C:\Users\ShopFloor\ShopDB" for
// weeks while the probe was actually writing into a different profile
// and the service was finding nothing.
var resultPath = UserContextSpool.PinnedResultPath();
if (exitCode == 0 && File.Exists(resultPath))
{
logger.LogInformation(
"The user-context probe finished; spool present at {Spool}.",
resultPath);
return true;
}
else if (exitCode == 0)
{
logger.LogWarning(
"The user-context probe reported success but no spool file is "
+ "present at {Spool}.", resultPath);
}
else if (exitCode == 2)
{
// The agent refused because it was elevated. Called out by name
// because as a bare exit code it reads like a probe fault and
// sends the reader looking for a drive or WMI problem that is
// not there - the fix is to start the tray unelevated, or let
// the scheduled task do the collecting.
logger.LogWarning(
"The user-context probe refused to run because this tray is "
+ "ELEVATED. Mapped drives are per logon token, so an elevated "
+ "probe would report the wrong drive table. Start the tray "
+ "unelevated, or rely on the 'ShopDB user-context probe' "
+ "scheduled task.");
}
else
{
logger.LogWarning(
"The user-context probe exited with code {ExitCode}; expected a "
+ "spool at {Spool}.", exitCode, resultPath);
}
return false;
}
catch (Exception exception)
{
// No longer silent. The agent logs its own detail, but if the process
// could not even be started the agent never ran and never logged.
logger.LogWarning(exception, "Could not run the user-context probe.");
return false;
}
finally
{
collectingUserContext = false;
}
}
private async void OnReportIssue(object? sender, EventArgs e)
@@ -61,13 +386,7 @@ internal sealed class TrayContext : ApplicationContext
includeScreenshot = dialog.IncludeScreenshot;
}
var agentExe = ResolveAgentExe();
if (agentExe is null)
{
ShowBalloon("Report an issue", "Could not find " + AgentExeName +
". Is the ShopDB Endpoint Agent installed?", ToolTipIcon.Error);
return;
}
var token = shutdownCancellation.Token;
// Capture AFTER the dialog closes so the report form is never in the
// shot. Stage into the interactive operator's own profile
@@ -97,16 +416,23 @@ internal sealed class TrayContext : ApplicationContext
ShowBalloon("Report an issue", "Sending your report to ShopDB...",
ToolTipIcon.Info);
int exitCode = await RunAgentAsync(agentExe, "--report \"" + payloadPath + "\"");
if (exitCode == 0)
// The agent checks the path is inside this user's own profile.
var response = await pipeClient.SendReportAsync(payloadPath, token);
if (response.Ok)
ShowBalloon("Report an issue", "Your report was sent to ShopDB.",
ToolTipIcon.Info);
else if (response.Data is null)
ShowBalloon("Report an issue", AgentUnreachableText, ToolTipIcon.Error);
else
ShowBalloon("Report an issue",
"Report send failed (exit " + exitCode +
"Report send failed (" + response.Error +
"). See the agent log under ProgramData\\ShopDB.",
ToolTipIcon.Warning);
}
catch (OperationCanceledException) when (token.IsCancellationRequested)
{
// Tray is exiting.
}
catch (Exception error)
{
ShowBalloon("Report an issue", "Could not send the report: " +
@@ -145,23 +471,15 @@ internal sealed class TrayContext : ApplicationContext
}
}
// Runs shopdbagent.exe --once out of process so the operator can push a
// fresh report without waiting for the next cadence tick. The exit code is
// the agent's own contract: 0 = reported OK, non-zero = failed (bad key,
// host removed, network) and the detail is in the rolling agent log.
// Asks the agent over the pipe to run telemetry once so the operator can
// push a fresh report without waiting for the next cadence tick. The agent
// answers ok or an error; the detail is in the rolling agent log.
private async void OnSendStatus(object? sender, EventArgs e)
{
if (sending)
return;
var agentExe = ResolveAgentExe();
if (agentExe is null)
{
ShowBalloon("Send Status", "Could not find " + AgentExeName +
". Is the ShopDB Endpoint Agent installed?", ToolTipIcon.Error);
return;
}
var token = shutdownCancellation.Token;
sending = true;
trayIcon.Text = "ShopDB Endpoint Agent (sending status...)";
ShowBalloon("Send Status", "Sending this PC's status to ShopDB...",
@@ -169,25 +487,45 @@ internal sealed class TrayContext : ApplicationContext
try
{
int exitCode = await RunAgentAsync(agentExe, "--once");
if (exitCode == 0)
// Probe BEFORE reporting, and await it. The operator clicking this
// has just changed something - remapped a drive, signed in - and
// expects the result to show up. sendstatus reads the spool file, so a
// probe started after it, or not awaited, would publish the previous
// probe's answer and the operator would click again and see the same
// stale card.
//
// Not fatal if it fails: a report with older session data still
// carries the heartbeat, inventory and events, which is most of what
// this button is for.
await CollectUserContextAsync();
var response = await pipeClient.SendStatusAsync(token);
if (response.Ok)
ShowBalloon("Send Status", "Status sent to ShopDB.",
ToolTipIcon.Info);
else if (response.Data is null)
ShowBalloon("Send Status", AgentUnreachableText, ToolTipIcon.Error);
else
ShowBalloon("Send Status",
"Status send failed (exit " + exitCode +
"Status send failed (" + response.Error +
"). See the agent log under ProgramData\\ShopDB.",
ToolTipIcon.Warning);
}
catch (OperationCanceledException) when (token.IsCancellationRequested)
{
// Tray is exiting.
return;
}
catch (Exception error)
{
ShowBalloon("Send Status", "Could not run the agent: " +
ShowBalloon("Send Status", "Could not reach the agent: " +
error.Message, ToolTipIcon.Error);
}
finally
{
sending = false;
trayIcon.Text = "ShopDB Endpoint Agent";
if (!token.IsCancellationRequested)
trayIcon.Text = "ShopDB Endpoint Agent";
}
}
@@ -249,6 +587,35 @@ internal sealed class TrayContext : ApplicationContext
startInfo.Environment.Remove(key);
}
// Full delay at logon, short delay when the session is already established.
//
// Environment.TickCount64 is machine uptime, not session uptime, which is a
// deliberate simplification: the two differ only when someone signs out and
// back in without rebooting, and treating that as "established" is harmless -
// the profile is already built and the drives are already mapped in that
// case too. A failure to read it falls back to the conservative long delay.
private TimeSpan ResolveStartupDelay()
{
try
{
var uptime = TimeSpan.FromMilliseconds(Environment.TickCount64);
if (uptime >= EstablishedSessionUptime)
{
logger.LogInformation(
"Session has been up {Uptime}; using the established-session probe delay.",
uptime);
return UserContextEstablishedDelay;
}
}
catch (Exception exception)
{
logger.LogDebug(exception,
"Could not read system uptime; using the logon probe delay.");
}
return UserContextStartupDelay;
}
// Same directory first (the installer ships the tray next to the agent),
// then the known Program Files install directory as a fallback.
private static string? ResolveAgentExe()
@@ -316,6 +683,19 @@ internal sealed class TrayContext : ApplicationContext
{
if (disposing)
{
userContextTimer.Stop();
userContextTimer.Dispose();
// Cancel and let the listener unwind on its own. The channel opens
// and closes its handle inside each wait rather than holding one, so
// there is nothing here to dispose.
refreshCancellation.Cancel();
refreshCancellation.Dispose();
shutdownCancellation.Cancel();
fleetAlertWorker?.Dispose();
shutdownCancellation.Dispose();
trayIcon.Dispose();
brandIcon.Dispose();
}

281
agent/ShopDbEndpointAgent/Program.cs
View File

@@ -11,26 +11,156 @@ using ShopDbEndpointAgent.Core;
// --console run interactively in the foreground for debugging
// --once collect and report exactly once, then exit (smoke test / cron)
// --report <path> upload one operator issue report (JSON sidecar written by
// the tray), then exit. Keeps the collector key in the
// service so the tray never holds a secret.
// the tray), then exit.
// Not running as SYSTEM, --once and --report are pipe clients: they ask the
// running service (AgentPipeServer) to do the work and never load a secret.
//
// Secrets are never read here: AgentConfigProvider layers the registry
// (HKLM\SOFTWARE\GE\ShopDB: CollectorKey, BaseUrl) over the shipped, non-secret
// appsettings.json. The executable is intentionally named shopdbagent.exe so it
// appsettings.json, and the per-PC credential lives in CredentialStore. The executable is intentionally named shopdbagent.exe so it
// never collides with third-party agent.exe processes/services (for example the
// MTConnect agent this same build can monitor).
var runOnce = args.Any(a => string.Equals(a, "--once", StringComparison.OrdinalIgnoreCase));
var forceConsole = args.Any(a => string.Equals(a, "--console", StringComparison.OrdinalIgnoreCase));
var collectUserContext = args.Any(a =>
string.Equals(a, "--collect-usercontext", StringComparison.OrdinalIgnoreCase));
var reportPayloadPath = ArgValue(args, "--report");
var agentVersion = Assembly.GetExecutingAssembly()
.GetName().Version?.ToString() ?? "0.0.0";
// FIRST, ahead of everything else. This verb runs as the signed-in operator (the
// tray invokes it) and must not touch a single privileged resource on the way in.
//
// It used to sit below the bootstrap logger, the fault monitor, and the config
// load, which meant the operator-context process opened the SYSTEM-owned
// ProgramData log directory, attempted an EventLog source registration that only
// an administrator can perform, and read CollectorKey out of HKLM - all before it
// discovered it was only being asked to write a file into the user's own profile.
// On an EPM-managed host (CyberArk) those are exactly the calls a broker
// intercepts, and the registry read put the collector secret in a process that
// has no business holding it.
//
// It also has no config dependency by design: it only writes into the caller's
// own profile and never talks to the server, so it must still work on a machine
// whose CollectorKey has not been provisioned yet - otherwise session data would
// be missing on precisely the endpoints that are hardest to diagnose.
if (collectUserContext)
{
// First statement in this branch. The scheduled task runs this exe directly
// in the operator's interactive session, and shopdbagent.exe is a console
// app, so without this a black window flashes up on every collection cycle.
// Done before any logging or probing so the window is gone immediately
// rather than lingering for the life of the probe.
ConsoleWindow.Hide();
// Logs to the OPERATOR's profile, not the service's ProgramData directory.
// A non-privileged user cannot write to the SYSTEM-owned log directory, and
// the file sink degrades to a no-op when it cannot create its directory, so
// pointing this at the service location meant every probe failure was
// discarded unread - which is exactly how a broken probe looked identical to
// a probe that was never invoked.
using var probeLogProvider = new FileLoggerProvider(
FileLoggerProvider.UserProfileLogDirectory(),
filePrefix: "trayagent");
using var probeLoggerFactory = LoggerFactory.Create(
logging => logging.AddProvider(probeLogProvider));
var probeLogger = probeLoggerFactory.CreateLogger<UserContextProbe>();
ReportSinkProblem(probeLogProvider, "user-context probe");
// Refuse to collect from an elevated process. Drive letters are a property
// of the logon token, not the machine: an elevated process gets a different
// token to the operator's interactive one and enumerates a DIFFERENT, almost
// always empty, drive table. Measured on a live endpoint, the same two
// shares reported Unavailable from the elevated shell and OK from the
// operator's, in both Win32_NetworkConnection and Get-SmbMapping.
//
// Refusing here rather than filtering later is deliberate. An elevated probe
// does not fail - it succeeds and reports every mapped drive as broken,
// which is indistinguishable from a genuine outage at the far end. Better to
// write nothing and say why than to publish a confident wrong answer.
if (IsProcessElevated())
{
probeLogger.LogError(
"Refusing to collect user context: this process is elevated. Mapped drives " +
"are per-logon-token, so an elevated probe reports the wrong drive table. " +
"Run the '{Task}' scheduled task (RunLevel Limited) instead of invoking " +
"--collect-usercontext from an admin shell.",
"ShopDB user-context probe");
Console.Error.WriteLine(
"shopdbagent: refusing to collect user context from an elevated process; " +
"mapped drives would be wrong. Use the scheduled task.");
return 2;
}
var probe = new UserContextProbe(probeLogger);
return probe.Collect() ? 0 : 1;
}
// True when the current process holds the elevated (high-integrity) token.
static bool IsProcessElevated()
{
if (!OperatingSystem.IsWindows())
{
return false;
}
try
{
using var identity = System.Security.Principal.WindowsIdentity.GetCurrent();
var principal = new System.Security.Principal.WindowsPrincipal(identity);
return principal.IsInRole(System.Security.Principal.WindowsBuiltInRole.Administrator);
}
catch
{
// Cannot tell. Treat as not elevated so an unreadable token does not
// suppress collection on a host where the probe would have worked; the
// spool's own account stamp is still checked on the way back in.
return false;
}
}
// Not SYSTEM: a user (the tray, or an operator at a prompt) cannot read the
// credential or the key, so hand the request to the service over the pipe.
// Ahead of the bootstrap logger for the same reason as the probe above.
if ((runOnce || !string.IsNullOrWhiteSpace(reportPayloadPath)) && !IsRunningAsSystem())
return await RunPipeClientAsync(runOnce, reportPayloadPath);
static bool IsRunningAsSystem()
{
if (!OperatingSystem.IsWindows())
return true;
try
{
using var identity = System.Security.Principal.WindowsIdentity.GetCurrent();
return identity.IsSystem;
}
catch
{
return false;
}
}
// Exit code is the tray's contract: 0 = done, non-zero = failed (reason on stderr).
static async Task<int> RunPipeClientAsync(bool runOnce, string? reportPath)
{
var client = new AgentPipeClient();
var response = runOnce
? await client.SendStatusAsync(CancellationToken.None)
: await client.SendReportAsync(Path.GetFullPath(reportPath!), CancellationToken.None);
if (response.Ok)
return 0;
Console.Error.WriteLine("shopdbagent: " + response.Error);
return response.Data is null ? 3 : 1;
}
// Bootstrap logger first so config loading (including the registry read that
// CyberArk/EPM may gate) is fully diagnosable before the host starts.
using var bootstrapFileLogger = new FileLoggerProvider(FileLoggerProvider.DefaultDirectory());
var bootstrapLogger = bootstrapFileLogger.CreateLogger("Startup");
ReportSinkProblem(bootstrapFileLogger, "service");
var faultMonitor = new AgentFaultMonitor(bootstrapLogger);
faultMonitor.Install();
@@ -41,15 +171,24 @@ var logDirectory = string.IsNullOrWhiteSpace(config.LogPath)
? FileLoggerProvider.DefaultDirectory()
: config.LogPath;
if (!config.IsUsable())
// BaseUrl is the only hard requirement: an enrolled PC runs without the shared
// key, and an unenrolled one can still request approval without it.
if (string.IsNullOrWhiteSpace(config.BaseUrl))
{
bootstrapLogger.LogError(
"Config incomplete: provision CollectorKey and BaseUrl in " +
"HKLM\\SOFTWARE\\GE\\ShopDB (or baseurl/apikey in appsettings.json for dev). " +
"Config incomplete: provision BaseUrl in " +
"HKLM\\SOFTWARE\\GE\\ShopDB (or baseurl in appsettings.json for dev). " +
"See the registry lines above to tell 'missing' from 'access denied'.");
return 1;
}
if (!config.HasSharedKey())
{
bootstrapLogger.LogWarning(
"No CollectorKey provisioned. Fine once this PC is enrolled; until then " +
"requests are unauthenticated and enrollment needs admin approval.");
}
var baseUrlProblem = config.BaseUrlProblem();
if (baseUrlProblem is not null)
{
@@ -65,8 +204,7 @@ try
if (!string.IsNullOrWhiteSpace(reportPayloadPath))
return await RunReportAsync(config, reportPayloadPath, logDirectory, faultMonitor);
await RunHostAsync(config, agentVersion, logDirectory, forceConsole);
return 0;
return await RunHostAsync(config, agentVersion, logDirectory, forceConsole);
}
catch (Exception error)
{
@@ -74,7 +212,7 @@ catch (Exception error)
return 1;
}
static async Task RunHostAsync(AgentConfig config, string agentVersion,
static async Task<int> RunHostAsync(AgentConfig config, string agentVersion,
string logDirectory, bool forceConsole)
{
var builder = Host.CreateApplicationBuilder();
@@ -85,10 +223,44 @@ static async Task RunHostAsync(AgentConfig config, string agentVersion,
ConfigureLogging(builder.Logging, logDirectory);
RegisterCoreServices(builder.Services, config, agentVersion);
// Enrollment is registered (and so started) first; the workers wait briefly
// for its first attempt so an enrollable PC reports on its own credential.
builder.Services.AddSingleton(sp => new EnrollmentService(
config,
sp.GetRequiredService<AgentHttpFactory>(),
sp.GetRequiredService<AgentCredentialProvider>(),
sp.GetRequiredService<CredentialStore>(),
agentVersion,
sp.GetRequiredService<ILogger<EnrollmentService>>()));
builder.Services.AddSingleton(sp => new AgentPipeServer(
config,
sp.GetRequiredService<TelemetryWorker>(),
sp.GetRequiredService<CollectorClient>(),
sp.GetRequiredService<FleetAlertClient>(),
sp.GetRequiredService<ICredentialProvider>(),
agentVersion,
sp.GetRequiredService<ILogger<AgentPipeServer>>()));
builder.Services.AddHostedService(sp => sp.GetRequiredService<EnrollmentService>());
builder.Services.AddHostedService(sp => sp.GetRequiredService<TelemetryWorker>());
builder.Services.AddHostedService(sp => sp.GetRequiredService<CommandWorker>());
builder.Services.AddHostedService(sp => sp.GetRequiredService<AgentPipeServer>());
await builder.Build().RunAsync();
var host = builder.Build();
// A newly issued credential reports straight away rather than at the next
// cadence tick, so the server sees the PC switch over.
var credentials = host.Services.GetRequiredService<AgentCredentialProvider>();
var collectSignal = host.Services.GetRequiredService<CollectSignal>();
credentials.Changed += (_, _) =>
{
if (credentials.Current is not null)
collectSignal.Request();
};
await host.RunAsync();
return 0;
}
static async Task<int> RunOnceAsync(AgentConfig config, string agentVersion,
@@ -111,8 +283,8 @@ static async Task<int> RunOnceAsync(AgentConfig config, string agentVersion,
}
// Uploads one operator issue report described by a tray-written JSON sidecar,
// then deletes the payload folder. Runs out of process from the tray so the
// collector key stays in the service binary. The exit code is the tray's
// then deletes the sidecar. SYSTEM only; a user-context --report goes through
// the pipe (RunPipeClientAsync) and the service validates the path. The exit code is the tray's
// contract: 0 = uploaded, non-zero = failed (detail in the agent log).
static async Task<int> RunReportAsync(AgentConfig config, string payloadPath,
string logDirectory, AgentFaultMonitor faultMonitor)
@@ -191,13 +363,60 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config
sp.GetRequiredService<ILogger<PowerShellUtilityRunner>>()));
}
// Persisted per-channel event watermark so each Windows event record is
// reported exactly once, instead of re-sending the newest window of every
// channel on every cadence tick.
services.AddSingleton(sp => new EventWatermarkStore(
sp.GetRequiredService<ILogger<EventWatermarkStore>>()));
// Session-scoped collection. The agent is LocalSystem, so mapped drives and
// user-owned listeners are invisible to it. The tray, which already runs as
// the operator, writes the spool file; the service only reads it.
//
// UserContextProbe is deliberately NOT registered here. The service must
// never run the probe: descending into the session with CreateProcessAsUser
// is the reverted design, and leaving a resolvable dependency in the
// container is an invitation to reintroduce it. The probe is constructed
// only on the --collect-usercontext path, which the tray invokes.
services.AddSingleton(sp => new UserContextSpool(
sp.GetRequiredService<ILogger<UserContextSpool>>()));
// The ask-the-tray side of the same split. Registering this is NOT a
// loophole around the paragraph above: the service still cannot collect
// session data, it can only request that the process which can does so, and
// then read the file that appears. The probe stays out of this container.
services.AddSingleton(sp => new UserContextRefreshChannel(
sp.GetRequiredService<ILogger<UserContextRefreshChannel>>()));
services.AddSingleton<ICollector>(sp => new EndpointCollector(
sp.GetRequiredService<ILogger<EndpointCollector>>(),
config,
sp.GetService<IAgentUtilityRunner>()));
sp.GetService<IAgentUtilityRunner>(),
sp.GetRequiredService<EventWatermarkStore>(),
sp.GetRequiredService<UserContextSpool>()));
// Per-PC credential (if enrolled) or shared key, swapped at runtime. One
// factory builds every HttpClient so pins and auth cannot be skipped.
services.AddSingleton(sp => new CredentialStore(
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Credentials")));
services.AddSingleton(sp => new AgentCredentialProvider(
config,
sp.GetRequiredService<CredentialStore>(),
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Credentials")));
services.AddSingleton<ICredentialProvider>(sp =>
sp.GetRequiredService<AgentCredentialProvider>());
services.AddSingleton(sp => new AgentHttpFactory(
config,
sp.GetRequiredService<ICredentialProvider>(),
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Http")));
services.AddSingleton(sp => new CollectorClient(
config, sp.GetRequiredService<ILogger<CollectorClient>>()));
sp.GetRequiredService<AgentHttpFactory>(),
sp.GetRequiredService<ILogger<CollectorClient>>()));
services.AddSingleton(sp => new FleetAlertClient(
sp.GetRequiredService<AgentHttpFactory>(),
sp.GetRequiredService<ILoggerFactory>().CreateLogger(nameof(FleetAlertClient))));
services.AddSingleton(sp => new TelemetryWorker(
sp.GetRequiredService<ICollector>(),
@@ -206,23 +425,30 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config
agentVersion,
sp.GetRequiredService<ILogger<TelemetryWorker>>(),
sp.GetService<IAgentUtilityRunner>(),
sp.GetRequiredService<CollectSignal>()));
sp.GetRequiredService<CollectSignal>(),
sp.GetRequiredService<UserContextSpool>(),
sp.GetRequiredService<UserContextRefreshChannel>(),
sp.GetService<EnrollmentService>()));
// Command channel: outbound long-poll for allowlisted WebUI commands, run
// through the fail-closed executor. Hosted alongside telemetry but on its
// own loop so a held-open poll never delays a report.
services.AddSingleton(sp => new CommandClient(
config, sp.GetRequiredService<ILogger<CommandClient>>()));
sp.GetRequiredService<AgentHttpFactory>(),
sp.GetRequiredService<ILogger<CommandClient>>()));
services.AddSingleton(sp => new CommandExecutor(
sp.GetRequiredService<ILogger<CommandExecutor>>(),
config));
config,
sp.GetRequiredService<AgentHttpFactory>()));
services.AddSingleton(sp => new CommandWorker(
sp.GetRequiredService<CommandClient>(),
sp.GetRequiredService<CommandExecutor>(),
sp.GetRequiredService<ILogger<CommandWorker>>(),
sp.GetRequiredService<CollectSignal>()));
sp.GetRequiredService<CollectSignal>(),
config,
sp.GetService<EnrollmentService>()));
}
static void ConfigureLogging(ILoggingBuilder logging, string logDirectory)
@@ -231,3 +457,22 @@ static void ConfigureLogging(ILoggingBuilder logging, string logDirectory)
logging.AddProvider(new FileLoggerProvider(logDirectory));
logging.SetMinimumLevel(LogLevel.Information);
}
// Say so, loudly, when a file sink could not open its directory.
//
// FileLoggerProvider never throws on construction and silently drops lines when
// its directory is unusable. That is the right behaviour - logging must not take
// the agent down - but it made two different failures indistinguishable: a probe
// that ran and failed, and a probe whose log had nowhere to go. The provider
// records the reason; this is the part that was missing, the caller actually
// asking for it. stderr is the only sink guaranteed to exist at this point, so
// that is where the complaint goes.
static void ReportSinkProblem(FileLoggerProvider provider, string which)
{
if (provider.SinkProblem is null)
return;
Console.Error.WriteLine(
$"shopdbagent: the {which} log sink is unusable and log lines will be "
+ $"dropped - {provider.SinkProblem}");
}

249
agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj
View File

@@ -7,7 +7,88 @@
<Nullable>enable</Nullable>
<RootNamespace>ShopDbEndpointAgent</RootNamespace>
<AssemblyName>shopdbagent</AssemblyName>
<!-- 1.6.3: operator issue reports stage under the interactive operator's
<!-- 1.10.1: the collect-usercontext verb hides its console window. The agent
is a console exe, which the tray used to mask by launching the probe with
CreateNoWindow; the scheduled task runs the exe directly in the operator's
session, so a black window appeared on every 30-minute collection. -->
<!-- 1.10.0: user context is collected by a scheduled task in the operator's
own logon session, not by a process the elevated installer launches.
Drive letters are per logon token: an elevated probe enumerates a
different, usually empty, drive table, so an install performed by an
admin produced a spool reporting every mapped drive as Unavailable while
the operator's own session showed OK. The installer no longer starts the
tray via explorer.exe (which de-elevates but does NOT change account) and
instead registers 'ShopDB user-context probe' against BUILTIN\Users at
RunLevel Limited, verifies it by reading the task back, and records the
outcome in the install ledger. The collect-usercontext verb now refuses
to run elevated (exit 2) rather than writing a confident wrong answer,
the probe stamps its token elevation, and the service rejects any spool
stamped elevated. Probe runs log the drive and listener counts. -->
<!-- 1.9.0: the spool is finally written where the service looks for it. The
probe script resolved its output directory from $env:USERPROFILE while
the reader used the pinned C:\Users\ShopFloor path; under a brokered or
re-parented tray launch those differ, so the probe reported success and
wrote the file somewhere the service never read. Both sides now derive
the path from one helper (UserContextSpool.PinnedResultPath). Collect()
verifies the file exists on disk before reporting success, because a
zero exit code from powershell.exe proved nothing about the write.
Minor rather than patch because the spool handoff itself changed shape
(AGENT-CONTRACT.md 5): the spool is now PERSISTED and overwritten each
run rather than consumed on read. The collector cache is in-memory only,
so consume-on-read plus a service restart blanked mapped drives until the
next tray cycle. Max age tightened 24h -> 90m now that staleness, not
deletion, is the retirement mechanism. A spool whose username stamp is
absent or not ShopFloor is rejected outright - wrong-session drive data
is worse than none.
The collect-usercontext branch moved to the top of Program.cs: it used to
open the SYSTEM-owned log directory, register an EventLog source, and
read CollectorKey out of HKLM before discovering it only needed to write
a file into the caller's own profile - all as the operator, and all
exactly what an EPM broker intercepts. Log filenames unified on
<date>-<component>.log across service, tray, and installer. A log sink
that cannot open its directory now says so on stderr instead of
discarding every line unread. Pairs with tray 1.2.0. -->
<!-- 1.8.1: user-context logging and spool location are split from the
service's. The probe runs as the signed-in operator but logged into
%ProgramData%\ShopDB\EndpointAgent\logs, which is SYSTEM-owned and not
writable from a user session; the file sink swallows that failure, so a
probe that failed and a probe that never ran looked identical - no log
line either way. The probe and the tray now log to
C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log, which the
operator owns. The spool path is pinned to C:\Users\ShopFloor\ShopDB
instead of being resolved through WTSQueryUserToken (a privileged call
endpoint protection blocks, after which writer and reader disagreed
silently about the location). The service now CONSUMES the spool -
removing it after a successful parse - so a dead tray stops re-reporting
stale drives instead of pinning them forever. The probe itself is
launched with -EncodedCommand rather than a staged .ps1 run via -File:
a quarantined or missing temp script makes powershell.exe print usage
and exit -196608 without running a statement. -->
<!-- 1.8.0: user-context collection is inverted. The service no longer
descends into the operator's session with CreateProcessAsUser to run the
probe; that path needs to duplicate a token, is what endpoint protection
watches for, and produced no spool file at all on a live endpoint. The
tray, which is already running as the operator, now invokes
shopdbagent.exe with the collect-usercontext verb on its own cadence and
the service only reads the spool. Requires the 1.1.0 tray: an older tray
the new verb, so usercontext stays null until the tray is updated.
Spool freshness moved from delete-around-read to the in-file collectedat
stamp with a max age, because writer and reader are now independent
processes and a delete-then-read would race the tray and discard good
results. -->
<!-- 1.7.0: curated events are harvested against a persisted per-channel
Windows record-id watermark and carry eventrecordid, so one real event
is reported and stored exactly once instead of once per cadence tick
for as long as it stayed in the newest-25 window. Adds WER 1000/1002,
event XML fields, listening ports, an on-demand network_connections
command, pendingreboot, disk freepercent, and populates cpupercent/
memorypercent/idleseconds (previously declared but always null).
Session-scoped data (mapped drives, user-owned listeners, idle time)
comes from a probe run in the interactive user's session via a spool
file, because none of it exists in the agent's LocalSystem session 0.
run_script user-context waits are now bounded; the previous Infinite
wait could pin the command worker on a hung script.
1.6.3: operator issue reports stage under the interactive operator's
%USERPROFILE%\ShopDBScreenshots (resolved in the ShopFloor tray session)
instead of %TEMP%, which on locked-down endpoints redirected to another
user's AppData\Local\Temp and left the SYSTEM-context report upload
@@ -50,7 +131,171 @@
(AgentConfig.LaunchPaths, shipped defaults overridable per site).
1.1.0: generic, data-driven service_*/process_* actions carry their
target over the wire, so new policies/apps need no agent rebuild. -->
<Version>1.6.3</Version>
<!-- 1.13.0: run_script honors a per-script working directory. The script
library gains an optional workingdirectory, the server carries it in the
run_script descriptor, and ScriptDeliveryClaim gains WorkingDirectory
(AGENT-CONTRACT.md). The point is corporate tooling shipped by Intune: a
script that shells out to a vendor CLI needs to run beside that CLI, not
in the staging folder. Empty means the existing staging path, so older
packages and every existing script keep their current behavior - this is
additive, hence minor. Fails closed: a named directory that does not
exist on the endpoint aborts the run with a clear error rather than
silently falling back to staging, because a script that ran in the wrong
folder is worse than one that did not run. The directory confers no
rights of its own - it is a cwd, not a grant, and the run still happens
in the script's configured context. Server-side validation rejects UNC
paths and traversal before the value is ever persisted. No tray change -
tray stays 1.4.0. -->
<!-- 1.12.0: startup jitter before first contact. Both loops previously made
their first request the instant the service started, and the service is
installed start=auto, so a patch reboot put the whole fleet on the wire
together - twice per machine (a telemetry report AND a command poll).
jitterseconds did not help: it spreads the gap BETWEEN runs, not first
contact. A 700-endpoint burst drove TLS handshakes to 1482ms worst case
and pinned CPU at 305% while the database sat idle at 3 active queries;
spreading the same arrivals over 30s cut worst-case handshake to 63.7ms
with no server-side change. New startupjitterseconds (default 30, 0
disables) is applied by StartupJitter.Next in both TelemetryWorker and
CommandWorker. Randomized in milliseconds rather than whole seconds
because a 30s window at second granularity re-clumps a large fleet into
30 buckets of simultaneous arrivals. Clamped to 300s and warned about
when out of range: the command channel is how an operator would correct
a bad value, so an unbounded delay would remove the means of recovery.
Costs up to startupjitterseconds of extra latency before a rebooted
endpoint picks up a queued command, on top of a reboot the machine was
already unreachable for. The once-only console switch bypasses the delay
entirely (it calls RunOnceAsync directly), so smoke tests stay instant.
appsettings schema gains a key; older packages without it get the
AgentConfig default. No tray change - tray stays 1.4.0. -->
<!-- 1.11.0: on-demand user-context refresh. "Refresh Status" (collect_now)
and the tray's "Send Status" re-sent whatever was already in the spool,
so a drive the operator had just unmapped kept reporting OK until the
next 30-minute probe and no amount of clicking changed it. The service
still does not - and must not - collect session data itself; it signals
Global\ShopDbEndpointAgent.UserContextRefresh and waits up to 12s for the
spool's collectedat stamp to ADVANCE before collecting, which is the only
proof the file in front of it answers this request and not the previous
one. The wait is bounded and non-fatal by design: telemetry is the
liveness signal, so a timeout reports slightly stale session data rather
than skipping the report. UserContextSpool.TryReadCollectedAt reads the
stamp outside the usability gates, so a spool rejected as expired or
wrong-account does not burn the full timeout every cycle. Minor because
the spool handoff gains a second channel (AGENT-CONTRACT.md 5) and
because the installer now registers the probe task to the ShopFloor USER
rather than BUILTIN\Users: a group-principal task cannot be started by a
member of that group, so Start-ScheduledTask returned access denied and
the task only ever ran on its own trigger. Requires tray 1.4.0 - the tray
is the side that listens, so an older tray simply never answers. -->
<!-- 1.13.1: coordinated release with tray 1.6.0 (fleet alert styling update).
No agent code changes, but deployed together for version consistency. -->
<!-- 1.13.2: coordinated release with tray 1.6.3 (auto-close when inactive, crash fix).
No agent code changes, but deployed together for version consistency. -->
<!-- 1.13.3: coordinated release with tray 1.6.4 (deadlock fix).
No agent code changes, but deployed together for version consistency. -->
<!-- 1.13.4: coordinated release with tray 1.6.5 (resource leak fixes).
No agent code changes, but deployed together for version consistency. -->
<!-- 1.14.0: truthful script verdicts and script output capture. A run_script
command was judged solely on PowerShell's exit code, and PowerShell
treats most errors as non-terminating: it writes the error, runs on, and
still exits zero. An account-provisioning script therefore reported
succeeded having provisioned nothing, and because no output was captured
there was no evidence to contradict it. The claim now carries errormode
(AGENT-CONTRACT.md 7.4); strict prepends $ErrorActionPreference='Stop'
AFTER digest verification so the hash still validates delivered bytes.
Default stays permissive so bodies authored against the old semantics
keep working. stdout/stderr is captured on both paths - redirected
streams for SYSTEM, a transcript file under the ShopFloor profile for
user-session runs, since CreateProcessAsUser cannot inherit our pipes
across the session boundary - always logged locally and returned to the
server only on failure. Minor because the claim gains a field and an
older agent ignoring it keeps today's behaviour. Supersedes the 1.13.1
to 1.13.4 tray-coordination bumps merged from wj, which carried no agent
code changes. Tray is at 1.6.5 from wj. -->
<!-- 1.15.0: successful runs can now return their output to the server, gated
by a per-script captureoutput flag on the claim. Failures still always
return output; success previously never did, which left a green badge as
the only evidence a run produced and no way to tell "did the work" from
"PowerShell exited cleanly". Opt-in rather than always-on because output
carries whatever the body printed, and that is not worth persisting
server-side for every routine run. Minor because the claim gains a field
that defaults false: an agent or server too old to know it keeps today's
withholding behaviour, which is the safe direction to fail. -->
<!-- 1.16.0: adds collect_log, which reads a plaintext log file from a
caller-supplied path and returns its contents for the Log Collector tab.
Strictly read-only: opened for reading with FileShare.ReadWrite|Delete,
because the logs worth collecting are the ones a running application
currently holds open and a stricter share mode fails on exactly those.
Bounded rather than unlimited - only the last 2 MB is returned, and only
for an allowlisted set of text extensions; a binary or oversized file is
refused rather than mangled into an unreadable pane. The path is
re-validated here even though the server already checked it, since this
process runs as SYSTEM and the path arrived over the network: environment
variables are expanded BEFORE the checks so %VAR% cannot smuggle a
traversal or UNC path past them. Minor because it is a new action name -
an older agent asked for it fails closed on the unknown-action default. -->
<!-- 1.17.0: application inventory delta optimization. Backend adds
app_fingerprint column (SHA256 of displayname|version|publisher) and
switches from delete-all-insert-all to delta updates. Agent side not yet
implemented - this release only fixes critical backend race conditions:
advisory locks prevent TOCTOU races in both app delta and event dedup,
UNIQUE constraint on (assetid, app_fingerprint) prevents duplicate
fingerprints. Reduces database writes by 99% once agent sends fingerprints.
Backend is backward compatible - legacy agents without fingerprints use
delete-all-insert-all fallback. Minor for critical race condition fixes. -->
<!-- 1.18.0: agent-side fingerprint computation. Computes SHA256 fingerprint
for each application (displayname|displayversion|publisher) and includes
in collector payload. Backend receives fingerprints and switches from
delete-all-insert-all to delta mode (only touch changed apps). Expected
impact: 99% reduction in database writes (1.92M/day to 19K/day). Matches
backend algorithm implemented in 1.17.0. Minor for feature completion. -->
<!-- 1.19.0: endpoint hardening. TLS without a pinned thumbprint now uses
normal Windows chain validation instead of accepting any certificate.
run_script and update_app stage into SYSTEM/Administrators-only
directories under %ProgramData%\ShopDB\EndpointAgent with random file
names, and the script hash is checked in memory and again over the
exact bytes written, so a standard user can no longer swap a body
between verification and execution. User-session transcripts moved out
of the operator's profile into that protected area, and SYSTEM never
follows a reparse point to read or delete them. SYSTEM scripts drain
stdout/stderr while running (no pipe deadlock), cap captured output,
and are killed with their process tree after 5 minutes. Strict mode
inserts after using/#Requires/param headers, keeps line numbers, and
writes UTF-8 with BOM. A user-session run with no exit code is a
failure. collect_log refuses answer files, agent config/state, key
material and reparse points, and rechecks the handle's final path.
Certificate renewal at start is bounded to 15s and exits non-zero so
SCM recovery restarts the service. -->
<!-- 1.20.0: per-PC endpoint credentials. The shared fleet collector key was
readable by any standard user on every bay, so one bay leaked the whole
fleet. The agent now enrolls itself (POST /api/endpoints/enroll with
hostname, BIOS serial, DeviceId and a persisted nonce) and receives its
own credential, stored DPAPI LocalMachine under a SYSTEM/Administrators
directory and sent as X-Endpoint-Credential. The shared key stays the
fallback until the PC is enrolled, so the build also runs against an
older server (a 404 on enroll parks enrollment for 6 hours). A revoked
credential (CREDENTIAL_INVALID) is deleted and the PC re-enrolls without
a restart, and a script or package download refused that way retries
once. Server trust moves to SPKI SHA-256 pins (the ServerPins registry
value when it exists, even empty; otherwise serverpins in
appsettings.json), checked by one AgentHttpFactory
that now builds every HttpClient, including script and package downloads
that previously had their own unpinned handlers. The tray holds no
secret any more: it reaches the service over the local named pipe
ShopDB.EndpointAgent.v1 (ping, sendstatus, sendreport, alerts, ackalert),
and the once and report switches become pipe clients when not SYSTEM. A report path must sit inside the
caller's own profile. appsettings.json is restricted to SYSTEM and
Administrators once the pipe is up. The 1.19 client-certificate renewal
and expiry monitor are removed, and with them the non-zero exit at start.
Pairs with tray 1.7.0.
1.20.1: no behaviour of its own. Two fixes landed after the 1.20.0
binaries had already been built and shipped to the display scope -
the fast retry while an enrollment is pending, and the fleet alert
key rename - so the rebuild carries different bytes under the same
version. Detection is by hash and copes, but two files calling
themselves 1.20.0.0 cannot be told apart by anyone reading a
version number, so the version moves. Pairs with tray 1.7.1. -->
<Version>1.20.1</Version>
<InvariantGlobalization>true</InvariantGlobalization>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
<SelfContained>true</SelfContained>

2
agent/ShopDbEndpointAgent/appsettings.json
View File

@@ -1,8 +1,10 @@
{
"baseurl": "",
"apikey": "REPLACE_WITH_COLLECTOR_KEY",
"serverpins": [],
"cadenceseconds": 900,
"jitterseconds": 60,
"startupjitterseconds": 30,
"adoptonfirstrun": true,
"utilitiespath": "",
"logpath": "",

355
agent/installer/install-shopdbagent.ps1
View File

@@ -1,355 +0,0 @@
# install-shopdbagent.ps1
# ShopDB Endpoint Agent installer for the test endpoint. Run from an ELEVATED
# (admin) PowerShell on the endpoint. Provisions the registry secrets, copies
# the agent files to Program Files, runs a one-shot smoke test, then installs
# and starts the Windows service.
#
# Secrets are provisioned to HKLM\SOFTWARE\GE\ShopDB. On locked-down hosts
# (CyberArk/EPM) the agent may be blocked from READING that hive, so the same
# values are also stamped into the installed appsettings.json as a fallback.
#
# The agent and tray are published as single-file self-contained executables,
# so copying the two .exe files plus appsettings.json is the complete payload.
[CmdletBinding()]
param(
# ShopDB server root URL for this instance. REQUIRED - there is no default,
# because a default here means one site's hostname shipped to every site
# (ADR-015) and, worse, a working install pointed at somebody else's server.
#
# https is required. The collector key travels in the X-API-Key header and
# every telemetry payload follows it, so plain http puts both on the wire in
# clear. Pass -AllowInsecureHttp only for a local test instance.
[Parameter(Mandatory = $true)]
[string] $BaseUrl,
# Endpoints-only collector key: this instance's .env COLLECTOR_API_KEY_ENDPOINTS.
# REQUIRED and never defaulted - a key in source is a key in git history, and
# this one is also the command channel's kill switch.
[Parameter(Mandatory = $true)]
[string] $CollectorKey,
# Permit an http:// BaseUrl. For a local test instance only.
[switch] $AllowInsecureHttp,
[string] $InstallDir = "C:\Program Files\ShopDB\EndpointAgent",
[string] $ServiceName = "ShopDbEndpointAgent",
# Interactive tray app that shows the system-tray icon and the on-demand
# "Send Status now" action. Separate process from the SYSTEM service (a
# service cannot draw a tray icon in a user session). Pass -NoTray to skip.
[string] $TrayExeName = "shopdb-endpoint-tray.exe",
[switch] $NoTray,
# Add a Microsoft Defender path exclusion for the install directory. The
# agent and tray are unsigned single-file self-contained executables, which
# Defender heuristics (and the tray's GDI/user32 icon P/Invoke) can flag as
# a false positive and quarantine, killing the tray before its icon appears.
# Pass -NoDefenderExclusion to skip (e.g. where policy manages exclusions
# centrally or the binaries are code-signed).
[switch] $NoDefenderExclusion
)
# Refuse cleartext unless it was asked for explicitly. The key is a bearer
# credential for the whole fleet's telemetry and command channel.
if ($BaseUrl -notmatch '^https://' -and -not $AllowInsecureHttp) {
throw "BaseUrl must be https:// (got '$BaseUrl'). Pass -AllowInsecureHttp for a local test instance."
}
$ErrorActionPreference = "Stop"
$sourceDir = Split-Path -Parent $MyInvocation.MyCommand.Path
function Assert-Admin {
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
throw "This script must be run from an elevated (Run as administrator) PowerShell."
}
}
Assert-Admin
if ([string]::IsNullOrWhiteSpace($BaseUrl)) {
throw "BaseUrl is not set. Pass -BaseUrl 'http://<host>:<port>' when running the script."
}
# Write HKLM values through the explicit 64-bit registry view. If this script
# runs under 32-bit PowerShell (some management tooling launches x86), the
# PSDrive HKLM: silently redirects writes into WOW6432Node. The 64-bit agent
# service and 64-bit Explorer (the logon Run key) then read the 64-bit view and
# see nothing, so config "goes missing" and the tray never autostarts. Pinning
# the view removes that dependency on the launching shell's bitness.
function Set-Hklm64Value {
param(
[Parameter(Mandatory)] [string] $SubKey,
[Parameter(Mandatory)] [string] $Name,
[Parameter(Mandatory)] [string] $Value
)
$base = [Microsoft.Win32.RegistryKey]::OpenBaseKey(
[Microsoft.Win32.RegistryHive]::LocalMachine,
[Microsoft.Win32.RegistryView]::Registry64)
try {
$key = $base.CreateSubKey($SubKey)
try {
$key.SetValue($Name, $Value, [Microsoft.Win32.RegistryValueKind]::String)
}
finally { $key.Dispose() }
}
finally { $base.Dispose() }
}
# Tear down any previously installed agent BEFORE touching files. The agent and
# tray are single-file self-contained exes; while the old service or tray is
# running it holds those .exe files open, so Copy-Item over them fails with a
# file-in-use error. Stopping the service, deleting it, and killing any lingering
# agent/tray processes up front makes the install a clean overwrite. Best-effort
# throughout: a missing service or process is the normal fresh-install case.
function Uninstall-PreviousAgent {
param(
[Parameter(Mandatory)] [string] $Service,
[Parameter(Mandatory)] [string] $TrayExe,
[Parameter(Mandatory)] [string] $TargetDir
)
$existing = Get-Service -Name $Service -ErrorAction SilentlyContinue
if ($existing) {
Write-Host " stopping and removing existing service $Service ..."
if ($existing.Status -ne "Stopped") {
try { Stop-Service -Name $Service -Force -ErrorAction Stop }
catch { Write-Warning " could not stop service: $($_.Exception.Message)" }
}
# sc.exe delete only marks the service for deletion; the SCM finishes the
# removal once every handle closes. Give it a moment so the subsequent
# sc.exe create does not hit a "marked for deletion" (1072) error.
sc.exe delete $Service | Out-Null
Start-Sleep -Seconds 2
}
# Kill any lingering worker or tray processes that would keep the exes
# locked. Match by the installed exe base names; the service host process is
# shopdbagent, the tray is the TrayExe base name.
$trayBase = [System.IO.Path]::GetFileNameWithoutExtension($TrayExe)
foreach ($name in @("shopdbagent", $trayBase)) {
$procs = Get-Process -Name $name -ErrorAction SilentlyContinue
foreach ($p in $procs) {
Write-Host " killing lingering process $($p.ProcessName) (pid $($p.Id)) ..."
try { Stop-Process -Id $p.Id -Force -ErrorAction Stop }
catch { Write-Warning " could not kill $($p.ProcessName): $($_.Exception.Message)" }
}
}
# Also kill anything still running out of the target install directory,
# regardless of exe name. A tray from an older build (different exe name) or
# one launched from a stale path would otherwise survive the name-based sweep
# above, keep its exe locked, and leave a duplicate icon once step 5 starts
# the fresh tray. Match on the executable path so we only touch our own dir.
$targetPrefix = $TargetDir.TrimEnd('\') + '\'
foreach ($p in (Get-Process -ErrorAction SilentlyContinue)) {
$path = $null
try { $path = $p.Path } catch { $path = $null }
if ($path -and $path.StartsWith($targetPrefix, [System.StringComparison]::OrdinalIgnoreCase)) {
Write-Host " killing process from install dir $($p.ProcessName) (pid $($p.Id)) ..."
try { Stop-Process -Id $p.Id -Force -ErrorAction Stop }
catch { Write-Warning " could not kill $($p.ProcessName): $($_.Exception.Message)" }
}
}
# Brief settle so file handles are released before the copy overwrites them.
Start-Sleep -Milliseconds 500
}
Write-Host "== ShopDB Endpoint Agent install ==" -ForegroundColor Cyan
Write-Host "BaseUrl : $BaseUrl"
Write-Host "InstallDir : $InstallDir"
Write-Host "ServiceName : $ServiceName"
# 0. Clear the Mark-of-the-Web on the SOURCE payload first. The operator
# typically extracts installer.zip into Downloads, so every file carries a
# Zone.Identifier ADS. SmartScreen reads that and blocks the unsigned exes with
# "Windows protected your PC" the first time they run - including the [3/5]
# smoke test below, which launches shopdbagent.exe out of $sourceDir before the
# installed copy exists. Unblocking the source folder up front keeps the smoke
# test (and any manual run from the extract folder) clean; the installed copies
# get unblocked again in step 2a after they land in Program Files. Best-effort:
# a file with no MotW is a no-op.
Write-Host "`n[0a] Clearing Mark-of-the-Web on the extracted installer files ..."
Get-ChildItem -Path $sourceDir -File | ForEach-Object {
try { Unblock-File -Path $_.FullName -ErrorAction Stop }
catch { Write-Warning " could not unblock $($_.Name): $($_.Exception.Message)" }
}
# 0. Uninstall any previous agent first, so the file copy is a clean overwrite
# and does not fail on exes locked by a running service or tray.
Write-Host "`n[0/5] Removing any previously installed agent ..."
Uninstall-PreviousAgent -Service $ServiceName -TrayExe $TrayExeName -TargetDir $InstallDir
# 1. Provision registry secrets (64-bit view, see Set-Hklm64Value).
$regPath = "HKLM:\SOFTWARE\GE\ShopDB"
Write-Host "`n[1/5] Provisioning registry $regPath (Registry64) ..."
Set-Hklm64Value -SubKey "SOFTWARE\GE\ShopDB" -Name "BaseUrl" -Value $BaseUrl
Set-Hklm64Value -SubKey "SOFTWARE\GE\ShopDB" -Name "CollectorKey" -Value $CollectorKey
# 2. Copy agent files to the install directory. The agent and tray are
# single-file self-contained executables, so the .exe plus appsettings.json is
# the whole payload; there are no side-by-side runtime DLLs to copy.
Write-Host "[2/5] Copying agent files to $InstallDir ..."
New-Item -ItemType Directory -Force -Path $InstallDir | Out-Null
$agentSource = Join-Path $sourceDir "shopdbagent.exe"
if (-not (Test-Path $agentSource)) {
throw "shopdbagent.exe not found in $sourceDir. Unzip the full installer package and run from that folder."
}
Copy-Item -Path $agentSource -Destination $InstallDir -Force
$appsettings = Join-Path $sourceDir "appsettings.json"
$installedSettings = Join-Path $InstallDir "appsettings.json"
if (Test-Path $appsettings) {
Copy-Item -Path $appsettings -Destination $InstallDir -Force
}
# Copy the interactive tray app next to the agent so its same-dir resolution
# finds shopdbagent.exe for the "Send Status now" action. Optional: absent on a
# service-only (headless) build.
$trayInstalled = $null
if (-not $NoTray) {
$traySource = Join-Path $sourceDir $TrayExeName
if (Test-Path $traySource) {
Copy-Item -Path $traySource -Destination $InstallDir -Force
$trayInstalled = Join-Path $InstallDir $TrayExeName
Write-Host " copied tray app $TrayExeName"
}
else {
Write-Warning " tray app $TrayExeName not found in $sourceDir; skipping tray (service still installs)"
}
}
# Fallback for locked-down endpoints: some hosts (CyberArk/EPM) gate the HKLM
# hive so the agent cannot READ the values the registry step just wrote. Stamp
# BaseUrl/CollectorKey into the installed appsettings.json too. The agent reads
# appsettings first, then lets the registry override it, so this is a safe
# no-op where the registry read works and the rescue where it does not.
if (Test-Path $installedSettings) {
try {
$json = Get-Content $installedSettings -Raw | ConvertFrom-Json
$json.baseurl = $BaseUrl
$json.apikey = $CollectorKey
$json | ConvertTo-Json -Depth 10 | Set-Content -Path $installedSettings -Encoding UTF8
Write-Host " stamped BaseUrl/CollectorKey into appsettings.json (registry-read fallback)"
}
catch {
Write-Warning " could not stamp appsettings.json: $($_.Exception.Message)"
}
}
$exePath = Join-Path $InstallDir "shopdbagent.exe"
# 2a. Strip the Mark-of-the-Web from the copied files. Windows tags files
# extracted from a downloaded zip with a Zone.Identifier alternate data stream;
# SmartScreen reads that and blocks the unsigned exe with "Windows protected
# your PC" the first time it runs, and the operator sees only a "Don't run"
# button. Unblock-File removes the ADS so the smoke test and the tray launch
# start clean. Best-effort: a file with no MotW is a no-op.
Write-Host "[2a/5] Clearing Mark-of-the-Web on installed files ..."
Get-ChildItem -Path $InstallDir -File | ForEach-Object {
try { Unblock-File -Path $_.FullName -ErrorAction Stop }
catch { Write-Warning " could not unblock $($_.Name): $($_.Exception.Message)" }
}
# 2b. Defender path exclusion for the install dir. Unsigned single-file
# self-contained binaries (and the tray's icon P/Invoke) can trip Defender
# heuristics and get quarantined, which is why the tray dies before showing an
# icon. Excluding the fixed install directory is the least-privilege fix short
# of code-signing. Best-effort: Set-MpPreference is absent on servers without
# Defender and can be blocked by tamper protection; a failure is not fatal.
if (-not $NoDefenderExclusion) {
Write-Host "[2b/5] Adding Defender path exclusion for $InstallDir ..."
if (Get-Command Add-MpPreference -ErrorAction SilentlyContinue) {
try {
Add-MpPreference -ExclusionPath $InstallDir -ErrorAction Stop
Write-Host " excluded $InstallDir from Defender scanning."
}
catch {
Write-Warning " could not add Defender exclusion (tamper protection or policy?): $($_.Exception.Message)"
Write-Warning " if the tray is quarantined, code-sign the binaries or add the exclusion via policy."
}
}
else {
Write-Host " Defender cmdlets not present; skipping exclusion."
}
}
# 3. One-shot smoke test (exact production collection path, then exits).
Write-Host "[3/5] Running one-shot smoke test (--once) ..."
$proc = Start-Process -FilePath $exePath -ArgumentList "--once" -WorkingDirectory $InstallDir -Wait -PassThru -NoNewWindow
Write-Host " smoke test exit code: $($proc.ExitCode)"
$logDir = Join-Path $env:ProgramData "ShopDB\EndpointAgent\logs"
$logFile = Join-Path $logDir ("agent-{0}.log" -f (Get-Date -Format "yyyyMMdd"))
if (Test-Path $logFile) {
Write-Host " --- last log lines ---" -ForegroundColor DarkGray
Get-Content $logFile -Tail 20 | ForEach-Object { Write-Host " $_" -ForegroundColor DarkGray }
}
if ($proc.ExitCode -ne 0) {
throw "Smoke test failed (exit $($proc.ExitCode)). Check $logFile before installing the service."
}
# 4. Install the Windows service. Any previous instance was already stopped,
# deleted, and its processes killed in step 0, so this is a clean create. Guard
# against the rare case where the SCM has not finished a pending deletion yet.
Write-Host "[4/5] Installing service $ServiceName ..."
$existing = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
if ($existing) {
Write-Host " service still present (pending deletion); removing again ..."
if ($existing.Status -ne "Stopped") { Stop-Service -Name $ServiceName -Force }
sc.exe delete $ServiceName | Out-Null
Start-Sleep -Seconds 2
}
$binPath = '"' + $exePath + '"'
sc.exe create $ServiceName binPath= $binPath start= auto DisplayName= "ShopDB Endpoint Agent" | Out-Null
sc.exe description $ServiceName "Collects endpoint telemetry for ShopDB and runs allowlisted WebUI commands." | Out-Null
sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/5000 | Out-Null
Start-Service -Name $ServiceName
Write-Host " service started."
# 5. Register the interactive tray app to launch at user logon (per-machine).
Write-Host "[5/5] Registering tray app for logon ..."
if ($trayInstalled -and (Test-Path $trayInstalled)) {
# Register in the 64-bit Run key so 64-bit Explorer fires it at logon. A
# 32-bit write would land in WOW6432Node\...\Run, which Explorer ignores.
Set-Hklm64Value -SubKey "SOFTWARE\Microsoft\Windows\CurrentVersion\Run" `
-Name "ShopDbEndpointTray" -Value ('"' + $trayInstalled + '"')
Write-Host " tray app registered to start at logon (Registry64): $trayInstalled"
# Start it now so the icon appears without waiting for the next logon. The
# Run key only fires at logon, so a fresh install would otherwise show no
# tray until the operator signs out and back in.
#
# The installer runs elevated, so a direct Start-Process would launch the
# tray ELEVATED too. That is wrong for a tray app (it should run at the
# user's medium integrity) and, on EPM-managed hosts (CyberArk), elevating an
# unsigned exe triggers a broker prompt that comes back as Win32 1223
# "The operation was canceled by the user" - which is exactly what we hit.
# Launching through explorer.exe hands the exe to the already-running user
# shell, which spawns it de-elevated in the interactive session, sidestepping
# the elevation broker entirely.
$interactiveUser = $env:USERNAME
if ($interactiveUser -and $interactiveUser -ne "SYSTEM") {
try {
# explorer.exe <path> re-parents the launch under the user shell at
# medium integrity; it returns immediately regardless of the tray's
# own lifetime, so there is nothing to wait on here.
Start-Process -FilePath "explorer.exe" -ArgumentList "`"$trayInstalled`"" | Out-Null
Write-Host " tray app started for the current session (de-elevated via explorer)."
}
catch {
Write-Warning " could not start tray now: $($_.Exception.Message); it will start at next logon."
}
}
else {
Write-Host " no interactive user session; tray will start at next logon."
}
}
else {
Write-Host " tray skipped (headless install)."
}
Write-Host "`nInstall complete." -ForegroundColor Green
Write-Host "Service '$ServiceName' is running and set to auto-start."
Write-Host "Logs: $logDir"

76
agent/test-fingerprint.ps1 Normal file
View File

@@ -0,0 +1,76 @@
# Test fingerprint computation matches Python backend
# Expected: SHA256("Google Chrome|117.0.5938.132|Google LLC")
$code = @"
using System;
using System.Security.Cryptography;
using System.Text;
public class FingerprintTester
{
public static string ComputeFingerprint(string displayName, string displayVersion, string publisher)
{
var normalized = $"{displayName}|{displayVersion}|{publisher}";
var bytes = Encoding.UTF8.GetBytes(normalized);
var hash = SHA256.HashData(bytes);
return Convert.ToHexString(hash).ToLowerInvariant();
}
}
"@
Add-Type -TypeDefinition $code
# Test vectors from backend
$tests = @(
@{
Name = "Google Chrome"
Version = "117.0.5938.132"
Publisher = "Google LLC"
Description = "Standard application"
},
@{
Name = "Microsoft Office"
Version = "16.0.16827.20166"
Publisher = "Microsoft Corporation"
Description = "Office suite"
},
@{
Name = "Test App No Version"
Version = ""
Publisher = "Test Publisher"
Description = "Empty version string"
},
@{
Name = "Test App Empty"
Version = ""
Publisher = ""
Description = "Empty strings"
}
)
Write-Host "C# Fingerprint Algorithm Test" -ForegroundColor Cyan
Write-Host "==============================" -ForegroundColor Cyan
Write-Host ""
foreach ($test in $tests) {
$fp = [FingerprintTester]::ComputeFingerprint($test.Name, $test.Version, $test.Publisher)
Write-Host "Test: $($test.Description)" -ForegroundColor Yellow
Write-Host " Input: '$($test.Name)|$($test.Version)|$($test.Publisher)'"
Write-Host " Fingerprint: $fp" -ForegroundColor Green
Write-Host " Length: $($fp.Length) chars" -ForegroundColor Gray
Write-Host ""
}
# Verify format
$testFp = [FingerprintTester]::ComputeFingerprint("Test", "1.0", "Publisher")
if ($testFp.Length -eq 64) {
Write-Host "Format check: PASS (64 characters)" -ForegroundColor Green
} else {
Write-Host "Format check: FAIL (expected 64, got $($testFp.Length))" -ForegroundColor Red
}
if ($testFp -match '^[0-9a-f]{64}$') {
Write-Host "Hex check: PASS (lowercase hexadecimal)" -ForegroundColor Green
} else {
Write-Host "Hex check: FAIL (invalid characters)" -ForegroundColor Red
}

119
agent/utilities/shopdb-dellupdates.ps1 Normal file
View File

@@ -0,0 +1,119 @@
<#
.SYNOPSIS
ShopDB agent utility: reports pending Dell Command | Update items.
.DESCRIPTION
Discovered and executed by PowerShellUtilityRunner on the telemetry cadence.
Emits a single JSON object on stdout, which the agent attaches to the report
as `dellupdates`. Never throws and never writes anything but JSON: a failure
here must not break a telemetry run, so every error path still emits a valid
object with a `status` explaining why the list is empty.
THIS SCRIPT DOES NOT SCAN. It only reads the log a previous scan wrote. The
scan itself is fired by the \ShopDB-DellCommandScan scheduled task, which is
what pins dcu-cli.exe's -outputLog to the path below. Keeping the two apart
is deliberate: a telemetry collector that shelled out to dcu-cli would block
the cadence for minutes and hammer Dell's servers every 15 minutes.
OUTPUT CONTRACT (keep in sync with plugins/endpoints/plugin.py):
status unknown | ok | error - "unknown" means no scan has ever run,
which is NOT the same as "ok" with an empty list (known clean).
scannedat ISO-8601 UTC timestamp of the scan log's last write, or null.
count total pending updates found.
updates up to $MaxUpdates entries of { name, severity, version }.
truncated true when count exceeded $MaxUpdates.
Dell's own long description text is deliberately NOT carried: it is
sentence-length marketing prose per update, which would add roughly an order
of magnitude to every report for no operational value. Name plus severity is
what an operator decides on.
#>
$ErrorActionPreference = 'Stop'
# Cap the list rather than the payload. A machine that is years behind can have
# dozens of pending updates, and an unbounded list would bloat every report on
# the cadence. Mirrors the MaxReportedConnections precedent in CommandExecutor:
# a truncated answer is more useful than a rejected payload.
$MaxUpdates = 25
# Must match the -outputLog path set in the \ShopDB-DellCommandScan task
# definition. ShopDB-initiated scans write HERE rather than to Dell's default
# location, which is what keeps them distinguishable from the corporate-managed
# DCU tasks that also run on these machines.
$ScanLog = Join-Path $env:ProgramData 'ShopDB\EndpointAgent\dellcommand\scan.xml'
function Write-Result {
param($Status, $ScannedAt, $Count, $Updates, $Truncated)
# Compress so the payload carries no pretty-printing whitespace. Depth 3
# covers the nested updates array.
[pscustomobject]@{
status = $Status
scannedat = $ScannedAt
count = $Count
updates = @($Updates)
truncated = [bool]$Truncated
} | ConvertTo-Json -Depth 3 -Compress
}
try {
if (-not (Test-Path -LiteralPath $ScanLog)) {
# No scan has run yet. This is the normal state for a freshly adopted
# endpoint, not an error, but it must stay distinct from "scanned and
# found nothing" or the UI would claim a machine is up to date when
# nobody has ever checked.
Write-Result -Status 'unknown' -ScannedAt $null -Count 0 -Updates @() -Truncated $false
return
}
$logFile = Get-Item -LiteralPath $ScanLog
$scannedAt = $logFile.LastWriteTimeUtc.ToString('o')
[xml]$report = Get-Content -LiteralPath $ScanLog -Raw
# dcu-cli's scan report shape has varied across DCU versions, so select the
# update nodes by name anywhere in the tree rather than by a fixed path.
$nodes = @($report.SelectNodes('//update'))
if ($nodes.Count -eq 0) {
$nodes = @($report.SelectNodes('//Update'))
}
$updates = foreach ($node in $nodes) {
# Attribute casing also varies by version; take whichever is present and
# fall back rather than emitting a null-filled row.
$name = $node.name
if (-not $name) { $name = $node.Name }
if (-not $name) { $name = $node.display }
if (-not $name) { continue }
$severity = $node.urgency
if (-not $severity) { $severity = $node.Urgency }
if (-not $severity) { $severity = $node.criticality }
$version = $node.version
if (-not $version) { $version = $node.Version }
[pscustomobject]@{
name = [string]$name
severity = if ($severity) { [string]$severity } else { 'unknown' }
version = if ($version) { [string]$version } else { $null }
}
}
$updates = @($updates)
$count = $updates.Count
$truncated = $count -gt $MaxUpdates
if ($truncated) {
$updates = $updates[0..($MaxUpdates - 1)]
}
Write-Result -Status 'ok' -ScannedAt $scannedAt -Count $count `
-Updates $updates -Truncated $truncated
}
catch {
# Emit a valid object rather than failing: the runner would log a non-JSON
# utility and drop it, which looks identical to "never ran" on the server.
# An explicit error status tells an operator the scan log exists but could
# not be read, which is a different problem from no scan at all.
Write-Result -Status 'error' -ScannedAt $null -Count 0 -Updates @() -Truncated $false
}

7
deploy/ci/buildkit-remote.yaml
View File

@@ -14,9 +14,10 @@
# Deployment (not a per-job container) means the build cache in the PVC survives
# across runs, so the heavy frontend npm build is only paid in full once.
#
# Cameron: apply once with `kubectl apply -f deploy/ci/buildkit-remote.yaml`.
# This changes cluster infra only. It does NOT modify any application code or
# any branch of the repo.
# Apply once with `kubectl apply -f deploy/ci/buildkit-remote.yaml`. This
# changes cluster infra only. It does NOT modify any application code or any
# branch of the repo. A site that builds its own image with plain `docker
# build` or `docker compose build` needs none of this.
---
apiVersion: v1
kind: Namespace

7
deploy/site-profile-universal.json
View File

@@ -3,22 +3,17 @@
"plugins": [
"backups",
"computers",
"employees",
"endpoints",
"geenforce",
"guides",
"knowledgebase",
"machines",
"measuringtools",
"network",
"notifications",
"printedparts",
"printers",
"slides",
"tools",
"usb",
"warranty"
],
"locked": [],
"_comment": "The profile the released Windows installer is built from. Every bundled plugin that carries a manifest, so ONE exe serves any site: the wizard offers all of them and the operator ticks what that site uses. Plugins left unticked are never installed, and 'flask plugin prune-schema' drops their tables at provisioning (ADR-014). 'applications' is deliberately absent - it is manifest-less core and always ships. Build with: deploy/windows/installer/build-installer.sh deploy/site-profile-universal.json <repo>. Use site-profile.example.json instead only when a site genuinely needs a lean build; see ADR-013."
"_comment": "Every plugin this branch bundles, which is what a receiving site installs by default. It is the profile docs/DEPLOY.md provisions from: 'flask plugin apply-profile deploy/site-profile-universal.json'. A plugin named here that is not on disk fails the command loudly, so this file and plugins/ must agree - they did not on 2026-09-19, when this still named five plugins that had been left on the site line, and provisioning a fresh site failed with PluginError. 'applications' is deliberately absent: it is manifest-less core and always ships. A site that wants fewer than all twelve copies site-profile.example.json and prunes; 'flask plugin prune-schema' drops the tables of anything left out (ADR-014, ADR-013)."
}

6
deploy/windows/installer/.gitignore vendored
View File

@@ -1,6 +0,0 @@
# Build output and staged payload - regenerable, and ~220MB.
bundle/
Output/
# Generated by build-installer.sh from the staged bundle.
plugins.iss
version.iss

247
deploy/windows/installer/README.md
View File

@@ -1,247 +0,0 @@
# Windows installer
Builds a single self-contained `.exe` that installs ShopDB-Flask on an
**air-gapped** Windows Server. Nothing here ever touches the network at install
time: Python, the wheels, the SPA and (optionally) MySQL all ship inside it.
Lives with the application on purpose. The installer depends on app internals -
`flask plugin` verbs, `site-profile.json`, `MOUNT_PATH`, the plugin registry -
so a separate repo would drift out of step with the thing it installs.
## Files
| File | What it is |
|---|---|
| `shopdb-preflight.ps1` | Stage 1. Read-only. Changes nothing, reports what this server is missing. |
| `shopdb-install.ps1` | Stages 0 and 2-5 plus `uninstall`. All the actual work. |
| `shopdb-admin.ps1` | Operator console installed alongside the app: status, restart, logs, backup, plugins. |
| `ShopDBFlask.iss` | Inno Setup wizard. A thin wrapper - it collects input and runs the stages. |
| `build-installer.sh` | Stages the bundle from a site profile. |
| `make-branding.py` | Generates wizard artwork and the icon from `frontend/public/*.svg`. |
| `*.bmp`, `shopdb.ico` | Generated artwork, committed so a Windows build box needs no Python. |
| `build-installer.ps1` | The same build, natively on Windows. No Bash needed. |
| `bundle-lock.json` | The exact third-party payload this installer ships. Reviewed by commit. |
| `bundle-lock.ps1` | Creates and checks that lock. Also runs on the target server. |
| `refresh-bundle-lock.ps1` | Regenerates the lock, after showing what changed. |
| `verify_bundle_lock.py` | The same check for the Bash builder, so Linux needs no pwsh. |
## Building
Two builders, same result. Use whichever machine you are on - `build-installer.ps1`
does the whole job natively so a Windows work PC needs no Bash.
```bash
# Linux / WSL
./build-installer.sh ../../site-profile.example.json
```
```powershell
# Windows
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
```
Both stage the app tree, build the SPA twice, write `plugins.iss`, copy the
installer scripts **from this directory**, and then verify the third-party
payload against `bundle-lock.json`. They exit non-zero if it does not match.
The payload itself is added by hand, and no longer needs Windows to produce:
```
bundle/wheels/ pip download -r requirements.txt --only-binary=:all: \
--platform win_amd64 --python-version 314 \
--implementation cp --abi cp314 -d wheels
bundle/python/ python-3.14.x-amd64.exe
bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below)
bundle/vcredist/ VC_redist.x64.exe (MySQL will not install without it)
bundle/mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)
bundle/mysqlclient/ mysql.exe, mysqldump.exe + libcrypto/libssl (backups)
```
Then compile on Windows:
```
iscc ShopDBFlask.iss
```
**Inno Setup 6.6.0 or newer.** The wizard uses the built-in `windows11` custom
style, which earlier versions reject. The script fails at compile time with that
sentence rather than with a bare "WizardStyle is invalid".
The wheelhouse is **cp314-locked**. A different Python minor version means a
different wheelhouse; the installer will not use a Python it did not install.
## What is pinned, and where
Three layers, because no one of them covers the whole problem.
| Layer | Covers | Enforced |
|---|---|---|
| `requirements.txt` sha256 per package | every wheel is genuinely what upstream published | `pip --require-hashes` at install; aborts on mismatch |
| `bundle-lock.json` | the EXACT payload: wheels, Python installer, MSIs | both builders, and again on the server before anything runs |
| git | the application tree | code review |
`--require-hashes` alone is not enough. pip lists every artifact of a pinned
version - `cffi 2.1.0` has 100 hashes - so it proves the wheel is genuine, not
that it is the wheel this bundle was built and tested with. It also ignores
extra files in the wheelhouse, and says nothing about the Python installer or
the MSIs, all of which run as SYSTEM on the target server.
So `bundle-lock.json` records an exact file set with a sha256 and a byte size
each, and verification is **set equality**: a missing file, an unexpected extra
file, or changed content all fail. There is no install-time override.
The lockfile is `--universal`, so one file serves Linux (dev, Docker, CI) and the
Windows wheelhouse. A Linux-only resolve had silently omitted `colorama`, a
win32-only dependency of `click` - which in hash-checking mode is a hard error
rather than a quiet omission.
### Verifying the installer itself
`bundle-lock.json` is **inside** the thing it describes, so it proves the payload
was not altered between build and install - not that the `.exe` you received is
the one that was built. That needs something out of band. Two options, in order
of preference:
1. **Authenticode-sign the `.exe`** with a GE code-signing certificate. Windows
then shows a real publisher instead of "Unknown", which is also what stops an
operator learning to click through the SmartScreen warning.
2. **Publish a sha256 per release** through a different channel than the file
itself, and have the receiving site check it:
`Get-FileHash ShopDBFlask_Installer_*.exe -Algorithm SHA256`
Neither is wired up yet. Until one is, an installer is only as trustworthy as
the share it arrived on.
### Changing what ships
```powershell
.\refresh-bundle-lock.ps1 # show what changed, write nothing
.\refresh-bundle-lock.ps1 -Yes # write it
```
Then **commit `bundle-lock.json`**. That commit is the review - it is the only
place a change to what runs as SYSTEM on a customer's server becomes visible to
a human. `refresh-bundle-lock.ps1` refuses to overwrite an existing lock until
you have seen the diff, for that reason.
To stage a bundle before its lock exists: `ALLOW_UNLOCKED=1` (Bash) or
`-AllowUnlocked` (PowerShell). Bundles built that way must not be shipped.
Verifying a live server, months later and offline:
```powershell
shopdb-admin.ps1 verify
```
## Servers this installer did not build
It is built for greenfield: its own Python, its own venv, its own IIS objects.
Its upgrade path assumes the thing being upgraded came out of a previous run.
Two guards keep it from damaging a server that was deployed by hand.
**Python minor version.** An existing venv is reused, which is right for a repair
or an upgrade. It is wrong when the venv belongs to a different Python - the
wheelhouse is tagged for one minor version, so pip would die at the first
compiled package, *after* Python was installed and the app tree replaced. The
installer compares the two up front and stops with both version numbers.
**IIS objects.** Switching deployment method removes the other method's artifact,
which is correct when the installer owns both and dangerous when it does not: a
wrong `-MountAlias` would delete a live mount with no prompt. It now refuses
unless there is a version stamp proving it made the install, or you pass
`-AdoptExisting`. The refusal lists exactly what it would have removed.
An existing `web.config` is never overwritten in either case.
For the West Jefferson production server specifically, this is a **migration, not
an upgrade** - prod runs Python 3.13 against a hand-built deployment, so it needs
a deliberate window, a database backup, and web.config reconciled by hand.
## Bill of materials
Every build stages a CycloneDX 1.6 SBOM at `sbom.cdx.json`, inside the
application tree, so it installs onto the server with the app. Both ecosystems,
in one document:
- **Python** - every pin in `requirements.txt`, with the sha256 the installer
enforces. Environment markers are ignored: a `sys_platform == 'win32'`
dependency still installs on the target.
- **npm** - every package in `frontend/package-lock.json`. Build-only packages
are marked `scope: excluded` rather than dropped, so "not here" is
distinguishable from "not looked for".
It ships to the server because an air-gapped site cannot be scanned from
anywhere else. When a CVE lands, the answer is already on the box:
```powershell
shopdb-admin.ps1 verify # counts, and which bundle this is
shopdb-admin.ps1 verify -Path leaflet # is that component here, at what version
```
Generated by `scripts/generate_sbom.py` from files that are already pinned and
committed, so it is a translation rather than a scan - no network, no extra
toolchain on the build box, and byte-identical output for the same inputs. It is
deliberately not in `bundle-lock.json`: its provenance is git, not the payload.
## Client IP addresses
IIS does not set `X-Forwarded-For` on its own, and HttpPlatformHandler connects
from loopback. Without a rule, **every client reads as 127.0.0.1** - so the
GE-Enforce IP allowlist, the dashboard's visitor-location lookup and per-host
login rate limiting all stop working, silently.
The wizard asks, because the two answers are mutually exclusive:
- **Clients connect directly** (`-ClientIpSource direct`, the default) - installs
URL Rewrite from the bundle and sets `X-Forwarded-For` from `REMOTE_ADDR`.
Overwriting the header is what stops a client spoofing its own.
- **A proxy sits in front** (`-ClientIpSource proxy`) - leaves the rule off.
Behind ARR or a load balancer `REMOTE_ADDR` is the *proxy*, so applying the
rule would discard the real client IP.
An existing `web.config` is never overwritten - it is the one file on a server
that legitimately carries hand-edits. The installer reports what it found
instead.
## Deployment methods
Chosen in the wizard, and the bundle carries a SPA build for each because Vite
compiles the base path in - it cannot be switched at install time.
- **Its own site** on a port (default 8090).
- **Subpath** under an existing site, e.g. `http://<server-fqdn>/shopdb/`. Needs
no new DNS record. Three things must agree - the IIS application alias,
`MOUNT_PATH` in `.env`, and the SPA's build-time base - so the alias is fixed
per bundle (`SUBPATH_ALIAS`, default `shopdb`) and the installer refuses if the
bundle's build does not match what was asked for.
Switching between methods removes the other one's IIS artifact and reconciles
`MOUNT_PATH` and `CORS_ORIGINS`, so a server never ends up with both.
## Upgrades
Run a newer installer over an existing install. It:
- backs the database up first, **verifies** the dump is complete, and refuses to
migrate if it cannot;
- restores from that backup if migrations fail, and reports honestly that DDL
the failed migration committed cannot be undone;
- refuses to run a bundle older than what is installed;
- keeps `.env` unless new credentials are supplied, and copies it aside first;
- stops the app pool before replacing files, then starts it again.
Whether a run is an upgrade is decided by probing the **target database**, not by
whether the app directory exists - a rebuilt server pointed at an existing
database is an upgrade, and treating it as fresh would drop tables.
## Testing notes
Verified end to end on Windows Server 2025 against both a bundled MySQL 8.4 LTS and
an existing MySQL 5.6: fresh install, upgrade, re-run idempotency, failure and
rollback, uninstall, and both deployment methods including switching between
them.
**Not yet verified:** a fully air-gapped run with the network disabled at the
hypervisor, and any load in a real browser (all HTTP checks so far used curl,
which sends no `Origin` header - so `CORS_ORIGINS` is untested in anger).

1340
deploy/windows/installer/ShopDBFlask.iss
View File

File diff suppressed because it is too large Load Diff

301
deploy/windows/installer/build-installer.ps1
View File

@@ -1,301 +0,0 @@
<#
.SYNOPSIS
Stage a lean per-site installer bundle on Windows, and verify its third-party
payload against bundle-lock.json.
.DESCRIPTION
The Windows equivalent of build-installer.sh, for a work PC with no Bash. It
does the whole job natively - it does NOT shell out to build-site.sh - so the
only tools needed are Python, Node and (to compile) Inno Setup.
What it does, in order:
1. Resolves the site's plugin closure from its profile.
2. Builds the SPA twice, because Vite compiles the base path in and it
therefore cannot be chosen at install time: once for /<alias> (subpath
deployment) and once for / (its own site).
3. Stages the backend tree: core, the chosen plugins only, and the runtime
files a deployable tree needs.
4. Writes plugins.iss so the wizard's plugin page matches the payload.
5. Copies the installer scripts from THIS directory.
6. Verifies wheels\, python\ and the MSIs against bundle-lock.json, and
FAILS if the bundle is not exactly what the lock describes.
Step 6 is the point of the script. A bundle that does not match its lock is
not shipped, and "the wheelhouse was missing" is caught here rather than by
an operator halfway through installing on a server with no network.
.PARAMETER Profile
Path to the site profile (see deploy\site-profile.example.json).
.PARAMETER RepoRoot
The repository. Defaults to four levels up from this script, which is correct
for a normal checkout.
.PARAMETER SkipFrontend
Reuse the SPA builds already staged in the bundle. For iterating on the
installer itself, where two Vite builds per run is most of the wall clock.
Never use it for a bundle you intend to ship.
.PARAMETER AllowUnlocked
Stage the bundle and report payload problems WITHOUT failing. For assembling
a bundle before its lock exists. A bundle built this way must not be shipped;
run refresh-bundle-lock.ps1, commit the lock, then build again without this.
.EXAMPLE
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
.\build-installer.ps1 -Profile C:\sites\wjf.json -SkipFrontend
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)] [string] $Profile,
[string] $RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path,
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
[string] $SubpathAlias = 'shopdb',
[switch] $SkipFrontend,
[switch] $AllowUnlocked
)
$ErrorActionPreference = 'Stop'
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
function Step { param($m) Write-Host ''; Write-Host "==> $m" -ForegroundColor Cyan }
function Die { param($m, $fix = '') Write-Host ''; Write-Host " $m" -ForegroundColor Red
if ($fix) { Write-Host " $fix" -ForegroundColor Yellow }; exit 1 }
function Invoke-Tool {
# Runs a build tool and stops on a non-zero exit. npm writes progress to
# stderr on SUCCESS, so stderr alone must never be treated as failure.
param([string] $Exe, [string[]] $Arguments, [string] $WorkDir, [string] $What)
Push-Location $WorkDir
try {
$prev = $ErrorActionPreference
$ErrorActionPreference = 'Continue'
& $Exe @Arguments 2>&1 | ForEach-Object { Say " $_" 'DarkGray' }
$code = $LASTEXITCODE
$ErrorActionPreference = $prev
if ($code -ne 0) { Die "$What failed (exit $code)" }
} finally { Pop-Location }
}
if (-not (Test-Path $Profile)) { Die "profile not found: $Profile" }
if (-not (Test-Path $RepoRoot)) { Die "repo not found: $RepoRoot" }
$Profile = (Resolve-Path $Profile).Path
$RepoRoot = (Resolve-Path $RepoRoot).Path
$AppOut = Join-Path $BundleRoot 'app'
$python = (Get-Command python -ErrorAction SilentlyContinue)
if (-not $python) { $python = Get-Command py -ErrorAction SilentlyContinue }
if (-not $python) { Die 'Python not found on PATH' 'Install Python 3.14 and re-open the shell.' }
$npm = Get-Command npm.cmd -ErrorAction SilentlyContinue
if (-not $npm -and -not $SkipFrontend) { Die 'npm not found on PATH' 'Install Node, or pass -SkipFrontend to reuse the staged SPA.' }
Say ''
Say " repo : $RepoRoot"
Say " profile : $Profile"
Say " bundle : $BundleRoot"
# --- 1. plugin closure ------------------------------------------------------
# Same resolver the Linux builder uses, so both produce the same set from one
# profile instead of two implementations of the closure rules.
Step 'Resolving plugin closure'
$closure = (& $python.Source (Join-Path $RepoRoot 'scripts\resolve_plugin_closure.py') $Profile $RepoRoot)
if ($LASTEXITCODE -ne 0 -or -not $closure) { Die 'could not resolve the plugin closure from the profile' }
$closure = $closure.Trim()
Say " $closure" 'White'
# --- 2. frontend ------------------------------------------------------------
$frontend = Join-Path $RepoRoot 'frontend'
$subStaged = Join-Path $BundleRoot 'spa-subpath'
$rootStaged= Join-Path $BundleRoot 'spa-root'
if ($SkipFrontend) {
if (-not (Test-Path $subStaged) -or -not (Test-Path $rootStaged)) {
Die '-SkipFrontend was passed but no SPA build is staged' 'Run once without it.'
}
Say ''
Say ' Reusing the staged SPA builds (-SkipFrontend). NOT shippable if the frontend changed.' 'Yellow'
} else {
Step "Building the SPA for /$SubpathAlias/"
$env:SITE_PLUGINS = $closure
$env:VITE_BASE_PATH = "/$SubpathAlias/"
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'subpath frontend build' }
finally { Remove-Item Env:\VITE_BASE_PATH -ErrorAction SilentlyContinue }
Remove-Item $subStaged -Recurse -Force -ErrorAction SilentlyContinue
Copy-Item (Join-Path $frontend 'dist') $subStaged -Recurse -Force
# The alias is fixed at BUILD time; the installer reads this and refuses to
# publish under a different one rather than serving a page that cannot load
# its own assets.
Set-Content -Path (Join-Path $subStaged '.alias') -Value $SubpathAlias -Encoding ASCII
# Root build LAST, so frontend\dist is left in the state a developer expects.
Step 'Building the SPA for /'
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'root frontend build' }
finally { Remove-Item Env:\SITE_PLUGINS -ErrorAction SilentlyContinue }
Remove-Item $rootStaged -Recurse -Force -ErrorAction SilentlyContinue
Copy-Item (Join-Path $frontend 'dist') $rootStaged -Recurse -Force
}
# --- 3. backend tree --------------------------------------------------------
Step "Staging the application tree"
Remove-Item $AppOut -Recurse -Force -ErrorAction SilentlyContinue
New-Item -ItemType Directory -Path (Join-Path $AppOut 'plugins') -Force | Out-Null
Copy-Item (Join-Path $RepoRoot 'shopdb') $AppOut -Recurse -Force
foreach ($name in $closure.Split(',')) {
$src = Join-Path $RepoRoot ('plugins\' + $name.Trim())
if (-not (Test-Path $src)) { Die "plugin in the closure is not on disk: $name" }
Copy-Item $src (Join-Path $AppOut 'plugins') -Recurse -Force
}
# Runtime files a deployable tree needs beyond the Python packages. Without
# these the tree imports but cannot be run or migrated.
foreach ($f in @('wsgi.py', 'requirements.txt')) {
Copy-Item (Join-Path $RepoRoot $f) $AppOut -Force
}
Copy-Item (Join-Path $RepoRoot 'migrations') $AppOut -Recurse -Force
# ONLY web.config, not all of deploy\. The bundle is staged INSIDE deploy\, so
# copying the whole tree would recurse into its own output, and the rest of
# deploy\ is installer source that has no business on an application server.
# shopdb-install.ps1 reads it from exactly this path.
$cfgSrc = Join-Path $RepoRoot 'deploy\windows\web.config'
if (Test-Path $cfgSrc) {
New-Item -ItemType Directory -Path (Join-Path $AppOut 'deploy\windows') -Force | Out-Null
Copy-Item $cfgSrc (Join-Path $AppOut 'deploy\windows') -Force
}
# A CycloneDX SBOM of everything this tree depends on, Python and npm together.
# Staged INTO the tree so it installs onto the server with the application: an
# air-gapped site cannot be scanned remotely, so the only way to answer "are we
# exposed to this CVE, and where" is for the answer to be sitting on the box.
Step 'Generating SBOM'
& $python.Source (Join-Path $RepoRoot 'scripts\generate_sbom.py') $RepoRoot `
-o (Join-Path $AppOut 'sbom.cdx.json') | ForEach-Object { Say " $_" 'White' }
if ($LASTEXITCODE -ne 0) { Die 'SBOM generation failed' }
# Docs the running site serves, plus the runbooks an air-gapped server has no
# other way to reach. Without openapi.json and llms.txt the self-hosted /api/docs
# page is broken on every installed server.
Step 'Staging docs'
$docsOut = Join-Path $AppOut 'docs'
New-Item -ItemType Directory -Path $docsOut -Force | Out-Null
foreach ($doc in @('openapi.json', 'llms.txt', 'api-inventory.json',
'INSTALL-WINDOWS.md', 'OPERATE-WINDOWS.md',
'BACKUP-RESTORE.md', 'UPGRADE.md')) {
$src = Join-Path $RepoRoot ('docs\' + $doc)
if (Test-Path $src) { Copy-Item $src $docsOut -Force; Say " $doc" }
}
# Stage the profile INTO the tree: `flask plugin apply-profile` at provisioning
# reads the same profile the tree was staged from, so the installed plugin set
# and the shipped plugin code cannot drift.
Copy-Item $Profile (Join-Path $AppOut 'site-profile.json') -Force
# The installer expects frontend\dist and frontend\dist-subpath.
$feOut = Join-Path $AppOut 'frontend'
New-Item -ItemType Directory -Path $feOut -Force | Out-Null
Copy-Item $rootStaged (Join-Path $feOut 'dist') -Recurse -Force
Copy-Item $subStaged (Join-Path $feOut 'dist-subpath') -Recurse -Force
Get-ChildItem $AppOut -Recurse -Directory -Filter '__pycache__' -ErrorAction SilentlyContinue |
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
Get-ChildItem $AppOut -Recurse -File -Filter '*.pyc' -ErrorAction SilentlyContinue |
Remove-Item -Force -ErrorAction SilentlyContinue
# --- 4. plugins.iss ---------------------------------------------------------
Step 'Writing plugins.iss'
$shipped = (Get-ChildItem (Join-Path $AppOut 'plugins') -Directory | Select-Object -ExpandProperty Name) -join ','
$aliasBuilt = ''
$aliasFile = Join-Path $feOut 'dist-subpath\.alias'
if (Test-Path $aliasFile) { $aliasBuilt = (Get-Content $aliasFile -TotalCount 1).Trim() }
@"
; GENERATED by build-installer.ps1 - do not edit.
; The plugins present in bundle\app\plugins at build time.
#define AvailablePlugins "$shipped"
; The alias the subpath SPA was built for, or empty if this bundle has no
; subpath build - in which case the wizard must not offer that option.
#define SubpathAlias "$aliasBuilt"
"@ | Set-Content -Path (Join-Path $PSScriptRoot 'plugins.iss') -Encoding ASCII
Say " $shipped" 'White'
# The product version, read from the code rather than restated in the .iss.
Step 'Writing version.iss'
$initText = Get-Content (Join-Path $RepoRoot 'shopdb\__init__.py') -Raw
if ($initText -notmatch "(?m)^__version__\s*=\s*'([^']+)'") { Die 'could not read __version__ from shopdb\__init__.py' }
$appVersion = $Matches[1]
@"
; GENERATED by build-installer.ps1 from shopdb/__init__.py - do not edit.
#define AppVersion "$appVersion"
"@ | Set-Content -Path (Join-Path $PSScriptRoot 'version.iss') -Encoding ASCII
Say " $appVersion" 'White'
# --- 5. installer scripts ---------------------------------------------------
# From THIS directory, which is the reviewed copy under version control. They
# used to be copied from a downloads folder, so the logic that shipped was not
# the logic that was committed and the build only worked on one machine.
Step 'Copying installer scripts'
foreach ($f in @('shopdb-install.ps1', 'shopdb-preflight.ps1', 'bundle-lock.ps1')) {
$src = Join-Path $PSScriptRoot $f
if (-not (Test-Path $src)) { Die "installer script missing from the repo: $f" }
Copy-Item $src $BundleRoot -Force
Say " $f"
}
# --- 6. payload verification ------------------------------------------------
Step 'Verifying the third-party payload against bundle-lock.json'
$lockPath = Join-Path $PSScriptRoot 'bundle-lock.json'
$lock = Read-BundleLock $lockPath
if (-not $lock) {
$msg = "no bundle-lock.json at $lockPath"
if ($AllowUnlocked) { Say " $msg - continuing because -AllowUnlocked was passed" 'Yellow' }
else {
Die $msg @'
Add the wheels and installers to the bundle, then:
.\refresh-bundle-lock.ps1 (review the list)
.\refresh-bundle-lock.ps1 -Yes (write it)
and commit bundle-lock.json. Pass -AllowUnlocked to stage a bundle without one -
it must not be shipped.
'@
}
} else {
$problems = Test-BundleLock -BundleRoot $BundleRoot -Lock $lock
if ($problems.Count -eq 0) {
Say (" payload matches the lock ({0}, {1})" -f `
(Get-JsonProperty $lock 'pythontag' 'unknown'), (Get-JsonProperty $lock 'platform' 'unknown')) 'Green'
# Ships WITH the bundle: the installer re-checks the payload on the
# target server before running any of it, so tampering between build and
# install is caught too.
Copy-Item $lockPath $BundleRoot -Force
} else {
Say ''
foreach ($p in $problems) { Say " $p" 'Red' }
Say ''
if ($AllowUnlocked) {
Say ' -AllowUnlocked: continuing anyway. DO NOT SHIP this bundle.' 'Yellow'
} else {
Die ("{0} payload problem(s) - the bundle is not what the lock describes" -f $problems.Count) @'
Either the payload is wrong (fix the bundle) or it changed on purpose (run
refresh-bundle-lock.ps1, read the diff, and commit the new lock).
'@
}
}
}
# --- summary ----------------------------------------------------------------
Write-Host ''
Say " Bundle staged at: $BundleRoot" 'White'
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'vcredist', 'mysqlclient', 'mysql')) {
$p = Join-Path $BundleRoot $d
if (Test-Path $p) {
$mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB)
Say (" {0,-22} {1,8:N1} MB" -f $d, $mb)
} else {
Say (" {0,-22} {1}" -f $d, 'absent') 'DarkGray'
}
}
Write-Host ''
Say ' Compile with: iscc ShopDBFlask.iss' 'White'
Say ' (Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)' 'DarkGray'
Write-Host ''

141
deploy/windows/installer/build-installer.sh
View File

@@ -1,141 +0,0 @@
#!/bin/bash
# Stage a lean per-site bundle next to ShopDBFlask.iss, ready for Inno Setup.
#
# The bundle is built FOR ONE SITE from its plugin profile (ADR-013): plugins the
# site did not choose are absent from the payload entirely. Build one installer
# per site, not one universal installer.
#
# Usage: build-installer.sh <site-profile.json> [repo-path]
#
# The wheelhouse cannot be built here. Wheels are cp314 win_amd64 and must be
# produced ON Windows with the matching Python:
# pip download -r requirements.txt -d wheels --only-binary=:all:
# Copy that wheels\ directory in before compiling.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROFILE="${1:?usage: build-installer.sh <site-profile.json> [repo-path]}"
REPO="${2:-$HOME/projects/shopdb-flask}"
BUNDLE="$HERE/bundle"
[ -f "$PROFILE" ] || { echo "profile not found: $PROFILE"; exit 1; }
[ -d "$REPO" ] || { echo "repo not found: $REPO"; exit 1; }
echo "==> Staging lean app tree from $PROFILE"
rm -rf "$BUNDLE/app"
bash "$REPO/scripts/build-site.sh" "$PROFILE" "$BUNDLE/app"
# build-site.sh emits the SPA as frontend-dist; the installer's web.config and
# static route expect frontend\dist.
if [ -d "$BUNDLE/app/frontend-dist" ]; then
mkdir -p "$BUNDLE/app/frontend"
rm -rf "$BUNDLE/app/frontend/dist"
mv "$BUNDLE/app/frontend-dist" "$BUNDLE/app/frontend/dist"
fi
# The /shopdb-based build, used when the operator picks the subpath deployment.
if [ -d "$BUNDLE/app/frontend-dist-subpath" ]; then
rm -rf "$BUNDLE/app/frontend/dist-subpath"
mv "$BUNDLE/app/frontend-dist-subpath" "$BUNDLE/app/frontend/dist-subpath"
fi
# Tell the .iss which plugins this bundle actually carries, so the wizard's
# plugin page always matches the payload instead of a hand-maintained list.
echo "==> Writing plugins.iss"
PLUGINS=$(ls "$BUNDLE/app/plugins" 2>/dev/null | tr '\n' ',' | sed 's/,$//')
# The subpath SPA is built with its base path compiled in, so whether the wizard
# can OFFER a subpath install is a property of the bundle, not a runtime choice.
SUBPATH_ALIAS_BUILT=""
if [ -f "$BUNDLE/app/frontend/dist-subpath/.alias" ]; then
SUBPATH_ALIAS_BUILT="$(cat "$BUNDLE/app/frontend/dist-subpath/.alias")"
fi
cat > "$HERE/plugins.iss" <<EOF
; GENERATED by build-installer.sh - do not edit.
; The plugins present in bundle\\app\\plugins at build time.
#define AvailablePlugins "$PLUGINS"
; The alias the subpath SPA was built for, or empty if this bundle has no
; subpath build - in which case the wizard must not offer that option.
#define SubpathAlias "$SUBPATH_ALIAS_BUILT"
EOF
echo " $PLUGINS"
# The product version, read from the code rather than restated here. A hardcoded
# AppVersion in the .iss had drifted two minor versions from shopdb/__init__.py.
echo "==> Writing version.iss"
APPVERSION=$(sed -n "s/^__version__ = '\\(.*\\)'/\\1/p" "$REPO/shopdb/__init__.py" | head -1)
[ -n "$APPVERSION" ] || { echo "could not read __version__ from shopdb/__init__.py"; exit 1; }
cat > "$HERE/version.iss" <<EOF
; GENERATED by build-installer.sh from shopdb/__init__.py - do not edit.
#define AppVersion "$APPVERSION"
EOF
echo " $APPVERSION"
# From THIS directory, which is the reviewed copy under version control. These
# used to be copied from $HOME/Downloads, so the installer logic that shipped was
# not the logic that was committed, and the build only worked on one machine.
echo "==> Copying installer scripts"
mkdir -p "$BUNDLE"
for f in shopdb-install.ps1 shopdb-preflight.ps1 bundle-lock.ps1; do
[ -f "$HERE/$f" ] || { echo "installer script missing from the repo: $f"; exit 1; }
cp "$HERE/$f" "$BUNDLE/"
done
echo ""
echo "Bundle staged at: $BUNDLE"
for d in app wheels python httpplatformhandler urlrewrite vcredist mysqlclient mysql; do
if [ -d "$BUNDLE/$d" ]; then
printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)"
else
printf ' %-20s absent\n' "$d"
fi
done
echo ""
echo " plugins shipped: $(ls "$BUNDLE/app/plugins" 2>/dev/null | tr '\n' ' ')"
# --- payload verification ---------------------------------------------------
# The third-party payload is the part git does not record: the wheels, the Python
# installer and the MSIs that run as SYSTEM on the target server. It must be
# EXACTLY what bundle-lock.json describes - no missing file, no stale extra wheel
# left over from a previous build, no changed content - or this is not a bundle
# anyone reviewed. Previously a missing wheelhouse printed MISSING and the script
# still exited 0, so an empty bundle compiled into a shippable installer and the
# failure surfaced on an air-gapped server with no way to fix it.
#
# ALLOW_UNLOCKED=1 downgrades this to a warning, for assembling a bundle before
# its lock exists. A bundle built that way must not be shipped.
echo ""
echo "==> Verifying the third-party payload against bundle-lock.json"
if python3 "$HERE/verify_bundle_lock.py" "$BUNDLE" "$HERE/bundle-lock.json"; then
echo " payload matches the lock"
# Ships WITH the bundle: the installer re-checks the payload on the target
# server before running any of it, so tampering between build and install is
# caught too.
cp "$HERE/bundle-lock.json" "$BUNDLE/"
elif [ "${ALLOW_UNLOCKED:-0}" = "1" ]; then
echo ""
echo " ALLOW_UNLOCKED=1: continuing anyway. DO NOT SHIP this bundle."
else
echo ""
echo " The bundle is not what the lock describes."
echo ""
echo " Add the missing pieces by hand:"
echo " wheels/ pip download -r requirements.txt --only-binary=:all: \\"
echo " --platform win_amd64 --python-version 314 \\"
echo " --implementation cp --abi cp314 -d wheels"
echo " python/ python-3.14.x-amd64.exe"
echo " httpplatformhandler/ httpPlatformHandler_amd64.msi"
echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)"
echo " vcredist/ VC_redist.x64.exe (MySQL requires it)"
echo " mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)"
echo ""
echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:"
echo " pwsh ./refresh-bundle-lock.ps1 # review the diff"
echo " pwsh ./refresh-bundle-lock.ps1 -Yes # write it"
echo ""
echo " To stage a bundle before its lock exists: ALLOW_UNLOCKED=1 $0 ..."
exit 1
fi
echo ""
echo "Then compile on Windows: iscc ShopDBFlask.iss"
echo "(Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)"

240
deploy/windows/installer/bundle-lock.json
View File

@@ -1,240 +0,0 @@
{
"schema": 1,
"generated": "2026-08-04T23:11:22Z",
"pythontag": "cp314",
"platform": "win_amd64",
"note": "Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.",
"payloads": {
"httpplatformhandler": {
"files": {
"httpPlatformHandler_amd64.msi": {
"sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef",
"size": 557056
}
},
"required": true
},
"python": {
"files": {
"python-3.14.6-amd64.exe": {
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd",
"size": 30774112
}
},
"required": true
},
"vcredist": {
"files": {
"VC_redist.x64.exe": {
"sha256": "cc0ff0eb1dc3f5188ae6300faef32bf5beeba4bdd6e8e445a9184072096b713b",
"size": 25635768
}
},
"required": false
},
"mysql": {
"files": {
"mysql-8.4.6-winx64.msi": {
"sha256": "868885b2e221409f0170729bb30b2b2ce83440a1a4c735d19b3c225b51045762",
"size": 135081984
}
},
"required": false
},
"wheels": {
"files": {
"email_validator-2.3.0-py3-none-any.whl": {
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4",
"size": 35604
},
"cachelib-0.13.0-py3-none-any.whl": {
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516",
"size": 20914
},
"flask_caching-2.4.0-py3-none-any.whl": {
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486",
"size": 28727
},
"typing_extensions-4.15.0-py3-none-any.whl": {
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548",
"size": 44614
},
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033",
"size": 239835
},
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f",
"size": 159634
},
"colorama-0.4.6-py2.py3-none-any.whl": {
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6",
"size": 25335
},
"Flask_Migrate-4.1.0-py3-none-any.whl": {
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d",
"size": 21237
},
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581",
"size": 15341
},
"flask_marshmallow-1.5.0-py3-none-any.whl": {
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6",
"size": 12161
},
"tzdata-2026.3-py2.py3-none-any.whl": {
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931",
"size": 348168
},
"python_dotenv-1.2.2-py3-none-any.whl": {
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a",
"size": 22101
},
"dnspython-2.8.0-py3-none-any.whl": {
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af",
"size": 331094
},
"pymysql-1.1.3-py3-none-any.whl": {
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289",
"size": 45356
},
"packaging-26.3-py3-none-any.whl": {
"sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c",
"size": 129956
},
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb",
"size": 22698
},
"urllib3-2.7.0-py3-none-any.whl": {
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897",
"size": 131087
},
"marshmallow-4.3.0-py3-none-any.whl": {
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46",
"size": 49148
},
"waitress-3.0.2-py3-none-any.whl": {
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e",
"size": 56232
},
"idna-3.13-py3-none-any.whl": {
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3",
"size": 68629
},
"flask-3.1.3-py3-none-any.whl": {
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c",
"size": 103424
},
"werkzeug-3.1.8-py3-none-any.whl": {
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50",
"size": 226459
},
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0",
"size": 25125
},
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258",
"size": 16582
},
"alembic-1.18.4-py3-none-any.whl": {
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a",
"size": 263893
},
"tabulate-0.10.0-py3-none-any.whl": {
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3",
"size": 39814
},
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb",
"size": 187937
},
"jinja2-3.1.6-py3-none-any.whl": {
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67",
"size": 134899
},
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30",
"size": 3840395
},
"certifi-2026.4.22-py3-none-any.whl": {
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a",
"size": 135707
},
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5",
"size": 2144204
},
"blinker-1.9.0-py3-none-any.whl": {
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc",
"size": 8458
},
"pycparser-3.0-py3-none-any.whl": {
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992",
"size": 48172
},
"flask_cors-6.0.2-py3-none-any.whl": {
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a",
"size": 13257
},
"requests-2.33.1-py3-none-any.whl": {
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a",
"size": 64947
},
"pyjwt-2.12.1-py3-none-any.whl": {
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c",
"size": 29726
},
"click-8.3.3-py3-none-any.whl": {
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613",
"size": 110502
},
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49",
"size": 18197576
},
"mako-1.3.12-py3-none-any.whl": {
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9",
"size": 78521
},
"itsdangerous-2.2.0-py3-none-any.whl": {
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef",
"size": 16234
}
},
"required": true
},
"urlrewrite": {
"files": {
"rewrite_amd64_en-US.msi": {
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a",
"size": 6078464
}
},
"required": false
},
"mysqlclient": {
"files": {
"libcrypto-3-x64.dll": {
"sha256": "e84ec61fc1b07a3c899ae39c1dc3f38591cdd310c043a388e7ab40a6828aa297",
"size": 5187216
},
"mysqldump.exe": {
"sha256": "329410b7a8ae3d68d38e6f5b781610770854d3f11b975060e231dae62c20dc46",
"size": 7142528
},
"mysql.exe": {
"sha256": "bd6880253c1853fc17b1e19129d6e5d9f9c86a2dccf2c7c9e6b878a10baa9770",
"size": 7177336
},
"libssl-3-x64.dll": {
"sha256": "e24fcd6c8ec31a14c97f520b5b960ed360c55ddf0935040eff297e511697633e",
"size": 788616
}
},
"required": false
}
}
}

261
deploy/windows/installer/bundle-lock.ps1
View File

@@ -1,261 +0,0 @@
<#
.SYNOPSIS
Hash manifest for the installer's third-party payload: create it, and check a
bundle against it.
.DESCRIPTION
Dot-source this. It defines two functions and runs nothing on its own:
New-BundleLock hash a staged bundle and return the lock object
Test-BundleLock compare a staged bundle against a lock, return problems
WHAT IT COVERS, and why pip's own hash checking is not enough.
requirements.txt carries a sha256 for every wheel, so pip refuses an artifact
upstream did not publish. Three gaps remain, and all three are what actually
goes wrong with a hand-assembled offline bundle:
1. pip lists EVERY artifact of a pinned version - cffi 2.1.0 alone has 100
hashes. It proves the wheel is genuine, not that it is the wheel this
bundle was built and tested with.
2. pip ignores extra files in the wheelhouse. A stale wheel left behind by
a previous build sits there unnoticed until a resolve picks it up.
3. pip says nothing about the rest of the payload - the Python installer,
the HttpPlatformHandler MSI, URL Rewrite, MySQL. Those are executables
that run as SYSTEM on the target server and were, until this file, the
only unverified thing the installer would run.
So the lock records an exact file set with a sha256 and a byte size each, and
verification is SET EQUALITY: a missing file, an unexpected extra file, or a
changed file all fail. Nothing is skipped and nothing is "close enough".
The app tree is deliberately NOT covered. It is built from the repository on
every run and changes with every commit; hashing it would make the lock churn
constantly and train everyone to regenerate it without reading it. Git is the
record for the app tree. This file is the record for everything that comes
from outside the repository.
.NOTES
Stock Windows PowerShell 5.1, and also runs under pwsh on Linux so the Bash
builder can call the same checker.
#>
# Payload directories under the bundle root. 'required' means the installer
# cannot work without it; the optional ones are per-deployment choices, and an
# absent optional directory is fine. A PRESENT directory is always checked in
# full, optional or not.
$script:BundlePayloads = @(
@{ Name = 'wheels'; Required = $true; What = 'Python wheels for the offline install' }
@{ Name = 'python'; Required = $true; What = 'the Python installer' }
@{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' }
@{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' }
@{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' }
@{ Name = 'vcredist'; Required = $false; What = 'the Visual C++ runtime MySQL requires' }
@{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' }
)
function Get-JsonProperty {
# shopdb-install.ps1 runs under Set-StrictMode 2.0, where reading a property
# that does not exist on a PSCustomObject THROWS instead of returning $null.
# A truncated or hand-edited bundle-lock.json would therefore blow up with
# "Property 'payloads' cannot be found" rather than saying what is wrong with
# the lock. Every read of parsed JSON goes through here.
param($Object, [string] $Name, $Default = $null)
if ($null -eq $Object) { return $Default }
$prop = $Object.PSObject.Properties[$Name]
if ($null -eq $prop) { return $Default }
return $prop.Value
}
function Get-FileDigest {
param([string] $Path)
$sha = [System.Security.Cryptography.SHA256]::Create()
$stream = [System.IO.File]::OpenRead($Path)
try { return (-join ($sha.ComputeHash($stream) | ForEach-Object { $_.ToString('x2') })) }
finally { $stream.Dispose(); $sha.Dispose() }
}
function Get-PayloadFiles {
<#
Every file in the directory, keyed by its path RELATIVE to that directory
with forward slashes, so a lock generated on Windows reads the same from
the Bash builder.
The root comes from Get-Item, NOT Resolve-Path, and the prefix is checked
before it is trimmed. Both matter, and a real install proved it:
Inno extracts the bundle under C:\Users\ADMINI~1\AppData\Local\Temp\... -
an 8.3 SHORT path. Resolve-Path kept that short form while Get-ChildItem
returned the long one (Administrator), so the root was five characters
shorter than the prefix it was slicing off. Every relative path came out
mangled - 'wheels/heels/flask.whl' - and the verifier reported all 96 files
as simultaneously missing and unexpected. The payload was fine; the
comparison was not.
Get-Item and Get-ChildItem go through the same provider, so their path
forms agree. The StartsWith guard means that if they ever disagree again
this fails loudly instead of inventing paths.
#>
param([string] $Dir)
$out = @{}
if (-not (Test-Path $Dir)) { return $out }
$rootItem = Get-Item -LiteralPath $Dir
$root = $rootItem.FullName.TrimEnd('\', '/')
foreach ($f in (Get-ChildItem -LiteralPath $rootItem.FullName -Recurse -File)) {
if (-not $f.FullName.StartsWith($root, [System.StringComparison]::OrdinalIgnoreCase)) {
throw ("cannot place '{0}' beneath '{1}' - path forms disagree (8.3 short name?)" -f $f.FullName, $root)
}
$rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/')
$out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length }
}
return $out
}
function New-BundleLock {
<#
Hash a staged bundle. The caller writes the result to bundle-lock.json;
this returns the object so a caller can diff it against the committed lock
before overwriting anything.
#>
param(
[Parameter(Mandatory = $true)] [string] $BundleRoot,
[string] $PythonTag = 'cp314',
[string] $Platform = 'win_amd64'
)
$payloads = @{}
foreach ($p in $script:BundlePayloads) {
$dir = Join-Path $BundleRoot $p.Name
if (-not (Test-Path $dir)) {
if ($p.Required) { throw ("payload directory is missing: {0} ({1})" -f $p.Name, $p.What) }
continue
}
$files = Get-PayloadFiles $dir
if ($files.Count -eq 0 -and $p.Required) {
throw ("payload directory is empty: {0} ({1})" -f $p.Name, $p.What)
}
$payloads[$p.Name] = @{ required = $p.Required; files = $files }
}
return [ordered]@{
schema = 1
generated = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
pythontag = $PythonTag
platform = $Platform
note = 'Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.'
payloads = $payloads
}
}
function Test-BundleLock {
<#
Compare a staged bundle against a lock. Returns an array of problem
strings - EMPTY means the bundle is exactly what the lock describes.
Returning problems rather than throwing is deliberate: an operator fixing a
wheelhouse wants the whole list at once, not one failure per rebuild.
#>
param(
[Parameter(Mandatory = $true)] [string] $BundleRoot,
[Parameter(Mandatory = $true)] $Lock
)
$problems = @()
$payloads = Get-JsonProperty $Lock 'payloads'
# ,@(...) for the same reason as the final return: a bare one-element array
# unrolls to a string, and the caller's .Count then measures the wrong thing.
if ($null -eq $payloads) { return ,@('bundle-lock.json has no "payloads" section') }
foreach ($p in $script:BundlePayloads) {
$name = $p.Name
$dir = Join-Path $BundleRoot $name
$present = Test-Path $dir
$entry = Get-JsonProperty $payloads $name
$locked = ($null -ne $entry)
if (-not $locked) {
# Not in the lock at all. An unlocked directory that exists is a
# payload nobody reviewed, which is exactly what this is here to stop.
if ($present) { $problems += "$name/ is present but is not in bundle-lock.json - regenerate the lock" }
elseif ($p.Required) { $problems += "$name/ is required but is in neither the bundle nor the lock" }
continue
}
if (-not $present) {
if ($p.Required -or (Get-JsonProperty $entry 'required' $false)) {
$problems += "$name/ is in the lock but missing from the bundle ($($p.What))"
}
continue
}
$expected = @{}
$lockedFiles = Get-JsonProperty $entry 'files'
if ($null -ne $lockedFiles) {
foreach ($prop in $lockedFiles.PSObject.Properties) { $expected[$prop.Name] = $prop.Value }
}
$actual = Get-PayloadFiles $dir
# Sorted so the report reads the same way twice, and so it matches the
# order verify_bundle_lock.py produces.
foreach ($rel in ($expected.Keys | Sort-Object)) {
if (-not $actual.ContainsKey($rel)) { $problems += "$name/$rel is in the lock but missing from the bundle"; continue }
if ($actual[$rel].sha256 -ne $expected[$rel].sha256) {
$problems += "$name/$rel does NOT match the lock (expected sha256 $($expected[$rel].sha256.Substring(0,12))..., got $($actual[$rel].sha256.Substring(0,12))...)"
} elseif ([int64] $actual[$rel].size -ne [int64] $expected[$rel].size) {
# Cannot happen for a matching sha256, so it means the lock itself
# was hand-edited. Say so rather than passing it.
$problems += "$name/$rel size disagrees with the lock - the lock has been edited by hand"
}
}
foreach ($rel in ($actual.Keys | Sort-Object)) {
if (-not $expected.ContainsKey($rel)) { $problems += "$name/$rel is in the bundle but NOT in the lock (unexpected extra file)" }
}
}
$problems += Test-WheelhouseCoversRequirements -BundleRoot $BundleRoot
# The comma keeps this an ARRAY through the return. Without it PowerShell
# unrolls an empty result to $null and a single result to a bare string, and
# every caller that asks for .Count then behaves differently depending on how
# many problems there happen to be.
return ,$problems
}
function Test-WheelhouseCoversRequirements {
<#
The lock records what IS in the wheelhouse, not what the application NEEDS.
Without this an incomplete wheelhouse gets locked, blessed, and shipped,
and the install fails on an air-gapped server.
Not hypothetical: assembling the wheelhouse anywhere other than Windows
silently omits colorama, a win32-only dependency of click, because pip
evaluates environment markers against the machine doing the downloading
rather than the machine being targeted. Markers are therefore IGNORED here
- a requirement guarded by sys_platform == 'win32' is precisely the one
that has to be present.
#>
param([Parameter(Mandatory = $true)] [string] $BundleRoot)
$wheels = Join-Path $BundleRoot 'wheels'
$reqs = Join-Path $BundleRoot 'app\requirements.txt'
if (-not (Test-Path $wheels) -or -not (Test-Path $reqs)) { return @() }
$have = @(Get-ChildItem $wheels -File -ErrorAction SilentlyContinue | ForEach-Object { $_.Name.ToLower() })
$problems = @()
$pins = @{}
foreach ($line in (Get-Content $reqs)) {
$trimmed = $line.Trim()
if (-not $trimmed -or $trimmed.StartsWith('#')) { continue }
if ($trimmed -match '^([A-Za-z0-9._-]+)==([^\s;\\]+)') {
# PEP 427 wheel filename form: runs of non-alphanumerics become one _.
$pins[([regex]::Replace($Matches[1], '[^A-Za-z0-9.]+', '_')).ToLower()] = $Matches[2]
}
}
foreach ($name in ($pins.Keys | Sort-Object)) {
$prefix = "$name-$($pins[$name])-"
if (-not ($have | Where-Object { $_.StartsWith($prefix) })) {
$problems += ("wheels/ has no wheel for {0}=={1}, which requirements.txt pins " +
"(a marked-out dependency still installs on Windows)") -f $name, $pins[$name]
}
}
return $problems
}
function Read-BundleLock {
param([Parameter(Mandatory = $true)] [string] $Path)
if (-not (Test-Path $Path)) { return $null }
return (Get-Content $Path -Raw | ConvertFrom-Json)
}

159
deploy/windows/installer/make-branding.py
View File

@@ -1,159 +0,0 @@
#!/usr/bin/env python3
"""Generate Inno Setup wizard artwork from the app's own brand assets.
Everything here is derived from frontend/public/*.svg so the installer and the
running application are visibly the same product. Nothing is redrawn by hand.
Inno stretches artwork to fit and does not resample well, so render at the exact
sizes it asks for and supply the 125%/250% variants for high-DPI displays.
WizardImageFile 164x314, 192x386, 384x772
WizardSmallImageFile 55x55, 64x64, 138x138
SetupIconFile .ico with 16/24/32/48/64/128/256
Usage: python3 make-branding.py [output-dir]
"""
import io
import sys
from pathlib import Path
import cairosvg
from PIL import Image, ImageDraw, ImageFont
ASSETS = Path.home() / "projects/shopdb-flask/frontend/public"
OUT = Path(sys.argv[1] if len(sys.argv) > 1 else Path(__file__).parent)
# Sampled from the application's own palette so the installer does not look like
# a different product wearing the same badge.
NAVY = (10, 34, 74) # deep base
BLUE = (16, 74, 150) # GE blue
CYAN = (0, 158, 224) # accent
WHITE = (255, 255, 255)
MUTED = (176, 197, 226)
def render_svg(name, width=None, height=None):
png = cairosvg.svg2png(url=str(ASSETS / name), output_width=width, output_height=height)
return Image.open(io.BytesIO(png)).convert("RGBA")
def recolour(img, colour):
"""Replace RGB while keeping the alpha mask. Source marks are dark-on-light;
on a dark panel they must be inverted or they disappear."""
solid = Image.new("RGBA", img.size, colour + (255,))
solid.putalpha(img.getchannel("A"))
return solid
def font(size, bold=False):
for path in (
f"/usr/share/fonts/truetype/dejavu/DejaVuSans{'-Bold' if bold else ''}.ttf",
f"/usr/share/fonts/truetype/liberation/LiberationSans{'-Bold' if bold else '-Regular'}.ttf",
):
if Path(path).exists():
return ImageFont.truetype(path, size)
return ImageFont.load_default()
def vertical_gradient(size, top, bottom):
w, h = size
img = Image.new("RGB", size)
draw = ImageDraw.Draw(img)
for y in range(h):
t = y / max(1, h - 1)
# Ease the ramp so the middle does not look flat.
t = t * t * (3 - 2 * t)
draw.line(
[(0, y), (w, y)],
fill=tuple(int(top[i] + (bottom[i] - top[i]) * t) for i in range(3)),
)
return img
def banner(w, h):
img = vertical_gradient((w, h), BLUE, NAVY)
draw = ImageDraw.Draw(img)
k = w / 164.0 # scale factor from the 100% design
# Faint diagonal wash: stops the flat area under the text reading as empty.
glow = Image.new("RGBA", (w, h), (0, 0, 0, 0))
gd = ImageDraw.Draw(glow)
gd.polygon([(0, int(h * 0.52)), (w, int(h * 0.30)), (w, h), (0, h)],
fill=(255, 255, 255, 10))
img = Image.alpha_composite(img.convert("RGBA"), glow).convert("RGB")
draw = ImageDraw.Draw(img)
margin = int(22 * k)
# GE Aerospace wordmark at the top, above the product name - the corporate
# mark leads, the product sits under it. (Previously the bare monogram was
# here and the wordmark was stranded at the bottom.)
mark_w = w - (margin * 2)
mark = render_svg("ge-aerospace-logo.svg", mark_w, int(mark_w * 32 / 138))
mark = recolour(mark, WHITE)
img.paste(mark, (margin, int(34 * k)), mark)
# Product name, directly beneath it.
y = int(34 * k) + mark.height + int(30 * k)
draw.text((margin, y), "ShopDB", font=font(int(26 * k), bold=True), fill=WHITE)
y += int(31 * k)
# Hairline rule, then the descriptor. Cheap way to look considered.
draw.rectangle([margin, y, margin + int(30 * k), y + max(1, int(2 * k))], fill=CYAN)
y += int(14 * k)
for line in ("Asset management", "for the shop floor"):
draw.text((margin, y), line, font=font(int(10.5 * k)), fill=MUTED)
y += int(15 * k)
# Accent bar flush to the bottom edge.
bar = max(2, int(4 * k))
draw.rectangle([0, h - bar, w, h], fill=CYAN)
return img
def small(size):
"""Header mark on every page after the welcome page. White plate so it sits
correctly on the wizard's own header, in light or dark mode."""
img = Image.new("RGB", (size, size), WHITE)
m = int(size * 0.80)
mono = recolour(render_svg("ge-monogram.svg", m, m), BLUE)
off = (size - m) // 2
img.paste(mono, (off, off), mono)
return img
def icon(path):
"""Installer icon. Rounded navy tile with the monogram, so it reads at 16px
instead of turning into mush."""
base = 256
img = Image.new("RGBA", (base, base), (0, 0, 0, 0))
d = ImageDraw.Draw(img)
d.rounded_rectangle([0, 0, base - 1, base - 1], radius=int(base * 0.22), fill=BLUE + (255,))
d.rounded_rectangle([0, 0, base - 1, int(base * 0.5)], radius=int(base * 0.22),
fill=(30, 96, 175, 255))
d.rounded_rectangle([0, int(base * 0.3), base - 1, base - 1], radius=int(base * 0.22),
fill=BLUE + (255,))
m = int(base * 0.62)
mono = recolour(render_svg("ge-monogram.svg", m, m), WHITE)
img.paste(mono, ((base - m) // 2, (base - m) // 2), mono)
img.save(path, sizes=[(16, 16), (24, 24), (32, 32), (48, 48), (64, 64),
(128, 128), (256, 256)])
def main():
OUT.mkdir(parents=True, exist_ok=True)
made = []
for w, h, name in [(164, 314, "wizard-image.bmp"),
(192, 386, "wizard-image@125.bmp"),
(384, 772, "wizard-image@250.bmp")]:
banner(w, h).save(OUT / name, "BMP"); made.append((name, f"{w}x{h}"))
for s, name in [(55, "wizard-small.bmp"), (64, "wizard-small@125.bmp"),
(138, "wizard-small@250.bmp")]:
small(s).save(OUT / name, "BMP"); made.append((name, f"{s}x{s}"))
icon(OUT / "shopdb.ico"); made.append(("shopdb.ico", "multi-size"))
for name, dims in made:
print(f" {name:<26} {dims:<10} {(OUT / name).stat().st_size // 1024} KB")
if __name__ == "__main__":
main()

110
deploy/windows/installer/refresh-bundle-lock.ps1
View File

@@ -1,110 +0,0 @@
<#
.SYNOPSIS
Regenerate bundle-lock.json from the staged bundle, after showing what would
change.
.DESCRIPTION
Run this on the machine that assembled the wheelhouse, once the bundle holds
the payload you intend to ship:
bundle\wheels\ the wheels, built with the matching Python
bundle\python\ the Python installer
bundle\httpplatformhandler\ the IIS module MSI
bundle\urlrewrite\ URL Rewrite MSI (optional)
bundle\mysql\ MySQL MSI (optional)
Then COMMIT the resulting bundle-lock.json. That commit is the review: it is
the only place a change to what runs as SYSTEM on a customer's server becomes
visible to a human. A lock regenerated and committed without reading the diff
provides nothing, so this refuses to overwrite an existing lock until you
have seen the change and passed -Yes.
.EXAMPLE
.\refresh-bundle-lock.ps1 # show the diff, write nothing
.\refresh-bundle-lock.ps1 -Yes # write it
.NOTES
Building the wheelhouse itself no longer requires Windows. From any machine:
pip download -r requirements.txt -d wheels --only-binary=:all: `
--platform win_amd64 --python-version 314 --implementation cp --abi cp314
Do it wherever you like; this script records what came out.
#>
[CmdletBinding()]
param(
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
[string] $LockPath = (Join-Path $PSScriptRoot 'bundle-lock.json'),
[string] $PythonTag = 'cp314',
[string] $Platform = 'win_amd64',
[switch] $Yes
)
$ErrorActionPreference = 'Stop'
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
if (-not (Test-Path $BundleRoot)) {
Say "bundle not found: $BundleRoot" 'Red'
Say 'Stage it first with build-installer.ps1 (or build-installer.sh), then add' 'Yellow'
Say 'the wheels and installers by hand.' 'Yellow'
exit 1
}
Say ''
Say " Hashing $BundleRoot" 'Cyan'
$new = New-BundleLock -BundleRoot $BundleRoot -PythonTag $PythonTag -Platform $Platform
foreach ($name in $new.payloads.Keys) {
Say (" {0,-22} {1,4} files" -f $name, $new.payloads[$name].files.Count)
}
$old = Read-BundleLock $LockPath
if (-not $old) {
Say ''
Say ' No existing lock - this will be the first one.' 'Yellow'
} else {
# Diff by file, per payload, so the operator sees exactly which artifacts
# changed rather than "the lock is different".
Say ''
Say ' Changes against the committed lock:' 'Cyan'
$changes = 0
foreach ($name in $new.payloads.Keys) {
$oldFiles = @{}
if ($old.payloads.PSObject.Properties.Name -contains $name) {
foreach ($p in $old.payloads.$name.files.PSObject.Properties) { $oldFiles[$p.Name] = $p.Value.sha256 }
}
$newFiles = $new.payloads[$name].files
foreach ($rel in ($newFiles.Keys | Sort-Object)) {
if (-not $oldFiles.ContainsKey($rel)) { Say " + $name/$rel" 'Green'; $changes++ }
elseif ($oldFiles[$rel] -ne $newFiles[$rel].sha256) { Say " ~ $name/$rel (content changed)" 'Yellow'; $changes++ }
}
foreach ($rel in ($oldFiles.Keys | Sort-Object)) {
if (-not $newFiles.ContainsKey($rel)) { Say " - $name/$rel" 'Red'; $changes++ }
}
}
foreach ($p in $old.payloads.PSObject.Properties.Name) {
if (-not $new.payloads.Contains($p)) { Say " - $p/ (whole payload gone)" 'Red'; $changes++ }
}
if ($changes -eq 0) {
Say ' none - the bundle already matches the lock' 'Green'
exit 0
}
Say ''
Say (" {0} change(s)." -f $changes) 'White'
}
if (-not $Yes) {
Say ''
Say ' Nothing written. Read the list above, then re-run with -Yes.' 'Yellow'
Say ' Commit the resulting bundle-lock.json - that commit IS the review.' 'Yellow'
exit 2
}
# ConvertTo-Json defaults to a depth of 2, which silently flattens the per-file
# entries into "System.Collections.Hashtable" strings and produces a lock that
# verifies against nothing.
$new | ConvertTo-Json -Depth 8 | Set-Content -Path $LockPath -Encoding UTF8
Say ''
Say " Written: $LockPath" 'Green'
Say ' Commit it.' 'Green'

1045
deploy/windows/installer/shopdb-admin.ps1
View File

File diff suppressed because it is too large Load Diff

2805
deploy/windows/installer/shopdb-install.ps1
View File

File diff suppressed because it is too large Load Diff

562
deploy/windows/installer/shopdb-preflight.ps1
View File

@@ -1,562 +0,0 @@
<#
.SYNOPSIS
ShopDB-Flask installer - Stage 1: read-only preflight.
.DESCRIPTION
Discovers everything the installer needs to know about this box and reports
it. Makes NO changes: no installs, no config edits, no service restarts.
Safe to run on a production server.
Written for stock Windows PowerShell 5.1 (Windows Server ships it). No
pwsh-only syntax, no external modules, no network access.
.PARAMETER SitePort
The port the ShopDB site will listen on. Default 8090 (the runbook's example;
the classic ASP site keeps 8080).
.PARAMETER AppRoot
Intended install directory. Default C:\shopdb-flask.
.PARAMETER Json
Emit machine-readable JSON instead of the human report. Later installer
stages consume this.
.EXAMPLE
powershell -ExecutionPolicy Bypass -File .\shopdb-preflight.ps1
powershell -ExecutionPolicy Bypass -File .\shopdb-preflight.ps1 -Json > preflight.json
.NOTES
Exit 0 = no blocking problems. Exit 1 = at least one FAIL.
#>
[CmdletBinding()]
param(
[int] $SitePort = 8090,
[string] $AppRoot = 'C:\shopdb-flask',
# Needed so the port check can tell OUR site apart from a stranger's.
[string] $SiteName = 'shopdb-flask',
[switch] $Json,
# Machine-readable output for a GUI caller: one record per line,
# STATUS|AREA|CHECK|DETAIL|FIX
# The console rendering below aligns columns with padding spaces, which only
# works in a fixed-width font at console width. A GUI must do its own layout,
# so give it DATA and let it decide - do not make it parse a formatted table.
# (-Json exists too, but Inno's Pascal Script has no JSON parser.)
[switch] $Delimited
)
Set-StrictMode -Version 2.0
$ErrorActionPreference = 'Stop'
# --- result collection -------------------------------------------------------
# Every check appends one record. Status is PASS / WARN / FAIL / INFO.
$script:Results = New-Object System.Collections.ArrayList
$script:IisPresent = $false
function Add-Result {
param(
[string] $Area,
[string] $Check,
[ValidateSet('PASS','WARN','FAIL','INFO','SKIP')] [string] $Status,
[string] $Detail,
[string] $Fix = ''
)
$null = $script:Results.Add([PSCustomObject]@{
Area = $Area
Check = $Check
Status = $Status
Detail = $Detail
Fix = $Fix
})
}
# Wrap a check so one failure cannot abort the whole run. On an unfamiliar box
# an unexpected exception is itself a finding, not a crash.
function Invoke-Check {
param([string] $Area, [string] $Check, [scriptblock] $Body)
try { & $Body }
catch {
Add-Result $Area $Check 'WARN' "check could not run: $($_.Exception.Message)" `
'Report this output; the installer needs to handle this box shape.'
}
}
# =============================================================================
# 1. Operator context
# =============================================================================
Invoke-Check 'System' 'Elevation' {
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
$adm = (New-Object Security.Principal.WindowsPrincipal($id)).IsInRole(
[Security.Principal.WindowsBuiltInRole]::Administrator)
if ($adm) { Add-Result 'System' 'Elevation' 'PASS' 'running as Administrator' }
else {
Add-Result 'System' 'Elevation' 'FAIL' 'not elevated' `
'Re-run PowerShell as Administrator. IIS and service changes require it.'
}
}
Invoke-Check 'System' 'Windows version' {
$os = Get-CimInstance Win32_OperatingSystem
$name = $os.Caption
$ver = $os.Version
# ProductType: 1 = workstation, 2 = domain controller, 3 = server
$isServer = ($os.ProductType -ne 1)
$detail = "$name (build $ver), $(if ($isServer) {'Server'} else {'Client'})"
$supported = $false
if ($isServer -and [version]$ver -ge [version]'10.0.17763') { $supported = $true } # 2019+
if (-not $isServer -and [version]$ver -ge [version]'10.0.19045') { $supported = $true } # Win10 22H2+
if ($supported) { Add-Result 'System' 'Windows version' 'PASS' $detail }
else {
Add-Result 'System' 'Windows version' 'FAIL' $detail `
'Supported: Windows Server 2019/2022+, or Windows 10 22H2 / 11 Pro+.'
}
# Client SKUs must be Pro/Enterprise/Education for IIS.
if (-not $isServer -and $name -match 'Home') {
Add-Result 'System' 'Windows edition' 'FAIL' 'Windows Home edition' `
'IIS is not available on Home editions. Pro or higher is required.'
}
}
Invoke-Check 'System' 'Architecture' {
if ([Environment]::Is64BitOperatingSystem) {
Add-Result 'System' 'Architecture' 'PASS' '64-bit'
} else {
Add-Result 'System' 'Architecture' 'FAIL' '32-bit' `
'The bundled Python and wheels are 64-bit (win_amd64) only.'
}
}
Invoke-Check 'System' 'PowerShell version' {
$v = $PSVersionTable.PSVersion
Add-Result 'System' 'PowerShell version' 'INFO' "$v"
if ($v.Major -lt 5) {
Add-Result 'System' 'PowerShell version' 'FAIL' "$v" `
'PowerShell 5.1 or later is required.'
}
}
# =============================================================================
# 2. Disk and ports
# =============================================================================
Invoke-Check 'Disk' 'Free space' {
$drive = (Split-Path -Qualifier $AppRoot)
$d = Get-CimInstance Win32_LogicalDisk -Filter "DeviceID='$drive'"
if ($null -eq $d) {
Add-Result 'Disk' 'Free space' 'FAIL' "drive $drive not found" `
"Choose an -AppRoot on an existing volume."
return
}
$freeGB = [math]::Round($d.FreeSpace / 1GB, 1)
$detail = "$freeGB GB free on $drive"
if ($freeGB -ge 5) { Add-Result 'Disk' 'Free space' 'PASS' $detail }
else { Add-Result 'Disk' 'Free space' 'FAIL' $detail 'At least 5 GB is required.' }
}
Invoke-Check 'Disk' 'AppRoot' {
if (Test-Path $AppRoot) {
$existing = @(Get-ChildItem $AppRoot -Force -ErrorAction SilentlyContinue)
if ($existing.Count -gt 0) {
$hasEnv = Test-Path (Join-Path $AppRoot '.env')
if ($hasEnv) {
# An existing install is the NORMAL state for an upgrade. Reporting
# it as a warning makes a routine update look like a problem.
$ver = ''
$vf = Join-Path $AppRoot '.installed-version'
if (Test-Path $vf) { $ver = ' version ' + (Get-Content $vf -TotalCount 1).Trim() }
Add-Result 'Disk' 'AppRoot' 'INFO' `
"existing ShopDB-Flask install found$ver - it will be upgraded in place, and your settings and database are kept"
} else {
Add-Result 'Disk' 'AppRoot' 'WARN' "$AppRoot exists and is not empty" `
'Confirm this directory is safe to install into.'
}
} else { Add-Result 'Disk' 'AppRoot' 'PASS' "$AppRoot exists and is empty" }
} else { Add-Result 'Disk' 'AppRoot' 'PASS' "$AppRoot does not exist yet" }
}
function Test-PortFree {
param([int] $Port)
# Get-NetTCPConnection is the reliable listener check on Server 2012R2+.
try {
$listening = @(Get-NetTCPConnection -State Listen -LocalPort $Port -ErrorAction SilentlyContinue)
return ($listening.Count -eq 0)
} catch {
# Fall back to a bind attempt if the cmdlet is unavailable.
try {
$l = New-Object System.Net.Sockets.TcpListener([System.Net.IPAddress]::Any, $Port)
$l.Start(); $l.Stop(); return $true
} catch { return $false }
}
}
Invoke-Check 'Network' 'Site port' {
if (Test-PortFree $SitePort) {
Add-Result 'Network' 'Site port' 'PASS' "TCP $SitePort is free"
} else {
$owner = ''
try {
$c = Get-NetTCPConnection -State Listen -LocalPort $SitePort -ErrorAction SilentlyContinue | Select-Object -First 1
if ($c) { $owner = " (pid $($c.OwningProcess): $((Get-Process -Id $c.OwningProcess -ErrorAction SilentlyContinue).ProcessName))" }
} catch { }
# Is the listener OUR OWN site? On a reinstall or upgrade the port is held
# by the very application being upgraded, and blocking on that makes the
# installer refuse to update anything it previously installed.
$ours = $false
try {
Import-Module WebAdministration -ErrorAction SilentlyContinue
$site = Get-Website -Name $SiteName -ErrorAction SilentlyContinue
if ($site) {
foreach ($b in $site.bindings.Collection) {
# ${} is required: "$SitePort:" parses as a DRIVE-qualified variable.
if ($b.bindingInformation -match ":${SitePort}:") { $ours = $true }
}
}
} catch { }
if ($ours) {
Add-Result 'Network' 'Site port' 'INFO' `
"TCP $SitePort is used by the existing $SiteName site - this will be upgraded in place"
} else {
# WARN, not FAIL. This check runs before the operator has reached the
# Address page, so it is testing the DEFAULT port, not necessarily
# the one they intend to use. Blocking here would refuse an install
# over a conflict the very next page lets them resolve.
Add-Result 'Network' 'Site port' 'WARN' "TCP $SitePort is in use$owner" `
"Pick a different port on the Address page later in this wizard, or stop whatever is holding it."
}
}
}
# =============================================================================
# 3. IIS
# =============================================================================
Invoke-Check 'IIS' 'Installed' {
$svc = Get-Service -Name W3SVC -ErrorAction SilentlyContinue
$script:IisPresent = ($null -ne $svc)
if ($null -eq $svc) {
Add-Result 'IIS' 'Installed' 'FAIL' 'W3SVC service not found' `
'Install IIS. Server: Install-WindowsFeature Web-Server -IncludeManagementTools. Client: enable Internet Information Services in Windows Features.'
return
}
Add-Result 'IIS' 'Installed' 'PASS' "W3SVC present, status $($svc.Status)"
if ($svc.Status -ne 'Running') {
Add-Result 'IIS' 'Running' 'WARN' "W3SVC is $($svc.Status)" 'Start-Service W3SVC'
}
}
Invoke-Check 'IIS' 'WebAdministration module' {
$m = Get-Module -ListAvailable -Name WebAdministration
if ($m) { Add-Result 'IIS' 'WebAdministration module' 'PASS' 'available' }
else {
Add-Result 'IIS' 'WebAdministration module' 'FAIL' 'not available' `
'Install the IIS management tools (Web-Mgmt-Console / IIS Management Scripts and Tools).'
}
}
Invoke-Check 'IIS' 'HttpPlatformHandler' {
# The handler registers itself as a global module. Check the module list.
$appcmd = Join-Path $env:windir 'system32\inetsrv\appcmd.exe'
if (-not (Test-Path $appcmd)) {
Add-Result 'IIS' 'HttpPlatformHandler' 'SKIP' 'appcmd.exe not present (IIS not installed)' `
'Re-run this preflight after installing IIS.'
return
}
$modules = & $appcmd list module 2>$null
if ($modules -match 'httpPlatformHandler') {
Add-Result 'IIS' 'HttpPlatformHandler' 'PASS' 'installed'
} else {
# NOT a blocker: the MSI is in the bundle and stage 4 installs it. This
# was a FAIL, which - once the preflight page started blocking on any
# failure - stopped the wizard dead over something the installer was
# about to do by itself, with no way forward but to go and install it by
# hand. Nothing the installer SUPPLIES may be a blocker.
Add-Result 'IIS' 'HttpPlatformHandler' 'INFO' 'not installed yet' `
'The installer installs it from the bundle. No action needed.'
}
}
Invoke-Check 'IIS' 'Locked config sections' {
# The authoritative source is applicationHost.config. `appcmd list config
# /section:X` prints the section CONTENTS, not its lock state, so grepping
# that output silently reports every section as unlocked.
$cfg = Join-Path $env:windir 'system32\inetsrv\config\applicationHost.config'
if (-not (Test-Path $cfg)) {
Add-Result 'IIS' 'Locked config sections' 'SKIP' 'applicationHost.config not found'
return
}
foreach ($name in @('handlers','httpPlatform')) {
$line = Select-String -Path $cfg -Pattern ('<section name="' + $name + '"') |
Select-Object -First 1
if ($null -eq $line) {
# httpPlatform is registered by the HttpPlatformHandler MSI. Before
# that, `appcmd unlock config /section:system.webServer/httpPlatform`
# fails with "Unknown config section".
Add-Result 'IIS' "Section $name" 'SKIP' 'section not registered yet' `
'Install HttpPlatformHandler FIRST; only then can this section be unlocked.'
} elseif ($line.Line -match 'overrideModeDefault="Deny"') {
Add-Result 'IIS' "Section $name" 'WARN' 'locked (overrideModeDefault="Deny")' `
"Installer must run: appcmd unlock config /section:system.webServer/$name (else IIS 500.19)"
} else {
Add-Result 'IIS' "Section $name" 'PASS' 'not locked'
}
}
}
Invoke-Check 'IIS' 'URL Rewrite module' {
$appcmd = Join-Path $env:windir 'system32\inetsrv\appcmd.exe'
if (-not (Test-Path $appcmd)) {
Add-Result 'IIS' 'URL Rewrite module' 'SKIP' 'IIS not installed; cannot check'
return
}
$modules = & $appcmd list module 2>$null
if ($modules -match 'RewriteModule') {
Add-Result 'IIS' 'URL Rewrite module' 'PASS' 'installed (the X-Forwarded-For rule can be enabled)'
} else {
# Not a FAIL: it is only needed for -ClientIpSource direct, and the
# installer carries the MSI and installs it itself. Worth reporting
# because without the rule IIS sends no X-Forwarded-For at all, so every
# client reads as 127.0.0.1 and the GE-Enforce IP allowlist, the
# visitor-location lookup and per-host login rate limiting go quiet.
Add-Result 'IIS' 'URL Rewrite module' 'INFO' 'not installed' `
'The installer installs it from the bundle when -ClientIpSource is direct. Behind a reverse proxy that already sets X-Forwarded-For, use -ClientIpSource proxy and leave it out.'
}
}
Invoke-Check 'IIS' 'Existing sites' {
if (-not $script:IisPresent) {
Add-Result 'IIS' 'Existing sites' 'SKIP' 'IIS not installed'
return
}
try {
Import-Module WebAdministration -ErrorAction Stop
$sites = @(Get-Website)
if ($sites.Count -eq 0) { Add-Result 'IIS' 'Existing sites' 'INFO' 'none' ; return }
$desc = ($sites | ForEach-Object {
$b = ($_.bindings.Collection | ForEach-Object { $_.bindingInformation }) -join ','
"$($_.Name) [$($_.State)] $b"
}) -join '; '
Add-Result 'IIS' 'Existing sites' 'INFO' $desc
# Adoption sites typically run the classic ASP shopdb here already.
if ($desc -match '8080') {
Add-Result 'IIS' 'Classic ASP site' 'INFO' 'a site is bound on 8080 (likely the classic ASP shopdb)' `
'Install ShopDB as a separate site on its own port; do not disturb this one.'
}
} catch {
Add-Result 'IIS' 'Existing sites' 'WARN' "could not enumerate: $($_.Exception.Message)" ''
}
}
# =============================================================================
# 4. MySQL (detect BEFORE offering bundled vs existing)
# =============================================================================
Invoke-Check 'MySQL' 'Service' {
$svcs = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^MySQL' -or $_.DisplayName -match 'MySQL' })
if ($svcs.Count -eq 0) {
Add-Result 'MySQL' 'Service' 'INFO' 'no MySQL service found' `
'Bundled MySQL 8.4 LTS is the appropriate choice on this box.'
return
}
foreach ($s in $svcs) {
Add-Result 'MySQL' 'Service' 'WARN' "$($s.Name) ($($s.DisplayName)) is $($s.Status)" `
'MySQL already present. Default to the EXISTING-server option; installing bundled MySQL will collide on port 3306.'
}
}
Invoke-Check 'MySQL' 'Port 3306' {
if (Test-PortFree 3306) {
Add-Result 'MySQL' 'Port 3306' 'INFO' 'nothing listening on 3306'
} else {
Add-Result 'MySQL' 'Port 3306' 'WARN' 'something is listening on 3306' `
'Bundled MySQL cannot use the default port. Use the existing server, or pick another port.'
}
}
Invoke-Check 'MySQL' 'Backup client' {
# mysqldump is what takes the mandatory pre-upgrade backup. Without it every
# upgrade skips the backup - and skips it AFTER the application pool has been
# stopped and the tree replaced, so the site is down and there is nothing to
# restore from. A site whose database is on another server typically has no
# MySQL client installed here at all, which is exactly the case that needs it.
$names = @('mysqldump.exe')
$found = ''
foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'),
(Join-Path $AppRoot 'mysqlclient'),
'C:\MySQL84', 'C:\Program Files\MySQL', 'C:\mysql56\bin',
'C:\Program Files (x86)\MySQL')) {
if (-not (Test-Path $root)) { continue }
$hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue |
Select-Object -First 1
if ($hit) { $found = $hit.FullName; break }
}
if ($found) {
Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)"
} else {
Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' `
'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. It is not on this server yet - the installer stages its own copy, so this normally resolves itself during installation.'
}
}
Invoke-Check 'MySQL' 'Version and config' {
# Find mysqld.exe via the service binary path; read the version and locate my.ini.
$svc = Get-CimInstance Win32_Service -ErrorAction SilentlyContinue |
Where-Object { $_.PathName -match 'mysqld' } | Select-Object -First 1
if ($null -eq $svc) { return }
$path = $svc.PathName
$exe = ''
if ($path -match '"([^"]+mysqld[^"]*)"') { $exe = $matches[1] }
elseif ($path -match '(\S+mysqld\S*)') { $exe = $matches[1] }
$ver = ''
if ($exe -and (Test-Path $exe)) {
try { $ver = (& $exe --version 2>$null | Out-String).Trim() } catch { }
}
if ($ver) { Add-Result 'MySQL' 'Version' 'INFO' $ver }
# my.ini path is passed as --defaults-file in the service command line.
$ini = ''
if ($path -match '--defaults-file="?([^"]+\.ini)"?') { $ini = $matches[1] }
if ($ini -and (Test-Path $ini)) {
Add-Result 'MySQL' 'Config file' 'INFO' $ini
# MySQL 5.6 needs three flags or `flask db upgrade` dies with error 1071.
$is56 = ($ver -match '\b5\.6\.')
if ($is56) {
$content = Get-Content $ini -Raw
$need = @('innodb_file_per_table','innodb_file_format','innodb_large_prefix')
$missing = @()
foreach ($k in $need) { if ($content -notmatch $k) { $missing += $k } }
if ($missing.Count -eq 0) {
# Present in the FILE is not the same as ACTIVE. MySQL must be
# restarted for them to take effect, and the app's own
# `flask db-utils preflight` queries the live server - trust that.
Add-Result 'MySQL' '5.6 index flags' 'WARN' 'all three present in my.ini' `
'Present in the file only. They take effect after a MySQL RESTART, which interrupts the classic ASP app. Confirm with SHOW VARIABLES or flask db-utils preflight.'
} else {
# WARN, not FAIL. This inspects the LOCAL MySQL, which may not be the
# database the operator is about to install against - a bundled 8.4,
# or a remote server. Blocking the wizard here refused an install
# over a server that had nothing to do with it. Stage 3 runs
# 'flask db-utils preflight' against the database actually chosen,
# which is the check that can genuinely block.
Add-Result 'MySQL' '5.6 index flags' 'WARN' ("missing: " + ($missing -join ', ')) `
"Add to [mysqld] in $ini and restart MySQL, or 'flask db upgrade' fails with error 1071. NOTE: restarting interrupts the classic ASP app."
}
}
}
}
# =============================================================================
# 5. Python (detect, but the installer uses its OWN bundled interpreter)
# =============================================================================
Invoke-Check 'Python' 'On PATH' {
$cmd = Get-Command python -ErrorAction SilentlyContinue
if ($null -eq $cmd) {
Add-Result 'Python' 'On PATH' 'INFO' 'no python on PATH' `
'Expected. The installer supplies its own interpreter.'
return
}
$v = ''
try { $v = (& $cmd.Source --version 2>&1 | Out-String).Trim() } catch { }
$detail = "$v at $($cmd.Source)"
# A per-user install under %LOCALAPPDATA% is unreadable by the IIS app-pool
# identity. That produces a 500 with an empty HttpPlatform log.
if ($cmd.Source -like "$env:LOCALAPPDATA*") {
Add-Result 'Python' 'On PATH' 'WARN' "$detail (PER-USER install)" `
'The IIS app-pool identity cannot read %LOCALAPPDATA%. The installer must install Python for ALL USERS and use absolute paths.'
} elseif ($cmd.Source -like '*WindowsApps*') {
Add-Result 'Python' 'On PATH' 'WARN' "$detail (Microsoft Store)" `
'Store Python misbehaves under service identities. The installer will not use it.'
} else {
Add-Result 'Python' 'On PATH' 'INFO' $detail `
'Not used by the installer, but a manual `flask` command later would resolve to this interpreter.'
}
}
Invoke-Check 'Python' 'Registered installs' {
$found = @()
foreach ($hive in @('HKLM:\SOFTWARE\Python\PythonCore','HKCU:\SOFTWARE\Python\PythonCore')) {
if (Test-Path $hive) {
foreach ($k in Get-ChildItem $hive -ErrorAction SilentlyContinue) {
$ip = Join-Path $k.PSPath 'InstallPath'
if (Test-Path $ip) {
$loc = (Get-ItemProperty $ip -ErrorAction SilentlyContinue).'(default)'
$scope = if ($hive -like 'HKLM*') { 'all-users' } else { 'per-user' }
$found += "$($k.PSChildName) ($scope) $loc"
}
}
}
}
if ($found.Count -eq 0) { Add-Result 'Python' 'Registered installs' 'INFO' 'none' }
else { Add-Result 'Python' 'Registered installs' 'INFO' ($found -join '; ') }
}
# =============================================================================
# Report
# =============================================================================
$fails = @($script:Results | Where-Object { $_.Status -eq 'FAIL' })
$warns = @($script:Results | Where-Object { $_.Status -eq 'WARN' })
$skips = @($script:Results | Where-Object { $_.Status -eq 'SKIP' })
if ($Delimited) {
# Data only. No padding, no colour, no alignment - the caller lays it out.
# Pipes are stripped from field values so the record can be split naively.
foreach ($r in $script:Results) {
$fix = ''
if ($r.Fix) { $fix = $r.Fix }
$fields = @($r.Status, $r.Area, $r.Check, $r.Detail, $fix) | ForEach-Object {
([string]$_) -replace '\|', '/' -replace '\s*\r?\n\s*', ' '
}
Write-Output ($fields -join '|')
}
} elseif ($Json) {
[PSCustomObject]@{
Timestamp = (Get-Date).ToString('s')
Computer = $env:COMPUTERNAME
SitePort = $SitePort
AppRoot = $AppRoot
Failures = $fails.Count
Warnings = $warns.Count
Skipped = $skips.Count
Results = $script:Results
} | ConvertTo-Json -Depth 5
} else {
Write-Host ''
Write-Host 'ShopDB-Flask preflight' -ForegroundColor Cyan
Write-Host (" host {0} site port {1} approot {2}" -f $env:COMPUTERNAME, $SitePort, $AppRoot)
Write-Host ''
$area = ''
foreach ($r in $script:Results) {
if ($r.Area -ne $area) { $area = $r.Area; Write-Host "[$area]" -ForegroundColor White }
$colour = 'Gray'
if ($r.Status -eq 'PASS') { $colour = 'Green' }
if ($r.Status -eq 'WARN') { $colour = 'Yellow' }
if ($r.Status -eq 'FAIL') { $colour = 'Red' }
if ($r.Status -eq 'SKIP') { $colour = 'DarkGray' }
Write-Host (" {0,-5} {1,-28} {2}" -f $r.Status, $r.Check, $r.Detail) -ForegroundColor $colour
if ($r.Fix -and $r.Status -ne 'PASS' -and $r.Status -ne 'INFO' -and $r.Status -ne 'SKIP') {
Write-Host (" -> {0}" -f $r.Fix) -ForegroundColor DarkGray
}
}
Write-Host ''
if ($fails.Count -eq 0) {
Write-Host "No blocking problems. $($warns.Count) warning(s), $($skips.Count) skipped." -ForegroundColor Green
} else {
Write-Host "$($fails.Count) blocking problem(s), $($warns.Count) warning(s), $($skips.Count) skipped." -ForegroundColor Red
}
if ($skips.Count -gt 0) {
Write-Host " Skipped checks were NOT verified. Re-run once their prerequisite is installed." -ForegroundColor DarkGray
}
Write-Host ''
}
if ($fails.Count -gt 0) { exit 1 } else { exit 0 }

BIN
deploy/windows/installer/shopdb.ico
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 52 KiB

BIN
deploy/windows/installer/swatch-bad.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 822 B

BIN
deploy/windows/installer/swatch-ok.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 822 B

BIN
deploy/windows/installer/swatch-warn.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 822 B

130
deploy/windows/installer/tests/test-install-state.ps1
View File

@@ -1,130 +0,0 @@
# Tests the installer's durable install record.
#
# The record answers one question: "is this a re-run of MY install?". Getting it
# wrong is destructive in one direction - answering "first provisioning" for a
# server holding real data lets prune-schema --force loose on it - and a dead
# end in the other, which is what it was built to fix.
#
# Run directly: pwsh -File test-install-state.ps1
# pytest runs it through tests/test_installer_state.py wherever pwsh exists.
Set-StrictMode -Version 2.0
$ErrorActionPreference = 'Stop'
$Here = Split-Path -Parent $MyInvocation.MyCommand.Path
$Installer = Join-Path (Split-Path -Parent $Here) 'shopdb-install.ps1'
if (-not (Test-Path $Installer)) { throw "installer not found at $Installer" }
$Sandbox = Join-Path ([System.IO.Path]::GetTempPath()) ('shopdb-state-' + [System.Guid]::NewGuid().ToString('N'))
# Load ONLY the state block, so the rest of the installer does not run. Bounded
# by the same markers the installer uses, and it fails loudly if either moves
# rather than silently testing nothing.
$lines = Get-Content $Installer
$start = ($lines | Select-String -Pattern '^\$script:StateFile ' | Select-Object -First 1)
$end = ($lines | Select-String -Pattern '^function Invoke-Native' | Select-Object -First 1)
if (-not $start -or -not $end) { throw 'could not locate the install-record block in shopdb-install.ps1' }
$block = $lines[($start.LineNumber - 1)..($end.LineNumber - 2)] -join "`n"
if ($block -notmatch 'function Test-FirstProvisioning') { throw 'extracted block does not contain the state functions' }
# Stand-ins for the surrounding script.
$script:Created = New-Object System.Collections.ArrayList
function Write-Log { param($m, $l = 'INFO') }
function Protect-File { param([string] $Path, [switch] $Directory) }
function Test-SamePath {
param([string] $A, [string] $B)
if (-not $A) { return -not $B }
return ($A.TrimEnd('\', '/').ToLowerInvariant() -eq $B.TrimEnd('\', '/').ToLowerInvariant())
}
if (-not $env:ProgramData) { $env:ProgramData = [System.IO.Path]::GetTempPath() }
Invoke-Expression $block
$script:Failures = 0
function Check {
param([string] $Name, $Expected, $Actual)
if ($Expected -eq $Actual) {
Write-Host (" PASS " + $Name)
} else {
Write-Host (" FAIL {0}: expected [{1}], got [{2}]" -f $Name, $Expected, $Actual)
$script:Failures++
}
}
function Reset-Case {
param([switch] $Stamped)
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
New-Item -ItemType Directory -Path $Sandbox -Force | Out-Null
$script:AppRoot = $Sandbox
$script:StateFile = Join-Path $Sandbox 'install-state.json'
$script:InstallState = $null
if ($Stamped) { Set-Content (Join-Path $Sandbox '.installed-version') '0.7.0' }
}
Write-Host 'brand new server'
Reset-Case
Initialize-InstallState
Check 'first provisioning' $true (Test-FirstProvisioning)
Check 'record written' $true (Test-Path $script:StateFile)
Write-Host 'retry of a failed first install'
$script:InstallState = $null
Initialize-InstallState
Check 'still first provisioning' $true (Test-FirstProvisioning)
Write-Host 'genuine upgrade after a completed install'
Complete-InstallState
$script:InstallState = $null
Initialize-InstallState
Check 'no longer first provisioning' $false (Test-FirstProvisioning)
Write-Host 'install predating the record - must take the safe side'
Reset-Case -Stamped
Initialize-InstallState
Check 'treated as established' $false (Test-FirstProvisioning)
Write-Host 'record naming a different directory'
Reset-Case
Initialize-InstallState
$other = Get-Content $script:StateFile -Raw | ConvertFrom-Json
$other.approot = 'C:\somewhere-else'
($other | ConvertTo-Json -Depth 6) | Set-Content $script:StateFile
$script:InstallState = $null
Check 'foreign record rejected' $null (Get-InstallState)
Write-Host 'corrupt record'
Reset-Case
Set-Content $script:StateFile '{ this is not json'
$script:InstallState = $null
Check 'corrupt record rejected' $null (Get-InstallState)
Reset-Case
Set-Content $script:StateFile '{ this is not json'
Initialize-InstallState
Check 'recovers and rewrites' $true ($null -ne $script:InstallState)
Write-Host 'created items survive a crash'
Reset-Case
Initialize-InstallState
Track 'site' 'shopdb-flask'
Track 'apppool' 'shopdbflask'
Track 'site' 'shopdb-flask'
$script:InstallState = $null
Initialize-InstallState
Check 'site remembered' 'shopdb-flask' ((Get-CreatedItems 'site') -join ',')
Check 'apppool remembered' 'shopdbflask' ((Get-CreatedItems 'apppool') -join ',')
# A zero-length array returned from a function unrolls to $null, and $null.Count
# is fatal under StrictMode. This is the check that catches losing the comma.
Check 'unknown kind is an empty array' 0 ((Get-CreatedItems 'firewall')).Count
Write-Host 'no record at all'
$script:InstallState = $null
Check 'empty list, not null' 0 ((Get-CreatedItems 'site')).Count
Check 'not first provisioning' $false (Test-FirstProvisioning)
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
if ($script:Failures -gt 0) {
Write-Host ("{0} check(s) failed" -f $script:Failures)
exit 1
}
Write-Host 'all checks passed'
exit 0

171
deploy/windows/installer/verify_bundle_lock.py
View File

@@ -1,171 +0,0 @@
#!/usr/bin/env python3
"""Check a staged installer bundle against bundle-lock.json.
Prints one line per problem and exits non-zero if there are any. Exits 0 only
when the bundle's third-party payload is EXACTLY what the lock describes: no
missing file, no unexpected extra file, no changed content.
Why this exists alongside bundle-lock.ps1, which does the same job:
- bundle-lock.ps1 is canonical. It runs at INSTALL time on the target server,
where PowerShell is the only thing guaranteed to be present - Python is not
installed until stage 2, and verifying the payload after running part of it
would defeat the purpose.
- This file lets the Linux builder (build-installer.sh) do the same check
without adding pwsh as a build dependency.
The two are kept honest by tests/test_bundle_lock.py, which runs BOTH against
the same fixtures and fails if they disagree.
Usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>
"""
import hashlib
import json
import os
import re
import sys
# Must match $script:BundlePayloads in bundle-lock.ps1.
PAYLOADS = [
('wheels', True, 'Python wheels for the offline install'),
('python', True, 'the Python installer'),
('httpplatformhandler', True, 'the IIS module that launches waitress'),
('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'),
('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'),
('vcredist', False, 'the Visual C++ runtime MySQL requires'),
('mysql', False, 'MySQL, for the bundled-database option'),
]
def digest(path):
sha = hashlib.sha256()
with open(path, 'rb') as fh:
for chunk in iter(lambda: fh.read(1024 * 1024), b''):
sha.update(chunk)
return sha.hexdigest()
def payload_files(directory):
"""Every file under the directory, keyed by forward-slashed relative path."""
found = {}
if not os.path.isdir(directory):
return found
for root, _dirs, files in os.walk(directory):
for name in files:
full = os.path.join(root, name)
rel = os.path.relpath(full, directory).replace(os.sep, '/')
found[rel] = {'sha256': digest(full), 'size': os.path.getsize(full)}
return found
def normalize(name):
"""PEP 427 wheel filename form: runs of non-alphanumerics become one _."""
return re.sub(r'[^A-Za-z0-9.]+', '_', name).lower()
def requirement_pins(requirements_path):
"""Every 'name==version' pinned in a lockfile, including marked-out ones.
Markers are deliberately IGNORED. A requirement guarded by
sys_platform == 'win32' is exactly the case that must be present, because the
target is Windows and the wheelhouse is usually assembled somewhere else.
"""
pins = {}
with open(requirements_path) as fh:
for line in fh:
line = line.strip()
if not line or line.startswith('#'):
continue
match = re.match(r'^([A-Za-z0-9._-]+)==([^\s;\\]+)', line)
if match:
pins[normalize(match.group(1))] = match.group(2)
return pins
def check_wheelhouse_covers_requirements(bundle_root):
"""The lock records what IS in the wheelhouse, not what the app NEEDS.
Without this, an incomplete wheelhouse gets locked and blessed, and the
install fails on an air-gapped server. That is not hypothetical: assembling
the wheelhouse on Linux silently omits colorama, a win32-only dependency of
click, because pip evaluates environment markers against the machine doing
the downloading rather than the machine being targeted.
"""
wheels = os.path.join(bundle_root, 'wheels')
requirements = os.path.join(bundle_root, 'app', 'requirements.txt')
if not os.path.isdir(wheels) or not os.path.exists(requirements):
return []
have = os.listdir(wheels)
problems = []
for name, version in sorted(requirement_pins(requirements).items()):
prefix = '%s-%s-' % (name, version)
if not any(f.lower().startswith(prefix) for f in have):
problems.append(
'wheels/ has no wheel for %s==%s, which requirements.txt pins '
'(a marked-out dependency still installs on Windows)' % (name, version))
return problems
def verify(bundle_root, lock):
problems = []
locked = lock.get('payloads')
if not locked:
return ['bundle-lock.json has no "payloads" section']
for name, required, what in PAYLOADS:
directory = os.path.join(bundle_root, name)
present = os.path.isdir(directory)
if name not in locked:
if present:
problems.append(
'%s/ is present but is not in bundle-lock.json - regenerate the lock' % name)
elif required:
problems.append(
'%s/ is required but is in neither the bundle nor the lock' % name)
continue
if not present:
if required or locked[name].get('required'):
problems.append('%s/ is in the lock but missing from the bundle (%s)' % (name, what))
continue
expected = locked[name].get('files', {})
actual = payload_files(directory)
for rel, want in sorted(expected.items()):
got = actual.get(rel)
if got is None:
problems.append('%s/%s is in the lock but missing from the bundle' % (name, rel))
elif got['sha256'] != want['sha256']:
problems.append(
'%s/%s does NOT match the lock (expected sha256 %s..., got %s...)'
% (name, rel, want['sha256'][:12], got['sha256'][:12]))
elif int(got['size']) != int(want['size']):
# Impossible for a matching sha256, so the lock was hand-edited.
problems.append(
'%s/%s size disagrees with the lock - the lock has been edited by hand'
% (name, rel))
for rel in sorted(actual):
if rel not in expected:
problems.append(
'%s/%s is in the bundle but NOT in the lock (unexpected extra file)'
% (name, rel))
problems.extend(check_wheelhouse_covers_requirements(bundle_root))
return problems
def main():
if len(sys.argv) != 3:
sys.exit('usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>')
bundle_root, lock_path = sys.argv[1], sys.argv[2]
if not os.path.exists(lock_path):
print('no bundle-lock.json at %s' % lock_path)
return 1
with open(lock_path) as fh:
lock = json.load(fh)
problems = verify(bundle_root, lock)
for problem in problems:
print(problem)
return 1 if problems else 0
if __name__ == '__main__':
sys.exit(main())

BIN
deploy/windows/installer/wizard-image.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 151 KiB

BIN
deploy/windows/installer/wizard-image@125.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 217 KiB

BIN
deploy/windows/installer/wizard-image@250.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 869 KiB

BIN
deploy/windows/installer/wizard-small.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 9.1 KiB

BIN
deploy/windows/installer/wizard-small@125.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 12 KiB

BIN
deploy/windows/installer/wizard-small@250.bmp
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 56 KiB

422
deploy/windows/shopdb-diagnose.py
View File

@@ -1,422 +0,0 @@
"""Collect everything needed to diagnose a ShopDB-Flask stage 5 failure.
Stage 5 is the smoke test: the installer asks IIS for the site and expects 200.
When it does not get one, the cause is always in one of four places, and this
script reads all four in one pass so the answer arrives in a single round trip:
1. What IIS actually answers, and with which status code and error page.
2. Whether the config sections httpPlatformHandler needs are unlocked.
3. Whether the app-pool identity can read the app and run its venv.
4. Whether the app itself imports and starts.
Run it ON THE SERVER, as Administrator:
C:\\Python314\\python.exe shopdb-diagnose.py
It writes shopdb-diagnose-<timestamp>.txt next to itself and prints the path.
Send that file back.
SECRETS: the report never contains them. Values from .env (database password,
SECRET_KEY, JWT_SECRET_KEY) are read first, then scrubbed out of every section
of the report before it is written, including command output and tracebacks
that might quote them.
Standard library only, so it runs on the bundled runtime or any system Python.
"""
import os
import re
import socket
import subprocess
import sys
import time
from datetime import datetime
APP_ROOT = os.environ.get('SHOPDB_APPROOT', r'C:\shopdb-flask')
ALIAS = 'shopdb'
TIMEOUT = 25
# Filled from .env, then scrubbed from the whole report.
SECRETS = []
WINDIR = os.environ.get('WINDIR', r'C:\Windows')
# Sysnative gives a 32-bit process the real 64-bit System32. Harmless on 64-bit.
APPCMD_CANDIDATES = [
os.path.join(WINDIR, 'Sysnative', 'inetsrv', 'appcmd.exe'),
os.path.join(WINDIR, 'System32', 'inetsrv', 'appcmd.exe'),
]
def find_appcmd():
for path in APPCMD_CANDIDATES:
if os.path.isfile(path):
return path
return None
class Report(object):
def __init__(self):
self.chunks = []
def head(self, title):
self.chunks.append('\n' + '=' * 72 + '\n' + title + '\n' + '=' * 72)
def line(self, text=''):
self.chunks.append(str(text))
def block(self, title, body):
self.chunks.append('\n--- %s ---' % title)
if body is None or str(body).strip() == '':
self.chunks.append('(no output)')
else:
self.chunks.append(str(body).rstrip())
def text(self):
return '\n'.join(self.chunks) + '\n'
def run(cmd, timeout=60):
"""Run a command, return combined output. Never raises."""
try:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, shell=False)
out, _ = proc.communicate(timeout=timeout)
text = out.decode('utf-8', 'replace') if out else ''
return '[exit %s]\n%s' % (proc.returncode, text)
except subprocess.TimeoutExpired:
try:
proc.kill()
except Exception:
pass
return '[TIMED OUT after %ss]' % timeout
except Exception as exc:
return '[could not run: %s]' % exc
def powershell(script, timeout=90):
exe = os.path.join(WINDIR, 'Sysnative', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
if not os.path.isfile(exe):
exe = 'powershell.exe'
return run([exe, '-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', script],
timeout=timeout)
def load_secrets():
"""Read .env so its secret VALUES can be scrubbed from the report."""
env_path = os.path.join(APP_ROOT, '.env')
found = {}
if not os.path.isfile(env_path):
return found, None
try:
with open(env_path, 'r', encoding='utf-8', errors='replace') as handle:
raw = handle.read()
except Exception as exc:
return found, '[could not read .env: %s]' % exc
for line in raw.splitlines():
line = line.strip()
if not line or line.startswith('#') or '=' not in line:
continue
key, value = line.split('=', 1)
key, value = key.strip(), value.strip().strip('"').strip("'")
found[key] = value
if not value:
continue
upper = key.upper()
if 'SECRET' in upper or 'PASSWORD' in upper or 'TOKEN' in upper or 'KEY' in upper:
SECRETS.append(value)
if upper == 'DATABASE_URL':
# mysql+pymysql://user:PASSWORD@host/db -- the password only.
match = re.match(r'^[^:]+://([^:@/]+):([^@]+)@', value)
if match:
SECRETS.append(match.group(2))
return found, raw
def scrub(text):
"""Remove every known secret value from the report."""
for secret in SECRETS:
if secret and len(secret) >= 4:
text = text.replace(secret, '<REDACTED>')
# Catch a password inside any URL that did not come from .env.
text = re.sub(r'(://[^:@/\s]+:)[^@\s]+(@)', r'\1<REDACTED>\2', text)
return text
def http_probe(url):
"""Fetch a URL, returning status, headers and body even for an error page."""
import urllib.error
import urllib.request
started = time.time()
try:
request = urllib.request.Request(url, headers={'User-Agent': 'shopdb-diagnose'})
with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
body = response.read(4000).decode('utf-8', 'replace')
return {'status': response.status, 'reason': response.reason,
'headers': dict(response.headers), 'body': body,
'seconds': time.time() - started}
except urllib.error.HTTPError as exc:
body = ''
try:
body = exc.read(4000).decode('utf-8', 'replace')
except Exception:
pass
return {'status': exc.code, 'reason': exc.reason,
'headers': dict(exc.headers or {}), 'body': body,
'seconds': time.time() - started}
except Exception as exc:
return {'status': None, 'reason': '%s: %s' % (type(exc).__name__, exc),
'headers': {}, 'body': '', 'seconds': time.time() - started}
def summarise_iis_error(body):
"""Pull the meaningful bits out of an IIS error page."""
if not body:
return None
import html
# style and script blocks first: their contents survive plain tag stripping
# and drag CSS into the summary.
flat = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', body)
flat = re.sub(r'(?s)<!--.*?-->', ' ', flat)
flat = re.sub(r'(?s)<[^>]*>', ' ', flat)
# Any unterminated tag left by the 4000-byte body truncation.
flat = re.sub(r'(?s)<[^>]*$', ' ', flat)
flat = html.unescape(flat)
flat = re.sub(r'\s+', ' ', flat).strip()
hints = []
for pattern in (r'\b\d{3}\.\d+\b', r'0x[0-9a-fA-F]{8}',
r'Error Code[^.]{0,80}', r'Config (?:Error|File)[^.]{0,120}',
r'Requested URL[^.]{0,120}', r'Physical Path[^.]{0,120}'):
for match in re.findall(pattern, flat):
cleaned = re.sub(r'\s+', ' ', match).strip(' :-')
if cleaned and cleaned not in hints:
hints.append(cleaned)
return {'flat': flat[:1200], 'hints': hints}
def main():
report = Report()
stamp = datetime.now().strftime('%Y%m%d-%H%M%S')
env_values, env_raw = load_secrets()
report.line('ShopDB-Flask stage 5 diagnostic')
report.line('generated %s' % datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
report.line('host %s' % socket.gethostname())
report.line('app root %s' % APP_ROOT)
report.line('python %s' % sys.version.replace('\n', ' '))
report.line('process is %d-bit' % (64 if sys.maxsize > 2 ** 32 else 32))
# ---------------------------------------------------------------- 1. HTTP
report.head('1. WHAT IIS ANSWERS')
report.line('This is the single most important section. The status code names')
report.line('the fault: 500.19 = config locked, 502.3/503 = the app did not')
report.line('start, 404 = application or handler mapping missing.')
hostname = socket.gethostname()
targets = [
'http://localhost/%s/' % ALIAS,
'http://127.0.0.1/%s/' % ALIAS,
'http://[::1]/%s/' % ALIAS,
'http://%s/%s/' % (hostname, ALIAS),
'http://localhost/',
]
for url in targets:
result = http_probe(url)
report.line('\n%s' % url)
report.line(' status : %s %s' % (result['status'], result['reason']))
report.line(' time : %.1fs' % result['seconds'])
server = result['headers'].get('Server')
if server:
report.line(' server : %s' % server)
summary = summarise_iis_error(result['body'])
if summary and summary['hints']:
report.line(' hints : %s' % ' | '.join(summary['hints'][:8]))
if summary and summary['flat']:
report.line(' body : %s' % summary['flat'][:600])
# localhost resolving to ::1 first has bitten this install before.
report.block('name resolution for localhost', run(
['nslookup', 'localhost'], timeout=20))
# ------------------------------------------------------------- 2. IIS state
report.head('2. IIS STATE')
appcmd = find_appcmd()
if not appcmd:
report.line('appcmd.exe NOT FOUND - is the IIS role installed?')
else:
report.line('appcmd: %s' % appcmd)
report.block('sites', run([appcmd, 'list', 'sites']))
report.block('applications', run([appcmd, 'list', 'apps']))
report.block('app pools', run([appcmd, 'list', 'apppools']))
report.block('worker processes (empty means nothing is running)',
run([appcmd, 'list', 'wp']))
report.block('modules: httpPlatformHandler present?',
run([appcmd, 'list', 'modules']))
# overrideMode tells us whether the unlock actually took effect.
#
# allowedServerVariables is in this list because it caused a 500.52 that
# the first two sections could not explain: it is Deny by default, so an
# <allowedServerVariables> block in the app's web.config is rejected
# before httpPlatformHandler runs. Checking only the sections we unlock
# would have missed the one we do not.
for section in ('system.webServer/handlers',
'system.webServer/httpPlatform',
'system.webServer/rewrite/allowedServerVariables',
'system.webServer/rewrite/rules'):
report.block('lock state of %s' % section,
run([appcmd, 'list', 'config', '/section:%s' % section,
'/text:*']))
report.block('W3SVC / WAS services', powershell(
"Get-Service W3SVC,WAS | Format-Table Name,Status,StartType -AutoSize | Out-String"))
report.block('listeners on port 80', powershell(
"Get-NetTCPConnection -LocalPort 80 -State Listen -EA SilentlyContinue | "
"Format-Table LocalAddress,LocalPort,OwningProcess -AutoSize | Out-String"))
report.block('app pool detail', powershell(
"Import-Module WebAdministration -EA SilentlyContinue; "
"Get-Item IIS:\\AppPools\\shopdbflask -EA SilentlyContinue | "
"Select-Object name,state,managedRuntimeVersion,enable32BitAppOnWin64,"
"@{n='identity';e={$_.processModel.identityType}} | Format-List | Out-String"))
# --------------------------------------------------------- 3. app + config
report.head('3. APPLICATION AND CONFIG')
web_config = os.path.join(APP_ROOT, 'web.config')
if os.path.isfile(web_config):
try:
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
report.block('web.config', handle.read())
except Exception as exc:
report.block('web.config', '[could not read: %s]' % exc)
else:
report.block('web.config', 'MISSING at %s' % web_config)
if env_raw is None:
report.block('.env', 'MISSING at %s' % os.path.join(APP_ROOT, '.env'))
else:
# Keys and non-secret values only. Secret values are scrubbed anyway.
lines = []
for key in sorted(env_values):
upper = key.upper()
secretish = ('SECRET' in upper or 'PASSWORD' in upper
or 'TOKEN' in upper or 'KEY' in upper
or upper == 'DATABASE_URL')
if secretish:
lines.append('%s = <set, %d chars>' % (key, len(env_values[key])))
else:
lines.append('%s = %s' % (key, env_values[key]))
report.block('.env (secret values withheld)', '\n'.join(lines))
venv_python = os.path.join(APP_ROOT, 'venv', 'Scripts', 'python.exe')
report.line('\nvenv python exists: %s' % os.path.isfile(venv_python))
if os.path.isfile(venv_python):
# The exact failure stage 3 used to hit. Proves the app imports.
report.block('venv: import shopdb', run(
[venv_python, '-c',
'import shopdb; print("import OK"); '
'app = shopdb.create_app(); print("create_app OK")'], timeout=120))
report.block('venv: waitress present', run(
[venv_python, '-c', 'import waitress; print(waitress.__version__)'],
timeout=60))
# What httpPlatformHandler is told to launch, and whether it exists.
if os.path.isfile(web_config):
try:
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
raw = handle.read()
match = re.search(r'processPath\s*=\s*"([^"]+)"', raw)
args = re.search(r'arguments\s*=\s*"([^"]*)"', raw)
if match:
path = os.path.expandvars(match.group(1))
report.line('\nhttpPlatform processPath : %s' % match.group(1))
report.line(' resolved : %s' % path)
report.line(' exists : %s' % os.path.isfile(path))
if args:
report.line('httpPlatform arguments : %s' % args.group(1))
except Exception as exc:
report.line('[could not parse web.config: %s]' % exc)
# ------------------------------------------------------------- 4. app logs
report.head('4. APPLICATION LOGS')
log_dir = os.path.join(APP_ROOT, 'logs')
if not os.path.isdir(log_dir):
report.line('MISSING: %s' % log_dir)
else:
entries = []
for name in sorted(os.listdir(log_dir)):
full = os.path.join(log_dir, name)
try:
entries.append((os.path.getmtime(full), full, name,
os.path.getsize(full)))
except OSError:
pass
if not entries:
report.line('%s is EMPTY.' % log_dir)
report.line('No stdout log at all means httpPlatformHandler never')
report.line('launched python - look at the pool identity and ACLs.')
for _, full, name, size in sorted(entries, reverse=True)[:5]:
if size == 0:
report.block('%s (0 bytes)' % name,
'EMPTY - python was launched but wrote nothing.')
continue
try:
with open(full, 'r', encoding='utf-8', errors='replace') as handle:
tail = handle.readlines()[-60:]
report.block('%s (%d bytes, last 60 lines)' % (name, size),
''.join(tail))
except Exception as exc:
report.block(name, '[could not read: %s]' % exc)
# ---------------------------------------------------------------- 5. ACLs
report.head('5. PERMISSIONS')
report.line('The pool runs as "IIS AppPool\\shopdbflask". It needs RX on the')
report.line('tree, Modify on logs and instance, and Read on .env.')
for target in (APP_ROOT, log_dir, os.path.join(APP_ROOT, '.env'),
os.path.join(APP_ROOT, 'venv', 'Scripts')):
if os.path.exists(target):
report.block('icacls %s' % target, run(['icacls', target], timeout=40))
# ------------------------------------------------------------ 6. event log
report.head('6. EVENT LOG')
report.block('recent application errors', powershell(
"Get-WinEvent -FilterHashtable @{LogName='Application';"
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
"Where-Object { $_.ProviderName -match 'HttpPlatform|IIS|W3SVC|WAS|\\.NET' "
"-or $_.LevelDisplayName -eq 'Error' } | Select-Object -First 25 "
"TimeCreated,ProviderName,LevelDisplayName,Message | Format-List | Out-String",
timeout=180))
report.block('system log: WAS / W3SVC', powershell(
"Get-WinEvent -FilterHashtable @{LogName='System';"
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
"Where-Object { $_.ProviderName -match 'WAS|W3SVC|HTTP' } | "
"Select-Object -First 20 TimeCreated,ProviderName,LevelDisplayName,Message | "
"Format-List | Out-String", timeout=180))
# ------------------------------------------------------------------ write
body = scrub(report.text())
out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'shopdb-diagnose-%s.txt' % stamp)
try:
with open(out_path, 'w', encoding='utf-8') as handle:
handle.write(body)
except Exception:
out_path = os.path.join(os.environ.get('TEMP', r'C:\Windows\Temp'),
'shopdb-diagnose-%s.txt' % stamp)
with open(out_path, 'w', encoding='utf-8') as handle:
handle.write(body)
print('')
print('Report written to:')
print(' %s' % out_path)
print('')
print('%d secret value(s) were scrubbed from it.' % len(SECRETS))
print('Send that file back.')
return 0
if __name__ == '__main__':
sys.exit(main())

140
deploy/windows/web.config
View File

@@ -1,140 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
IIS site config for shopdb-flask via HttpPlatformHandler.
IIS launches waitress (a Windows-friendly WSGI server; gunicorn does NOT run
on Windows) and forwards requests to it on a private loopback port that IIS
assigns via %HTTP_PLATFORM_PORT%. One process serves both /api and the built
Vue SPA (frontend/dist), so no separate static site is needed.
Prerequisites on the box:
- HttpPlatformHandler IIS module installed
(https://www.iis.net/downloads/microsoft/httpplatformhandler)
- URL Rewrite module installed (only for the optional X-Forwarded-For rule)
- Python 3.14 + a venv at APP_ROOT\venv with requirements.txt + waitress
- Secrets live in APP_ROOT\.env (wsgi.py load_dotenv() reads it). Keep them
OUT of this file. Lock .env ACLs to the app pool identity + admins.
Replace APP_ROOT (C:\shopdb-flask below) with the real deploy path. The IIS
site's physical path MUST be APP_ROOT (where wsgi.py lives).
-->
<configuration>
<system.webServer>
<handlers>
<add name="httpplatformhandler" path="*" verb="*"
modules="httpPlatformHandler" resourceType="Unspecified" />
</handlers>
<httpPlatform
processPath="C:\shopdb-flask\venv\Scripts\waitress-serve.exe"
arguments="--port=%HTTP_PLATFORM_PORT% --host=127.0.0.1 --threads=8 --trusted-proxy=127.0.0.1 --trusted-proxy-headers=x-forwarded-for --trusted-proxy-count=1 wsgi:app"
stdoutLogEnabled="true"
stdoutLogFile="C:\shopdb-flask\logs\httpplatform"
startupTimeLimit="120"
startupRetryCount="3">
<environmentVariables>
<!-- FLASK_ENV MUST be production here or wsgi.py defaults to the dev
config (SQL echo, debug, wrong DB URL). Real secrets go in .env. -->
<environmentVariable name="FLASK_ENV" value="production" />
<environmentVariable name="PYTHONPATH" value="C:\shopdb-flask" />
<!-- Subpath method only: when this web.config sits in an IIS
Application (e.g. /ops) under an existing site instead of its own
site, tell the app its mount path. Must match the alias the
Application was created with AND the VITE_BASE_PATH the frontend
was built with ('/ops/'). Omit for the own-site method.
<environmentVariable name="MOUNT_PATH" value="/ops" />
-->
</environmentVariables>
</httpPlatform>
<!--
Forward the real client IP, so the audit log, the kiosk visitor-location
feature (IP -> business unit), the GE-Enforce IP allowlist and per-host
login rate limiting all see the caller rather than the loopback address
HttpPlatformHandler connects from.
IIS does not set X-Forwarded-For on its own. Without the rule below there
is no such header at all, and every client looks like 127.0.0.1 - so the
allowlist and the visitor-location lookup silently stop working.
ONLY CORRECT WHEN IIS IS DIRECTLY EXPOSED. It overwrites the header with
REMOTE_ADDR, which is what stops a client spoofing its own X-Forwarded-For.
Behind a reverse proxy (ARR, a load balancer) REMOTE_ADDR is the PROXY, so
this rule would destroy the real client IP - there, leave it disabled and
let the proxy set the header.
It ships DISABLED because it needs the URL Rewrite module; enabled without
it, IIS returns HTTP 500.19 ("configuration section not well-formed").
There is deliberately NO <allowedServerVariables> block below. Setting a
server variable requires that variable to be allowed, but the section
system.webServer/rewrite/allowedServerVariables ships with
overrideModeDefault="Deny", so declaring it in an application's own
web.config is refused outright: IIS answered 500.52 with error 0x80070021,
"this configuration section cannot be used at this path", BEFORE it ever
reached httpPlatformHandler - so python was never launched and the stdout
log stayed empty, which looks like an application fault and is not one.
The installer instead allows the single variable at server level, which
grants exactly HTTP_X_FORWARDED_FOR rather than unlocking the section and
letting every site on the machine declare arbitrary server variables.
The installer handles both: -ClientIpSource direct installs URL Rewrite
from the bundle, allows the variable, and enables this; -ClientIpSource
proxy leaves it alone. By hand: install URL Rewrite, run
appcmd set config /section:system.webServer/rewrite/allowedServerVariables ^
/+"[name='HTTP_X_FORWARDED_FOR']" /commit:apphost
then delete the two marker lines below.
-->
<!-- SHOPDB-CLIENTIP-BEGIN
<rewrite>
<rules>
<rule name="Set X-Forwarded-For" stopProcessing="false">
<match url=".*" />
<serverVariables>
<set name="HTTP_X_FORWARDED_FOR" value="{REMOTE_ADDR}" />
</serverVariables>
<action type="None" />
</rule>
</rules>
</rewrite>
SHOPDB-CLIENTIP-END -->
</system.webServer>
<!--
Installer downloads: serve /installers/* as IIS static files instead of
forwarding them to Flask. The handler above is path="*", so without this a
request for /installers/Foo.exe goes to waitress, which has no such route
(SPA fallback), and large binaries would stream through a Python thread.
This <location> clears the httpPlatformHandler for that one subpath and puts
the static file handler back, so IIS serves the bytes directly (kernel-mode,
range/resume, no Python thread held).
Requires a physical folder at APP_ROOT\installers (the site's physical path
is APP_ROOT). Drop the installer binaries there, e.g. robocopy them from the
classic wwwroot\installers. The stored installpath 'installers/Foo.exe' then
resolves to <mount>/installers/Foo.exe (e.g. /shopdb/installers/Foo.exe).
.exe/.msi are given an explicit MIME map; if the parent site has a Request
Filtering rule that denies executable extensions, also allow them there.
-->
<location path="installers">
<system.webServer>
<handlers>
<clear />
<add name="StaticFile" path="*" verb="*"
modules="StaticFileModule" resourceType="File"
requireAccess="Read" />
</handlers>
<staticContent>
<remove fileExtension=".exe" />
<mimeMap fileExtension=".exe" mimeType="application/octet-stream" />
<remove fileExtension=".msi" />
<mimeMap fileExtension=".msi" mimeType="application/octet-stream" />
</staticContent>
</system.webServer>
</location>
</configuration>

8
docker-compose.airgap.yml
View File

@@ -3,7 +3,7 @@
# For a site with NO internet. Nothing is built or pulled here: the images are
# built on a connected box (scripts/build-offline-bundle.ps1), shipped as a
# tarball, and `docker load`ed at the site. This file only RUNS pre-loaded
# images. See docs/DEPLOY-AIRGAP.md for the full runbook.
# images. See the DEPLOY-AIRGAP page in the project wiki for the full runbook.
#
# Differences from docker-compose.yml (the connected/build template):
# - api uses `image:` (a loaded image), never `build: .` (build needs the net).
@@ -32,9 +32,9 @@ x-app-env: &app-env
LOG_LEVEL: ${LOG_LEVEL:-INFO}
ZABBIX_URL: ${ZABBIX_URL:-}
ZABBIX_TOKEN: ${ZABBIX_TOKEN:-}
# Channel label for a distributable installer. Empty on the stable (main)
# bundle, 'DEV' on the edge (dev-be) bundle, which lights up the sidebar
# badge. Sourced from .env, which build-channel-installer.ps1 pre-fills.
# Channel label for a distributable installer. Empty on the stable bundle,
# 'DEV' on the edge bundle, which lights up the sidebar badge. Sourced from
# .env, which build-channel-installer.ps1 pre-fills.
SHOPDB_ENV_LABEL: ${SHOPDB_ENV_LABEL:-}
services:

2
docker-compose.channel.yml
View File

@@ -1,7 +1,7 @@
# Channel overlay for side-by-side stable / bleeding-edge stacks.
#
# The base docker-compose.yml is channel-agnostic. This overlay adds only the
# pieces that differ between a stable (main) and an edge (dev-be) stack so the
# pieces that differ between a stable and an edge stack so the
# two can run at once on one dev workstation without colliding:
#
# - SHOPDB_ENV_LABEL is passed into the api container. When non-empty the

112
docker-compose.prebuilt.yml Normal file
View File

@@ -0,0 +1,112 @@
# Docker Compose using pre-built frontend
# Build frontend on host first: cd frontend && npm run build
# Then: docker compose -f docker-compose.prebuilt.yml build
services:
db:
image: mysql:8.4
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
ports:
- "127.0.0.1:3306:3306"
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: shopdb_flask
MYSQL_USER: shopdb
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
volumes:
- db_data:/var/lib/mysql
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
# Schema, plugin bootstrap and seed data - runs once before API starts
migrate:
build:
context: .
dockerfile: Dockerfile.prebuilt
restart: "no"
depends_on:
db:
condition: service_healthy
environment:
- FLASK_ENV=${FLASK_ENV:-production}
- DATABASE_URL=mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask?charset=utf8mb4
- SECRET_KEY=${SECRET_KEY}
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
- CORS_ORIGINS=${CORS_ORIGINS}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
command:
- sh
- -c
- >
flask db upgrade &&
/app/scripts/bootstrap-plugins.sh &&
flask plugin upgrade-all &&
flask seed permissions &&
flask seed settings &&
flask seed reference-data
volumes:
- ./plugins:/app/plugins:ro
- ./scripts:/app/scripts:ro
- instance_data:/app/instance
api:
build:
context: .
dockerfile: Dockerfile.prebuilt
ports:
- "${API_PORT:-5001}:5001"
environment:
- DATABASE_URL=mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask
- SECRET_KEY=${SECRET_KEY}
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
- FLASK_ENV=${FLASK_ENV:-production}
- FLASK_DEBUG=${FLASK_DEBUG:-0}
- CORS_ORIGINS=${CORS_ORIGINS}
- SHOPDB_ENV_LABEL=${SHOPDB_ENV_LABEL}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- ZABBIX_URL=${ZABBIX_URL:-}
- ZABBIX_TOKEN=${ZABBIX_TOKEN:-}
- COLLECTOR_API_KEY=${COLLECTOR_API_KEY:-}
- COLLECTOR_API_KEY_ENDPOINTS=${COLLECTOR_API_KEY_ENDPOINTS:-}
volumes:
- instance_data:/app/instance
- ./plugins:/app/plugins:ro
- ./scripts:/app/scripts:ro
- ./certs:/app/certs:ro
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
restart: unless-stopped
command:
- gunicorn
- --bind
- 0.0.0.0:5001
- --worker-class
- gevent
- --workers
- "4"
- --worker-connections
- "2000"
- --timeout
- "60"
- --access-logfile
- "-"
- --access-logformat
- '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
- --error-logfile
- "-"
- --certfile
- /app/certs/server.crt
- --keyfile
- /app/certs/server.key
- wsgi:app
volumes:
db_data:
instance_data:

8
docker-compose.yml
View File

@@ -63,10 +63,10 @@ services:
ZABBIX_TOKEN: ${ZABBIX_TOKEN:-}
COLLECTOR_API_KEY: ${COLLECTOR_API_KEY:-}
COLLECTOR_API_KEY_ENDPOINTS: ${COLLECTOR_API_KEY_ENDPOINTS:-}
# This file is the source-build dev stack (wilm/dev-be); stable and edge
# ship through the pull/airgap compose files, not this one. So default the
# badge label to DEV here - a raw source build is by definition not a
# release - while still allowing an override from .env.
# This file is the source-build stack. Released channels ship through
# the pull/airgap compose files, not this one. So default the badge label
# to DEV here - a raw source build is by definition not a release - while
# still allowing an override from .env.
SHOPDB_ENV_LABEL: ${SHOPDB_ENV_LABEL:-DEV}
ports:
- "${API_PORT:-5001}:5001"

291
docs/ADOPTING-AT-ANOTHER-SITE.md Normal file
View File

@@ -0,0 +1,291 @@
<!-- docmeta
purpose: howto
audience: adopter
status: current
summary: Point the asset reporter and EventSaver at your own ShopDB, by Intune or by hand.
loadwith:
-->
# Adopting the fleet tools at another site
Two things run on shop-floor PCs and talk to a ShopDB instance: the **asset
reporter**, which posts what a PC is to the collector API, and **EventSaver**, a
screensaver that shows a slideshow served by ShopDB.
Neither is site-specific by design. What is site-specific is the URL, the
credential, and which PCs it runs on - and all three are inputs, not code.
**Getting them onto a PC is your site's job, not ShopDB's.** ShopDB serves the
data; something else lays the files down and keeps them there. This page covers
the two delivery paths that do not assume a tool you may not have: Microsoft
Intune, and doing it by hand. If you already run a configuration-enforcement
tool on the floor, the end state below is what it has to produce.
Read [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) first if you have not: it is one
page and it is what makes the rest of this make sense.
## Where these files come from
| Artifact | Where it is |
|---|---|
| `Report-AssetToShopDB.ps1` | `plugins/computers/client/`, so it versions with the collector contract it implements. It names no site: the server comes from `HKLM:\SOFTWARE\GE\ShopDB` `BaseUrl` or `-ApiUrl` |
| `Report-PrintersToShopDB.ps1` | `plugins/printers/client/`. Optional - it records what print queues a bay actually has |
| EventSaver (`EventSaver.cs`, `EventSaver.ini`) | `plugins/slides/client/`. One C# file that builds with the in-box .NET Framework compiler, so a site can rebuild it rather than trust a binary. The compiled `EventSaver.scr` is a release asset, not a file in the repository. Neither file names a site |
---
## Part 1: the asset reporter
### What it does
Runs as SYSTEM and POSTs what the PC is - hostname, BIOS serial, PC type,
logged-in user, make and model, OS, last boot, every routable NIC, the machine
number where one is configured - to
`POST https://<your-shopdb>/api/collector/computers`. The payload contract is
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md). A field your site does not collect
is simply absent; the server upserts on `hostname` and leaves the rest alone.
### What you must provide
| Input | Where it comes from |
|---|---|
| Server URL | `-ApiUrl`, or `HKLM:\SOFTWARE\GE\ShopDB` value `BaseUrl`. A script downloaded from Settings has your URL already in it |
| A credential | `HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey` for the shared key, or the PC's own credential once it has enrolled. Never stamped into the script. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| Your routable ranges | A **server** setting, `collector_corporate_ranges`. The bay reports every NIC and takes no view; the server picks which one is the PC's address |
The reporter still accepts `-AllowedRanges` and reads a `CollectorRanges`
registry value, and **ignores both**, logging a note when either is set. Choosing
the NIC on the bay was wrong - a bay cannot know a site's addressing - so it
moved to the server. The parameters remain only so an existing scheduled task
does not fail on an unknown argument. Set the server setting.
**Shortcut: download it pre-configured.** Settings > Computers > Asset reporter
generates the script with your server's URL already in the parameter defaults,
and shows its SHA-256. The credential is deliberately not included.
### The credential to start with
Mint a managed personal access token scoped to `collector.ingest` and nothing
else, so a token recovered off a shop-floor PC cannot read your asset register.
Plan to move off it onto per-PC credentials.
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole story and the only page that
describes it.
### Deploying it
The three paths below are alternatives - pick one. All of them end with the same
two facts on the PC: the script exists somewhere it can run from, and the
credential is in the registry.
#### Option A: Microsoft Intune
**A1. The key, as a platform script.** Devices > Scripts and remediations >
Platform scripts > Add > Windows 10 and later. Run as SYSTEM, do not run in the
64-bit context only.
```powershell
# Set-ShopdbCollectorKey.ps1 (Intune platform script, runs once per device)
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
exit 0
```
Putting a token in an Intune script body means anyone who can read the Intune
configuration can read the token. That is an argument for the `collector.ingest`
scope, not against Intune: the blast radius of that token is one API. Restrict
the registry key's ACL to SYSTEM and Administrators so an interactive user on the
bay cannot read it.
**A2. The reporter, as a remediation on a schedule.** Devices > Scripts and
remediations > Remediations. A remediation is the natural fit because it re-runs
on a schedule, which is exactly what reporting is.
*Detection script* - forces the remediation to run every cycle:
```powershell
# Always "not compliant": reporting is not a state to converge on, it is an
# event that should happen on every schedule tick.
exit 1
```
*Remediation script* - the reporter itself, with your URL:
```powershell
& "$PSScriptRoot\Report-AssetToShopDB.ps1" `
-ApiUrl 'https://shopdb.example.net/api/collector/computers'
exit 0
```
Assign it with a daily schedule. Run as SYSTEM, 64-bit.
**A3. Intune with DSC (Machine Configuration).** If your estate is already
governed by Azure Machine Configuration, express the *credential* as
configuration and leave the *reporting* to a scheduled task, because a report is
an event and DSC converges state:
```powershell
Configuration ShopdbCollector
{
Import-DscResource -ModuleName PSDesiredStateConfiguration
Node localhost
{
Registry CollectorKey
{
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
ValueName = 'CollectorKey'
ValueData = 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
ValueType = 'String'
Ensure = 'Present'
}
Registry CollectorUrl
{
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
ValueName = 'BaseUrl'
ValueData = 'https://shopdb.example.net'
ValueType = 'String'
Ensure = 'Present'
}
# ...and a Script resource whose SetScript registers the scheduled task
# below (option B has the three lines). Pass it no -ApiUrl: BaseUrl
# above is where the script reads the server from, and one source beats
# two that can disagree.
}
}
```
Do not add a `CollectorRanges` resource. That value is read and ignored; the
ranges are a server setting now.
#### Option B: manual
For a pilot, a single bay, or an estate with no management plane:
```powershell
# As administrator, once per PC.
New-Item -Path 'C:\ProgramData\ShopDB' -ItemType Directory -Force | Out-Null
Copy-Item .\Report-AssetToShopDB.ps1 'C:\ProgramData\ShopDB\'
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument @'
-NoProfile -ExecutionPolicy Bypass -File "C:\ProgramData\ShopDB\Report-AssetToShopDB.ps1" -ApiUrl "https://shopdb.example.net/api/collector/computers"
'@
$trigger = New-ScheduledTaskTrigger -Daily -At 7am
Register-ScheduledTask -TaskName 'ShopDB asset report' -Action $action -Trigger $trigger `
-User 'SYSTEM' -RunLevel Highest -Force
Start-ScheduledTask -TaskName 'ShopDB asset report'
```
Lock down `C:\ProgramData\ShopDB` if you put it there: a script a user can edit,
run by SYSTEM on a schedule, is a local privilege escalation. Break inheritance
and grant Administrators and SYSTEM only.
### Verifying, and what to do when it fails
Open Computers in the web UI and find the hostname: it should be there with its
serial, type and a last-reported timestamp within one schedule tick. The same
answer over the API needs a logged-in session, because the read side is
permissioned (`computers.view`) even though the ingest side is not:
```powershell
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>' `
-Headers @{ Authorization = "Bearer $jwt" }
```
The collector key does NOT work here - it grants `collector.ingest` and nothing
else, which is the point of it. The script also logs to `C:\Logs\Shopfloor`.
Every failure mode and its fix is the table in
[COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md).
---
## Part 2: EventSaver
### What you have to decide
[EVENTSAVER](EVENTSAVER.md) is what EventSaver is, how it picks slides and how to
diagnose it. Read it first; adopting it adds only three decisions:
1. **Your feed URL**, in `EventSaver.ini`:
`url=https://<your-shopdb>/api/slides/feed?surface=shopfloor`.
2. **Ship the ini with the binary.** There is no compiled-in fallback, so a
missing ini is a blank screen. Check any `.scr` you were handed rather than
built: an early build still has another site's file server baked in, which is
exactly why the fallback was removed (ADR-015).
3. **Which PCs get it.** A screensaver over a live display is a defect. Exclude
kiosks, lobby displays and metrology bays.
If your configuration tool enforces the ini by hash, your own hand edits on a
bay revert - that is the feature working. Edit the copy your tool deploys from,
and update the hash it detects on:
```powershell
(Get-FileHash .\configs\EventSaver.ini -Algorithm SHA256).Hash
```
### Deploying it
#### Option A: Intune
Package the three files as a Win32 app (`EventSaver.scr`, `EventSaver.ini`, and
the enable script) with this install command:
```powershell
# install.ps1 for the Win32 app
Copy-Item .\EventSaver.scr 'C:\Windows\System32\EventSaver.scr' -Force
Copy-Item .\EventSaver.ini 'C:\Windows\System32\EventSaver.ini' -Force
exit 0
```
Detection rule: file `C:\Windows\System32\EventSaver.scr` exists **and** matches
the expected SHA-256, so a changed binary reinstalls.
Enable it per user with a configuration profile (Settings catalog > Screen
saver) or a user-context script setting the three values under option B. With
Machine Configuration/DSC, the two files are `File` resources - but DSC runs as
SYSTEM, so the per-user setting needs the logged-on user's hive or a
default-user template, which is usually more trouble than a login script.
#### Option B: manual
```powershell
Copy-Item .\EventSaver.scr 'C:\Windows\System32\' -Force
Copy-Item .\EventSaver.ini 'C:\Windows\System32\' -Force # with YOUR url=
# Per user, in the user's own session:
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name SCRNSAVE.EXE -Value 'C:\Windows\System32\EventSaver.scr'
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveActive -Value '1'
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveTimeOut -Value '480'
```
Those three values are the whole "enable" step in every option above: a
screensaver is a per-user setting, so whatever deploys the binary still has to
reach the user hive.
### Verifying
Run `EventSaver.scr /c` on the PC - it reports which source it resolved and
where it is caching - then open that feed URL in a browser on the same PC. The
symptom table in [EVENTSAVER](EVENTSAVER.md) reads the answers.
---
## Which path to choose
| You have | Use |
|---|---|
| Intune across the estate | Option A. A remediation for the reporter, a Win32 app for EventSaver. |
| A configuration-enforcement tool on the floor | Its own idiom, producing the end state option B describes: the script on disk, the credential in the registry, a task as SYSTEM. |
| Neither, or a pilot of a few bays | Option B, then move to A once it earns its place. |
These are not exclusive, and on a real floor they usually are not. Where two
tools both converge the same PC, decide which owns which layer before deploying
both: the deciding question is which one you would look at first to answer "why
is this bay wrong".

89
docs/API-REFERENCE.md Normal file
View File

@@ -0,0 +1,89 @@
<!-- docmeta
purpose: reference
audience: integrator, developer
status: current
summary: Where the generated API spec is, what llms.txt is for, and the surface and auth orientation a spec cannot give.
loadwith: none
-->
# API reference
**The reference is generated and served by the app. This page is a pointer to
it, plus the two things a specification cannot tell you.**
There used to be a hand-written endpoint list here. It competed with the
generated spec, lost, and went stale in between - so it was cut rather than
maintained. Everything it listed is in the spec, derived from the code: every
operation with its auth, its query parameters, its response shape and
harvested examples. Counts are in [PROJECT-MAP](PROJECT-MAP.md), which is
generated; nothing of that kind belongs on this page.
## Where the generated reference is
| Reach it at | What it is |
|---|---|
| `GET /api/docs` | The reference, rendered with Redoc. The bundle is vendored in the image, so it renders with no network - which is the only reason it is usable on an air-gapped site. |
| `GET /api/docs/openapi.json` | The raw OpenAPI 3.1 spec. Also tracked as `docs/openapi.json`, so it can be read from a checkout without running anything; regenerate with `scripts/gen_openapi.py` after any API change. |
| `GET /api/docs/llms.txt` | The short version, for a machine caller. Also [llms.txt](llms.txt) in this corpus. |
| `GET /api/docs/try` | Swagger UI over the same spec, where Execute sends a real request as whoever is signed in on that browser. Off by default (`API_DOCS_TRYIT_ENABLED`) and not something a production site turns on without deciding to. |
[llms.txt](llms.txt) is not a second spec. It is the orientation a caller needs
before the spec is useful: the response envelope, how authentication is
obtained and sent, the pagination and filtering conventions, and the handful of
recipes people get wrong. Read it first, then look operations up in the spec.
## Which surface owns what
The spec lists every operation; it does not say which contract governs one.
There are three surfaces, and the third is everything else.
| Surface | Who calls it | Where its semantics live |
|---|---|---|
| **Fleet and client contracts** | Shop-floor PCs, the endpoint agent, kiosks, displays and printer installers - machines, not browsers | [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md), [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md), [COLLECTOR-AUTH](COLLECTOR-AUTH.md), [PRINTER-INSTALLER](PRINTER-INSTALLER.md), [EVENTSAVER](EVENTSAVER.md) |
| **Bulk import** | A script loading a site's data. There is no separate import surface: it drives the same endpoints the UI does, with a header that preserves the source rows' timestamps | [IMPORT-API](IMPORT-API.md) |
| **Core UI API** | The Vue frontend, and anything else holding a token | [CONTRACT-STABILITY](CONTRACT-STABILITY.md) for what is safe to depend on |
## How auth works at a glance
Four credentials, and which one an operation takes follows from its surface.
The spec states it per operation; this is the shape behind it.
- **JWT** - a login token or a Personal Access Token. The core UI API.
- **Scoped service token** - `X-API-Key` or `Authorization: Bearer`, minted and
revoked per purpose (`collector.ingest`, `collector.ingest.endpoints`). A
token can be bound to specific resources and is refused anything outside
them. This is what the fleet uses.
- **A per-PC credential** - an enrolled endpoint holds its own secret, and
revoking one endpoint costs nothing to the rest. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md)
and [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md).
- **A kiosk device token, or an address inside the kiosk's allowlist** - a
credential for a screen, not authorization for a person. It says which
cabinet is asking; the badge on the request says who is standing at it. Both
kiosks are off by default and answer 404, not 403, when they are off.
Two rules cover almost everything else:
- **Every read is behind a permission.** Reading is not free, and a site
decides what a signed-out visitor may see by granting the seeded `guest` role
the read permissions it chooses - in the same role grid it uses for staff.
- **Every mutation needs a JWT and a permission or role.** None are public.
## What is reachable with no credential
This is the inventory a firewall or a deployment reviewer asks for, and it is
the one question the spec answers badly - it is a question about the whole
surface, not about any one operation.
Do not read it off a page somebody maintains. It is enumerated, with a reason
per entry, in `tests/test_core/test_public_read_surface.py`, which fails the
build when a new public read appears without being added to the list. That file
is the answer, and it is the answer because it cannot go stale without turning
CI red.
## See also
- [CONFIG](CONFIG.md) - the environment variables and settings behind these
endpoints, including the public settings allowlist.
- [CONTRACT-STABILITY](CONTRACT-STABILITY.md) - what a sister site may depend
on, and how tightly to pin.
- [DEPLOY](DEPLOY.md) - the site-exposure surface a deploy reviewer needs.

158
docs/ASSET-COMPOSITION.md Normal file
View File

@@ -0,0 +1,158 @@
<!-- docmeta
purpose: explanation
audience: developer, adopter
status: current
summary: When several devices share one identifier, file each under a parent asset - when to, and what it buys.
loadwith:
-->
# Composition: several devices under one parent
When more than one physical device answers to a single identifier, model each
device as its own asset and file it under a parent. This page is the pattern:
when to reach for it, what you get for free, and how to make the collector do it
without writing code. The mechanism itself is not new - `partof` and its
propagation rails are the ADR-001 platform contract; the recipe is.
## The problem it solves
A machine number is supposed to identify one thing. Sometimes it does not.
At the reference site, several Telesis part markers serve one operation number:
0613, 0615 and PRTMK01 each have more than one. Their configurations differ, most
often by COM port. Treating the operation as the device collapsed them into a
single record, and the damage was quiet:
- Their backups overwrote each other, so the history read as one device
flip-flopping between configurations that were really two devices.
- No question about an individual device could be asked at all - how many there
are, which port one is on, which one failed.
- Every PC driving one contested the single `controls` link to the operation,
so whichever reported last appeared to own it.
None of that announced itself. It surfaced as duplicate backup rows, weeks
later.
## When to use it
Reach for composition when ALL of these hold:
- Several physical devices share one identifier, and that is CORRECT - not a
numbering mistake. Two PCs carrying the same machine number by accident is a
fault to fix on the PC, not a model to build. `flask relationships
check-shared-machines` tells the two apart.
- The devices are individually interesting: they fail, get replaced, carry
their own configuration or calibration.
- Something already identifies each device. One device per PC is the easy case,
because the PC names it.
Do NOT reach for it when the parent is simply a location. A room holding six
printers wants a Location, not a parent asset.
## What you get
Filing a device `partof` a parent buys behaviour already built:
- **Control propagates.** `controls` propagates through `partof` (ADR-001, and
seeded by `flask seed reference-data`). A PC controlling a device therefore
controls its parent by inheritance, so the PC does NOT need - and must not
have - a direct link to the parent. That is what stops several devices
contesting a link only one can hold.
- **Position inherits.** Map-position resolution walks `partof` first, so a
device with no coordinates of its own shows at its parent's position.
- **History separates.** Anything keyed on the asset - backup revisions,
relationships, notes, audit - is per device instead of merged.
## Making the collector do it
A PC that drives a subordinate device declares it in
`plugins/computers/pctypemap.py`:
```python
SUBORDINATE_DEVICE_MAP = {
'gea-shopfloor-partmarker': {
'assettype': 'machine', # core AssetType for the device
'typename': 'Part Marker', # type within that vocabulary
'description': 'Telesis part marker',
'suffix': 'PARTMARKER', # device asset number = <PC>-<suffix>
'partof': True, # file under the reported machine number
'label': 'collector:partmarker',
},
}
```
`partof: True` is the switch that matters. It files the device under the
operation the PC reports AND stops that PC claiming the operation directly.
Leave it False for a device that does not share an identifier - a CMM is a
subordinate device too, but no two CMMs answer to one number, so it links to
its PC and nothing more.
`label` is the relationship origin marker. Only rows carrying it are archived
by a collector push, so links made by hand are never touched. **Do not change
an existing label**: those exact strings are in production databases.
A site adds or retargets an entry without a code change, per ADR-015:
```
Setting: subordinatedevice_<pctype> category: pctypemapping
Value: {"assettype": "machine", "typename": "Marking Laser",
"suffix": "LASER", "partof": true, "label": "collector:partmarker"}
```
A malformed override falls back to the built-in default rather than failing the
collector push - a bad setting must not stop a bay reporting its inventory.
## Making a backup kind follow the device
A backup posted by a PC that drives a device belongs to the DEVICE, not the
parent. `plugins/backups/services/registry.py` resolves this through
`markerforsource`: it finds the PC by the reported `sourcehostname`, follows the
PC's active device link, and files against what it finds. It falls back to the
machine number whenever the device cannot be resolved - no hostname on the
payload, a lean build without the computers plugin, or a PC that has not
reported yet - because filing under the parent beats rejecting a backup.
Two things that matter if you add a kind:
- **`sourcehostname` is load-bearing**, not informational. Without it a backup
files against the parent and merges two devices' histories.
- **A revision chain is (asset, kind, source hostname)**, so two devices on one
parent keep separate chains even before they become separate assets.
## Finding the next one
```
flask relationships check-shared-machines
```
Read-only. Lists every machine number claimed by more than one PC and separates
the two cases by whether child assets exist:
```
0615 11 PCs, 11 child asset(s) - modelled
2026 2 PCs, NO child assets - PCONE, PCTWO
```
The first is composition working. The second is two PCs carrying one number -
fix that on the PC.
## The parent is not disposable
Once devices are filed under it, the parent is doing a real job even though it
may look empty: it is the thing that says those devices belong together.
Deactivating it (`isactive = 0`, which is what "delete" does in the UI) breaks
filing, because the resolver requires an ACTIVE parent - every subsequent
report warns and files nothing.
A hard SQL `DELETE` is worse: `backuprevisions.assetid` is `ON DELETE CASCADE`,
so it destroys history still attached to the parent.
If the parent looks wrong in a list - an operation number appearing among
machines - that is a CLASSIFICATION question, not a deletion one.
## See also
- ADR-001 - relationship types, propagation rails, position inheritance
- ADR-015 - why the device map is settings-overridable
- [COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) - the collector contract
and the part-marker case end to end

52
docs/BACKUP-KINDS.md Normal file
View File

@@ -0,0 +1,52 @@
<!-- docmeta
purpose: reference
audience: developer
status: current
summary: How to add a backup kind to the backups plugin - the BackupKind members and what a new kind inherits for free.
loadwith:
-->
# Adding a fleet config backup kind
The recipe for adding a kind to the `backups` plugin - the per-PC CONFIG
backups the collector takes off shopfloor machines (NTLARS settings,
part-marker configs). Backing up the ShopDB database itself is
[BACKUP-RESTORE.md](BACKUP-RESTORE.md); why a Backup badge reads green or red
is [BACKUP-STALENESS.md](BACKUP-STALENESS.md), and you do not need it here.
---
A kind is one class in `plugins/backups/services/registry.py`, added to
`REGISTRY` at the bottom of that file:
```python
REGISTRY = {k.key: k for k in (NtlarsKind(), PartMarkerKind())}
```
Subclass `BackupKind` and override what applies. `NtlarsKind` is the fullest
example (parseable, renderable, has an info panel); `PartMarkerKind` is the
lean one.
| Member | What it is |
|--------|------------|
| `key` | Wire value. This is what lands in `backuprevisions.backupkind` and what the fleet table shows on the badge. Keep it short and lowercase. |
| `displayname` | Human label for the UI. |
| `storagebackend` | `'shopdb'` (bytes live in the DB, deduped on a semantic hash) or `'share'` (file lives on the SMB share, deduped on a raw byte hash). |
| `assettypes` | Which core asset types this kind can attach to; `['*']` for any. |
| `emptytext` | Text when an asset has no revisions of this kind. `None` HIDES the panel, which is the right default - a kind applies to an asset TYPE, but whether a given machine ever has that backup is per-machine. A part-marker panel on all 144 machines is noise. |
| `parse(raw)` | Opaque kinds return `None`; parseable kinds return a projection dict. |
| `formats()` / `render(...)` | Download formats, for `shopdb` kinds. |
| `resolveassetid(payload)` | Map a collector payload to the asset the backup belongs to. |
| `infopanel()` / `buildinfo(...)` | Optional at-a-glance card (ADR-010). Declared by the KIND, not hardcoded in the plugin, so a successor technology ships its own card by adding a class. |
| `sharedir(...)` | Conventional UNC directory for a `share` kind. Advisory - the authoritative path is whatever the collector reported, because the PC is what actually wrote the file. |
### What you do NOT have to touch
The **Backups stopped** dashboard card needs no change. It reads
`backuprevisions` generically: newest revision per chain, whatever the kind, and
shows `backupkind` as the detail line. A new kind inherits the card with no
work. The staleness rule needs no change either - it is time-based and
kind-agnostic ([BACKUP-STALENESS.md](BACKUP-STALENESS.md)).
Making a kind file against the DEVICE rather than the parent asset it hangs
under is [ASSET-COMPOSITION.md](ASSET-COMPOSITION.md).

202
docs/BACKUP-RESTORE.md Normal file
View File

@@ -0,0 +1,202 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Back up and restore a ShopDB instance - the database plus instance/ - on Docker or external MySQL.
loadwith:
-->
# Backup and restore the ShopDB database
This is the ShopDB instance itself. For the per-PC CONFIG backups the collector
takes off shopfloor machines, see [BACKUP-KINDS.md](BACKUP-KINDS.md).
Each site owns its own data (single-tenant, ADR-004), so backups are the site's
responsibility. A complete backup is two parts, and **a database-only backup is
not a backup**: restoring the database without the matching `instance/`
directory leaves the app pointing at floor plans and logos that no longer exist.
| Item | Location | Why |
|------|----------|-----|
| Database | MySQL `shopdb_flask` | All asset, user, audit and settings data. |
| `instance/branding/` | `instance/` dir | Uploaded logos and favicon. |
| `instance/modelimages/` | `instance/` dir | Uploaded vendor-model photos. |
| `instance/` floor plans | `instance/` dir | Uploaded map blueprints. |
| `instance/plugins.json` | `instance/` dir | Which plugins this site enabled. |
| `.env` | repo root (offline, secured) | Secrets needed to bring the stack back up. Store separately from the data backup, in a secrets manager. |
## The shipped Docker backup overlay
`docker-compose.backup.yml` is an opt-in per-site overlay that automates all of
the above. Turn it on with:
```bash
docker compose -f docker-compose.yml -f docker-compose.backup.yml \
--profile backup up -d
```
It adds three cooperating services: `backup-store`, a resident restic REST
server holding every snapshot in its own volume, deliberately NOT published to
the host so backups are reachable only over the compose network; `backup-watcher`,
which actuates the "Back up now" sentinel the api drops, because the api
container cannot run docker itself; and `backup`, the same one-shot run manually
(`run --rm backup`) for sites that prefer a host scheduler.
Each run captures one recoverable set: `db.sql.gz` (a non-blocking
`mysqldump --single-transaction`), a restic snapshot of `instance/`, and
`manifest.json` recording the core and plugin Alembic revisions plus the image
tag at capture time. That manifest is what makes a restore a CHECKED operation:
`scripts/backup/restore.sh` refuses to load a set onto code whose schema is
older than the set's, rather than leaving the site half-broken with rows
referencing columns the live schema has not applied. It does NOT roll back code
- restore the code first, run `flask db upgrade` and `flask plugin upgrade-all`,
then load the data.
Retention is restic-native: `--keep-daily 5 --keep-weekly 4 --keep-monthly 6`.
`RESTIC_REPOSITORY` defaults to the bundled store and can point at S3, Azure
Blob or a mounted share instead; credentials come from the site's env or secret
store, never committed.
## Backing up by hand
### Database (Docker)
```bash
docker compose exec -T db mysqldump \
-u root -p"${MYSQL_ROOT_PASSWORD}" \
--single-transaction --routines --triggers \
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
```
`--single-transaction` gives a consistent dump without locking the tables (InnoDB).
### Database (external MySQL, no container)
```bash
mysqldump -h <host> -u <user> -p \
--single-transaction --routines --triggers \
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
```
### instance directory
```bash
tar czf instance-$(date +%F).tar.gz instance/
```
Recommended cadence: nightly database dump to offsite storage, 14-day
retention; `instance/` captured on the same schedule (and always right before an
upgrade). Verify a restore quarterly.
## Restore
Restoring replaces the current database contents. Do it into a known-empty or a
throwaway target first if you are unsure.
### Step 1: Bring up the stack (or a fresh one)
```bash
cp .env.example .env # or restore your saved .env
# ensure MYSQL_* and DATABASE_URL match the dump's database name (shopdb_flask)
docker compose up -d db
```
Wait for the `db` container to report healthy (`docker compose ps`).
### Step 2: Load the database dump
```bash
gunzip -c shopdb-2026-07-10.sql.gz | \
docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask
```
For an external MySQL:
```bash
gunzip -c shopdb-2026-07-10.sql.gz | mysql -h <host> -u <user> -p shopdb_flask
```
If the target database does not exist yet, create it as utf8mb4 first (matching
the schema charset):
```sql
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
```
### Step 3: Restore the instance directory
**On Docker, `instance/` is a named volume, not the host directory.** The api
service mounts `instance_data:/app/instance`, so extracting the tarball into
`./instance` on the host restores nothing the application can see. The data has
to go into the volume:
```bash
tar xzf instance-2026-07-10.tar.gz # unpacks ./instance on the host
docker compose up -d api # creates the container and volume
docker compose cp instance/. api:/app/instance # copy into the volume
docker compose restart api
```
Check it landed before going on:
```bash
docker compose exec api ls /app/instance
```
`plugins.json` must be there. Without it the site comes back with its plugins
disabled and database rows pointing at uploaded files that are no longer
present.
A site that has replaced the named volume with its own bind mount
(`- ./instance:/app/instance`) restores by unpacking the tarball on the host and
nothing else. Check `docker-compose.yml` rather than assuming which one you have.
For a non-docker deploy, unpack the tarball over the application directory:
`instance/` is a plain directory there.
### Step 4: Reconcile migrations
```bash
docker compose exec api flask db upgrade
docker compose exec api flask plugin upgrade-all
```
For a non-docker deploy, the same two commands without the prefix.
`flask db upgrade` is a safety net: if the dump predates the current code, this
applies only the core Alembic chain. `flask plugin upgrade-all` then applies any
newer per-plugin migrations (each bundled plugin owns its own chain, ADR-008);
without it, plugin-owned tables stay un-migrated. If the dump is at the same
version both are no-ops.
### Step 5: Verify
- Log in with a known account.
- Confirm the floor map renders (branding and map blueprints resolve from
`instance/`).
- Spot-check a few asset records and the audit log.
- `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .`
should return a `VALIDATION_ERROR`, not a 500.
## What is lost if `.env` is lost
`.env` is not in the database dump, and rebuilding it is not simply retyping it:
| Value | If it is lost |
|---|---|
| `SECRET_KEY`, `JWT_SECRET_KEY` | Every issued token and session becomes invalid. Users log in again; managed API tokens must be reissued, which means every collector client needs its key replaced. Recoverable, but it is a fleet-wide job. |
| `DATABASE_URL` password | Recoverable: reset the MySQL user's password and write the new one in. |
| `MYSQL_ROOT_PASSWORD` | Recoverable through MySQL's own reset procedure, which requires stopping the server. |
| `RESTIC_PASSWORD` | NOT recoverable. Every restic snapshot in the repository is encrypted with it; without it the backups are unreadable by anyone, including you. |
| `ZABBIX_TOKEN` and similar integration tokens | Reissue at the far end. Nothing else breaks. |
So back it up with the database, to somewhere as protected as the dump. A dump
without its `.env` restores the data and locks everyone out of it.
## See also
- [BACKUP-KINDS.md](BACKUP-KINDS.md) - the other backups: per-PC config
snapshots taken by the collector. A different subject entirely.
- [DEPLOY.md](DEPLOY.md) - first-time deploy
- [UPGRADE.md](UPGRADE.md) - upgrade procedure (back up first)
- [CONFIG.md](CONFIG.md) - environment variables and Setting keys

104
docs/BACKUP-STALENESS.md Normal file
View File

@@ -0,0 +1,104 @@
<!-- docmeta
purpose: explanation
audience: developer, operator
status: current
summary: Why the Backups-stopped card lists a chain, why it stays quiet about one, and why staleness is measured from lastseenat.
loadwith:
-->
# How the backups card decides good vs stale
The `backups` plugin's health verdict, as shown on the dashboard's **Backups
stopped** card. Read it when the card looks wrong, or before changing the rule.
Adding a new kind needs none of this - see [BACKUP-KINDS.md](BACKUP-KINDS.md).
## The trap this is built around
**A revision is only written when the config CHANGES.** Dedup compares against
the latest revision for the chain `(asset, kind, sourcehostname)`, so a machine
whose config has been stable for six months has a six-month-old newest revision
and is perfectly healthy. The question worth answering is not "when was the last
backup taken" - it is "is this still being checked".
So the timestamp the card uses is `backuprevisions.lastseenat`: the last time
the collector CONFIRMED this config, whether or not anything changed. It moves
on every successful collection; the revision does not.
It is a card of exceptions rather than a column of raw dates because a raw date
read as neglect. At the default `backups_intervalhours` of **24** the collector
only attempts once a day, so a day-old confirmation IS the healthy steady state -
and the date made the reader do arithmetic against a setting they would have to
go and look up.
## What it does now
`GET /api/backups/dashboard/stale` returns one row per chain that has gone
quiet, worst first:
| Field | Meaning |
|-------|---------|
| `assetid`, `assetnumber` | The asset the backup belongs to. |
| `backupkind` | Which kind stopped. Chains fail independently, so the kind is part of the identity, not a summary. |
| `sourcehostname` | The PC that was doing the collecting. |
| `lastseenat` | When this chain was last CONFIRMED still current (ISO). |
| `quietdays` | How long it has been quiet, which is what the card sorts on. |
`lastseenat` is reported rather than the capture time because they answer
different questions and one cannot stand for both: the first proves the check is
running, the second only says what is stored. Collapsing them was what made a
healthy machine look neglected.
## The chain is the unit, not the host
A marker bay runs both an NTLARS backup and a part-marker backup. Reporting one
verdict per host would let a healthy kind mask a dead one: the part-marker
capture could stop for a month and the host would look fine on the strength of
its NTLARS row. So the card is keyed on the CHAIN - asset, kind and source PC
together - because that is the unit that fails independently.
## Silence is not a verdict
A chain whose newest revision has never been confirmed - `lastseenat` NULL,
which means a backfilled row or one written before the column existed - is not
stale, it is UNMEASURABLE, and it is skipped rather than listed. Substituting
`collectedat` for it looked right and was wrong: on a real fleet it reported
every machine whose config had simply been stable for months as a stopped
backup, so the whole board lit up and said the site was broken when it was fine.
A chain becomes measurable the first time its PC posts, and the card says
nothing about it until then.
The card is equally silent about an asset that has NEVER been backed up. Whether
one SHOULD be is a question the asset record cannot answer - most machines carry
no NTLARS at all - and guessing would list a hundred healthy assets, which is
the noise that makes a board worth ignoring.
Known consequence: a device retired from a PC leaves its last revision behind,
so that chain stays listed until the revision is pruned. That is the honest
failure direction - it complains about a backup that genuinely is not running
any more, rather than going quiet about one.
The threshold is the backups plugin's own `backups_staledays` setting (default
**3**), read through `plugins.backups.services.staleness.staledays()` rather
than re-derived, so there is ONE definition of stale. `0` disables the card.
## Why time-based rather than per-kind
Every kind answers the same question the same way: something confirmed this
recently, or it did not. Making the rule per-kind would mean each new kind has
to define health before it can appear, for no gain. If a kind ever needs its own
window - a weekly backup that should not be judged on a 3-day rule - add a
threshold override on `BackupKind` and have the rule prefer it; the
skip-the-unmeasurable contract stays as it is.
## Where the code lives
- `plugins/backups/services/staleness.py` - `staledays()` (the shared
threshold), `latestperchain()` (newest revision per chain), `stalechains()`
(the verdict)
- `plugins/backups/api/routes.py` - `dashboard_stale`, a thin route over that
rule so it stays testable without an auth layer in the way
- `plugins/backups/plugin.py` - the `backups-stale` dashboard widget
declaration, including the `backups_staledays` setting
Adding a kind is [BACKUP-KINDS.md](BACKUP-KINDS.md); backing up the ShopDB
database itself is [BACKUP-RESTORE.md](BACKUP-RESTORE.md), a different subject.

292
docs/COLLECTOR-AUTH.md Normal file
View File

@@ -0,0 +1,292 @@
<!-- docmeta
purpose: reference
audience: adopter, integrator, developer
status: current
summary: The two ways a shop-floor PC authenticates to the collector API - its own per-PC credential, or the legacy shared key.
loadwith:
-->
# Collector authentication
**This page is the one authoritative description of how a fleet client proves
who it is.** Every other page links here rather than restating it.
There are two credentials, and which one a PC uses is a property of that PC, not
of the site:
| | Per-PC credential | Shared collector key |
|---|---|---|
| Status | **current** | **legacy - still works, do not build on it** |
| Header | `X-Endpoint-Credential` | `X-API-Key` (or `Authorization: Bearer`) |
| Identifies | this one PC | any holder of the key |
| Issued by | enrollment, once per PC | an admin, once per fleet |
| Revocable | per PC | only fleet-wide |
| Blast radius if it leaks | one PC's report | every PC's report |
A PC that has enrolled sends its credential and nothing else. A PC that has not
enrolled sends the shared key, exactly as the whole fleet used to. **Never both
on one request.**
Per-PC credentials need the `endpoints` plugin on the server and an endpoint
agent (or reporter script) that knows how to enroll. A site running older
clients is on the shared key and keeps working; that is the point of keeping
both.
> **Both halves are in this tree.** `plugins/endpoints/api/agentauth.py` resolves
> the credential and decides the shared-key fall-through,
> `plugins/endpoints/api/enrollment.py` carries enroll, approve, reject, revoke,
> allow-reenroll and the migration waves,
> `plugins/endpoints/models/credential.py` holds the three tables, and
> `shopdb/core/api/collector.py` reads the header on the generic collector
> route. The C# agent sends it
> (`agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs`) and both shipped
> PowerShell reporters fall back to the shared key on their own. The decision
> record is [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md); the wire
> contract is `plugins/endpoints/AGENT-CONTRACT.md` section 9, and the fleet
> rollout order is `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
---
## 1. Which endpoints take which
| Endpoint | Per-PC credential | Shared key |
|---|---|---|
| `POST /api/collector/computers` | yes | yes |
| `POST /api/collector/printers` | yes | yes |
| `POST /api/collector/endpoints` (and `/api/endpoints/collect`) | yes | yes |
| `POST /api/collector/<other plugin>` | only if that plugin's schema declares `acceptsagentcredential` | yes |
| `POST /api/collector/pc`, `/apps`, `/heartbeat`, `/bulk` | no | yes, see section 5 |
| `GET /api/collector/status` | no | yes |
| `GET /api/collector/_schemas` | no | no - JWT only, it is for humans |
A plugin's own unattended routes are outside this table. They authenticate with
a scoped managed service token, or with a source-IP allowlist, and they accept
neither a per-PC credential nor a collector key. That is the distinction people
get wrong: the collector credential says WHICH PC is writing a report, a service
token says WHAT an integration is allowed to do.
## 2. How the server decides
For a collector endpoint whose plugin schema declares `acceptsagentcredential`:
1. `X-Endpoint-Credential` present? Resolve it to an asset. Unknown or revoked
is `401 CREDENTIAL_INVALID`. The asset the credential resolves to is the asset
the request acts for, whatever hostname the body carries.
2. Otherwise fall through to shared-key auth (section 4).
3. If the shared key authenticated but the named PC **has enrolled**, the
request is refused `401 SHARED_KEY_REFUSED`. A PC with its own secret must
use it; a shared-key call in its name is an older agent or somebody else.
4. A payload that names a different asset than the credential is
`403 ASSET_MISMATCH`.
A PC that has NEVER enrolled has no credential row, is never refused by step 3,
and keeps reporting on the shared key indefinitely. That is what carries an
un-migrated fleet.
### Error codes
| HTTP | Code | Meaning | What the client does |
|---|---|---|---|
| 401 | `CREDENTIAL_INVALID` | unknown or revoked credential | delete the stored credential, retry once with the shared key, wake enrollment |
| 401 | `SHARED_KEY_REFUSED` | shared key sent for an enrolled PC, or the shared key is switched off | use the credential; if there is none, enroll |
| 403 | `ASSET_MISMATCH` | the credential's asset is not the one the request names | log and stop. Do not retry the same request |
| 401 | `Invalid API key` | the shared key is wrong or missing | fix the key |
| 500 | `Collector API key not configured` | server-side: no env key set and no managed token presented | a SERVER config gap, not a client problem |
`ASSET_MISMATCH` also covers a hostname that is on no computer record at all.
The code and status are the same because nothing changes for the client, but the
message says which of the two it is and names the hostname. An admin fixes it by
setting the hostname on the enrolled computer.
## 3. Per-PC credentials (current)
### Shape and storage
A credential is `sdbe_` plus 43 base64url characters (32 random bytes). The
server stores only its SHA-256 plus the first 12 characters for display; the full
secret exists exactly once, in the enrollment response.
On the PC it lives in
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
`LocalMachine`, in a folder ACL'd to SYSTEM and Administrators only. The file
records the `baseurl` it was issued for, and the credential must never be sent
anywhere else. `enroll.json` beside it holds the enrollment nonce in plain text.
### Enrolling
`POST {baseurl}/api/endpoints/enroll` with `hostname` and `serialnumber`
(required), plus an optional `deviceid`, an optional `agentversion`, and a
persisted `enrollnonce` of 32 to 128 characters from `A-Za-z0-9_-`.
The route never refuses a call for its authentication. A PC installed without the
shared key still records a request an admin can approve. The shared key only
decides whether a credential is issued on the spot.
Because it takes no auth and writes rows, it carries the two bounds an open door
needs: a per-IP fixed window of 300 calls a minute, sized for a whole shop floor
behind one address rather than for one PC, and a cap on how many PENDING rows
may exist at once (`plugin.endpoints.maxpendingenrollments`, default 500). The
cap turns away only a hostname and serial never seen before, and it drops
abandoned rows before it refuses a real PC, so neither bound can lock out a
machine already in the queue.
| Status | Body | What it means |
|---|---|---|
| 201 | `{"status":"enrolled","credential":"sdbe_...","assetid":123}` | issued. Store it and switch headers; no restart |
| 202 | `{"status":"pending","requestid":45,"reason":"serial mismatch"}` | queued for an admin. Retry on a backoff |
| 403 | `ENROLL_NONCE_MISMATCH` | the approved request belongs to a different nonce. An admin must reject and re-approve |
A credential is auto-issued only when ALL of these hold: the shared key
authenticated the call, `plugin.endpoints.autoenroll` is on, the hostname matches
exactly one computer, the serial on record matches (trimmed, case-insensitive),
and the asset has no active credential. Otherwise the request is pending with one
of: `unknown hostname`, `duplicate hostname`, `no serial on record`,
`serial mismatch`, `already enrolled`, `awaiting approval`, `nonce changed`,
`rejected`, `reenroll blocked`.
There is one request row per (hostname, serial number); repeated calls update it.
The nonce is what an approval binds to, so a second machine claiming the same
hostname and serial cannot collect the approved credential. A REJECTED row binds
to the identity instead of to the nonce: any nonce from that hostname and serial
stays rejected, so a reinstall cannot argue its way back into the queue, and
approving the request is the only way out of a rejection.
### Revoking, rotating, rolling back
Revoking sets `revokedat` on the asset's one credential row. Re-issuing updates
that same row, so an asset never carries two credentials.
By default a revoked PC **falls back to the shared key**. That is deliberate: a
revoke is also how an admin rotates a credential, and how a PC is rolled back to
an older agent that only knows the shared key. Denying by default would silence
that PC instead of rolling it back.
- `blockreenroll` on the revoke stops the PC enrolling again until an admin calls
`allow-reenroll`. That is what makes a revoke stick for a compromised PC.
- `plugin.endpoints.revokeddeniessharedkey` (default **off**) makes a revoke a
real eviction: the PC reports nothing until it enrolls again. Turn it on only
once the fleet is off the shared key, because it also closes the rollback path.
Admin API, for reference: `GET /api/endpoints/enrollment` (needs
`endpoints.fleet`), and the audited admin-role writes
`.../requests/<id>/approve`, `.../requests/<id>/reject`,
`.../assets/<assetid>/revoke`, `.../assets/<assetid>/allow-reenroll`.
### Server pins
A pinned agent accepts the TLS handshake if and only if the leaf certificate's
SubjectPublicKeyInfo SHA-256 is in its pin set; with no pins it requires a clean
Windows chain validation.
`HKLM:\SOFTWARE\GE\ShopDB` `ServerPins` (REG_MULTI_SZ, either registry view) is
authoritative whenever the value exists, **even when empty** - an empty value
means "registry authoritative, no pins". Only when it is absent from both views
do the `serverpins` entries in the agent's `appsettings.json` apply. The agent
logs which source is in effect at startup and on every change, and re-reads the
set within about a minute with no restart.
The Server Certificate settings page shows the pin; the client kit has
`Get-ShopDBServerPin.ps1` to compute one and `Set-ShopDBServerPins.ps1` to write
the set (`-None` for an empty authoritative value, `-Clear` to delete it).
## 4. The shared collector key (legacy)
### What counts as one
- A **managed API token** (`shopdb_pat_...`) scoped to ONLY `collector.ingest`,
or to `collector.ingest.<pluginname>` for one collector. This is the form to
use. It can be minted, rotated and revoked from Settings > API Tokens, and its
use shows in `lastusedat` and the audit log.
- An **environment key**: `COLLECTOR_API_KEY_<PLUGINNAME>` (per plugin,
uppercased) or `COLLECTOR_API_KEY` (shared fallback). Resolution is per-plugin
first, then shared. These stay supported as a bootstrap and legacy path.
An unscoped token, or one scoped to some other permission, is NOT a collector
credential and is rejected. `collector.ingest` gates no normal route and is
denied on every role-gated endpoint, so a scoped token that leaks can submit
collector payloads and do nothing else - not even with an admin owner, because
scoping suspends the admin bypass.
### Header only, fail-closed
The key travels in `X-API-Key` (Bearer also accepted for a managed token).
**The `?api_key=<key>` querystring form was removed on every collector
endpoint** - querystring keys leak into access logs, proxy history and browser
history. A caller still putting it in the URL gets 401.
If neither an env key nor a managed token is available server-side, every request
is refused. An unconfigured server never silently accepts unauthenticated data.
### Minting one
1. Settings > API Tokens > New Token.
2. Check **Restrict permissions**, tick only **Submit collector payloads (fleet
reporting)** (`collector.ingest`). Name it, optionally set an expiry, Create.
3. Copy the `shopdb_pat_...` secret, shown once, and deliver it to the fleet
(section 6).
If you prefer a non-admin owner, create a low-privilege user whose role holds
only `collector.ingest` and let it mint its own token; the scope ceiling then
caps anything it mints. The admin-minted route is simpler and equally contained.
### Rotating one
1. Mint the replacement.
2. Deliver it (section 6) - one value per site.
3. Watch the new token's `lastusedat` climb and the old one's go stale.
4. Revoke the old token. Revocation is immediate.
The endpoint agent's key has its own kill switch: Settings > Endpoint agent key
generates a key scoped to `collector.ingest.endpoints`, revokes any previous one
in the same request, and switches off the `.env` fallback for endpoints
(`plugin.endpoints.envkeyfallback`). Revoking it stops all endpoint-agent traffic
without touching the computers or printers collectors.
### Switching it off
`plugin.endpoints.sharedkeyallowed` (default on) gates the shared key for the
endpoints collector. Turning it off is the last step of a migration, not the
first: every PC that has not enrolled stops reporting the moment it flips.
## 5. The four legacy core routes
`POST /api/collector/pc`, `/apps`, `/heartbeat` and `/bulk` predate both the
generic collector contract and per-PC credentials. They take their target
hostname from the body and rewrite that PC's logged-in user, boot time, serial
number and installed apps, so the shared key is an **any-host write key** on
them and no per-PC credential narrows it. `plugin.endpoints.sharedkeyallowed`
does not reach them.
`collector.legacysharedkeyrefused` (default **false**, today's behaviour) closes
them to the shared key; a collector-scoped managed token still works. Every
shared-key use on these routes is logged whether the gate is on or off, so read
the "Legacy collector route used with the shared key" warnings for a while first
- whatever still calls them stops working the moment it is on. Neither shipped
reporter is affected; they post to `/api/collector/computers` and
`/api/collector/printers`.
Per ADR-006 the legacy `/pc` path is deprecated and slated for removal before
v1.0. New integrations target `POST /api/collector/<plugin>`.
## 6. Delivering the shared key to a PC
Do NOT hardcode a key in a script that lives on a file share: a literal key there
is published to everyone with share read. In order of preference:
1. **`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`**, written at imaging
time, by DSC, or by an Intune platform script. This is
where both shipped reporters look. Restrict the key's ACL to SYSTEM and
Administrators so an interactive or kiosk user cannot read it.
2. **A per-site config value** the machine account already reads -
`collectorApiKey` in `C:\Enrollment\site-config.json`. Rotation is one value
per site, no re-image and no script redeploy.
3. **A machine-local file** staged at imaging
(`C:\Enrollment\collector-api-key.txt`), ACL'd to SYSTEM. Tighter blast radius
- the key never sits on the share at all - but rotation means touching every
PC.
Never put a token in a scheduled task's `-Argument` string: task XML is
world-readable. Never put one in a manifest: manifests sync broadly.
A per-PC credential needs none of this. It is issued to the PC by enrollment and
never leaves it.

147
docs/COLLECTOR-CLIENT-SETUP.md Normal file
View File

@@ -0,0 +1,147 @@
<!-- docmeta
purpose: howto
audience: operator, adopter
status: current
summary: Get the shipped asset and printer reporters running on a fleet, prove it, and fix it when a bay returns an error.
loadwith: COLLECTOR-AUTH.md
-->
# Running the collector reporters on a fleet
Follow this to put PC reporting on a fleet that does not have it yet, or to move
an existing fleet off a classic-ASP reporter. The payload contract is
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) and the credential choice is
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); you should not need either open to work
through this page.
Adopting the whole tool set at a plant that has none of it is a wider job -
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md).
---
## 1. Use the shipped reporters
Do not write your own. Two scripts ship in this repository and version with the
contract they implement:
| Script | Reports | Posts to |
|---|---|---|
| `plugins/computers/client/Report-AssetToShopDB.ps1` | hostname, BIOS serial, pc-type, logged-in user, make/model, OS, last boot, machine number, device id, every routable NIC | `POST /api/collector/computers` |
| `plugins/printers/client/Report-PrintersToShopDB.ps1` | every real print queue on the host | `POST /api/collector/printers` |
Both run as SYSTEM, always exit 0 so "last run result" stays clean, and log to
`C:\Logs\Shopfloor\`. Neither names a site: ADR-015 makes a site name in product
code a defect, and these ship everywhere.
Each reads its configuration in this order:
1. Its own parameters (`-ApiUrl`, `-ApiKey`), which a scheduled task's arguments
can supply for testing.
2. `HKLM:\SOFTWARE\GE\ShopDB` - `BaseUrl` for the server and `CollectorKey` for
the shared key. This is the secret store every shipped reporter reads, and
whatever images or configures the bay is what provisions it.
With neither, the script logs the reason and exits without posting.
**Which credential it sends** is decided on the PC, not in configuration: if the
PC holds a per-PC credential it is sent as `X-Endpoint-Credential` and the shared
key is not sent at all; otherwise the shared key goes as `X-API-Key`. If the
server refuses the credential as invalid or revoked, the reporter retries the
same POST once with the shared key, which is what makes a revoke a rollback
rather than an outage.
**Shortcut for a single site:** Settings > Computers > Asset reporter generates
`Report-AssetToShopDB.ps1` with your server's URL already in the parameter
defaults, and shows its SHA-256. The key is deliberately not included.
## 2. Prepare the server
1. Enable the `computers` plugin (and `printers`, if you want observed queues).
2. Mint the credential the fleet will start on - a managed token scoped to
`collector.ingest`. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 4 has the
steps. An un-enrolled fleet needs this even if you intend to move to per-PC
credentials, because enrollment itself auto-issues only to a call the shared
key authenticated.
3. Set `collector_corporate_ranges` to this site's routable CIDRs. Leave it blank
and the server falls back to the NIC the bay says carries the default route,
which is a guess on any bay with two NICs.
4. Map your imaging pc-types. Unmapped values only warn, but every report warns
until you do.
## 3. Deliver the key to the PCs
`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`, ACL'd to SYSTEM and
Administrators. How it gets there depends on what you run - an imaging task,
Intune, DSC or by hand - and the options with their trade-offs are in
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 6.
Two rules regardless: never bake the key into a script that lives on a file
share, and never put it in a manifest or a scheduled task's arguments.
## 4. Schedule the reporter
Reporting is an EVENT on a schedule, not a state to converge on. Two placements,
pick one:
- **As a scheduled task**, registered at imaging time, by Intune, or by hand.
SYSTEM, RunLevel Highest, at startup plus an interval.
- **As an entry in whatever enforces configuration on your floor**, run off a
share every cycle with no local copy. Order the key-provisioning step BEFORE
it so the key is present the same cycle the first report fires.
Either way, run it ALONGSIDE whatever reports today during rollout. Do not
remove the old path until the new one is proven.
## 5. Stage the rollout
1. Leave the existing reporting in place. Provision `CollectorKey` and enable the
plugins.
2. Add the reporter on a handful of pilot bays, one per pc-type. It cannot break
enforcement: it never throws and always exits 0.
3. Verify on the server. Pilot PCs appear or update, `action` is
`created`/`updated`, and every warning is understood - map any unmapped
`pctype`, add missing OS strings to the `operatingsystems` vocabulary.
Cross-check the client log's `OK` lines against the audit log.
4. Roll it out fleet-wide. From the `common` scope, every pc-type gets it.
5. Once ShopDB is the system of record and has survived a full imaging cycle,
retire the old reporter.
6. Migrate to per-PC credentials, then switch the shared key off. That is its
own rollout and deliberately comes last: read
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) for what a PC does to enroll and what an
admin approves, and `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`
for the order the shared key has to be cleared out of the places it lives.
Both shipped reporters already send a credential when one is present and fall
back to the shared key when it is not, so a PC that has not enrolled keeps
reporting and nothing has to be redeployed to start.
## 6. Verify one PC
```powershell
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>'
```
The PC should exist with its serial, type and last-reported timestamp. The
client log is `C:\Logs\Shopfloor\report-asset-YYYYMMDD.log` (printer queues:
`report-printers-YYYYMMDD.log`); it records the POST line and the server's
response verbatim, including which credential it used.
## 7. When a bay does not report
| Symptom | HTTP | Cause / fix |
|---|---|---|
| `Invalid API key` | 401 | The `X-API-Key` header is missing or wrong. Confirm the client key matches `COLLECTOR_API_KEY_COMPUTERS` (or the shared `COLLECTOR_API_KEY`) or is the managed token you minted. Check the client actually read it. The querystring form no longer works. |
| `Collector API key not configured` | 500 | Fail-closed: no env key set server-side and the caller presented no managed token. A SERVER config gap, not a client problem. |
| `Endpoint credential is unknown or revoked` | 401 | The PC's credential was revoked, or the database was restored past its issue. The reporter falls back to the shared key on its own; let the PC enroll again to restore it. |
| `The shared key is not accepted for this endpoint` | 401 | This PC has enrolled, so it must use its own credential - or the shared key is switched off site-wide. Check that `credential.bin` is present and readable by SYSTEM. |
| `The request names a different endpoint than the credential` | 403 | The hostname in the payload is on a different asset than the credential, or on no computer record at all. The message says which. Do not retry; fix the record. |
| `No collector registered for plugin computers` | 404 | The plugin is disabled or not loaded on that instance. Enable it. |
| `hostname is required` / `No data provided` | 400 | Empty body, non-JSON body, or missing identity field. Check `-ContentType 'application/json'`. |
| `Internal error processing collector payload` | 500 | Generic by design - the server does not leak the cause. The real error is in the server log (`Collector upsert failed for computers`). |
| `queues is required; send an empty array for a host with no queues` | 400 | The printers payload omitted `queues` entirely. Absent and empty are not the same thing here. |
| A bay's observed queues all vanished | 200 | It reported `queues: []`, which legitimately clears the host. Check the client log for `posting NOTHING` (a failed enumeration correctly sends nothing) versus an empty set. If the bay really has queues, fix the enumeration on that host. |
| Observed queues show as `unknown` | 200 | The queue matched no printer asset by port address or name. Add the printer to ShopDB with its IP and re-read - matching happens at read time, so the bay does not need to report again. |
| `warnings` present but `action` is created/updated | 200 | Soft issues only; the row WAS written. Common: unmapped `pctype`, unknown `osname`, unknown app name, `pcsubtype` not stored. No action needed unless the warning matters to you. |
| The wrong IP is recorded as primary | 200 | `collector_corporate_ranges` is blank or does not cover this bay. The bay reports every NIC and takes no view; set the ranges. |
Client-side logs are under `C:\Logs\Shopfloor\`. Server-side, the Flask app log
carries upsert failures and per-plugin schema failures.

85
docs/COLLECTOR-INTEGRATION.md Normal file
View File

@@ -0,0 +1,85 @@
<!-- docmeta
purpose: explanation
audience: operator, adopter, integrator, developer
status: current
summary: What the collector is, what it guarantees, and which of the three collector pages to open next.
loadwith:
-->
# The collector
The collector is how the shop-floor fleet pushes what it knows about itself into
ShopDB. It replaced the classic-ASP `api.asp?action=updateCompleteAsset` path.
This page is the shape and the guarantees. It is short on purpose: the detail
lives in three pages below, and you almost never need more than one of them.
## Which page you want
| You are | Open |
|---|---|
| Rolling reporting out to a fleet, or a bay is returning an error | [COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md) |
| Writing or changing a client, or answering "what does this field do" | [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) |
| Deciding how a PC proves who it is, minting or revoking a credential | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
| Asking why the contract is shaped this way | [ADR-006](adr/ADR-006-collector-contract.md) |
Where all of this sits relative to EventSaver, the backup collectors and the
kiosks is [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md).
## The shape
One dynamic route, `POST /api/collector/<plugin>`, serves every enabled plugin
that returns a collector schema. A plugin declares its identity field and its
accepted fields; the dispatcher handles auth, parsing, the audit row and the
error envelope, and the plugin handles the upsert. Adding a collector is adding a
schema, not adding a route.
Two plugins use it on a normal fleet:
- `computers` - what this PC is. This is the record everything else hangs off.
- `printers` - what print queues this PC actually has.
The `endpoints` plugin adds a third, much larger payload for sites running the
endpoint agent.
## What it guarantees
**Idempotent.** The same report twice changes nothing. A PC is matched by its
identity field, so re-imaging a bay updates its record instead of duplicating it,
and existing asset relationships survive.
**Patch-style, where it says so.** On the computers payload an omitted field
leaves the stored value alone, so a report from a bay that could not read its
BIOS never blanks a serial number. The printers payload is deliberately the
opposite: the reported set REPLACES the host's queues, because "what does this
bay have right now" is not a question about history.
**Warnings, not failures, for soft problems.** An unmapped pc-type, an unknown OS
string or an unknown application name is reported back in `warnings` and the row
is still written. The alternative - refusing the report - loses the whole PC over
one unrecognised string.
**Observed is never assigned.** Nothing a PC reports becomes a statement of what
it SHOULD have. A bay that installed the wrong printer cannot make itself right
by reporting it; turning observed state into assigned state is always a person's
decision.
**The report says what the PC is, not where it is on the network.** The bay
reports every NIC and takes no view of which matters. Only the server knows the
site's addressing, so only the server picks.
**Fail-closed on authentication.** A server with no credential configured refuses
every request rather than silently accepting unauthenticated data. Credentials go
in a header, never a querystring, so they stay out of access logs and proxy
history.
## The one thing that changed recently
A PC used to authenticate with a key the whole fleet shared. It now
authenticates as ITSELF, with a per-PC credential issued by enrollment and
revocable one PC at a time. The shared key still works for PCs that have not
enrolled, and is refused for any PC that has.
If you are adopting this product today, plan on per-PC credentials and treat the
shared key as the bootstrap. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole
story.

399
docs/COLLECTOR-PAYLOADS.md Normal file
View File

@@ -0,0 +1,399 @@
<!-- docmeta
purpose: reference
audience: integrator, developer
status: current
summary: Field-by-field contract for the computers and printers collector payloads, and what the server does with each.
loadwith: COLLECTOR-AUTH.md
-->
# Collector payload contract
What a fleet client may send to `POST /api/collector/<plugin>` and what the
server does with each field. Authentication is a separate subject and lives in
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); the rationale for the contract shape is
[ADR-006](adr/ADR-006-collector-contract.md).
Source of truth is `get_collector_schema` in each plugin
(`plugins/computers/plugin.py`, `plugins/printers/plugin.py`). If you change a
payload, change it there and re-check the table here.
---
## 1. The generic endpoint
One dynamic route serves every enabled plugin that returns a collector schema.
For the PC fleet that is `POST /api/collector/computers`.
Request flow inside `generic_collect`:
1. Look up the plugin's schema. Unknown or disabled plugin -> `404 No collector
registered for plugin <plugin>`.
2. Authenticate (COLLECTOR-AUTH).
3. Parse the JSON body. Missing or non-object -> `400 No data provided`.
4. Resolve the identity field the schema names (`hostname` for both collectors
below). Missing or blank -> `400 <identityfield> is required`.
5. Idempotent upsert: the plugin's `apply_collector_payload` finds the existing
asset by identity and updates it, or inserts a new one. The same identity on a
later submission updates the same row - re-imaging a PC does not duplicate it,
and existing asset relationships are preserved.
6. Audit: every accepted submission writes an `AuditLog` row (`action`
created/updated, entity `Collector`, `details = collector:<plugin>
action=<action>`), then commits.
### Response
HTTP 200, in the standard envelope, with the collector result under `data`:
```json
{
"status": "success",
"data": {
"status": "ok",
"action": "created",
"assetid": 12345,
"identityvalue": "SHOPPC2335",
"warnings": ["unknown operating system: Windows 11 Enterprise 23H2 (build 22631)"]
},
"message": "computers collector created",
"meta": { "timestamp": "...", "requestid": "..." }
}
```
`action` is `created`, `updated` or `noop`. `warnings` is a list of soft problems
(unmapped pc-type, unknown OS, unknown app, un-stored pcsubtype) that did NOT
fail the request - the row was still written.
Errors use the same envelope with `status: "error"` and the detail under
`data.error`:
```json
{ "status": "error", "data": { "error": { "code": "UNAUTHORIZED", "message": "Invalid API key" } }, "meta": { } }
```
A plugin raises `ValueError` for controlled validation failures and its message
is returned verbatim (400). Any other exception is caught, logged server-side,
the transaction rolled back, and the caller gets a generic `500 Internal error
processing collector payload` with no internal detail - by design. Check the
server log to see what actually failed.
### Schema discovery
`GET /api/collector/_schemas` returns the schema for every enabled plugin. It is
JWT-protected, not key-protected: it is for humans and tooling introspecting what
payloads are accepted, not for the headless collectors themselves.
---
## 2. Computers payload
Identity field `hostname`. Every other field is optional and patch-style: an
omitted field leaves the existing value untouched, so a bare report never blanks
a column.
| Payload field | Type | Server behaviour (`apply_collector_payload`) |
|---|---|---|
| `hostname` (required) | string | Identity. Matches `Computer.hostname` (case-insensitive), then falls back to `Asset.assetnumber`. New asset created if no match. |
| `machinenumber` | string | Identifies the MACHINE this PC drives, never the PC. A new PC always takes `assetnumber = hostname`, and an existing PC's `assetnumber` is left alone. The number resolves a machine asset and builds a PC -> machine `controls` relationship; an unknown number warns rather than creating a machine. The placeholder `9999` and empty string link nothing. See section 3. |
| `deviceid` | string | Identifies the DEVICE hanging off this PC, read from `C:\Enrollment\asset-id.txt`. It names the asset and nothing else - it does NOT say what kind of device it is, because the pc-type already does - so the same file serves a part marker, a Keyence, a Genspect or anything declared later. SEPARATE from `machinenumber` on purpose: "which bay is this" and "which device is this" are different facts, and `machinenumber` is what a site's own bay-gated configuration keys off, so naming a device there would silently stop every bay-gated rule from matching. An unknown value warns rather than minting; a value resolving to the wrong KIND of asset is refused with the asset named. See section 4. |
| `measuringtoolid` | string | The 0.12.0 name for the same thing, read from `C:\Enrollment\measuringtool-id.txt`. Still accepted as an alias so a bay already staged with that file keeps reporting; `deviceid` wins when both arrive. Prefer `asset-id.txt` for anything new. |
| `pctype` | string | `gea-shopfloor-*` imaging type -> `Computer.computertypeid` via the configurable `pctypemap` settings. Unmapped value -> warning, not error. See section 5. |
| `pcsubtype` | string | Accepted but not stored yet -> warning. |
| `serialnumber` | string | `Asset.serialnumber`. |
| `loggedinuser` | string | `Computer.loggedinuser`. (`currentuser` is also accepted as a legacy alias.) |
| `lastboottime` | ISO-8601 datetime | `Computer.lastboottime`. Unparseable -> warning. |
| `lastcheckin` | ISO-8601 datetime | Accepted (heartbeat semantics). `lastreporteddate` is set server-side on every call regardless. |
| `interfaces` | array of `{ipaddress, macaddress, adaptername, isdefaultroute}` | EVERY connected hardware NIC with a routable IPv4, in no meaningful order. One `Communication` row per NIC. See "Which NIC is the PC's address" below. |
| `ipaddress` | string | The one-address form, kept for bays on an older collector. A payload carrying both wins on `interfaces`. |
| `vendorname` | string | `Computer.vendorid`. Vendor row auto-created if missing (free vocab). |
| `modelnumber` | string | `Computer.modelnumberid`, scoped to the vendor when known. Model row auto-created if missing. |
| `osname` | string | `Computer.osid`. Controlled vocab: looked up in `operatingsystems`, NOT auto-created. Unknown value -> warning (row still written, `osid` left unset). |
| `installedsoftware` | array of `{name, version}` | `ComputerInstalledApp` rows for applications shopdb already tracks. Unknown app name -> warning, skipped. |
| `accessprotocols` | array of strings | Remote-access protocols this PC actually exposes, by catalog name (`VNC`, `WinRM`, `RDP`). PRESENCE of the key drives the sync: reported protocols are activated, catalogued ones not reported are deactivated. Omit the key entirely to leave existing rows alone - most came from the legacy `isvnc`/`iswinrm` migration. |
`defaultprinter` and `printers` are **retired**. See section 6.
### Which NIC is the PC's address
The bay reports every NIC and takes no view. The server picks, in this order:
1. The first reported address inside a CIDR in the `collector_corporate_ranges`
setting.
2. Else the NIC the bay flagged `isdefaultroute`.
3. Else the first reported.
Step 3 is a guess, and configuring the ranges is what makes it stop being one.
Choosing on the bay was the wrong place for it: a bay cannot know a site's
addressing, so a CMM with a fast link to its controller reported the machine-side
address as the PC's and never sent the corporate one at all.
Rows are keyed on the MAC where the bay sent one, because an address moves
between adapters and a MAC does not, and fall back to the address for a bay that
reports none. Exactly one row is flagged primary, and only among the rows this
report touched. A NIC the bay did NOT report is left alone: a NIC that is down at
check-in is not evidence the record is wrong, and hand-entered detail lives on
those same rows.
## 3. PC -> machine links, and part markers
The reported `machinenumber` says which machine the PC drives. It is not the
PC's own identifier, and writing it as one is a defect this contract used to
describe: `assets.assetnumber` is uniquely indexed and the machine already holds
that value, so the insert failed and the bay got a 500 on every report, forever.
A PC takes its hostname as its asset number and keeps it.
- Link: PC -> machine as `controls`, tagged
`assetrelationships.label = 'collector:machine'`. Only tagged rows are ever
archived by a push, so a link made by hand is never touched.
- Unknown number: warns and links nothing. A machine is never invented, because
a mistyped number would create equipment nobody can account for.
- A second PC reporting a machine another PC holds is a CLAIM, not a handover.
A PC imaged for a machine carries that number before it reaches the floor, so
the PC currently running the machine keeps the link while it is still alive -
reporting within 24 hours (`MACHINE_CLAIM_QUIET_HOURS`) and still `In Use`.
The challenger is recorded as a dormant link, which also marks the claim as
already announced so it does not alert every cycle.
- Handover completes on its own once the incumbent has been quiet past that
window, or immediately when someone moves it off `In Use` - which is the
one-step way to force a swap at the moment it happens. The old link is
archived, never deleted, so "which PC ran this machine in June" stays
answerable. The retired PC's STATUS is deliberately not changed: the collector
cannot tell shelved from broken from re-imaged.
- Alerts for handover and contested claims are gated on the
`computers_machinelink_alerts` setting, which ships OFF. A site may run
several PCs on one machine number legitimately, and there the alerts fire on
correct data. Warnings ride in the collector response either way.
A PC reporting `pctype = gea-shopfloor-partmarker` is handled differently,
because several markers can serve one machine number and an operation holds any
number of them. Such a PC gets its own Part Marker machine asset
(`<HOSTNAME>-PARTMARKER`), `controls` it, and the marker is filed `partof` the
operation the PC reported. Both links are tagged `collector:partmarker`. The PC
does NOT claim the operation directly: `controls` propagates through `partof`,
so control still follows from controlling the marker, without two markers
contesting a link only one can hold. Backups from a marker PC file against the
marker rather than the operation.
That is one instance of a general pattern - several devices under one parent.
[ASSET-COMPOSITION](ASSET-COMPOSITION.md) covers when to use it, what
propagation buys you, and how to declare a new device type without a code change.
## 4. PC -> device links (`asset-id.txt`)
Some pc-types mean the PC drives an attached DEVICE: a part marker, or a
metrology instrument such as a CMM, Keyence, Genspect or wax-trace unit.
`SUBORDINATE_DEVICE_MAP` in `plugins/computers/pctypemap.py` declares which, and
a site can add its own through a `subordinatedevice_<pctype>` setting without a
code change.
**On this line only the part-marker case does anything.** The four metrology
pc-types declare `assettype: measuring_tool`, and the plugin that owned that
asset class is not bundled here (ADR-018: metrology instruments are machines).
The collector detects the absence, skips the link, and says so in the response
warnings - it does not fail the report, and it does not mint anything. File a
CMM as a machine and link it by hand.
`C:\Enrollment\asset-id.txt` names that device: one line, the device's
`assetnumber`. It does NOT record what kind of device it is - the pc-type already
does - so a bay declared later needs no new file and no client change.
Resolution runs most-stable-identity first, and ADOPTS before it mints:
1. **`deviceid`** - the asset named in `asset-id.txt`.
2. A prior collector link from this PC, reactivated.
3. An existing device this PC already controls that the collector did NOT
create - a legacy `MT-####` row, or one somebody made by hand. Adopting
stamps the label so it is ours from then on.
4. Mint, only when none of the above matched.
**Why the file exists, and why it is step 1.** Every other identity here is
derived from the PC. Step 2 looks for a link from THIS PC asset; step 4 mints
`<PC number>-<SUFFIX>`, and a PC's asset number is its hostname. All of them
survive a re-image and NONE of them survives a PC SWAP: a new hostname is a new
PC asset with no prior link and a predicted number that has never existed, so
the same physical device gets a second record while the first keeps its config
and backup history under a dead PC's name. That is how 43 legacy `MT-####`
tools ended up shadowed by minted twins, three records deep in places. Treat
anything minted as a placeholder to be reconciled.
Two guards on the file's contents:
- An **unresolvable** value warns and links nothing, rather than inventing a
phantom device nobody can account for.
- A value resolving to the **wrong kind** of asset is refused, naming the asset
it hit. A machine number pasted into `asset-id.txt` would otherwise file that
MACHINE under a device label the collector also owns - a link that reads as an
instrument, or as a marker, everywhere downstream.
A part marker additionally stays `partof` the operation from `machinenumber`,
because several markers serve one operation. Naming the marker in `asset-id.txt`
changes which marker asset the PC controls; it does not change which operation
that marker files under.
## 5. pc-type mapping (configurable per site)
`pctype` (e.g. `gea-shopfloor-cmm`) maps to a shopdb Computer Type through
`pctypemap_<pxetype>` settings. The built-in defaults in
`plugins/computers/pctypemap.py` are seeded on plugin install and the collector
reads them, so existing enrollment keeps working; add a setting of that name to
map an imaging type of your own. Unmapped pc-types produce a warning, not a
failure, because a bay must not stop reporting its inventory over a naming
question.
## 6. PC -> printer relationships are NOT set here (retired)
**A collector report has never been allowed to set a printer assignment, and as
of 2026-08-24 it no longer can.** `defaultprinter` and `printers` used to be
accepted on the computers payload and written as `defaultprinter`/`connectedto`
relationship rows. That contradicted "observed is not assigned" below, and this
contract carried both claims at once while the code followed the wrong one.
If you are still sending either key: **your push still succeeds** - they are
ignored, and the response carries a warning naming them. To report what a PC
actually has, use `POST /api/collector/printers`; to turn that into an
assignment, a person calls
`POST /api/printers/assignments/seed-from-observed/<assetid>` after reviewing the
comparison.
## 7. Printers payload (observed print queues)
`POST /api/collector/printers`. ShopDB has always known what a bay SHOULD have -
`GET /api/printers/for-host/<hostname>`, applied by `Set-ShopdbPrinters.ps1` -
and has never known what it actually has. This payload is that missing half.
Same dispatcher, same auth, same audit row; only the payload and the plugin
differ.
- Schema + apply: `get_collector_schema` / `apply_collector_payload` in
`plugins/printers/plugin.py`.
- Client: `plugins/printers/client/Report-PrintersToShopDB.ps1` (SYSTEM context,
`Type=PS1` / DetectionMethod `Always` manifest entry, logs to
`C:\Logs\Shopfloor\report-printers-YYYYMMDD.log`, always exits 0).
- Storage: `printerobservedqueues`, one row per observed queue, owned by the
printers plugin. A separate table from the assignment on purpose.
The identity field is `hostname`. Unlike the computers payload this one is NOT
patch-style: `queues` is required, and the reported set replaces everything
previously recorded for that host.
| Payload field | Type | Server behaviour |
|---|---|---|
| `hostname` (required) | string | Identity of the report. `COMPUTERNAME` or its FQDN; matched case-insensitively. Rows are keyed by the NAME, so a bay that reports before its PC record exists still records everything - the report resolves to an asset the moment that record appears. An unknown hostname is a warning, never an error. |
| `queues` (required) | array of objects | Every real print queue on the host. This REPLACES the host's previous set. An empty array is a valid report meaning "this bay has no queues" and clears the host's rows. A missing key is rejected (400), because one client bug that dropped the field would otherwise erase the fleet's observed state host by host. |
| `queues[].queuename` (required) | string | Windows printer name. A queue with no name is skipped with a warning; a repeated name within one report is dropped with a warning (Windows cannot hold two queues of one name). |
| `queues[].drivername` | string | Driver name verbatim, as the INF spells it. Compared against the assignment's driver to detect drift. |
| `queues[].portname` | string | Windows port name. Also tried as a match key, since a port created outside the client is usually named after the host address. |
| `queues[].portaddress` | string | `PrinterHostAddress` of a TCP/IP port - an IP or FQDN. The PRIMARY key for matching a queue to a printer asset. Omit it for a non-TCP port (USB, WSD, redirected); such a queue is still worth reporting and matches on name alone. |
| `queues[].isdefault` | boolean | True on the one queue that is the user's default. If several arrive true, the first is kept and the rest are cleared with a warning - a bay has exactly one default, and keeping both would leave a seed picking one at random. |
| `queues[].isshared` | boolean | True when the queue is shared off this PC. Recorded, not acted on. |
| `observedat` | ISO-8601 datetime | Accepted and ignored, as is any other client timestamp. The server stamps `observedat` at ingest, one value for the whole report, so a bay with a wrong clock cannot report itself fresh or stale. |
Response `data` is the standard collector shape: `action` is always `updated`
(this endpoint replaces rows and creates no asset - calling an identical
re-report `noop` would hide that the bay is still checking in), `assetid` is the
resolved PC or `null`, `extra.queuecount` is how many rows were stored, and
`warnings` carries the soft problems above.
```
POST /api/collector/printers
X-Endpoint-Credential: sdbe_...
Content-Type: application/json
{
"hostname": "WORKSTATION01",
"queues": [
{
"queuename": "Bay Label Printer",
"drivername": "Generic / Text Only",
"portname": "IP_192.0.2.40",
"portaddress": "192.0.2.40",
"isdefault": true
},
{
"queuename": "Office Laser",
"drivername": "HP Universal Printing PCL 6",
"portname": "IP_192.0.2.41",
"portaddress": "192.0.2.41",
"isdefault": false
}
]
}
```
### The latest report replaces the previous one
This is CURRENT STATE, not history. Each report deletes every row previously
recorded for that hostname and writes the reported set in its place, inside the
dispatcher's transaction. Nothing accumulates, "what does this bay have" is
never a question about time, and a queue removed from a bay disappears from
ShopDB on the next cycle without anyone tidying up.
That has one sharp edge, and it belongs to the client: an empty `queues` array
is a legitimate report that wipes the host's rows. A client whose enumeration
FAILED must therefore send nothing at all rather than an empty list. Reporting
nothing loses one cycle; reporting `[]` after a WMI hiccup deletes real state
and reads as a bay that lost its printers. `Report-PrintersToShopDB.ps1` tracks
this with an `$enumerated` flag and exits without posting when both the cmdlet
and the WMI fallback failed.
### Observed is not assigned
Stated plainly, because it is the whole point of keeping two tables:
**A collector report NEVER becomes an assignment.** Nothing on this path writes
a `usesprinter` or `defaultprinter` row. The moment a drifted bay's observed
state is treated as correct, enforcement stops meaning anything - a bay that
installed the wrong printer would make itself right simply by reporting it.
Observed state becomes assigned state only when a person asks for it, through
`POST /api/printers/assignments/seed-from-observed/<assetid>` (requires
`printers.edit`), after reviewing the comparison. That route refuses rather than
guesses: it will not seed from an ambiguous host, will not seed queues that
match no printer unless told to, and will not write an empty set.
### Matching, and what "unknown" means
Nothing is matched at ingest - the raw observed strings are stored as reported,
and resolution to a printer asset happens at READ time, so a printer added to
ShopDB tomorrow matches yesterday's report without the bay reporting again.
At read time each queue is matched by PORT ADDRESS first (an IP or FQDN names
one device unambiguously), then by queue name. There is no fuzzier fallback: a
queue that matches nothing is reported as `unknown` rather than guessed, because
a wrong match seeds a wrong assignment, which is worse than no assignment.
### Reading it back
| Endpoint | Purpose |
|---|---|
| `GET /api/printers/observed/<hostname>` | What the host last reported, every queue classified against the assignment it resolves to (`matching`, `drifted`, `extra`, `missing`, `unknown`), with `observedat`, a summary, and a `seedcandidate` preview. Read-only. Needs `printers.view`. |
| `POST /api/printers/assignments/seed-from-observed/<assetid>` | The one path from observed to assigned, human-triggered. Needs `printers.edit`. Normally posted against the MACHINE, so the assignment survives a reimage. |
Server-side the printers key resolves per-plugin first: set
`COLLECTOR_API_KEY_PRINTERS` if you scope env keys per collector, or rely on the
shared `COLLECTOR_API_KEY`. A site that has scoped the computers key per-plugin
and set no printers key gets 401 here until one of the two is in place. A
per-PC credential and a `collector.ingest` managed token both work here exactly
as they do for computers.
## 8. Porting from classic `api.asp`
The fleet's classic-ASP reporter posts form fields to
`api.asp?action=updateCompleteAsset`. Map them to the collector JSON as follows:
| Classic `updateCompleteAsset` form field | Collector field |
|---|---|
| `hostname` | `hostname` |
| `machineNo` | `machinenumber` |
| `deviceId` | `deviceid` |
| `pcType` | `pctype` |
| `serialNumber` | `serialnumber` |
| `loggedInUser` | `loggedinuser` |
| `lastBootUpTime` / `lastBootTime` | `lastboottime` |
| `manufacturer` | `vendorname` |
| `model` | `modelnumber` |
| `osVersion` | `osname` |
| `installedApps` | `installedsoftware` |
Not carried over (no current home in the computers schema): warranty fields, DNC
config, VNC/WinRM flags as booleans (report them through `accessprotocols`
instead). The classic reporter's full `networkInterfaces` array maps onto
`interfaces`.

352
docs/CONFIG.md Normal file
View File

@@ -0,0 +1,352 @@
<!-- docmeta
purpose: reference
audience: operator, adopter, developer
status: current
summary: Every environment variable and Settings key, what it defaults to and which one to reach for.
loadwith:
-->
# Configuration reference
**To find one setting: search this page for its exact key.** Every key appears
in exactly one table. Nothing here has to be read in order.
Two places, and the split is deliberate:
- **Environment variables** hold **secrets and deploy-time wiring**: database
credentials, signing keys, CORS origins, ports, API keys. Read once at boot by
`shopdb/config.py`, which is authoritative - where this page and that file
disagree, the file wins.
- **The Settings table** holds **site preferences**: branding, ServiceNow links,
search toggles, facility identity. Seeded by `flask seed settings`, edited in
the UI under Settings or the setup wizard, changeable at runtime with no
restart. `build_default_settings` in `shopdb/core/api/settings.py` is
authoritative for the defaults.
Rule of thumb: if leaking it would be a security incident, it is an environment
variable. Never put a secret in the Settings table.
**Part 1, environment variables:** [Flask core](#flask-core) |
[Database](#database) | [Auth rate limiting](#authentication-rate-limiting) |
[Collector ingest](#collector-ingest-adr-006) | [Zabbix](#zabbix-printer-supply-monitoring) |
[Employee directory DB](#employee-directory-database-optional-read-only) |
[docker-compose only](#docker-compose-only)
**Part 2, Settings keys:** [site](#site) (facility name, timezone, base URL,
retention) | [branding](#branding) | [printing](#printing) (QR and label
targets) | [map](#map) (superseded) | [integrations](#integrations)
(ServiceNow, Zabbix, Dell warranty) | [email](#email) | [audit](#audit) |
[auth](#auth) (SAML placeholders) | [tools](#tools) (Secure Send) |
[identifiers](#identifiers-dynamic) | [search](#search-dynamic) |
[Custom fields](#custom-fields)
---
## Part 1: Environment variables
Copy `.env.example` to `.env` and fill in values. In `production`
(`FLASK_ENV=production`), `ProductionConfig.validate()` refuses to boot if
`SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, or `CORS_ORIGINS` are missing or
set to the dev defaults.
### Flask core
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `FLASK_APP` | No | `wsgi.py` | Entry point for the `flask` CLI. |
| `FLASK_ENV` | Yes | `development` | `production` for live sites (triggers `validate()`). Other values: `development`, `testing`. |
| `SECRET_KEY` | Yes (prod) | dev default | Flask session/signing key. Generate: `python -c "import secrets; print(secrets.token_urlsafe(64))"`. |
| `JWT_SECRET_KEY` | Yes (prod) | dev default | JWT signing key. Different value from `SECRET_KEY`. |
| `JWT_ACCESS_TOKEN_EXPIRES` | No | `3600` | Access-token TTL in seconds. |
| `JWT_REFRESH_TOKEN_EXPIRES` | No | `2592000` | Refresh-token TTL in seconds (30 days). |
| `CORS_ORIGINS` | Yes (prod) | `http://localhost:5173` | Comma-separated explicit origins. Wildcard `*` is rejected in production. |
| `LOG_LEVEL` | No | `INFO` | Logging verbosity. |
### Database
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `DATABASE_URL` | Yes (prod) | dev localhost URL | `mysql+pymysql://<user>:<pass>@<host>:<port>/<db>?charset=utf8mb4`. Keep `?charset=utf8mb4`. |
### Authentication rate limiting
IP-based fixed-window limit on the login endpoint, defense-in-depth atop the
per-account lockout. Uses the cache extension, which is per-process, so the
limit is approximate across multiple gunicorn workers.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `AUTH_RATELIMIT_ENABLED` | No | `True` | Set `False` to disable (TestingConfig disables it). |
| `AUTH_RATELIMIT_MAX` | No | `30` | Max login attempts per source IP per window before 429. |
| `AUTH_RATELIMIT_WINDOW_SECONDS` | No | `300` | Window length in seconds. |
### Collector ingest (ADR-006)
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `COLLECTOR_API_KEY` | No | (empty) | Shared key for `/api/collector/*`. Endpoint fails closed (denies) when unset and no managed token is presented. Sent as the `X-API-Key` header. |
| `COLLECTOR_API_KEY_<PLUGIN>` | No | (empty) | Per-plugin override, e.g. `COLLECTOR_API_KEY_COMPUTERS`. Checked before the shared key. |
These env keys are the bootstrap/legacy fallback. A managed API token scoped to
`collector.ingest` is preferred and is minted, rotated and revoked from
Settings > API Tokens; see
[COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) (Managed collector tokens).
### Zabbix (printer supply monitoring)
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `ZABBIX_ENABLED` | No | `false` | Enable the Zabbix integration. |
| `ZABBIX_URL` | No | (empty) | Zabbix API URL. |
| `ZABBIX_TOKEN` | No | (empty) | Zabbix API bearer token. |
These are the boot-time wiring; the Settings entries (`zabbix_enabled`,
`zabbix_url`, `zabbix_token`) are preferred for runtime changes.
### Employee directory database (optional, read-only)
Separate HR lookup DB, read-only. This is where staff names come from: global
search resolves the employee domain through it, and the audit log uses it to
show a name instead of a bare SSO. Nothing here is required - leave it unset and
both surfaces fall back to SSOs. There is no safe default for the password; an
unset one fails loud rather than trying a guessed credential.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `EMPLOYEE_DB_HOST` | No | `localhost` | HR DB host. |
| `EMPLOYEE_DB_USER` | No | (empty) | HR DB user. |
| `EMPLOYEE_DB_PASSWORD` | No | (empty) | HR DB password. No safe default. |
| `EMPLOYEE_DB_NAME` | No | `wjf_employees` | HR DB name. |
### docker-compose only
Read by `docker-compose.yml`, not by the Flask app directly.
| Variable | Required | Default | Notes |
|----------|----------|---------|-------|
| `MYSQL_ROOT_PASSWORD` | Yes | (none) | Root password for the bundled MySQL container. |
| `MYSQL_PASSWORD` | Yes | (none) | App-user password; must match the `DATABASE_URL` password. |
| `MYSQL_PORT` | No | `3306` | Host port for MySQL. Bound to `127.0.0.1` only. |
| `API_PORT` | No | `5001` | Host port for the API container. |
The opt-in backup overlay adds `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`,
`SHOPDB_SITE` and optional cloud credentials - see
[BACKUP-RESTORE.md](BACKUP-RESTORE.md).
---
## Part 2: Settings table keys
Seeded by `flask seed settings` (idempotent; re-running adds anything missing).
Values are stored as strings and typed by `valuetype`. Secrets in this table
(anything whose key contains `password`, `token`, or `secret`) are masked when
read back through the API.
### site
| Key | Default | Notes |
|-----|---------|-------|
| `setup_complete` | `false` | Set true once the first-run wizard finishes; gates the `/setup` route. |
| `employee_directory_mode` | `selfhosted` | `external` reads a separate HR database (see `EMPLOYEE_DB_*`). No plugin on this line keeps a self-hosted directory, so `selfhosted` resolves no names - set `external`, or accept bare SSOs. |
| `audit_retentiondays` | `0` | Delete audit entries older than this many days. 0 keeps everything forever. Saving a period deletes nothing by itself: purge from this page or run `flask audit purge --apply`. |
| `site_base_url` | (empty) | Public base URL for QR codes and absolute links. Scheme + host, optionally including the subpath a site is served under (`https://host/shopdb`). Blank = use the browsing origin. |
| `facility_name` | (empty) | Facility name in the dashboard header. Blank = frontend falls back to `ShopDB`. |
| `pc_access_domain` | `device.geaerospace.net` | Domain appended to a PC hostname for remote-access links. Blank = hostname as-is. |
| `employeeid_pattern` | `^\d{9}$` | Regex a search term must match to be treated as an employee id. Invalid regex falls back to the default and never 500s. |
| `printer_hostname_template` | `Printer-{ip}.printer.geaerospace.net` | Printer hostname template. `{ip}` is the dash-separated IP address. |
| `contact_email_domain` | `geaerospace.com` | Email domain appended to a support contact's SSO to build email (`sso@domain`) and Teams-chat links. Blank hides the contact action buttons. |
| `dualpath_single_machine` | `true` | Treat a Dualpath pair (a dual-bay machine with one controller) as a single machine in the machines list, dashboard/report counts, and the floor map (the secondary bay is hidden). The data model always keeps both bay records; detail pages stay per-bay with a sibling banner. `false` lists and counts both bays separately. |
| `site_timezone` | `America/New_York` | IANA timezone for the site. Notification times are stored UTC and converted for display by `frontend/src/utils/datetime.js`, so they are entered and shown in THIS zone, not the viewer's browser zone; daily-reset notification expiry (`expirymode=dailytime`) is computed here too. Public-readable so kiosks can resolve it. |
### branding
Blank values fall back to the shipped GE default asset, so an un-reconfigured
install still renders. Upload replacements at Settings > Branding, which saves
them under `instance/branding/`.
| Key | Default | Notes |
|-----|---------|-------|
| `site_logo` | `/ge-aerospace-logo.svg` | Header and login-page logo. |
| `qr_logo` | `/ge-monogram.svg` | Logo composited in printer QR labels. Blank = no overlay. |
| `badge_logo` | `/ge-aerospace-logo.svg` | Logo on the machine badge print page. |
| `site_favicon` | (empty) | Browser tab favicon. Blank = shipped `/favicon.svg`. |
| `brand_primary_color` | (empty) | Primary brand color as a CSS color value (maps to `--primary`). Blank = built-in theme color. |
| `brand_primary_dark_color` | (empty) | Primary hover/active color (maps to `--primary-dark`). Blank = auto-derived by darkening the primary color ~15%. |
| `brand_accent_color` | (empty) | Accent color for secondary buttons and badges (maps to `--secondary`). Blank = built-in theme color. |
| `brand_sidebar_color` | (empty) | Sidebar background color (maps to `--sidebar-bg`). Blank = built-in theme color. |
### printing
| Key | Default | Notes |
|-----|---------|-------|
| `qr_target_printer` | (empty) | Custom URL template for printer QR labels. Blank = link to the printer page on this instance. Placeholders: `{printerid}`, `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{ip}`, `{hostname}`. |
| `qr_target_machine` | (empty) | Custom URL template for machine labels. Blank = link to the machine page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
| `qr_target_computer` | (empty) | Custom URL template for computer labels. Blank = link to the computer page. Same placeholders as machine. |
| `qr_target_network_device` | (empty) | Custom URL template for network-device labels. Blank = link to the device page. Same placeholders as machine. |
| `label_default_style` | `card` | Default asset-label layout when a label first opens: `card` (badge with image and identity) or `plain` (just the code and a caption). |
| `label_default_codetype` | `qr` | Default asset-label code type when a label first opens: `qr` or `barcode` (CODE128). |
| `label_default_encodes_machine` | `assetnumber` | What a machine label encodes by default. |
| `label_default_encodes_computer` | `assetpage` | What a computer label encodes by default. |
| `label_default_encodes_printer` | `assetpage` | What a printer label encodes by default. |
| `label_default_encodes_network_device` | `assetpage` | What a network-device label encodes by default. |
Values for those four `label_default_encodes_*` keys: `assetpage`,
`assetnumber`, `serialnumber`, or `custom` (the matching `qr_target_<type>`
template). All of it is overridable on the label page itself.
There is no `measuring_tool` label type on this branch. Metrology instruments
are machines (ADR-018), so use the machine label keys, and the measuring-tools
plugin is not bundled here - `shopdb/core/api/settings.py` declares four
`qr_target_*` and four `label_default_encodes_*` keys to match.
| Route | What it does |
|---|---|
| `/print/asset-label/<assettype>/<id>` | One label. Public, like the other `/print/*` pages. `<assettype>` is `machine`, `computer`, `printer` or `network_device`; `<id>` is the asset's plugin id. |
| `/print/asset-label-batch/<assettype>` | A multi-selection of one type onto ULINE sheets: 6-up 3 in x 3 in, or a dense 72-up mini-label, with a start-cell offset for reusing partial sheets. Reached from the "Print Labels" button on each asset list page. |
### map
**SUPERSEDED as of 0.11.0. Editing these four keys changes nothing on the map.**
A blueprint and its pixel dimensions are properties of a LEVEL now (ADR-017).
They are retained only so a downgrade finds the blueprint it had; the levels
migration copied their values onto the first level. Manage plans in
**Settings > Buildings and levels** - see [FLOOR-MAP.md](FLOOR-MAP.md).
| Key | Default | Superseded by |
|-----|---------|-------|
| `map_blueprint_light` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintlight` |
| `map_blueprint_dark` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintdark` |
| `map_width` | `3300` | `maplevels.mapwidth` |
| `map_height` | `2550` | `maplevels.mapheight` |
### integrations
| Key | Default | Notes |
|-----|---------|-------|
| `servicenow_enabled` | `true` | Enable ServiceNow ticket recognition and links. Disabled = tickets render as plain text. |
| `servicenow_search_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. |
| `servicenow_ticket_prefixes` | `GEINC,GECHG,GERIT,GESCT` | Comma-separated prefixes recognized as ServiceNow tickets. |
| `servicenow_incident_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct incident URL if your instance has one. |
| `servicenow_change_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct change URL if your instance has one. |
| `zabbix_enabled` | `false` | Enable Zabbix for printer supply monitoring. |
| `zabbix_url` | (empty) | Zabbix API URL. |
| `zabbix_token` | (empty) | Zabbix API token (masked). |
| `warranty_dell_enabled` | `false` | Enable Dell warranty (service-tag) lookups. |
| `warranty_dell_clientid` | (empty) | Dell TechDirect API client id. |
| `warranty_dell_clientsecret` | (empty) | Dell TechDirect API client secret (masked). |
| `warranty_dell_tokenurl` | (empty) | Dell OAuth token URL. Blank = Dell default. |
| `warranty_dell_apiurl` | (empty) | Dell warranty API URL. Blank = Dell default. |
### email
| Key | Default | Notes |
|-----|---------|-------|
| `smtp_enabled` | `false` | Enable email notifications and alerts. |
| `smtp_host` | (empty) | SMTP server hostname. |
| `smtp_port` | `587` | SMTP port (587 TLS, 465 SSL, 25 plain). |
| `smtp_username` | (empty) | SMTP auth username. |
| `smtp_password` | (empty) | SMTP auth password (masked). |
| `smtp_use_tls` | `true` | Use TLS for the SMTP connection. |
| `smtp_from_address` | (empty) | From address for outgoing email. |
| `smtp_from_name` | `ShopDB` | From name for outgoing email. |
| `alert_recipients` | (empty) | Default alert/report recipients (comma-separated). |
The mailer (`shopdb/utils/mailer.py`) reads these settings-first, falling back to
matching `SMTP_*` environment variables only when at least one of them is set.
When `smtp_enabled` is false or `smtp_host` is blank, every send is a graceful
no-op that logs a warning, so an unconfigured site never crashes. The password
is never logged.
Three flows send mail, and **there is no scheduler in this app** - sending is
always on-demand:
- **Welcome email**, on `POST /api/users`. Best-effort: the account is created
even if the send fails (the response carries a `warning`). The new account is
flagged `mustchangepassword`, so the frontend forces `/change-password` before
the app; changing it clears the flag and resets lockout counters. Opt out with
`sendwelcome: false` in the create body.
- **Test email**, `POST /api/settings/test-email` (`settings.edit`), behind the
"Send Test Email" button. A real SMTP error is surfaced with the password
scrubbed out.
- **Alerts and report delivery**, `POST /api/reports/email` (`reports.export`),
taking `{subject, columns, rows, intro?, to?}` and mailing the rows as an HTML
table. Recipients default to `alert_recipients`. To automate a recurring send,
point an external cron job at this endpoint with a PAT scoped to
`reports.export`.
### audit
Retention is `audit_retentiondays`, in the [site](#site) section. There is only
one such key; anything writing `audit_retention_days` is writing to nothing.
### auth
**SAML is not implemented yet.** These keys exist so a site can be configured
ahead of the feature, and the settings page says so on screen. Nothing reads
them: there is no SAML login route and no assertion is accepted, so sign-in is
local accounts only whatever is set here. In particular `saml_allow_local_login`
does NOT disable password login - the toggle is locked on until there is another
way in, because a control that appears to close the local door while it stays
open is worse than no control.
| Key | Default | Notes |
|-----|---------|-------|
| `saml_enabled` | `false` | Reserved. Enable SAML SSO once it ships. |
| `saml_idp_metadata_url` | (empty) | Reserved. SAML IdP metadata URL. |
| `saml_entity_id` | (empty) | Reserved. SAML SP entity id. |
| `saml_acs_url` | (empty) | Reserved. SAML Assertion Consumer Service URL. |
| `saml_allow_local_login` | `true` | Reserved, and locked on. Local login cannot be turned off while it is the only way in. |
| `saml_auto_create_users` | `true` | Reserved. Auto-create users on first SAML login. |
| `saml_admin_group` | (empty) | Reserved. SAML group name that grants the admin role. |
**Personal API tokens (PATs) have no configuration here**, and
`JWT_ACCESS_TOKEN_EXPIRES` does not apply to them. They are created and revoked
from Settings > API Tokens; the token model is documented once, in
[IMPORT-API.md](IMPORT-API.md#1-authentication).
### tools
| Key | Default | Notes |
|-----|---------|-------|
| `securesend_allow_guests` | `false` | Let people with no account create Secure Send links. Public (readable before login) so the create page can show a guest the form instead of a sign-in wall. Off by default: a guest secret has no creator, so the audit trail can say when and from where but never who. Receiving a secret never needs an account either way. |
[SECURE-SEND.md](SECURE-SEND.md) is authoritative for the rest of it - guest
rate limiting, why the read fails closed, and why the size limit is not a tuning
knob.
### identifiers (dynamic)
One boolean key per asset identifier per asset type, keyed
`identifier_<name>_<assettype>_enabled` (default `true`). Admins choose which
optional identifiers show on which asset types. See ADR-001. The exact set is
generated from `IDENTIFIER_LABELS` x `IDENTIFIER_ASSETTYPES` in
`shopdb/core/api/settings.py`.
### search (dynamic)
One boolean key per search domain, keyed `search_<type>_enabled` (default
`true`). Toggles whether a domain appears in global search results. The set is
generated from `SEARCH_DOMAINS` in `shopdb/core/api/settings.py`.
Search terms are matched word-wise: a multi-word query returns rows containing
EVERY word, each word anywhere in the searched fields, in any order ("CSF Roles"
matches a row with "CSF" and "Roles" in different columns). Quoting does not
force a contiguous phrase.
## Custom fields
Site-defined extra attributes per asset type (Settings > Custom Fields, table
`customfields`). Each field has a `searchable` flag (default off). When on, the
field's stored values are matched by global search and a hit routes to the
owning asset's detail page. The asset's `search_<type>_enabled` domain toggle
still applies, so a custom-field hit on a computer only shows when the computer
search domain is enabled. Inactive or non-searchable fields are never matched.
---
## See also
- [DEPLOY.md](DEPLOY.md) - per-site deployment runbook
- [UPGRADE.md](UPGRADE.md) - upgrading an existing site
- [BACKUP-RESTORE.md](BACKUP-RESTORE.md) - backup and restore
- `GET /api/docs` - what endpoints exist and their auth, generated from the code

149
docs/CONTRACT-STABILITY.md Normal file
View File

@@ -0,0 +1,149 @@
<!-- docmeta
purpose: explanation
audience: adopter, integrator
status: current
summary: How stable each part of the plugin contract is today, and how tightly a sister site should pin.
loadwith: none
-->
# Path to contract 1.0
This page is an honest read of how stable the plugin contract is today, for a
sister site deciding how much to build on it. It is derived from the ADRs and
the live code, not aspiration. It does NOT define the contract - what each hook
takes and returns is in [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md), and the versioning
rules are in [ADR-002](adr/ADR-002-plugin-versioning.md). What it adds is a
judgement about how much weight each part will bear.
## Current version
The plugin contract version is declared in `shopdb/__init__.py` as
`__contract_version__`, and the current value is in
[PROJECT-MAP.md](PROJECT-MAP.md), which is generated. It is pre-1.0, which under
semver means any 0.x minor bump is allowed to break the contract, and this
project has used that latitude - see 0.19.0 below.
The product release version (`__version__`) is a separate series with its own
bump rules; see [ADR-007](adr/ADR-007-product-versioning-and-releases.md).
Do not pin against it for compatibility - pin against `__contract_version__`.
### 0.x history
Recorded in the comment block in `shopdb/__init__.py`:
| Version | Change | Kind |
|---------|--------|------|
| 0.3.0 | `shopdb.api` expanded to the full plugin import surface (db, cache, model bases, core models, response + pagination helpers, `employee_connection`) so plugins stop importing internal core paths | additive (minor) |
| 0.4.0 | Removed the never-implemented `get_searchable_fields` hook (search is a core concern over the asset model) and wired `get_dashboard_widgets` to a real consumer (`/api/dashboard/widgets`) | pre-1.0 contract reduction |
| 0.6.0 | Added the `get_reports` hook, consumed by `GET /api/reports` to merge plugin report cards into the Reports hub | additive optional hook (minor) |
| 0.7.0 | Added the four ADR-010 frontend-contribution hooks (`get_settings_cards`, `get_asset_panels`, `get_map_overlays`, `get_asset_presentation`), consumed by the `GET /api/pluginui/*` endpoints | additive optional hooks (minor) |
| 0.9.0 | Exposed the dualpath pair-resolution helpers on `shopdb.api` for the machines plugin | additive surface (minor) |
| 0.10.0 | Added the `get_permissions` hook so plugins declare their own RBAC permissions; the catalog is resolved dynamically from core + enabled plugins | additive optional hook (minor) |
| 0.11.0 | Added `service_token_authorized(scope)` to `shopdb.api` so a plugin's unattended endpoints can authorize a scoped managed service token without importing core token internals | additive surface (minor) |
| 0.12.0 | Added the mailer helpers (`send_email`, `send_alert`) to `shopdb.api` | additive surface (minor) |
| 0.13.0 | Added the `User` model to the `shopdb.api` surface | additive surface (minor) |
| 0.14.0 | Added `send_webhook` to the `shopdb.api` surface | additive surface (minor) |
| 0.15.0 | Added `authorized_service_token` / the `SupportTeam` model to the `shopdb.api` surface | additive surface (minor) |
| 0.16.0 | Added the `get_settings_defaults` hook so a plugin declares the Setting rows it owns; the framework seeds them at install, at enable, and on `flask plugin upgrade-all`, files a first-time write under the declared category, and honours `public: True` for pages that render before login | additive optional hook (minor) |
| 0.17.0 | Added `DashboardDefault` to the `shopdb.api` surface, so a plugin can resolve a display without reaching into core | additive surface (minor) |
| 0.18.0 | Added `DISPLAY_ROLES`, `DISPLAY_ROLE_PATHS` and `normalize_display_role`, and finally exported `DashboardDefault`, which 0.17.0 imported but left out of `__all__`. The role vocabulary became the kiosk's own - `Dashboard`, `Lobby`, `3DPrintRoom` - so a plugin holding its own copy of that map reads core's instead of drifting from it | additive surface (minor) |
| 0.19.0 | **BREAKING.** `get_dashboard_widgets` returns DATA AND SHAPE, not a component name. The old shape (`name` + `component` + `size`) named a Vue component per widget, which cannot survive a lean build - a plugin's component may never be staged into the frontend bundle (ADR-013) - and five plugins were declaring widgets that pointed at components nobody had written. A card now declares `id` / `title` / `endpoint` / `render` / `severity` / `permission` / `empty` / `position`; see PLUGIN-HOOKS.md. Also added `send_upload` so a plugin serving user-supplied bytes gets the headers that keep an SVG from running as script | **contract change (minor, pre-1.0)** |
| 0.20.0 | The `Asset` model gained `levelid` (which floor plan its `mapx`/`mapy` are pixels of) and `mapverifiedat`; `Location` gained `levelid` too, and `resolve_asset_position` now returns the `levelid` belonging to whichever source supplied the coordinates. A plugin that writes a map position MUST write the level with it - a position without one renders as "level unknown" rather than being drawn on the default drawing, because a marker on the wrong floor plan looks correct (ADR-017) | additive surface (minor) |
| 0.21.0 | Added the `get_guest_grantable_permissions` hook: a plugin names the WRITE permissions a site may grant to the seeded `guest` role, which is who anyone without a token now holds. Reads (`.view`/`.read`) are grantable without a plugin saying anything; anything else is refused. Also exposed `require_any_permission` on `shopdb.api` for a read two different rights should reach | additive optional hook + surface (minor) |
| 0.22.0 | Added the `register_import_hook` surface: a plugin owns a VIRTUAL COLUMN on an importable CSV table - a header the bulk importer accepts but never writes - plus a post-write row hook that turns the value into a plugin concept. The computers plugin uses it for `machineassetnumber` on `computers.csv`, which builds the PC->machine controls link, so a site writes the machine number in the PC's own row instead of hand-building a separate relationships sheet. Core never learns what the column means. `RowWarning` is exported with it so a hook can report a soft failure without aborting the import | additive surface (minor) |
| 0.23.0 | Added `mint_service_token`, `list_service_tokens` and `revoke_service_tokens` so a plugin can provision, list and revoke a single-scope managed service token without importing core token internals. The endpoints plugin uses them for a UI-managed, regenerable agent key scoped to `collector.ingest.endpoints` alone - narrow on purpose, so rotating or revoking it cannot disturb the computers collector, and revocation is an instant kill switch on agent ingest | additive surface (minor) |
| 0.24.0 | Added `collector_authorized(pluginname)` so a plugin can gate an agent-facing route that lives OUTSIDE the generic collector blueprint on the same per-plugin key and kill switch the ingest route uses. The endpoints command channel needs it: the agent long-polls `/api/endpoints/commands/next` and posts results back, and those are agent callbacks rather than JWT routes. Without it a plugin would reach into core collector internals to authorize them | additive surface (minor) |
| 0.25.0 | Added `caller_holds(permission)`, which answers whether the current caller holds a permission WITHOUT refusing the request - the complement to `require_permission` for a route that serves everyone but should only offer what the caller can act on. The endpoints services panel is the case: it emitted Start/Stop row actions to every reader, because the emission never consulted the caller. Clicking them was always refused server-side, so nothing was exploitable, but offering a read-only viewer controls over a shop-floor PC is the wrong signal, and it becomes a real problem once a site grants that panel to guests. Guests are included in the answer, since a signed-out visitor holds whatever the guest role was granted | additive surface (minor) |
| 0.26.0 | Dashboard cards may declare `shortname`. The dashboard merged its per-source cards into one worst-first queue, and a card's `title` is written as a heading ("Endpoints with crashes or shutdowns") which as a row badge ran wider than the hostname it qualified. Optional, defaults to the title, so a plugin that does not declare one is unaffected | additive field (minor) |
| 0.27.0 | Added `Script` so a plugin can read the ShopDB-hosted script library that backs endpoint `run_script` delivery without importing core model internals. The endpoints plugin resolves a delivered script by name and serves its body by digest to the agent | additive surface (minor) |
| 0.29.0 | Added an optional `group` to `get_asset_panels`, naming which detail-page tab a panel belongs to: `details` (the default) / `monitors` / `software` / `activity`. The detail pages are tabbed, and a page that placed panels by knowing their ids would defeat the hook. Additive: an absent or unrecognised group resolves to `details` server-side, so every existing plugin keeps working with no change | additive surface (minor) |
| 0.28.0 | Added `client_ip` and `strip_port` so a plugin does not hand-roll caller-IP extraction. Two plugins had, and both read the first `X-Forwarded-For` hop, which the caller supplies - so a rate limit keyed on it was defeated by rotating the header. The helper resolves from `remote_addr`, which ProxyFix has already filled from the header when `TRUSTED_PROXY_COUNT` says a proxy is really in front | additive surface (minor) |
| 0.31.0 | The asset-panel vocabulary a plugin may declare grew four optional things: two more tab groups, `serviceprocess` and `reports`, beside the 0.29.0 set; a `format: 'datetime'` on a column or field, so a panel stops shipping pre-formatted date strings the browser cannot re-localise; a `map.body` key, so a keyvalue field can name where its value lives in a nested payload; and a `tabs` section that carries `columns`/`rows`/`empty` and renders as a table, so one card can hold two tables that share no key (volume capacity and physical drive health are the case). All optional and all resolved server-side when absent, so a plugin declaring none of them is unaffected | additive surface (minor) |
The source comment block documents 0.3.0, 0.4.0, 0.6.0, 0.7.0, 0.9.0, 0.10.0,
0.11.0, 0.16.0, 0.18.0 and 0.19.0; 0.12.0 through 0.15.0 and 0.17.0 are recorded
in this table only.
**0.19.0 is the one to read before pinning.** It is the only entry in this
series that took something away, and it shipped before it was recorded here -
which is precisely the reason to pin tight rather than to trust that a minor
bump is safe. Earlier points
(0.1.x / 0.2.x) predate that recorded rationale; `PluginMeta`'s fallback
`core_version` default of `>=0.2.0,<1.0.0` is the only remaining trace of the
0.2 baseline.
## Settled surface
These are unlikely to break before 1.0. A plugin can depend on them with
reasonable confidence; a breaking change to any would be a major bump and would
land with a new or amended ADR. What each one does is in
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md); this is only the stability judgement.
- The required hooks and the manifest schema: `meta` / `PluginMeta`,
`get_blueprint`, `get_models`.
- Plugin plumbing: `init_app`, `get_cli_commands`, `get_services`, the settings
helpers (`get_setting` / `set_setting`), and the four lifecycle hooks.
- The UI-contribution hooks: `get_navigation_items`, `get_dashboard_widgets`,
`get_reports`, `get_permissions`, and the ADR-010 set (`get_settings_cards`,
`get_asset_panels`, `get_map_overlays`, `get_asset_presentation`).
- The collector pair, `get_collector_schema` + `apply_collector_payload`, per
[ADR-006](adr/ADR-006-collector-contract.md).
- The setup-wizard declarations: `get_provisioning_note`, `get_config_schema`.
- The `shopdb.api` import surface itself: adding a name is minor, removing one
is major.
The asset model itself - `Asset`, `AssetType`, `AssetStatus`,
`AssetRelationship`, and the shared reference models - is the platform contract
locked in [ADR-001](adr/ADR-001-asset-as-platform-contract.md).
## Expected churn before 1.0
Known-unstable areas. Building on these means expecting rework.
| Area | Status | Reference |
|------|--------|-----------|
| Frontend renderers (residual) | The four data-only hooks and their `/api/pluginui/*` consumers are settled (0.7.0). The generic core renderers are landing incrementally: the settings-cards rail/landing renderer ships with 0.7.0; the asset-panel, map-overlay, and search-presentation renderers are wired opt-in per the ADR adoption plan. Real component-backed panels (bespoke charts, custom overlays) remain deferred to Option C (build-time glob discovery). | [ADR-010](adr/ADR-010-frontend-plugin-hooks.md) (ACCEPTED) |
| Per-plugin migrations | Brand new. The per-plugin Alembic engine exists and every bundled plugin now carries a chain, but the pattern has one release of production mileage, not years. | [ADR-008](adr/ADR-008-plugin-migration-ownership.md) (2026-07-10) |
| Pip distribution | Deferred to v2. External plugins install by clone / submodule / symlink; there is no entry-point discovery and no automatic update path yet. | [ADR-003](adr/ADR-003-plugin-distribution.md) |
## Bump rules
From [ADR-002](adr/ADR-002-plugin-versioning.md), applied to `__contract_version__`:
| Bump | Trigger |
|------|---------|
| major | Breaking change to the `BasePlugin` ABC, the `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`). Removing a name from `shopdb.api` is major. |
| minor | Additive change: a new optional hook, a new field on a contract model with a default, a new name added to `shopdb.api`. |
| patch | Bug fix with no change to the contract surface. |
### Deprecation policy
ADR-002 defines the bump classification above but is **silent** on any
deprecation window or notice period for pre-1.0 removals. In practice a removal
is simply a major (or, pre-1.0, a breaking minor) bump: the hook or name is
gone, and the loader fails loud in dev or excludes the mismatched plugin in prod
(see the load-time table in [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md)).
Because the ADR says nothing about a grace period, this document proposes none;
the honest guidance for sister sites is the mitigation that already exists - pin
a tight `core_version` range and re-test before widening it.
## Criteria for declaring 1.0
Provisional. This is the maintainer's working list, not a committed checklist,
and it will move. ADR-002's own open question ("When does the framework declare
1.0.0?") ties 1.0 to the `Machine` retirement from ADR-001 and the framework
being "ready for sister sites"; the items below expand that intent.
- Frontend hook contract defined via its own ADR (asset-detail panels, map
markers, search results), closing the biggest churn item above.
- `Machine` / legacy-model retirement complete per ADR-001, so the asset model
is the only contract.
- At least two external plugins running in production at a second site - the
bar ADR-003 already sets before pip distribution and sister-site readiness
are considered justified.
- The per-plugin migration pattern (ADR-008) proven across real upgrades, not
just fresh installs.
- No contract bump needed for N consecutive product releases (N to be fixed
when the list is firmed up), showing the surface has actually settled.
Until then: pre-1.0, pin tight, re-test on every framework bump.

139
docs/CSV-IMPORT.md Normal file
View File

@@ -0,0 +1,139 @@
<!-- docmeta
purpose: howto
audience: adopter, operator
status: current
summary: Load a new site's starting data from generated CSV templates, with no developer and no source database.
loadwith:
-->
# Loading a site's data from spreadsheets
**Four import pages.** Spreadsheet and no developer: this page. The HTTP wire
contract: [IMPORT-API](IMPORT-API.md). How to shape a per-site loader around
it: [IMPORT-ADOPTION](IMPORT-ADOPTION.md). Source is the classic-ASP shopdb:
[IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) on top of the contract.
---
## The short version
```bash
cd C:\shopdb-flask # or your install directory
venv\Scripts\flask csv templates --out csv-templates
```
Fill in the templates. Then:
```bash
venv\Scripts\flask csv import --dir csv-templates
```
That **checks only** and changes nothing. It tells you what it would create and
what is wrong. When you are happy:
```bash
venv\Scripts\flask csv import --dir csv-templates --commit
```
---
## Write names, not numbers
This is the part that makes the difference. Every column that points at another
table accepts the **name** of the thing:
```
assetnumber,name,assettypeid,statusid,locationid
CMM-01,Zeiss Contura,Machine,Active,Gage Lab
MILL-07,Haas VF-2,Machine,Active,Bay 3
```
`assettypeid` gets `Machine`. `locationid` gets `Gage Lab`. You never
have to import a file, read back the numbers it generated, and paste them into
the next one.
The column keeps its database name so it matches the rest of the system, but
the value is whatever you actually know. Numeric ids still work if you have
them - useful when re-importing something this system exported.
Names resolve **across files in the same run**, so `assets.csv` can reference a
location that only exists because `locations.csv` was loaded moments earlier.
## Nothing is half-imported
Every row is checked before anything is written. If one row is wrong, nothing is
written at all - you fix the file and run it again. A mistake on line 400 never
leaves 399 rows loaded.
## Running it twice is safe
Each file is matched on a natural key - `assetnumber` for assets, `locationname`
for locations, and so on. Re-importing an edited file **updates** those rows
rather than creating second copies. Correcting a spreadsheet and re-running is
the expected workflow, not a mistake.
## What the errors look like
```
assets: 0 new, 0 updated, 1 problem(s)
line 4, column 'locationid': nothing in locations is named 'Bay 9'
- add it to locations.csv, or import that file first
```
Line, column, value, and what to do. Not a foreign key constraint violation.
---
## What you can import
Fourteen tables. `flask csv templates` writes one file per table and `--dir`
imports them in dependency order, which is the order the generated folder lists
them in: assetstatuses, assettypes, locationtypes, modeltypes, computertypes,
machinetypes, businessunits, locations, vendors, models, operatingsystems,
assets, computers, machines. The authoritative list, with each table's natural
key, is `IMPORTABLE` in `shopdb/core/services/csvimport.py`.
You only need the files you have; skip any you do not care about. Use `--file`
with `--table` for a single file out of order.
**User accounts are deliberately not importable.** Passwords do not belong in a
spreadsheet, in either direction. Create the first administrator through the
first-run page and the rest in the application.
## The templates are generated, not maintained
`flask csv templates` builds them from the live database schema each time, so
every column offered exists, every required one is marked, and every foreign key
says which file it refers to. Do not hand-maintain a template set: the one that
was tried had invented columns on seven of eleven tables and named a table that
does not exist, while looking entirely plausible. A test fails the build if the
generated set ever drifts from the schema.
## Editing the files
- **UTF-8**, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
- Lines starting with `#` are ignored, so the notes at the top of each template
can stay where they are.
- **The example rows are not comments. Delete them.** Each template carries a
worked example set as LIVE rows, so that a freshly downloaded folder imports
cleanly as a demo. Import it with your own data still in it and you get
`PC-001`, `CNC-014`, `Dell` and `Bay 3` in your database. The template says so
in its own header.
- Booleans are `1` or `0`.
- Dates are `YYYY-MM-DD` (`2026-08-04`). `YYYY-MM-DD HH:MM:SS` also works, as do
`DD/MM/YYYY` and `MM/DD/YYYY`.
- Leave a cell **empty** for "no value". Not `NULL`, not `N/A`.
- Quote anything containing a comma: `"Bay 3, North"`.
## If it will not run
**"the 'assets' table does not exist in this database"** - the schema has not
been created. Run `flask db upgrade` first, and check `DATABASE_URL` points at
the site you meant.
**"unknown column(s): ..."** - a column that does not exist, usually from an
older template. Regenerate with `flask csv templates`; the message lists what
the table does accept.
**"required column(s) missing: ..."** - a column that must be present has been
deleted from the header. Regenerate and copy your data across.

108
docs/DEPENDENCY-SECURITY.md Normal file
View File

@@ -0,0 +1,108 @@
<!-- docmeta
purpose: howto
audience: developer
status: current
summary: Answer "are we exposed to this advisory" from the shipped SBOM, and what can and cannot fail a build.
loadwith:
-->
# Dependency security
Two scripts, one question each.
| question | script | output |
|---|---|---|
| what are we running | `scripts/generate_sbom.py` | `sbom.cdx.json`, CycloneDX 1.6, shipped with every release |
| are we exposed | `scripts/scan_sbom.py` | a report, and an exit code |
The SBOM was already generated and installed onto every server. Nothing read
it, so "are we exposed to this advisory" was still a person going and looking.
The scanner closes that.
## Running it
```bash
venv/bin/python scripts/scan_sbom.py # report, exit 0
venv/bin/python scripts/scan_sbom.py --gate # exit 1 on a shipped finding
venv/bin/python scripts/scan_sbom.py --json out.json # machine-readable too
```
With no `--sbom`, it generates one first, so there is never a stale input.
It needs outbound HTTPS to `api.osv.dev`. **Run it where the repository lives,
not on a customer's server** - a deployed instance is frequently air-gapped and
has nothing to query. That is also why the SBOM ships as a file: it is the
artefact a security team scans on their own terms.
## Why OSV
It takes a purl and returns the advisories that apply. No account, no API key,
no agent installed anywhere, no vulnerability database to keep current. The
SBOM is already the input it wants, so this is a lookup against what shipped
rather than a second scan of the tree that could disagree with it.
## Shipped and build-only are not the same finding
Two thirds of the components in the SBOM are build tooling - bundlers, test
runners, their transitive dependencies. They never reach a server, and the SBOM
already marks them `scope: excluded`.
Both are reported. **Only shipped components can fail a build.**
This is deliberate. At the time of writing, the highest-severity match in the
whole tree is a CRITICAL against the test runner, which is not installed
anywhere a release reaches. A gate that fails on that teaches everybody to pass
`--no-verify` within a month, and then it catches nothing at all.
## Severity is taken, not computed
Where an advisory records a plain word - GHSA gives LOW, MODERATE, HIGH,
CRITICAL - that word is used. Where there is only a CVSS vector, the report says
UNKNOWN rather than parsing the vector into a score it might get wrong.
UNKNOWN is shown and never gates. A guessed score is worse than an honest gap,
because somebody would act on it.
## The allowlist is a record, not a mute button
`scripts/vulnerability-allowlist.json` holds advisories that were reviewed and
accepted. An entry carries the advisory id, the reason, who accepted it, the
date, and what would invalidate the reasoning.
A good reason names the precondition that is absent. The file has held exactly
one entry so far, and how it ended is the point.
A PyJWT advisory rated HIGH - algorithm confusion, forged HS256 tokens - needs a
verifier configured with both symmetric and asymmetric algorithms AND a raw JSON
JWK passed as the key. This application configures neither: there is no
`JWT_ALGORITHM` or `JWT_DECODE_ALGORITHMS` anywhere in the tree, so
flask-jwt-extended runs HS256-only against a string secret. That was written
down, with `invalidatedby` recording what would make it untrue - SSO issuing
tokens verified against a JWK.
Then PyJWT was upgraded, the advisory stopped matching, and the entry was
deleted. The allowlist is empty again, which is the state to aim for.
`invalidatedby` is the field that earns its keep. A reason stops being true when
the code that made it true changes, and nobody re-reads an allowlist looking for
that. Write down what would break it.
Upgrading is almost always cheaper than defending a decision to skip it. Do not
add an entry to make a build pass.
## What the report tells you to do
The report ends with a per-component upgrade rollup: the smallest upgrade that
clears every advisory against that component, and how many it clears.
"Smallest" is doing real work there. An advisory patched on several release
branches lists a fix per branch - nanoid's is fixed in both 3.3.18 and 5.1.16 -
and taking the highest recommends a major upgrade when a patch on the current
branch would do. Where the package is transitive that major may not even be
reachable: nanoid arrives under postcss, pinned `^3.3.11`, a range 5.x cannot
satisfy. So the target is the lowest published fix above the version in use.
Fixes on older branches are ignored rather than offered as a downgrade.
A target on a different major version is still labelled `MAJOR`, but now that
label means something: no fix exists on the current branch, so the upgrade is
genuinely a different job - it can be ESM-only, or drop an API.

180
docs/DEPLOY-AIRGAP.md Normal file
View File

@@ -0,0 +1,180 @@
<!-- docmeta
purpose: howto
audience: adopter, developer
status: current
summary: Build the stack on a connected box, carry one tarball in, and run a Docker site with no internet at all.
loadwith: none
-->
# Air-gapped Docker deploy
> **Topology: Docker, at a site with no internet at all.** The connected
> equivalent is [DEPLOY](DEPLOY.md).
> **Where this sits:** **install** -> operate -> upgrade.
> Later versions: [Upgrading to a new version](#upgrading-to-a-new-version) on
> this page, not [UPGRADE](UPGRADE.md), which assumes a checkout to pull.
The site cannot `pip install`, `npm ci`, or `docker pull`, so nothing is built
or pulled there. You build a fully self-contained image on a **connected** box,
ship it as one tarball, and `docker load` + run it at the site.
`docker-compose.airgap.yml` is the counterpart to `docker-compose.yml` and
differs in four ways that matter:
- **`image:` not `build: .`** - the site runs a loaded image; it never builds.
- **No `./plugins` bind mount** - the image already carries every plugin baked
in. Binding a host `./plugins` (which does not exist at an image-only site)
would mask the baked plugins with an empty dir and load **zero** plugins.
- **A one-shot `migrate` service** runs `db upgrade` + `plugin upgrade-all` +
seed before `api` starts, so `up -d` alone brings up a working site.
- **No `./certs` mount and no TLS flags** - `api` runs the image's own gunicorn
command and serves plain http on 5001. Put TLS in front of it, or in the
container the way `docker-compose.yml` does.
**A ready-made package exists too.** `scripts/build-channel-installer.ps1`
wraps this same tarball with `docker-compose.airgap.yml`, a pre-filled `.env`
and an `Install-ShopDB.ps1`, for an operator who has no repository at all. Read
the script's own header for what it expects on the build box.
---
## 1. Build the bundle (connected box)
On a box with clean access to Docker Hub + PyPI + npm, from the repo root:
```powershell
pwsh scripts/build-offline-bundle.ps1 -Version <version>
```
This builds `shopdb-flask:<version>` (frontend + all Python deps baked in), pulls
`mysql:8.4`, and writes:
- `shopdb-stack-<version>.tar.gz` - both images in one archive
- `shopdb-stack-<version>.tar.gz.sha256` - checksum to verify after transfer
The image is self-contained: **no pip/npm/registry access is needed at the
site.**
### Building behind an inspecting proxy
If the build box is itself behind an inspecting corporate proxy, the in-build
`pip`/`npm` fails with `CERTIFICATE_VERIFY_FAILED`: the build container has its
own certificate store, and the host's `PIP_CERT` / `NODE_EXTRA_CA_CERTS` do NOT
carry into a `docker build`. Easiest fix is to build from a box with clean
internet. Otherwise the corporate root CA must be trusted **inside** the build,
which is a Dockerfile change and is not wired today - see
[DEVELOPMENT-SETUP](DEVELOPMENT-SETUP.md) section 0b for the all-roots PEM
bundle.
---
## 2. Transfer + verify
Carry `shopdb-stack-<version>.tar.gz` (+ the `.sha256`), plus
`docker-compose.airgap.yml` and `.env.example`, to the site on approved media.
Verify the archive survived the trip:
```powershell
# PowerShell
(Get-FileHash shopdb-stack-<version>.tar.gz -Algorithm SHA256).Hash.ToLower()
# compare against the .sha256 file
```
```bash
# Linux site
sha256sum -c shopdb-stack-<version>.tar.gz.sha256
```
---
## 3. Load + run (air-gapped site)
```bash
# 1) Load both images into the local Docker.
docker load -i shopdb-stack-<version>.tar.gz
docker image ls | grep -E 'shopdb-flask|mysql' # confirm both present
# 2) Configure the site.
cp .env.example .env
# Edit .env - REQUIRED:
# IMAGE_TAG=<version> # MUST match the loaded image tag
# MYSQL_ROOT_PASSWORD=... # strong, unique
# MYSQL_PASSWORD=... # strong, unique (the app's db user)
# SECRET_KEY=... # 32+ random bytes
# JWT_SECRET_KEY=... # 32+ random bytes, different from SECRET_KEY
# CORS_ORIGINS=https://shopdb.site.example # the site's browser origin(s)
# Optional: API_PORT (default 5001), LOG_LEVEL, ZABBIX_URL/ZABBIX_TOKEN.
# 3) Bring it up. The migrate one-shot runs the schema + seed, then api starts.
docker compose -f docker-compose.airgap.yml up -d
# 4) Watch the one-shot finish (it exits 0 when the schema + seed are done).
docker compose -f docker-compose.airgap.yml logs -f migrate
```
`IMAGE_TAG` in `.env` must equal the loaded tag; otherwise compose looks for an
image that was never loaded and `api`/`migrate` will not start.
`docker-compose.airgap.yml` falls back to `0.7.0` when `IMAGE_TAG` is unset,
which is almost certainly not the version you just built - so set it, rather
than relying on the default.
---
## 4. Create the first admin
Seeding creates permissions, settings, and reference data, but not a login. Make
one admin (choose the password; it is not automatable):
```bash
docker compose -f docker-compose.airgap.yml exec api \
flask seed admin --username <username> --email <email> --password <password>
# Omit --password to have a strong one generated and printed once.
```
---
## 5. Verify
```bash
docker compose -f docker-compose.airgap.yml ps # db + api "running", migrate "exited (0)"
docker compose -f docker-compose.airgap.yml logs api # gunicorn started, no tracebacks
curl -sf http://localhost:${API_PORT:-5001}/api/dashboard/health # or browse the site origin
```
Then log in at the site origin with the admin created in step 4.
---
## Upgrading to a new version
Build a new bundle on the connected box (`-Version <newversion>`), transfer, then at the
site:
```bash
docker load -i shopdb-stack-<newversion>.tar.gz
# set IMAGE_TAG=<newversion> in .env
docker compose -f docker-compose.airgap.yml up -d
```
The `migrate` one-shot re-runs `db upgrade` + `plugin upgrade-all` + seed (all
idempotent) against the existing data before the new `api` starts. Back up first
([BACKUP-RESTORE](BACKUP-RESTORE.md)); the `db_data` volume persists across
upgrades.
---
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| `service api is not running` / `manifest ... not found` | The image was not loaded, or `IMAGE_TAG` in `.env` does not match a loaded image. `docker image ls`, fix `IMAGE_TAG`. Also: never use `docker-compose.yml` here - its `build: .` needs internet. |
| Site loads but **no plugins / empty nav** | You used the wrong compose file. `docker-compose.yml` bind-mounts `./plugins` (absent here) over the baked plugins. Use `docker-compose.airgap.yml`. |
| `api` never starts, `migrate` shows an error | Read `logs migrate`. A DB-connection error means `db` is not healthy yet (`logs db`) or `MYSQL_PASSWORD` in `.env` differs from what the `db` volume was first initialised with. A fresh site with a stale `db_data` volume needs the volume removed (`docker compose ... down -v` - DESTROYS data). |
| `SECRET_KEY must be set` (and similar) on `up` | A required `.env` var is empty. Fill every REQUIRED key in step 3. |
| Build fails on the connected box at `pip`/`npm` | Proxy certificate - see "Building behind an inspecting proxy" above. |
## See also
- [DEPLOY](DEPLOY.md) - the connected stack, the environment variables, and the
reverse-proxy and client-address reference
- [BACKUP-RESTORE](BACKUP-RESTORE.md) - back up before every image swap
- [CONFIG](CONFIG.md) - what each `.env` value means

Some files were not shown because too many files have changed in this diff Show More