Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0ded9207ab | ||
|
|
581baafee0 | ||
|
|
a97346ed81 | ||
|
|
762d1b7210 | ||
|
|
6c00b10618 | ||
|
|
32d3f4b29d | ||
|
|
9d6342b3ab | ||
|
|
e685852c2e | ||
|
|
b108b985ea | ||
|
|
c4d8542888 | ||
|
|
70f10c6fa0 | ||
|
|
bf6b353b52 | ||
|
|
ecdbe53ef6 | ||
|
|
534f179ee6 | ||
| 98398f7839 | |||
| cb3975049c | |||
| fd513c08eb | |||
| cf95daf3bb | |||
| 8ad6511f14 | |||
| 088648011f | |||
| 1466f9c3b0 |
@@ -17,7 +17,3 @@ venv
|
||||
.gitignore
|
||||
*.log
|
||||
.env.local
|
||||
|
||||
# Installer build artifacts (not part of the app image)
|
||||
installer-arch-dev/build
|
||||
installer-arch-dev/dist
|
||||
|
||||
22
.dockerignore.prebuilt
Normal file
@@ -0,0 +1,22 @@
|
||||
# Dockerignore for pre-built frontend approach
|
||||
# frontend/dist is INCLUDED (not ignored) for this build
|
||||
|
||||
**/node_modules
|
||||
frontend/src/.plugins-staged
|
||||
frontend/src/router/routes.gen.js
|
||||
|
||||
# Python caches
|
||||
**/__pycache__
|
||||
**/*.pyc
|
||||
.venv
|
||||
venv
|
||||
|
||||
# Local dev and VCS
|
||||
.git
|
||||
.gitignore
|
||||
*.log
|
||||
.env.local
|
||||
|
||||
# Installer artifacts
|
||||
installer-arch-dev/build
|
||||
installer-arch-dev/dist
|
||||
61
.env.example
@@ -4,7 +4,11 @@
|
||||
# refuses to boot when SECRET_KEY, JWT_SECRET_KEY, DATABASE_URL, or
|
||||
# CORS_ORIGINS are missing or use the dev defaults.
|
||||
#
|
||||
# See the DEPLOY wiki page for the full per-site deployment runbook.
|
||||
# This template covers what a container deploy sets. Docker is the shipping
|
||||
# path for a receiving site: docker-compose.yml builds from source,
|
||||
# docker-compose.pull.yml runs a published image, and
|
||||
# docker-compose.airgap.yml runs an image loaded from a tarball. See the
|
||||
# DEPLOY wiki page for the full per-site runbook.
|
||||
|
||||
# ---- Flask ----
|
||||
FLASK_APP=wsgi.py
|
||||
@@ -37,11 +41,12 @@ DATABASE_URL=mysql+pymysql://shopdb:CHANGE_ME@db:3306/shopdb_flask?charset=utf8m
|
||||
CORS_ORIGINS=http://localhost:5173
|
||||
|
||||
# ---- Reverse proxy (optional, default 0 = no proxy) ----
|
||||
# Set to the number of proxies actually in front of this instance - 1 when IIS
|
||||
# or nginx serves it under a subpath such as /shopdb. Without it the app sees
|
||||
# the proxy's connection instead of the browser's (scheme http, the proxy's
|
||||
# host and IP), so any absolute URL it builds is wrong at the browser. That is
|
||||
# what made a redirect on the endpoints list unfollowable behind IIS.
|
||||
# Set to the number of proxies actually in front of this instance - 1 when
|
||||
# nginx or another front end terminates TLS for the container. Without it the
|
||||
# app sees the proxy's connection instead of the browser's (scheme http, the
|
||||
# proxy's host and IP), so any absolute URL it builds is wrong at the browser.
|
||||
# That is what made a redirect on the endpoints list unfollowable behind a
|
||||
# proxy.
|
||||
#
|
||||
# Leave it at 0 when nothing fronts the app. It makes X-Forwarded-* trusted,
|
||||
# and a client can send those headers itself - the login rate limit and the
|
||||
@@ -56,15 +61,22 @@ JWT_REFRESH_TOKEN_EXPIRES=2592000
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# ---- docker-compose only ----
|
||||
# These are read by docker-compose.yml; not used by the Flask app directly.
|
||||
# These are read by the compose files; not used by the Flask app directly.
|
||||
MYSQL_ROOT_PASSWORD=CHANGE_ME_ROOT_PASSWORD
|
||||
MYSQL_PASSWORD=CHANGE_ME_APP_PASSWORD
|
||||
MYSQL_PORT=3306
|
||||
API_PORT=5001
|
||||
# Air-gapped deploy only (docker-compose.airgap.yml): the loaded image tag,
|
||||
# which MUST match what build-offline-bundle.ps1 -Version produced. Ignored by
|
||||
# the connected build template (docker-compose.yml builds from source).
|
||||
# the connected build template (docker-compose.yml builds from source) and by
|
||||
# the pull template (which uses SHOPDB_IMAGE instead).
|
||||
IMAGE_TAG=0.7.0
|
||||
# Pull deploy only (docker-compose.pull.yml): the full image reference to pull.
|
||||
# SHOPDB_IMAGE=ghcr.io/<org>/shopdb-flask:stable
|
||||
#
|
||||
# Non-empty lights a channel badge in the sidebar. Leave unset on a real site;
|
||||
# the dev stacks set it to DEV.
|
||||
# SHOPDB_ENV_LABEL=
|
||||
|
||||
# ---- Zabbix integration (optional, for printer supply monitoring) ----
|
||||
ZABBIX_URL=
|
||||
@@ -89,36 +101,25 @@ ZABBIX_TOKEN=
|
||||
# (flask plugin stamp-bundled) and pinning keys above.
|
||||
# PLUGIN_REQUIRE_SIGNED=false
|
||||
#
|
||||
# Dev-only: directories whose UNSIGNED plugins are trusted, honored ONLY under
|
||||
# DEBUG/TESTING (external-repo/symlink dev). Production ignores this.
|
||||
# PLUGIN_DEV_TRUST_DIRS=/home/dev/my-plugin-repo
|
||||
#
|
||||
# Read-only folder the app pulls plugin artifacts from (a SharePoint-synced or
|
||||
# copied shelf). The app reads this folder; it never speaks any network.
|
||||
# PLUGIN_SHELF_DIR=/srv/shopdb/plugin-shelf
|
||||
|
||||
# ---- Employee directory database (optional, read-only) ----
|
||||
# Separate HR/employee lookup DB consumed by the notifications plugin and the
|
||||
# public shopfloor kiosks. Leave unset if the feature is not used; there is no
|
||||
# safe default for the password, so an unset password fails loud.
|
||||
# Separate HR/employee lookup DB. Core reads it in two places: the employee
|
||||
# domain of global search (shopdb/core/api/search.py) and audit-log name
|
||||
# resolution (shopdb/core/api/auditlogs.py). Leave unset if the site has no
|
||||
# such database; there is no safe default for the password, so an unset
|
||||
# password fails loud rather than connecting as someone else.
|
||||
# EMPLOYEE_DB_HOST=
|
||||
# EMPLOYEE_DB_USER=
|
||||
# EMPLOYEE_DB_PASSWORD=
|
||||
# EMPLOYEE_DB_NAME=wjf_employees
|
||||
|
||||
# ---- cmmc_usb database (USB check-in/out) ----
|
||||
# Separate read-write MySQL DB used by the USB plugin to track device
|
||||
# check-in/out, lockers, and the check-in/out log. Leave unset if the feature
|
||||
# is not used; there is no safe default for the password, so an unset password
|
||||
# fails loud.
|
||||
# CMMC_USB_DB_HOST=
|
||||
# CMMC_USB_DB_USER=
|
||||
# CMMC_USB_DB_PASSWORD=
|
||||
# CMMC_USB_DB_NAME=cmmc_usb
|
||||
# EMPLOYEE_DB_NAME=employees
|
||||
|
||||
# ---- Subpath deployment (optional) ----
|
||||
# Serve the app under a URL prefix instead of the server root, e.g. as an IIS
|
||||
# Application at /ops under an existing site. The frontend must be rebuilt with
|
||||
# the matching base: VITE_BASE_PATH=/ops/ npm run build. Leave unset when the
|
||||
# app owns its own site/port (the default). See the INSTALL-WINDOWS-IIS wiki page.
|
||||
# Serve the app under a URL prefix instead of the server root, e.g. behind an
|
||||
# nginx location at /ops. wsgi.py reads this to mount the WSGI app under the
|
||||
# prefix. The frontend must be rebuilt with the matching base:
|
||||
# VITE_BASE_PATH=/ops/ npm run build. Leave unset when the app owns its own
|
||||
# host/port, which is the container default.
|
||||
# MOUNT_PATH=/ops
|
||||
|
||||
6
.gitattributes
vendored
@@ -4,9 +4,3 @@
|
||||
*.sh text eol=lf
|
||||
scripts/check-naming-and-style.sh text eol=lf
|
||||
.githooks/pre-commit text eol=lf
|
||||
|
||||
# Batch files must be CRLF. cmd.exe parses a lone LF inconsistently around
|
||||
# labels and goto, so an LF-only .cmd can run its happy path and then jump
|
||||
# nowhere on the error path - which is the path that matters in a promote
|
||||
# script.
|
||||
*.cmd text eol=crlf
|
||||
|
||||
5
.github/copilot-instructions.md
vendored
@@ -67,6 +67,11 @@ name. Spell out: `config context manager request response environment utilities`
|
||||
`migrations/versions/`; each plugin owns its own chain under
|
||||
`plugins/<name>/migrations/`. New schema = a new migration with an idempotent
|
||||
guard and a real downgrade. Migrations must run clean on strict MySQL 8.
|
||||
Any model column/table change ships WITH its migration in the SAME change, and
|
||||
bringing up a new image MUST run `flask db upgrade` then
|
||||
`flask plugin upgrade-all` before serving traffic. A model that references a
|
||||
column its migration has not applied yet 500s every query on that table
|
||||
(`1054 Unknown column`), so code and schema must never be deployed out of step.
|
||||
- **Asset model is the platform contract.** Physical things are an `Asset` plus a
|
||||
plugin subtype row linked by `assetid` (FK, `ON DELETE CASCADE`). Consumables
|
||||
with quantities are standalone tables, not assets.
|
||||
|
||||
122
.github/workflows/ci.yml
vendored
@@ -6,11 +6,12 @@
|
||||
# gates - proves those checks actually RAN rather than skipping
|
||||
# frontend - vitest + Vue build
|
||||
# lean-build - ADR-013: a per-site build carries only its plugins
|
||||
# migrations-mysql - the REAL multi-site deploy path: fresh flask db upgrade
|
||||
# + every plugin's chain on utf8mb4 MySQL 8, idempotent on
|
||||
# a second run. The pytest suite only exercises SQLite
|
||||
# create_all(), so this is what catches an Alembic
|
||||
# regression on MySQL before it ships.
|
||||
# api-contract - the published spec matches the code that serves it
|
||||
#
|
||||
# The MySQL-8 migration job is NOT here; it runs on the internal CI server. See
|
||||
# the note at the bottom of this file. It was listed here for a while after it
|
||||
# moved, which reads as coverage this workflow does not have - do not re-add it
|
||||
# to this list without re-adding the job to a runner that can host it.
|
||||
|
||||
name: CI
|
||||
|
||||
@@ -53,6 +54,60 @@ jobs:
|
||||
- uses: actions/checkout@v5
|
||||
- run: bash scripts/check-naming-and-style.sh
|
||||
|
||||
api-contract:
|
||||
# The published API contract has to match the code that serves it. These
|
||||
# gates catch a route that exists and is undocumented, a query parameter the
|
||||
# handler reads that the spec omits, and a public route nobody declared
|
||||
# public. They already work; until now nothing ran them except by hand, and
|
||||
# the spec had silently drifted by three operations.
|
||||
runs-on: arc-runner-set
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.14'
|
||||
cache: pip
|
||||
- run: pip install -r requirements-dev.txt
|
||||
- run: |
|
||||
python -m pytest -q \
|
||||
tests/test_api_inventory_parity.py \
|
||||
tests/test_openapi_spec.py \
|
||||
tests/test_openapi_examples.py \
|
||||
tests/test_core/test_public_read_surface.py
|
||||
- name: Seed the plugin registry the generator needs
|
||||
# gen_openapi.py refuses to run while a bundled plugin is disabled, and
|
||||
# instance/plugins.json is gitignored, so a clean checkout has none.
|
||||
# The step used to pass only because the pytest step above writes that
|
||||
# file at import time and never removes it - green by accident of step
|
||||
# order, and red the moment the steps are reordered, with an error
|
||||
# about disabled plugins that has nothing to do with the spec.
|
||||
run: |
|
||||
python - <<'PY'
|
||||
import datetime, json, pathlib
|
||||
names = sorted(d.name for d in pathlib.Path('plugins').iterdir()
|
||||
if (d / 'manifest.json').is_file())
|
||||
now = datetime.datetime.now().isoformat()
|
||||
out = pathlib.Path('instance'); out.mkdir(exist_ok=True)
|
||||
(out / 'plugins.json').write_text(json.dumps({'plugins': {
|
||||
n: {'name': n, 'version': '1.0.0', 'installed_at': now,
|
||||
'enabled': True, 'migrations_applied': [], 'config': {}}
|
||||
for n in names}}, indent=2))
|
||||
print('registry seeded with', len(names), 'plugins')
|
||||
PY
|
||||
|
||||
- name: The spec must be regenerated, not just passing
|
||||
# --refresh is derived from the code, so if it changes anything the
|
||||
# committed spec was stale. Fail here rather than let a generated client
|
||||
# be built from a file that no longer describes the API.
|
||||
run: |
|
||||
python scripts/gen_openapi.py --refresh
|
||||
if ! git diff --quiet docs/openapi.json docs/api-inventory.json; then
|
||||
echo "docs/openapi.json is stale. Run:"
|
||||
echo " python scripts/gen_openapi.py --refresh"
|
||||
git --no-pager diff --stat docs/openapi.json docs/api-inventory.json
|
||||
exit 1
|
||||
fi
|
||||
|
||||
gates:
|
||||
# Prove the gates RAN. Several skip themselves when the file they read is
|
||||
# absent, which is correct in this checkout (docs/ becomes the wiki) and was
|
||||
@@ -88,9 +143,31 @@ jobs:
|
||||
- run: npm run build
|
||||
|
||||
lean-build:
|
||||
# ADR-013 Phase 5: prove a per-site build carries only its chosen plugins.
|
||||
# Builds a lean site (machines + printers) and asserts an omitted plugin's
|
||||
# code is absent from the bundle - the delete-a-plugin guarantee in CI.
|
||||
# ADR-013: prove a per-site build carries only its chosen plugins.
|
||||
#
|
||||
# The split below is SYNTHETIC, and deliberately so. Every plugin in this
|
||||
# repository ships on this line, so there is no "omitted" set to borrow from
|
||||
# a real site - and a job that iterated an empty omitted list would assert
|
||||
# nothing while staying green, which is the exact failure the omission
|
||||
# guarantee exists to catch. So the profile names a genuine SUBSET of the
|
||||
# bundled plugins and the assertions prove the SUBSETTING MECHANISM: what
|
||||
# the profile chooses is staged, and what it leaves out is not.
|
||||
#
|
||||
# Chosen (7): backups computers endpoints machines network notifications
|
||||
# printers
|
||||
# Omitted (5): guides knowledgebase slides tools warranty
|
||||
#
|
||||
# The omitted five are picked because nothing depends on them, so the
|
||||
# closure stays exactly the chosen set. 'applications' is manifest-less core
|
||||
# and ships in every build (ADR-013). endpoints hard-depends on computers,
|
||||
# which is already chosen, so the closure adds nothing - if that changes,
|
||||
# resolve_plugin_closure.py pulls the dependency in and the absence
|
||||
# assertions below start failing, which is the signal to update this list
|
||||
# rather than to loosen the check.
|
||||
#
|
||||
# Verified by planting a leak, per the standing rule that a green gate may
|
||||
# be vacuous: adding 'guides' to the profile and rebuilding makes both the
|
||||
# GuidesList assertion and the plugins/guides directory assertion fail.
|
||||
runs-on: arc-runner-set
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
@@ -100,21 +177,29 @@ jobs:
|
||||
cache: npm
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
- run: cd frontend && npm ci
|
||||
- name: Build lean site (machines + printers)
|
||||
- name: Build lean site (seven-plugin subset)
|
||||
run: |
|
||||
printf '{ "site": "ci-lean", "plugins": ["machines", "printers"] }' \
|
||||
printf '%s' '{ "site": "ci-lean", "plugins": ["backups", "computers", "endpoints", "machines", "network", "notifications", "printers"] }' \
|
||||
> /tmp/lean-profile.json
|
||||
bash scripts/build-site.sh /tmp/lean-profile.json /tmp/leansite
|
||||
- name: Assert omitted plugin code is absent, chosen present
|
||||
run: |
|
||||
assets=/tmp/leansite/frontend-dist/assets
|
||||
for code in PartsKiosk ManifestEditor USBLabelBatch KnowledgeBaseDetail; do
|
||||
# One component per OMITTED plugin: guides, knowledgebase, slides,
|
||||
# tools, warranty. Each name is proven to exist by the chosen-plugin
|
||||
# loop on a full build, so these are known-good grep targets rather
|
||||
# than strings that would pass by being misspelled.
|
||||
for code in GuidesList KnowledgeBaseDetail SlideManager ToolsIndex \
|
||||
WarrantiesList; do
|
||||
if grep -rqoh "$code" "$assets"/*.js; then
|
||||
echo "FAIL: omitted-plugin code '$code' leaked into the lean bundle"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
for code in MachineDetail PrinterDetail; do
|
||||
# One component per CHOSEN plugin, so a plugin silently dropped from
|
||||
# the bundle fails here instead of on the floor.
|
||||
for code in BackupHistory PCsList EndpointsList MachineDetail \
|
||||
NetworkDevicesList NotificationsList PrinterDetail; do
|
||||
grep -rqoh "$code" "$assets"/*.js || {
|
||||
echo "FAIL: chosen-plugin code '$code' missing from the lean bundle"
|
||||
exit 1; }
|
||||
@@ -124,8 +209,17 @@ jobs:
|
||||
grep -rqoh "ApplicationsList" "$assets"/*.js || {
|
||||
echo "FAIL: core page 'ApplicationsList' missing from the lean bundle"
|
||||
exit 1; }
|
||||
test -d /tmp/leansite/plugins/machines
|
||||
test ! -d /tmp/leansite/plugins/printedparts
|
||||
for plugin in backups computers endpoints machines network \
|
||||
notifications printers; do
|
||||
test -d "/tmp/leansite/plugins/$plugin" || {
|
||||
echo "FAIL: chosen plugin '$plugin' missing from the backend tree"
|
||||
exit 1; }
|
||||
done
|
||||
for plugin in guides knowledgebase slides tools warranty; do
|
||||
test ! -d "/tmp/leansite/plugins/$plugin" || {
|
||||
echo "FAIL: omitted plugin '$plugin' staged into the backend tree"
|
||||
exit 1; }
|
||||
done
|
||||
echo "lean build verified: only chosen plugins present"
|
||||
|
||||
# NOTE: the MySQL-8 migration/seed job (fresh `flask db upgrade` + every
|
||||
|
||||
12
.github/workflows/publish-images.yml
vendored
@@ -76,8 +76,18 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Set up Buildx
|
||||
# Connect to a persistent, out-of-cluster BuildKit deployment instead of
|
||||
# spinning up a builder against a local Docker daemon. The arc-runner-set
|
||||
# pods have no Docker daemon (no /var/run/docker.sock) and cannot run a
|
||||
# privileged DinD sidecar under cluster policy, so we point Buildx at a
|
||||
# remote buildkitd Service. The endpoint is provided as a repo/org secret
|
||||
# (e.g. tcp://buildkit.buildkit.svc.cluster.local:1234), so it is not
|
||||
# hardcoded here. See deploy/ci/buildkit-remote.yaml for the deployment.
|
||||
- name: Set up Buildx (remote BuildKit)
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
driver: remote
|
||||
endpoint: ${{ secrets.BUILDKIT_ENDPOINT }}
|
||||
|
||||
- name: Build and push channel image
|
||||
uses: docker/build-push-action@v6
|
||||
|
||||
1
.gitignore
vendored
@@ -83,7 +83,6 @@ secrets.yml
|
||||
*_secret
|
||||
*_secrets
|
||||
credentials.json
|
||||
scripts/site_imports/wjf/idmap.json
|
||||
|
||||
# work-PC publication clone: bundle drop folder for the transfer pipeline
|
||||
_transfer/
|
||||
|
||||
@@ -24,6 +24,24 @@
|
||||
- **Index names**: `idx_<table>_<column>`. Underscores allowed here only because indexes are infrastructure, not data.
|
||||
- Good: `idx_machines_locationid`
|
||||
|
||||
### Migrations and deploy ordering
|
||||
|
||||
- **A model change ships WITH its migration in the same change.** Adding or
|
||||
changing a column/table on a SQLAlchemy model without the matching Alembic
|
||||
migration is a broken change, even if it passes tests against a freshly built
|
||||
schema.
|
||||
- **Code and schema must never deploy out of step.** A model that references a
|
||||
column its migration has not applied yet makes MySQL raise
|
||||
`1054 Unknown column`, which 500s every query on that table (this is not a
|
||||
degraded read - the whole endpoint fails).
|
||||
- **Bringing up a new image MUST run migrations before serving traffic**, in
|
||||
order: `flask db upgrade` (core chain) then `flask plugin upgrade-all`
|
||||
(per-plugin chains). A rebuilt image with new model code but an
|
||||
un-migrated database will 500 until this runs.
|
||||
- Core schema lives in `migrations/versions/`; each plugin owns its own chain
|
||||
under `plugins/<name>/migrations/`. New schema = a new migration with an
|
||||
idempotent guard and a real downgrade, clean on strict MySQL 8.
|
||||
|
||||
### Allowed acronyms (closed list, expand by PR)
|
||||
|
||||
- **Universal**: `id`, `url`, `api`, `http`, `https`, `json`, `jwt`, `sql`, `os`, `ip`, `dns`, `csv`, `pdf`, `cors`, `ttl`, `uuid`, `html`, `css`, `orm`
|
||||
@@ -151,17 +169,60 @@ Tests are required for:
|
||||
- New plugin hooks (contract test)
|
||||
- Schema migrations (before/after data integrity)
|
||||
|
||||
## Adding or changing an API route
|
||||
|
||||
The published contract is generated from the code, so it goes stale the moment
|
||||
a route changes and nobody regenerates it. A stale spec is worse than no spec:
|
||||
it is what a client generator, an agent, or the MCP server in `tools/mcp/` will
|
||||
believe.
|
||||
|
||||
After adding or changing a route:
|
||||
|
||||
python scripts/gen_openapi.py --refresh
|
||||
|
||||
That rewrites `docs/api-inventory.json` and `docs/openapi.json` from the live
|
||||
app. It fills in what is derived and leaves prose you have written alone, so run
|
||||
it freely and then replace any seeded summary with the reason the route exists.
|
||||
|
||||
If it prints `review` against your route, it could not find a gate on the view.
|
||||
Do not hand-edit the inventory - the next refresh overwrites it. Either gate the
|
||||
view, or, if the route is genuinely public, name it in `PUBLIC_ENDPOINTS` in
|
||||
`tests/test_core/test_public_read_surface.py` with the reason. The generator
|
||||
reads that list, so the published contract and the enforced gate cannot
|
||||
disagree.
|
||||
|
||||
The `api-contract` CI job runs the parity gates and then regenerates and fails
|
||||
if anything changed, so a stale spec fails the pull request rather than being
|
||||
discovered months later.
|
||||
|
||||
## Branches: what belongs on main, and what does not
|
||||
|
||||
`main` is the product every site runs and the publication mirror exports.
|
||||
`<site>` branches hold work that is not ready for other sites to see. There are
|
||||
two: `wj` here, and `wilm` on the corporate GitHub, where the other site
|
||||
develops the .NET endpoint agent.
|
||||
`main` is the curated product other sites adopt, and what the publication
|
||||
mirror exports. It is a release decision: something reaching `main` says other
|
||||
sites should take it. **`main` is not what this site runs.**
|
||||
|
||||
Each branch answers to one instance:
|
||||
|
||||
| branch | runs on | contains |
|
||||
|---|---|---|
|
||||
| `dev` | the dev instance | work being validated |
|
||||
| `wj` | this site's PRODUCTION | everything this site runs |
|
||||
| `main` | nothing directly | the curated release for other sites |
|
||||
| `wilm` | the other site | their work, on the corporate GitHub |
|
||||
|
||||
So work flows `dev` -> `wj` -> `main`: land it on `dev`, validate it on the dev
|
||||
instance, take it to `wj` for production, and promote to `main` what you are
|
||||
happy for another site to run. Promoting first and back-filling the site branch
|
||||
is the wrong way round - it puts unvalidated work in front of other sites and
|
||||
leaves production behind.
|
||||
|
||||
`wilm` is not visible from the dev box because that machine has no account on
|
||||
the corporate GitHub, which is why `fetch-wilm-branch.cmd` exists to bring it
|
||||
back for review. It sits on the publication lineage, so it shares no ancestor
|
||||
with `main`/`wj`/`dev` and cannot be merged locally.
|
||||
|
||||
A site branch is not a fork. Both feed `main`, and the sooner they do it the
|
||||
cheaper it is. A branch the dev box cannot see is still a branch: `wilm` is not
|
||||
visible from here because this machine has no GitHub account, which is why
|
||||
`fetch-wilm-branch.cmd` on the share exists to bring it back for review.
|
||||
cheaper it is.
|
||||
|
||||
**Most work is product.** ADR-015 gives site differences three homes on `main`:
|
||||
a setting with a neutral default, seed data, or a site-namespaced directory
|
||||
|
||||
42
Dockerfile
@@ -1,13 +1,19 @@
|
||||
# shopdb-flask single-site container.
|
||||
#
|
||||
# One image, one site. Per ADR-004, each adopting facility runs its own
|
||||
# stack with its own DB, secrets, and enabled-plugin list. This image
|
||||
# bundles all 13 catalog plugins (computers, employees, geenforce,
|
||||
# knowledgebase, machines, measuringtools, network, notifications,
|
||||
# printedparts, printers, slides, usb, warranty); a site installs + enables
|
||||
# the ones it wants with `flask plugin install <name>` (or, declaratively,
|
||||
# `flask plugin apply-profile <profile.json>`). Per ADR-013 a future lean
|
||||
# build stages only the chosen plugin directories into this image.
|
||||
# stack with its own DB, secrets, and enabled-plugin list. This image bundles
|
||||
# the whole catalog - backups, computers, endpoints, guides, knowledgebase,
|
||||
# machines, network, notifications, printers, slides, tools, warranty - and a
|
||||
# site installs + enables the ones it wants with `flask plugin install <name>`
|
||||
# (or, declaratively, `flask plugin apply-profile <profile.json>`).
|
||||
#
|
||||
# WHY no count here: a hand-typed number goes stale the moment a plugin is
|
||||
# added or dropped, and nothing in the test suite polices a count outside
|
||||
# docs/. The generated PROJECT-MAP wiki page is the authority on what ships.
|
||||
#
|
||||
# Per ADR-013 a lean build stages only the chosen plugin directories into this
|
||||
# image; scripts/build-site.sh plus scripts/resolve_plugin_closure.py are that
|
||||
# mechanism.
|
||||
#
|
||||
# The frontend is built in a first stage and its dist output is copied into
|
||||
# the final image so Flask can serve the SPA (register_frontend_routes in
|
||||
@@ -118,6 +124,20 @@ COPY migrations/ ./migrations/
|
||||
COPY scripts/ ./scripts/
|
||||
COPY wsgi.py ./
|
||||
|
||||
# The two files under docs/ that the APP SERVES at runtime, not documentation.
|
||||
# shopdb/core/api/docs.py resolves both relative to the repo root, so without
|
||||
# them /api/docs renders the Redoc shell and then 404s fetching its own spec -
|
||||
# which is what every Docker deployment did. Copied by name rather than copying
|
||||
# docs/ wholesale: the rest of that tree is prose and has no business in a lean
|
||||
# runtime image.
|
||||
COPY docs/openapi.json docs/llms.txt ./docs/
|
||||
|
||||
# Fail the BUILD on a syntax error rather than the pod at runtime. Docker never
|
||||
# imports the application, so a broken .py sails through every layer and only
|
||||
# surfaces as a crash-looping container: observed, four restarts before anyone
|
||||
# noticed. compileall touches nothing at runtime and costs a couple of seconds.
|
||||
RUN python -m compileall -q shopdb plugins migrations wsgi.py
|
||||
|
||||
# Built SPA from stage 1. Flask serves it via register_frontend_routes.
|
||||
COPY --from=frontendbuild /build/frontend/dist ./frontend/dist
|
||||
|
||||
@@ -139,10 +159,16 @@ HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||
-H "Content-Type: application/json" -d '{}' \
|
||||
| grep -q "VALIDATION_ERROR" || exit 1
|
||||
|
||||
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
|
||||
# compose file overrides this CMD, but an image run without one (a plain
|
||||
# `docker run`, a site's own orchestrator) would otherwise still get the
|
||||
# blocking sync default and the same starvation.
|
||||
CMD ["gunicorn", \
|
||||
"--bind", "0.0.0.0:5001", \
|
||||
"--worker-class", "gthread", \
|
||||
"--workers", "4", \
|
||||
"--timeout", "60", \
|
||||
"--threads", "8", \
|
||||
"--timeout", "30", \
|
||||
"--access-logfile", "-", \
|
||||
"--error-logfile", "-", \
|
||||
"wsgi:app"]
|
||||
|
||||
47
Dockerfile.prebuilt
Normal file
@@ -0,0 +1,47 @@
|
||||
# Dockerfile for pre-built frontend
|
||||
# Use this when frontend/dist is already built on the host
|
||||
# Avoids npm issues with corporate proxy
|
||||
|
||||
FROM python:3.14-slim AS base
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
default-libmysqlclient-dev \
|
||||
pkg-config \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Python dependencies
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir -r requirements.txt \
|
||||
&& pip install --no-cache-dir gunicorn gevent
|
||||
|
||||
# Copy application code
|
||||
COPY shopdb/ ./shopdb/
|
||||
COPY plugins/ ./plugins/
|
||||
COPY migrations/ ./migrations/
|
||||
COPY scripts/ ./scripts/
|
||||
COPY wsgi.py ./
|
||||
|
||||
# Copy pre-built frontend
|
||||
COPY frontend/dist ./frontend/dist
|
||||
|
||||
# Make scripts executable
|
||||
RUN chmod +x /app/scripts/docker-entrypoint.sh /app/scripts/bootstrap-plugins.sh
|
||||
|
||||
# Create instance directory
|
||||
RUN mkdir -p /app/instance
|
||||
|
||||
# Expose port
|
||||
EXPOSE 5001
|
||||
|
||||
# Set environment
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV FLASK_APP=shopdb
|
||||
|
||||
# Entrypoint runs migrations and starts gunicorn
|
||||
ENTRYPOINT ["/app/scripts/docker-entrypoint.sh"]
|
||||
82
README.md
@@ -9,13 +9,13 @@ ShopDB tracks and manages:
|
||||
- **PCs** - Shopfloor computers, engineering workstations
|
||||
- **Printers** - Network printers with Zabbix supply integration
|
||||
- **Network devices** - Switches, routers, and the subnet browser
|
||||
- **Measuring tools** - Gage-lab instruments with calibration tracking
|
||||
- **Applications** - Software deployed across the shop floor, with per-PC install tracking
|
||||
- **Employees** - Directory, recognition and training notifications
|
||||
- **Warranties** - Coverage records with Dell warranty lookups
|
||||
- **USB devices** - CMMC check-in/out tracking
|
||||
- **Knowledge Base** - Documentation and troubleshooting guides
|
||||
- **GE-Enforce manifests** - Imaging/software manifest editing and fleet compliance
|
||||
- **Endpoints** - RMM telemetry reported by the ShopDB agent: metrics, services, inventory, Windows events
|
||||
- **Backups** - Per-asset configuration backups with revision history and diffs
|
||||
- **Warranties** - Coverage records with vendor warranty lookups
|
||||
- **Knowledge Base and Guides** - Reference articles and step-by-step procedures with working checklists
|
||||
- **Notifications** - Shopfloor announcements, planned changes and incident notices
|
||||
- **Slides** - Lobby display and shopfloor screensaver slideshows
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -108,12 +108,13 @@ SQLite). Do not run dev or production against SQLite.
|
||||
|
||||
### Distribution
|
||||
|
||||
Source lives here. The Windows installer ships as a release asset rather than a
|
||||
file in the repository - it is around 240 MB, well past what a repository takes
|
||||
and well inside what a release asset does. There is no package or image
|
||||
registry; a site installs from that `.exe` or builds from source.
|
||||
Source lives here, and a site stands up ShopDB with Docker. Three compose
|
||||
files cover the three situations: `docker-compose.yml` builds the image from
|
||||
this checkout, `docker-compose.pull.yml` runs an image published to a
|
||||
registry, and `docker-compose.airgap.yml` runs an image loaded from a tarball
|
||||
at a site with no internet.
|
||||
|
||||
### Fast path (Docker)
|
||||
### Standing up a site
|
||||
|
||||
The Docker image builds the Vue frontend and serves it from the API container,
|
||||
so a container deploy needs no separate Node build step.
|
||||
@@ -174,42 +175,19 @@ project wiki.
|
||||
To import a site's legacy data, use the HTTP import surface: an admin API
|
||||
token plus the IMPORT-API page in the project wiki drive the whole migration
|
||||
through documented endpoints (`X-Import-Mode` preserves original timestamps).
|
||||
`scripts/site_imports/wjf/` is the reference site's loader, kept as a worked
|
||||
example of the whole sequence.
|
||||
`scripts/site_imports/` holds the shared import machinery - the resolver, the
|
||||
idempotent journal, the approval packet builder and the WINC machine-adoption
|
||||
writer that the core import routes call.
|
||||
|
||||
### Which deployment route
|
||||
|
||||
| Target | Use |
|
||||
|---|---|
|
||||
| **Windows Server + IIS** (how sister sites run) | **the INSTALL-WINDOWS page in the project wiki** - one installer `.exe`, offline, no manual IIS work. Day 2: the OPERATE-WINDOWS page in the project wiki |
|
||||
| Linux / Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki |
|
||||
| Linux / Docker, connected | the DEPLOY page in the project wiki |
|
||||
| Docker, connected | the DEPLOY page in the project wiki (`docker-compose.yml` or `docker-compose.pull.yml`) |
|
||||
| Docker, air-gapped | the DEPLOY-AIRGAP page in the project wiki (`docker-compose.airgap.yml`) |
|
||||
|
||||
For every environment variable and Setting key see the CONFIG page in the project wiki.
|
||||
|
||||
### What the Windows installer requires
|
||||
|
||||
The installer checks all of this before it changes anything and refuses rather
|
||||
than half-installing.
|
||||
|
||||
| | Minimum | Notes |
|
||||
|---|---|---|
|
||||
| Windows Server | **2019** (build 10.0.17763) | Ships **IIS 10.0**, which is therefore the earliest IIS supported. Server 2016 is also IIS 10.0 but falls below the build floor and is refused. |
|
||||
| Windows client (test boxes) | **10 22H2** (build 10.0.19045) or 11 | **Pro or higher** - Home has no IIS at all. |
|
||||
| Architecture | 64-bit | The payload is cp314 win_amd64. |
|
||||
| IIS Web Server role | installed beforehand | `Install-WindowsFeature -Name Web-Server -IncludeManagementTools`. Needs no internet. |
|
||||
| Free disk | 5 GB | Refused below this. 40 GB is comfortable once uploads and config backups accumulate. |
|
||||
| Database | bundled MySQL 8.4 LTS, or your own | Bring credentials if you use an existing server. |
|
||||
|
||||
The binding constraint is the operating system rather than IIS: the payload is
|
||||
Python 3.14 plus HttpPlatformHandler and URL Rewrite, and the handler itself runs
|
||||
on older IIS. An older host is untested rather than known-broken.
|
||||
|
||||
If you must deploy onto something older, the INSTALL-WINDOWS-IIS page in the project wiki
|
||||
is the manual procedure and has no OS gate - it even documents MySQL 5.6, which is
|
||||
that era of machine. The trade is stated there: it produces a server the
|
||||
installer will not subsequently upgrade.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables (`.env`):
|
||||
@@ -235,7 +213,7 @@ The REST API follows standard conventions:
|
||||
| DELETE | `/api/machines/:id` | Soft delete machine |
|
||||
|
||||
Each asset plugin exposes the same CRUD pattern on its own prefix
|
||||
(`/api/computers`, `/api/printers`, `/api/network`, `/api/measuringtools`),
|
||||
(`/api/computers`, `/api/printers`, `/api/network`, `/api/machines`),
|
||||
and cross-cutting asset endpoints live under `/api/assets`.
|
||||
|
||||
Query parameters for list endpoints:
|
||||
@@ -244,7 +222,7 @@ Query parameters for list endpoints:
|
||||
- `sort` - Sort field
|
||||
- `dir` - Sort direction (asc/desc)
|
||||
- `search` - Search term
|
||||
- `assettype` - Filter by asset type (computer, printer, machine, networkdevice, measuringtool)
|
||||
- `assettype` - Filter by asset type (computer, printer, machine, networkdevice)
|
||||
|
||||
## Where to start
|
||||
|
||||
@@ -261,21 +239,21 @@ The image bundles these plugins; only the ones a site installs are loaded. The
|
||||
current count and each plugin's version and migration head are in
|
||||
the PROJECT-MAP page in the project wiki, which is generated:
|
||||
|
||||
- **backups** - Machine configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
|
||||
- **backups** - Per-asset configuration backups (NTLARS/DNC, CMM, part marker, UDC) with revision history and diffs
|
||||
- **computers** - Shopfloor PCs and workstations, collector fleet ingest
|
||||
- **employees** - Employee directory
|
||||
- **geenforce** - GE-Enforce imaging/software manifests and fleet compliance
|
||||
- **endpoints** - RMM telemetry from the ShopDB agent: metrics, services, inventory, Windows events
|
||||
- **guides** - Step-by-step procedures with images, working checklists and printable output
|
||||
- **knowledgebase** - Documentation and troubleshooting articles
|
||||
- **machines** - CNC, CMM, and other shop-floor machines
|
||||
- **measuringtools** - Gage-lab instruments with calibration tracking
|
||||
- **knowledgebase** - Documentation and troubleshooting guides
|
||||
- **network** - Network devices and subnets
|
||||
- **notifications** - Shopfloor notifications and recognition feed
|
||||
- **printedparts** - 3D-printed part catalogue, kiosk issue tracking and stock alerts
|
||||
- **notifications** - Shopfloor notifications. The employee-facing types (recognition, training, recertification) need the employees plugin, which this image does not bundle
|
||||
- **printers** - Extended printer management with Zabbix integration
|
||||
- **slides** - TV/kiosk slideshows
|
||||
- **tools** - Tech Tools: barcode and QR generation laid out for real label stock
|
||||
- **usb** - CMMC USB check-in/out tracking
|
||||
- **warranty** - Warranty records with Dell lookups
|
||||
- **slides** - Lobby display and shopfloor screensaver slideshows
|
||||
- **tools** - Tech Tools: barcode/QR label generation and one-view secure send
|
||||
- **warranty** - Warranty records with derived coverage status
|
||||
|
||||
Applications is not in that list because it is not a plugin: it is core, has no
|
||||
manifest, and always ships.
|
||||
|
||||
## Legacy Migration
|
||||
|
||||
|
||||
@@ -84,12 +84,23 @@ event channel and system WMI.
|
||||
## Tray app (system-tray icon)
|
||||
|
||||
The tray is a per-user process, not a service, so it starts at sign-in rather
|
||||
than under `sc.exe`. `install-shopdbagent.ps1` handles it: it copies
|
||||
`shopdb-endpoint-tray.exe` next to the agent, registers an HKCU logon autostart
|
||||
(`Software\Microsoft\Windows\CurrentVersion\Run\ShopDbEndpointTray`), and
|
||||
launches it for the current user. Run the installer from the operator's own
|
||||
session so the autostart lands in the right user hive. Pass `-NoTray` for a
|
||||
headless, service-only install.
|
||||
than under `sc.exe`. The generated `install.ps1` handles it: it copies
|
||||
`shopdb-endpoint-tray.exe` next to the agent and registers a machine-wide logon
|
||||
autostart (`HKLM\Software\Microsoft\Windows\CurrentVersion\Run\ShopDbEndpointTray`,
|
||||
64-bit view). The installer deliberately does NOT start the tray itself: it runs
|
||||
elevated, so any process it starts inherits an admin token, and mapped drives are
|
||||
per logon token - an elevated tray probes the wrong drive table. The Run key only
|
||||
fires at logon, which would leave the tray dead until sign-out on a machine
|
||||
someone is already using, so the installer also registers a transient
|
||||
`ShopDB tray first launch` scheduled task under `S-1-5-32-545` at `RunLevel
|
||||
Limited`, runs it once, and unregisters it. That task is the only mechanism that
|
||||
can reach into an existing interactive session without carrying the installer's
|
||||
token. Pass `-NoTray` for a headless, service-only install.
|
||||
|
||||
User-context collection does not depend on the tray. The installer registers a
|
||||
`ShopDB user-context probe` scheduled task against the Users group at
|
||||
`RunLevel Limited`, which runs `shopdbagent.exe --collect-usercontext` in the
|
||||
operator's own session at logon and every 30 minutes.
|
||||
|
||||
Right-click the tray icon for:
|
||||
|
||||
@@ -143,7 +154,7 @@ logic itself changes.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
- Rolling file log: `%ProgramData%\ShopDB\EndpointAgent\logs\agent-YYYYMMDD.log`
|
||||
- Rolling file log: `%ProgramData%\ShopDB\EndpointAgent\logs\YYYYMMDD-agent.log`
|
||||
(14-day retention).
|
||||
- Windows Event Log: agent runtime failures and unhandled exceptions are written
|
||||
to the `Application` log under source `ShopDB Endpoint Agent`. The source is
|
||||
|
||||
@@ -13,10 +13,40 @@ public sealed class AgentConfig
|
||||
[JsonPropertyName("apikey")] public string ApiKey { get; set; } = "";
|
||||
[JsonPropertyName("cadenceseconds")] public int CadenceSeconds { get; set; } = 900;
|
||||
|
||||
// Legacy pin: SHA-1 thumbprint of the server leaf certificate (hex, no
|
||||
// separators). Honoured with serverpins only while the registry ServerPins
|
||||
// value is absent; breaks on every cert renewal, so prefer an SPKI pin.
|
||||
// See AgentHttpFactory.
|
||||
[JsonPropertyName("servercertthumbprint")] public string? ServerCertThumbprint { get; set; }
|
||||
|
||||
// Server public key pins: lowercase hex SHA-256 of the leaf certificate's
|
||||
// SubjectPublicKeyInfo. Fallback only: HKLM\SOFTWARE\GE\ShopDB ServerPins
|
||||
// (REG_MULTI_SZ), when it exists in either view, replaces these entirely,
|
||||
// even when empty. Any pin in effect = the leaf key must match one; chain
|
||||
// and name errors are then ignored. None = normal Windows chain validation.
|
||||
[JsonPropertyName("serverpins")] public List<string> ServerPins { get; set; } = new();
|
||||
|
||||
// Jitter spreads a fleet's reports so they do not arrive in lockstep
|
||||
// (thundering herd). Actual delay = cadence +/- up to jitterseconds.
|
||||
[JsonPropertyName("jitterseconds")] public int JitterSeconds { get; set; } = 60;
|
||||
|
||||
// One-time random delay before FIRST contact, applied by both the telemetry
|
||||
// and command loops at startup. JitterSeconds only spreads the gap BETWEEN
|
||||
// runs, so without this every agent in the fleet still makes its first
|
||||
// request the instant its service starts - and the service is installed
|
||||
// start=auto, so a patch reboot starts them all together.
|
||||
//
|
||||
// Measured on a 700-endpoint burst (devnotes/gevent-sizing-simulation.md):
|
||||
// simultaneous arrival drove TLS handshakes to 1482ms worst case and pinned
|
||||
// CPU at 305% while the database sat idle at 3 active queries. Spreading the
|
||||
// same arrivals over 30s cut worst-case handshake to 63.7ms. The server is
|
||||
// not the constraint; simultaneity is.
|
||||
//
|
||||
// Costs up to this many seconds before a rebooted endpoint picks up a queued
|
||||
// command. That window is on top of a reboot the machine was already
|
||||
// unreachable for. 0 disables the delay.
|
||||
[JsonPropertyName("startupjitterseconds")] public int StartupJitterSeconds { get; set; } = 30;
|
||||
|
||||
[JsonPropertyName("adoptonfirstrun")] public bool AdoptOnFirstRun { get; set; } = true;
|
||||
|
||||
// Directory holding site-provided PowerShell "agent utilities" the runtime
|
||||
@@ -89,42 +119,52 @@ public sealed class AgentConfig
|
||||
return bare;
|
||||
}
|
||||
|
||||
// Per-name allowlist of pre-staged scripts for the generic run_script
|
||||
// action. Exactly like LaunchPaths, the wire never carries a script body or
|
||||
// path - only a script NAME the agent already holds here. A name absent
|
||||
// from this map fails closed. There are no shipped defaults: a script is a
|
||||
// site-specific artifact, so an empty map means run_script does nothing
|
||||
// until a site stages scripts and lists them in appsettings.json. Keyed by
|
||||
// script name (case-insensitive, .ps1 optional).
|
||||
[JsonPropertyName("scriptpaths")]
|
||||
public Dictionary<string, string> ScriptPaths { get; set; } = new();
|
||||
// REMOVED DELIBERATELY - DO NOT REINTRODUCE: 'scriptpaths' / ScriptPaths.
|
||||
//
|
||||
// This used to be a per-name map of PRE-STAGED scripts on the endpoint
|
||||
// (typically under C:\ProgramData\ShopDB\EndpointAgent\scripts), which
|
||||
// run_script would look up by name and execute. It was superseded by
|
||||
// server-hosted delivery and left behind as dead config: nothing read the
|
||||
// map, so it was an execution path that appeared supported but was not.
|
||||
//
|
||||
// It is gone rather than merely unused because reviving it is a security
|
||||
// regression, and dead code invites revival. That model had NO integrity
|
||||
// check - no digest, no provenance - so whoever could write into the staging
|
||||
// directory chose what the agent ran as SYSTEM. ProgramData is world-
|
||||
// writable by default unless the tree is explicitly tightened, which made
|
||||
// the gate "is this name in appsettings.json", not "is this content
|
||||
// trusted".
|
||||
//
|
||||
// The supported path is run_script with a server-issued delivery descriptor:
|
||||
// the body is fetched from ShopDB over the collector credential and verified
|
||||
// against a server-pinned SHA-256 before it executes, so the content is
|
||||
// authenticated and the action is attributable in the command log. If you
|
||||
// need a script to run on an endpoint, publish it server-side and send it
|
||||
// through that path. If you need to trigger something already installed on
|
||||
// the box, use scheduledtask_run, which is far narrower.
|
||||
|
||||
// The effective script map: appsettings.json entries only (no defaults).
|
||||
// Keys are normalized to bare names so a lookup tolerates 'reset' or
|
||||
// 'reset.ps1'.
|
||||
public IReadOnlyDictionary<string, string> ResolveScriptPaths()
|
||||
{
|
||||
var resolved = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var pair in ScriptPaths)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(pair.Key) || string.IsNullOrWhiteSpace(pair.Value))
|
||||
continue;
|
||||
resolved[NormalizeScript(pair.Key)] = pair.Value;
|
||||
}
|
||||
return resolved;
|
||||
}
|
||||
|
||||
private static string NormalizeScript(string name)
|
||||
{
|
||||
var bare = (name ?? "").Trim();
|
||||
if (bare.EndsWith(".ps1", StringComparison.OrdinalIgnoreCase))
|
||||
bare = bare[..^4];
|
||||
return bare;
|
||||
}
|
||||
// Approved scheduled task paths for the scheduledtask_run action. The wire
|
||||
// carries a free-text task path so the WebUI can expose it as a single
|
||||
// field; ScheduledTaskPolicy validates it against this list and fails closed
|
||||
// when it does not match. An entry matches that exact task path, or acts as
|
||||
// a folder prefix for anything beneath it.
|
||||
//
|
||||
// Empty means "use the shipped Corporate Tools defaults"
|
||||
// (ScheduledTaskPolicy.DefaultApprovedTaskPaths). Setting it REPLACES those
|
||||
// defaults, so a site can narrow the set as well as widen it without an
|
||||
// agent rebuild. Keep it tight: every entry here is something an operator
|
||||
// holding endpoints.deploy can start, in whatever context the task itself
|
||||
// registered.
|
||||
[JsonPropertyName("approvedtaskpaths")]
|
||||
public List<string> ApprovedTaskPaths { get; set; } = new();
|
||||
|
||||
public bool IsUsable()
|
||||
=> !string.IsNullOrWhiteSpace(BaseUrl)
|
||||
&& !string.IsNullOrWhiteSpace(ApiKey)
|
||||
=> !string.IsNullOrWhiteSpace(BaseUrl) && HasSharedKey();
|
||||
|
||||
// True when a real shared collector key is provisioned (not the shipped
|
||||
// placeholder). An enrolled agent runs without one.
|
||||
public bool HasSharedKey()
|
||||
=> !string.IsNullOrWhiteSpace(ApiKey)
|
||||
&& ApiKey != "REPLACE_WITH_COLLECTOR_KEY";
|
||||
|
||||
// Validates the BaseUrl scheme before any request is made. A plaintext
|
||||
|
||||
269
agent/ShopDbEndpointAgent.Core/AgentCredentials.cs
Normal file
@@ -0,0 +1,269 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// This PC's own endpoint credential, issued once by POST /api/endpoints/enroll.
|
||||
// Readable only by SYSTEM and Administrators (DPAPI LocalMachine inside a
|
||||
// protected directory). The JSON key names are a contract: SYSTEM PowerShell
|
||||
// (upgrade-shopdbagent.ps1, the reporters) reads the same file with
|
||||
// [System.Security.Cryptography.ProtectedData]::Unprotect(bytes, $null, 'LocalMachine').
|
||||
public sealed class AgentCredential
|
||||
{
|
||||
[JsonPropertyName("credential")] public string Credential { get; set; } = "";
|
||||
[JsonPropertyName("assetid")] public int AssetId { get; set; }
|
||||
[JsonPropertyName("issuedat")] public string IssuedAt { get; set; } = "";
|
||||
[JsonPropertyName("baseurl")] public string BaseUrl { get; set; } = "";
|
||||
|
||||
// First 12 chars, the same prefix the server shows. Never log more.
|
||||
[JsonIgnore]
|
||||
public string Prefix => Credential.Length <= 12 ? Credential : Credential[..12];
|
||||
}
|
||||
|
||||
// Why the server refused an agent request. Parsed from the error envelope.
|
||||
public static class AgentAuthErrorCodes
|
||||
{
|
||||
public const string CredentialInvalid = "CREDENTIAL_INVALID";
|
||||
public const string SharedKeyRefused = "SHARED_KEY_REFUSED";
|
||||
public const string AssetMismatch = "ASSET_MISMATCH";
|
||||
}
|
||||
|
||||
// Current auth material for outbound requests. Swaps at runtime: enrollment
|
||||
// sets a credential, a CREDENTIAL_INVALID clears it, neither needs a restart.
|
||||
public interface ICredentialProvider
|
||||
{
|
||||
// Null = not enrolled (or the stored credential belongs to another server).
|
||||
AgentCredential? Current { get; }
|
||||
|
||||
// Shared fleet collector key, null when not provisioned.
|
||||
string? SharedKey { get; }
|
||||
|
||||
// Raised after Current changes (set or cleared).
|
||||
event EventHandler? Changed;
|
||||
|
||||
// Raised when the enrollment service should run now.
|
||||
event EventHandler? EnrollmentNeeded;
|
||||
|
||||
// Called by the auth handler for every 401/403 carrying an auth code.
|
||||
// sentCredential is the credential the failed request carried, or null
|
||||
// when it went out on the shared key.
|
||||
void ReportRejected(string code, string? sentCredential);
|
||||
}
|
||||
|
||||
public sealed class AgentCredentialProvider : ICredentialProvider
|
||||
{
|
||||
private readonly object gate = new();
|
||||
private readonly CredentialStore? store;
|
||||
private readonly ILogger logger;
|
||||
private AgentCredential? current;
|
||||
|
||||
public AgentCredentialProvider(AgentConfig config, CredentialStore? store, ILogger logger)
|
||||
{
|
||||
this.store = store;
|
||||
this.logger = logger;
|
||||
SharedKey = config.HasSharedKey() ? config.ApiKey : null;
|
||||
|
||||
var loaded = store?.TryLoad();
|
||||
if (loaded is not null && !SameServer(loaded.BaseUrl, config.BaseUrl))
|
||||
{
|
||||
// Never send a credential to a server that did not issue it. Kept on
|
||||
// disk so pointing BaseUrl back restores it; a new enroll overwrites.
|
||||
logger.LogWarning(
|
||||
"Stored credential {Prefix} was issued by {Issuer}, not {BaseUrl}; not using it.",
|
||||
loaded.Prefix, loaded.BaseUrl, config.BaseUrl);
|
||||
loaded = null;
|
||||
}
|
||||
current = loaded;
|
||||
if (current is not null)
|
||||
logger.LogInformation("Using endpoint credential {Prefix} (asset {AssetId}).",
|
||||
current.Prefix, current.AssetId);
|
||||
}
|
||||
|
||||
public AgentCredential? Current
|
||||
{
|
||||
get { lock (gate) return current; }
|
||||
}
|
||||
|
||||
public string? SharedKey { get; }
|
||||
|
||||
public event EventHandler? Changed;
|
||||
public event EventHandler? EnrollmentNeeded;
|
||||
|
||||
// Persist then swap. Throws when the store cannot be written: a credential
|
||||
// held only in memory is lost at restart and the server will not reissue.
|
||||
public void Set(AgentCredential credential)
|
||||
{
|
||||
store?.Save(credential);
|
||||
lock (gate) current = credential;
|
||||
logger.LogInformation("Endpoint credential {Prefix} stored (asset {AssetId}).",
|
||||
credential.Prefix, credential.AssetId);
|
||||
Changed?.Invoke(this, EventArgs.Empty);
|
||||
}
|
||||
|
||||
public void ReportRejected(string code, string? sentCredential)
|
||||
{
|
||||
if (code == AgentAuthErrorCodes.CredentialInvalid && sentCredential is not null)
|
||||
{
|
||||
bool cleared = false;
|
||||
lock (gate)
|
||||
{
|
||||
// Only the credential that was refused. A late response for an
|
||||
// old credential must not wipe a freshly issued one.
|
||||
if (current is not null && current.Credential == sentCredential)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"Server refused credential {Prefix} (revoked or unknown); deleting it and re-enrolling.",
|
||||
current.Prefix);
|
||||
current = null;
|
||||
cleared = true;
|
||||
}
|
||||
}
|
||||
if (cleared)
|
||||
{
|
||||
try
|
||||
{
|
||||
store?.Delete();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogError(error, "Could not delete the refused credential file.");
|
||||
}
|
||||
Changed?.Invoke(this, EventArgs.Empty);
|
||||
EnrollmentNeeded?.Invoke(this, EventArgs.Empty);
|
||||
}
|
||||
}
|
||||
else if (code == AgentAuthErrorCodes.SharedKeyRefused && sentCredential is null && Current is null)
|
||||
{
|
||||
logger.LogWarning("Server refused the shared key for this PC; requesting enrollment.");
|
||||
EnrollmentNeeded?.Invoke(this, EventArgs.Empty);
|
||||
}
|
||||
else if (code == AgentAuthErrorCodes.AssetMismatch)
|
||||
{
|
||||
logger.LogError(
|
||||
"Server reported ASSET_MISMATCH: this PC's credential belongs to a different asset than the request named.");
|
||||
}
|
||||
}
|
||||
|
||||
private static bool SameServer(string issued, string configured)
|
||||
=> string.Equals(Normalize(issued), Normalize(configured), StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
private static string Normalize(string url) => (url ?? "").Trim().TrimEnd('/');
|
||||
}
|
||||
|
||||
// credential.bin and enroll.json under %ProgramData%\ShopDB\EndpointAgent\credential.
|
||||
public sealed class CredentialStore
|
||||
{
|
||||
public const string DirectoryName = "credential";
|
||||
public const string CredentialFileName = "credential.bin";
|
||||
public const string EnrollFileName = "enroll.json";
|
||||
|
||||
private const int MaxFileBytes = 64 * 1024;
|
||||
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
};
|
||||
|
||||
private readonly ILogger logger;
|
||||
|
||||
public CredentialStore(ILogger logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
private static string StoreDirectory => ProtectedFileSystem.EnsureDirectory(DirectoryName);
|
||||
|
||||
public AgentCredential? TryLoad()
|
||||
{
|
||||
try
|
||||
{
|
||||
var path = Path.Combine(StoreDirectory, CredentialFileName);
|
||||
if (ProtectedFileSystem.TryGetAttributes(path) is null)
|
||||
return null;
|
||||
|
||||
var blob = ReadAllBytesNoFollow(path);
|
||||
var json = ProtectedData.Unprotect(blob, null, DataProtectionScope.LocalMachine);
|
||||
var credential = JsonSerializer.Deserialize<AgentCredential>(json, JsonOptions);
|
||||
if (credential is null || !credential.Credential.StartsWith("sdbe_", StringComparison.Ordinal))
|
||||
{
|
||||
logger.LogWarning("Stored credential file is not a usable credential; ignoring it.");
|
||||
return null;
|
||||
}
|
||||
return credential;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogError(error, "Could not read the stored endpoint credential; running unenrolled.");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public void Save(AgentCredential credential)
|
||||
{
|
||||
var dir = StoreDirectory;
|
||||
var json = JsonSerializer.SerializeToUtf8Bytes(credential);
|
||||
var blob = ProtectedData.Protect(json, null, DataProtectionScope.LocalMachine);
|
||||
var target = Path.Combine(dir, CredentialFileName);
|
||||
var temp = Path.Combine(dir, CredentialFileName + ".new");
|
||||
using (var file = ProtectedFileSystem.CreateFile(temp))
|
||||
{
|
||||
file.Write(blob);
|
||||
file.Flush(flushToDisk: true);
|
||||
}
|
||||
// Moved file keeps its protected DACL.
|
||||
File.Move(temp, target, overwrite: true);
|
||||
}
|
||||
|
||||
public void Delete()
|
||||
=> ProtectedFileSystem.DeleteFileNoFollow(Path.Combine(StoreDirectory, CredentialFileName));
|
||||
|
||||
// The enroll nonce is generated once and kept for the life of the PC, so
|
||||
// an approved request can only be claimed by the agent that made it.
|
||||
public string GetOrCreateEnrollNonce()
|
||||
{
|
||||
var path = Path.Combine(StoreDirectory, EnrollFileName);
|
||||
if (ProtectedFileSystem.TryGetAttributes(path) is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var doc = JsonDocument.Parse(ReadAllBytesNoFollow(path));
|
||||
if (doc.RootElement.TryGetProperty("enrollnonce", out var value)
|
||||
&& value.GetString() is string nonce && nonce.Length >= 32)
|
||||
return nonce;
|
||||
logger.LogWarning("enroll.json has no usable nonce; generating a new one.");
|
||||
}
|
||||
catch (Exception error) when (error is JsonException or IOException or UnauthorizedAccessException)
|
||||
{
|
||||
logger.LogWarning(error, "enroll.json unreadable; generating a new nonce.");
|
||||
}
|
||||
}
|
||||
|
||||
var fresh = Base64Url(RandomNumberGenerator.GetBytes(32));
|
||||
var body = JsonSerializer.SerializeToUtf8Bytes(new
|
||||
{
|
||||
enrollnonce = fresh,
|
||||
createdat = DateTime.UtcNow.ToString("o"),
|
||||
});
|
||||
using (var file = ProtectedFileSystem.CreateFile(path))
|
||||
{
|
||||
file.Write(body);
|
||||
file.Flush(flushToDisk: true);
|
||||
}
|
||||
return fresh;
|
||||
}
|
||||
|
||||
internal static string Base64Url(byte[] bytes)
|
||||
=> Convert.ToBase64String(bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_');
|
||||
|
||||
private static byte[] ReadAllBytesNoFollow(string path)
|
||||
{
|
||||
using var stream = ProtectedFileSystem.OpenReadNoFollow(path);
|
||||
if (stream.Length > MaxFileBytes)
|
||||
throw new InvalidDataException($"{path} is larger than expected");
|
||||
using var buffer = new MemoryStream();
|
||||
stream.CopyTo(buffer);
|
||||
return buffer.ToArray();
|
||||
}
|
||||
}
|
||||
387
agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs
Normal file
@@ -0,0 +1,387 @@
|
||||
using System.Net;
|
||||
using System.Net.Security;
|
||||
using System.Security.Cryptography;
|
||||
using System.Security.Cryptography.X509Certificates;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Win32;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// What an HttpClient is for. Decides pooling, timeout and auth.
|
||||
public enum AgentHttpChannel
|
||||
{
|
||||
Collector,
|
||||
Command,
|
||||
FleetAlert,
|
||||
Download,
|
||||
Enroll,
|
||||
}
|
||||
|
||||
// The ONLY place the agent builds an HttpClient or a handler. Every channel gets
|
||||
// the same server certificate check (ServerPinSet) and the same per-request auth
|
||||
// (AgentAuthHandler), so a new channel cannot forget either.
|
||||
public sealed class AgentHttpFactory
|
||||
{
|
||||
private readonly AgentConfig config;
|
||||
private readonly ICredentialProvider credentials;
|
||||
private readonly ILogger logger;
|
||||
private readonly ServerPinSet pins;
|
||||
|
||||
public AgentHttpFactory(AgentConfig config, ICredentialProvider credentials, ILogger logger)
|
||||
{
|
||||
this.config = config;
|
||||
this.credentials = credentials;
|
||||
this.logger = logger;
|
||||
pins = ServerPinSet.Load(config, logger);
|
||||
}
|
||||
|
||||
public ICredentialProvider Credentials => credentials;
|
||||
|
||||
// Caller owns and disposes the client (it disposes the handler chain).
|
||||
public HttpClient CreateClient(AgentHttpChannel channel, TimeSpan? timeout = null)
|
||||
{
|
||||
// Lifetime cap: pooled TLS sessions were validated against the pins of
|
||||
// their handshake, so recycle them for a pin change to reach new ones.
|
||||
var sockets = new SocketsHttpHandler
|
||||
{
|
||||
PooledConnectionLifetime = TimeSpan.FromMinutes(2),
|
||||
SslOptions = new SslClientAuthenticationOptions
|
||||
{
|
||||
RemoteCertificateValidationCallback = (_, cert, _, errors)
|
||||
=> pins.Validate(cert, errors, channel),
|
||||
},
|
||||
};
|
||||
|
||||
switch (channel)
|
||||
{
|
||||
case AgentHttpChannel.Command:
|
||||
// Long-poll: retire pooled TLS connections faster than the
|
||||
// server keep-alive, or a reused dead socket yields "unexpected
|
||||
// EOF or 0 bytes from the transport stream" (1.6.2).
|
||||
sockets.PooledConnectionIdleTimeout = TimeSpan.FromSeconds(1);
|
||||
break;
|
||||
case AgentHttpChannel.FleetAlert:
|
||||
sockets.PooledConnectionIdleTimeout = TimeSpan.FromSeconds(90);
|
||||
break;
|
||||
}
|
||||
|
||||
var auth = new AgentAuthHandler(credentials, logger,
|
||||
sharedKeyOnly: channel == AgentHttpChannel.Enroll)
|
||||
{
|
||||
InnerHandler = sockets,
|
||||
};
|
||||
|
||||
return new HttpClient(auth, disposeHandler: true)
|
||||
{
|
||||
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
|
||||
Timeout = timeout ?? DefaultTimeout(channel),
|
||||
};
|
||||
}
|
||||
|
||||
private static TimeSpan DefaultTimeout(AgentHttpChannel channel) => channel switch
|
||||
{
|
||||
// Claim is held open server-side (~25s); clear that plus slack.
|
||||
AgentHttpChannel.Command => TimeSpan.FromSeconds(60),
|
||||
AgentHttpChannel.Download => TimeSpan.FromMinutes(10),
|
||||
_ => TimeSpan.FromSeconds(30),
|
||||
};
|
||||
}
|
||||
|
||||
// Adds exactly one auth header per request, read at send time so a credential
|
||||
// issued or revoked mid-run takes effect on the next request. Watches 401/403
|
||||
// replies for the auth error codes and reports them to the provider.
|
||||
public sealed class AgentAuthHandler : DelegatingHandler
|
||||
{
|
||||
public const string CredentialHeader = "X-Endpoint-Credential";
|
||||
public const string SharedKeyHeader = "X-API-Key";
|
||||
|
||||
private readonly ICredentialProvider credentials;
|
||||
private readonly ILogger logger;
|
||||
private readonly bool sharedKeyOnly;
|
||||
|
||||
public AgentAuthHandler(ICredentialProvider credentials, ILogger logger, bool sharedKeyOnly = false)
|
||||
{
|
||||
this.credentials = credentials;
|
||||
this.logger = logger;
|
||||
this.sharedKeyOnly = sharedKeyOnly;
|
||||
}
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(
|
||||
HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
request.Headers.Remove(CredentialHeader);
|
||||
request.Headers.Remove(SharedKeyHeader);
|
||||
|
||||
var credential = sharedKeyOnly ? null : credentials.Current?.Credential;
|
||||
if (credential is not null)
|
||||
request.Headers.TryAddWithoutValidation(CredentialHeader, credential);
|
||||
else if (credentials.SharedKey is string sharedKey)
|
||||
request.Headers.TryAddWithoutValidation(SharedKeyHeader, sharedKey);
|
||||
|
||||
var response = await base.SendAsync(request, cancellationToken);
|
||||
|
||||
if (response.StatusCode is HttpStatusCode.Unauthorized or HttpStatusCode.Forbidden)
|
||||
{
|
||||
try
|
||||
{
|
||||
// Buffered so the caller can still read the body.
|
||||
await response.Content.LoadIntoBufferAsync();
|
||||
var body = await response.Content.ReadAsStringAsync(cancellationToken);
|
||||
var code = AgentServerError.TryParse(body)?.Code;
|
||||
if (!string.IsNullOrEmpty(code))
|
||||
credentials.ReportRejected(code, credential);
|
||||
}
|
||||
catch (Exception error) when (error is not OperationCanceledException)
|
||||
{
|
||||
logger.LogDebug(error, "Could not read an auth failure body");
|
||||
}
|
||||
}
|
||||
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
// The server's error envelope: {"status":"error","data":{"error":{"code","message"}}}.
|
||||
public sealed record AgentServerError(string Code, string Message)
|
||||
{
|
||||
public static AgentServerError? TryParse(string? body)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(body))
|
||||
return null;
|
||||
try
|
||||
{
|
||||
using var doc = JsonDocument.Parse(body);
|
||||
var root = doc.RootElement;
|
||||
if (root.ValueKind != JsonValueKind.Object)
|
||||
return null;
|
||||
if (root.TryGetProperty("data", out var data) && data.ValueKind == JsonValueKind.Object)
|
||||
root = data;
|
||||
if (!root.TryGetProperty("error", out var error) || error.ValueKind != JsonValueKind.Object)
|
||||
return null;
|
||||
var code = error.TryGetProperty("code", out var c) && c.ValueKind == JsonValueKind.String
|
||||
? c.GetString() ?? "" : "";
|
||||
var message = error.TryGetProperty("message", out var m) && m.ValueKind == JsonValueKind.String
|
||||
? m.GetString() ?? "" : "";
|
||||
return new AgentServerError(code, message);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Server certificate trust. Pins are hex SHA-256 of the leaf SubjectPublicKeyInfo.
|
||||
// HKLM\SOFTWARE\GE\ShopDB ServerPins is AUTHORITATIVE whenever the value exists
|
||||
// in either view, even empty (empty = no pins, Windows chain validation), so a
|
||||
// removed or rotated-out pin really goes away. Only when the value is absent in
|
||||
// both views do appsettings serverpins and the legacy SHA-1 servercertthumbprint
|
||||
// apply. With any pin in effect only a matching leaf is accepted and chain/name
|
||||
// errors are ignored (self-signed servers on the air gap). Without pins:
|
||||
// SslPolicyErrors.None. Registry re-read at most every RefreshInterval on the
|
||||
// handshake path, so a GE-Enforce fix or a rotation lands without a restart.
|
||||
public sealed class ServerPinSet
|
||||
{
|
||||
public const string RegistryValue = "ServerPins";
|
||||
public static readonly TimeSpan RefreshInterval = TimeSpan.FromSeconds(60);
|
||||
|
||||
// Present = the value exists in at least one view.
|
||||
private sealed record RegistryRead(bool Present, List<string> Values)
|
||||
{
|
||||
public string Raw => Present ? "registry:" + string.Join("|", Values) : "absent";
|
||||
}
|
||||
|
||||
private sealed record Snapshot(HashSet<string> SpkiPins, string? LegacyThumbprint, bool FromRegistry, string RawRegistry)
|
||||
{
|
||||
public string Signature => (FromRegistry ? "registry:" : "appsettings:")
|
||||
+ string.Join(",", SpkiPins.OrderBy(p => p, StringComparer.Ordinal))
|
||||
+ (LegacyThumbprint is null ? "" : "+legacy");
|
||||
public bool Pinned => SpkiPins.Count > 0 || LegacyThumbprint is not null;
|
||||
public string Source => FromRegistry ? "registry" : "appsettings.json";
|
||||
}
|
||||
|
||||
private readonly HashSet<string> configPins;
|
||||
private readonly string? legacyThumbprint;
|
||||
private readonly ILogger logger;
|
||||
private readonly Func<bool, RegistryRead?> readRegistry;
|
||||
private readonly object refreshLock = new();
|
||||
private volatile Snapshot current;
|
||||
private long nextRefreshTicks;
|
||||
|
||||
private ServerPinSet(HashSet<string> configPins, string? legacyThumbprint, ILogger logger,
|
||||
Func<bool, RegistryRead?> readRegistry)
|
||||
{
|
||||
this.configPins = configPins;
|
||||
this.legacyThumbprint = legacyThumbprint;
|
||||
this.logger = logger;
|
||||
this.readRegistry = readRegistry;
|
||||
current = Build(readRegistry(false) ?? new RegistryRead(false, new List<string>()));
|
||||
nextRefreshTicks = Environment.TickCount64 + (long)RefreshInterval.TotalMilliseconds;
|
||||
}
|
||||
|
||||
public bool IsPinned => Current().Pinned;
|
||||
|
||||
public static ServerPinSet Load(AgentConfig config, ILogger logger)
|
||||
{
|
||||
var pins = new HashSet<string>(StringComparer.Ordinal);
|
||||
foreach (var pin in config.ServerPins ?? new List<string>())
|
||||
AddPin(pins, pin, "appsettings.json", logger);
|
||||
|
||||
string? thumbprint = null;
|
||||
if (!string.IsNullOrWhiteSpace(config.ServerCertThumbprint))
|
||||
thumbprint = config.ServerCertThumbprint.Replace(":", "").Replace(" ", "").Trim().ToUpperInvariant();
|
||||
|
||||
var set = new ServerPinSet(pins, thumbprint, logger, refresh => ReadRegistryPins(logger, refresh));
|
||||
var snapshot = set.current;
|
||||
logger.LogInformation(
|
||||
"Server certificate trust from {Source}: {Pins} SPKI pin(s){Legacy}; {Mode}.",
|
||||
snapshot.Source, snapshot.SpkiPins.Count,
|
||||
snapshot.LegacyThumbprint is null ? "" : " + legacy thumbprint",
|
||||
snapshot.Pinned ? "pinned" : "Windows chain validation");
|
||||
if (snapshot.FromRegistry && (pins.Count > 0 || thumbprint is not null))
|
||||
logger.LogInformation(
|
||||
"Registry {Value} is set; ignoring appsettings.json serverpins/servercertthumbprint.", RegistryValue);
|
||||
return set;
|
||||
}
|
||||
|
||||
// Cheap tick compare per handshake; registry read only when the window lapsed.
|
||||
private Snapshot Current()
|
||||
{
|
||||
if (Environment.TickCount64 < Interlocked.Read(ref nextRefreshTicks))
|
||||
return current;
|
||||
lock (refreshLock)
|
||||
{
|
||||
if (Environment.TickCount64 < nextRefreshTicks)
|
||||
return current;
|
||||
var read = readRegistry(true);
|
||||
Interlocked.Exchange(ref nextRefreshTicks,
|
||||
Environment.TickCount64 + (long)RefreshInterval.TotalMilliseconds);
|
||||
var previous = current;
|
||||
if (read is null || read.Raw == previous.RawRegistry)
|
||||
return previous;
|
||||
|
||||
var next = Build(read);
|
||||
current = next;
|
||||
if (next.Signature != previous.Signature)
|
||||
logger.LogInformation(
|
||||
"Server pins changed, source {Source}: {Count} SPKI pin(s) [{Pins}]{Legacy}; {Mode}.",
|
||||
next.Source,
|
||||
next.SpkiPins.Count,
|
||||
string.Join(", ", next.SpkiPins.OrderBy(p => p, StringComparer.Ordinal).Select(p => p[..12])),
|
||||
next.LegacyThumbprint is null ? "" : " + legacy thumbprint",
|
||||
next.Pinned ? "pinned" : "Windows chain validation");
|
||||
return next;
|
||||
}
|
||||
}
|
||||
|
||||
// Malformed-pin warnings only fire when the raw registry text changes.
|
||||
private Snapshot Build(RegistryRead read)
|
||||
{
|
||||
if (!read.Present)
|
||||
return new Snapshot(new HashSet<string>(configPins, StringComparer.Ordinal),
|
||||
legacyThumbprint, FromRegistry: false, read.Raw);
|
||||
|
||||
var pins = new HashSet<string>(StringComparer.Ordinal);
|
||||
foreach (var pin in read.Values)
|
||||
AddPin(pins, pin, "registry", logger);
|
||||
return new Snapshot(pins, LegacyThumbprint: null, FromRegistry: true, read.Raw);
|
||||
}
|
||||
|
||||
public bool Validate(X509Certificate? certificate, SslPolicyErrors errors, AgentHttpChannel channel)
|
||||
{
|
||||
if (certificate is null)
|
||||
{
|
||||
logger.LogWarning("Server provided no certificate ({Channel})", channel);
|
||||
return false;
|
||||
}
|
||||
|
||||
var snapshot = Current();
|
||||
if (!snapshot.Pinned)
|
||||
{
|
||||
if (errors == SslPolicyErrors.None)
|
||||
return true;
|
||||
logger.LogWarning(
|
||||
"Server certificate rejected on {Channel} channel: {Errors} ({Subject})",
|
||||
channel, errors, certificate.Subject);
|
||||
return false;
|
||||
}
|
||||
|
||||
var owned = certificate as X509Certificate2;
|
||||
var leaf = owned ?? new X509Certificate2(certificate);
|
||||
try
|
||||
{
|
||||
var spki = SpkiSha256(leaf);
|
||||
if (snapshot.SpkiPins.Contains(spki))
|
||||
return true;
|
||||
if (snapshot.LegacyThumbprint is not null
|
||||
&& string.Equals(leaf.GetCertHashString(), snapshot.LegacyThumbprint, StringComparison.OrdinalIgnoreCase))
|
||||
return true;
|
||||
|
||||
logger.LogError(
|
||||
"Server certificate does not match any pin on {Channel} channel: spki {Spki} ({Subject})",
|
||||
channel, spki, leaf.Subject);
|
||||
return false;
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (owned is null)
|
||||
leaf.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
public static string SpkiSha256(X509Certificate2 certificate)
|
||||
=> Convert.ToHexString(SHA256.HashData(certificate.PublicKey.ExportSubjectPublicKeyInfo()))
|
||||
.ToLowerInvariant();
|
||||
|
||||
private static void AddPin(HashSet<string> pins, string? raw, string source, ILogger logger)
|
||||
{
|
||||
var pin = (raw ?? "").Replace(":", "").Replace(" ", "").Trim().ToLowerInvariant();
|
||||
if (pin.Length == 0)
|
||||
return;
|
||||
if (pin.Length != 64 || !pin.All(Uri.IsHexDigit))
|
||||
{
|
||||
logger.LogWarning("Ignoring malformed server pin from {Source}: not 64 hex chars", source);
|
||||
return;
|
||||
}
|
||||
pins.Add(pin);
|
||||
}
|
||||
|
||||
// Startup keeps whatever views read. A refresh returns null on any failed view
|
||||
// so a transient denial cannot silently swap the pin source (and the server with it).
|
||||
private static RegistryRead? ReadRegistryPins(ILogger logger, bool refresh)
|
||||
{
|
||||
var present = false;
|
||||
var found = new List<string>();
|
||||
foreach (var view in new[] { RegistryView.Registry64, RegistryView.Registry32 })
|
||||
{
|
||||
try
|
||||
{
|
||||
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, view);
|
||||
using var key = machine.OpenSubKey(AgentConfigProvider.RegistryPath);
|
||||
var value = key?.GetValue(RegistryValue);
|
||||
if (value is not null)
|
||||
present = true;
|
||||
switch (value)
|
||||
{
|
||||
case string[] many:
|
||||
found.AddRange(many);
|
||||
break;
|
||||
case string one:
|
||||
found.AddRange(one.Split(new[] { ',', ';' }, StringSplitOptions.RemoveEmptyEntries));
|
||||
break;
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
if (refresh)
|
||||
{
|
||||
logger.LogDebug(error, "Registry re-read of {Value} in {View} view failed; keeping pins", RegistryValue, view);
|
||||
return null;
|
||||
}
|
||||
logger.LogWarning(error, "Registry read of {Value} in {View} view failed", RegistryValue, view);
|
||||
}
|
||||
}
|
||||
return new RegistryRead(present, found);
|
||||
}
|
||||
}
|
||||
180
agent/ShopDbEndpointAgent.Core/AgentPipeClient.cs
Normal file
@@ -0,0 +1,180 @@
|
||||
using System.IO.Pipes;
|
||||
using System.Security.Principal;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Wire constants for the local agent pipe. One UTF-8 JSON object per line:
|
||||
// request, response, close.
|
||||
public static class AgentPipeProtocol
|
||||
{
|
||||
public const string PipeName = "ShopDB.EndpointAgent.v1";
|
||||
public const int MaxRequestBytes = 64 * 1024;
|
||||
public const int MaxResponseBytes = 4 * 1024 * 1024;
|
||||
public static readonly TimeSpan ConnectTimeout = TimeSpan.FromSeconds(3);
|
||||
public static readonly TimeSpan RequestReadTimeout = TimeSpan.FromSeconds(10);
|
||||
}
|
||||
|
||||
// Ok/Error as the agent answered. Data is the whole response object (ping
|
||||
// fields such as agentversion/enrolled/assetid live there). Data is null when
|
||||
// the agent could not be reached or answered garbage; Error then says why.
|
||||
public sealed record PipeResponse(bool Ok, string? Error, JsonElement? Data);
|
||||
|
||||
public sealed class AgentPipeException : Exception
|
||||
{
|
||||
public AgentPipeException(string message) : base(message)
|
||||
{
|
||||
}
|
||||
|
||||
public AgentPipeException(string message, Exception inner) : base(message, inner)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
// Client for the SYSTEM agent's local pipe. Used by the tray (which holds no
|
||||
// secret) and by shopdbagent.exe --once/--report when not running as SYSTEM.
|
||||
// Only GetAlertsAsync throws; the others report failure through PipeResponse.
|
||||
public sealed class AgentPipeClient
|
||||
{
|
||||
// Send Status waits on the user-context refresh plus the upload retries.
|
||||
private static readonly TimeSpan LongOperationTimeout = TimeSpan.FromMinutes(3);
|
||||
private static readonly TimeSpan ShortOperationTimeout = TimeSpan.FromSeconds(45);
|
||||
|
||||
private static readonly JsonSerializerOptions AlertOptions = new()
|
||||
{
|
||||
Converters = { new TolerantFleetAlertListConverter() },
|
||||
};
|
||||
|
||||
public Task<PipeResponse> PingAsync(CancellationToken cancellationToken)
|
||||
=> SendSafeAsync(new { op = "ping" }, ShortOperationTimeout, cancellationToken);
|
||||
|
||||
public Task<PipeResponse> SendStatusAsync(CancellationToken cancellationToken)
|
||||
=> SendSafeAsync(new { op = "sendstatus" }, LongOperationTimeout, cancellationToken);
|
||||
|
||||
public Task<PipeResponse> SendReportAsync(string spoolPath, CancellationToken cancellationToken)
|
||||
=> SendSafeAsync(new { op = "sendreport", spoolpath = spoolPath }, LongOperationTimeout,
|
||||
cancellationToken);
|
||||
|
||||
public async Task<IReadOnlyList<FleetAlert>> GetAlertsAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var response = await SendAsync(new { op = "alerts" }, ShortOperationTimeout, cancellationToken);
|
||||
if (!response.Ok)
|
||||
throw new AgentPipeException(response.Error ?? "agent refused the alerts request");
|
||||
|
||||
if (response.Data is not JsonElement data
|
||||
|| !data.TryGetProperty("alerts", out var alerts)
|
||||
|| alerts.ValueKind != JsonValueKind.Array)
|
||||
throw new AgentPipeException("agent response has no alerts array");
|
||||
|
||||
try
|
||||
{
|
||||
return alerts.Deserialize<List<FleetAlert>>(AlertOptions) ?? new List<FleetAlert>();
|
||||
}
|
||||
catch (JsonException error)
|
||||
{
|
||||
throw new AgentPipeException("agent alerts could not be parsed", error);
|
||||
}
|
||||
}
|
||||
|
||||
public Task<PipeResponse> AckAlertAsync(int notificationId, CancellationToken cancellationToken)
|
||||
=> SendSafeAsync(new { op = "ackalert", notificationid = notificationId },
|
||||
ShortOperationTimeout, cancellationToken);
|
||||
|
||||
private static async Task<PipeResponse> SendSafeAsync(
|
||||
object request, TimeSpan timeout, CancellationToken cancellationToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
return await SendAsync(request, timeout, cancellationToken);
|
||||
}
|
||||
catch (AgentPipeException error)
|
||||
{
|
||||
return new PipeResponse(false, error.Message, null);
|
||||
}
|
||||
}
|
||||
|
||||
// Throws AgentPipeException on any transport or framing failure, and
|
||||
// OperationCanceledException only when the caller cancelled.
|
||||
private static async Task<PipeResponse> SendAsync(
|
||||
object request, TimeSpan timeout, CancellationToken cancellationToken)
|
||||
{
|
||||
using var timeoutSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
|
||||
timeoutSource.CancelAfter(timeout);
|
||||
var token = timeoutSource.Token;
|
||||
|
||||
try
|
||||
{
|
||||
// Identification: the agent may learn who we are, never act as us.
|
||||
await using var pipe = new NamedPipeClientStream(".", AgentPipeProtocol.PipeName,
|
||||
PipeDirection.InOut, PipeOptions.Asynchronous, TokenImpersonationLevel.Identification);
|
||||
|
||||
try
|
||||
{
|
||||
await pipe.ConnectAsync((int)AgentPipeProtocol.ConnectTimeout.TotalMilliseconds, token);
|
||||
}
|
||||
catch (TimeoutException error)
|
||||
{
|
||||
throw new AgentPipeException("agent not reachable (is the ShopDB Endpoint Agent service running?)", error);
|
||||
}
|
||||
|
||||
var line = JsonSerializer.SerializeToUtf8Bytes(request);
|
||||
await pipe.WriteAsync(line, token);
|
||||
await pipe.WriteAsync(new[] { (byte)'\n' }, token);
|
||||
await pipe.FlushAsync(token);
|
||||
|
||||
var body = await ReadLineAsync(pipe, AgentPipeProtocol.MaxResponseBytes, token)
|
||||
?? throw new AgentPipeException("agent closed the pipe without answering");
|
||||
|
||||
using var doc = JsonDocument.Parse(body);
|
||||
var root = doc.RootElement;
|
||||
if (root.ValueKind != JsonValueKind.Object)
|
||||
throw new AgentPipeException("agent response is not a JSON object");
|
||||
|
||||
var ok = root.TryGetProperty("ok", out var okValue) && okValue.ValueKind == JsonValueKind.True;
|
||||
var errorText = root.TryGetProperty("error", out var errorValue)
|
||||
&& errorValue.ValueKind == JsonValueKind.String
|
||||
? errorValue.GetString() : null;
|
||||
return new PipeResponse(ok, ok ? null : errorText ?? "agent reported failure", root.Clone());
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (OperationCanceledException error)
|
||||
{
|
||||
throw new AgentPipeException("agent did not answer in time", error);
|
||||
}
|
||||
catch (AgentPipeException)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException or JsonException
|
||||
or InvalidDataException)
|
||||
{
|
||||
throw new AgentPipeException("agent pipe failed: " + error.Message, error);
|
||||
}
|
||||
}
|
||||
|
||||
// One newline-terminated UTF-8 line, or null on EOF before any byte.
|
||||
// Throws InvalidDataException past maxBytes. Shared with the server.
|
||||
internal static async Task<string?> ReadLineAsync(Stream stream, int maxBytes, CancellationToken token)
|
||||
{
|
||||
var buffer = new MemoryStream();
|
||||
var chunk = new byte[4096];
|
||||
while (true)
|
||||
{
|
||||
var read = await stream.ReadAsync(chunk, token);
|
||||
if (read == 0)
|
||||
return buffer.Length == 0 ? null : Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
|
||||
|
||||
var newline = Array.IndexOf(chunk, (byte)'\n', 0, read);
|
||||
var take = newline >= 0 ? newline : read;
|
||||
if (buffer.Length + take > maxBytes)
|
||||
throw new InvalidDataException($"pipe message exceeds {maxBytes} bytes");
|
||||
buffer.Write(chunk, 0, take);
|
||||
if (newline >= 0)
|
||||
return Encoding.UTF8.GetString(buffer.GetBuffer(), 0, (int)buffer.Length);
|
||||
}
|
||||
}
|
||||
}
|
||||
562
agent/ShopDbEndpointAgent.Core/AgentPipeServer.cs
Normal file
@@ -0,0 +1,562 @@
|
||||
using System.IO.Pipes;
|
||||
using System.Security.AccessControl;
|
||||
using System.Security.Principal;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Win32;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// \\.\pipe\ShopDB.EndpointAgent.v1: the tray's only way to reach the server.
|
||||
// The tray holds no secret; it asks the SYSTEM agent to send status, upload a
|
||||
// report, list alerts or acknowledge one, and the agent does it with its own
|
||||
// credential. Protocol in AgentPipeProtocol / AgentPipeClient.
|
||||
//
|
||||
// ACL: SYSTEM and Administrators full control, INTERACTIVE read/write (no
|
||||
// create-instance, so a user cannot add instances to our pipe). The first
|
||||
// instance is created with FirstPipeInstance so a process that squatted the
|
||||
// name before the service started is refused rather than joined.
|
||||
public sealed class AgentPipeServer : BackgroundService
|
||||
{
|
||||
public const int MaxInstances = 8;
|
||||
|
||||
private const int MaxScreenshotBytes = 25 * 1024 * 1024;
|
||||
private static readonly TimeSpan LongOperationTimeout = TimeSpan.FromMinutes(2);
|
||||
private static readonly TimeSpan ShortOperationTimeout = TimeSpan.FromSeconds(40);
|
||||
|
||||
private static readonly JsonSerializerOptions PayloadOptions = new()
|
||||
{
|
||||
PropertyNameCaseInsensitive = true,
|
||||
};
|
||||
|
||||
private readonly AgentConfig config;
|
||||
private readonly TelemetryWorker telemetry;
|
||||
private readonly CollectorClient collector;
|
||||
private readonly FleetAlertClient alerts;
|
||||
private readonly ICredentialProvider credentials;
|
||||
private readonly string agentVersion;
|
||||
private readonly ILogger<AgentPipeServer> logger;
|
||||
private readonly SemaphoreSlim statusGate = new(1, 1);
|
||||
private readonly SemaphoreSlim reportGate = new(1, 1);
|
||||
|
||||
public AgentPipeServer(AgentConfig config, TelemetryWorker telemetry, CollectorClient collector,
|
||||
FleetAlertClient alerts, ICredentialProvider credentials, string agentVersion,
|
||||
ILogger<AgentPipeServer> logger)
|
||||
{
|
||||
this.config = config;
|
||||
this.telemetry = telemetry;
|
||||
this.collector = collector;
|
||||
this.alerts = alerts;
|
||||
this.credentials = credentials;
|
||||
this.agentVersion = agentVersion;
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
var slots = new SemaphoreSlim(MaxInstances, MaxInstances);
|
||||
bool first = true;
|
||||
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await slots.WaitAsync(stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
NamedPipeServerStream pipe;
|
||||
try
|
||||
{
|
||||
pipe = CreateInstance(first);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
slots.Release();
|
||||
logger.LogError(error,
|
||||
first
|
||||
? "Could not create the agent pipe; another process may own the name. Retrying in 30s."
|
||||
: "Could not create an agent pipe instance. Retrying in 30s.");
|
||||
if (!await SafeDelay(TimeSpan.FromSeconds(30), stoppingToken))
|
||||
break;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (first)
|
||||
{
|
||||
first = false;
|
||||
logger.LogInformation("Agent pipe listening on {Pipe}", AgentPipeProtocol.PipeName);
|
||||
RestrictAppSettingsAcl();
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await pipe.WaitForConnectionAsync(stoppingToken);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
pipe.Dispose();
|
||||
slots.Release();
|
||||
if (stoppingToken.IsCancellationRequested)
|
||||
break;
|
||||
logger.LogWarning(error, "Agent pipe connection failed");
|
||||
if (!await SafeDelay(TimeSpan.FromSeconds(1), stoppingToken))
|
||||
break;
|
||||
continue;
|
||||
}
|
||||
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
await HandleConnectionAsync(pipe, stoppingToken);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Agent pipe request ended with an error");
|
||||
}
|
||||
finally
|
||||
{
|
||||
pipe.Dispose();
|
||||
slots.Release();
|
||||
}
|
||||
}, CancellationToken.None);
|
||||
}
|
||||
}
|
||||
|
||||
private static NamedPipeServerStream CreateInstance(bool first)
|
||||
{
|
||||
var security = new PipeSecurity();
|
||||
security.AddAccessRule(new PipeAccessRule(
|
||||
new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null),
|
||||
PipeAccessRights.FullControl, AccessControlType.Allow));
|
||||
security.AddAccessRule(new PipeAccessRule(
|
||||
new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null),
|
||||
PipeAccessRights.FullControl, AccessControlType.Allow));
|
||||
security.AddAccessRule(new PipeAccessRule(
|
||||
new SecurityIdentifier(WellKnownSidType.InteractiveSid, null),
|
||||
PipeAccessRights.ReadWrite, AccessControlType.Allow));
|
||||
|
||||
var options = PipeOptions.Asynchronous;
|
||||
if (first)
|
||||
options |= PipeOptions.FirstPipeInstance;
|
||||
|
||||
return NamedPipeServerStreamAcl.Create(AgentPipeProtocol.PipeName, PipeDirection.InOut,
|
||||
MaxInstances, PipeTransmissionMode.Byte, options, 4096, 4096, security);
|
||||
}
|
||||
|
||||
private async Task HandleConnectionAsync(NamedPipeServerStream pipe, CancellationToken stoppingToken)
|
||||
{
|
||||
string? line;
|
||||
using (var readTimeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken))
|
||||
{
|
||||
readTimeout.CancelAfter(AgentPipeProtocol.RequestReadTimeout);
|
||||
try
|
||||
{
|
||||
line = await AgentPipeClient.ReadLineAsync(pipe, AgentPipeProtocol.MaxRequestBytes,
|
||||
readTimeout.Token);
|
||||
}
|
||||
catch (InvalidDataException)
|
||||
{
|
||||
await WriteAsync(pipe, Failure("request too large"), stoppingToken);
|
||||
return;
|
||||
}
|
||||
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
logger.LogDebug("Agent pipe client sent no request within {Timeout}",
|
||||
AgentPipeProtocol.RequestReadTimeout);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (line is null)
|
||||
return;
|
||||
|
||||
// Impersonation needs a completed read first, which we just did.
|
||||
var (callerName, callerSid) = IdentifyCaller(pipe);
|
||||
|
||||
string op;
|
||||
JsonElement request;
|
||||
try
|
||||
{
|
||||
using var doc = JsonDocument.Parse(line);
|
||||
request = doc.RootElement.Clone();
|
||||
op = request.ValueKind == JsonValueKind.Object
|
||||
&& request.TryGetProperty("op", out var opValue) && opValue.ValueKind == JsonValueKind.String
|
||||
? opValue.GetString() ?? ""
|
||||
: "";
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
await WriteAsync(pipe, Failure("request is not valid JSON"), stoppingToken);
|
||||
return;
|
||||
}
|
||||
|
||||
object response;
|
||||
try
|
||||
{
|
||||
response = op switch
|
||||
{
|
||||
"ping" => Ping(),
|
||||
"sendstatus" => await SendStatusAsync(callerName, stoppingToken),
|
||||
"sendreport" => await SendReportAsync(request, callerName, callerSid, stoppingToken),
|
||||
"alerts" => await ListAlertsAsync(stoppingToken),
|
||||
"ackalert" => await AckAlertAsync(request, callerName, stoppingToken),
|
||||
_ => Failure("unknown op"),
|
||||
};
|
||||
}
|
||||
catch (OperationCanceledException) when (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
response = Failure("the agent timed out talking to ShopDB");
|
||||
}
|
||||
|
||||
await WriteAsync(pipe, response, stoppingToken);
|
||||
}
|
||||
|
||||
private (string? Name, SecurityIdentifier? Sid) IdentifyCaller(NamedPipeServerStream pipe)
|
||||
{
|
||||
string? name = null;
|
||||
SecurityIdentifier? sid = null;
|
||||
try
|
||||
{
|
||||
name = pipe.GetImpersonationUserName();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "GetImpersonationUserName failed");
|
||||
}
|
||||
try
|
||||
{
|
||||
pipe.RunAsClient(() =>
|
||||
{
|
||||
using var identity = WindowsIdentity.GetCurrent(TokenAccessLevels.Query);
|
||||
sid = identity.User;
|
||||
});
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Could not read the pipe caller's SID");
|
||||
}
|
||||
return (name, sid);
|
||||
}
|
||||
|
||||
private object Ping()
|
||||
{
|
||||
var current = credentials.Current;
|
||||
return new
|
||||
{
|
||||
ok = true,
|
||||
agentversion = agentVersion,
|
||||
enrolled = current is not null,
|
||||
assetid = current?.AssetId,
|
||||
};
|
||||
}
|
||||
|
||||
private async Task<object> SendStatusAsync(string? caller, CancellationToken stoppingToken)
|
||||
{
|
||||
if (!await statusGate.WaitAsync(0, stoppingToken))
|
||||
return Failure("a status send is already running");
|
||||
try
|
||||
{
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
|
||||
timeout.CancelAfter(LongOperationTimeout);
|
||||
logger.LogInformation("Status send requested over the pipe by {Caller}", caller);
|
||||
var result = await telemetry.RunOnceAsync(config.AdoptOnFirstRun, timeout.Token);
|
||||
logger.LogInformation("Pipe status send complete: action={Action}", result.Action);
|
||||
return Success();
|
||||
}
|
||||
catch (CollectorException error)
|
||||
{
|
||||
logger.LogWarning("Pipe status send failed: HTTP {Status}", error.StatusCode);
|
||||
return Failure(ServerFailure(error));
|
||||
}
|
||||
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
|
||||
&& !stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
logger.LogWarning(error, "Pipe status send failed");
|
||||
return Failure("could not reach ShopDB");
|
||||
}
|
||||
finally
|
||||
{
|
||||
statusGate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
// Does what --report does, but only with files inside the CALLER's own
|
||||
// profile: the pipe is open to every interactive user and this process is
|
||||
// SYSTEM, so an unchecked path would upload any file SYSTEM can read. The
|
||||
// sidecar is not deleted here (the tray removes it); deleting in a user-
|
||||
// writable tree as SYSTEM is a redirection target.
|
||||
private async Task<object> SendReportAsync(JsonElement request, string? caller,
|
||||
SecurityIdentifier? callerSid, CancellationToken stoppingToken)
|
||||
{
|
||||
if (callerSid is null)
|
||||
return Failure("could not identify the calling user");
|
||||
|
||||
var spoolPath = request.TryGetProperty("spoolpath", out var pathValue)
|
||||
&& pathValue.ValueKind == JsonValueKind.String
|
||||
? pathValue.GetString()
|
||||
: null;
|
||||
if (string.IsNullOrWhiteSpace(spoolPath))
|
||||
return Failure("spoolpath is required");
|
||||
|
||||
var profile = ProfilePathFor(callerSid);
|
||||
if (profile is null)
|
||||
return Failure("no local profile for the calling user");
|
||||
|
||||
if (!await reportGate.WaitAsync(0, stoppingToken))
|
||||
return Failure("a report upload is already running");
|
||||
try
|
||||
{
|
||||
OperatorReportPayload? payload;
|
||||
try
|
||||
{
|
||||
using var sidecar = OpenCallerFile(spoolPath, profile, AgentPipeProtocol.MaxRequestBytes);
|
||||
payload = await JsonSerializer.DeserializeAsync<OperatorReportPayload>(
|
||||
sidecar, PayloadOptions, stoppingToken);
|
||||
}
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException
|
||||
or JsonException or ArgumentException
|
||||
or NotSupportedException)
|
||||
{
|
||||
logger.LogWarning("Refused report sidecar from {Caller}: {Reason}", caller, error.Message);
|
||||
return Failure("report file refused: " + error.Message);
|
||||
}
|
||||
if (payload is null)
|
||||
return Failure("report file is empty");
|
||||
|
||||
FileStream? screenshot = null;
|
||||
try
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(payload.ScreenshotPath))
|
||||
{
|
||||
if (!payload.ScreenshotPath.EndsWith(".png", StringComparison.OrdinalIgnoreCase))
|
||||
return Failure("screenshot must be a .png file");
|
||||
try
|
||||
{
|
||||
screenshot = OpenCallerFile(payload.ScreenshotPath, profile, MaxScreenshotBytes);
|
||||
}
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException
|
||||
or ArgumentException or NotSupportedException)
|
||||
{
|
||||
logger.LogWarning("Refused report screenshot from {Caller}: {Reason}",
|
||||
caller, error.Message);
|
||||
return Failure("screenshot refused: " + error.Message);
|
||||
}
|
||||
}
|
||||
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
|
||||
timeout.CancelAfter(LongOperationTimeout);
|
||||
var uploaded = await collector.UploadOperatorReportAsync(
|
||||
payload.Reason, payload.Details, screenshot,
|
||||
payload.ScreenshotPath is null ? null : Path.GetFileName(payload.ScreenshotPath),
|
||||
Environment.MachineName, timeout.Token);
|
||||
logger.LogInformation("Operator report from {Caller} uploaded: id={ReportId}",
|
||||
caller, uploaded?.OperatorReportId);
|
||||
return Success();
|
||||
}
|
||||
catch (CollectorException error)
|
||||
{
|
||||
logger.LogWarning("Operator report upload failed: HTTP {Status}", error.StatusCode);
|
||||
return Failure(ServerFailure(error));
|
||||
}
|
||||
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
|
||||
&& !stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
logger.LogWarning(error, "Operator report upload failed");
|
||||
return Failure("could not reach ShopDB");
|
||||
}
|
||||
finally
|
||||
{
|
||||
screenshot?.Dispose();
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
reportGate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<object> ListAlertsAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
|
||||
timeout.CancelAfter(ShortOperationTimeout);
|
||||
var list = await alerts.FetchAlertsAsync(Environment.MachineName, timeout.Token);
|
||||
return new { ok = true, alerts = list.ToArray() };
|
||||
}
|
||||
catch (CollectorException error)
|
||||
{
|
||||
logger.LogDebug("Alert fetch for the tray failed: HTTP {Status}", error.StatusCode);
|
||||
return Failure(ServerFailure(error));
|
||||
}
|
||||
catch (Exception error) when (error is HttpRequestException or TaskCanceledException or JsonException
|
||||
&& !stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
logger.LogDebug(error, "Alert fetch for the tray failed");
|
||||
return Failure("could not reach ShopDB");
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<object> AckAlertAsync(JsonElement request, string? caller, CancellationToken stoppingToken)
|
||||
{
|
||||
if (!request.TryGetProperty("notificationid", out var idValue)
|
||||
|| idValue.ValueKind != JsonValueKind.Number
|
||||
|| !idValue.TryGetInt32(out var notificationId)
|
||||
|| notificationId <= 0)
|
||||
return Failure("notificationid must be a positive integer");
|
||||
|
||||
try
|
||||
{
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
|
||||
timeout.CancelAfter(ShortOperationTimeout);
|
||||
// 0 = "the caller": the server resolves it from credential or hostname.
|
||||
var assetId = credentials.Current?.AssetId ?? 0;
|
||||
await alerts.AcknowledgeAlertAsync(notificationId, assetId, Environment.MachineName, timeout.Token);
|
||||
logger.LogInformation("Alert {NotificationId} acknowledged by {Caller}", notificationId, caller);
|
||||
return Success();
|
||||
}
|
||||
catch (CollectorException error)
|
||||
{
|
||||
logger.LogWarning("Alert {NotificationId} acknowledge failed: HTTP {Status}",
|
||||
notificationId, error.StatusCode);
|
||||
return Failure(ServerFailure(error));
|
||||
}
|
||||
catch (Exception error) when (error is HttpRequestException or TaskCanceledException
|
||||
&& !stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
logger.LogWarning(error, "Alert {NotificationId} acknowledge failed", notificationId);
|
||||
return Failure("could not reach ShopDB");
|
||||
}
|
||||
}
|
||||
|
||||
// Profile root for a SID from ProfileList. Null when absent.
|
||||
private static string? ProfilePathFor(SecurityIdentifier sid)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64);
|
||||
using var key = machine.OpenSubKey(
|
||||
@"SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList\" + sid.Value);
|
||||
if (key?.GetValue("ProfileImagePath") is not string raw || string.IsNullOrWhiteSpace(raw))
|
||||
return null;
|
||||
return Path.GetFullPath(Environment.ExpandEnvironmentVariables(raw)).TrimEnd('\\');
|
||||
}
|
||||
catch
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Open a caller-named file only if it sits inside the caller's profile, no
|
||||
// directory from the profile down is a reparse point, the file itself is
|
||||
// not one, and the opened handle's final path is still inside the profile.
|
||||
private static FileStream OpenCallerFile(string path, string profile, int maxBytes)
|
||||
{
|
||||
if (!Path.IsPathFullyQualified(path) || path.StartsWith(@"\\", StringComparison.Ordinal))
|
||||
throw new UnauthorizedAccessException("path must be a local absolute path");
|
||||
|
||||
var full = Path.GetFullPath(path);
|
||||
var root = profile + @"\";
|
||||
if (!full.StartsWith(root, StringComparison.OrdinalIgnoreCase))
|
||||
throw new UnauthorizedAccessException("path is outside the caller's profile");
|
||||
|
||||
var directory = Path.GetDirectoryName(full);
|
||||
while (directory is not null && directory.Length >= profile.Length)
|
||||
{
|
||||
if (ProtectedFileSystem.IsReparsePoint(directory))
|
||||
throw new UnauthorizedAccessException("path crosses a reparse point");
|
||||
directory = Path.GetDirectoryName(directory);
|
||||
}
|
||||
|
||||
var stream = ProtectedFileSystem.OpenReadNoFollow(full);
|
||||
try
|
||||
{
|
||||
var final = ProtectedFileSystem.FinalPath(stream.SafeFileHandle);
|
||||
if (final is null || !final.StartsWith(root, StringComparison.OrdinalIgnoreCase))
|
||||
throw new UnauthorizedAccessException("path resolves outside the caller's profile");
|
||||
if (stream.Length > maxBytes)
|
||||
throw new IOException($"file is larger than {maxBytes} bytes");
|
||||
return stream;
|
||||
}
|
||||
catch
|
||||
{
|
||||
stream.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
// Only SYSTEM (the service) and Administrators read appsettings.json now:
|
||||
// no user-context process needs it once the tray talks through this pipe.
|
||||
private void RestrictAppSettingsAcl()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var identity = WindowsIdentity.GetCurrent();
|
||||
if (!identity.IsSystem)
|
||||
return;
|
||||
|
||||
var path = Path.Combine(AppContext.BaseDirectory, "appsettings.json");
|
||||
if (ProtectedFileSystem.TryGetAttributes(path) is not FileAttributes attributes
|
||||
|| attributes.HasFlag(FileAttributes.ReparsePoint))
|
||||
return;
|
||||
|
||||
var security = new FileSecurity();
|
||||
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null),
|
||||
FileSystemRights.FullControl, AccessControlType.Allow));
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
new SecurityIdentifier(WellKnownSidType.BuiltinAdministratorsSid, null),
|
||||
FileSystemRights.FullControl, AccessControlType.Allow));
|
||||
new FileInfo(path).SetAccessControl(security);
|
||||
logger.LogInformation("appsettings.json restricted to SYSTEM and Administrators.");
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogWarning(error, "Could not restrict the appsettings.json ACL");
|
||||
}
|
||||
}
|
||||
|
||||
private static string ServerFailure(CollectorException error)
|
||||
=> string.IsNullOrEmpty(error.ServerCode)
|
||||
? $"ShopDB returned HTTP {error.StatusCode}"
|
||||
: $"ShopDB returned HTTP {error.StatusCode} {error.ServerCode}";
|
||||
|
||||
private static object Success() => new { ok = true };
|
||||
|
||||
private static object Failure(string error) => new { ok = false, error };
|
||||
|
||||
private static async Task WriteAsync(Stream pipe, object response, CancellationToken token)
|
||||
{
|
||||
try
|
||||
{
|
||||
var bytes = JsonSerializer.SerializeToUtf8Bytes(response);
|
||||
await pipe.WriteAsync(bytes, token);
|
||||
await pipe.WriteAsync(new[] { (byte)'\n' }, token);
|
||||
await pipe.FlushAsync(token);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Client went away.
|
||||
}
|
||||
}
|
||||
|
||||
private static async Task<bool> SafeDelay(TimeSpan delay, CancellationToken token)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(delay, token);
|
||||
return true;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,8 +5,9 @@ using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Posts one EndpointReport to POST {baseurl}/api/collector/endpoints with the
|
||||
// X-API-Key header (plugins/endpoints/AGENT-CONTRACT.md section 1), unwrapping
|
||||
// Posts one EndpointReport to POST {baseurl}/api/endpoints/collect (per-PC
|
||||
// credential) or /api/collector/endpoints (shared key), auth added by
|
||||
// AgentHttpFactory (plugins/endpoints/AGENT-CONTRACT.md section 1), unwrapping
|
||||
// the ShopDB envelope { status, data, meta } to return the CollectorResponse.
|
||||
//
|
||||
// Resilience: transient failures (network errors, timeouts, 5xx, 429) are
|
||||
@@ -27,17 +28,15 @@ public sealed class CollectorClient : IDisposable
|
||||
private readonly int maxAttempts;
|
||||
private readonly Random jitter = new();
|
||||
|
||||
public CollectorClient(AgentConfig config, ILogger<CollectorClient> logger,
|
||||
private readonly ICredentialProvider credentials;
|
||||
|
||||
public CollectorClient(AgentHttpFactory httpFactory, ILogger<CollectorClient> logger,
|
||||
int maxAttempts = 4)
|
||||
{
|
||||
this.logger = logger;
|
||||
this.maxAttempts = Math.Max(1, maxAttempts);
|
||||
http = new HttpClient
|
||||
{
|
||||
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
|
||||
Timeout = TimeSpan.FromSeconds(30),
|
||||
};
|
||||
http.DefaultRequestHeaders.Add("X-API-Key", config.ApiKey);
|
||||
credentials = httpFactory.Credentials;
|
||||
http = httpFactory.CreateClient(AgentHttpChannel.Collector);
|
||||
}
|
||||
|
||||
public async Task<CollectorResponse> SendAsync(
|
||||
@@ -76,8 +75,12 @@ public sealed class CollectorClient : IDisposable
|
||||
private async Task<CollectorResponse> SendOnceAsync(
|
||||
EndpointReport report, CancellationToken token)
|
||||
{
|
||||
using var response = await http.PostAsJsonAsync(
|
||||
"api/collector/endpoints", report, JsonOptions, token);
|
||||
// Enrolled agents post to the credential route; the shared key keeps
|
||||
// the core collector route so this build still reports to an older server.
|
||||
var route = credentials.Current is not null
|
||||
? "api/endpoints/collect"
|
||||
: "api/collector/endpoints";
|
||||
using var response = await http.PostAsJsonAsync(route, report, JsonOptions, token);
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
@@ -91,7 +94,7 @@ public sealed class CollectorClient : IDisposable
|
||||
|
||||
// Fetch the server-resolved watch list (policy-attached processes/services)
|
||||
// for this hostname, GET /api/endpoints/monitors/resolved. Shares the same
|
||||
// X-API-Key credential as telemetry. Best-effort: returns null on any
|
||||
// auth as telemetry. Best-effort: returns null on any
|
||||
// failure so the caller falls back to the last-known/local list rather than
|
||||
// blanking monitoring on a transient blip. A single attempt, no retry - the
|
||||
// next telemetry cycle retries naturally, and a stale-but-usable watch list
|
||||
@@ -134,49 +137,77 @@ public sealed class CollectorClient : IDisposable
|
||||
|
||||
// Uploads one operator issue report (free-text reason/details plus an
|
||||
// optional PNG screenshot) to POST /api/endpoints/operatorreports as
|
||||
// multipart/form-data, authorized by the same X-API-Key collector key as
|
||||
// telemetry. The server resolves this hostname to its asset. A single
|
||||
// multipart/form-data, authorized the same way as telemetry. The server
|
||||
// resolves this hostname (or the credential) to its asset. A single
|
||||
// attempt: the tray shows the operator the outcome and can retry by hand,
|
||||
// and a stuck retry loop holding a screenshot in memory is worse than a
|
||||
// clear failure. Returns the parsed upload response (new row id) or throws.
|
||||
public async Task<OperatorReportUploadResponse?> UploadOperatorReportAsync(
|
||||
string reason, string details, string? screenshotPath, string hostname,
|
||||
CancellationToken token)
|
||||
{
|
||||
FileStream? imageStream = null;
|
||||
try
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(screenshotPath) && File.Exists(screenshotPath))
|
||||
imageStream = File.OpenRead(screenshotPath);
|
||||
return await UploadOperatorReportAsync(reason, details, imageStream,
|
||||
screenshotPath is null ? null : Path.GetFileName(screenshotPath),
|
||||
hostname, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
imageStream?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
// Same upload from an already opened screenshot stream. The pipe path opens
|
||||
// the file itself (no reparse follow, final path checked) and hands it here.
|
||||
public async Task<OperatorReportUploadResponse?> UploadOperatorReportAsync(
|
||||
string reason, string details, Stream? screenshot, string? screenshotName,
|
||||
string hostname, CancellationToken token)
|
||||
{
|
||||
using var form = new MultipartFormDataContent();
|
||||
form.Add(new StringContent(reason ?? string.Empty), "reason");
|
||||
form.Add(new StringContent(details ?? string.Empty), "details");
|
||||
form.Add(new StringContent(hostname ?? string.Empty), "hostname");
|
||||
|
||||
StreamContent? imageContent = null;
|
||||
FileStream? imageStream = null;
|
||||
try
|
||||
if (screenshot is not null)
|
||||
{
|
||||
if (!string.IsNullOrWhiteSpace(screenshotPath) && File.Exists(screenshotPath))
|
||||
{
|
||||
imageStream = File.OpenRead(screenshotPath);
|
||||
imageContent = new StreamContent(imageStream);
|
||||
imageContent.Headers.ContentType =
|
||||
new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
|
||||
form.Add(imageContent, "screenshot",
|
||||
Path.GetFileName(screenshotPath));
|
||||
}
|
||||
|
||||
using var response = await http.PostAsync(
|
||||
"api/endpoints/operatorreports", form, token);
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
throw new CollectorException((int)response.StatusCode, body);
|
||||
|
||||
var envelope = JsonSerializer.Deserialize<OperatorReportUploadEnvelope>(
|
||||
body, JsonOptions);
|
||||
return envelope?.Data;
|
||||
}
|
||||
finally
|
||||
{
|
||||
imageContent?.Dispose();
|
||||
imageStream?.Dispose();
|
||||
// Not disposed with the form: the caller owns the stream.
|
||||
var imageContent = new StreamContent(new NonClosingStream(screenshot));
|
||||
imageContent.Headers.ContentType =
|
||||
new System.Net.Http.Headers.MediaTypeHeaderValue("image/png");
|
||||
form.Add(imageContent, "screenshot", screenshotName ?? "screenshot.png");
|
||||
}
|
||||
|
||||
using var response = await http.PostAsync(
|
||||
"api/endpoints/operatorreports", form, token);
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
throw new CollectorException((int)response.StatusCode, body);
|
||||
|
||||
var envelope = JsonSerializer.Deserialize<OperatorReportUploadEnvelope>(
|
||||
body, JsonOptions);
|
||||
return envelope?.Data;
|
||||
}
|
||||
|
||||
private sealed class NonClosingStream : Stream
|
||||
{
|
||||
private readonly Stream inner;
|
||||
public NonClosingStream(Stream inner) => this.inner = inner;
|
||||
public override bool CanRead => inner.CanRead;
|
||||
public override bool CanSeek => inner.CanSeek;
|
||||
public override bool CanWrite => false;
|
||||
public override long Length => inner.Length;
|
||||
public override long Position { get => inner.Position; set => inner.Position = value; }
|
||||
public override void Flush() { }
|
||||
public override int Read(byte[] buffer, int offset, int count) => inner.Read(buffer, offset, count);
|
||||
public override ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken cancellationToken = default)
|
||||
=> inner.ReadAsync(buffer, cancellationToken);
|
||||
public override long Seek(long offset, SeekOrigin origin) => inner.Seek(offset, origin);
|
||||
public override void SetLength(long value) => throw new NotSupportedException();
|
||||
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
|
||||
}
|
||||
|
||||
public void Dispose() => http.Dispose();
|
||||
@@ -193,9 +224,15 @@ public sealed class CollectorException : Exception
|
||||
StatusCode >= 400 && StatusCode < 500
|
||||
&& StatusCode != (int)HttpStatusCode.TooManyRequests;
|
||||
|
||||
public string Body { get; }
|
||||
|
||||
// Error code from the server's JSON envelope (e.g. CREDENTIAL_INVALID), or null.
|
||||
public string? ServerCode => AgentServerError.TryParse(Body)?.Code;
|
||||
|
||||
public CollectorException(int statusCode, string body)
|
||||
: base($"collector returned HTTP {statusCode}: {body}")
|
||||
{
|
||||
StatusCode = statusCode;
|
||||
Body = body;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -10,9 +10,8 @@ namespace ShopDbEndpointAgent.Core;
|
||||
// GET /api/endpoints/commands/next, which the server holds open until a command
|
||||
// is queued (HTTP 200) or the hold window elapses (HTTP 204). After running the
|
||||
// claimed command locally the agent posts the outcome to
|
||||
// POST /api/endpoints/commands/{id}/result. Both share the endpoints collector
|
||||
// key via X-API-Key, so the command channel rides the same kill switch as
|
||||
// telemetry and adds no second secret.
|
||||
// POST /api/endpoints/commands/{id}/result. Both use the same auth as telemetry
|
||||
// (per-PC credential, else the shared collector key), added by AgentHttpFactory.
|
||||
public sealed class CommandClient : IDisposable
|
||||
{
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
@@ -22,33 +21,13 @@ public sealed class CommandClient : IDisposable
|
||||
};
|
||||
|
||||
private readonly HttpClient http;
|
||||
private readonly SocketsHttpHandler handler;
|
||||
private readonly ILogger<CommandClient> logger;
|
||||
|
||||
public CommandClient(AgentConfig config, ILogger<CommandClient> logger)
|
||||
public CommandClient(AgentHttpFactory httpFactory, ILogger<CommandClient> logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
// The command channel long-polls: the server holds each claim open for a
|
||||
// hold window and then, between windows, the pooled TLS connection sits
|
||||
// idle. Gunicorn's keep-alive closes an idle socket after a couple of
|
||||
// seconds, so a connection reused after that yields "unexpected EOF or 0
|
||||
// bytes from the transport stream" on the next TLS handshake. Retire
|
||||
// pooled connections faster than the server's keep-alive so a socket the
|
||||
// server has already dropped is never handed back out; the lifetime cap
|
||||
// is a hard backstop so no connection lingers indefinitely.
|
||||
handler = new SocketsHttpHandler
|
||||
{
|
||||
PooledConnectionIdleTimeout = TimeSpan.FromSeconds(1),
|
||||
PooledConnectionLifetime = TimeSpan.FromMinutes(2),
|
||||
};
|
||||
http = new HttpClient(handler, disposeHandler: true)
|
||||
{
|
||||
BaseAddress = new Uri(config.BaseUrl.TrimEnd('/') + "/"),
|
||||
// Longer than the collector client: the claim request is held open
|
||||
// server-side (~25s), so the read timeout must clear that plus slack.
|
||||
Timeout = TimeSpan.FromSeconds(60),
|
||||
};
|
||||
http.DefaultRequestHeaders.Add("X-API-Key", config.ApiKey);
|
||||
// Pooling tuned for the long-poll lives in AgentHttpFactory (Command).
|
||||
http = httpFactory.CreateClient(AgentHttpChannel.Command);
|
||||
}
|
||||
|
||||
// Claim the next pending command for this hostname, or null when the server
|
||||
|
||||
@@ -41,15 +41,20 @@ public sealed class CommandWorker : BackgroundService
|
||||
private readonly CommandExecutor executor;
|
||||
private readonly ILogger<CommandWorker> logger;
|
||||
private readonly CollectSignal? collectSignal;
|
||||
private readonly AgentConfig? config;
|
||||
private readonly string hostname;
|
||||
private readonly EnrollmentService? enrollment;
|
||||
|
||||
public CommandWorker(CommandClient client, CommandExecutor executor,
|
||||
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null)
|
||||
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null,
|
||||
AgentConfig? config = null, EnrollmentService? enrollment = null)
|
||||
{
|
||||
this.enrollment = enrollment;
|
||||
this.client = client;
|
||||
this.executor = executor;
|
||||
this.logger = logger;
|
||||
this.collectSignal = collectSignal;
|
||||
this.config = config;
|
||||
hostname = Environment.MachineName;
|
||||
}
|
||||
|
||||
@@ -57,6 +62,30 @@ public sealed class CommandWorker : BackgroundService
|
||||
{
|
||||
logger.LogInformation("Command channel started for host {Host}", hostname);
|
||||
|
||||
// Spread first contact across the fleet. Every agent otherwise polls the
|
||||
// instant its service starts, and the service is start=auto, so a patch
|
||||
// reboot puts the whole fleet on the wire together. See StartupJitter.
|
||||
var startupDelay = StartupJitter.Next(config?.StartupJitterSeconds ?? 0, logger);
|
||||
if (startupDelay > TimeSpan.Zero)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Delaying first command poll by {Delay}ms (startup jitter)",
|
||||
(int)startupDelay.TotalMilliseconds);
|
||||
await SafeDelay(startupDelay, stoppingToken);
|
||||
}
|
||||
|
||||
if (enrollment is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
await enrollment.WaitForFirstAttemptAsync(TimeSpan.FromSeconds(20), stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
var pollStarted = System.Diagnostics.Stopwatch.StartNew();
|
||||
|
||||
51
agent/ShopDbEndpointAgent.Core/ConsoleWindow.cs
Normal file
@@ -0,0 +1,51 @@
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Hides the console window this process was given.
|
||||
//
|
||||
// shopdbagent.exe is a console executable, so anything that starts it in an
|
||||
// interactive session gets a visible black window. That was invisible while the
|
||||
// tray owned the probe - it launched with CreateNoWindow - but the user-context
|
||||
// scheduled task runs the exe directly in the operator's session, where the
|
||||
// window is very much visible and pops up on every collection cycle.
|
||||
//
|
||||
// The window is hidden rather than the exe rebuilt as a WinExe: the agent is
|
||||
// also a service host and a command-line tool, and both of those want a real
|
||||
// console for --once, --report and service debugging.
|
||||
public static class ConsoleWindow
|
||||
{
|
||||
private const int SW_HIDE = 0;
|
||||
|
||||
[DllImport("kernel32.dll")]
|
||||
private static extern nint GetConsoleWindow();
|
||||
|
||||
[DllImport("user32.dll")]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool ShowWindow(nint window, int command);
|
||||
|
||||
// Best-effort: a process with no console (already detached, or redirected by
|
||||
// a caller) simply has nothing to hide, and failing to hide a window must
|
||||
// never stop the collection that window belongs to.
|
||||
public static void Hide()
|
||||
{
|
||||
if (!OperatingSystem.IsWindows())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var window = GetConsoleWindow();
|
||||
if (window != nint.Zero)
|
||||
{
|
||||
ShowWindow(window, SW_HIDE);
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Nothing useful to do or say: there is no log sink yet at the point
|
||||
// this runs, and a visible window is cosmetic next to losing data.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,7 @@ public sealed class EndpointReport
|
||||
[JsonPropertyName("ipaddress")] public string? IpAddress { get; set; }
|
||||
[JsonPropertyName("machinenumber")] public string? MachineNumber { get; set; }
|
||||
[JsonPropertyName("agentversion")] public string? AgentVersion { get; set; }
|
||||
[JsonPropertyName("trayversion")] public string? TrayVersion { get; set; }
|
||||
[JsonPropertyName("lastboottime")] public DateTimeOffset? LastBootTime { get; set; }
|
||||
[JsonPropertyName("uptimeseconds")] public long? UptimeSeconds { get; set; }
|
||||
[JsonPropertyName("idleseconds")] public long? IdleSeconds { get; set; }
|
||||
@@ -43,11 +44,57 @@ public sealed class EndpointReport
|
||||
// collection: both Uninstall registry hives, {displayname, displayversion,
|
||||
// publisher}. Server logs it as an unpersisted key until the SBOM store lands.
|
||||
[JsonPropertyName("applications")] public object[]? Applications { get; set; }
|
||||
|
||||
// True when Windows is holding a reboot. A machine that has been waiting to
|
||||
// reboot for weeks explains stalled patching and "the fix did not take"
|
||||
// reports, and on a shop floor PC nobody reboots voluntarily.
|
||||
[JsonPropertyName("pendingreboot")] public bool? PendingReboot { get; set; }
|
||||
|
||||
// Which signal(s) drove pendingreboot, so the server can tell a box staged
|
||||
// mid-servicing (pendingfilerename) from one with a stale servicing key.
|
||||
// Empty array = checked, nothing pending. Omitted = the check failed.
|
||||
[JsonPropertyName("rebootreasons")] public string[]? RebootReasons { get; set; }
|
||||
|
||||
// hiberbootenabled (fast startup configured) and lastboottype (what the
|
||||
// last boot actually did). Uptime is not trustworthy when these disagree.
|
||||
[JsonPropertyName("bootinfo")] public object? BootInfo { get; set; }
|
||||
|
||||
// Defender health. Persisted but deliberately not surfaced: known false
|
||||
// positives in this environment.
|
||||
[JsonPropertyName("defender")] public object? Defender { get; set; }
|
||||
|
||||
// Physical drive health from two sources: MSFT_PhysicalDisk (the only one
|
||||
// that reports NVMe) and Win32_DiskDrive.Status (ATA/SATA and fallback).
|
||||
// Each entry carries a precomputed predictfailure so an alert is a boolean
|
||||
// test rather than a string match against an enum. Null predictfailure
|
||||
// means the drive did not answer - unknown, not healthy.
|
||||
[JsonPropertyName("diskhealth")] public object[]? DiskHealth { get; set; }
|
||||
|
||||
// Session-scoped data gathered by a probe running as the signed-in user:
|
||||
// mapped drives and user-owned listeners. Null when nobody is signed in or
|
||||
// the probe did not produce a result this cycle - absent means "not
|
||||
// observed", which is deliberately distinct from an empty list.
|
||||
[JsonPropertyName("usercontext")] public object? UserContext { get; set; }
|
||||
|
||||
// Pending Dell Command | Update results, produced by the shopdb-dellupdates
|
||||
// agent utility rather than collected in C#: the data comes from dcu-cli's
|
||||
// own scan output, so the utility that parses it belongs next to the task
|
||||
// that writes it. Null means no scan has ever run (or the utility is not
|
||||
// installed), which is deliberately distinct from a scan that found nothing
|
||||
// pending - the first is "unknown", the second is "known clean".
|
||||
[JsonPropertyName("dellupdates")] public object? DellUpdates { get; set; }
|
||||
}
|
||||
|
||||
public sealed class EndpointEventRecord
|
||||
{
|
||||
[JsonPropertyName("eventid")] public int EventId { get; set; }
|
||||
|
||||
// The Windows event record id: monotonic and unique within its channel on a
|
||||
// given machine. It is both the agent's harvest watermark and the server's
|
||||
// dedupe key, so the same record can never be stored twice. Null only when
|
||||
// the source record did not expose one.
|
||||
[JsonPropertyName("eventrecordid")] public long? EventRecordId { get; set; }
|
||||
|
||||
[JsonPropertyName("eventchannel")] public string? EventChannel { get; set; }
|
||||
[JsonPropertyName("eventsource")] public string? EventSource { get; set; }
|
||||
[JsonPropertyName("eventtime")] public DateTimeOffset? EventTime { get; set; }
|
||||
@@ -135,6 +182,28 @@ public sealed class AppDeliveryClaim
|
||||
// downloads with its collector credential; Sha256 pins the exact bytes;
|
||||
// RunContext is 'system' (the agent's own service account) or 'user' (the
|
||||
// interactive logged-on session), decided per script in the ShopDB library.
|
||||
// WorkingDirectory is the optional folder to run the script IN. Empty means the
|
||||
// agent's staging directory, which is what every script did before the field
|
||||
// existed. It is there so a script can sit beside a corporate tool already
|
||||
// installed on the endpoint and invoke it by relative path; it confers no
|
||||
// rights, since the body is already arbitrary PowerShell that could change its
|
||||
// own directory.
|
||||
// ErrorMode is 'default' or 'strict'. Default is what PowerShell does unaided:
|
||||
// a non-terminating error is written and the script continues, so powershell.exe
|
||||
// still exits 0 and the command reports SUCCEEDED even when the body did none of
|
||||
// its work. Strict makes the agent prepend $ErrorActionPreference = 'Stop' so the
|
||||
// first error terminates and the exit code means what an operator reads it to
|
||||
// mean. Per script, never global: strict changes execution semantics, and a body
|
||||
// written to tolerate a failing cleanup or to probe with a cmdlet that errors on
|
||||
// absence would start aborting halfway. An absent or unrecognised value is
|
||||
// treated as default, so an old server that never sends the field is safe.
|
||||
// CaptureOutput decides whether a SUCCESSFUL run returns its output to the
|
||||
// server. Failures always return it, because a failure has to be diagnosable
|
||||
// without walking to the machine. Success is opt-in per script: output carries
|
||||
// whatever the body printed, and that is not worth persisting server-side for a
|
||||
// routine green run. The endpoint log keeps the full copy regardless, so this
|
||||
// only governs what crosses the wire. Absent means false, which is the same
|
||||
// behaviour as before the field existed.
|
||||
public sealed class ScriptDeliveryClaim
|
||||
{
|
||||
[JsonPropertyName("scriptid")] public int ScriptId { get; set; }
|
||||
@@ -142,6 +211,9 @@ public sealed class ScriptDeliveryClaim
|
||||
[JsonPropertyName("installpath")] public string? InstallPath { get; set; }
|
||||
[JsonPropertyName("sha256")] public string? Sha256 { get; set; }
|
||||
[JsonPropertyName("runcontext")] public string? RunContext { get; set; }
|
||||
[JsonPropertyName("workingdirectory")] public string? WorkingDirectory { get; set; }
|
||||
[JsonPropertyName("errormode")] public string? ErrorMode { get; set; }
|
||||
[JsonPropertyName("captureoutput")] public bool CaptureOutput { get; set; }
|
||||
}
|
||||
|
||||
// Envelope for the claim long-poll: { status, data: { endpointcommandid, action } }.
|
||||
|
||||
@@ -27,7 +27,15 @@ public static class CuratedEventCatalog
|
||||
new(55, "System", "NTFS file system corruption"),
|
||||
new(7011, "System", "Service Control Manager timeout"),
|
||||
new(10016, "System", "DCOM permission error"),
|
||||
// Windows Error Reporting on the Application channel: 1000 is the
|
||||
// faulting-application record (app, version, faulting module, exception
|
||||
// code), 1001 is the WER bucket record Windows pairs with it, and 1002
|
||||
// is an unresponsive app rather than a crash. The structured detail for
|
||||
// all three arrives in the record's EventData, which the collector lifts
|
||||
// into `fields`.
|
||||
new(1000, "Application", "Application error / faulting module"),
|
||||
new(1001, "Application", "Application error / bugcheck report"),
|
||||
new(1002, "Application", "Application hang"),
|
||||
};
|
||||
|
||||
// The distinct channels the collector must read.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
using System.Diagnostics;
|
||||
using System.Diagnostics.Eventing.Reader;
|
||||
using System.Management;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Win32;
|
||||
@@ -19,16 +22,28 @@ public sealed class EndpointCollector : ICollector
|
||||
// keep the payload bounded on a chatty Security log.
|
||||
private const int MaxEventsPerChannel = 25;
|
||||
|
||||
// The spool file is read every cycle. Reading it is a small file read, and
|
||||
// the tray - not this cadence - decides how often the data is refreshed, so
|
||||
// throttling the read here would only delay already-collected data reaching
|
||||
// the server without saving any real work.
|
||||
|
||||
private readonly ILogger<EndpointCollector> logger;
|
||||
private readonly AgentConfig config;
|
||||
private readonly IAgentUtilityRunner? utilities;
|
||||
private readonly EventWatermarkStore? watermarks;
|
||||
private readonly UserContextSpool? userContextSpool;
|
||||
|
||||
private object? lastUserContext;
|
||||
|
||||
public EndpointCollector(ILogger<EndpointCollector> logger, AgentConfig config,
|
||||
IAgentUtilityRunner? utilities = null)
|
||||
IAgentUtilityRunner? utilities = null, EventWatermarkStore? watermarks = null,
|
||||
UserContextSpool? userContextSpool = null)
|
||||
{
|
||||
this.logger = logger;
|
||||
this.config = config;
|
||||
this.utilities = utilities;
|
||||
this.watermarks = watermarks;
|
||||
this.userContextSpool = userContextSpool;
|
||||
}
|
||||
|
||||
public EndpointReport Collect(string agentVersion, bool adopt)
|
||||
@@ -37,6 +52,7 @@ public sealed class EndpointCollector : ICollector
|
||||
{
|
||||
Hostname = Environment.MachineName,
|
||||
AgentVersion = agentVersion,
|
||||
TrayVersion = GetTrayVersion(),
|
||||
Adopt = adopt ? true : null,
|
||||
LoggedInUser = GetLoggedInUser(),
|
||||
SerialNumber = GetSerialNumber(),
|
||||
@@ -48,23 +64,260 @@ public sealed class EndpointCollector : ICollector
|
||||
MachineNumber = GetMachineNumber(),
|
||||
LastBootTime = GetLastBootTime(),
|
||||
Services = GetMonitoredTargets(),
|
||||
Ports = GetListeningPorts(),
|
||||
Interfaces = GetInterfaces(),
|
||||
Events = GetCuratedEvents(),
|
||||
Disks = GetDisks(),
|
||||
DiskHealth = GetDiskHealth(),
|
||||
Bios = GetBios(),
|
||||
Memory = GetMemory(),
|
||||
Processor = GetProcessor(),
|
||||
OsInfo = GetOsInfo(),
|
||||
Domain = GetDomain(),
|
||||
Applications = GetApplications(),
|
||||
PendingReboot = GetPendingReboot(),
|
||||
RebootReasons = GetRebootReasons(),
|
||||
BootInfo = GetBootInfo(),
|
||||
Defender = GetDefender(),
|
||||
UserContext = GetUserContext(),
|
||||
};
|
||||
|
||||
if (report.LastBootTime is DateTimeOffset boot)
|
||||
report.UptimeSeconds = (long)(DateTimeOffset.UtcNow - boot).TotalSeconds;
|
||||
|
||||
report.CpuPercent = GetCpuPercent();
|
||||
report.MemoryPercent = GetMemoryPercent();
|
||||
report.IdleSeconds = GetIdleSecondsFromUserContext();
|
||||
|
||||
return report;
|
||||
}
|
||||
|
||||
// Instantaneous CPU load from the _Total performance instance.
|
||||
//
|
||||
// Uses the WMI formatted-data class rather than PerformanceCounter because a
|
||||
// PerformanceCounter reads 0 on its first sample and needs a second sample a
|
||||
// second later to mean anything. Sleeping a second inside the collect path
|
||||
// would delay the heartbeat on every tick for one number. The formatted-data
|
||||
// class is already averaged by the OS, so a single read is usable.
|
||||
private double? GetCpuPercent()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
"SELECT PercentProcessorTime FROM " +
|
||||
"Win32_PerfFormattedData_PerfOS_Processor WHERE Name='_Total'");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
var value = ToLong(mo["PercentProcessorTime"]);
|
||||
if (value is long percent)
|
||||
return Math.Clamp(percent, 0, 100);
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "CPU percent query failed");
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Percentage of physical memory in use. Reported alongside the absolute
|
||||
// byte counts in the memory object because the percentage is what an alert
|
||||
// threshold keys off, and deriving it server-side would break whenever the
|
||||
// byte fields are absent.
|
||||
private double? GetMemoryPercent()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
"SELECT TotalVisibleMemorySize, FreePhysicalMemory " +
|
||||
"FROM Win32_OperatingSystem");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
var totalKb = ToLong(mo["TotalVisibleMemorySize"]);
|
||||
var freeKb = ToLong(mo["FreePhysicalMemory"]);
|
||||
if (totalKb is long total && total > 0 && freeKb is long free)
|
||||
{
|
||||
var used = (double)(total - free) / total * 100.0;
|
||||
return Math.Round(Math.Clamp(used, 0, 100), 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Memory percent query failed");
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Idle time is measured by the user-context probe, because GetLastInputInfo
|
||||
// is per session and this service runs in session 0. Null when no probe
|
||||
// result is available, which correctly reads as "not observed" rather than
|
||||
// "the machine has been idle for zero seconds".
|
||||
private long? GetIdleSecondsFromUserContext()
|
||||
{
|
||||
if (lastUserContext is not Dictionary<string, object?> context)
|
||||
return null;
|
||||
if (!context.TryGetValue("idleseconds", out var raw) || raw is null)
|
||||
return null;
|
||||
|
||||
try
|
||||
{
|
||||
if (raw is JsonElement element)
|
||||
{
|
||||
return element.ValueKind == JsonValueKind.Number
|
||||
&& element.TryGetInt64(out var parsed) ? parsed : null;
|
||||
}
|
||||
return Convert.ToInt64(raw);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Could not read idleseconds from the user context");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Listening TCP ports with the owning process name.
|
||||
//
|
||||
// Only listeners, never established connections: listeners answer "what
|
||||
// service is this machine offering" (the DNC port, the MTConnect adapter),
|
||||
// which is stable and small. Established connections churn constantly and
|
||||
// would make every report differ from the last for no diagnostic gain, so
|
||||
// the full table is exposed as an on-demand command instead.
|
||||
private const int MaxListeners = 100;
|
||||
|
||||
private object[] GetListeningPorts()
|
||||
{
|
||||
try
|
||||
{
|
||||
var listeners = TcpTableReader.GetListeners();
|
||||
if (listeners.Count == 0)
|
||||
return Array.Empty<object>();
|
||||
|
||||
var names = GetProcessNames();
|
||||
var result = new List<object>();
|
||||
|
||||
foreach (var listener in listeners)
|
||||
{
|
||||
if (result.Count >= MaxListeners)
|
||||
{
|
||||
logger.LogDebug(
|
||||
"Listener list truncated at {Limit} entries.", MaxListeners);
|
||||
break;
|
||||
}
|
||||
|
||||
names.TryGetValue(listener.ProcessId, out var processName);
|
||||
result.Add(new
|
||||
{
|
||||
localaddress = listener.LocalAddress,
|
||||
localport = listener.LocalPort,
|
||||
processid = listener.ProcessId,
|
||||
processname = processName,
|
||||
protocol = "tcp",
|
||||
});
|
||||
}
|
||||
|
||||
return result.ToArray();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Listening port enumeration failed");
|
||||
return Array.Empty<object>();
|
||||
}
|
||||
}
|
||||
|
||||
// Snapshot the process table once and index by id. Resolving each listener
|
||||
// individually would be an O(n) call per row and can race with process exit.
|
||||
private Dictionary<int, string> GetProcessNames()
|
||||
{
|
||||
var names = new Dictionary<int, string>();
|
||||
try
|
||||
{
|
||||
foreach (var process in Process.GetProcesses())
|
||||
{
|
||||
try
|
||||
{
|
||||
names[process.Id] = process.ProcessName;
|
||||
}
|
||||
catch
|
||||
{
|
||||
// The process exited between enumeration and read.
|
||||
}
|
||||
finally
|
||||
{
|
||||
process.Dispose();
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Process name snapshot failed");
|
||||
}
|
||||
return names;
|
||||
}
|
||||
|
||||
// Session-scoped data, read from the spool file the tray writes. The service
|
||||
// only reads: it cannot collect this itself, because mapped drives and idle
|
||||
// time do not exist in session 0. Between reads the previous result is
|
||||
// repeated, because a mapped drive stays mapped and dropping the key would
|
||||
// read on the server as "the drive went away". The spool's own age limit is
|
||||
// what eventually retires a result, so a logged-off machine goes quiet
|
||||
// instead of reporting the last operator forever.
|
||||
private object? GetUserContext()
|
||||
{
|
||||
if (userContextSpool is null)
|
||||
return null;
|
||||
|
||||
var resultPath = userContextSpool.TryGetResultPath(out var reason);
|
||||
if (resultPath is null)
|
||||
{
|
||||
logger.LogDebug("Skipping the user-context read: {Reason}", reason);
|
||||
return lastUserContext;
|
||||
}
|
||||
|
||||
// Peek rather than consume: the probe overwrites the spool on every run,
|
||||
// so the newest result is always the one on disk. The file deliberately
|
||||
// survives a service restart, which the previous delete-on-read did not -
|
||||
// lastUserContext is in-memory only, so a restart used to leave the agent
|
||||
// with no drives to report until the tray's next probe up to 30 minutes
|
||||
// later. lastUserContext still repeats the previous value between reads,
|
||||
// because the tray collects on a slower cadence than telemetry and
|
||||
// dropping the key in between would read on the server as "the drives
|
||||
// went away".
|
||||
var collected = userContextSpool.ConsumeResult(resultPath);
|
||||
if (collected is not null)
|
||||
lastUserContext = collected;
|
||||
|
||||
return lastUserContext;
|
||||
}
|
||||
|
||||
// The interactive tray app ships and updates independently of this service,
|
||||
// so the agent reports the tray's file version as its own field. Resolve the
|
||||
// tray exe beside the agent first (the normal install layout), then the
|
||||
// default install dir; degrade to null when it is absent or unreadable so a
|
||||
// tray-less install simply omits the field rather than failing the report.
|
||||
private const string TrayExeName = "shopdb-endpoint-tray.exe";
|
||||
private const string DefaultInstallDir = @"C:\Program Files\ShopDB\EndpointAgent";
|
||||
|
||||
private string? GetTrayVersion()
|
||||
{
|
||||
try
|
||||
{
|
||||
var beside = Path.Combine(AppContext.BaseDirectory, TrayExeName);
|
||||
var installed = Path.Combine(DefaultInstallDir, TrayExeName);
|
||||
var trayExe = File.Exists(beside) ? beside
|
||||
: File.Exists(installed) ? installed
|
||||
: null;
|
||||
if (trayExe is null)
|
||||
return null;
|
||||
return FileVersionInfo.GetVersionInfo(trayExe).FileVersion;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogDebug(ex, "could not resolve tray version");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Combined watch list: interactive processes (by image name) and Windows
|
||||
// services (by service key name). Both land in report.Services with a shared
|
||||
// {name, app, kind, state, running, count} shape so the server/UI renders
|
||||
@@ -267,6 +520,7 @@ public sealed class EndpointCollector : ICollector
|
||||
private object[] GetInterfaces()
|
||||
{
|
||||
var result = new List<object>();
|
||||
var linkstate = GetLinkState();
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
@@ -292,6 +546,10 @@ public sealed class EndpointCollector : ICollector
|
||||
var gateway = gateways is null || gateways.Length == 0
|
||||
? null : gateways[0];
|
||||
|
||||
// Paired by MAC from MSFT_NetAdapter. Absent when that class is
|
||||
// unavailable, which must leave the rest of the entry intact.
|
||||
linkstate.TryGetValue(NormalizeMac(mac) ?? "", out var link);
|
||||
|
||||
result.Add(new
|
||||
{
|
||||
alias = mo["Description"]?.ToString()?.Trim() ?? "",
|
||||
@@ -303,6 +561,10 @@ public sealed class EndpointCollector : ICollector
|
||||
gateway = gateway ?? "",
|
||||
subnet = subnet ?? "",
|
||||
dhcp = mo["DHCPEnabled"] as bool? ?? false,
|
||||
// Null rather than 0/false when link state was not
|
||||
// readable: an unknown duplex must not read as half.
|
||||
linkspeedbps = link.Speed,
|
||||
fullduplex = link.FullDuplex,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -422,40 +684,33 @@ public sealed class EndpointCollector : ICollector
|
||||
}
|
||||
}
|
||||
|
||||
// Walk each curated channel newest-first, keep only the curated IDs for
|
||||
// that channel, cap per channel. Reads Security/System/Application - the
|
||||
// old scaffold read System only and filtered by the wrong IDs.
|
||||
// Walk each curated channel newest-first and report ONLY records newer than
|
||||
// the stored watermark, so a record crosses the wire exactly once.
|
||||
//
|
||||
// Two things changed here versus the original scaffold. First, filtering by
|
||||
// event id now happens inside the event log service through an XPath query
|
||||
// instead of walking every entry in the channel from C#; on a busy Security
|
||||
// log that is the difference between touching a handful of records and
|
||||
// hundreds of thousands. Second, EventLogReader (not the legacy EventLog
|
||||
// API) exposes RecordId, which is what makes exactly-once possible.
|
||||
//
|
||||
// First contact on a channel has no watermark, so it sends a bounded tail
|
||||
// (MaxEventsPerChannel) for initial context rather than the whole backlog.
|
||||
// The same cap also bounds a burst: if more than the cap arrived since the
|
||||
// last cycle, the NEWEST are taken and the watermark still jumps to the
|
||||
// newest, so the payload can never balloon.
|
||||
private EndpointEventRecord[] GetCuratedEvents()
|
||||
{
|
||||
var records = new List<EndpointEventRecord>();
|
||||
foreach (var channel in CuratedEventCatalog.Channels)
|
||||
{
|
||||
var wanted = CuratedEventCatalog.IdsForChannel(channel);
|
||||
if (wanted.Count == 0)
|
||||
continue;
|
||||
|
||||
try
|
||||
{
|
||||
using var log = new EventLog(channel);
|
||||
int kept = 0;
|
||||
for (int i = log.Entries.Count - 1;
|
||||
i >= 0 && kept < MaxEventsPerChannel; i--)
|
||||
{
|
||||
EventLogEntry entry;
|
||||
try { entry = log.Entries[i]; } catch { continue; }
|
||||
|
||||
int id = (int)(entry.InstanceId & 0xFFFF);
|
||||
if (!wanted.Contains(id))
|
||||
continue;
|
||||
|
||||
records.Add(new EndpointEventRecord
|
||||
{
|
||||
EventId = id,
|
||||
EventChannel = channel,
|
||||
EventSource = entry.Source,
|
||||
EventTime = new DateTimeOffset(
|
||||
entry.TimeGenerated.ToUniversalTime(), TimeSpan.Zero),
|
||||
EventMessage = Truncate(entry.Message, 1000),
|
||||
});
|
||||
kept++;
|
||||
}
|
||||
records.AddRange(ReadChannel(channel, wanted));
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
@@ -464,9 +719,172 @@ public sealed class EndpointCollector : ICollector
|
||||
logger.LogDebug(error, "Event channel {Channel} unreadable", channel);
|
||||
}
|
||||
}
|
||||
|
||||
watermarks?.Save();
|
||||
return records.ToArray();
|
||||
}
|
||||
|
||||
private IEnumerable<EndpointEventRecord> ReadChannel(
|
||||
string channel, IReadOnlySet<int> wanted)
|
||||
{
|
||||
long watermark = watermarks?.GetWatermark(channel) ?? 0;
|
||||
bool seeding = watermarks is null || !watermarks.HasWatermark(channel);
|
||||
|
||||
var query = new EventLogQuery(channel, PathType.LogName, BuildIdFilter(wanted))
|
||||
{
|
||||
// Newest-first so the cap keeps the most recent records and we can
|
||||
// stop as soon as we reach the watermark.
|
||||
ReverseDirection = true,
|
||||
};
|
||||
|
||||
var found = new List<EndpointEventRecord>();
|
||||
long newestSeen = 0;
|
||||
|
||||
using var reader = new EventLogReader(query);
|
||||
while (found.Count < MaxEventsPerChannel)
|
||||
{
|
||||
EventRecord? entry;
|
||||
try { entry = reader.ReadEvent(); }
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Event read failed on {Channel}", channel);
|
||||
break;
|
||||
}
|
||||
|
||||
if (entry is null)
|
||||
break;
|
||||
|
||||
using (entry)
|
||||
{
|
||||
long recordId = entry.RecordId ?? 0;
|
||||
if (recordId > newestSeen)
|
||||
newestSeen = recordId;
|
||||
|
||||
// Reverse order means everything from here down is older than
|
||||
// the watermark and has already been reported.
|
||||
if (!seeding && recordId != 0 && recordId <= watermark)
|
||||
break;
|
||||
|
||||
found.Add(ToEventRecord(entry, channel, recordId));
|
||||
}
|
||||
}
|
||||
|
||||
// A cleared log restarts record ids low; reseed rather than go silent.
|
||||
if (!seeding && newestSeen > 0
|
||||
&& watermarks is not null && watermarks.ResetIfStale(channel, newestSeen))
|
||||
{
|
||||
return found;
|
||||
}
|
||||
|
||||
if (newestSeen > 0)
|
||||
watermarks?.Advance(channel, newestSeen);
|
||||
|
||||
return found;
|
||||
}
|
||||
|
||||
private EndpointEventRecord ToEventRecord(
|
||||
EventRecord entry, string channel, long recordId)
|
||||
{
|
||||
string? message = null;
|
||||
try { message = entry.FormatDescription(); }
|
||||
catch (Exception error)
|
||||
{
|
||||
// A provider whose message DLL is missing cannot render text; the
|
||||
// id, time, and source are still worth reporting.
|
||||
logger.LogDebug(error, "Event message not renderable on {Channel}", channel);
|
||||
}
|
||||
|
||||
return new EndpointEventRecord
|
||||
{
|
||||
EventId = entry.Id,
|
||||
EventRecordId = recordId == 0 ? null : recordId,
|
||||
EventChannel = channel,
|
||||
EventSource = entry.ProviderName,
|
||||
EventTime = entry.TimeCreated is DateTime created
|
||||
? new DateTimeOffset(created.ToUniversalTime(), TimeSpan.Zero)
|
||||
: null,
|
||||
EventMessage = Truncate(message, 1000),
|
||||
Fields = GetEventFields(entry, channel),
|
||||
};
|
||||
}
|
||||
|
||||
// Bounds on the structured field extraction. The rendered message is often
|
||||
// a paragraph of prose while the useful specifics (which account, which
|
||||
// logon type, which faulting module) live in EventData, so it is worth
|
||||
// carrying - but a single record can declare a lot of elements and some
|
||||
// providers emit large blobs, so both the count and each value are capped.
|
||||
private const int MaxEventFields = 30;
|
||||
private const int MaxEventFieldLength = 512;
|
||||
|
||||
// Pull the provider's structured payload out of the event XML into the
|
||||
// existing `fields` dictionary, which the server already stores as
|
||||
// payloadjson. Most providers use <EventData><Data Name="x">v</Data></>;
|
||||
// some use <UserData> with an arbitrary provider-defined subtree, so both
|
||||
// are walked. Unnamed <Data> elements keep their positional index as the
|
||||
// key, because for several catalog IDs position IS the meaning.
|
||||
//
|
||||
// Never throws: a provider with malformed or unreadable XML degrades to no
|
||||
// fields rather than losing the whole record.
|
||||
private Dictionary<string, object>? GetEventFields(EventRecord entry, string channel)
|
||||
{
|
||||
try
|
||||
{
|
||||
var xml = entry.ToXml();
|
||||
if (string.IsNullOrWhiteSpace(xml))
|
||||
return null;
|
||||
|
||||
var document = System.Xml.Linq.XDocument.Parse(xml);
|
||||
var root = document.Root;
|
||||
if (root is null)
|
||||
return null;
|
||||
|
||||
var container =
|
||||
root.Elements().FirstOrDefault(e => e.Name.LocalName == "EventData")
|
||||
?? root.Elements().FirstOrDefault(e => e.Name.LocalName == "UserData");
|
||||
if (container is null)
|
||||
return null;
|
||||
|
||||
var fields = new Dictionary<string, object>();
|
||||
int unnamed = 0;
|
||||
foreach (var element in container.Descendants())
|
||||
{
|
||||
if (fields.Count >= MaxEventFields)
|
||||
break;
|
||||
if (element.HasElements)
|
||||
continue;
|
||||
|
||||
var value = element.Value;
|
||||
if (string.IsNullOrWhiteSpace(value))
|
||||
continue;
|
||||
|
||||
var name = element.Attribute("Name")?.Value;
|
||||
if (string.IsNullOrWhiteSpace(name))
|
||||
{
|
||||
name = element.Name.LocalName == "Data"
|
||||
? "data" + unnamed++
|
||||
: element.Name.LocalName;
|
||||
}
|
||||
|
||||
fields[name!] = Truncate(value, MaxEventFieldLength) ?? string.Empty;
|
||||
}
|
||||
|
||||
return fields.Count > 0 ? fields : null;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Event XML not parseable on {Channel}", channel);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// An XPath the event log service evaluates itself, e.g.
|
||||
// *[System[(EventID=4624 or EventID=4625)]]
|
||||
private static string BuildIdFilter(IReadOnlySet<int> wanted)
|
||||
{
|
||||
var clauses = string.Join(" or ", wanted.Select(id => "EventID=" + id));
|
||||
return "*[System[(" + clauses + ")]]";
|
||||
}
|
||||
|
||||
// Installed-software SBOM. Mirrors the Assess-Orchestrator applications
|
||||
// collection exactly: enumerate both Uninstall registry hives (64-bit and
|
||||
// WOW6432Node), keep only entries with a DisplayName, and emit
|
||||
@@ -518,11 +936,15 @@ public sealed class EndpointCollector : ICollector
|
||||
if (!seen.Add(displayName + "\u0000" + displayVersion))
|
||||
continue;
|
||||
|
||||
var fingerprint = ComputeApplicationFingerprint(
|
||||
displayName, displayVersion, publisher);
|
||||
|
||||
result.Add(new
|
||||
{
|
||||
displayname = displayName,
|
||||
displayversion = displayVersion,
|
||||
publisher,
|
||||
fingerprint,
|
||||
});
|
||||
}
|
||||
catch (Exception error)
|
||||
@@ -537,6 +959,256 @@ public sealed class EndpointCollector : ICollector
|
||||
}
|
||||
}
|
||||
|
||||
// Compute SHA256 fingerprint for application inventory delta optimization.
|
||||
// Format: SHA256(displayname|displayversion|publisher) as lowercase hex string.
|
||||
// Matches backend algorithm in plugins/endpoints/plugin.py and backfill script.
|
||||
// Empty strings are preserved (not converted to null) to match backend behavior.
|
||||
private static string ComputeApplicationFingerprint(
|
||||
string displayName, string displayVersion, string publisher)
|
||||
{
|
||||
var normalized = $"{displayName}|{displayVersion}|{publisher}";
|
||||
var bytes = Encoding.UTF8.GetBytes(normalized);
|
||||
var hash = SHA256.HashData(bytes);
|
||||
return Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
|
||||
// Whether Windows is waiting on a reboot.
|
||||
//
|
||||
// There is no single API for this, so we check the three signals Windows
|
||||
// itself uses: the CBS servicing flag, the Windows Update auto-update flag,
|
||||
// and a queued file-rename operation. Any one of them is sufficient. Checked
|
||||
// by key EXISTENCE rather than value, which is how these flags work - the
|
||||
// key is created when a reboot is needed and deleted once it happens.
|
||||
// Link speed and duplex, keyed by MAC so the caller can pair them onto the
|
||||
// matching GetInterfaces() entry. Deliberately a separate query: the
|
||||
// IPEnabled config class above does not carry link state, and a NIC that
|
||||
// negotiated 100Mb half-duplex on a gigabit switch port is a cable/port
|
||||
// fault that otherwise presents as "the machine is slow sometimes".
|
||||
private Dictionary<string, (long? Speed, bool? FullDuplex)> GetLinkState()
|
||||
{
|
||||
var byMac = new Dictionary<string, (long?, bool?)>(
|
||||
StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
// MSFT_NetAdapter carries duplex; Win32_NetworkAdapter does not. It
|
||||
// lives in the same Storage-style optional namespace family, so a
|
||||
// failure here must not take the interface list with it.
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
@"root\StandardCimv2",
|
||||
"SELECT PermanentAddress, Speed, FullDuplex, MediaConnectState " +
|
||||
"FROM MSFT_NetAdapter");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
var mac = NormalizeMac(mo["PermanentAddress"]?.ToString());
|
||||
if (mac is null)
|
||||
continue;
|
||||
byMac[mac] = (ToLong(mo["Speed"]), mo["FullDuplex"] as bool?);
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Net adapter link state query failed");
|
||||
}
|
||||
|
||||
return byMac;
|
||||
}
|
||||
|
||||
// WMI reports MACs in several shapes (00:11:22, 001122AABBCC). Strip to hex
|
||||
// so the two sources can be matched at all.
|
||||
private static string? NormalizeMac(string? raw)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(raw))
|
||||
return null;
|
||||
var hex = new string(Array.FindAll(raw.ToCharArray(), Uri.IsHexDigit));
|
||||
return hex.Length == 12 ? hex : null;
|
||||
}
|
||||
|
||||
// Whether the last start was a true cold boot or a fast-startup resume.
|
||||
//
|
||||
// This matters because uptime LIES under fast startup: the box reports days
|
||||
// of uptime after what the operator experienced as a shutdown, and a
|
||||
// "reboot" that was really a hybrid resume does not clear the driver or
|
||||
// kernel state that a support call is usually trying to clear.
|
||||
//
|
||||
// Two distinct facts, deliberately not collapsed:
|
||||
// hiberbootenabled - fast startup is CONFIGURED (registry, always readable)
|
||||
// lastboottype - what the last boot ACTUALLY did (Kernel-Boot event 27)
|
||||
// The registry alone cannot tell you what the last boot did, and the event
|
||||
// may have aged out of the log, so neither substitutes for the other.
|
||||
private object? GetBootInfo()
|
||||
{
|
||||
bool? hiberboot = null;
|
||||
try
|
||||
{
|
||||
using var power = Registry.LocalMachine.OpenSubKey(
|
||||
@"SYSTEM\CurrentControlSet\Control\Session Manager\Power");
|
||||
if (power?.GetValue("HiberbootEnabled") is int raw)
|
||||
hiberboot = raw != 0;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Hiberboot registry read failed");
|
||||
}
|
||||
|
||||
string? lastboottype = null;
|
||||
try
|
||||
{
|
||||
// Kernel-Boot 27 BootType: 0 cold, 1 hybrid (fast startup),
|
||||
// 2 resume from hibernate.
|
||||
var query = new EventLogQuery(
|
||||
"System", PathType.LogName,
|
||||
"*[System[Provider[@Name='Microsoft-Windows-Kernel-Boot'] " +
|
||||
"and (EventID=27)]]")
|
||||
{ ReverseDirection = true };
|
||||
using var reader = new EventLogReader(query);
|
||||
using var record = reader.ReadEvent();
|
||||
if (record?.Properties.Count > 0)
|
||||
{
|
||||
lastboottype = ToInt(record.Properties[0].Value) switch
|
||||
{
|
||||
0 => "Cold",
|
||||
1 => "Fast startup",
|
||||
2 => "Resume",
|
||||
_ => null,
|
||||
};
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Boot type event read failed");
|
||||
}
|
||||
|
||||
if (hiberboot is null && lastboottype is null)
|
||||
return null;
|
||||
|
||||
return new
|
||||
{
|
||||
hiberbootenabled = hiberboot,
|
||||
lastboottype = lastboottype ?? "Unknown",
|
||||
};
|
||||
}
|
||||
|
||||
// Defender status. Collected and persisted but NOT surfaced: this
|
||||
// environment is known to produce false positives, so wiring it to a card
|
||||
// or an alert now would train operators to ignore the card. Stored so the
|
||||
// signal is already there when someone is ready to interpret it.
|
||||
private object? GetDefender()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
@"root\Microsoft\Windows\Defender",
|
||||
"SELECT AMServiceEnabled, RealTimeProtectionEnabled, " +
|
||||
"AntivirusSignatureAge, AntispywareSignatureAge, " +
|
||||
"QuickScanAge FROM MSFT_MpComputerStatus");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
return new
|
||||
{
|
||||
serviceenabled = mo["AMServiceEnabled"] as bool?,
|
||||
realtimeenabled = mo["RealTimeProtectionEnabled"] as bool?,
|
||||
// Ages in days. Null when Defender did not report rather
|
||||
// than 0, which would read as "updated today".
|
||||
signatureagedays = ToLong(mo["AntivirusSignatureAge"]),
|
||||
antispywareagedays = ToLong(mo["AntispywareSignatureAge"]),
|
||||
quickscanagedays = ToLong(mo["QuickScanAge"]),
|
||||
};
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
// A third-party AV box or a stripped image has no Defender WMI
|
||||
// provider at all. That is normal here, not a fault.
|
||||
logger.LogDebug(error, "Defender status query failed");
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Which signal(s) drove pendingreboot=true, in report order.
|
||||
//
|
||||
// pendingreboot is a single bool OR-ed from three sources, so it cannot say
|
||||
// WHY. That distinction is the point: PendingFileRenameOperations is the
|
||||
// stronger signal - files are staged to be replaced on next boot, so the
|
||||
// box is mid-servicing - whereas a lone CBS or Windows Update key can sit
|
||||
// set for weeks on a machine nobody has rebooted. Same boolean, very
|
||||
// different urgency.
|
||||
//
|
||||
// Empty array = checked, nothing pending. Null = the check failed, matching
|
||||
// pendingreboot's own null-is-unknown rule.
|
||||
private string[]? GetRebootReasons()
|
||||
{
|
||||
try
|
||||
{
|
||||
var reasons = new List<string>();
|
||||
|
||||
if (LocalMachineKeyExists(
|
||||
@"SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending"))
|
||||
{
|
||||
reasons.Add("componentbasedservicing");
|
||||
}
|
||||
|
||||
if (LocalMachineKeyExists(
|
||||
@"SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired"))
|
||||
{
|
||||
reasons.Add("windowsupdate");
|
||||
}
|
||||
|
||||
using var sessionManager = Registry.LocalMachine.OpenSubKey(
|
||||
@"SYSTEM\CurrentControlSet\Control\Session Manager");
|
||||
if (sessionManager?.GetValue("PendingFileRenameOperations") is string[] pending
|
||||
&& pending.Length > 0)
|
||||
{
|
||||
reasons.Add("pendingfilerename");
|
||||
}
|
||||
|
||||
return reasons.ToArray();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Reboot reason check failed");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private bool? GetPendingReboot()
|
||||
{
|
||||
try
|
||||
{
|
||||
const string cbs =
|
||||
@"SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending";
|
||||
const string windowsUpdate =
|
||||
@"SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired";
|
||||
|
||||
if (LocalMachineKeyExists(cbs) || LocalMachineKeyExists(windowsUpdate))
|
||||
return true;
|
||||
|
||||
using var sessionManager = Registry.LocalMachine.OpenSubKey(
|
||||
@"SYSTEM\CurrentControlSet\Control\Session Manager");
|
||||
if (sessionManager?.GetValue("PendingFileRenameOperations") is string[] pending
|
||||
&& pending.Length > 0)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
// Null, not false: an unreadable registry means we do not know, and
|
||||
// reporting a confident "no reboot pending" would be worse than
|
||||
// reporting nothing.
|
||||
logger.LogDebug(error, "Pending reboot check failed");
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool LocalMachineKeyExists(string path)
|
||||
{
|
||||
using var key = Registry.LocalMachine.OpenSubKey(path);
|
||||
return key is not null;
|
||||
}
|
||||
|
||||
// Fixed local disks with capacity/free so the server can flag a PC that is
|
||||
// running out of space. Win32_LogicalDisk DriveType 3 = local fixed disk;
|
||||
// network/removable drives are skipped. Bytes stay as longs; the server can
|
||||
@@ -562,6 +1234,14 @@ public sealed class EndpointCollector : ICollector
|
||||
freebytes = free,
|
||||
usedbytes = size is long total && free is long available
|
||||
? total - available : (long?)null,
|
||||
// Precomputed so an alert threshold is a direct comparison.
|
||||
// A 40GB free reading means nothing without the capacity it
|
||||
// came from, and the server would have to guard against a
|
||||
// null or zero size on every evaluation to derive it.
|
||||
freepercent = size is long capacity && capacity > 0
|
||||
&& free is long remaining
|
||||
? Math.Round((double)remaining / capacity * 100.0, 1)
|
||||
: (double?)null,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -572,6 +1252,129 @@ public sealed class EndpointCollector : ICollector
|
||||
return result.ToArray();
|
||||
}
|
||||
|
||||
// Physical drive health, so a failing disk is caught before it takes the
|
||||
// machine and whatever was local on it.
|
||||
//
|
||||
// Two sources, because neither covers the fleet alone:
|
||||
//
|
||||
// MSFT_PhysicalDisk (Storage namespace) is the only one that reports NVMe.
|
||||
// The classic SMART class, MSStorageDriver_FailurePredictStatus, is ATA/SATA
|
||||
// only and returns NOTHING on an NVMe box - verified on a two-NVMe machine
|
||||
// here, where it yielded zero instances while the drives were present and
|
||||
// healthy. Collecting only that class would have reported an empty disk
|
||||
// health list on every modern endpoint and looked like "no problems".
|
||||
//
|
||||
// Win32_DiskDrive.Status is the old reliable string ("OK"/"Pred Fail") and
|
||||
// covers anything the Storage namespace does not, including machines where
|
||||
// that namespace is missing or blocked.
|
||||
//
|
||||
// Keyed by DeviceId so the server can pair the two views per drive the same
|
||||
// way mapped drives are paired.
|
||||
private object[] GetDiskHealth()
|
||||
{
|
||||
var result = new List<object>();
|
||||
|
||||
// MSFT_PhysicalDisk returns raw enum INTEGERS through
|
||||
// ManagementObjectSearcher, not the friendly strings Get-PhysicalDisk
|
||||
// prints. Verified on a live box: Health=0 Op=2 for a healthy NVMe.
|
||||
// Mapping them here keeps the wire self-describing; shipping bare
|
||||
// integers would make the server guess at an undocumented enum.
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
@"root\Microsoft\Windows\Storage",
|
||||
"SELECT DeviceId, FriendlyName, MediaType, BusType, " +
|
||||
"HealthStatus, OperationalStatus FROM MSFT_PhysicalDisk");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
var health = ToInt(mo["HealthStatus"]);
|
||||
result.Add(new
|
||||
{
|
||||
deviceid = mo["DeviceId"]?.ToString()?.Trim() ?? "",
|
||||
model = mo["FriendlyName"]?.ToString()?.Trim() ?? "",
|
||||
mediatype = PhysicalDiskMediaType(ToInt(mo["MediaType"])),
|
||||
bustype = PhysicalDiskBusType(ToInt(mo["BusType"])),
|
||||
healthstatus = PhysicalDiskHealth(health),
|
||||
// The actionable bit, precomputed so an alert is a direct
|
||||
// boolean test rather than a string comparison against an
|
||||
// enum the server would have to keep in sync with this file.
|
||||
// Null when health is unreadable: unknown is not healthy,
|
||||
// but it is not a confident failure either.
|
||||
predictfailure = health is null ? (bool?)null : health != 0,
|
||||
source = "physicaldisk",
|
||||
});
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Physical disk health query failed");
|
||||
}
|
||||
|
||||
// Win32_DiskDrive as the ATA/SATA and fallback view. Status is a plain
|
||||
// string here, no enum mapping needed.
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher(
|
||||
"SELECT DeviceID, Model, Status FROM Win32_DiskDrive");
|
||||
foreach (var mo in searcher.Get())
|
||||
{
|
||||
var status = mo["Status"]?.ToString()?.Trim() ?? "";
|
||||
result.Add(new
|
||||
{
|
||||
deviceid = mo["DeviceID"]?.ToString()?.Trim() ?? "",
|
||||
model = mo["Model"]?.ToString()?.Trim() ?? "",
|
||||
mediatype = "",
|
||||
bustype = "",
|
||||
healthstatus = status,
|
||||
// "OK" is the only healthy value this class reports;
|
||||
// "Pred Fail" is the SMART trip. An empty status means the
|
||||
// drive did not answer, which is unknown rather than failed.
|
||||
predictfailure = status.Length == 0
|
||||
? (bool?)null
|
||||
: !status.Equals("OK", StringComparison.OrdinalIgnoreCase),
|
||||
source = "diskdrive",
|
||||
});
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Disk drive status query failed");
|
||||
}
|
||||
|
||||
return result.ToArray();
|
||||
}
|
||||
|
||||
// MSFT_PhysicalDisk HealthStatus: 0 Healthy, 1 Warning, 2 Unhealthy.
|
||||
private static string PhysicalDiskHealth(int? value) => value switch
|
||||
{
|
||||
0 => "Healthy",
|
||||
1 => "Warning",
|
||||
2 => "Unhealthy",
|
||||
_ => "Unknown",
|
||||
};
|
||||
|
||||
// MSFT_PhysicalDisk MediaType: 3 HDD, 4 SSD, 5 SCM.
|
||||
private static string PhysicalDiskMediaType(int? value) => value switch
|
||||
{
|
||||
3 => "HDD",
|
||||
4 => "SSD",
|
||||
5 => "SCM",
|
||||
_ => "Unknown",
|
||||
};
|
||||
|
||||
// MSFT_PhysicalDisk BusType. Only the values that actually appear on an
|
||||
// endpoint are named; anything else stays numeric rather than guessed at.
|
||||
private static string PhysicalDiskBusType(int? value) => value switch
|
||||
{
|
||||
3 => "ATA",
|
||||
7 => "USB",
|
||||
8 => "RAID",
|
||||
9 => "iSCSI",
|
||||
11 => "SATA",
|
||||
17 => "NVMe",
|
||||
_ => value is null ? "Unknown" : value.Value.ToString(),
|
||||
};
|
||||
|
||||
// Firmware identity from Win32_BIOS: vendor, version string, and release
|
||||
// date. Lets the server track BIOS/UEFI levels for a fleet update campaign.
|
||||
private object? GetBios()
|
||||
@@ -708,6 +1511,14 @@ public sealed class EndpointCollector : ICollector
|
||||
catch { return null; }
|
||||
}
|
||||
|
||||
private static int? ToInt(object? value)
|
||||
{
|
||||
if (value is null)
|
||||
return null;
|
||||
try { return Convert.ToInt32(value); }
|
||||
catch { return null; }
|
||||
}
|
||||
|
||||
// Win32 CIM datetimes (yyyymmddHHMMSS.ffffff+zzz) render as an ISO-8601 UTC
|
||||
// string; anything unparseable passes through untouched so nothing is lost.
|
||||
private string? ToBiosDate(string? raw)
|
||||
|
||||
357
agent/ShopDbEndpointAgent.Core/EnrollmentService.cs
Normal file
@@ -0,0 +1,357 @@
|
||||
using System.Management;
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Win32;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Gets this PC its own endpoint credential (POST /api/endpoints/enroll) and
|
||||
// keeps trying until it has one. Workers run on the shared key meanwhile, so an
|
||||
// unenrolled or pending PC keeps reporting.
|
||||
//
|
||||
// 201 enrolled store credential, swap provider, idle until woken
|
||||
// 202 pending PendingSchedule: 1 min, then 5 min, then 15 min
|
||||
// 404 server predates enrollment: idle, retry every 6 h
|
||||
// other back off 1, 5, 15 min (cap); a 429 or 5xx lands here, so a
|
||||
// rate limit drops out of the fast pending phase for that try
|
||||
//
|
||||
// Woken early by the provider on CREDENTIAL_INVALID (credential deleted) and on
|
||||
// SHARED_KEY_REFUSED while unenrolled, floored at one attempt a minute.
|
||||
public sealed class EnrollmentService : BackgroundService
|
||||
{
|
||||
public const string DeviceIdValue = "DeviceId";
|
||||
|
||||
// Retry while pending, by how long this request has been pending. An admin
|
||||
// usually approves within a couple of minutes and a tech is standing there,
|
||||
// so ask often at first. Jitter is added, never subtracted, so MinWakeInterval
|
||||
// cannot clip it back into lockstep across a room of PCs.
|
||||
private static readonly (TimeSpan Waited, TimeSpan Retry, TimeSpan Jitter)[] PendingSchedule =
|
||||
{
|
||||
(TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(1), TimeSpan.FromSeconds(8)),
|
||||
(TimeSpan.FromMinutes(30), TimeSpan.FromMinutes(5), TimeSpan.FromSeconds(20)),
|
||||
(TimeSpan.MaxValue, TimeSpan.FromMinutes(15), TimeSpan.FromMinutes(2)),
|
||||
};
|
||||
private static readonly TimeSpan UnsupportedRetry = TimeSpan.FromHours(6);
|
||||
private static readonly TimeSpan[] ErrorBackoff =
|
||||
{
|
||||
TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(5), TimeSpan.FromMinutes(15),
|
||||
};
|
||||
private static readonly TimeSpan MinWakeInterval = TimeSpan.FromMinutes(1);
|
||||
|
||||
private readonly AgentConfig config;
|
||||
private readonly AgentHttpFactory httpFactory;
|
||||
private readonly AgentCredentialProvider credentials;
|
||||
private readonly CredentialStore store;
|
||||
private readonly string agentVersion;
|
||||
private readonly ILogger<EnrollmentService> logger;
|
||||
private readonly SemaphoreSlim wake = new(0, 1);
|
||||
private readonly TaskCompletionSource firstAttempt =
|
||||
new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
private readonly Random jitter = new();
|
||||
|
||||
public EnrollmentService(AgentConfig config, AgentHttpFactory httpFactory,
|
||||
AgentCredentialProvider credentials, CredentialStore store, string agentVersion,
|
||||
ILogger<EnrollmentService> logger)
|
||||
{
|
||||
this.config = config;
|
||||
this.httpFactory = httpFactory;
|
||||
this.credentials = credentials;
|
||||
this.store = store;
|
||||
this.agentVersion = agentVersion;
|
||||
this.logger = logger;
|
||||
credentials.EnrollmentNeeded += (_, _) => Wake();
|
||||
}
|
||||
|
||||
// Workers await this (bounded) before first contact so a PC that can enroll
|
||||
// right away does its first report on its own credential.
|
||||
public async Task WaitForFirstAttemptAsync(TimeSpan limit, CancellationToken token)
|
||||
{
|
||||
try
|
||||
{
|
||||
await firstAttempt.Task.WaitAsync(limit, token);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
public void Wake()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (wake.CurrentCount == 0)
|
||||
wake.Release();
|
||||
}
|
||||
catch (SemaphoreFullException)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
int failures = 0;
|
||||
var lastAttempt = DateTime.MinValue;
|
||||
// Start of the current pending run; reset on a new request id so a
|
||||
// reject-and-ask-again (or a revoke) gets the fast phase over again.
|
||||
DateTime? pendingSince = null;
|
||||
int? pendingRequest = null;
|
||||
using var client = httpFactory.CreateClient(AgentHttpChannel.Enroll);
|
||||
|
||||
try
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
if (credentials.Current is not null)
|
||||
{
|
||||
pendingSince = null;
|
||||
pendingRequest = null;
|
||||
firstAttempt.TrySetResult();
|
||||
await wake.WaitAsync(stoppingToken);
|
||||
continue;
|
||||
}
|
||||
|
||||
var sinceLast = DateTime.UtcNow - lastAttempt;
|
||||
if (sinceLast < MinWakeInterval)
|
||||
await Task.Delay(MinWakeInterval - sinceLast, stoppingToken);
|
||||
|
||||
lastAttempt = DateTime.UtcNow;
|
||||
var (outcome, requestId) = await TryEnrollAsync(client, stoppingToken);
|
||||
firstAttempt.TrySetResult();
|
||||
|
||||
TimeSpan delay;
|
||||
switch (outcome)
|
||||
{
|
||||
case EnrollOutcome.Enrolled:
|
||||
failures = 0;
|
||||
pendingSince = null;
|
||||
pendingRequest = null;
|
||||
continue;
|
||||
case EnrollOutcome.Pending:
|
||||
failures = 0;
|
||||
if (pendingSince is null
|
||||
|| (requestId is not null && requestId != pendingRequest))
|
||||
{
|
||||
pendingSince = DateTime.UtcNow;
|
||||
pendingRequest = requestId;
|
||||
}
|
||||
delay = PendingDelay(DateTime.UtcNow - pendingSince.Value);
|
||||
logger.LogDebug("Next enrollment attempt in {Delay}.", delay);
|
||||
break;
|
||||
case EnrollOutcome.Unsupported:
|
||||
failures = 0;
|
||||
pendingSince = null;
|
||||
pendingRequest = null;
|
||||
delay = UnsupportedRetry;
|
||||
break;
|
||||
default:
|
||||
delay = ErrorBackoff[Math.Min(failures, ErrorBackoff.Length - 1)];
|
||||
failures++;
|
||||
break;
|
||||
}
|
||||
|
||||
// A wake cuts the wait short (credential refused, shared key refused).
|
||||
await wake.WaitAsync(delay, stoppingToken);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
finally
|
||||
{
|
||||
firstAttempt.TrySetResult();
|
||||
}
|
||||
}
|
||||
|
||||
private enum EnrollOutcome { Enrolled, Pending, Unsupported, Failed }
|
||||
|
||||
private TimeSpan PendingDelay(TimeSpan waited)
|
||||
{
|
||||
foreach (var step in PendingSchedule)
|
||||
{
|
||||
if (waited < step.Waited)
|
||||
return step.Retry
|
||||
+ TimeSpan.FromSeconds(jitter.NextDouble() * step.Jitter.TotalSeconds);
|
||||
}
|
||||
return PendingSchedule[^1].Retry;
|
||||
}
|
||||
|
||||
private async Task<(EnrollOutcome Outcome, int? RequestId)> TryEnrollAsync(
|
||||
HttpClient client, CancellationToken token)
|
||||
{
|
||||
var hostname = Environment.MachineName;
|
||||
var serial = ReadBiosSerial();
|
||||
if (string.IsNullOrWhiteSpace(serial))
|
||||
{
|
||||
logger.LogError("Enrollment skipped: no BIOS serial number from WMI Win32_BIOS.");
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
|
||||
string nonce;
|
||||
try
|
||||
{
|
||||
nonce = store.GetOrCreateEnrollNonce();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogError(error, "Enrollment skipped: could not read or create enroll.json.");
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
|
||||
var request = new
|
||||
{
|
||||
hostname,
|
||||
serialnumber = serial,
|
||||
deviceid = ReadDeviceId(),
|
||||
enrollnonce = nonce,
|
||||
agentversion = agentVersion,
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
using var response = await client.PostAsJsonAsync("api/endpoints/enroll", request, token);
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NotFound
|
||||
&& AgentServerError.TryParse(body)?.Code is null or "" or "NOT_FOUND")
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Server has no enrollment endpoint (HTTP 404); staying on the shared key, retry in {Delay}.",
|
||||
UnsupportedRetry);
|
||||
return (EnrollOutcome.Unsupported, null);
|
||||
}
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.Created)
|
||||
{
|
||||
var result = ParseResult(body);
|
||||
var secret = result.GetString("credential");
|
||||
if (secret is null || !secret.StartsWith("sdbe_", StringComparison.Ordinal))
|
||||
{
|
||||
logger.LogError("Enrollment returned 201 without a usable credential.");
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
|
||||
var credential = new AgentCredential
|
||||
{
|
||||
Credential = secret,
|
||||
AssetId = result.GetInt("assetid") ?? 0,
|
||||
IssuedAt = DateTime.UtcNow.ToString("o"),
|
||||
BaseUrl = config.BaseUrl.Trim().TrimEnd('/'),
|
||||
};
|
||||
try
|
||||
{
|
||||
credentials.Set(credential);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
// Shown once by the server; an admin must revoke and let it re-enroll.
|
||||
logger.LogError(error,
|
||||
"Enrolled as asset {AssetId} but could not store credential {Prefix}. " +
|
||||
"Revoke it in ShopDB so this PC can enroll again.",
|
||||
credential.AssetId, credential.Prefix);
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
return (EnrollOutcome.Enrolled, null);
|
||||
}
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.Accepted)
|
||||
{
|
||||
var result = ParseResult(body);
|
||||
var requestId = result.GetInt("requestid");
|
||||
logger.LogInformation(
|
||||
"Enrollment pending (request {RequestId}): {Reason}.",
|
||||
requestId, result.GetString("reason") ?? "no reason given");
|
||||
return (EnrollOutcome.Pending, requestId);
|
||||
}
|
||||
|
||||
var refusal = AgentServerError.TryParse(body);
|
||||
if (refusal?.Code == "ENROLL_NONCE_MISMATCH")
|
||||
{
|
||||
logger.LogError(
|
||||
"Enrollment refused: the approved request belongs to a different enroll nonce. " +
|
||||
"An admin must reject it so this PC can request again.");
|
||||
// No request id: keeps the current pending run and its schedule.
|
||||
return (EnrollOutcome.Pending, null);
|
||||
}
|
||||
|
||||
logger.LogWarning("Enrollment failed: HTTP {Status} {Code} {Message}",
|
||||
(int)response.StatusCode, refusal?.Code, refusal?.Message);
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
catch (Exception error) when (
|
||||
error is HttpRequestException or TaskCanceledException or JsonException
|
||||
&& !token.IsCancellationRequested)
|
||||
{
|
||||
logger.LogWarning("Enrollment request failed: {Message}", error.Message);
|
||||
return (EnrollOutcome.Failed, null);
|
||||
}
|
||||
}
|
||||
|
||||
// Accepts the bare body or the success envelope's data object.
|
||||
private static ResultReader ParseResult(string body)
|
||||
{
|
||||
using var doc = JsonDocument.Parse(body);
|
||||
var root = doc.RootElement;
|
||||
if (root.ValueKind == JsonValueKind.Object
|
||||
&& root.TryGetProperty("data", out var data) && data.ValueKind == JsonValueKind.Object)
|
||||
root = data;
|
||||
return new ResultReader(root.Clone());
|
||||
}
|
||||
|
||||
private readonly struct ResultReader
|
||||
{
|
||||
private readonly JsonElement root;
|
||||
public ResultReader(JsonElement root) => this.root = root;
|
||||
|
||||
public string? GetString(string name)
|
||||
=> root.ValueKind == JsonValueKind.Object
|
||||
&& root.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String
|
||||
? value.GetString() : null;
|
||||
|
||||
public int? GetInt(string name)
|
||||
=> root.ValueKind == JsonValueKind.Object
|
||||
&& root.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.Number
|
||||
&& value.TryGetInt32(out var number)
|
||||
? number : null;
|
||||
}
|
||||
|
||||
private string? ReadBiosSerial()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var searcher = new ManagementObjectSearcher("SELECT SerialNumber FROM Win32_BIOS");
|
||||
foreach (var item in searcher.Get())
|
||||
{
|
||||
var value = item["SerialNumber"]?.ToString()?.Trim();
|
||||
if (!string.IsNullOrWhiteSpace(value))
|
||||
return value.Length > 128 ? value[..128] : value;
|
||||
}
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogWarning(error, "WMI Win32_BIOS query failed");
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private string? ReadDeviceId()
|
||||
{
|
||||
foreach (var view in new[] { RegistryView.Registry64, RegistryView.Registry32 })
|
||||
{
|
||||
try
|
||||
{
|
||||
using var machine = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, view);
|
||||
using var key = machine.OpenSubKey(AgentConfigProvider.RegistryPath);
|
||||
if (key?.GetValue(DeviceIdValue) is string value && !string.IsNullOrWhiteSpace(value))
|
||||
return value.Trim();
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Registry read of {Value} in {View} view failed", DeviceIdValue, view);
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
132
agent/ShopDbEndpointAgent.Core/EventWatermarkStore.cs
Normal file
@@ -0,0 +1,132 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Remembers the newest Windows event record id already reported for each
|
||||
// curated channel, so a collect cycle harvests ONLY records the server has not
|
||||
// seen.
|
||||
//
|
||||
// Why this exists: the collector used to re-read the newest N records of every
|
||||
// channel on every cycle, and the server ingest appends event rows without a
|
||||
// dedupe check. A single crash therefore landed in the database once per
|
||||
// cadence tick for as long as it stayed inside that window - up to 96 identical
|
||||
// rows a day per event. Carrying a watermark makes the steady state "only what
|
||||
// is new", which is also what makes the full event XML affordable to send.
|
||||
//
|
||||
// The watermark is persisted under %ProgramData% (not in memory) because a
|
||||
// service restart would otherwise re-report the whole window again, which is
|
||||
// exactly the crash-loop case where duplicate events are most likely.
|
||||
//
|
||||
// Record ids are per channel and monotonic on a given machine. A cleared or
|
||||
// rebuilt event log restarts them at 1; that is handled by treating a stored
|
||||
// watermark HIGHER than the newest available record as stale and resetting it,
|
||||
// otherwise the channel would go permanently silent.
|
||||
public sealed class EventWatermarkStore
|
||||
{
|
||||
private const string StoreFileName = "eventwatermark.json";
|
||||
|
||||
// %ProgramData%\ShopDB\EndpointAgent - the same root the rolling agent log
|
||||
// lives under, so all agent-owned state sits in one place.
|
||||
public static string DefaultStateDirectory => Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"ShopDB", "EndpointAgent");
|
||||
|
||||
private readonly string storePath;
|
||||
private readonly ILogger<EventWatermarkStore> logger;
|
||||
private readonly Dictionary<string, long> watermarks;
|
||||
|
||||
public EventWatermarkStore(ILogger<EventWatermarkStore> logger,
|
||||
string? stateDirectory = null)
|
||||
{
|
||||
this.logger = logger;
|
||||
var directory = string.IsNullOrWhiteSpace(stateDirectory)
|
||||
? DefaultStateDirectory
|
||||
: stateDirectory;
|
||||
storePath = Path.Combine(directory, StoreFileName);
|
||||
watermarks = Load();
|
||||
}
|
||||
|
||||
// The last record id reported for this channel, or 0 when the channel has
|
||||
// never been harvested. 0 means "first contact": the caller sends a bounded
|
||||
// tail for initial context rather than the entire backlog.
|
||||
public long GetWatermark(string channel)
|
||||
=> watermarks.TryGetValue(channel, out var value) ? value : 0;
|
||||
|
||||
public bool HasWatermark(string channel)
|
||||
=> watermarks.ContainsKey(channel);
|
||||
|
||||
// Advance the channel forward only. Never moves backwards, so an
|
||||
// out-of-order read cannot cause the same record to be reported twice.
|
||||
public void Advance(string channel, long recordId)
|
||||
{
|
||||
if (recordId <= 0)
|
||||
return;
|
||||
if (watermarks.TryGetValue(channel, out var current) && current >= recordId)
|
||||
return;
|
||||
watermarks[channel] = recordId;
|
||||
}
|
||||
|
||||
// A cleared or rebuilt log restarts record ids near 1. When the newest
|
||||
// record available is below the stored watermark the store is stale, so
|
||||
// drop it and let the channel re-seed instead of going silent forever.
|
||||
public bool ResetIfStale(string channel, long newestAvailableRecordId)
|
||||
{
|
||||
if (newestAvailableRecordId <= 0)
|
||||
return false;
|
||||
if (!watermarks.TryGetValue(channel, out var current) || current <= newestAvailableRecordId)
|
||||
return false;
|
||||
|
||||
logger.LogInformation(
|
||||
"Event channel {Channel} record id went backwards ({Stored} -> {Newest}); log was cleared, reseeding",
|
||||
channel, current, newestAvailableRecordId);
|
||||
watermarks.Remove(channel);
|
||||
return true;
|
||||
}
|
||||
|
||||
// Best-effort persist. A failure here costs at most a repeated report of
|
||||
// records already sent, so it must never abort a collection.
|
||||
public void Save()
|
||||
{
|
||||
try
|
||||
{
|
||||
var directory = Path.GetDirectoryName(storePath);
|
||||
if (!string.IsNullOrWhiteSpace(directory))
|
||||
Directory.CreateDirectory(directory);
|
||||
|
||||
// Write-then-rename so a crash mid-write cannot leave a truncated
|
||||
// file that would reseed every channel on the next start.
|
||||
var temporaryPath = storePath + ".tmp";
|
||||
File.WriteAllText(temporaryPath,
|
||||
JsonSerializer.Serialize(watermarks));
|
||||
File.Move(temporaryPath, storePath, overwrite: true);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error, "Could not persist event watermarks to {Path}", storePath);
|
||||
}
|
||||
}
|
||||
|
||||
private Dictionary<string, long> Load()
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!File.Exists(storePath))
|
||||
return new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
var loaded = JsonSerializer.Deserialize<Dictionary<string, long>>(
|
||||
File.ReadAllText(storePath));
|
||||
return loaded is null
|
||||
? new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase)
|
||||
: new Dictionary<string, long>(loaded, StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
// A corrupt store reseeds every channel, which costs one bounded
|
||||
// tail per channel - acceptable versus failing the collection.
|
||||
logger.LogWarning(error,
|
||||
"Event watermark store unreadable at {Path}; reseeding channels", storePath);
|
||||
return new Dictionary<string, long>(StringComparer.OrdinalIgnoreCase);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -14,32 +14,74 @@ public sealed class FileLoggerProvider : ILoggerProvider
|
||||
private readonly string directory;
|
||||
private readonly LogLevel minLevel;
|
||||
private readonly int retentionDays;
|
||||
private readonly string filePrefix;
|
||||
private readonly object gate = new();
|
||||
private readonly ConcurrentDictionary<string, FileLogger> loggers = new();
|
||||
private string currentDateStamp = "";
|
||||
private StreamWriter? writer;
|
||||
|
||||
// Why the sink can be unusable, or null when it opened normally. The
|
||||
// constructor cannot throw (see below), but a caller that has a second,
|
||||
// known-writable place to complain to needs to be able to ask.
|
||||
public string? SinkProblem { get; private set; }
|
||||
|
||||
public FileLoggerProvider(string directory, LogLevel minLevel = LogLevel.Information,
|
||||
int retentionDays = 14)
|
||||
int retentionDays = 14, string filePrefix = "agent")
|
||||
{
|
||||
this.directory = directory;
|
||||
this.minLevel = minLevel;
|
||||
this.retentionDays = retentionDays;
|
||||
this.filePrefix = filePrefix;
|
||||
// Best-effort: creating the log directory can fail on a locked-down host
|
||||
// or when a non-privileged user targets a SYSTEM-owned path. Never let
|
||||
// that abort construction; Write() degrades to a no-op if the sink is
|
||||
// unusable.
|
||||
//
|
||||
// The reason is KEPT rather than discarded. A silently dead sink is what
|
||||
// made the user-context probe undiagnosable: running as the operator it
|
||||
// could not write into the SYSTEM-owned ProgramData log directory, so a
|
||||
// probe that failed and a probe that never ran produced identical
|
||||
// silence. Callers that have a writable fallback can now surface this.
|
||||
try { Directory.CreateDirectory(directory); }
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException) { }
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
SinkProblem = $"{directory}: {error.Message}";
|
||||
}
|
||||
}
|
||||
|
||||
// The default log directory when config leaves LogPath empty:
|
||||
// %ProgramData%\ShopDB\EndpointAgent\logs
|
||||
//
|
||||
// SERVICE-ONLY. This path is SYSTEM-owned and a signed-in operator cannot
|
||||
// write to it, so anything running in the user's session must use
|
||||
// UserProfileLogDirectory() instead.
|
||||
public static string DefaultDirectory()
|
||||
=> Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"ShopDB", "EndpointAgent", "logs");
|
||||
|
||||
// The shop-floor operator account. Every shop-floor endpoint uses this same
|
||||
// local account name, so the path is fixed rather than resolved through the
|
||||
// interactive token: the token lookup is the part that fails on a hardened
|
||||
// endpoint, and when it fails the writer and the reader silently disagree
|
||||
// about where the spool lives. If a site ever uses a different account only
|
||||
// user-context is lost - telemetry, commands and inventory are unaffected.
|
||||
public const string ShopFloorUserName = "ShopFloor";
|
||||
|
||||
public static string ShopFloorProfileDirectory()
|
||||
=> Path.Combine(DrivesRoot, "Users", ShopFloorUserName);
|
||||
|
||||
private static string DrivesRoot =>
|
||||
Path.GetPathRoot(Environment.GetFolderPath(Environment.SpecialFolder.Windows))
|
||||
?? @"C:\";
|
||||
|
||||
// Where the TRAY and the user-context probe log:
|
||||
// C:\Users\ShopFloor\ShopDB\Logs
|
||||
// The operator owns this directory, so it is writable from the user session
|
||||
// where ProgramData is not.
|
||||
public static string UserProfileLogDirectory()
|
||||
=> Path.Combine(ShopFloorProfileDirectory(), "ShopDB", "Logs");
|
||||
|
||||
public ILogger CreateLogger(string categoryName)
|
||||
=> loggers.GetOrAdd(categoryName, name => new FileLogger(this, name));
|
||||
|
||||
@@ -85,7 +127,14 @@ public sealed class FileLoggerProvider : ILoggerProvider
|
||||
writer?.Flush();
|
||||
writer?.Dispose();
|
||||
currentDateStamp = stamp;
|
||||
var path = Path.Combine(directory, $"agent-{stamp}.log");
|
||||
// One convention for every log this codebase writes:
|
||||
// "<yyyyMMdd>-<prefix>.log", so agent, tray and installer logs sort
|
||||
// chronologically when they share a directory and can be collected with
|
||||
// a single date-leading glob. The service previously used the opposite
|
||||
// order, which meant a support request for "the logs from the 5th" had to
|
||||
// know which component wrote which shape.
|
||||
var name = $"{stamp}-{filePrefix}.log";
|
||||
var path = Path.Combine(directory, name);
|
||||
writer = new StreamWriter(new FileStream(
|
||||
path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
|
||||
{
|
||||
@@ -99,7 +148,12 @@ public sealed class FileLoggerProvider : ILoggerProvider
|
||||
try
|
||||
{
|
||||
var cutoff = DateTime.Now.AddDays(-retentionDays);
|
||||
foreach (var file in Directory.EnumerateFiles(directory, "agent-*.log"))
|
||||
// Both shapes: retention has to keep covering the "<prefix>-<date>"
|
||||
// files written before the convention was unified, or an upgraded
|
||||
// endpoint would keep its pre-upgrade logs forever.
|
||||
string[] patterns = [$"*-{filePrefix}.log", $"{filePrefix}-*.log"];
|
||||
foreach (var pattern in patterns)
|
||||
foreach (var file in Directory.EnumerateFiles(directory, pattern))
|
||||
{
|
||||
if (File.GetLastWriteTime(file) < cutoff)
|
||||
File.Delete(file);
|
||||
|
||||
167
agent/ShopDbEndpointAgent.Core/FleetAlertClient.cs
Normal file
@@ -0,0 +1,167 @@
|
||||
using System.Net;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Polls the fleet alert channel to fetch active notifications for this endpoint.
|
||||
// The server may publish site-wide alerts (maintenance windows, policy changes)
|
||||
// or endpoint-specific warnings (outdated agent version, failed telemetry push).
|
||||
// The agent polls GET /api/endpoints/alerts?hostname=X every 60 seconds when idle
|
||||
// and deserializes the JSON response into FleetAlert DTOs. Same auth as
|
||||
// telemetry and commands (AgentHttpFactory). In 1.20 the service calls this and
|
||||
// serves the tray over the local pipe (AgentPipeServer).
|
||||
public sealed class FleetAlertClient : IDisposable
|
||||
{
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private readonly HttpClient http;
|
||||
private readonly ILogger logger;
|
||||
|
||||
public FleetAlertClient(AgentHttpFactory httpFactory, ILogger logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
http = httpFactory.CreateClient(AgentHttpChannel.FleetAlert);
|
||||
}
|
||||
|
||||
// Tray compatibility until tray 1.7.0 moves to AgentPipeClient: shared key
|
||||
// only, no credential. Remove with the last tray that calls it.
|
||||
public FleetAlertClient(AgentConfig config, ILogger logger)
|
||||
: this(new AgentHttpFactory(config, new SharedKeyOnlyProvider(config), logger), logger)
|
||||
{
|
||||
}
|
||||
|
||||
private sealed class SharedKeyOnlyProvider : ICredentialProvider
|
||||
{
|
||||
public SharedKeyOnlyProvider(AgentConfig config)
|
||||
=> SharedKey = config.HasSharedKey() ? config.ApiKey : null;
|
||||
public AgentCredential? Current => null;
|
||||
public string? SharedKey { get; }
|
||||
public event EventHandler? Changed { add { } remove { } }
|
||||
public event EventHandler? EnrollmentNeeded { add { } remove { } }
|
||||
public void ReportRejected(string code, string? sentCredential) { }
|
||||
}
|
||||
|
||||
public async Task<List<FleetAlert>> FetchAlertsAsync(
|
||||
string hostname, CancellationToken token)
|
||||
{
|
||||
var url = "api/endpoints/alerts?hostname="
|
||||
+ Uri.EscapeDataString(hostname);
|
||||
using var response = await http.GetAsync(url, token);
|
||||
|
||||
if (response.StatusCode == HttpStatusCode.NoContent)
|
||||
return new List<FleetAlert>();
|
||||
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
if (!response.IsSuccessStatusCode)
|
||||
throw new CollectorException((int)response.StatusCode, body);
|
||||
|
||||
var envelope = JsonSerializer.Deserialize<FleetAlertEnvelope>(
|
||||
body, JsonOptions);
|
||||
if (envelope?.Data is FleetAlertList parsed)
|
||||
foreach (var skipped in parsed.Skipped)
|
||||
logger.LogWarning("Skipped malformed fleet alert: {Reason}", skipped);
|
||||
return envelope?.Data ?? new List<FleetAlert>();
|
||||
}
|
||||
|
||||
public async Task AcknowledgeAlertAsync(
|
||||
int notificationId, int assetId, string hostname, CancellationToken token)
|
||||
{
|
||||
var url = $"api/endpoints/alerts/{notificationId}/acknowledge";
|
||||
var payload = new
|
||||
{
|
||||
assetid = assetId,
|
||||
hostname
|
||||
};
|
||||
var json = JsonSerializer.Serialize(payload, JsonOptions);
|
||||
var content = new StringContent(json, System.Text.Encoding.UTF8, "application/json");
|
||||
|
||||
using var response = await http.PostAsync(url, content, token);
|
||||
var body = await response.Content.ReadAsStringAsync(token);
|
||||
|
||||
if (!response.IsSuccessStatusCode)
|
||||
throw new CollectorException((int)response.StatusCode, body);
|
||||
}
|
||||
|
||||
public void Dispose() => http.Dispose();
|
||||
}
|
||||
|
||||
// Server-sent strings may be JSON null; setters coerce to "" so consumers
|
||||
// never see null. Timeout null or <= 0 means no timeout.
|
||||
public sealed class FleetAlert
|
||||
{
|
||||
private string typeName = "";
|
||||
private string typeColor = "";
|
||||
private string title = "";
|
||||
private string message = "";
|
||||
|
||||
[JsonPropertyName("notificationid")] public int NotificationId { get; set; }
|
||||
[JsonPropertyName("type")] public string TypeName { get => typeName; set => typeName = value ?? ""; }
|
||||
[JsonPropertyName("typecolor")] public string TypeColor { get => typeColor; set => typeColor = value ?? ""; }
|
||||
[JsonPropertyName("title")] public string Title { get => title; set => title = value ?? ""; }
|
||||
[JsonPropertyName("message")] public string Message { get => message; set => message = value ?? ""; }
|
||||
[JsonPropertyName("timeout")] public int? Timeout { get; set; }
|
||||
|
||||
[JsonIgnore] public int TimeoutMinutes => Timeout is > 0 ? Timeout.Value : 0;
|
||||
}
|
||||
|
||||
public sealed class FleetAlertEnvelope
|
||||
{
|
||||
[JsonPropertyName("status")] public string Status { get; set; } = "";
|
||||
|
||||
[JsonPropertyName("data")]
|
||||
[JsonConverter(typeof(TolerantFleetAlertListConverter))]
|
||||
public List<FleetAlert>? Data { get; set; }
|
||||
}
|
||||
|
||||
// List of alerts that parsed, plus a note per element that did not. The
|
||||
// caller logs Skipped; one malformed alert must not drop the rest.
|
||||
public sealed class FleetAlertList : List<FleetAlert>
|
||||
{
|
||||
public List<string> Skipped { get; } = new();
|
||||
}
|
||||
|
||||
// Deserializes the data array one element at a time. An element that fails
|
||||
// to bind, or has no usable notificationid, is recorded and skipped.
|
||||
public sealed class TolerantFleetAlertListConverter : JsonConverter<List<FleetAlert>>
|
||||
{
|
||||
public override List<FleetAlert>? Read(
|
||||
ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
|
||||
{
|
||||
if (reader.TokenType == JsonTokenType.Null)
|
||||
return null;
|
||||
if (reader.TokenType != JsonTokenType.StartArray)
|
||||
throw new JsonException("Fleet alert data is not an array.");
|
||||
|
||||
var list = new FleetAlertList();
|
||||
var index = 0;
|
||||
while (reader.Read() && reader.TokenType != JsonTokenType.EndArray)
|
||||
{
|
||||
using var element = JsonDocument.ParseValue(ref reader);
|
||||
try
|
||||
{
|
||||
var alert = element.RootElement.Deserialize<FleetAlert>(options);
|
||||
if (alert is null || alert.NotificationId <= 0)
|
||||
list.Skipped.Add($"element {index}: missing notificationid");
|
||||
else
|
||||
list.Add(alert);
|
||||
}
|
||||
catch (Exception exception) when (exception is JsonException or InvalidOperationException)
|
||||
{
|
||||
list.Skipped.Add($"element {index}: {exception.Message}");
|
||||
}
|
||||
index++;
|
||||
}
|
||||
return list;
|
||||
}
|
||||
|
||||
public override void Write(
|
||||
Utf8JsonWriter writer, List<FleetAlert> value, JsonSerializerOptions options)
|
||||
{
|
||||
JsonSerializer.Serialize(writer, value.ToArray(), options);
|
||||
}
|
||||
}
|
||||
190
agent/ShopDbEndpointAgent.Core/FleetAlertStore.cs
Normal file
@@ -0,0 +1,190 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Local JSON file store for tracking acknowledged fleet alerts.
|
||||
// Stores notification IDs that have been acknowledged so they don't
|
||||
// reappear in future polls. Lives under %LOCALAPPDATA% because the tray runs
|
||||
// as the standard operator, who cannot write to %ProgramData%. Read-modify-write
|
||||
// is serialized across processes by a named mutex.
|
||||
public sealed class FleetAlertStore
|
||||
{
|
||||
private const int MaxIds = 1000;
|
||||
private const string MutexName = @"Local\ShopDB-FleetAlertStore";
|
||||
private static readonly TimeSpan MutexWait = TimeSpan.FromSeconds(5);
|
||||
|
||||
private static readonly string DataDir = Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData),
|
||||
"ShopDB",
|
||||
"FleetAlerts");
|
||||
|
||||
private static readonly string StoreFile = Path.Combine(DataDir, "acknowledged-alerts.json");
|
||||
|
||||
// Pre-1.6.11 location. Standard users could not write it, but an elevated
|
||||
// run may have left IDs there; merged once.
|
||||
private static readonly string LegacyStoreFile = Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"ShopDB",
|
||||
"FleetAlerts",
|
||||
"acknowledged-alerts.json");
|
||||
|
||||
private readonly object fileLock = new();
|
||||
private readonly ILogger logger;
|
||||
private HashSet<int> acknowledged = new();
|
||||
|
||||
public FleetAlertStore(ILogger logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
Load();
|
||||
}
|
||||
|
||||
public bool IsAcknowledged(int notificationId)
|
||||
{
|
||||
lock (fileLock)
|
||||
{
|
||||
return acknowledged.Contains(notificationId);
|
||||
}
|
||||
}
|
||||
|
||||
public void MarkAcknowledged(int notificationId)
|
||||
{
|
||||
lock (fileLock)
|
||||
{
|
||||
WithFileMutex(() =>
|
||||
{
|
||||
// Merge what another instance may have written since our load.
|
||||
var onDisk = ReadFile(StoreFile);
|
||||
if (onDisk?.AcknowledgedIds is not null)
|
||||
acknowledged.UnionWith(onDisk.AcknowledgedIds);
|
||||
|
||||
acknowledged.Add(notificationId);
|
||||
Prune();
|
||||
Save(onDisk?.MigratedLegacy ?? true);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private void Load()
|
||||
{
|
||||
lock (fileLock)
|
||||
{
|
||||
WithFileMutex(() =>
|
||||
{
|
||||
var stored = ReadFile(StoreFile);
|
||||
acknowledged = stored?.AcknowledgedIds?.ToHashSet() ?? new HashSet<int>();
|
||||
|
||||
if (stored?.MigratedLegacy == true)
|
||||
return;
|
||||
|
||||
var legacy = ReadFile(LegacyStoreFile);
|
||||
var merged = legacy?.AcknowledgedIds?.Count(id => acknowledged.Add(id)) ?? 0;
|
||||
if (merged > 0)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Merged {Count} acknowledged alert ID(s) from legacy store {Path}.",
|
||||
merged, LegacyStoreFile);
|
||||
Prune();
|
||||
}
|
||||
if (merged > 0 || stored is not null)
|
||||
Save(migratedLegacy: true);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private StoredData? ReadFile(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
return null;
|
||||
return JsonSerializer.Deserialize<StoredData>(File.ReadAllText(path));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not read acknowledged alerts store {Path}.", path);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private void Save(bool migratedLegacy)
|
||||
{
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(DataDir);
|
||||
|
||||
var data = new StoredData
|
||||
{
|
||||
AcknowledgedIds = acknowledged.ToList(),
|
||||
LastUpdated = DateTime.UtcNow,
|
||||
MigratedLegacy = migratedLegacy,
|
||||
};
|
||||
|
||||
var json = JsonSerializer.Serialize(data, new JsonSerializerOptions
|
||||
{
|
||||
WriteIndented = true
|
||||
});
|
||||
|
||||
var temp = StoreFile + ".tmp";
|
||||
File.WriteAllText(temp, json);
|
||||
File.Move(temp, StoreFile, overwrite: true);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogError(ex,
|
||||
"Could not save acknowledged alerts store {Path}; acknowledged alerts may reappear.",
|
||||
StoreFile);
|
||||
}
|
||||
}
|
||||
|
||||
// Runs action under the cross-process mutex. If the mutex cannot be had,
|
||||
// logs and runs anyway: a rare clobber beats never recording an ack.
|
||||
private void WithFileMutex(Action action)
|
||||
{
|
||||
Mutex? mutex = null;
|
||||
var owned = false;
|
||||
try
|
||||
{
|
||||
mutex = new Mutex(false, MutexName);
|
||||
try
|
||||
{
|
||||
owned = mutex.WaitOne(MutexWait);
|
||||
}
|
||||
catch (AbandonedMutexException)
|
||||
{
|
||||
owned = true;
|
||||
}
|
||||
if (!owned)
|
||||
logger.LogWarning("Timed out waiting for fleet alert store mutex; proceeding without it.");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex, "Could not open fleet alert store mutex; proceeding without it.");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
action();
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (owned)
|
||||
mutex!.ReleaseMutex();
|
||||
mutex?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
private void Prune()
|
||||
{
|
||||
// Keep only newest IDs to prevent unbounded growth
|
||||
if (acknowledged.Count > MaxIds)
|
||||
acknowledged = acknowledged.OrderByDescending(id => id).Take(MaxIds).ToHashSet();
|
||||
}
|
||||
|
||||
private sealed class StoredData
|
||||
{
|
||||
public List<int> AcknowledgedIds { get; set; } = new();
|
||||
public DateTime LastUpdated { get; set; }
|
||||
public bool MigratedLegacy { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
using System.ComponentModel;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Security.Principal;
|
||||
using System.Text;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
@@ -18,7 +20,11 @@ namespace ShopDbEndpointAgent.Core;
|
||||
// Only process_start uses this. Stop/status/service actions stay as SYSTEM.
|
||||
public static class InteractiveProcessLauncher
|
||||
{
|
||||
public readonly record struct LaunchOutcome(bool Launched, string? Error, int? ExitCode = null);
|
||||
// TimedOut=true means the process launched fine but was still running when
|
||||
// the caller's wait budget expired. It is deliberately NOT killed, so the
|
||||
// work still completes; the caller just stops blocking on it.
|
||||
public readonly record struct LaunchOutcome(
|
||||
bool Launched, string? Error, int? ExitCode = null, bool TimedOut = false);
|
||||
|
||||
private const int TokenDuplicate = 0x0002;
|
||||
private const int TokenQuery = 0x0008;
|
||||
@@ -34,6 +40,10 @@ public static class InteractiveProcessLauncher
|
||||
|
||||
private const uint Infinite = 0xFFFFFFFF;
|
||||
|
||||
// WaitForSingleObject returns this when the budget expires with the handle
|
||||
// still unsignalled.
|
||||
private const uint WaitTimeout = 0x00000102;
|
||||
|
||||
// Returns Launched=true when CreateProcessAsUser succeeds. Never throws;
|
||||
// failures come back as Launched=false with a descriptive Error so the
|
||||
// caller can surface a clean CommandResult.
|
||||
@@ -51,9 +61,17 @@ public static class InteractiveProcessLauncher
|
||||
// needs the actual outcome (run_script in the user session) can report
|
||||
// completed/failed instead of fire-and-forget "launched". GUI callers
|
||||
// (process_start) leave it false and return as soon as the window is up.
|
||||
//
|
||||
// waitTimeout bounds that wait. This matters because the caller may be the
|
||||
// telemetry loop, and telemetry IS the heartbeat: a script that hangs in the
|
||||
// user session used to block here forever (the wait was hardcoded to
|
||||
// Infinite), which would stall reporting and make a perfectly healthy PC
|
||||
// read as offline in the WebUI. On expiry the child is left running and the
|
||||
// outcome comes back with TimedOut=true. Null means wait forever and should
|
||||
// only be used by a caller that is not on a timed path.
|
||||
public static LaunchOutcome LaunchInActiveSession(
|
||||
string applicationPath, string? arguments, string? workingDirectory,
|
||||
bool waitForExit = false)
|
||||
bool waitForExit = false, TimeSpan? waitTimeout = null)
|
||||
{
|
||||
var sessionId = WTSGetActiveConsoleSessionId();
|
||||
if (sessionId == InvalidSessionId)
|
||||
@@ -129,11 +147,25 @@ public static class InteractiveProcessLauncher
|
||||
}
|
||||
|
||||
int? exitCode = null;
|
||||
bool timedOut = false;
|
||||
if (waitForExit && processInformation.hProcess != IntPtr.Zero)
|
||||
{
|
||||
WaitForSingleObject(processInformation.hProcess, Infinite);
|
||||
if (GetExitCodeProcess(processInformation.hProcess, out var code))
|
||||
uint milliseconds = waitTimeout is TimeSpan budget
|
||||
? (uint)Math.Clamp(budget.TotalMilliseconds, 0, uint.MaxValue - 1)
|
||||
: Infinite;
|
||||
|
||||
if (WaitForSingleObject(processInformation.hProcess, milliseconds)
|
||||
== WaitTimeout)
|
||||
{
|
||||
// Still running. Leave it alone - killing it would abandon
|
||||
// half-done work, and a file-spooling probe will still write
|
||||
// its result for the next cycle to pick up.
|
||||
timedOut = true;
|
||||
}
|
||||
else if (GetExitCodeProcess(processInformation.hProcess, out var code))
|
||||
{
|
||||
exitCode = unchecked((int)code);
|
||||
}
|
||||
}
|
||||
|
||||
if (processInformation.hProcess != IntPtr.Zero)
|
||||
@@ -141,7 +173,7 @@ public static class InteractiveProcessLauncher
|
||||
if (processInformation.hThread != IntPtr.Zero)
|
||||
CloseHandle(processInformation.hThread);
|
||||
|
||||
return new LaunchOutcome(true, null, exitCode);
|
||||
return new LaunchOutcome(true, null, exitCode, timedOut);
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -154,6 +186,93 @@ public static class InteractiveProcessLauncher
|
||||
}
|
||||
}
|
||||
|
||||
// Profile directory of the active console user, for example
|
||||
// C:\Users\ShopFloor. The agent runs as LocalSystem, so expanding
|
||||
// %USERPROFILE% here would return SYSTEM's own profile under
|
||||
// config\systemprofile rather than the operator's. The interactive user's
|
||||
// token is the only reliable source, and this class already knows how to
|
||||
// obtain it. Returns null with a reason when nobody is signed in.
|
||||
public static string? TryGetActiveUserProfileDirectory(out string? error)
|
||||
{
|
||||
var sessionId = WTSGetActiveConsoleSessionId();
|
||||
if (sessionId == InvalidSessionId)
|
||||
{
|
||||
error = "No active console session (no interactive user is signed in).";
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!WTSQueryUserToken(sessionId, out var userToken))
|
||||
{
|
||||
error = "Could not obtain the interactive user token: "
|
||||
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
uint size = 0;
|
||||
GetUserProfileDirectory(userToken, null, ref size);
|
||||
if (size == 0)
|
||||
{
|
||||
error = "Could not size the user profile directory: "
|
||||
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
|
||||
return null;
|
||||
}
|
||||
|
||||
var buffer = new StringBuilder((int)size);
|
||||
if (!GetUserProfileDirectory(userToken, buffer, ref size))
|
||||
{
|
||||
error = "Could not resolve the user profile directory: "
|
||||
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
|
||||
return null;
|
||||
}
|
||||
|
||||
error = null;
|
||||
return buffer.ToString();
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (userToken != IntPtr.Zero)
|
||||
CloseHandle(userToken);
|
||||
}
|
||||
}
|
||||
|
||||
// SID of the active console user, so SYSTEM can grant that one account
|
||||
// access to a file it stages. Null with a reason when nobody is signed in.
|
||||
public static SecurityIdentifier? TryGetActiveUserSid(out string? error)
|
||||
{
|
||||
var sessionId = WTSGetActiveConsoleSessionId();
|
||||
if (sessionId == InvalidSessionId)
|
||||
{
|
||||
error = "No active console session (no interactive user is signed in).";
|
||||
return null;
|
||||
}
|
||||
|
||||
if (!WTSQueryUserToken(sessionId, out var userToken))
|
||||
{
|
||||
error = "Could not obtain the interactive user token (WTSQueryUserToken failed): "
|
||||
+ new Win32Exception(Marshal.GetLastWin32Error()).Message;
|
||||
return null;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var identity = new WindowsIdentity(userToken);
|
||||
error = identity.User is null ? "Interactive user token carries no SID." : null;
|
||||
return identity.User;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
error = "Could not read the interactive user SID: " + exception.Message;
|
||||
return null;
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (userToken != IntPtr.Zero)
|
||||
CloseHandle(userToken);
|
||||
}
|
||||
}
|
||||
|
||||
private enum SecurityImpersonationLevel
|
||||
{
|
||||
SecurityAnonymous,
|
||||
@@ -222,6 +341,10 @@ public static class InteractiveProcessLauncher
|
||||
[DllImport("userenv.dll", SetLastError = true)]
|
||||
private static extern bool DestroyEnvironmentBlock(IntPtr environment);
|
||||
|
||||
[DllImport("userenv.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern bool GetUserProfileDirectory(
|
||||
IntPtr token, StringBuilder? profileDirectory, ref uint size);
|
||||
|
||||
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern bool CreateProcessAsUser(
|
||||
IntPtr token,
|
||||
|
||||
285
agent/ShopDbEndpointAgent.Core/ProtectedFileSystem.cs
Normal file
@@ -0,0 +1,285 @@
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Security.AccessControl;
|
||||
using System.Security.Principal;
|
||||
using System.Text;
|
||||
using Microsoft.Win32.SafeHandles;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// File access for paths the agent touches AS SYSTEM where a standard user could
|
||||
// otherwise interfere.
|
||||
//
|
||||
// %ProgramData% lets any user create files and folders beneath it, so a staging
|
||||
// path the agent does not lock down can be pre-created, swapped between the hash
|
||||
// check and execution, or replaced with a junction that redirects a SYSTEM read
|
||||
// or delete to an arbitrary file. Directories made here carry a protected DACL:
|
||||
// SYSTEM and BUILTIN\Administrators only, inheritance off. SIDs are well-known
|
||||
// strings so a localized OS resolves them the same way.
|
||||
internal static class ProtectedFileSystem
|
||||
{
|
||||
private static readonly SecurityIdentifier LocalSystemSid = new("S-1-5-18");
|
||||
private static readonly SecurityIdentifier AdministratorsSid = new("S-1-5-32-544");
|
||||
|
||||
private const uint GenericRead = 0x80000000;
|
||||
private const uint FileShareRead = 0x00000001;
|
||||
private const uint FileShareWrite = 0x00000002;
|
||||
private const uint FileShareDelete = 0x00000004;
|
||||
private const uint OpenExisting = 3;
|
||||
private const uint FileFlagOpenReparsePoint = 0x00200000;
|
||||
|
||||
// %ProgramData%\ShopDB\EndpointAgent. The Defender-excluded agent root.
|
||||
public static string AgentDataRoot => Path.Combine(
|
||||
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
|
||||
"ShopDB", "EndpointAgent");
|
||||
|
||||
// Create or re-secure <AgentDataRoot>\<name>. readerSid, when set, gets
|
||||
// list/traverse on the directory itself (not inherited) so a process in that
|
||||
// user's session can run from it and open files it was granted explicitly.
|
||||
// Throws when the directory cannot be proven safe; callers fail closed.
|
||||
public static string EnsureDirectory(string name, SecurityIdentifier? readerSid = null)
|
||||
{
|
||||
var root = AgentDataRoot;
|
||||
Directory.CreateDirectory(root);
|
||||
RefuseReparseChain(root);
|
||||
|
||||
var path = Path.Combine(root, name);
|
||||
var attributes = TryGetAttributes(path);
|
||||
if (attributes is FileAttributes existing
|
||||
&& (existing.HasFlag(FileAttributes.ReparsePoint)
|
||||
|| !existing.HasFlag(FileAttributes.Directory)))
|
||||
{
|
||||
// Link or planted file at our name. Removing a link never touches
|
||||
// its target.
|
||||
if (existing.HasFlag(FileAttributes.Directory))
|
||||
Directory.Delete(path);
|
||||
else
|
||||
File.Delete(path);
|
||||
attributes = null;
|
||||
}
|
||||
|
||||
if (attributes is null)
|
||||
new DirectoryInfo(path).Create(DirectorySecurityFor(readerSid));
|
||||
|
||||
var info = new DirectoryInfo(path);
|
||||
if (!IsTrustedOwner(info))
|
||||
{
|
||||
// Pre-created by someone else. The owner can always rewrite the
|
||||
// DACL, so re-securing is not enough: replace the directory.
|
||||
Directory.Delete(path, recursive: true);
|
||||
new DirectoryInfo(path).Create(DirectorySecurityFor(readerSid));
|
||||
info = new DirectoryInfo(path);
|
||||
if (!IsTrustedOwner(info))
|
||||
throw new IOException($"Staging directory {path} has an untrusted owner");
|
||||
}
|
||||
|
||||
if (IsReparsePoint(path))
|
||||
throw new IOException($"Staging directory {path} is a reparse point");
|
||||
|
||||
info.SetAccessControl(DirectorySecurityFor(readerSid));
|
||||
return path;
|
||||
}
|
||||
|
||||
// New file inside a protected directory. Any existing entry at the name is
|
||||
// removed first, then CreateNew so nothing can be slipped in between. The
|
||||
// DACL is protected; userSid, when set, is granted userRights explicitly.
|
||||
public static FileStream CreateFile(
|
||||
string path, SecurityIdentifier? userSid = null,
|
||||
FileSystemRights userRights = FileSystemRights.ReadAndExecute)
|
||||
{
|
||||
if (TryGetAttributes(path) is not null)
|
||||
File.Delete(path);
|
||||
|
||||
var security = new FileSecurity();
|
||||
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
LocalSystemSid, FileSystemRights.FullControl, AccessControlType.Allow));
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
AdministratorsSid, FileSystemRights.FullControl, AccessControlType.Allow));
|
||||
if (userSid is not null)
|
||||
security.AddAccessRule(new FileSystemAccessRule(
|
||||
userSid, userRights, AccessControlType.Allow));
|
||||
|
||||
return new FileInfo(path).Create(
|
||||
FileMode.CreateNew, FileSystemRights.Read | FileSystemRights.Write,
|
||||
FileShare.Read, 4096, FileOptions.None, security);
|
||||
}
|
||||
|
||||
// Attributes of the entry itself (a link reports its own, not its target's).
|
||||
public static FileAttributes? TryGetAttributes(string path)
|
||||
{
|
||||
try
|
||||
{
|
||||
return File.GetAttributes(path);
|
||||
}
|
||||
catch (FileNotFoundException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
catch (DirectoryNotFoundException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public static bool IsReparsePoint(string path)
|
||||
=> TryGetAttributes(path) is FileAttributes attributes
|
||||
&& attributes.HasFlag(FileAttributes.ReparsePoint);
|
||||
|
||||
// Open for reading without following a reparse point at the final
|
||||
// component, and refuse if the opened object is one. Share mode is wide so
|
||||
// files a running application holds open still read.
|
||||
public static FileStream OpenReadNoFollow(string path)
|
||||
{
|
||||
var handle = CreateFileW(path, GenericRead,
|
||||
FileShareRead | FileShareWrite | FileShareDelete, IntPtr.Zero,
|
||||
OpenExisting, FileFlagOpenReparsePoint, IntPtr.Zero);
|
||||
if (handle.IsInvalid)
|
||||
{
|
||||
var hr = Marshal.GetHRForLastWin32Error();
|
||||
handle.Dispose();
|
||||
throw Marshal.GetExceptionForHR(hr) ?? new IOException($"Could not open {path}");
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
if (!GetFileInformationByHandle(handle, out var information))
|
||||
throw Marshal.GetExceptionForHR(Marshal.GetHRForLastWin32Error())
|
||||
?? new IOException($"Could not query {path}");
|
||||
|
||||
var attributes = (FileAttributes)information.FileAttributes;
|
||||
if (attributes.HasFlag(FileAttributes.ReparsePoint))
|
||||
throw new UnauthorizedAccessException($"{path} is a reparse point");
|
||||
if (attributes.HasFlag(FileAttributes.Directory))
|
||||
throw new UnauthorizedAccessException($"{path} is a directory");
|
||||
|
||||
return new FileStream(handle, FileAccess.Read);
|
||||
}
|
||||
catch
|
||||
{
|
||||
handle.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
// The path the handle really refers to after every junction and link in
|
||||
// its ancestry is resolved. Null when it cannot be read.
|
||||
public static string? FinalPath(SafeFileHandle handle)
|
||||
{
|
||||
var buffer = new StringBuilder(1024);
|
||||
var length = GetFinalPathNameByHandleW(handle, buffer, (uint)buffer.Capacity, 0);
|
||||
if (length == 0)
|
||||
return null;
|
||||
if (length >= buffer.Capacity)
|
||||
{
|
||||
buffer = new StringBuilder((int)length + 1);
|
||||
length = GetFinalPathNameByHandleW(handle, buffer, (uint)buffer.Capacity, 0);
|
||||
if (length == 0 || length >= buffer.Capacity)
|
||||
return null;
|
||||
}
|
||||
|
||||
var final = buffer.ToString();
|
||||
if (final.StartsWith(@"\\?\UNC\", StringComparison.OrdinalIgnoreCase))
|
||||
return @"\\" + final[8..];
|
||||
if (final.StartsWith(@"\\?\", StringComparison.Ordinal))
|
||||
return final[4..];
|
||||
return final;
|
||||
}
|
||||
|
||||
// Read at most maxBytes of text from the head of a file, BOM-aware.
|
||||
public static string ReadTextHeadNoFollow(string path, int maxBytes)
|
||||
{
|
||||
using var stream = OpenReadNoFollow(path);
|
||||
var buffer = new byte[(int)Math.Min(maxBytes, Math.Max(stream.Length, 0))];
|
||||
var total = 0;
|
||||
int read;
|
||||
while (total < buffer.Length
|
||||
&& (read = stream.Read(buffer, total, buffer.Length - total)) > 0)
|
||||
total += read;
|
||||
|
||||
using var reader = new StreamReader(
|
||||
new MemoryStream(buffer, 0, total), Encoding.UTF8, true);
|
||||
return reader.ReadToEnd();
|
||||
}
|
||||
|
||||
// Delete the entry itself. DeleteFile removes a link, never its target;
|
||||
// a directory at a file path is left alone.
|
||||
public static void DeleteFileNoFollow(string path)
|
||||
{
|
||||
if (TryGetAttributes(path) is not FileAttributes attributes)
|
||||
return;
|
||||
if (attributes.HasFlag(FileAttributes.Directory))
|
||||
throw new IOException($"{path} is a directory, not a file");
|
||||
File.Delete(path);
|
||||
}
|
||||
|
||||
private static DirectorySecurity DirectorySecurityFor(SecurityIdentifier? readerSid)
|
||||
{
|
||||
var security = new DirectorySecurity();
|
||||
security.SetAccessRuleProtection(isProtected: true, preserveInheritance: false);
|
||||
const InheritanceFlags inherit =
|
||||
InheritanceFlags.ContainerInherit | InheritanceFlags.ObjectInherit;
|
||||
security.AddAccessRule(new FileSystemAccessRule(LocalSystemSid,
|
||||
FileSystemRights.FullControl, inherit, PropagationFlags.None,
|
||||
AccessControlType.Allow));
|
||||
security.AddAccessRule(new FileSystemAccessRule(AdministratorsSid,
|
||||
FileSystemRights.FullControl, inherit, PropagationFlags.None,
|
||||
AccessControlType.Allow));
|
||||
if (readerSid is not null)
|
||||
security.AddAccessRule(new FileSystemAccessRule(readerSid,
|
||||
FileSystemRights.ReadAndExecute, InheritanceFlags.None,
|
||||
PropagationFlags.None, AccessControlType.Allow));
|
||||
return security;
|
||||
}
|
||||
|
||||
private static bool IsTrustedOwner(DirectoryInfo info)
|
||||
{
|
||||
var owner = info.GetAccessControl(AccessControlSections.Owner)
|
||||
.GetOwner(typeof(SecurityIdentifier)) as SecurityIdentifier;
|
||||
return owner is not null
|
||||
&& (owner.Equals(LocalSystemSid) || owner.Equals(AdministratorsSid));
|
||||
}
|
||||
|
||||
// A junction anywhere from ProgramData down to the agent root would move
|
||||
// every "protected" directory somewhere else.
|
||||
private static void RefuseReparseChain(string root)
|
||||
{
|
||||
var programData = Environment.GetFolderPath(
|
||||
Environment.SpecialFolder.CommonApplicationData).TrimEnd('\\');
|
||||
var current = Path.GetFullPath(root).TrimEnd('\\');
|
||||
while (current.Length > programData.Length
|
||||
&& current.StartsWith(programData, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
if (IsReparsePoint(current))
|
||||
throw new IOException($"{current} is a reparse point; refusing to stage under it");
|
||||
current = Path.GetDirectoryName(current) ?? "";
|
||||
}
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct ByHandleFileInformation
|
||||
{
|
||||
public uint FileAttributes;
|
||||
public System.Runtime.InteropServices.ComTypes.FILETIME CreationTime;
|
||||
public System.Runtime.InteropServices.ComTypes.FILETIME LastAccessTime;
|
||||
public System.Runtime.InteropServices.ComTypes.FILETIME LastWriteTime;
|
||||
public uint VolumeSerialNumber;
|
||||
public uint FileSizeHigh;
|
||||
public uint FileSizeLow;
|
||||
public uint NumberOfLinks;
|
||||
public uint FileIndexHigh;
|
||||
public uint FileIndexLow;
|
||||
}
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern SafeFileHandle CreateFileW(
|
||||
string fileName, uint desiredAccess, uint shareMode, IntPtr securityAttributes,
|
||||
uint creationDisposition, uint flagsAndAttributes, IntPtr templateFile);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
private static extern bool GetFileInformationByHandle(
|
||||
SafeFileHandle file, out ByHandleFileInformation information);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern uint GetFinalPathNameByHandleW(
|
||||
SafeFileHandle file, StringBuilder filePath, uint filePathLength, uint flags);
|
||||
}
|
||||
159
agent/ShopDbEndpointAgent.Core/ScheduledTaskPolicy.cs
Normal file
@@ -0,0 +1,159 @@
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Guards which registered scheduled tasks the scheduledtask_run action may
|
||||
// start. The wire carries a free-text task path (so the WebUI can expose it as
|
||||
// one field, and a site can widen the set without an agent rebuild); this
|
||||
// policy is the safeguard that makes that free text safe to honor.
|
||||
//
|
||||
// WHY THIS EXISTS AT ALL. The agent runs as SYSTEM. A scheduled task is an
|
||||
// INDIRECTION: the task path names a definition, and that definition chooses
|
||||
// what actually runs. So "start the task named X" is only as safe as the set of
|
||||
// X the agent will accept - an unconstrained task name would be a general-
|
||||
// purpose "run anything already registered on this box, as whatever context it
|
||||
// asked for" primitive. That is a materially bigger grant than any other
|
||||
// allowlisted action, and it would be invisible in the audit trail (the command
|
||||
// log would read "scheduledtask_run \Foo" no matter what \Foo does).
|
||||
//
|
||||
// WHAT THIS DOES AND DOES NOT BUY. A prefix allowlist stops a compromised or
|
||||
// misconfigured SERVER from starting arbitrary tasks on the fleet - that is the
|
||||
// threat it actually addresses, and it is the realistic one. It does NOT stop an
|
||||
// attacker who can already register or rewrite task definitions on the endpoint:
|
||||
// such an attacker can define a task under an approved prefix and have the agent
|
||||
// start it. That is accepted, because anyone who can write task definitions on a
|
||||
// box already has the privilege the task would grant them. Local task definition
|
||||
// ACLs, not this policy, are the control for that.
|
||||
//
|
||||
// The approved-prefix list ships with the Corporate Tools set and is overridable
|
||||
// per site via AgentConfig.ApprovedTaskPaths, matching the LaunchPathPolicy
|
||||
// pattern so the future Site Agent Configuration tool can narrow or widen it
|
||||
// without a rebuild.
|
||||
public sealed class ScheduledTaskPolicy
|
||||
{
|
||||
// Shipped defaults: the Corporate Tools task set. Prefix entries end with a
|
||||
// backslash and match a whole folder; entries without one match that exact
|
||||
// task path only. Kept deliberately tight - widen per site, not here.
|
||||
//
|
||||
// The ShopDB-* entries are tasks ShopDB itself owns. They are separate task
|
||||
// definitions from any corporate-initiated equivalent ON PURPOSE: a
|
||||
// ShopDB-initiated Dell Command scan must write its results somewhere the
|
||||
// agent's collector can attribute to ShopDB, rather than into whatever
|
||||
// location a corporate-managed task already writes. Keeping the definitions
|
||||
// distinct is what makes "who asked for this scan" answerable at all - see
|
||||
// the note in commands.py about the DCU output path convention.
|
||||
public static IReadOnlyList<string> DefaultApprovedTaskPaths { get; } = new[]
|
||||
{
|
||||
@"\ReportIP",
|
||||
@"\SFLD-ApplyDSCConfig",
|
||||
@"\SFLD - Consume Credentials",
|
||||
@"\ShopDB-DellCommandScan",
|
||||
@"\ShopDB-DellCommandUpdate",
|
||||
};
|
||||
|
||||
private readonly IReadOnlyList<string> approvedPaths;
|
||||
|
||||
public ScheduledTaskPolicy(IEnumerable<string>? approvedPathsOverride = null)
|
||||
{
|
||||
var paths = (approvedPathsOverride ?? Enumerable.Empty<string>())
|
||||
.Where(path => !string.IsNullOrWhiteSpace(path))
|
||||
.Select(NormalizeTaskPath)
|
||||
.ToList();
|
||||
approvedPaths = paths.Count > 0 ? paths : DefaultApprovedTaskPaths;
|
||||
}
|
||||
|
||||
// Validates a candidate task path. On success taskPath is the normalized
|
||||
// path to hand to Task Scheduler; on failure reason explains the rejection
|
||||
// so the command result tells the operator WHY rather than just "denied".
|
||||
public bool TryValidate(string? candidate, out string taskPath, out string reason)
|
||||
{
|
||||
taskPath = "";
|
||||
reason = "";
|
||||
|
||||
if (string.IsNullOrWhiteSpace(candidate))
|
||||
{
|
||||
reason = "empty task path";
|
||||
return false;
|
||||
}
|
||||
|
||||
var normalized = NormalizeTaskPath(candidate);
|
||||
|
||||
// Traversal in a task path is meaningless to Task Scheduler but very
|
||||
// meaningful to a prefix check: "\Approved\..\Evil" starts with an
|
||||
// approved prefix as a STRING while naming something else entirely.
|
||||
// Reject rather than try to canonicalize it away.
|
||||
if (normalized.Contains(".."))
|
||||
{
|
||||
reason = "task path contains a traversal segment";
|
||||
return false;
|
||||
}
|
||||
|
||||
// Control characters and quotes have no place in a task path and are the
|
||||
// shape of an attempt to break out of the argument. The invocation uses
|
||||
// ArgumentList (no shell, no string concatenation) so this is defense in
|
||||
// depth rather than the primary control, but a task path that needs
|
||||
// these is not a task path we should be starting.
|
||||
foreach (var ch in normalized)
|
||||
{
|
||||
if (char.IsControl(ch) || ch == '"' || ch == '&' || ch == '|')
|
||||
{
|
||||
reason = "task path contains an illegal character";
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (!IsApproved(normalized))
|
||||
{
|
||||
reason = "task path is not in the approved task list";
|
||||
return false;
|
||||
}
|
||||
|
||||
taskPath = normalized;
|
||||
return true;
|
||||
}
|
||||
|
||||
// Task Scheduler paths are rooted and backslash-separated. Accept the
|
||||
// operator writing "ReportIP", "\ReportIP", or "/ReportIP" and settle on the
|
||||
// rooted form, so an allowlist entry does not hinge on how it was typed.
|
||||
private static string NormalizeTaskPath(string path)
|
||||
{
|
||||
var trimmed = (path ?? "").Trim().Replace('/', '\\');
|
||||
if (trimmed.Length == 0)
|
||||
return "";
|
||||
if (!trimmed.StartsWith('\\'))
|
||||
trimmed = "\\" + trimmed;
|
||||
// Collapse a trailing separator so "\Folder\" and "\Folder" compare
|
||||
// equal as exact entries; prefix semantics are handled in IsApproved.
|
||||
while (trimmed.Length > 1 && trimmed.EndsWith('\\'))
|
||||
trimmed = trimmed[..^1];
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
// An entry ending in a separator is a FOLDER prefix and matches anything
|
||||
// beneath it; any other entry must match the whole task path exactly. The
|
||||
// exact-match case is why "\Approved" does not silently authorize
|
||||
// "\ApprovedEvil".
|
||||
private bool IsApproved(string candidate)
|
||||
{
|
||||
foreach (var entry in approvedPaths)
|
||||
{
|
||||
if (entry.Length == 0)
|
||||
continue;
|
||||
|
||||
var isFolderPrefix = entry.EndsWith('\\');
|
||||
if (isFolderPrefix)
|
||||
{
|
||||
if (candidate.StartsWith(entry, StringComparison.OrdinalIgnoreCase))
|
||||
return true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (string.Equals(candidate, entry, StringComparison.OrdinalIgnoreCase))
|
||||
return true;
|
||||
|
||||
// Allow the folder form without its trailing separator to still act
|
||||
// as a folder: "\SFLD" approves "\SFLD\Thing" but never "\SFLDEvil".
|
||||
if (candidate.StartsWith(entry + "\\", StringComparison.OrdinalIgnoreCase))
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -15,8 +15,17 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Options" Version="8.0.0" />
|
||||
<PackageReference Include="Microsoft.Win32.Registry" Version="5.0.0" />
|
||||
<PackageReference Include="System.Management" Version="8.0.0" />
|
||||
<!-- DPAPI LocalMachine for the per-PC credential (CredentialStore). -->
|
||||
<PackageReference Include="System.Security.Cryptography.ProtectedData" Version="8.0.0" />
|
||||
<PackageReference Include="System.Diagnostics.EventLog" Version="8.0.0" />
|
||||
<PackageReference Include="System.ServiceProcess.ServiceController" Version="8.0.0" />
|
||||
<!-- EventWaitHandleAcl: creating the user-context refresh event with an
|
||||
explicit DACL. The service creates it as LocalSystem, and without a
|
||||
granted ACE the unelevated tray cannot open it. -->
|
||||
<PackageReference Include="System.Threading.AccessControl" Version="8.0.0" />
|
||||
<!-- FileSystem.AccessControl: protected DACLs on staging, credential and
|
||||
appsettings.json files (ProtectedFileSystem, AgentPipeServer). -->
|
||||
<PackageReference Include="System.IO.FileSystem.AccessControl" Version="5.0.0" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
51
agent/ShopDbEndpointAgent.Core/StartupJitter.cs
Normal file
@@ -0,0 +1,51 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// One-time startup delay, shared by TelemetryWorker and CommandWorker.
|
||||
//
|
||||
// Both loops otherwise make their first request the instant the service starts,
|
||||
// and the service is installed start=auto - so a patch-day reboot has an entire
|
||||
// fleet arriving together, twice per machine (a telemetry report AND a command
|
||||
// poll). AgentConfig.JitterSeconds does not help: it spreads the gap BETWEEN
|
||||
// runs, not the first contact.
|
||||
//
|
||||
// Lives here rather than in either worker so the two cannot drift apart, and so
|
||||
// the clamp is enforced in one place.
|
||||
internal static class StartupJitter
|
||||
{
|
||||
// Ceiling on the configured value. A fat-fingered appsettings.json must not
|
||||
// be able to park an endpoint for hours before it first checks in: the
|
||||
// command channel is how an operator would correct it, so an unbounded
|
||||
// delay would take away the means of recovery.
|
||||
private const int MaxStartupJitterSeconds = 300;
|
||||
|
||||
public static TimeSpan Next(int configuredSeconds, ILogger logger)
|
||||
{
|
||||
// Warn rather than clamp silently: an operator who deliberately set a
|
||||
// large value and got a smaller one has config and behaviour disagreeing
|
||||
// with no way to tell why. A negative value is a mistake too, so it is
|
||||
// worth the same line.
|
||||
if (configuredSeconds > MaxStartupJitterSeconds)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"startupjitterseconds {Configured} exceeds the {Max}s ceiling; using {Max}s",
|
||||
configuredSeconds, MaxStartupJitterSeconds, MaxStartupJitterSeconds);
|
||||
}
|
||||
else if (configuredSeconds < 0)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"startupjitterseconds {Configured} is negative; startup jitter disabled",
|
||||
configuredSeconds);
|
||||
}
|
||||
|
||||
int spread = Math.Clamp(configuredSeconds, 0, MaxStartupJitterSeconds);
|
||||
if (spread == 0)
|
||||
return TimeSpan.Zero;
|
||||
|
||||
// Milliseconds, not whole seconds: 700 endpoints over a 30s window would
|
||||
// otherwise land in 30 clumps of ~23 simultaneous arrivals, which is the
|
||||
// same pile-up in miniature.
|
||||
return TimeSpan.FromMilliseconds(Random.Shared.Next(0, spread * 1000));
|
||||
}
|
||||
}
|
||||
293
agent/ShopDbEndpointAgent.Core/StrictModePreamble.cs
Normal file
@@ -0,0 +1,293 @@
|
||||
using System.Text;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Inserts the strict error-mode statement into a PowerShell script body.
|
||||
//
|
||||
// The statement cannot simply go first: using statements, [CmdletBinding()] and
|
||||
// param(...) must precede every other statement or the script stops parsing.
|
||||
// So the scanner walks the leading header (whitespace, # and <# #> comments
|
||||
// including #Requires and comment help, using lines, attributes, the param
|
||||
// block) and inserts on the SAME line as the first real statement, followed by
|
||||
// '; ', so every line number PowerShell reports still matches the stored body.
|
||||
// Anything it cannot read with confidence (named begin/process/end blocks,
|
||||
// subexpressions or here-strings inside the header, unbalanced brackets, bytes
|
||||
// that are not valid text) returns null and the caller skips injection.
|
||||
internal static class StrictModePreamble
|
||||
{
|
||||
private static readonly UTF8Encoding StrictUtf8 = new(false, true);
|
||||
private static readonly UTF8Encoding Utf8WithBom = new(true);
|
||||
|
||||
private static readonly string[] NamedBlocks =
|
||||
{ "begin", "process", "end", "dynamicparam", "clean" };
|
||||
|
||||
// Returns the rewritten body as UTF-8 WITH BOM (Windows PowerShell 5.1
|
||||
// reads a BOM-less file as the ANSI code page), or null with a reason.
|
||||
public static byte[]? TryApply(byte[] body, string statement, out string? reason)
|
||||
{
|
||||
if (!TryDecode(body, out var text))
|
||||
{
|
||||
reason = "script is not valid UTF-8 or UTF-16 text";
|
||||
return null;
|
||||
}
|
||||
|
||||
var index = FindInsertionPoint(text, out reason);
|
||||
if (index < 0)
|
||||
return null;
|
||||
|
||||
var rewritten = text.Insert(index, statement + "; ");
|
||||
var preamble = Utf8WithBom.GetPreamble();
|
||||
var encoded = Utf8WithBom.GetBytes(rewritten);
|
||||
var result = new byte[preamble.Length + encoded.Length];
|
||||
preamble.CopyTo(result, 0);
|
||||
encoded.CopyTo(result, preamble.Length);
|
||||
return result;
|
||||
}
|
||||
|
||||
private static bool TryDecode(byte[] body, out string text)
|
||||
{
|
||||
text = "";
|
||||
try
|
||||
{
|
||||
if (body.Length >= 3 && body[0] == 0xEF && body[1] == 0xBB && body[2] == 0xBF)
|
||||
text = StrictUtf8.GetString(body, 3, body.Length - 3);
|
||||
else if (body.Length >= 2 && body[0] == 0xFF && body[1] == 0xFE)
|
||||
text = new UnicodeEncoding(false, false, true).GetString(body, 2, body.Length - 2);
|
||||
else if (body.Length >= 2 && body[0] == 0xFE && body[1] == 0xFF)
|
||||
text = new UnicodeEncoding(true, false, true).GetString(body, 2, body.Length - 2);
|
||||
else
|
||||
text = StrictUtf8.GetString(body);
|
||||
return true;
|
||||
}
|
||||
catch (DecoderFallbackException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Index of the first statement after the script header, or -1.
|
||||
internal static int FindInsertionPoint(string text, out string? reason)
|
||||
{
|
||||
reason = null;
|
||||
var index = SkipTrivia(text, 0);
|
||||
while (index >= 0 && MatchWord(text, index, "using"))
|
||||
{
|
||||
var lineEnd = text.IndexOf('\n', index);
|
||||
index = SkipTrivia(text, lineEnd < 0 ? text.Length : lineEnd);
|
||||
}
|
||||
if (index < 0)
|
||||
return Uncertain("unterminated block comment in header", out reason);
|
||||
|
||||
var cursor = index;
|
||||
while (cursor < text.Length && text[cursor] == '[')
|
||||
{
|
||||
cursor = SkipBalanced(text, cursor);
|
||||
if (cursor < 0)
|
||||
return Uncertain("could not read attribute in header", out reason);
|
||||
cursor = SkipTrivia(text, cursor);
|
||||
if (cursor < 0)
|
||||
return Uncertain("unterminated block comment in header", out reason);
|
||||
}
|
||||
|
||||
if (MatchWord(text, cursor, "param"))
|
||||
{
|
||||
var open = SkipTrivia(text, cursor + "param".Length);
|
||||
if (open < 0 || open >= text.Length || text[open] != '(')
|
||||
return Uncertain("could not read param block", out reason);
|
||||
var close = SkipBalanced(text, open);
|
||||
if (close < 0)
|
||||
return Uncertain("could not read param block", out reason);
|
||||
index = SkipTrivia(text, close);
|
||||
if (index < 0)
|
||||
return Uncertain("unterminated block comment after param block", out reason);
|
||||
}
|
||||
// Brackets not followed by param are a type literal starting the first
|
||||
// statement, so index stays in front of them.
|
||||
|
||||
if (index >= text.Length)
|
||||
return Uncertain("no statements after the script header", out reason);
|
||||
|
||||
foreach (var block in NamedBlocks)
|
||||
{
|
||||
if (MatchWord(text, index, block))
|
||||
return Uncertain($"script uses a named '{block}' block", out reason);
|
||||
}
|
||||
|
||||
return index;
|
||||
}
|
||||
|
||||
private static int Uncertain(string why, out string? reason)
|
||||
{
|
||||
reason = why;
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Skip whitespace, line comments and block comments. -1 on an unterminated
|
||||
// block comment.
|
||||
private static int SkipTrivia(string text, int index)
|
||||
{
|
||||
while (index < text.Length)
|
||||
{
|
||||
var character = text[index];
|
||||
if (char.IsWhiteSpace(character) || character == '\uFEFF')
|
||||
{
|
||||
index++;
|
||||
}
|
||||
else if (character == '<' && index + 1 < text.Length && text[index + 1] == '#')
|
||||
{
|
||||
var end = text.IndexOf("#>", index + 2, StringComparison.Ordinal);
|
||||
if (end < 0)
|
||||
return -1;
|
||||
index = end + 2;
|
||||
}
|
||||
else if (character == '#')
|
||||
{
|
||||
var lineEnd = text.IndexOf('\n', index);
|
||||
index = lineEnd < 0 ? text.Length : lineEnd + 1;
|
||||
}
|
||||
else
|
||||
{
|
||||
break;
|
||||
}
|
||||
}
|
||||
return index;
|
||||
}
|
||||
|
||||
private static bool MatchWord(string text, int index, string word)
|
||||
{
|
||||
if (index < 0 || index + word.Length > text.Length)
|
||||
return false;
|
||||
if (string.Compare(text, index, word, 0, word.Length, StringComparison.OrdinalIgnoreCase) != 0)
|
||||
return false;
|
||||
var after = index + word.Length;
|
||||
if (after == text.Length)
|
||||
return true;
|
||||
var next = text[after];
|
||||
return !(char.IsLetterOrDigit(next) || next == '_' || next == '-');
|
||||
}
|
||||
|
||||
private static bool IsSingleQuote(char character)
|
||||
=> character is '\'' or '\u2018' or '\u2019' or '\u201A' or '\u201B';
|
||||
|
||||
private static bool IsDoubleQuote(char character)
|
||||
=> character is '"' or '\u201C' or '\u201D' or '\u201E';
|
||||
|
||||
// From an opening ( [ or { return the index just past its matching closer,
|
||||
// honoring quotes and comments. -1 when unsure.
|
||||
private static int SkipBalanced(string text, int index)
|
||||
{
|
||||
var closers = new Stack<char>();
|
||||
while (index < text.Length)
|
||||
{
|
||||
var character = text[index];
|
||||
switch (character)
|
||||
{
|
||||
case '(':
|
||||
closers.Push(')');
|
||||
index++;
|
||||
break;
|
||||
case '[':
|
||||
closers.Push(']');
|
||||
index++;
|
||||
break;
|
||||
case '{':
|
||||
closers.Push('}');
|
||||
index++;
|
||||
break;
|
||||
case ')':
|
||||
case ']':
|
||||
case '}':
|
||||
if (closers.Count == 0 || closers.Pop() != character)
|
||||
return -1;
|
||||
index++;
|
||||
if (closers.Count == 0)
|
||||
return index;
|
||||
break;
|
||||
case '`':
|
||||
index += 2;
|
||||
break;
|
||||
case '@' when index + 1 < text.Length
|
||||
&& (IsSingleQuote(text[index + 1]) || IsDoubleQuote(text[index + 1])):
|
||||
// Here-string; not worth reading in a header.
|
||||
return -1;
|
||||
case '<' when index + 1 < text.Length && text[index + 1] == '#':
|
||||
var end = text.IndexOf("#>", index + 2, StringComparison.Ordinal);
|
||||
if (end < 0)
|
||||
return -1;
|
||||
index = end + 2;
|
||||
break;
|
||||
case '#':
|
||||
if (index > 0 && (char.IsLetterOrDigit(text[index - 1]) || text[index - 1] == '_'))
|
||||
return -1;
|
||||
var lineEnd = text.IndexOf('\n', index);
|
||||
if (lineEnd < 0)
|
||||
return -1;
|
||||
index = lineEnd + 1;
|
||||
break;
|
||||
default:
|
||||
if (IsSingleQuote(character))
|
||||
{
|
||||
index = SkipSingleQuoted(text, index + 1);
|
||||
if (index < 0)
|
||||
return -1;
|
||||
}
|
||||
else if (IsDoubleQuote(character))
|
||||
{
|
||||
index = SkipDoubleQuoted(text, index + 1);
|
||||
if (index < 0)
|
||||
return -1;
|
||||
}
|
||||
else
|
||||
{
|
||||
index++;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
private static int SkipSingleQuoted(string text, int index)
|
||||
{
|
||||
while (index < text.Length)
|
||||
{
|
||||
if (IsSingleQuote(text[index]))
|
||||
{
|
||||
if (index + 1 < text.Length && IsSingleQuote(text[index + 1]))
|
||||
{
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
return index + 1;
|
||||
}
|
||||
index++;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
private static int SkipDoubleQuoted(string text, int index)
|
||||
{
|
||||
while (index < text.Length)
|
||||
{
|
||||
var character = text[index];
|
||||
if (character == '`')
|
||||
{
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
if (character == '$' && index + 1 < text.Length && text[index + 1] == '(')
|
||||
return -1;
|
||||
if (IsDoubleQuote(character))
|
||||
{
|
||||
if (index + 1 < text.Length && IsDoubleQuote(text[index + 1]))
|
||||
{
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
return index + 1;
|
||||
}
|
||||
index++;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
128
agent/ShopDbEndpointAgent.Core/TcpTableReader.cs
Normal file
@@ -0,0 +1,128 @@
|
||||
using System.Net;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// TCP connection state with owning process ids, via GetExtendedTcpTable.
|
||||
//
|
||||
// Why P/Invoke instead of shelling out to netstat.exe: parsing netstat text
|
||||
// output means spawning a process on every telemetry tick and depending on the
|
||||
// column layout of a localized command. GetExtendedTcpTable is the API netstat
|
||||
// itself calls, returns process ids directly, and costs a single allocation.
|
||||
//
|
||||
// Listeners are the cheap, bounded half of this (a handful of rows, stable over
|
||||
// time) and ride the normal cadence. The full connection table is unbounded and
|
||||
// churns constantly, so it stays on-demand.
|
||||
public static class TcpTableReader
|
||||
{
|
||||
private const int AF_INET = 2;
|
||||
private const uint ERROR_INSUFFICIENT_BUFFER = 122;
|
||||
private const int TCP_TABLE_OWNER_PID_ALL = 5;
|
||||
|
||||
public enum TcpState
|
||||
{
|
||||
Closed = 1,
|
||||
Listen = 2,
|
||||
SynSent = 3,
|
||||
SynReceived = 4,
|
||||
Established = 5,
|
||||
FinWait1 = 6,
|
||||
FinWait2 = 7,
|
||||
CloseWait = 8,
|
||||
Closing = 9,
|
||||
LastAck = 10,
|
||||
TimeWait = 11,
|
||||
DeleteTcb = 12,
|
||||
}
|
||||
|
||||
public readonly record struct TcpRow(
|
||||
string LocalAddress,
|
||||
int LocalPort,
|
||||
string RemoteAddress,
|
||||
int RemotePort,
|
||||
TcpState State,
|
||||
int ProcessId);
|
||||
|
||||
// Full IPv4 TCP table. Throws only on catastrophic interop failure; callers
|
||||
// treat an empty list as "not observed".
|
||||
public static List<TcpRow> GetTcpTable()
|
||||
{
|
||||
var rows = new List<TcpRow>();
|
||||
var buffer = IntPtr.Zero;
|
||||
var size = 0;
|
||||
|
||||
try
|
||||
{
|
||||
// First call sizes the buffer. It is expected to fail with
|
||||
// INSUFFICIENT_BUFFER; anything else is a real error.
|
||||
var result = GetExtendedTcpTable(
|
||||
IntPtr.Zero, ref size, false, AF_INET, TCP_TABLE_OWNER_PID_ALL, 0);
|
||||
if (result != ERROR_INSUFFICIENT_BUFFER && result != 0)
|
||||
return rows;
|
||||
|
||||
buffer = Marshal.AllocHGlobal(size);
|
||||
result = GetExtendedTcpTable(
|
||||
buffer, ref size, false, AF_INET, TCP_TABLE_OWNER_PID_ALL, 0);
|
||||
if (result != 0)
|
||||
return rows;
|
||||
|
||||
var entryCount = Marshal.ReadInt32(buffer);
|
||||
var rowSize = Marshal.SizeOf<MIB_TCPROW_OWNER_PID>();
|
||||
var cursor = IntPtr.Add(buffer, 4);
|
||||
|
||||
for (var index = 0; index < entryCount; index++)
|
||||
{
|
||||
var row = Marshal.PtrToStructure<MIB_TCPROW_OWNER_PID>(cursor);
|
||||
rows.Add(new TcpRow(
|
||||
FormatAddress(row.localAddress),
|
||||
FormatPort(row.localPort),
|
||||
FormatAddress(row.remoteAddress),
|
||||
FormatPort(row.remotePort),
|
||||
(TcpState)row.state,
|
||||
(int)row.owningProcessId));
|
||||
cursor = IntPtr.Add(cursor, rowSize);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (buffer != IntPtr.Zero)
|
||||
Marshal.FreeHGlobal(buffer);
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
public static List<TcpRow> GetListeners()
|
||||
=> GetTcpTable().FindAll(row => row.State == TcpState.Listen);
|
||||
|
||||
private static string FormatAddress(uint address)
|
||||
=> new IPAddress(BitConverter.GetBytes(address)).ToString();
|
||||
|
||||
// The port arrives as a big-endian value packed into the low two bytes of a
|
||||
// host-order uint, so the bytes must be swapped rather than simply cast.
|
||||
private static int FormatPort(uint port)
|
||||
{
|
||||
var bytes = BitConverter.GetBytes(port);
|
||||
return (bytes[0] << 8) | bytes[1];
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct MIB_TCPROW_OWNER_PID
|
||||
{
|
||||
public uint state;
|
||||
public uint localAddress;
|
||||
public uint localPort;
|
||||
public uint remoteAddress;
|
||||
public uint remotePort;
|
||||
public uint owningProcessId;
|
||||
}
|
||||
|
||||
[DllImport("iphlpapi.dll", SetLastError = true)]
|
||||
private static extern uint GetExtendedTcpTable(
|
||||
IntPtr tcpTable,
|
||||
ref int size,
|
||||
bool order,
|
||||
int addressFamily,
|
||||
int tableClass,
|
||||
int reserved);
|
||||
}
|
||||
@@ -15,6 +15,11 @@ namespace ShopDbEndpointAgent.Core;
|
||||
// cadence tick rather than hammering the server.
|
||||
public sealed class TelemetryWorker : BackgroundService
|
||||
{
|
||||
// The one agent utility whose output is forwarded to the server today. The
|
||||
// name is the .ps1 filename without its extension, which is how
|
||||
// PowerShellUtilityRunner keys its results.
|
||||
public const string DellUpdatesUtilityName = "shopdb-dellupdates";
|
||||
|
||||
private readonly ICollector collector;
|
||||
private readonly CollectorClient client;
|
||||
private readonly IAgentUtilityRunner? utilities;
|
||||
@@ -22,12 +27,38 @@ public sealed class TelemetryWorker : BackgroundService
|
||||
private readonly string agentVersion;
|
||||
private readonly ILogger<TelemetryWorker> logger;
|
||||
private readonly CollectSignal? collectSignal;
|
||||
private readonly UserContextSpool? userContextSpool;
|
||||
private readonly UserContextRefreshChannel? userContextRefresh;
|
||||
private readonly EnrollmentService? enrollment;
|
||||
private readonly Random jitter = new();
|
||||
|
||||
// Upper bound on holding first contact for the enrollment attempt.
|
||||
private static readonly TimeSpan EnrollmentWaitLimit = TimeSpan.FromSeconds(20);
|
||||
|
||||
// How long to wait for the tray to answer a refresh request.
|
||||
//
|
||||
// Bounded, and bounded SHORT. Telemetry is this agent's liveness signal: a
|
||||
// report that arrives with slightly stale mapped drives is a working
|
||||
// endpoint, while a report that never arrives because we blocked on an
|
||||
// absent tray is an endpoint that looks dead. The probe itself is a single
|
||||
// short PowerShell run, so a tray that is alive answers well inside this;
|
||||
// anything longer means nobody is home.
|
||||
private static readonly TimeSpan UserContextRefreshTimeout = TimeSpan.FromSeconds(12);
|
||||
|
||||
// Poll interval while waiting for the stamp to advance. There is no
|
||||
// completion signal coming back the other way - the tray's answer IS the
|
||||
// rewritten spool file - so this is a poll by design, kept coarse enough to
|
||||
// be free and fine enough not to add visible latency.
|
||||
private static readonly TimeSpan UserContextPollInterval = TimeSpan.FromMilliseconds(400);
|
||||
|
||||
public TelemetryWorker(ICollector collector, CollectorClient client,
|
||||
AgentConfig config, string agentVersion, ILogger<TelemetryWorker> logger,
|
||||
IAgentUtilityRunner? utilities = null, CollectSignal? collectSignal = null)
|
||||
IAgentUtilityRunner? utilities = null, CollectSignal? collectSignal = null,
|
||||
UserContextSpool? userContextSpool = null,
|
||||
UserContextRefreshChannel? userContextRefresh = null,
|
||||
EnrollmentService? enrollment = null)
|
||||
{
|
||||
this.enrollment = enrollment;
|
||||
this.collector = collector;
|
||||
this.client = client;
|
||||
this.config = config;
|
||||
@@ -35,6 +66,77 @@ public sealed class TelemetryWorker : BackgroundService
|
||||
this.logger = logger;
|
||||
this.utilities = utilities;
|
||||
this.collectSignal = collectSignal;
|
||||
this.userContextSpool = userContextSpool;
|
||||
this.userContextRefresh = userContextRefresh;
|
||||
}
|
||||
|
||||
// Request a tray-side re-probe and wait, briefly, for the spool to be
|
||||
// rewritten. Never throws and never fails the run.
|
||||
//
|
||||
// The service does NOT collect session data itself and must not start doing
|
||||
// so: mapped drives belong to the operator's logon token, and a LocalSystem
|
||||
// probe reads a different, near-empty drive table. See the tombstone in
|
||||
// Program.cs. This method is the entire service-side involvement - ask, then
|
||||
// wait for the answer to appear on disk.
|
||||
private async Task RefreshUserContextAsync(CancellationToken token)
|
||||
{
|
||||
if (userContextSpool is null || userContextRefresh is null
|
||||
|| !OperatingSystem.IsWindows())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var resultPath = userContextSpool.TryGetResultPath(out _);
|
||||
if (resultPath is null)
|
||||
return;
|
||||
|
||||
// Captured BEFORE the request so "advanced" is measured against what we
|
||||
// would otherwise have reported. A missing spool reads as null, and any
|
||||
// stamp at all then counts as progress - that is the first-probe case.
|
||||
var before = userContextSpool.TryReadCollectedAt(resultPath);
|
||||
|
||||
if (!userContextRefresh.RequestRefresh())
|
||||
{
|
||||
// Could not signal at all. Note that the inverse does NOT mean a tray
|
||||
// is listening: the channel opens-or-creates and does not hold the
|
||||
// handle, so signalling into an empty machine succeeds by creating
|
||||
// the event and finding nobody waiting on it. Detecting an absent
|
||||
// tray is the timeout's job below, not this return value's.
|
||||
logger.LogDebug(
|
||||
"Could not request a user-context refresh; reporting the existing spool.");
|
||||
return;
|
||||
}
|
||||
|
||||
var deadline = DateTime.UtcNow + UserContextRefreshTimeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(UserContextPollInterval, token);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var after = userContextSpool.TryReadCollectedAt(resultPath);
|
||||
if (after is not null && (before is null || after > before))
|
||||
{
|
||||
logger.LogDebug(
|
||||
"User-context spool refreshed by the tray (collectedat={Stamp}).",
|
||||
after);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Timed out. Proceed anyway - deliberately. Session data is one card on
|
||||
// one page; telemetry carries heartbeat, inventory, events and command
|
||||
// results, and holding all of that hostage to a tray that did not answer
|
||||
// in twelve seconds trades a small inaccuracy for a total outage.
|
||||
logger.LogDebug(
|
||||
"The tray did not refresh the user-context spool within {Timeout}; "
|
||||
+ "reporting with the existing session data.",
|
||||
UserContextRefreshTimeout);
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
@@ -43,6 +145,37 @@ public sealed class TelemetryWorker : BackgroundService
|
||||
"Endpoint agent {Version} -> {BaseUrl}, cadence {Cadence}s +/-{Jitter}s",
|
||||
agentVersion, config.BaseUrl, config.CadenceSeconds, config.JitterSeconds);
|
||||
|
||||
// Spread first contact across the fleet. JitterSeconds only spreads the
|
||||
// gap BETWEEN runs, so without this the whole fleet reports the instant
|
||||
// its service starts. See StartupJitter.
|
||||
var startupDelay = StartupJitter.Next(config.StartupJitterSeconds, logger);
|
||||
if (startupDelay > TimeSpan.Zero)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Delaying first telemetry report by {Delay}ms (startup jitter)",
|
||||
(int)startupDelay.TotalMilliseconds);
|
||||
try
|
||||
{
|
||||
await Task.Delay(startupDelay, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (enrollment is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
await enrollment.WaitForFirstAttemptAsync(EnrollmentWaitLimit, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
bool firstRun = true;
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
@@ -111,6 +244,15 @@ public sealed class TelemetryWorker : BackgroundService
|
||||
config.Monitors.Processes.Length, config.Monitors.Services.Length);
|
||||
}
|
||||
|
||||
// Ask the tray to re-probe session data and WAIT for it, before
|
||||
// collecting. Ordering is the whole point: the collector reads the spool
|
||||
// file, so a refresh that lands after Collect() is a refresh the server
|
||||
// does not see until the NEXT cycle. That was the bug - "Send Status"
|
||||
// and "Refresh Status" both re-sent a spool nobody had rewritten, so a
|
||||
// drive the operator had just unmapped kept reporting OK for up to half
|
||||
// an hour and no amount of clicking changed it.
|
||||
await RefreshUserContextAsync(token);
|
||||
|
||||
var report = collector.Collect(agentVersion, adopt);
|
||||
|
||||
if (utilities is not null)
|
||||
@@ -118,6 +260,17 @@ public sealed class TelemetryWorker : BackgroundService
|
||||
var extra = utilities.RunAll(token);
|
||||
if (extra.Count > 0)
|
||||
logger.LogDebug("Loaded {Count} agent utility result(s)", extra.Count);
|
||||
|
||||
// NARROW ON PURPOSE. The runner executes every .ps1 in the utilities
|
||||
// directory, but only this one known key is attached to the report.
|
||||
// Forwarding the whole dictionary would mean any script dropped into
|
||||
// that directory starts uploading its output to the server on every
|
||||
// cadence - see the signing caveat in PowerShellUtilityRunner's
|
||||
// header. Widening this to the full map is the intended end state of
|
||||
// the IAgentUtilityRunner seam, but it needs script signing or an
|
||||
// allowlist first. Until then, add a key here per utility.
|
||||
if (extra.TryGetValue(DellUpdatesUtilityName, out var dellUpdates))
|
||||
report.DellUpdates = dellUpdates;
|
||||
}
|
||||
|
||||
logger.LogInformation(
|
||||
|
||||
362
agent/ShopDbEndpointAgent.Core/UserContextProbe.cs
Normal file
@@ -0,0 +1,362 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Collects the session-scoped data that does not exist in the agent's own
|
||||
// context, by running a short PowerShell probe IN THE CALLING PROCESS'S
|
||||
// SESSION and writing the result to the user's spool file.
|
||||
//
|
||||
// Collects only data that genuinely does not exist in the agent's own context:
|
||||
// netuse - mapped drives are per logon session; "net use" from
|
||||
// LocalSystem returns SYSTEM's (empty) mapping table, never the
|
||||
// operator's Z: to the file server.
|
||||
// userlisteners- TCP listeners owned by the signed-in user's processes, which
|
||||
// is what identifies an operator-launched DNC or CMM listener.
|
||||
// idleseconds - GetLastInputInfo is per session, so session 0 cannot answer
|
||||
// it. Sampled at probe time only, so treat it as a coarse
|
||||
// "was anyone at the machine" signal, not live presence.
|
||||
//
|
||||
// This runs from `shopdbagent.exe --collect-usercontext`, which the tray - and
|
||||
// only the tray - invokes, because the tray is already running as the signed-in
|
||||
// operator. The service does NOT call this: it previously tried to descend into
|
||||
// the session with CreateProcessAsUser, which is the direction that has to
|
||||
// duplicate a token, is what endpoint protection watches for, and produced
|
||||
// nothing on a live endpoint. Nothing here crosses a privilege boundary now;
|
||||
// the probe simply runs as whoever the tray is.
|
||||
//
|
||||
// The script is embedded rather than shipped as a .ps1 so the probe cannot drift
|
||||
// from the agent binary or be tampered with on disk between cycles.
|
||||
public sealed class UserContextProbe
|
||||
{
|
||||
// The spool directory is injected by BuildProbeScript rather than derived in
|
||||
// the script, so the writer cannot disagree with the reader. See
|
||||
// UserContextSpool.PinnedSpoolDirectory for why $env:USERPROFILE was wrong.
|
||||
private const string SpoolDirectoryToken = "__SHOPDB_SPOOL_DIR__";
|
||||
|
||||
private const string ProbeScriptTemplate = """
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$result = [ordered]@{}
|
||||
# Injected by the agent from UserContextSpool.PinnedSpoolDirectory() so the
|
||||
# writer and the reader resolve the same directory. Deliberately NOT
|
||||
# $env:USERPROFILE: the tray can inherit another account's environment on a
|
||||
# locked-down image, which silently wrote this file into the wrong profile.
|
||||
$spool = '__SHOPDB_SPOOL_DIR__'
|
||||
$target = Join-Path $spool 'usercontext.json'
|
||||
$temporary = "$target.tmp"
|
||||
|
||||
try {
|
||||
New-Item -ItemType Directory -Path $spool -Force | Out-Null
|
||||
|
||||
$netuse = @()
|
||||
try {
|
||||
Get-CimInstance -ClassName Win32_NetworkConnection -ErrorAction Stop |
|
||||
ForEach-Object {
|
||||
$netuse += [ordered]@{
|
||||
localname = $_.LocalName
|
||||
remotename = $_.RemoteName
|
||||
status = $_.Status
|
||||
username = $_.UserName
|
||||
persistent = [bool]$_.Persistent
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
$result['netuseerror'] = $_.Exception.Message
|
||||
}
|
||||
$result['netuse'] = $netuse
|
||||
|
||||
# Second, independent view of the same drives. Win32_NetworkConnection
|
||||
# is the "net use" table and Get-SmbMapping is the SMB client's own,
|
||||
# and they disagree: a share can be Unavailable in one and absent or
|
||||
# OK in the other. Neither is authoritative, so both are reported as
|
||||
# collected and the server decides what to show. Normalising them
|
||||
# here would discard exactly the disagreement that is diagnostic.
|
||||
$smbmapping = @()
|
||||
try {
|
||||
Get-SmbMapping -ErrorAction Stop |
|
||||
ForEach-Object {
|
||||
$smbmapping += [ordered]@{
|
||||
localpath = $_.LocalPath
|
||||
remotepath = $_.RemotePath
|
||||
status = [string]$_.Status
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
# Absent on older builds and in some session types. A missing
|
||||
# source is not a drive fault, so it is recorded and left out
|
||||
# rather than reported as a failed mapping.
|
||||
$result['smbmappingerror'] = $_.Exception.Message
|
||||
}
|
||||
$result['smbmapping'] = $smbmapping
|
||||
|
||||
$listeners = @()
|
||||
try {
|
||||
$ownprocesses = @{}
|
||||
Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
$ownprocesses[$_.Id] = $_.ProcessName
|
||||
}
|
||||
Get-NetTCPConnection -State Listen -ErrorAction Stop |
|
||||
Where-Object { $ownprocesses.ContainsKey($_.OwningProcess) } |
|
||||
ForEach-Object {
|
||||
$listeners += [ordered]@{
|
||||
localaddress = $_.LocalAddress
|
||||
localport = $_.LocalPort
|
||||
processid = $_.OwningProcess
|
||||
processname = $ownprocesses[$_.OwningProcess]
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
$result['userlistenerserror'] = $_.Exception.Message
|
||||
}
|
||||
$result['userlisteners'] = $listeners
|
||||
|
||||
# Idle time must be measured inside the user's session:
|
||||
# GetLastInputInfo is per session, so asking from the agent's
|
||||
# session 0 would report the service's own idle time, which is
|
||||
# meaningless. This is what distinguishes "powered on" from
|
||||
# "someone is actually working at this machine".
|
||||
try {
|
||||
# Built as a single-quoted string rather than a here-string: a
|
||||
# here-string terminator must sit at column 0, which cannot be
|
||||
# expressed inside an indented C# raw string literal.
|
||||
$signature = 'using System; using System.Runtime.InteropServices; ' +
|
||||
'public class ShopDbIdle { ' +
|
||||
'[StructLayout(LayoutKind.Sequential)] ' +
|
||||
'struct LASTINPUTINFO { public uint cbSize; public uint dwTime; } ' +
|
||||
'[DllImport("user32.dll")] ' +
|
||||
'static extern bool GetLastInputInfo(ref LASTINPUTINFO info); ' +
|
||||
'public static long IdleSeconds() { ' +
|
||||
'LASTINPUTINFO info = new LASTINPUTINFO(); ' +
|
||||
'info.cbSize = (uint)Marshal.SizeOf(info); ' +
|
||||
'if (!GetLastInputInfo(ref info)) return -1; ' +
|
||||
'return (long)((uint)Environment.TickCount - info.dwTime) / 1000; } }'
|
||||
Add-Type -TypeDefinition $signature -ErrorAction Stop
|
||||
$idle = [ShopDbIdle]::IdleSeconds()
|
||||
if ($idle -ge 0) { $result['idleseconds'] = $idle }
|
||||
} catch {
|
||||
$result['idlesecondserror'] = $_.Exception.Message
|
||||
}
|
||||
|
||||
$result['collectedat'] = (Get-Date).ToUniversalTime().ToString('o')
|
||||
$result['username'] = $env:USERNAME
|
||||
|
||||
# Stamp the token's elevation state. The username alone does not
|
||||
# settle it: the SAME account elevated and unelevated gets two
|
||||
# different logon tokens and therefore two different drive tables,
|
||||
# so a spool from an elevated ShopFloor session passes the account
|
||||
# check while still reporting every drive as broken.
|
||||
try {
|
||||
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
|
||||
$result['elevated'] = [bool] $principal.IsInRole(
|
||||
[Security.Principal.WindowsBuiltInRole]::Administrator)
|
||||
} catch {
|
||||
$result['elevatederror'] = $_.Exception.Message
|
||||
}
|
||||
|
||||
# Write to a temporary name and rename into place, so the reader can
|
||||
# never observe a half-written file and discard a good result.
|
||||
$result | ConvertTo-Json -Depth 4 -Compress |
|
||||
Set-Content -Path $temporary -Encoding UTF8 -NoNewline
|
||||
Move-Item -Path $temporary -Destination $target -Force
|
||||
} catch {
|
||||
exit 1
|
||||
}
|
||||
exit 0
|
||||
""";
|
||||
|
||||
// Substitute the resolved spool directory into the template. The path goes in
|
||||
// as a PowerShell single-quoted literal, so an embedded quote is doubled and
|
||||
// nothing else in the path is interpreted - backslashes and $ are literal
|
||||
// inside single quotes, which is exactly what a Windows path needs.
|
||||
private static string BuildProbeScript(string spoolDirectory)
|
||||
=> ProbeScriptTemplate.Replace(
|
||||
SpoolDirectoryToken, spoolDirectory.Replace("'", "''"));
|
||||
|
||||
// How long the probe is allowed to take. Generous: this no longer runs on
|
||||
// the telemetry path, so an unresponsive WMI call delays nothing but itself.
|
||||
private static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(60);
|
||||
|
||||
// powershell.exe's exit code when it rejects its command line and prints
|
||||
// usage instead of running anything. Named so the failure is recognisable.
|
||||
private const int PowerShellStartupFailure = -196608;
|
||||
|
||||
private readonly ILogger<UserContextProbe> logger;
|
||||
|
||||
public UserContextProbe(ILogger<UserContextProbe> logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
// Run the probe in THIS process's session and write the spool file.
|
||||
// Returns true only when the spool file is present on disk afterwards.
|
||||
//
|
||||
// Called from the --collect-usercontext entry point, which the tray runs as
|
||||
// the signed-in operator. Never throws: this is background collection and a
|
||||
// failure must not take the tray or the agent process down with it.
|
||||
public bool Collect()
|
||||
{
|
||||
var spoolDirectory = UserContextSpool.PinnedSpoolDirectory();
|
||||
var resultPath = UserContextSpool.PinnedResultPath();
|
||||
|
||||
try
|
||||
{
|
||||
// Logged before the launch, not after. When this goes wrong the
|
||||
// interesting question is always "which directory did it target",
|
||||
// and a failure later in this method must not take the answer with
|
||||
// it.
|
||||
logger.LogInformation(
|
||||
"Running the user-context probe as {User}; spool target {Path}",
|
||||
Environment.UserName, resultPath);
|
||||
|
||||
// The probe writes a .tmp beside the target and moves it into place,
|
||||
// so the directory has to exist before powershell.exe starts. The
|
||||
// script creates it too, but doing it here means a permission problem
|
||||
// is reported by the agent with a real exception rather than showing
|
||||
// up as an opaque non-zero exit code.
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(spoolDirectory);
|
||||
}
|
||||
catch (Exception error) when (error is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
logger.LogWarning(error,
|
||||
"Could not create the user-context spool directory at {Directory}",
|
||||
spoolDirectory);
|
||||
return false;
|
||||
}
|
||||
// Passed as -EncodedCommand rather than staged as a .ps1 and run with
|
||||
// -File. The script used to be written into %TEMP% and executed from
|
||||
// there, which put a dependency on that file still existing, and still
|
||||
// being readable, at the moment powershell.exe started. On a hardened
|
||||
// endpoint that window is not safe: endpoint protection quarantines a
|
||||
// freshly written script in the temp directory between the write and
|
||||
// the launch, and roaming/redirected profiles can make the path
|
||||
// unusable outright.
|
||||
//
|
||||
// When powershell.exe cannot use its -File argument it prints its usage
|
||||
// banner and exits -196608 (0xFFFD0000) WITHOUT running a single
|
||||
// statement. That is a startup failure, not a script failure, so the
|
||||
// script's own try/catch never runs, nothing is logged about drives,
|
||||
// and the operator sees a silent "no mapped drives reported" while the
|
||||
// log only shows the bare exit code. -EncodedCommand carries the script
|
||||
// in the command line itself, so there is no file to lose and no
|
||||
// execution-policy or quarantine step between us and the probe.
|
||||
var encoded = Convert.ToBase64String(
|
||||
System.Text.Encoding.Unicode.GetBytes(BuildProbeScript(spoolDirectory)));
|
||||
|
||||
var startInfo = new System.Diagnostics.ProcessStartInfo
|
||||
{
|
||||
FileName = "powershell.exe",
|
||||
Arguments =
|
||||
"-NoProfile -NonInteractive -WindowStyle Hidden -EncodedCommand "
|
||||
+ encoded,
|
||||
WorkingDirectory = Path.GetTempPath(),
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
};
|
||||
|
||||
using var process = System.Diagnostics.Process.Start(startInfo);
|
||||
if (process is null)
|
||||
{
|
||||
logger.LogWarning("The user-context probe could not be started.");
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!process.WaitForExit((int)ProbeTimeout.TotalMilliseconds))
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context probe did not finish within {Seconds} seconds.",
|
||||
ProbeTimeout.TotalSeconds);
|
||||
TryKill(process);
|
||||
return false;
|
||||
}
|
||||
|
||||
if (process.ExitCode != 0)
|
||||
{
|
||||
// 0xFFFD0000 is powershell.exe refusing its own command line and
|
||||
// printing usage: the probe never ran. Called out by name because
|
||||
// as a bare number it reads like a script error and sends the
|
||||
// reader looking for a WMI or drive fault that is not there.
|
||||
var detail = process.ExitCode == PowerShellStartupFailure
|
||||
? " (powershell.exe rejected its command line; the probe did not run)"
|
||||
: string.Empty;
|
||||
logger.LogWarning(
|
||||
"The user-context probe exited with code {ExitCode}.{Detail}",
|
||||
process.ExitCode, detail);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Exit code 0 is necessary but NOT sufficient. The probe used to be
|
||||
// trusted on its exit code alone, which meant a run that wrote its
|
||||
// file into a different profile entirely still reported success - the
|
||||
// log claimed a spool had been written for weeks while nothing was
|
||||
// ever there. Confirm the file exists where the reader will look.
|
||||
if (!File.Exists(resultPath))
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context probe reported success but no spool file is "
|
||||
+ "present at {Path}.", resultPath);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Byte count alone does not distinguish a healthy spool from a
|
||||
// well-formed empty one, and "0 drives" is the exact symptom this
|
||||
// whole path exists to diagnose. Summarising the contents means a
|
||||
// single log line from any given task run answers whether the probe
|
||||
// saw the operator's drives, without needing the spool itself.
|
||||
var summary = SummariseSpool(resultPath);
|
||||
logger.LogInformation(
|
||||
"The user-context probe wrote {Bytes} bytes to {Path} as {User}: {Summary}",
|
||||
new FileInfo(resultPath).Length, resultPath, Environment.UserName, summary);
|
||||
return true;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception, "The user-context probe failed.");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// One-line description of what the probe actually captured, for the run log.
|
||||
// Best-effort by design: this is diagnostic colour on an already-successful
|
||||
// run, so a parse problem here must never turn that run into a failure.
|
||||
private static string SummariseSpool(string resultPath)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var document = System.Text.Json.JsonDocument.Parse(
|
||||
File.ReadAllText(resultPath));
|
||||
var root = document.RootElement;
|
||||
|
||||
static int CountOf(System.Text.Json.JsonElement root, string name) =>
|
||||
root.TryGetProperty(name, out var value)
|
||||
&& value.ValueKind == System.Text.Json.JsonValueKind.Array
|
||||
? value.GetArrayLength()
|
||||
: 0;
|
||||
|
||||
var user = root.TryGetProperty("username", out var stamped)
|
||||
? stamped.GetString()
|
||||
: "(unstamped)";
|
||||
|
||||
return $"user={user} netuse={CountOf(root, "netuse")} "
|
||||
+ $"smbmapping={CountOf(root, "smbmapping")} "
|
||||
+ $"listeners={CountOf(root, "userlisteners")}";
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
return "(could not summarise)";
|
||||
}
|
||||
}
|
||||
|
||||
private void TryKill(System.Diagnostics.Process process)
|
||||
{
|
||||
try
|
||||
{
|
||||
process.Kill(entireProcessTree: true);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogDebug(exception, "Could not stop the timed-out user-context probe.");
|
||||
}
|
||||
}
|
||||
}
|
||||
179
agent/ShopDbEndpointAgent.Core/UserContextRefreshChannel.cs
Normal file
@@ -0,0 +1,179 @@
|
||||
using System.Runtime.Versioning;
|
||||
using System.Security.AccessControl;
|
||||
using System.Security.Principal;
|
||||
using System.Threading;
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// The service-to-tray request channel for an on-demand user-context refresh.
|
||||
//
|
||||
// WHY THIS EXISTS AT ALL. Mapped drives are a property of the operator's logon
|
||||
// token, not of the machine. The agent is a session-0 LocalSystem service, so it
|
||||
// CANNOT enumerate them - not by impersonation, not by CreateProcessAsUser, not
|
||||
// by starting the probe task itself. That approach was built once and reverted;
|
||||
// see the tombstone in Program.cs where UserContextProbe is deliberately kept
|
||||
// out of the service's DI container. The only process on the box that can see
|
||||
// the right drive table is one already running inside the operator's session:
|
||||
// the tray.
|
||||
//
|
||||
// So the service does not probe. It ASKS. This channel is the ask, and the
|
||||
// spool file remains the answer.
|
||||
//
|
||||
// WHY A NAMED EVENT rather than a pipe or a socket. The request carries no data
|
||||
// - it is a single bit, "please re-probe now" - and the reply already has a
|
||||
// transport (the spool file, whose collectedat stamp the service watches
|
||||
// advance). A kernel event is the smallest thing that crosses the session-0
|
||||
// boundary, needs no serialization, no port, no firewall rule, and no cleanup if
|
||||
// either side dies. A file-watcher trigger would also work but leaves litter on
|
||||
// disk when the service is killed mid-request.
|
||||
//
|
||||
// THE Global\ PREFIX IS LOAD-BEARING. Without it the name is scoped to the
|
||||
// caller's session, so the service (session 0) and the tray (session 1+) would
|
||||
// create two unrelated events with the same name and never see each other. That
|
||||
// failure is silent: both sides "work", nothing is ever refreshed.
|
||||
//
|
||||
// THE DACL IS ALSO LOAD-BEARING. The service creates the event as LocalSystem,
|
||||
// which by default produces an object a non-elevated user cannot even open. The
|
||||
// tray runs unelevated as ShopFloor by design (an ELEVATED tray is refused by
|
||||
// the probe, because it would read the wrong drive table). So the creator grants
|
||||
// Users explicit Synchronize + Modify rights, or the tray gets access-denied on
|
||||
// open and the refresh silently never happens.
|
||||
public sealed class UserContextRefreshChannel
|
||||
{
|
||||
// Session-global so session 0 and the interactive session share one object.
|
||||
public const string EventName = @"Global\ShopDbEndpointAgent.UserContextRefresh";
|
||||
|
||||
private readonly ILogger logger;
|
||||
|
||||
// Takes the base ILogger rather than ILogger<T>: both the service (generic
|
||||
// host) and the tray (its own file-backed logger named "Tray") construct
|
||||
// this, and the tray has no DI container to resolve a typed logger from.
|
||||
public UserContextRefreshChannel(ILogger logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
// Signal the tray that a fresh probe is wanted.
|
||||
//
|
||||
// Returns false when the event could not be opened or created, which is the
|
||||
// normal state on a machine with nobody signed in and no tray running. The
|
||||
// caller treats that as "no refresh available" and reports what it already
|
||||
// has - a stale mapped-drive list is worth strictly more than a missed
|
||||
// heartbeat, because telemetry IS the liveness signal.
|
||||
//
|
||||
// AutoReset, so exactly one waiting tray is released per request and the
|
||||
// event does not latch on for every subsequent listener iteration.
|
||||
[SupportedOSPlatform("windows")]
|
||||
public bool RequestRefresh()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var handle = OpenOrCreate(out var created);
|
||||
if (handle is null)
|
||||
return false;
|
||||
|
||||
handle.Set();
|
||||
logger.LogDebug(
|
||||
"Requested a user-context refresh on {Event} (created={Created})",
|
||||
EventName, created);
|
||||
return true;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
// Never throws to the caller. This sits directly on the telemetry
|
||||
// path, and a refresh is an enhancement to the report, not a
|
||||
// precondition for sending it.
|
||||
logger.LogDebug(error,
|
||||
"Could not request a user-context refresh on {Event}", EventName);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Tray side: block until the service asks for a refresh, the timeout expires,
|
||||
// or the token is cancelled. Returns true only for an actual request.
|
||||
[SupportedOSPlatform("windows")]
|
||||
public bool WaitForRequest(TimeSpan timeout, CancellationToken token)
|
||||
{
|
||||
try
|
||||
{
|
||||
using var handle = OpenOrCreate(out _);
|
||||
if (handle is null)
|
||||
return false;
|
||||
|
||||
var index = WaitHandle.WaitAny(
|
||||
new[] { handle, token.WaitHandle }, timeout);
|
||||
|
||||
// Index 1 is cancellation, WaitTimeout is the idle path; only index 0
|
||||
// is a genuine request.
|
||||
return index == 0;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error,
|
||||
"Could not wait on the user-context refresh channel {Event}", EventName);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Open the shared event, creating it if this process is first.
|
||||
//
|
||||
// Either side may start first - the service can boot before anyone signs in,
|
||||
// and the tray can start before the service finishes initializing - so both
|
||||
// sides use the same create-or-open call rather than designating an owner.
|
||||
// Whoever wins applies the DACL; the loser opens the existing object.
|
||||
[SupportedOSPlatform("windows")]
|
||||
private EventWaitHandle? OpenOrCreate(out bool created)
|
||||
{
|
||||
created = false;
|
||||
try
|
||||
{
|
||||
var security = new EventWaitHandleSecurity();
|
||||
|
||||
// BUILTIN\Users by SID, not by name: on a non-English Windows the
|
||||
// group is not called "Users", and this agent ships to whatever
|
||||
// locale the site runs. Same reasoning as the task registration.
|
||||
var users = new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null);
|
||||
security.AddAccessRule(new EventWaitHandleAccessRule(
|
||||
users,
|
||||
EventWaitHandleRights.Synchronize | EventWaitHandleRights.Modify,
|
||||
AccessControlType.Allow));
|
||||
|
||||
// FullControl for SYSTEM and Administrators so the service retains
|
||||
// ownership semantics regardless of which side created the object.
|
||||
var system = new SecurityIdentifier(WellKnownSidType.LocalSystemSid, null);
|
||||
security.AddAccessRule(new EventWaitHandleAccessRule(
|
||||
system, EventWaitHandleRights.FullControl, AccessControlType.Allow));
|
||||
|
||||
var admins = new SecurityIdentifier(
|
||||
WellKnownSidType.BuiltinAdministratorsSid, null);
|
||||
security.AddAccessRule(new EventWaitHandleAccessRule(
|
||||
admins, EventWaitHandleRights.FullControl, AccessControlType.Allow));
|
||||
|
||||
return EventWaitHandleAcl.Create(
|
||||
initialState: false,
|
||||
mode: EventResetMode.AutoReset,
|
||||
name: EventName,
|
||||
createdNew: out created,
|
||||
eventSecurity: security);
|
||||
}
|
||||
catch (UnauthorizedAccessException)
|
||||
{
|
||||
// The object exists with a DACL that does not admit this caller.
|
||||
// Retry read-side-only rights: the tray only ever needs to wait, and
|
||||
// an event created by an older agent build may not grant Modify.
|
||||
try
|
||||
{
|
||||
return EventWaitHandle.OpenExisting(EventName);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
logger.LogDebug(error,
|
||||
"The user-context refresh event {Event} exists but is not accessible",
|
||||
EventName);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
389
agent/ShopDbEndpointAgent.Core/UserContextSpool.cs
Normal file
@@ -0,0 +1,389 @@
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ShopDbEndpointAgent.Core;
|
||||
|
||||
// Reads the result file that the tray-run user-context probe writes into the
|
||||
// signed-in user's ShopDB spool folder.
|
||||
//
|
||||
// Why a file and not a return value: the agent is a session-0 LocalSystem
|
||||
// service, so anything session-scoped (mapped drives from "net use", per-user
|
||||
// network state) simply does not exist in its own context. The probe runs as the
|
||||
// operator under the tray, in a completely separate process with no pipe back to
|
||||
// the service. A file in the user's own profile is the handoff, and the user can
|
||||
// always write there without elevation.
|
||||
//
|
||||
// Freshness is judged from the 'collectedat' stamp inside the file, NOT by
|
||||
// deleting the file around each read. Writer and reader are now independent
|
||||
// processes on unrelated schedules, so a delete-then-read would race the tray
|
||||
// and usually destroy a perfectly good result before it was ever reported. The
|
||||
// file is therefore left alone and simply ignored once it ages out, which also
|
||||
// means a logged-off machine stops reporting session data instead of pinning the
|
||||
// last operator's mapped drives forever.
|
||||
public sealed class UserContextSpool
|
||||
{
|
||||
public const string SpoolFolderName = "ShopDB";
|
||||
public const string ResultFileName = "usercontext.json";
|
||||
|
||||
// The one definition of where the spool lives. Both the probe (writer) and
|
||||
// this class (reader) call it, so the two can no longer disagree.
|
||||
//
|
||||
// The writer used to build its own path from $env:USERPROFILE while the
|
||||
// reader used the pinned ShopFloor profile. On a locked-down image those are
|
||||
// not the same directory - the tray is launched through explorer.exe by an
|
||||
// elevated installer and can inherit another account's environment block, as
|
||||
// the tray already documents for %TEMP%. The probe then wrote a perfectly
|
||||
// good file into the wrong profile, exited 0, and the service found nothing:
|
||||
// no spool ever appeared and no mapped drive was ever reported.
|
||||
public static string PinnedSpoolDirectory()
|
||||
=> Path.Combine(FileLoggerProvider.ShopFloorProfileDirectory(), SpoolFolderName);
|
||||
|
||||
public static string PinnedResultPath()
|
||||
=> Path.Combine(PinnedSpoolDirectory(), ResultFileName);
|
||||
|
||||
// A probe result is small - a handful of mapped drives and listeners. A file
|
||||
// larger than this is either corrupt or hostile, and we will not parse it.
|
||||
private const long MaxResultBytes = 512 * 1024;
|
||||
|
||||
// A result older than this is treated as absent. Sized at roughly three tray
|
||||
// probe intervals (30 min each), so a single missed run never blanks the card
|
||||
// but genuinely stale session data retires quickly.
|
||||
//
|
||||
// This was 24h when the reader deleted the file after each read; the delete
|
||||
// was doing the real retiring and the age check was only a backstop. Now that
|
||||
// the spool persists, this value IS the staleness bound, so it is sized
|
||||
// against the writer's cadence rather than the server's quiet-report window.
|
||||
//
|
||||
// Not read from the server: this is an offline-safe local bound, and the
|
||||
// agent must retire stale session data even when it cannot reach ShopDB.
|
||||
private static readonly TimeSpan MaxResultAge = TimeSpan.FromMinutes(90);
|
||||
|
||||
private readonly ILogger<UserContextSpool> logger;
|
||||
|
||||
public UserContextSpool(ILogger<UserContextSpool> logger)
|
||||
{
|
||||
this.logger = logger;
|
||||
}
|
||||
|
||||
// Full path to the spool file. Fixed at C:\Users\ShopFloor\ShopDB, because
|
||||
// every shop-floor endpoint signs in as the same local ShopFloor account.
|
||||
//
|
||||
// This used to be resolved through WTSQueryUserToken/GetUserProfileDirectory,
|
||||
// and that lookup is precisely what fails on a hardened endpoint: querying
|
||||
// another session's token is a privileged operation that endpoint protection
|
||||
// blocks. When it failed the service simply reported "no session data" while
|
||||
// the tray was writing a perfectly good file, so writer and reader silently
|
||||
// disagreed about the location and the operator saw an empty card.
|
||||
//
|
||||
// The token lookup is kept ONLY as a fallback for a machine whose operator
|
||||
// account is named something else. If both are unavailable there is no
|
||||
// session data, which costs mapped drives and idle time but never affects
|
||||
// telemetry, commands or inventory.
|
||||
public string? TryGetResultPath(out string? reason)
|
||||
{
|
||||
var pinned = FileLoggerProvider.ShopFloorProfileDirectory();
|
||||
if (Directory.Exists(pinned))
|
||||
{
|
||||
reason = null;
|
||||
return PinnedResultPath();
|
||||
}
|
||||
|
||||
var profileDirectory =
|
||||
InteractiveProcessLauncher.TryGetActiveUserProfileDirectory(out var error);
|
||||
if (string.IsNullOrWhiteSpace(profileDirectory))
|
||||
{
|
||||
reason = $"No {FileLoggerProvider.ShopFloorUserName} profile at {pinned}, "
|
||||
+ "and the interactive profile could not be resolved either: "
|
||||
+ (error ?? "unknown error");
|
||||
return null;
|
||||
}
|
||||
|
||||
reason = null;
|
||||
return Path.Combine(profileDirectory, SpoolFolderName, ResultFileName);
|
||||
}
|
||||
|
||||
// Remove a result file. Only used to discard one that is corrupt or has aged
|
||||
// out; a valid result is left in place for the next read.
|
||||
public void ClearStaleResult(string resultPath)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (File.Exists(resultPath))
|
||||
{
|
||||
File.Delete(resultPath);
|
||||
logger.LogDebug(
|
||||
"Removed a stale user-context spool file at {Path}", resultPath);
|
||||
}
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
// Not fatal on its own, but it does mean we cannot trust whatever we
|
||||
// read next, so the caller must know.
|
||||
logger.LogWarning(exception,
|
||||
"Could not remove the stale user-context spool file at {Path}", resultPath);
|
||||
}
|
||||
}
|
||||
|
||||
public void EnsureSpoolDirectory(string resultPath)
|
||||
{
|
||||
var directory = Path.GetDirectoryName(resultPath);
|
||||
if (string.IsNullOrWhiteSpace(directory))
|
||||
return;
|
||||
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(directory);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception,
|
||||
"Could not create the user-context spool directory at {Directory}", directory);
|
||||
}
|
||||
}
|
||||
|
||||
// Read the probe result without consuming it. Returns null when there is no
|
||||
// usable, sufficiently recent result, which the caller reports as "no session
|
||||
// data this cycle" rather than as empty data - absent and empty mean
|
||||
// different things here.
|
||||
public Dictionary<string, object?>? TryReadResult(string resultPath)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!File.Exists(resultPath))
|
||||
return null;
|
||||
|
||||
var info = new FileInfo(resultPath);
|
||||
if (info.Length == 0)
|
||||
{
|
||||
logger.LogDebug("The user-context spool file was empty.");
|
||||
return null;
|
||||
}
|
||||
|
||||
if (info.Length > MaxResultBytes)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context spool file was {Bytes} bytes, above the {Limit} byte "
|
||||
+ "limit; discarding it.",
|
||||
info.Length, MaxResultBytes);
|
||||
return null;
|
||||
}
|
||||
|
||||
var json = File.ReadAllText(resultPath);
|
||||
if (string.IsNullOrWhiteSpace(json))
|
||||
return null;
|
||||
|
||||
var parsed = JsonSerializer.Deserialize<Dictionary<string, object?>>(json);
|
||||
if (parsed is null || parsed.Count == 0)
|
||||
return null;
|
||||
|
||||
if (IsExpired(parsed, resultPath))
|
||||
return null;
|
||||
|
||||
if (IsWrongAccount(parsed))
|
||||
return null;
|
||||
|
||||
if (IsElevatedCollection(parsed))
|
||||
return null;
|
||||
|
||||
return parsed;
|
||||
}
|
||||
catch (JsonException exception)
|
||||
{
|
||||
// A truncated file is the expected shape of this failure: the probe
|
||||
// was still writing when we looked. Next cycle gets a clean one.
|
||||
logger.LogWarning(exception,
|
||||
"The user-context spool file was not valid JSON; discarding it.");
|
||||
return null;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception,
|
||||
"Could not read the user-context spool file at {Path}", resultPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Read the spool WITHOUT removing it. The probe overwrites the file on every
|
||||
// run (write .tmp, move into place), so the newest result always replaces the
|
||||
// previous one and there is nothing for the reader to clean up.
|
||||
//
|
||||
// This used to delete the file after a successful read, to stop a dead tray
|
||||
// re-reporting the same drives forever. The collectedat age check already
|
||||
// does that job, and deleting created a worse problem: lastUserContext in the
|
||||
// collector is in-memory only, so a service restart wiped the cache while the
|
||||
// spool had already been consumed. The agent then reported NO mapped drives
|
||||
// until the tray's next probe - and the installer restarts the service and
|
||||
// the tray on every install, so a fresh install was guaranteed to show an
|
||||
// empty card at exactly the moment someone looked at it.
|
||||
//
|
||||
// The contents are the operator's own session data, visible to them in their
|
||||
// own context, so leaving the file in place costs nothing.
|
||||
public Dictionary<string, object?>? ConsumeResult(string resultPath)
|
||||
=> TryReadResult(resultPath);
|
||||
|
||||
// The collectedat stamp on disk, or null when there is no readable stamp.
|
||||
//
|
||||
// This is how the service PROVES a refresh actually happened rather than
|
||||
// assuming it. Signalling the tray is asynchronous and the probe takes a
|
||||
// second or two, so a caller that simply slept would either send the old
|
||||
// spool anyway or pad the delay and hope. Watching this value advance past
|
||||
// the pre-request stamp is the only honest way to know the file in front of
|
||||
// us is the answer to OUR request and not the previous one.
|
||||
//
|
||||
// Deliberately NOT routed through TryReadResult: this must report the stamp
|
||||
// of whatever is on disk, including a spool that is expired, elevated, or
|
||||
// from the wrong account. Those are all rejected later for reporting, but
|
||||
// for ordering we only care whether the file was rewritten - conflating "not
|
||||
// usable" with "not fresh" would make the service wait out the full timeout
|
||||
// on every cycle for a machine whose spool is being rejected for some other
|
||||
// reason.
|
||||
//
|
||||
// Also deliberately NOT the file's LastWriteTime, for the reason IsExpired
|
||||
// gives: a spool can be copied or restored with a roaming profile and carry
|
||||
// a misleading filesystem timestamp. The stamp inside the file is written by
|
||||
// the probe itself.
|
||||
public DateTimeOffset? TryReadCollectedAt(string resultPath)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!File.Exists(resultPath))
|
||||
return null;
|
||||
|
||||
var info = new FileInfo(resultPath);
|
||||
if (info.Length == 0 || info.Length > MaxResultBytes)
|
||||
return null;
|
||||
|
||||
var json = File.ReadAllText(resultPath);
|
||||
if (string.IsNullOrWhiteSpace(json))
|
||||
return null;
|
||||
|
||||
var parsed = JsonSerializer.Deserialize<Dictionary<string, object?>>(json);
|
||||
if (parsed is null
|
||||
|| !parsed.TryGetValue("collectedat", out var raw)
|
||||
|| raw is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (DateTimeOffset.TryParse(
|
||||
raw.ToString(),
|
||||
System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.RoundtripKind,
|
||||
out var collectedat))
|
||||
{
|
||||
return collectedat;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
// A torn read while the probe is mid-write is the expected failure
|
||||
// here and is not worth a warning: the caller polls, so the next
|
||||
// attempt a moment later sees the completed file.
|
||||
logger.LogDebug(exception,
|
||||
"Could not read the user-context collectedat stamp at {Path}", resultPath);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// Reject a spool that some other account produced.
|
||||
//
|
||||
// Pinning the path guarantees the file lands where the reader looks, but it
|
||||
// cannot guarantee WHO ran the probe. Mapped drives and idle time are per
|
||||
// logon session, so a probe that ran as an administrator (for example a tray
|
||||
// that inherited an elevated installer's environment) would enumerate that
|
||||
// account's drive table and write it into the operator's spool. That is worse
|
||||
// than reporting nothing, because the card would look populated and correct
|
||||
// while showing drives the operator does not have.
|
||||
//
|
||||
// The probe stamps $env:USERNAME into the file, so the mismatch is checkable.
|
||||
// Logged at warning, not debug: this is a misconfigured endpoint, not a quiet
|
||||
// absence of data.
|
||||
private bool IsWrongAccount(Dictionary<string, object?> parsed)
|
||||
{
|
||||
if (!parsed.TryGetValue("username", out var raw) || raw is null)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context spool has no username stamp; ignoring it because "
|
||||
+ "the session it describes cannot be confirmed.");
|
||||
return true;
|
||||
}
|
||||
|
||||
var username = raw.ToString();
|
||||
if (string.Equals(username, FileLoggerProvider.ShopFloorUserName,
|
||||
StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
logger.LogWarning(
|
||||
"The user-context spool was written by {Actual}, not {Expected}; "
|
||||
+ "ignoring it. Session data is per logon session, so this would "
|
||||
+ "report the wrong account's mapped drives.",
|
||||
username, FileLoggerProvider.ShopFloorUserName);
|
||||
return true;
|
||||
}
|
||||
|
||||
// An elevated probe runs under a different logon token to the operator's
|
||||
// interactive one and enumerates a different, usually empty, drive table.
|
||||
// The agent refuses to collect while elevated, so a spool stamped elevated
|
||||
// should not exist; this rejects any that predates that guard or arrives
|
||||
// from a hand-run --collect-usercontext in an admin shell.
|
||||
//
|
||||
// Note the asymmetry with IsWrongAccount: a MISSING stamp is accepted here
|
||||
// rather than rejected. An unstamped spool is simply one written by an older
|
||||
// agent, and treating that as elevated would blank out session data across a
|
||||
// fleet mid-upgrade. An unstamped spool has already had to pass the username
|
||||
// check to reach this point.
|
||||
private bool IsElevatedCollection(Dictionary<string, object?> parsed)
|
||||
{
|
||||
if (!parsed.TryGetValue("elevated", out var raw) || raw is null)
|
||||
return false;
|
||||
|
||||
var elevated = raw is JsonElement element
|
||||
? element.ValueKind == JsonValueKind.True
|
||||
: string.Equals(raw.ToString(), "true", StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
if (!elevated)
|
||||
return false;
|
||||
|
||||
logger.LogWarning(
|
||||
"The user-context spool was collected by an ELEVATED process; ignoring "
|
||||
+ "it. Mapped drives are per logon token, so an elevated probe reports "
|
||||
+ "the wrong drive table even under the right account. Collection should "
|
||||
+ "come from the 'ShopDB user-context probe' scheduled task, which runs "
|
||||
+ "at RunLevel Limited.");
|
||||
return true;
|
||||
}
|
||||
|
||||
// Age is taken from the stamp the probe wrote, not from the file's last
|
||||
// write time, because the file can be copied or restored with a roaming
|
||||
// profile and carry a misleading timestamp. A result with no readable stamp
|
||||
// is treated as expired: we cannot prove it is current, so we will not claim
|
||||
// it is.
|
||||
private bool IsExpired(Dictionary<string, object?> parsed, string resultPath)
|
||||
{
|
||||
if (!parsed.TryGetValue("collectedat", out var raw) || raw is null
|
||||
|| !DateTimeOffset.TryParse(
|
||||
raw.ToString(),
|
||||
System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.RoundtripKind,
|
||||
out var collectedat))
|
||||
{
|
||||
logger.LogDebug(
|
||||
"The user-context spool file has no usable collectedat stamp; ignoring it.");
|
||||
return true;
|
||||
}
|
||||
|
||||
var age = DateTimeOffset.UtcNow - collectedat;
|
||||
if (age <= MaxResultAge)
|
||||
return false;
|
||||
|
||||
logger.LogDebug(
|
||||
"The user-context spool file is {Hours} hours old, past the {Limit} hour limit.",
|
||||
Math.Round(age.TotalHours, 1), MaxResultAge.TotalHours);
|
||||
ClearStaleResult(resultPath);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -92,6 +92,57 @@ else
|
||||
var shortRoot = Path.Combine(@"C:\PROGRA~1", "LppFixture", "good.exe");
|
||||
Check("8.3 short name resolving into the approved root", true, shortRoot, policy);
|
||||
|
||||
// --- ScheduledTaskPolicy ----------------------------------------------------
|
||||
// Pure string logic, unlike LaunchPathPolicy: a task path names an entry in the
|
||||
// Task Scheduler namespace, not a file, so there is no disk state to plant. The
|
||||
// cases that matter are the ones where a candidate LOOKS like an approved entry
|
||||
// as a string but is not the same task.
|
||||
|
||||
void CheckTask(string label, bool expectAccept, string candidate, ScheduledTaskPolicy policy)
|
||||
{
|
||||
var ok = policy.TryValidate(candidate, out var full, out var reason);
|
||||
var pass = ok == expectAccept;
|
||||
if (!pass) failures++;
|
||||
var verdict = pass ? "PASS" : "FAIL";
|
||||
var got = ok ? $"accepted -> {full}" : $"rejected ({reason})";
|
||||
Console.WriteLine($"{verdict} {label}");
|
||||
Console.WriteLine($" want={(expectAccept ? "accept" : "reject")} got={got}");
|
||||
}
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine("--- ScheduledTaskPolicy ---");
|
||||
|
||||
var taskPolicy = new ScheduledTaskPolicy(new[] { @"\ReportIP", @"\SFLD\" });
|
||||
|
||||
CheckTask("exact approved task", true, @"\ReportIP", taskPolicy);
|
||||
// Task Scheduler is case-insensitive, so the policy must be too, or the same
|
||||
// task typed differently would be refused.
|
||||
CheckTask("approved task, different case", true, @"\reportip", taskPolicy);
|
||||
// The operator should not have to know the rooted form.
|
||||
CheckTask("approved task without leading separator", true, "ReportIP", taskPolicy);
|
||||
CheckTask("approved task with forward slashes", true, "/ReportIP", taskPolicy);
|
||||
// The prefix-suffix confusion this policy exists to prevent: "\ReportIP" must
|
||||
// NOT authorize "\ReportIPEvil".
|
||||
CheckTask("sibling task sharing an approved prefix", false, @"\ReportIPEvil", taskPolicy);
|
||||
CheckTask("unapproved task", false, @"\Evil", taskPolicy);
|
||||
CheckTask("empty task path", false, "", taskPolicy);
|
||||
// Folder entries authorize everything beneath them, but not a sibling folder
|
||||
// whose name merely starts the same way.
|
||||
CheckTask("task inside an approved folder", true, @"\SFLD\ApplyDSCConfig", taskPolicy);
|
||||
CheckTask("folder itself", true, @"\SFLD", taskPolicy);
|
||||
CheckTask("sibling folder sharing the prefix", false, @"\SFLDEvil\Thing", taskPolicy);
|
||||
// Traversal would let an approved prefix string-match something else entirely.
|
||||
CheckTask("traversal out of an approved folder", false, @"\SFLD\..\Evil", taskPolicy);
|
||||
// Defense in depth: the invocation uses ArgumentList, but a task path with
|
||||
// these characters is not one we should be starting.
|
||||
CheckTask("argument-injection characters", false, "\\ReportIP\" & calc", taskPolicy);
|
||||
CheckTask("control character", false, "\\Report\u0000IP", taskPolicy);
|
||||
// An empty override must fall back to the shipped defaults, never to "allow
|
||||
// everything".
|
||||
var defaultPolicy = new ScheduledTaskPolicy(Array.Empty<string>());
|
||||
CheckTask("shipped default is honored when override is empty", true, @"\ReportIP", defaultPolicy);
|
||||
CheckTask("empty override does not mean allow-all", false, @"\Evil", defaultPolicy);
|
||||
|
||||
Console.WriteLine();
|
||||
Console.WriteLine(failures == 0 ? "ALL EXPECTATIONS MET" : $"{failures} EXPECTATION(S) NOT MET");
|
||||
return failures == 0 ? 0 : 1;
|
||||
|
||||
@@ -16,5 +16,9 @@
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<Compile Include="../ShopDbEndpointAgent.Core/LaunchPathPolicy.cs" Link="LaunchPathPolicy.cs" />
|
||||
<!-- ScheduledTaskPolicy is pure string logic with no filesystem state to
|
||||
plant, so unlike LaunchPathPolicy it would run anywhere. It is linked
|
||||
here anyway to keep both allowlist policies provable from one harness. -->
|
||||
<Compile Include="../ShopDbEndpointAgent.Core/ScheduledTaskPolicy.cs" Link="ScheduledTaskPolicy.cs" />
|
||||
</ItemGroup>
|
||||
</Project>
|
||||
|
||||
457
agent/ShopDbEndpointAgent.Tray/FleetAlertForm.cs
Normal file
@@ -0,0 +1,457 @@
|
||||
using System.Drawing;
|
||||
using System.Drawing.Drawing2D;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ShopDbEndpointAgent.Core;
|
||||
|
||||
namespace ShopDbEndpointAgent.Tray;
|
||||
|
||||
// Displays fleet-wide alert notifications, relayed by the local agent, in a toast-style
|
||||
// window. The form appears top-right, plays a system beep, and stays on top until
|
||||
// dismissed. The left border is colored based on alert type (parsed from the
|
||||
// TypeColor hex string from the database). The operator acknowledges the alert by
|
||||
// clicking the button or the X with the mouse. The form steals focus when shown,
|
||||
// so neither button is a tab stop, there is no Accept/Cancel button, keyboard
|
||||
// clicks are ignored, and mouse clicks are ignored until ArmDelay has passed:
|
||||
// keystrokes meant for another app must not acknowledge an unseen alert.
|
||||
// Closing any other way (server withdrew it, timeout, shutdown) is NOT an
|
||||
// acknowledgement.
|
||||
internal sealed class FleetAlertForm : Form
|
||||
{
|
||||
private const int MaxMessageLength = 8000;
|
||||
private const string DefaultTypeName = "Alert";
|
||||
private const string DefaultTitle = "(no title)";
|
||||
private static readonly TimeSpan ArmDelay = TimeSpan.FromMilliseconds(1500);
|
||||
private static readonly Color DefaultTypeColor = Color.Gray;
|
||||
|
||||
private readonly FleetAlert alert;
|
||||
private readonly AgentPipeClient? client;
|
||||
private readonly ILogger? logger;
|
||||
private System.Windows.Forms.Timer? countdownTimer;
|
||||
private System.Windows.Forms.Timer? statusCheckTimer;
|
||||
private readonly CancellationTokenSource statusCheckCancellation = new();
|
||||
private int remainingSeconds;
|
||||
private Label? countdownLabel;
|
||||
private TextBox? messageBox;
|
||||
private readonly List<Font> fonts = new();
|
||||
private DateTime armedAtUtc = DateTime.MaxValue;
|
||||
private bool acknowledged;
|
||||
|
||||
public event EventHandler? OnAcknowledgeClicked;
|
||||
|
||||
public FleetAlertForm(
|
||||
FleetAlert alert, AgentPipeClient? client = null, ILogger? logger = null)
|
||||
{
|
||||
this.alert = alert ?? throw new ArgumentNullException(nameof(alert));
|
||||
this.client = client;
|
||||
this.logger = logger;
|
||||
|
||||
SuspendLayout();
|
||||
AutoScaleDimensions = new SizeF(96F, 96F);
|
||||
AutoScaleMode = AutoScaleMode.Dpi;
|
||||
Text = string.Empty;
|
||||
FormBorderStyle = FormBorderStyle.None;
|
||||
TopMost = true;
|
||||
ShowInTaskbar = false;
|
||||
StartPosition = FormStartPosition.Manual;
|
||||
ClientSize = new Size(400, 320);
|
||||
BackColor = Color.White;
|
||||
KeyPreview = false;
|
||||
|
||||
BuildLayout();
|
||||
ResumeLayout(false);
|
||||
|
||||
Load += FleetAlertForm_Load;
|
||||
Shown += FleetAlertForm_Shown;
|
||||
Resize += (s, e) => ApplyRoundedCorners();
|
||||
FormClosing += FleetAlertForm_FormClosing;
|
||||
}
|
||||
|
||||
private void FleetAlertForm_FormClosing(object? sender, FormClosingEventArgs e)
|
||||
{
|
||||
countdownTimer?.Stop();
|
||||
statusCheckTimer?.Stop();
|
||||
statusCheckCancellation.Cancel();
|
||||
}
|
||||
|
||||
private void FleetAlertForm_Load(object? sender, EventArgs e)
|
||||
{
|
||||
// Size is final (DPI scaled) by Load; place and shape here.
|
||||
var screen = Screen.PrimaryScreen?.WorkingArea ?? Screen.PrimaryScreen?.Bounds ?? new Rectangle(0, 0, 1920, 1080);
|
||||
Location = new Point(screen.Right - Width - 20, screen.Top + 20);
|
||||
ApplyRoundedCorners();
|
||||
|
||||
System.Media.SystemSounds.Beep.Play();
|
||||
|
||||
if (client != null)
|
||||
{
|
||||
StartStatusCheck();
|
||||
}
|
||||
}
|
||||
|
||||
private void FleetAlertForm_Shown(object? sender, EventArgs e)
|
||||
{
|
||||
armedAtUtc = DateTime.UtcNow + ArmDelay;
|
||||
if (messageBox != null)
|
||||
{
|
||||
ActiveControl = messageBox;
|
||||
messageBox.Select(0, 0);
|
||||
}
|
||||
}
|
||||
|
||||
private void StartStatusCheck()
|
||||
{
|
||||
statusCheckTimer = new System.Windows.Forms.Timer { Interval = 30000 };
|
||||
statusCheckTimer.Tick += StatusCheckTimer_Tick;
|
||||
statusCheckTimer.Start();
|
||||
}
|
||||
|
||||
private void StatusCheckTimer_Tick(object? sender, EventArgs e)
|
||||
{
|
||||
if (client == null)
|
||||
return;
|
||||
|
||||
var token = statusCheckCancellation.Token;
|
||||
if (token.IsCancellationRequested)
|
||||
return;
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var alerts = await client.GetAlertsAsync(token)
|
||||
.ConfigureAwait(false);
|
||||
var stillActive = alerts.Any(a => a.NotificationId == alert.NotificationId);
|
||||
|
||||
if (!stillActive && !token.IsCancellationRequested && IsHandleCreated && !IsDisposed)
|
||||
{
|
||||
BeginInvoke(new Action(() =>
|
||||
{
|
||||
statusCheckTimer?.Stop();
|
||||
Close();
|
||||
}));
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (token.IsCancellationRequested)
|
||||
{
|
||||
// Form is closing
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
// Agent unreachable or refused: keep the form open
|
||||
logger?.LogDebug(exception,
|
||||
"Status check for fleet alert {NotificationId} failed.", alert.NotificationId);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void ApplyRoundedCorners()
|
||||
{
|
||||
using var path = new GraphicsPath();
|
||||
int radius = LogicalToDeviceUnits(8);
|
||||
var rect = new Rectangle(0, 0, Width, Height);
|
||||
|
||||
path.AddArc(rect.X, rect.Y, radius, radius, 180, 90);
|
||||
path.AddArc(rect.Right - radius, rect.Y, radius, radius, 270, 90);
|
||||
path.AddArc(rect.Right - radius, rect.Bottom - radius, radius, radius, 0, 90);
|
||||
path.AddArc(rect.X, rect.Bottom - radius, radius, radius, 90, 90);
|
||||
path.CloseFigure();
|
||||
|
||||
var old = Region;
|
||||
Region = new Region(path);
|
||||
old?.Dispose();
|
||||
}
|
||||
|
||||
// Not wired (1.6.2 made alerts manual-only). If wired, a timeout closes the
|
||||
// form WITHOUT acknowledging, so the alert can be surfaced again.
|
||||
private void StartCountdown()
|
||||
{
|
||||
var minutes = alert.TimeoutMinutes;
|
||||
if (minutes <= 0)
|
||||
return;
|
||||
|
||||
remainingSeconds = minutes * 60;
|
||||
|
||||
countdownTimer = new System.Windows.Forms.Timer { Interval = 1000 };
|
||||
countdownTimer.Tick += CountdownTimer_Tick;
|
||||
countdownTimer.Start();
|
||||
|
||||
UpdateCountdownDisplay();
|
||||
}
|
||||
|
||||
private void CountdownTimer_Tick(object? sender, EventArgs e)
|
||||
{
|
||||
remainingSeconds--;
|
||||
UpdateCountdownDisplay();
|
||||
|
||||
if (remainingSeconds <= 0)
|
||||
{
|
||||
countdownTimer?.Stop();
|
||||
Close();
|
||||
}
|
||||
}
|
||||
|
||||
private void UpdateCountdownDisplay()
|
||||
{
|
||||
if (countdownLabel != null)
|
||||
{
|
||||
int minutes = remainingSeconds / 60;
|
||||
int seconds = remainingSeconds % 60;
|
||||
countdownLabel.Text = $"Closes in {minutes}:{seconds:D2}";
|
||||
}
|
||||
}
|
||||
|
||||
private Font CreateFont(string family, float size, FontStyle style = FontStyle.Regular)
|
||||
{
|
||||
var font = new Font(family, size, style);
|
||||
fonts.Add(font);
|
||||
return font;
|
||||
}
|
||||
|
||||
private static string SafeText(string? value, string fallback)
|
||||
{
|
||||
return string.IsNullOrWhiteSpace(value) ? fallback : value.Trim();
|
||||
}
|
||||
|
||||
private static string PrepareMessage(string? value)
|
||||
{
|
||||
var text = value ?? string.Empty;
|
||||
if (text.Length > MaxMessageLength)
|
||||
text = text.Substring(0, MaxMessageLength) + "...";
|
||||
return text.Replace("\r\n", "\n").Replace('\r', '\n').Replace("\n", "\r\n");
|
||||
}
|
||||
|
||||
private void BuildLayout()
|
||||
{
|
||||
// Dock order: Fill added first, outermost edges added last.
|
||||
var contentPanel = new Panel
|
||||
{
|
||||
Dock = DockStyle.Fill,
|
||||
BackColor = Color.White,
|
||||
};
|
||||
Controls.Add(contentPanel);
|
||||
|
||||
var borderPanel = new Panel
|
||||
{
|
||||
BackColor = ParseTypeColor(alert.TypeColor),
|
||||
Dock = DockStyle.Left,
|
||||
Width = 5,
|
||||
};
|
||||
Controls.Add(borderPanel);
|
||||
|
||||
var messagePanel = new Panel
|
||||
{
|
||||
Dock = DockStyle.Fill,
|
||||
Padding = new Padding(20, 4, 12, 4),
|
||||
BackColor = Color.White,
|
||||
};
|
||||
contentPanel.Controls.Add(messagePanel);
|
||||
|
||||
messageBox = new TextBox
|
||||
{
|
||||
Text = PrepareMessage(alert.Message),
|
||||
Dock = DockStyle.Fill,
|
||||
Multiline = true,
|
||||
ReadOnly = true,
|
||||
WordWrap = true,
|
||||
ScrollBars = ScrollBars.Vertical,
|
||||
BorderStyle = BorderStyle.None,
|
||||
BackColor = Color.White,
|
||||
Font = CreateFont("Segoe UI", 9.75f),
|
||||
ForeColor = Color.FromArgb(85, 85, 85),
|
||||
TabIndex = 0,
|
||||
};
|
||||
messagePanel.Controls.Add(messageBox);
|
||||
|
||||
var titleLabel = new Label
|
||||
{
|
||||
Text = SafeText(alert.Title, DefaultTitle),
|
||||
Dock = DockStyle.Top,
|
||||
Height = 50,
|
||||
Padding = new Padding(16, 8, 16, 0),
|
||||
Font = CreateFont("Segoe UI", 11, FontStyle.Bold),
|
||||
ForeColor = Color.FromArgb(51, 51, 51),
|
||||
AutoSize = false,
|
||||
AutoEllipsis = true,
|
||||
UseMnemonic = false,
|
||||
};
|
||||
contentPanel.Controls.Add(titleLabel);
|
||||
|
||||
var buttonPanel = new Panel
|
||||
{
|
||||
Dock = DockStyle.Bottom,
|
||||
Height = 46,
|
||||
BackColor = Color.White,
|
||||
};
|
||||
contentPanel.Controls.Add(buttonPanel);
|
||||
|
||||
var acknowledgeButton = new MouseOnlyButton(IsArmed)
|
||||
{
|
||||
Text = "Acknowledge",
|
||||
Width = 140,
|
||||
Height = 32,
|
||||
Left = (395 - 140) / 2,
|
||||
Top = 7,
|
||||
Anchor = AnchorStyles.None,
|
||||
BackColor = Color.FromArgb(76, 175, 80),
|
||||
ForeColor = Color.White,
|
||||
FlatStyle = FlatStyle.Flat,
|
||||
Font = CreateFont("Segoe UI", 9, FontStyle.Bold),
|
||||
Cursor = Cursors.Hand,
|
||||
TabStop = false,
|
||||
UseMnemonic = false,
|
||||
};
|
||||
acknowledgeButton.FlatAppearance.BorderSize = 0;
|
||||
acknowledgeButton.FlatAppearance.MouseOverBackColor = Color.FromArgb(69, 160, 73);
|
||||
acknowledgeButton.Click += AcknowledgeButton_Click;
|
||||
buttonPanel.Controls.Add(acknowledgeButton);
|
||||
|
||||
var footerPanel = new Panel
|
||||
{
|
||||
Dock = DockStyle.Bottom,
|
||||
Height = 40,
|
||||
BackColor = Color.White,
|
||||
};
|
||||
contentPanel.Controls.Add(footerPanel);
|
||||
|
||||
countdownLabel = new Label
|
||||
{
|
||||
Text = "Manual acknowledgment required",
|
||||
Dock = DockStyle.Fill,
|
||||
TextAlign = ContentAlignment.MiddleCenter,
|
||||
Font = CreateFont("Segoe UI", 8.25f),
|
||||
ForeColor = Color.FromArgb(153, 153, 153),
|
||||
UseMnemonic = false,
|
||||
};
|
||||
footerPanel.Controls.Add(countdownLabel);
|
||||
|
||||
var footerBorder = new Panel
|
||||
{
|
||||
Dock = DockStyle.Top,
|
||||
Height = 1,
|
||||
BackColor = Color.FromArgb(238, 238, 238),
|
||||
};
|
||||
footerPanel.Controls.Add(footerBorder);
|
||||
|
||||
var headerPanel = new Panel
|
||||
{
|
||||
Dock = DockStyle.Top,
|
||||
Height = 45,
|
||||
BackColor = Color.FromArgb(245, 245, 245),
|
||||
};
|
||||
contentPanel.Controls.Add(headerPanel);
|
||||
|
||||
var typeLabel = new Label
|
||||
{
|
||||
Text = SafeText(alert.TypeName, DefaultTypeName).ToUpperInvariant(),
|
||||
Dock = DockStyle.Fill,
|
||||
Padding = new Padding(12, 0, 0, 0),
|
||||
TextAlign = ContentAlignment.MiddleLeft,
|
||||
AutoEllipsis = true,
|
||||
Font = CreateFont("Segoe UI", 10, FontStyle.Bold),
|
||||
ForeColor = Color.FromArgb(51, 51, 51),
|
||||
UseMnemonic = false,
|
||||
};
|
||||
headerPanel.Controls.Add(typeLabel);
|
||||
|
||||
var closeButton = new MouseOnlyButton(IsArmed)
|
||||
{
|
||||
Text = "X",
|
||||
Dock = DockStyle.Right,
|
||||
Width = 40,
|
||||
FlatStyle = FlatStyle.Flat,
|
||||
Font = CreateFont("Segoe UI", 11),
|
||||
ForeColor = Color.FromArgb(102, 102, 102),
|
||||
BackColor = Color.Transparent,
|
||||
Cursor = Cursors.Hand,
|
||||
TabStop = false,
|
||||
UseMnemonic = false,
|
||||
};
|
||||
closeButton.FlatAppearance.BorderSize = 0;
|
||||
closeButton.FlatAppearance.MouseOverBackColor = Color.FromArgb(224, 224, 224);
|
||||
closeButton.Click += CloseButton_Click;
|
||||
headerPanel.Controls.Add(closeButton);
|
||||
}
|
||||
|
||||
private bool IsArmed() => DateTime.UtcNow >= armedAtUtc;
|
||||
|
||||
private static Color ParseTypeColor(string? hexColor)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(hexColor))
|
||||
return DefaultTypeColor;
|
||||
|
||||
var hex = hexColor.Trim().TrimStart('#');
|
||||
if (hex.Length != 6 || !int.TryParse(hex, System.Globalization.NumberStyles.AllowHexSpecifier,
|
||||
System.Globalization.CultureInfo.InvariantCulture, out var rgb))
|
||||
return DefaultTypeColor;
|
||||
|
||||
return Color.FromArgb((rgb >> 16) & 0xFF, (rgb >> 8) & 0xFF, rgb & 0xFF);
|
||||
}
|
||||
|
||||
private void Acknowledge()
|
||||
{
|
||||
if (acknowledged)
|
||||
return;
|
||||
acknowledged = true;
|
||||
|
||||
countdownTimer?.Stop();
|
||||
OnAcknowledgeClicked?.Invoke(this, EventArgs.Empty);
|
||||
DialogResult = DialogResult.OK;
|
||||
Close();
|
||||
}
|
||||
|
||||
private void AcknowledgeButton_Click(object? sender, EventArgs e) => Acknowledge();
|
||||
|
||||
private void CloseButton_Click(object? sender, EventArgs e) => Acknowledge();
|
||||
|
||||
protected override void Dispose(bool disposing)
|
||||
{
|
||||
if (disposing)
|
||||
{
|
||||
countdownTimer?.Stop();
|
||||
countdownTimer?.Dispose();
|
||||
statusCheckTimer?.Stop();
|
||||
statusCheckTimer?.Dispose();
|
||||
statusCheckCancellation.Cancel();
|
||||
statusCheckCancellation.Dispose();
|
||||
foreach (var font in fonts)
|
||||
{
|
||||
font.Dispose();
|
||||
}
|
||||
fonts.Clear();
|
||||
}
|
||||
base.Dispose(disposing);
|
||||
}
|
||||
|
||||
// Raises Click only for a left mouse release on the button once armed.
|
||||
// Space, Enter, mnemonics and PerformClick all route through OnClick
|
||||
// without passing OnMouseUp, so they are dropped.
|
||||
private sealed class MouseOnlyButton : Button
|
||||
{
|
||||
private readonly Func<bool> isArmed;
|
||||
private bool inMouseUp;
|
||||
|
||||
public MouseOnlyButton(Func<bool> isArmed)
|
||||
{
|
||||
this.isArmed = isArmed;
|
||||
}
|
||||
|
||||
protected override void OnMouseUp(MouseEventArgs mevent)
|
||||
{
|
||||
inMouseUp = true;
|
||||
try
|
||||
{
|
||||
base.OnMouseUp(mevent);
|
||||
}
|
||||
finally
|
||||
{
|
||||
inMouseUp = false;
|
||||
}
|
||||
}
|
||||
|
||||
protected override void OnClick(EventArgs e)
|
||||
{
|
||||
if (inMouseUp && isArmed())
|
||||
base.OnClick(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
269
agent/ShopDbEndpointAgent.Tray/FleetAlertWorker.cs
Normal file
@@ -0,0 +1,269 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ShopDbEndpointAgent.Core;
|
||||
|
||||
namespace ShopDbEndpointAgent.Tray;
|
||||
|
||||
// Asks the local agent for fleet-wide alerts every 60 seconds and displays new
|
||||
// alerts to the operator via FleetAlertForm. The agent holds the credential and
|
||||
// talks to the server; the tray only speaks the pipe. An alert is recorded in
|
||||
// FleetAlertStore only once the agent confirms the acknowledgement, so a failed
|
||||
// ack lets it reappear. Runs for the tray's lifetime on a background thread.
|
||||
internal sealed class FleetAlertWorker : IDisposable
|
||||
{
|
||||
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
|
||||
private static readonly TimeSpan StartupDelay = TimeSpan.FromSeconds(15);
|
||||
|
||||
// Agent down polls every cycle; warn once, then at most this often.
|
||||
private static readonly TimeSpan UnreachableLogInterval = TimeSpan.FromMinutes(15);
|
||||
|
||||
private readonly AgentPipeClient client;
|
||||
private readonly FleetAlertStore store;
|
||||
private readonly ILogger logger;
|
||||
private readonly SynchronizationContext? uiContext;
|
||||
private readonly CancellationTokenSource cancellation;
|
||||
private readonly Thread workerThread;
|
||||
|
||||
// Set and cleared on the UI thread only.
|
||||
private FleetAlertForm? openForm;
|
||||
|
||||
// Poller thread only.
|
||||
private DateTime lastUnreachableLogUtc = DateTime.MinValue;
|
||||
private int unreachableCount;
|
||||
|
||||
public FleetAlertWorker(
|
||||
AgentPipeClient client,
|
||||
FleetAlertStore store,
|
||||
ILogger logger,
|
||||
SynchronizationContext? uiContext = null)
|
||||
{
|
||||
this.client = client ?? throw new ArgumentNullException(nameof(client));
|
||||
this.store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
this.uiContext = uiContext ?? SynchronizationContext.Current;
|
||||
cancellation = new CancellationTokenSource();
|
||||
|
||||
workerThread = new Thread(Run)
|
||||
{
|
||||
IsBackground = true,
|
||||
Name = "ShopDB fleet alert poller",
|
||||
};
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Fleet alert worker starting; first poll in {Delay}, then every {Interval}.",
|
||||
StartupDelay, PollInterval);
|
||||
workerThread.Start();
|
||||
}
|
||||
|
||||
private void Run()
|
||||
{
|
||||
var token = cancellation.Token;
|
||||
if (token.WaitHandle.WaitOne(StartupDelay))
|
||||
return;
|
||||
|
||||
while (!token.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
PollAndDisplayAlerts(token);
|
||||
}
|
||||
catch (OperationCanceledException) when (token.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (AgentPipeException exception)
|
||||
{
|
||||
LogUnreachable(exception);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception, "Fleet alert poll failed; retrying on next cycle.");
|
||||
}
|
||||
|
||||
if (token.WaitHandle.WaitOne(PollInterval))
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
private void LogUnreachable(AgentPipeException exception)
|
||||
{
|
||||
unreachableCount++;
|
||||
var now = DateTime.UtcNow;
|
||||
if (now - lastUnreachableLogUtc < UnreachableLogInterval)
|
||||
return;
|
||||
lastUnreachableLogUtc = now;
|
||||
logger.LogWarning(
|
||||
"Could not get fleet alerts from the agent ({Count} failed poll(s)): {Error}. "
|
||||
+ "Retrying on next cycle.", unreachableCount, exception.Message);
|
||||
}
|
||||
|
||||
private void PollAndDisplayAlerts(CancellationToken token)
|
||||
{
|
||||
var alerts = client.GetAlertsAsync(token).GetAwaiter().GetResult();
|
||||
if (unreachableCount > 0)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Agent answered the alert poll again after {Count} failed poll(s).", unreachableCount);
|
||||
unreachableCount = 0;
|
||||
lastUnreachableLogUtc = DateTime.MinValue;
|
||||
}
|
||||
if (alerts is FleetAlertList { Skipped.Count: > 0 } parsed)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"Skipped {Count} malformed fleet alert(s): {Details}",
|
||||
parsed.Skipped.Count, string.Join("; ", parsed.Skipped));
|
||||
}
|
||||
if (alerts.Count == 0)
|
||||
{
|
||||
logger.LogDebug("No active fleet alerts.");
|
||||
return;
|
||||
}
|
||||
|
||||
var newAlerts = alerts
|
||||
.Where(alert => !store.IsAcknowledged(alert.NotificationId))
|
||||
.ToList();
|
||||
|
||||
if (newAlerts.Count == 0)
|
||||
{
|
||||
logger.LogDebug("All {Count} fleet alerts already acknowledged.", alerts.Count);
|
||||
return;
|
||||
}
|
||||
|
||||
logger.LogInformation("Found {Count} new fleet alert(s).", newAlerts.Count);
|
||||
|
||||
foreach (var alert in newAlerts)
|
||||
{
|
||||
if (token.IsCancellationRequested)
|
||||
return;
|
||||
|
||||
ShowAlertOnUIThread(alert, token);
|
||||
}
|
||||
}
|
||||
|
||||
private void ShowAlertOnUIThread(FleetAlert alert, CancellationToken token)
|
||||
{
|
||||
var shown = new ManualResetEventSlim(false);
|
||||
Exception? formException = null;
|
||||
|
||||
var showAction = new SendOrPostCallback(_ =>
|
||||
{
|
||||
try
|
||||
{
|
||||
if (token.IsCancellationRequested)
|
||||
return;
|
||||
|
||||
using var form = new FleetAlertForm(alert, client, logger);
|
||||
form.OnAcknowledgeClicked += (s, e) => OnAlertAcknowledged(alert, token);
|
||||
openForm = form;
|
||||
try
|
||||
{
|
||||
form.ShowDialog();
|
||||
}
|
||||
finally
|
||||
{
|
||||
openForm = null;
|
||||
}
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
formException = exception;
|
||||
}
|
||||
finally
|
||||
{
|
||||
shown.Set();
|
||||
}
|
||||
});
|
||||
|
||||
if (uiContext is not null)
|
||||
uiContext.Post(showAction, null);
|
||||
else
|
||||
showAction(null);
|
||||
|
||||
shown.Wait(token);
|
||||
|
||||
if (formException is not null)
|
||||
{
|
||||
logger.LogWarning(formException,
|
||||
"Could not display fleet alert {NotificationId}.", alert.NotificationId);
|
||||
}
|
||||
}
|
||||
|
||||
private void OnAlertAcknowledged(FleetAlert alert, CancellationToken token)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Fleet alert {NotificationId} acknowledged; sending to the agent.", alert.NotificationId);
|
||||
|
||||
// Off the UI thread. Recorded locally only on Ok, so a failed ack
|
||||
// shows the alert again on a later poll instead of losing it.
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var response = await client.AckAlertAsync(alert.NotificationId, token);
|
||||
if (response.Ok)
|
||||
{
|
||||
store.MarkAcknowledged(alert.NotificationId);
|
||||
logger.LogInformation("Fleet alert {NotificationId} acknowledged by the agent.",
|
||||
alert.NotificationId);
|
||||
}
|
||||
else
|
||||
{
|
||||
logger.LogWarning(
|
||||
"Agent did not acknowledge fleet alert {NotificationId}: {Error}. It may be shown again.",
|
||||
alert.NotificationId, response.Error);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (token.IsCancellationRequested)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Acknowledgement of fleet alert {NotificationId} cancelled by shutdown.",
|
||||
alert.NotificationId);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception,
|
||||
"Could not acknowledge fleet alert {NotificationId}; it may be shown again.",
|
||||
alert.NotificationId);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Closes a displayed alert without acknowledging it, so the form's status
|
||||
// check stops polling the agent on shutdown.
|
||||
public void CloseOpenAlert()
|
||||
{
|
||||
var form = openForm;
|
||||
if (form is null || form.IsDisposed)
|
||||
return;
|
||||
|
||||
try
|
||||
{
|
||||
if (form.InvokeRequired)
|
||||
form.BeginInvoke(new Action(form.Close));
|
||||
else
|
||||
form.Close();
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception, "Could not close open fleet alert on shutdown.");
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
cancellation.Cancel();
|
||||
CloseOpenAlert();
|
||||
|
||||
if (workerThread.IsAlive)
|
||||
{
|
||||
if (!workerThread.Join(TimeSpan.FromSeconds(5)))
|
||||
{
|
||||
logger.LogWarning("Fleet alert worker thread did not exit within timeout.");
|
||||
}
|
||||
}
|
||||
|
||||
cancellation.Dispose();
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,122 @@
|
||||
<UseWindowsForms>true</UseWindowsForms>
|
||||
<RootNamespace>ShopDbEndpointAgent.Tray</RootNamespace>
|
||||
<AssemblyName>shopdb-endpoint-tray</AssemblyName>
|
||||
<!-- 1.1.0: the tray now collects session-scoped user context. It invokes
|
||||
shopdbagent.exe with the collect-usercontext verb on a timer (once
|
||||
shortly after logon, then periodically) and the agent service reads the
|
||||
resulting spool file. The tray owns this because it is the only ShopDB
|
||||
process already running as the operator: mapped drives, user-owned
|
||||
listeners, and input idle time do not exist in the service's session 0.
|
||||
Requires agent 1.8.0 or newer: an older agent does not recognize the
|
||||
verb and falls through to starting a full host, so the tray and agent
|
||||
must be deployed together. -->
|
||||
<!-- 1.0.0: first explicitly stamped tray version. The tray and the agent
|
||||
service version independently (they are separate exes and can be pushed
|
||||
apart), so the tray carries its own <Version> and the agent reports it
|
||||
as trayversion alongside its own agentversion (AGENT-CONTRACT.md 2). -->
|
||||
<!-- 1.1.1: the tray now logs. Every user-context failure path was silent -
|
||||
a missing shopdbagent.exe returned bare, and the probe launch was
|
||||
wrapped in an empty catch - so "no mapped drives" produced no evidence
|
||||
anywhere on the machine. The tray writes
|
||||
C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log (the operator's
|
||||
own profile; it cannot write to ProgramData or Program Files) covering
|
||||
tray startup, agent-exe resolution, probe launch and probe exit code. -->
|
||||
<!-- 1.2.0: the tray stops claiming success it did not verify. The probe
|
||||
completion message printed a hardcoded spool path regardless of what
|
||||
happened, so a probe that exited 0 without writing anything read as a
|
||||
clean collection; it now reports the real pinned path and distinguishes
|
||||
spool-present, exit-0-but-no-file, and non-zero exit. Startup probe
|
||||
delay is chosen from session uptime: the 2-minute delay still applies at
|
||||
logon (drive mappings are not established yet), but a tray started into
|
||||
an already-running session - which is what an installer does - probes
|
||||
after 15 seconds instead of leaving a fresh install blind for the full
|
||||
30-minute cycle. A log sink that cannot open its directory raises a
|
||||
one-time balloon, since it cannot report its own failure through the
|
||||
log. Requires agent 1.9.0: the two carry the shared pinned spool path
|
||||
and must be deployed together. -->
|
||||
<!-- 1.3.0: the tray is no longer the authoritative user-context collector.
|
||||
The 'ShopDB user-context probe' scheduled task now owns collection,
|
||||
because it is the only mechanism that runs in the operator's own logon
|
||||
session at RunLevel Limited; the tray keeps its own probe cadence as
|
||||
belt-and-braces and writes the same spool. The installer no longer
|
||||
starts the tray directly (explorer.exe de-elevated it but did not
|
||||
change ACCOUNT, so an admin install produced an admin tray whose
|
||||
probe saw the wrong drive table). Autostart at logon is the Run key;
|
||||
install-time startup on a machine someone is already signed in to is
|
||||
a transient 'ShopDB tray first launch' scheduled task, which is the
|
||||
only way to enter the operator's existing session unelevated. Probe
|
||||
exit code 2 - the agent refusing to collect while elevated - is
|
||||
reported by name instead of as a bare code. Requires agent 1.10.1. -->
|
||||
<!-- 1.4.0: the tray answers on-demand refresh requests. A background
|
||||
listener waits on Global\ShopDbEndpointAgent.UserContextRefresh and
|
||||
runs the probe when the service asks, which is what makes a WebUI
|
||||
"Refresh Status" click produce current mapped drives instead of the
|
||||
last cadence's answer - the service cannot collect this itself at any
|
||||
price, since drive letters live on the operator's logon token, so the
|
||||
request has to be serviced HERE. Each probe is marshalled onto the UI
|
||||
thread: the cadence timer and a service request could otherwise both
|
||||
pass the in-flight guard and race the same spool file. "Send Status"
|
||||
now awaits a probe before invoking the once verb, because that verb
|
||||
reads the spool and would otherwise publish the previous probe's
|
||||
answer to an operator who just changed something and is watching
|
||||
for it. The probe
|
||||
body is one shared method used by all three callers rather than three
|
||||
copies of the exit-code handling. A tray that cannot open the channel
|
||||
logs and keeps its timer, so the failure mode is "refresh is not
|
||||
immediate", not "session data stops working". Requires agent 1.11.0. -->
|
||||
<!-- 1.6.1: fix crash when clicking acknowledge button. The countdown timer
|
||||
is now properly stopped before the form closes to prevent disposal issues. -->
|
||||
<!-- 1.6.2: disable auto-dismiss timer. Fleet alerts now require manual
|
||||
acknowledgment only. Footer shows "Manual acknowledgment required" instead
|
||||
of countdown timer. -->
|
||||
<!-- 1.6.3: auto-close when notification becomes inactive. Fleet alert forms now
|
||||
check every 30 seconds if the notification is still active on the server and
|
||||
automatically close if it has ended or been deactivated. -->
|
||||
<!-- 1.6.4: fix UI thread deadlock. Acknowledgment and status checks now run async
|
||||
properly without blocking the UI thread. Acknowledgment is marked locally first,
|
||||
then posted to server in background to prevent lockups. -->
|
||||
<!-- 1.6.5: fix resource leaks and disposal issues. GraphicsPath properly disposed,
|
||||
Font objects tracked and disposed, worker thread joined before client disposal,
|
||||
Screen.PrimaryScreen null dereference prevented, status check cancellation token
|
||||
propagated to prevent orphaned HTTP requests. -->
|
||||
<!-- 1.6.6: coordinated release with agent 1.13.5 (no tray changes).
|
||||
No tray code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.6.7: coordinated release with agent 1.14.1 (wilm merge).
|
||||
No tray code changes, but deployed together after wilm merge. -->
|
||||
<!-- 1.6.8: coordinated release with agent 1.14.2 (fleet alert scope fixes).
|
||||
No tray code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.6.9: coordinated release with agent 1.17.0 (app delta optimization).
|
||||
No tray code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.6.10: coordinated release with agent 1.18.0 (fingerprint computation).
|
||||
No tray code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.6.11: fleet alert hardening. The toast steals focus, so its buttons are
|
||||
no longer tab stops, there is no Accept/Cancel button, and acknowledge
|
||||
needs a mouse click at least 1.5s after the form is shown: keystrokes
|
||||
meant for another app cannot acknowledge an unseen alert. Long messages
|
||||
scroll instead of being cut off and '&' is shown literally. A null or
|
||||
malformed alert field no longer throws away the whole poll. A timeout
|
||||
closes without acknowledging. The acknowledged store moves to
|
||||
%LOCALAPPDATA%\ShopDB\FleetAlerts (the operator could not write the
|
||||
ProgramData copy, so acks were lost every logon), merging the old file
|
||||
once. Shutdown no longer blocks on the poller's sleeps. -->
|
||||
<!-- 1.7.0: the tray holds no credential. Fleet alerts, alert acknowledgement,
|
||||
Send Status and Report an issue all go through the SYSTEM agent's local
|
||||
pipe (ShopDB.EndpointAgent.v1) instead of reading the collector key and
|
||||
config or shelling the agent exe with the once and report verbs, so a
|
||||
standard user on the bay can no longer lift the fleet key through the
|
||||
tray's access path. The state.json asset-id reader is gone (the agent
|
||||
knows its own identity). An unreachable agent shows the agent-not-running
|
||||
balloon, and alert polling logs at a throttled rate and retries. An alert
|
||||
is recorded as acknowledged only when the agent confirms it, so a failed
|
||||
ack lets it reappear. The collect-usercontext probe is unchanged.
|
||||
Requires the agent release that hosts the pipe.
|
||||
|
||||
1.7.1: no behaviour of its own. The fleet alert key rename landed
|
||||
after the 1.7.0 binary was built and shipped to the display scope,
|
||||
so the rebuild carries different bytes under the same version. The
|
||||
version moves so the two can be told apart. Pairs with agent
|
||||
1.20.1. -->
|
||||
<Version>1.7.1</Version>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
|
||||
<SelfContained>true</SelfContained>
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
using System.Diagnostics;
|
||||
using System.Reflection;
|
||||
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
using ShopDbEndpointAgent.Core;
|
||||
|
||||
namespace ShopDbEndpointAgent.Tray;
|
||||
@@ -9,11 +11,11 @@ namespace ShopDbEndpointAgent.Tray;
|
||||
// and headless-friendly: no main window, the icon is the whole UI surface until
|
||||
// the operator opens the ticket dialog.
|
||||
//
|
||||
// "Send Status now" reuses the exact production collection path by shelling out
|
||||
// to the installed shopdbagent.exe with --once. That keeps the collector key and
|
||||
// all collection logic in the service binary: the tray never holds a secret and
|
||||
// never duplicates wiring, it just asks the agent to report on demand so the
|
||||
// ShopDB monitor page reflects the box immediately instead of at next cadence.
|
||||
// "Send Status now", "Report an issue" and fleet alerts all go through the
|
||||
// SYSTEM agent's local pipe (AgentPipeClient). The agent holds the credential
|
||||
// and all collection logic: the tray never reads a key or config and never
|
||||
// duplicates wiring, it just asks the agent to act on demand so the ShopDB
|
||||
// monitor page reflects the box immediately instead of at next cadence.
|
||||
internal sealed class TrayContext : ApplicationContext
|
||||
{
|
||||
// The headless agent/service executable. The tray ships next to it in the
|
||||
@@ -25,8 +27,98 @@ internal sealed class TrayContext : ApplicationContext
|
||||
private readonly Icon brandIcon;
|
||||
private bool sending;
|
||||
|
||||
private const string AgentUnreachableText =
|
||||
"Could not reach the ShopDB Endpoint Agent. Is the ShopDB Endpoint Agent service running?";
|
||||
|
||||
private readonly AgentPipeClient pipeClient = new();
|
||||
|
||||
// Cancels in-flight pipe calls from menu handlers when the tray exits.
|
||||
private readonly CancellationTokenSource shutdownCancellation = new();
|
||||
|
||||
// Session-scoped collection. The service is LocalSystem and cannot see the
|
||||
// operator's mapped drives, user-owned listeners, or input idle time, so the
|
||||
// tray - which is already the operator - collects them and leaves the result
|
||||
// in a spool file the service reads on its own cadence.
|
||||
//
|
||||
// Deliberately longer than the telemetry cadence: this data changes rarely
|
||||
// (a mapped drive is mapped at logon and stays mapped), and each run spawns
|
||||
// a short PowerShell process on a machine an operator is actively using.
|
||||
private static readonly TimeSpan UserContextInterval = TimeSpan.FromMinutes(30);
|
||||
|
||||
// Logon is the worst moment to add work: drive mappings and profile setup
|
||||
// are still settling, and a probe fired now would capture an incomplete
|
||||
// picture while competing with everything else starting up.
|
||||
private static readonly TimeSpan UserContextStartupDelay = TimeSpan.FromMinutes(2);
|
||||
|
||||
// ...but that reasoning only applies at logon. When the tray starts into a
|
||||
// session that has been up for a while - which is what an installer restart
|
||||
// or a manual relaunch does - nothing is settling and the full delay just
|
||||
// extends the window where the server has no session data. The installer
|
||||
// kills and restarts the tray on every install, so the 2-minute delay was
|
||||
// being paid at exactly the moment someone was watching the card to see
|
||||
// whether the install had worked.
|
||||
private static readonly TimeSpan UserContextEstablishedDelay = TimeSpan.FromSeconds(15);
|
||||
|
||||
// How long the session must have been up to count as established. Comfortably
|
||||
// past the logon storm, well short of any realistic install-to-check gap.
|
||||
private static readonly TimeSpan EstablishedSessionUptime = TimeSpan.FromMinutes(5);
|
||||
|
||||
private readonly System.Windows.Forms.Timer userContextTimer;
|
||||
private bool userContextStarted;
|
||||
private bool collectingUserContext;
|
||||
|
||||
// Lets the service ask this tray to re-probe on demand, so a "Refresh
|
||||
// Status" click in the WebUI produces genuinely current session data instead
|
||||
// of whatever the last 30-minute tick happened to capture. The service
|
||||
// cannot collect this itself at any price - mapped drives live on the
|
||||
// operator's logon token - so the only way for an on-demand refresh to be
|
||||
// accurate is for the request to end up here, in the operator's session.
|
||||
private readonly UserContextRefreshChannel? refreshChannel;
|
||||
private readonly CancellationTokenSource refreshCancellation = new();
|
||||
|
||||
// How long each wait blocks before looping. The wait returns false on
|
||||
// timeout and the loop simply waits again, so this is not a deadline for
|
||||
// anything - it just bounds how long the handle stays open, which lets a
|
||||
// channel that was somehow broken get re-opened on the next pass instead of
|
||||
// leaving the tray deaf until it restarts.
|
||||
private static readonly TimeSpan RefreshWaitTimeout = TimeSpan.FromMinutes(5);
|
||||
|
||||
// The UI thread's context, captured in the constructor. The listener runs on
|
||||
// a background thread, but every probe attempt is marshalled back here so
|
||||
// that collectingUserContext is only ever read or written from one thread.
|
||||
// Without this the timer tick and a service request could both pass the
|
||||
// guard and launch two probes at the same spool file.
|
||||
private readonly SynchronizationContext? uiContext;
|
||||
|
||||
// Fleet alerts are fetched through the agent pipe and displayed to the
|
||||
// operator as toast notifications in the top-right corner. The worker polls
|
||||
// every 60 seconds, filters out already-acknowledged alerts via the store,
|
||||
// and sends acknowledgements to the agent when the operator dismisses the form.
|
||||
private readonly FleetAlertStore? fleetAlertStore;
|
||||
private readonly FleetAlertWorker? fleetAlertWorker;
|
||||
|
||||
// The tray logs into the operator's own profile
|
||||
// (C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log), NOT into the
|
||||
// service's ProgramData directory. The operator cannot write there, and the
|
||||
// file sink fails silently when it cannot create its directory, so tray-side
|
||||
// failures previously left no trace anywhere. Keeping the two apart means a
|
||||
// tray problem is diagnosable from a file the operator can actually reach.
|
||||
private readonly FileLoggerProvider logProvider;
|
||||
private readonly ILogger logger;
|
||||
|
||||
public TrayContext()
|
||||
{
|
||||
logProvider = new FileLoggerProvider(
|
||||
FileLoggerProvider.UserProfileLogDirectory(),
|
||||
filePrefix: "trayagent");
|
||||
logger = logProvider.CreateLogger("Tray");
|
||||
|
||||
var startupDelay = ResolveStartupDelay();
|
||||
logger.LogInformation(
|
||||
"Tray {Version} started; user-context probe in {Delay}, then every {Interval}.",
|
||||
typeof(TrayContext).Assembly.GetName().Version?.ToString() ?? "unknown",
|
||||
startupDelay, UserContextInterval);
|
||||
|
||||
var menu = new ContextMenuStrip();
|
||||
menu.Items.Add("Send Status now", null, OnSendStatus);
|
||||
menu.Items.Add("Report an issue...", null, OnReportIssue);
|
||||
@@ -45,6 +137,239 @@ internal sealed class TrayContext : ApplicationContext
|
||||
ContextMenuStrip = menu,
|
||||
};
|
||||
trayIcon.DoubleClick += OnSendStatus;
|
||||
|
||||
// A dead log sink cannot report itself through the log. Complaining via
|
||||
// the logger would write the one message that explains the silence into
|
||||
// the sink that is swallowing messages, so the balloon is the only
|
||||
// channel that survives. Shown once at startup and never repeated: this
|
||||
// is a "your diagnostics are blind" notice for whoever is standing at the
|
||||
// machine, not a recurring nag about something the operator cannot fix.
|
||||
if (logProvider.SinkProblem is not null)
|
||||
{
|
||||
trayIcon.ShowBalloonTip(
|
||||
10000,
|
||||
"ShopDB Endpoint Agent",
|
||||
"Tray logging is unavailable, so tray diagnostics will not be "
|
||||
+ $"recorded. {logProvider.SinkProblem}",
|
||||
ToolTipIcon.Warning);
|
||||
}
|
||||
|
||||
userContextTimer = new System.Windows.Forms.Timer
|
||||
{
|
||||
Interval = (int)startupDelay.TotalMilliseconds,
|
||||
};
|
||||
userContextTimer.Tick += OnUserContextTimer;
|
||||
userContextTimer.Start();
|
||||
|
||||
uiContext = SynchronizationContext.Current;
|
||||
|
||||
// Best-effort. A tray that cannot open the channel still probes on its
|
||||
// timer, so the failure mode is "refresh is not immediate", not "session
|
||||
// data stops working" - which is why this is caught and logged rather
|
||||
// than allowed to take the tray down at startup.
|
||||
try
|
||||
{
|
||||
refreshChannel = new UserContextRefreshChannel(
|
||||
logProvider.CreateLogger(nameof(UserContextRefreshChannel)));
|
||||
StartRefreshListener();
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception,
|
||||
"Could not open the user-context refresh channel; on-demand "
|
||||
+ "refresh requests from the service will be ignored and session "
|
||||
+ "data will only update on the {Interval} timer.",
|
||||
UserContextInterval);
|
||||
}
|
||||
|
||||
// Fleet alert worker. Best-effort: if the worker fails to start, the tray
|
||||
// still runs and the other features work. An agent that is down is not a
|
||||
// startup failure; the worker retries the pipe every cycle.
|
||||
try
|
||||
{
|
||||
fleetAlertStore = new FleetAlertStore(
|
||||
logProvider.CreateLogger(nameof(FleetAlertStore)));
|
||||
fleetAlertWorker = new FleetAlertWorker(
|
||||
pipeClient,
|
||||
fleetAlertStore,
|
||||
logProvider.CreateLogger(nameof(FleetAlertWorker)),
|
||||
uiContext);
|
||||
fleetAlertWorker.Start();
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogWarning(exception,
|
||||
"Could not start the fleet alert worker; fleet-wide alerts will not "
|
||||
+ "be displayed. Other tray features are unaffected.");
|
||||
}
|
||||
}
|
||||
|
||||
// Waits for service refresh requests for the life of the tray.
|
||||
//
|
||||
// A dedicated background thread rather than a Task: this blocks on a kernel
|
||||
// wait handle essentially forever, which is exactly the workload that should
|
||||
// not sit on a thread-pool thread. IsBackground keeps it from holding the
|
||||
// process open at exit.
|
||||
private void StartRefreshListener()
|
||||
{
|
||||
var thread = new Thread(() =>
|
||||
{
|
||||
var token = refreshCancellation.Token;
|
||||
while (!token.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (!refreshChannel!.WaitForRequest(RefreshWaitTimeout, token))
|
||||
continue;
|
||||
|
||||
logger.LogInformation(
|
||||
"The service requested a user-context refresh.");
|
||||
|
||||
// Marshalled, not awaited: the service is not waiting on a
|
||||
// reply through this channel, it is watching the spool file's
|
||||
// collectedat stamp. Our only job is to get the probe started
|
||||
// on the UI thread and go back to waiting.
|
||||
var post = uiContext;
|
||||
if (post is not null)
|
||||
post.Post(_ => _ = CollectUserContextAsync(), null);
|
||||
else
|
||||
_ = CollectUserContextAsync();
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
// Never let a bad wait kill the listener - if this thread
|
||||
// exits, on-demand refresh is silently dead for the rest of
|
||||
// the session with no other symptom.
|
||||
logger.LogWarning(exception,
|
||||
"The user-context refresh listener failed; retrying.");
|
||||
try
|
||||
{
|
||||
Thread.Sleep(TimeSpan.FromSeconds(5));
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
{
|
||||
IsBackground = true,
|
||||
Name = "ShopDB user-context refresh listener",
|
||||
};
|
||||
|
||||
thread.Start();
|
||||
}
|
||||
|
||||
// Fires once after the startup delay, then re-arms at the steady-state
|
||||
// interval. Failures are swallowed on purpose: session data is a
|
||||
// nice-to-have, and an operator must never see a dialog or a crashed tray
|
||||
// because a probe could not read a drive mapping.
|
||||
private async void OnUserContextTimer(object? sender, EventArgs e)
|
||||
{
|
||||
if (!userContextStarted)
|
||||
{
|
||||
userContextStarted = true;
|
||||
userContextTimer.Interval = (int)UserContextInterval.TotalMilliseconds;
|
||||
}
|
||||
|
||||
await CollectUserContextAsync();
|
||||
}
|
||||
|
||||
// Runs the probe once and returns when the spool has been rewritten.
|
||||
//
|
||||
// Shared by all three callers - the cadence timer, an on-demand service
|
||||
// request, and "Send Status now" - so there is exactly one place that knows
|
||||
// how to invoke the probe and how to interpret its exit codes. Returns true
|
||||
// only when a spool file is actually on disk afterwards, which is what lets
|
||||
// Send Status decide whether it is worth waiting on.
|
||||
private async Task<bool> CollectUserContextAsync()
|
||||
{
|
||||
if (collectingUserContext)
|
||||
{
|
||||
// Another probe is already in flight against the same spool path.
|
||||
// Its result is at most a second or two old and satisfies whatever
|
||||
// this caller wanted, so coalescing is correct here, not a shortcut.
|
||||
logger.LogDebug(
|
||||
"A user-context probe is already running; skipping this request.");
|
||||
return false;
|
||||
}
|
||||
|
||||
var agentExe = ResolveAgentExe();
|
||||
if (agentExe is null)
|
||||
{
|
||||
// Previously a bare return. This is the single most likely reason for
|
||||
// "no mapped drives" with nothing at all in any log, so it must say so.
|
||||
logger.LogWarning(
|
||||
"Skipping the user-context probe: {Agent} was not found beside the tray "
|
||||
+ "({Beside}) or at {Installed}.",
|
||||
AgentExeName, AppContext.BaseDirectory, DefaultInstallDir);
|
||||
return false;
|
||||
}
|
||||
|
||||
collectingUserContext = true;
|
||||
try
|
||||
{
|
||||
logger.LogInformation("Running the user-context probe: {Agent}", agentExe);
|
||||
var exitCode = await RunAgentAsync(agentExe, "--collect-usercontext");
|
||||
|
||||
// Check the disk rather than announcing what the exit code implies.
|
||||
// This line used to print a hardcoded path whenever the child exited
|
||||
// 0, so it reported "spool written to C:\Users\ShopFloor\ShopDB" for
|
||||
// weeks while the probe was actually writing into a different profile
|
||||
// and the service was finding nothing.
|
||||
var resultPath = UserContextSpool.PinnedResultPath();
|
||||
if (exitCode == 0 && File.Exists(resultPath))
|
||||
{
|
||||
logger.LogInformation(
|
||||
"The user-context probe finished; spool present at {Spool}.",
|
||||
resultPath);
|
||||
return true;
|
||||
}
|
||||
else if (exitCode == 0)
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context probe reported success but no spool file is "
|
||||
+ "present at {Spool}.", resultPath);
|
||||
}
|
||||
else if (exitCode == 2)
|
||||
{
|
||||
// The agent refused because it was elevated. Called out by name
|
||||
// because as a bare exit code it reads like a probe fault and
|
||||
// sends the reader looking for a drive or WMI problem that is
|
||||
// not there - the fix is to start the tray unelevated, or let
|
||||
// the scheduled task do the collecting.
|
||||
logger.LogWarning(
|
||||
"The user-context probe refused to run because this tray is "
|
||||
+ "ELEVATED. Mapped drives are per logon token, so an elevated "
|
||||
+ "probe would report the wrong drive table. Start the tray "
|
||||
+ "unelevated, or rely on the 'ShopDB user-context probe' "
|
||||
+ "scheduled task.");
|
||||
}
|
||||
else
|
||||
{
|
||||
logger.LogWarning(
|
||||
"The user-context probe exited with code {ExitCode}; expected a "
|
||||
+ "spool at {Spool}.", exitCode, resultPath);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
// No longer silent. The agent logs its own detail, but if the process
|
||||
// could not even be started the agent never ran and never logged.
|
||||
logger.LogWarning(exception, "Could not run the user-context probe.");
|
||||
return false;
|
||||
}
|
||||
finally
|
||||
{
|
||||
collectingUserContext = false;
|
||||
}
|
||||
}
|
||||
|
||||
private async void OnReportIssue(object? sender, EventArgs e)
|
||||
@@ -61,13 +386,7 @@ internal sealed class TrayContext : ApplicationContext
|
||||
includeScreenshot = dialog.IncludeScreenshot;
|
||||
}
|
||||
|
||||
var agentExe = ResolveAgentExe();
|
||||
if (agentExe is null)
|
||||
{
|
||||
ShowBalloon("Report an issue", "Could not find " + AgentExeName +
|
||||
". Is the ShopDB Endpoint Agent installed?", ToolTipIcon.Error);
|
||||
return;
|
||||
}
|
||||
var token = shutdownCancellation.Token;
|
||||
|
||||
// Capture AFTER the dialog closes so the report form is never in the
|
||||
// shot. Stage into the interactive operator's own profile
|
||||
@@ -97,16 +416,23 @@ internal sealed class TrayContext : ApplicationContext
|
||||
ShowBalloon("Report an issue", "Sending your report to ShopDB...",
|
||||
ToolTipIcon.Info);
|
||||
|
||||
int exitCode = await RunAgentAsync(agentExe, "--report \"" + payloadPath + "\"");
|
||||
if (exitCode == 0)
|
||||
// The agent checks the path is inside this user's own profile.
|
||||
var response = await pipeClient.SendReportAsync(payloadPath, token);
|
||||
if (response.Ok)
|
||||
ShowBalloon("Report an issue", "Your report was sent to ShopDB.",
|
||||
ToolTipIcon.Info);
|
||||
else if (response.Data is null)
|
||||
ShowBalloon("Report an issue", AgentUnreachableText, ToolTipIcon.Error);
|
||||
else
|
||||
ShowBalloon("Report an issue",
|
||||
"Report send failed (exit " + exitCode +
|
||||
"Report send failed (" + response.Error +
|
||||
"). See the agent log under ProgramData\\ShopDB.",
|
||||
ToolTipIcon.Warning);
|
||||
}
|
||||
catch (OperationCanceledException) when (token.IsCancellationRequested)
|
||||
{
|
||||
// Tray is exiting.
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
ShowBalloon("Report an issue", "Could not send the report: " +
|
||||
@@ -145,23 +471,15 @@ internal sealed class TrayContext : ApplicationContext
|
||||
}
|
||||
}
|
||||
|
||||
// Runs shopdbagent.exe --once out of process so the operator can push a
|
||||
// fresh report without waiting for the next cadence tick. The exit code is
|
||||
// the agent's own contract: 0 = reported OK, non-zero = failed (bad key,
|
||||
// host removed, network) and the detail is in the rolling agent log.
|
||||
// Asks the agent over the pipe to run telemetry once so the operator can
|
||||
// push a fresh report without waiting for the next cadence tick. The agent
|
||||
// answers ok or an error; the detail is in the rolling agent log.
|
||||
private async void OnSendStatus(object? sender, EventArgs e)
|
||||
{
|
||||
if (sending)
|
||||
return;
|
||||
|
||||
var agentExe = ResolveAgentExe();
|
||||
if (agentExe is null)
|
||||
{
|
||||
ShowBalloon("Send Status", "Could not find " + AgentExeName +
|
||||
". Is the ShopDB Endpoint Agent installed?", ToolTipIcon.Error);
|
||||
return;
|
||||
}
|
||||
|
||||
var token = shutdownCancellation.Token;
|
||||
sending = true;
|
||||
trayIcon.Text = "ShopDB Endpoint Agent (sending status...)";
|
||||
ShowBalloon("Send Status", "Sending this PC's status to ShopDB...",
|
||||
@@ -169,25 +487,45 @@ internal sealed class TrayContext : ApplicationContext
|
||||
|
||||
try
|
||||
{
|
||||
int exitCode = await RunAgentAsync(agentExe, "--once");
|
||||
if (exitCode == 0)
|
||||
// Probe BEFORE reporting, and await it. The operator clicking this
|
||||
// has just changed something - remapped a drive, signed in - and
|
||||
// expects the result to show up. sendstatus reads the spool file, so a
|
||||
// probe started after it, or not awaited, would publish the previous
|
||||
// probe's answer and the operator would click again and see the same
|
||||
// stale card.
|
||||
//
|
||||
// Not fatal if it fails: a report with older session data still
|
||||
// carries the heartbeat, inventory and events, which is most of what
|
||||
// this button is for.
|
||||
await CollectUserContextAsync();
|
||||
|
||||
var response = await pipeClient.SendStatusAsync(token);
|
||||
if (response.Ok)
|
||||
ShowBalloon("Send Status", "Status sent to ShopDB.",
|
||||
ToolTipIcon.Info);
|
||||
else if (response.Data is null)
|
||||
ShowBalloon("Send Status", AgentUnreachableText, ToolTipIcon.Error);
|
||||
else
|
||||
ShowBalloon("Send Status",
|
||||
"Status send failed (exit " + exitCode +
|
||||
"Status send failed (" + response.Error +
|
||||
"). See the agent log under ProgramData\\ShopDB.",
|
||||
ToolTipIcon.Warning);
|
||||
}
|
||||
catch (OperationCanceledException) when (token.IsCancellationRequested)
|
||||
{
|
||||
// Tray is exiting.
|
||||
return;
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
ShowBalloon("Send Status", "Could not run the agent: " +
|
||||
ShowBalloon("Send Status", "Could not reach the agent: " +
|
||||
error.Message, ToolTipIcon.Error);
|
||||
}
|
||||
finally
|
||||
{
|
||||
sending = false;
|
||||
trayIcon.Text = "ShopDB Endpoint Agent";
|
||||
if (!token.IsCancellationRequested)
|
||||
trayIcon.Text = "ShopDB Endpoint Agent";
|
||||
}
|
||||
}
|
||||
|
||||
@@ -249,6 +587,35 @@ internal sealed class TrayContext : ApplicationContext
|
||||
startInfo.Environment.Remove(key);
|
||||
}
|
||||
|
||||
// Full delay at logon, short delay when the session is already established.
|
||||
//
|
||||
// Environment.TickCount64 is machine uptime, not session uptime, which is a
|
||||
// deliberate simplification: the two differ only when someone signs out and
|
||||
// back in without rebooting, and treating that as "established" is harmless -
|
||||
// the profile is already built and the drives are already mapped in that
|
||||
// case too. A failure to read it falls back to the conservative long delay.
|
||||
private TimeSpan ResolveStartupDelay()
|
||||
{
|
||||
try
|
||||
{
|
||||
var uptime = TimeSpan.FromMilliseconds(Environment.TickCount64);
|
||||
if (uptime >= EstablishedSessionUptime)
|
||||
{
|
||||
logger.LogInformation(
|
||||
"Session has been up {Uptime}; using the established-session probe delay.",
|
||||
uptime);
|
||||
return UserContextEstablishedDelay;
|
||||
}
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
logger.LogDebug(exception,
|
||||
"Could not read system uptime; using the logon probe delay.");
|
||||
}
|
||||
|
||||
return UserContextStartupDelay;
|
||||
}
|
||||
|
||||
// Same directory first (the installer ships the tray next to the agent),
|
||||
// then the known Program Files install directory as a fallback.
|
||||
private static string? ResolveAgentExe()
|
||||
@@ -316,6 +683,19 @@ internal sealed class TrayContext : ApplicationContext
|
||||
{
|
||||
if (disposing)
|
||||
{
|
||||
userContextTimer.Stop();
|
||||
userContextTimer.Dispose();
|
||||
|
||||
// Cancel and let the listener unwind on its own. The channel opens
|
||||
// and closes its handle inside each wait rather than holding one, so
|
||||
// there is nothing here to dispose.
|
||||
refreshCancellation.Cancel();
|
||||
refreshCancellation.Dispose();
|
||||
|
||||
shutdownCancellation.Cancel();
|
||||
fleetAlertWorker?.Dispose();
|
||||
shutdownCancellation.Dispose();
|
||||
|
||||
trayIcon.Dispose();
|
||||
brandIcon.Dispose();
|
||||
}
|
||||
|
||||
@@ -11,26 +11,156 @@ using ShopDbEndpointAgent.Core;
|
||||
// --console run interactively in the foreground for debugging
|
||||
// --once collect and report exactly once, then exit (smoke test / cron)
|
||||
// --report <path> upload one operator issue report (JSON sidecar written by
|
||||
// the tray), then exit. Keeps the collector key in the
|
||||
// service so the tray never holds a secret.
|
||||
// the tray), then exit.
|
||||
// Not running as SYSTEM, --once and --report are pipe clients: they ask the
|
||||
// running service (AgentPipeServer) to do the work and never load a secret.
|
||||
//
|
||||
// Secrets are never read here: AgentConfigProvider layers the registry
|
||||
// (HKLM\SOFTWARE\GE\ShopDB: CollectorKey, BaseUrl) over the shipped, non-secret
|
||||
// appsettings.json. The executable is intentionally named shopdbagent.exe so it
|
||||
// appsettings.json, and the per-PC credential lives in CredentialStore. The executable is intentionally named shopdbagent.exe so it
|
||||
// never collides with third-party agent.exe processes/services (for example the
|
||||
// MTConnect agent this same build can monitor).
|
||||
|
||||
var runOnce = args.Any(a => string.Equals(a, "--once", StringComparison.OrdinalIgnoreCase));
|
||||
var forceConsole = args.Any(a => string.Equals(a, "--console", StringComparison.OrdinalIgnoreCase));
|
||||
var collectUserContext = args.Any(a =>
|
||||
string.Equals(a, "--collect-usercontext", StringComparison.OrdinalIgnoreCase));
|
||||
var reportPayloadPath = ArgValue(args, "--report");
|
||||
|
||||
var agentVersion = Assembly.GetExecutingAssembly()
|
||||
.GetName().Version?.ToString() ?? "0.0.0";
|
||||
|
||||
// FIRST, ahead of everything else. This verb runs as the signed-in operator (the
|
||||
// tray invokes it) and must not touch a single privileged resource on the way in.
|
||||
//
|
||||
// It used to sit below the bootstrap logger, the fault monitor, and the config
|
||||
// load, which meant the operator-context process opened the SYSTEM-owned
|
||||
// ProgramData log directory, attempted an EventLog source registration that only
|
||||
// an administrator can perform, and read CollectorKey out of HKLM - all before it
|
||||
// discovered it was only being asked to write a file into the user's own profile.
|
||||
// On an EPM-managed host (CyberArk) those are exactly the calls a broker
|
||||
// intercepts, and the registry read put the collector secret in a process that
|
||||
// has no business holding it.
|
||||
//
|
||||
// It also has no config dependency by design: it only writes into the caller's
|
||||
// own profile and never talks to the server, so it must still work on a machine
|
||||
// whose CollectorKey has not been provisioned yet - otherwise session data would
|
||||
// be missing on precisely the endpoints that are hardest to diagnose.
|
||||
if (collectUserContext)
|
||||
{
|
||||
// First statement in this branch. The scheduled task runs this exe directly
|
||||
// in the operator's interactive session, and shopdbagent.exe is a console
|
||||
// app, so without this a black window flashes up on every collection cycle.
|
||||
// Done before any logging or probing so the window is gone immediately
|
||||
// rather than lingering for the life of the probe.
|
||||
ConsoleWindow.Hide();
|
||||
|
||||
// Logs to the OPERATOR's profile, not the service's ProgramData directory.
|
||||
// A non-privileged user cannot write to the SYSTEM-owned log directory, and
|
||||
// the file sink degrades to a no-op when it cannot create its directory, so
|
||||
// pointing this at the service location meant every probe failure was
|
||||
// discarded unread - which is exactly how a broken probe looked identical to
|
||||
// a probe that was never invoked.
|
||||
using var probeLogProvider = new FileLoggerProvider(
|
||||
FileLoggerProvider.UserProfileLogDirectory(),
|
||||
filePrefix: "trayagent");
|
||||
using var probeLoggerFactory = LoggerFactory.Create(
|
||||
logging => logging.AddProvider(probeLogProvider));
|
||||
|
||||
var probeLogger = probeLoggerFactory.CreateLogger<UserContextProbe>();
|
||||
ReportSinkProblem(probeLogProvider, "user-context probe");
|
||||
|
||||
// Refuse to collect from an elevated process. Drive letters are a property
|
||||
// of the logon token, not the machine: an elevated process gets a different
|
||||
// token to the operator's interactive one and enumerates a DIFFERENT, almost
|
||||
// always empty, drive table. Measured on a live endpoint, the same two
|
||||
// shares reported Unavailable from the elevated shell and OK from the
|
||||
// operator's, in both Win32_NetworkConnection and Get-SmbMapping.
|
||||
//
|
||||
// Refusing here rather than filtering later is deliberate. An elevated probe
|
||||
// does not fail - it succeeds and reports every mapped drive as broken,
|
||||
// which is indistinguishable from a genuine outage at the far end. Better to
|
||||
// write nothing and say why than to publish a confident wrong answer.
|
||||
if (IsProcessElevated())
|
||||
{
|
||||
probeLogger.LogError(
|
||||
"Refusing to collect user context: this process is elevated. Mapped drives " +
|
||||
"are per-logon-token, so an elevated probe reports the wrong drive table. " +
|
||||
"Run the '{Task}' scheduled task (RunLevel Limited) instead of invoking " +
|
||||
"--collect-usercontext from an admin shell.",
|
||||
"ShopDB user-context probe");
|
||||
Console.Error.WriteLine(
|
||||
"shopdbagent: refusing to collect user context from an elevated process; " +
|
||||
"mapped drives would be wrong. Use the scheduled task.");
|
||||
return 2;
|
||||
}
|
||||
|
||||
var probe = new UserContextProbe(probeLogger);
|
||||
return probe.Collect() ? 0 : 1;
|
||||
}
|
||||
|
||||
// True when the current process holds the elevated (high-integrity) token.
|
||||
static bool IsProcessElevated()
|
||||
{
|
||||
if (!OperatingSystem.IsWindows())
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
using var identity = System.Security.Principal.WindowsIdentity.GetCurrent();
|
||||
var principal = new System.Security.Principal.WindowsPrincipal(identity);
|
||||
return principal.IsInRole(System.Security.Principal.WindowsBuiltInRole.Administrator);
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Cannot tell. Treat as not elevated so an unreadable token does not
|
||||
// suppress collection on a host where the probe would have worked; the
|
||||
// spool's own account stamp is still checked on the way back in.
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Not SYSTEM: a user (the tray, or an operator at a prompt) cannot read the
|
||||
// credential or the key, so hand the request to the service over the pipe.
|
||||
// Ahead of the bootstrap logger for the same reason as the probe above.
|
||||
if ((runOnce || !string.IsNullOrWhiteSpace(reportPayloadPath)) && !IsRunningAsSystem())
|
||||
return await RunPipeClientAsync(runOnce, reportPayloadPath);
|
||||
|
||||
static bool IsRunningAsSystem()
|
||||
{
|
||||
if (!OperatingSystem.IsWindows())
|
||||
return true;
|
||||
try
|
||||
{
|
||||
using var identity = System.Security.Principal.WindowsIdentity.GetCurrent();
|
||||
return identity.IsSystem;
|
||||
}
|
||||
catch
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// Exit code is the tray's contract: 0 = done, non-zero = failed (reason on stderr).
|
||||
static async Task<int> RunPipeClientAsync(bool runOnce, string? reportPath)
|
||||
{
|
||||
var client = new AgentPipeClient();
|
||||
var response = runOnce
|
||||
? await client.SendStatusAsync(CancellationToken.None)
|
||||
: await client.SendReportAsync(Path.GetFullPath(reportPath!), CancellationToken.None);
|
||||
if (response.Ok)
|
||||
return 0;
|
||||
Console.Error.WriteLine("shopdbagent: " + response.Error);
|
||||
return response.Data is null ? 3 : 1;
|
||||
}
|
||||
|
||||
// Bootstrap logger first so config loading (including the registry read that
|
||||
// CyberArk/EPM may gate) is fully diagnosable before the host starts.
|
||||
using var bootstrapFileLogger = new FileLoggerProvider(FileLoggerProvider.DefaultDirectory());
|
||||
var bootstrapLogger = bootstrapFileLogger.CreateLogger("Startup");
|
||||
ReportSinkProblem(bootstrapFileLogger, "service");
|
||||
|
||||
var faultMonitor = new AgentFaultMonitor(bootstrapLogger);
|
||||
faultMonitor.Install();
|
||||
@@ -41,15 +171,24 @@ var logDirectory = string.IsNullOrWhiteSpace(config.LogPath)
|
||||
? FileLoggerProvider.DefaultDirectory()
|
||||
: config.LogPath;
|
||||
|
||||
if (!config.IsUsable())
|
||||
// BaseUrl is the only hard requirement: an enrolled PC runs without the shared
|
||||
// key, and an unenrolled one can still request approval without it.
|
||||
if (string.IsNullOrWhiteSpace(config.BaseUrl))
|
||||
{
|
||||
bootstrapLogger.LogError(
|
||||
"Config incomplete: provision CollectorKey and BaseUrl in " +
|
||||
"HKLM\\SOFTWARE\\GE\\ShopDB (or baseurl/apikey in appsettings.json for dev). " +
|
||||
"Config incomplete: provision BaseUrl in " +
|
||||
"HKLM\\SOFTWARE\\GE\\ShopDB (or baseurl in appsettings.json for dev). " +
|
||||
"See the registry lines above to tell 'missing' from 'access denied'.");
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (!config.HasSharedKey())
|
||||
{
|
||||
bootstrapLogger.LogWarning(
|
||||
"No CollectorKey provisioned. Fine once this PC is enrolled; until then " +
|
||||
"requests are unauthenticated and enrollment needs admin approval.");
|
||||
}
|
||||
|
||||
var baseUrlProblem = config.BaseUrlProblem();
|
||||
if (baseUrlProblem is not null)
|
||||
{
|
||||
@@ -65,8 +204,7 @@ try
|
||||
if (!string.IsNullOrWhiteSpace(reportPayloadPath))
|
||||
return await RunReportAsync(config, reportPayloadPath, logDirectory, faultMonitor);
|
||||
|
||||
await RunHostAsync(config, agentVersion, logDirectory, forceConsole);
|
||||
return 0;
|
||||
return await RunHostAsync(config, agentVersion, logDirectory, forceConsole);
|
||||
}
|
||||
catch (Exception error)
|
||||
{
|
||||
@@ -74,7 +212,7 @@ catch (Exception error)
|
||||
return 1;
|
||||
}
|
||||
|
||||
static async Task RunHostAsync(AgentConfig config, string agentVersion,
|
||||
static async Task<int> RunHostAsync(AgentConfig config, string agentVersion,
|
||||
string logDirectory, bool forceConsole)
|
||||
{
|
||||
var builder = Host.CreateApplicationBuilder();
|
||||
@@ -85,10 +223,44 @@ static async Task RunHostAsync(AgentConfig config, string agentVersion,
|
||||
|
||||
ConfigureLogging(builder.Logging, logDirectory);
|
||||
RegisterCoreServices(builder.Services, config, agentVersion);
|
||||
|
||||
// Enrollment is registered (and so started) first; the workers wait briefly
|
||||
// for its first attempt so an enrollable PC reports on its own credential.
|
||||
builder.Services.AddSingleton(sp => new EnrollmentService(
|
||||
config,
|
||||
sp.GetRequiredService<AgentHttpFactory>(),
|
||||
sp.GetRequiredService<AgentCredentialProvider>(),
|
||||
sp.GetRequiredService<CredentialStore>(),
|
||||
agentVersion,
|
||||
sp.GetRequiredService<ILogger<EnrollmentService>>()));
|
||||
builder.Services.AddSingleton(sp => new AgentPipeServer(
|
||||
config,
|
||||
sp.GetRequiredService<TelemetryWorker>(),
|
||||
sp.GetRequiredService<CollectorClient>(),
|
||||
sp.GetRequiredService<FleetAlertClient>(),
|
||||
sp.GetRequiredService<ICredentialProvider>(),
|
||||
agentVersion,
|
||||
sp.GetRequiredService<ILogger<AgentPipeServer>>()));
|
||||
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<EnrollmentService>());
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<TelemetryWorker>());
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<CommandWorker>());
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<AgentPipeServer>());
|
||||
|
||||
await builder.Build().RunAsync();
|
||||
var host = builder.Build();
|
||||
|
||||
// A newly issued credential reports straight away rather than at the next
|
||||
// cadence tick, so the server sees the PC switch over.
|
||||
var credentials = host.Services.GetRequiredService<AgentCredentialProvider>();
|
||||
var collectSignal = host.Services.GetRequiredService<CollectSignal>();
|
||||
credentials.Changed += (_, _) =>
|
||||
{
|
||||
if (credentials.Current is not null)
|
||||
collectSignal.Request();
|
||||
};
|
||||
|
||||
await host.RunAsync();
|
||||
return 0;
|
||||
}
|
||||
|
||||
static async Task<int> RunOnceAsync(AgentConfig config, string agentVersion,
|
||||
@@ -111,8 +283,8 @@ static async Task<int> RunOnceAsync(AgentConfig config, string agentVersion,
|
||||
}
|
||||
|
||||
// Uploads one operator issue report described by a tray-written JSON sidecar,
|
||||
// then deletes the payload folder. Runs out of process from the tray so the
|
||||
// collector key stays in the service binary. The exit code is the tray's
|
||||
// then deletes the sidecar. SYSTEM only; a user-context --report goes through
|
||||
// the pipe (RunPipeClientAsync) and the service validates the path. The exit code is the tray's
|
||||
// contract: 0 = uploaded, non-zero = failed (detail in the agent log).
|
||||
static async Task<int> RunReportAsync(AgentConfig config, string payloadPath,
|
||||
string logDirectory, AgentFaultMonitor faultMonitor)
|
||||
@@ -191,13 +363,60 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config
|
||||
sp.GetRequiredService<ILogger<PowerShellUtilityRunner>>()));
|
||||
}
|
||||
|
||||
// Persisted per-channel event watermark so each Windows event record is
|
||||
// reported exactly once, instead of re-sending the newest window of every
|
||||
// channel on every cadence tick.
|
||||
services.AddSingleton(sp => new EventWatermarkStore(
|
||||
sp.GetRequiredService<ILogger<EventWatermarkStore>>()));
|
||||
|
||||
// Session-scoped collection. The agent is LocalSystem, so mapped drives and
|
||||
// user-owned listeners are invisible to it. The tray, which already runs as
|
||||
// the operator, writes the spool file; the service only reads it.
|
||||
//
|
||||
// UserContextProbe is deliberately NOT registered here. The service must
|
||||
// never run the probe: descending into the session with CreateProcessAsUser
|
||||
// is the reverted design, and leaving a resolvable dependency in the
|
||||
// container is an invitation to reintroduce it. The probe is constructed
|
||||
// only on the --collect-usercontext path, which the tray invokes.
|
||||
services.AddSingleton(sp => new UserContextSpool(
|
||||
sp.GetRequiredService<ILogger<UserContextSpool>>()));
|
||||
|
||||
// The ask-the-tray side of the same split. Registering this is NOT a
|
||||
// loophole around the paragraph above: the service still cannot collect
|
||||
// session data, it can only request that the process which can does so, and
|
||||
// then read the file that appears. The probe stays out of this container.
|
||||
services.AddSingleton(sp => new UserContextRefreshChannel(
|
||||
sp.GetRequiredService<ILogger<UserContextRefreshChannel>>()));
|
||||
|
||||
services.AddSingleton<ICollector>(sp => new EndpointCollector(
|
||||
sp.GetRequiredService<ILogger<EndpointCollector>>(),
|
||||
config,
|
||||
sp.GetService<IAgentUtilityRunner>()));
|
||||
sp.GetService<IAgentUtilityRunner>(),
|
||||
sp.GetRequiredService<EventWatermarkStore>(),
|
||||
sp.GetRequiredService<UserContextSpool>()));
|
||||
|
||||
// Per-PC credential (if enrolled) or shared key, swapped at runtime. One
|
||||
// factory builds every HttpClient so pins and auth cannot be skipped.
|
||||
services.AddSingleton(sp => new CredentialStore(
|
||||
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Credentials")));
|
||||
services.AddSingleton(sp => new AgentCredentialProvider(
|
||||
config,
|
||||
sp.GetRequiredService<CredentialStore>(),
|
||||
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Credentials")));
|
||||
services.AddSingleton<ICredentialProvider>(sp =>
|
||||
sp.GetRequiredService<AgentCredentialProvider>());
|
||||
services.AddSingleton(sp => new AgentHttpFactory(
|
||||
config,
|
||||
sp.GetRequiredService<ICredentialProvider>(),
|
||||
sp.GetRequiredService<ILoggerFactory>().CreateLogger("Http")));
|
||||
|
||||
services.AddSingleton(sp => new CollectorClient(
|
||||
config, sp.GetRequiredService<ILogger<CollectorClient>>()));
|
||||
sp.GetRequiredService<AgentHttpFactory>(),
|
||||
sp.GetRequiredService<ILogger<CollectorClient>>()));
|
||||
|
||||
services.AddSingleton(sp => new FleetAlertClient(
|
||||
sp.GetRequiredService<AgentHttpFactory>(),
|
||||
sp.GetRequiredService<ILoggerFactory>().CreateLogger(nameof(FleetAlertClient))));
|
||||
|
||||
services.AddSingleton(sp => new TelemetryWorker(
|
||||
sp.GetRequiredService<ICollector>(),
|
||||
@@ -206,23 +425,30 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config
|
||||
agentVersion,
|
||||
sp.GetRequiredService<ILogger<TelemetryWorker>>(),
|
||||
sp.GetService<IAgentUtilityRunner>(),
|
||||
sp.GetRequiredService<CollectSignal>()));
|
||||
sp.GetRequiredService<CollectSignal>(),
|
||||
sp.GetRequiredService<UserContextSpool>(),
|
||||
sp.GetRequiredService<UserContextRefreshChannel>(),
|
||||
sp.GetService<EnrollmentService>()));
|
||||
|
||||
// Command channel: outbound long-poll for allowlisted WebUI commands, run
|
||||
// through the fail-closed executor. Hosted alongside telemetry but on its
|
||||
// own loop so a held-open poll never delays a report.
|
||||
services.AddSingleton(sp => new CommandClient(
|
||||
config, sp.GetRequiredService<ILogger<CommandClient>>()));
|
||||
sp.GetRequiredService<AgentHttpFactory>(),
|
||||
sp.GetRequiredService<ILogger<CommandClient>>()));
|
||||
|
||||
services.AddSingleton(sp => new CommandExecutor(
|
||||
sp.GetRequiredService<ILogger<CommandExecutor>>(),
|
||||
config));
|
||||
config,
|
||||
sp.GetRequiredService<AgentHttpFactory>()));
|
||||
|
||||
services.AddSingleton(sp => new CommandWorker(
|
||||
sp.GetRequiredService<CommandClient>(),
|
||||
sp.GetRequiredService<CommandExecutor>(),
|
||||
sp.GetRequiredService<ILogger<CommandWorker>>(),
|
||||
sp.GetRequiredService<CollectSignal>()));
|
||||
sp.GetRequiredService<CollectSignal>(),
|
||||
config,
|
||||
sp.GetService<EnrollmentService>()));
|
||||
}
|
||||
|
||||
static void ConfigureLogging(ILoggingBuilder logging, string logDirectory)
|
||||
@@ -231,3 +457,22 @@ static void ConfigureLogging(ILoggingBuilder logging, string logDirectory)
|
||||
logging.AddProvider(new FileLoggerProvider(logDirectory));
|
||||
logging.SetMinimumLevel(LogLevel.Information);
|
||||
}
|
||||
|
||||
// Say so, loudly, when a file sink could not open its directory.
|
||||
//
|
||||
// FileLoggerProvider never throws on construction and silently drops lines when
|
||||
// its directory is unusable. That is the right behaviour - logging must not take
|
||||
// the agent down - but it made two different failures indistinguishable: a probe
|
||||
// that ran and failed, and a probe whose log had nowhere to go. The provider
|
||||
// records the reason; this is the part that was missing, the caller actually
|
||||
// asking for it. stderr is the only sink guaranteed to exist at this point, so
|
||||
// that is where the complaint goes.
|
||||
static void ReportSinkProblem(FileLoggerProvider provider, string which)
|
||||
{
|
||||
if (provider.SinkProblem is null)
|
||||
return;
|
||||
|
||||
Console.Error.WriteLine(
|
||||
$"shopdbagent: the {which} log sink is unusable and log lines will be "
|
||||
+ $"dropped - {provider.SinkProblem}");
|
||||
}
|
||||
|
||||
@@ -7,7 +7,88 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<RootNamespace>ShopDbEndpointAgent</RootNamespace>
|
||||
<AssemblyName>shopdbagent</AssemblyName>
|
||||
<!-- 1.6.3: operator issue reports stage under the interactive operator's
|
||||
<!-- 1.10.1: the collect-usercontext verb hides its console window. The agent
|
||||
is a console exe, which the tray used to mask by launching the probe with
|
||||
CreateNoWindow; the scheduled task runs the exe directly in the operator's
|
||||
session, so a black window appeared on every 30-minute collection. -->
|
||||
<!-- 1.10.0: user context is collected by a scheduled task in the operator's
|
||||
own logon session, not by a process the elevated installer launches.
|
||||
Drive letters are per logon token: an elevated probe enumerates a
|
||||
different, usually empty, drive table, so an install performed by an
|
||||
admin produced a spool reporting every mapped drive as Unavailable while
|
||||
the operator's own session showed OK. The installer no longer starts the
|
||||
tray via explorer.exe (which de-elevates but does NOT change account) and
|
||||
instead registers 'ShopDB user-context probe' against BUILTIN\Users at
|
||||
RunLevel Limited, verifies it by reading the task back, and records the
|
||||
outcome in the install ledger. The collect-usercontext verb now refuses
|
||||
to run elevated (exit 2) rather than writing a confident wrong answer,
|
||||
the probe stamps its token elevation, and the service rejects any spool
|
||||
stamped elevated. Probe runs log the drive and listener counts. -->
|
||||
<!-- 1.9.0: the spool is finally written where the service looks for it. The
|
||||
probe script resolved its output directory from $env:USERPROFILE while
|
||||
the reader used the pinned C:\Users\ShopFloor path; under a brokered or
|
||||
re-parented tray launch those differ, so the probe reported success and
|
||||
wrote the file somewhere the service never read. Both sides now derive
|
||||
the path from one helper (UserContextSpool.PinnedResultPath). Collect()
|
||||
verifies the file exists on disk before reporting success, because a
|
||||
zero exit code from powershell.exe proved nothing about the write.
|
||||
Minor rather than patch because the spool handoff itself changed shape
|
||||
(AGENT-CONTRACT.md 5): the spool is now PERSISTED and overwritten each
|
||||
run rather than consumed on read. The collector cache is in-memory only,
|
||||
so consume-on-read plus a service restart blanked mapped drives until the
|
||||
next tray cycle. Max age tightened 24h -> 90m now that staleness, not
|
||||
deletion, is the retirement mechanism. A spool whose username stamp is
|
||||
absent or not ShopFloor is rejected outright - wrong-session drive data
|
||||
is worse than none.
|
||||
The collect-usercontext branch moved to the top of Program.cs: it used to
|
||||
open the SYSTEM-owned log directory, register an EventLog source, and
|
||||
read CollectorKey out of HKLM before discovering it only needed to write
|
||||
a file into the caller's own profile - all as the operator, and all
|
||||
exactly what an EPM broker intercepts. Log filenames unified on
|
||||
<date>-<component>.log across service, tray, and installer. A log sink
|
||||
that cannot open its directory now says so on stderr instead of
|
||||
discarding every line unread. Pairs with tray 1.2.0. -->
|
||||
<!-- 1.8.1: user-context logging and spool location are split from the
|
||||
service's. The probe runs as the signed-in operator but logged into
|
||||
%ProgramData%\ShopDB\EndpointAgent\logs, which is SYSTEM-owned and not
|
||||
writable from a user session; the file sink swallows that failure, so a
|
||||
probe that failed and a probe that never ran looked identical - no log
|
||||
line either way. The probe and the tray now log to
|
||||
C:\Users\ShopFloor\ShopDB\Logs\<datetime>-trayagent.log, which the
|
||||
operator owns. The spool path is pinned to C:\Users\ShopFloor\ShopDB
|
||||
instead of being resolved through WTSQueryUserToken (a privileged call
|
||||
endpoint protection blocks, after which writer and reader disagreed
|
||||
silently about the location). The service now CONSUMES the spool -
|
||||
removing it after a successful parse - so a dead tray stops re-reporting
|
||||
stale drives instead of pinning them forever. The probe itself is
|
||||
launched with -EncodedCommand rather than a staged .ps1 run via -File:
|
||||
a quarantined or missing temp script makes powershell.exe print usage
|
||||
and exit -196608 without running a statement. -->
|
||||
<!-- 1.8.0: user-context collection is inverted. The service no longer
|
||||
descends into the operator's session with CreateProcessAsUser to run the
|
||||
probe; that path needs to duplicate a token, is what endpoint protection
|
||||
watches for, and produced no spool file at all on a live endpoint. The
|
||||
tray, which is already running as the operator, now invokes
|
||||
shopdbagent.exe with the collect-usercontext verb on its own cadence and
|
||||
the service only reads the spool. Requires the 1.1.0 tray: an older tray
|
||||
the new verb, so usercontext stays null until the tray is updated.
|
||||
Spool freshness moved from delete-around-read to the in-file collectedat
|
||||
stamp with a max age, because writer and reader are now independent
|
||||
processes and a delete-then-read would race the tray and discard good
|
||||
results. -->
|
||||
<!-- 1.7.0: curated events are harvested against a persisted per-channel
|
||||
Windows record-id watermark and carry eventrecordid, so one real event
|
||||
is reported and stored exactly once instead of once per cadence tick
|
||||
for as long as it stayed in the newest-25 window. Adds WER 1000/1002,
|
||||
event XML fields, listening ports, an on-demand network_connections
|
||||
command, pendingreboot, disk freepercent, and populates cpupercent/
|
||||
memorypercent/idleseconds (previously declared but always null).
|
||||
Session-scoped data (mapped drives, user-owned listeners, idle time)
|
||||
comes from a probe run in the interactive user's session via a spool
|
||||
file, because none of it exists in the agent's LocalSystem session 0.
|
||||
run_script user-context waits are now bounded; the previous Infinite
|
||||
wait could pin the command worker on a hung script.
|
||||
1.6.3: operator issue reports stage under the interactive operator's
|
||||
%USERPROFILE%\ShopDBScreenshots (resolved in the ShopFloor tray session)
|
||||
instead of %TEMP%, which on locked-down endpoints redirected to another
|
||||
user's AppData\Local\Temp and left the SYSTEM-context report upload
|
||||
@@ -50,7 +131,171 @@
|
||||
(AgentConfig.LaunchPaths, shipped defaults overridable per site).
|
||||
1.1.0: generic, data-driven service_*/process_* actions carry their
|
||||
target over the wire, so new policies/apps need no agent rebuild. -->
|
||||
<Version>1.6.3</Version>
|
||||
<!-- 1.13.0: run_script honors a per-script working directory. The script
|
||||
library gains an optional workingdirectory, the server carries it in the
|
||||
run_script descriptor, and ScriptDeliveryClaim gains WorkingDirectory
|
||||
(AGENT-CONTRACT.md). The point is corporate tooling shipped by Intune: a
|
||||
script that shells out to a vendor CLI needs to run beside that CLI, not
|
||||
in the staging folder. Empty means the existing staging path, so older
|
||||
packages and every existing script keep their current behavior - this is
|
||||
additive, hence minor. Fails closed: a named directory that does not
|
||||
exist on the endpoint aborts the run with a clear error rather than
|
||||
silently falling back to staging, because a script that ran in the wrong
|
||||
folder is worse than one that did not run. The directory confers no
|
||||
rights of its own - it is a cwd, not a grant, and the run still happens
|
||||
in the script's configured context. Server-side validation rejects UNC
|
||||
paths and traversal before the value is ever persisted. No tray change -
|
||||
tray stays 1.4.0. -->
|
||||
<!-- 1.12.0: startup jitter before first contact. Both loops previously made
|
||||
their first request the instant the service started, and the service is
|
||||
installed start=auto, so a patch reboot put the whole fleet on the wire
|
||||
together - twice per machine (a telemetry report AND a command poll).
|
||||
jitterseconds did not help: it spreads the gap BETWEEN runs, not first
|
||||
contact. A 700-endpoint burst drove TLS handshakes to 1482ms worst case
|
||||
and pinned CPU at 305% while the database sat idle at 3 active queries;
|
||||
spreading the same arrivals over 30s cut worst-case handshake to 63.7ms
|
||||
with no server-side change. New startupjitterseconds (default 30, 0
|
||||
disables) is applied by StartupJitter.Next in both TelemetryWorker and
|
||||
CommandWorker. Randomized in milliseconds rather than whole seconds
|
||||
because a 30s window at second granularity re-clumps a large fleet into
|
||||
30 buckets of simultaneous arrivals. Clamped to 300s and warned about
|
||||
when out of range: the command channel is how an operator would correct
|
||||
a bad value, so an unbounded delay would remove the means of recovery.
|
||||
Costs up to startupjitterseconds of extra latency before a rebooted
|
||||
endpoint picks up a queued command, on top of a reboot the machine was
|
||||
already unreachable for. The once-only console switch bypasses the delay
|
||||
entirely (it calls RunOnceAsync directly), so smoke tests stay instant.
|
||||
appsettings schema gains a key; older packages without it get the
|
||||
AgentConfig default. No tray change - tray stays 1.4.0. -->
|
||||
<!-- 1.11.0: on-demand user-context refresh. "Refresh Status" (collect_now)
|
||||
and the tray's "Send Status" re-sent whatever was already in the spool,
|
||||
so a drive the operator had just unmapped kept reporting OK until the
|
||||
next 30-minute probe and no amount of clicking changed it. The service
|
||||
still does not - and must not - collect session data itself; it signals
|
||||
Global\ShopDbEndpointAgent.UserContextRefresh and waits up to 12s for the
|
||||
spool's collectedat stamp to ADVANCE before collecting, which is the only
|
||||
proof the file in front of it answers this request and not the previous
|
||||
one. The wait is bounded and non-fatal by design: telemetry is the
|
||||
liveness signal, so a timeout reports slightly stale session data rather
|
||||
than skipping the report. UserContextSpool.TryReadCollectedAt reads the
|
||||
stamp outside the usability gates, so a spool rejected as expired or
|
||||
wrong-account does not burn the full timeout every cycle. Minor because
|
||||
the spool handoff gains a second channel (AGENT-CONTRACT.md 5) and
|
||||
because the installer now registers the probe task to the ShopFloor USER
|
||||
rather than BUILTIN\Users: a group-principal task cannot be started by a
|
||||
member of that group, so Start-ScheduledTask returned access denied and
|
||||
the task only ever ran on its own trigger. Requires tray 1.4.0 - the tray
|
||||
is the side that listens, so an older tray simply never answers. -->
|
||||
<!-- 1.13.1: coordinated release with tray 1.6.0 (fleet alert styling update).
|
||||
No agent code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.13.2: coordinated release with tray 1.6.3 (auto-close when inactive, crash fix).
|
||||
No agent code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.13.3: coordinated release with tray 1.6.4 (deadlock fix).
|
||||
No agent code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.13.4: coordinated release with tray 1.6.5 (resource leak fixes).
|
||||
No agent code changes, but deployed together for version consistency. -->
|
||||
<!-- 1.14.0: truthful script verdicts and script output capture. A run_script
|
||||
command was judged solely on PowerShell's exit code, and PowerShell
|
||||
treats most errors as non-terminating: it writes the error, runs on, and
|
||||
still exits zero. An account-provisioning script therefore reported
|
||||
succeeded having provisioned nothing, and because no output was captured
|
||||
there was no evidence to contradict it. The claim now carries errormode
|
||||
(AGENT-CONTRACT.md 7.4); strict prepends $ErrorActionPreference='Stop'
|
||||
AFTER digest verification so the hash still validates delivered bytes.
|
||||
Default stays permissive so bodies authored against the old semantics
|
||||
keep working. stdout/stderr is captured on both paths - redirected
|
||||
streams for SYSTEM, a transcript file under the ShopFloor profile for
|
||||
user-session runs, since CreateProcessAsUser cannot inherit our pipes
|
||||
across the session boundary - always logged locally and returned to the
|
||||
server only on failure. Minor because the claim gains a field and an
|
||||
older agent ignoring it keeps today's behaviour. Supersedes the 1.13.1
|
||||
to 1.13.4 tray-coordination bumps merged from wj, which carried no agent
|
||||
code changes. Tray is at 1.6.5 from wj. -->
|
||||
<!-- 1.15.0: successful runs can now return their output to the server, gated
|
||||
by a per-script captureoutput flag on the claim. Failures still always
|
||||
return output; success previously never did, which left a green badge as
|
||||
the only evidence a run produced and no way to tell "did the work" from
|
||||
"PowerShell exited cleanly". Opt-in rather than always-on because output
|
||||
carries whatever the body printed, and that is not worth persisting
|
||||
server-side for every routine run. Minor because the claim gains a field
|
||||
that defaults false: an agent or server too old to know it keeps today's
|
||||
withholding behaviour, which is the safe direction to fail. -->
|
||||
<!-- 1.16.0: adds collect_log, which reads a plaintext log file from a
|
||||
caller-supplied path and returns its contents for the Log Collector tab.
|
||||
Strictly read-only: opened for reading with FileShare.ReadWrite|Delete,
|
||||
because the logs worth collecting are the ones a running application
|
||||
currently holds open and a stricter share mode fails on exactly those.
|
||||
Bounded rather than unlimited - only the last 2 MB is returned, and only
|
||||
for an allowlisted set of text extensions; a binary or oversized file is
|
||||
refused rather than mangled into an unreadable pane. The path is
|
||||
re-validated here even though the server already checked it, since this
|
||||
process runs as SYSTEM and the path arrived over the network: environment
|
||||
variables are expanded BEFORE the checks so %VAR% cannot smuggle a
|
||||
traversal or UNC path past them. Minor because it is a new action name -
|
||||
an older agent asked for it fails closed on the unknown-action default. -->
|
||||
<!-- 1.17.0: application inventory delta optimization. Backend adds
|
||||
app_fingerprint column (SHA256 of displayname|version|publisher) and
|
||||
switches from delete-all-insert-all to delta updates. Agent side not yet
|
||||
implemented - this release only fixes critical backend race conditions:
|
||||
advisory locks prevent TOCTOU races in both app delta and event dedup,
|
||||
UNIQUE constraint on (assetid, app_fingerprint) prevents duplicate
|
||||
fingerprints. Reduces database writes by 99% once agent sends fingerprints.
|
||||
Backend is backward compatible - legacy agents without fingerprints use
|
||||
delete-all-insert-all fallback. Minor for critical race condition fixes. -->
|
||||
<!-- 1.18.0: agent-side fingerprint computation. Computes SHA256 fingerprint
|
||||
for each application (displayname|displayversion|publisher) and includes
|
||||
in collector payload. Backend receives fingerprints and switches from
|
||||
delete-all-insert-all to delta mode (only touch changed apps). Expected
|
||||
impact: 99% reduction in database writes (1.92M/day to 19K/day). Matches
|
||||
backend algorithm implemented in 1.17.0. Minor for feature completion. -->
|
||||
<!-- 1.19.0: endpoint hardening. TLS without a pinned thumbprint now uses
|
||||
normal Windows chain validation instead of accepting any certificate.
|
||||
run_script and update_app stage into SYSTEM/Administrators-only
|
||||
directories under %ProgramData%\ShopDB\EndpointAgent with random file
|
||||
names, and the script hash is checked in memory and again over the
|
||||
exact bytes written, so a standard user can no longer swap a body
|
||||
between verification and execution. User-session transcripts moved out
|
||||
of the operator's profile into that protected area, and SYSTEM never
|
||||
follows a reparse point to read or delete them. SYSTEM scripts drain
|
||||
stdout/stderr while running (no pipe deadlock), cap captured output,
|
||||
and are killed with their process tree after 5 minutes. Strict mode
|
||||
inserts after using/#Requires/param headers, keeps line numbers, and
|
||||
writes UTF-8 with BOM. A user-session run with no exit code is a
|
||||
failure. collect_log refuses answer files, agent config/state, key
|
||||
material and reparse points, and rechecks the handle's final path.
|
||||
Certificate renewal at start is bounded to 15s and exits non-zero so
|
||||
SCM recovery restarts the service. -->
|
||||
<!-- 1.20.0: per-PC endpoint credentials. The shared fleet collector key was
|
||||
readable by any standard user on every bay, so one bay leaked the whole
|
||||
fleet. The agent now enrolls itself (POST /api/endpoints/enroll with
|
||||
hostname, BIOS serial, DeviceId and a persisted nonce) and receives its
|
||||
own credential, stored DPAPI LocalMachine under a SYSTEM/Administrators
|
||||
directory and sent as X-Endpoint-Credential. The shared key stays the
|
||||
fallback until the PC is enrolled, so the build also runs against an
|
||||
older server (a 404 on enroll parks enrollment for 6 hours). A revoked
|
||||
credential (CREDENTIAL_INVALID) is deleted and the PC re-enrolls without
|
||||
a restart, and a script or package download refused that way retries
|
||||
once. Server trust moves to SPKI SHA-256 pins (the ServerPins registry
|
||||
value when it exists, even empty; otherwise serverpins in
|
||||
appsettings.json), checked by one AgentHttpFactory
|
||||
that now builds every HttpClient, including script and package downloads
|
||||
that previously had their own unpinned handlers. The tray holds no
|
||||
secret any more: it reaches the service over the local named pipe
|
||||
ShopDB.EndpointAgent.v1 (ping, sendstatus, sendreport, alerts, ackalert),
|
||||
and the once and report switches become pipe clients when not SYSTEM. A report path must sit inside the
|
||||
caller's own profile. appsettings.json is restricted to SYSTEM and
|
||||
Administrators once the pipe is up. The 1.19 client-certificate renewal
|
||||
and expiry monitor are removed, and with them the non-zero exit at start.
|
||||
Pairs with tray 1.7.0.
|
||||
|
||||
1.20.1: no behaviour of its own. Two fixes landed after the 1.20.0
|
||||
binaries had already been built and shipped to the display scope -
|
||||
the fast retry while an enrollment is pending, and the fleet alert
|
||||
key rename - so the rebuild carries different bytes under the same
|
||||
version. Detection is by hash and copes, but two files calling
|
||||
themselves 1.20.0.0 cannot be told apart by anyone reading a
|
||||
version number, so the version moves. Pairs with tray 1.7.1. -->
|
||||
<Version>1.20.1</Version>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
|
||||
<SelfContained>true</SelfContained>
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
{
|
||||
"baseurl": "",
|
||||
"apikey": "REPLACE_WITH_COLLECTOR_KEY",
|
||||
"serverpins": [],
|
||||
"cadenceseconds": 900,
|
||||
"jitterseconds": 60,
|
||||
"startupjitterseconds": 30,
|
||||
"adoptonfirstrun": true,
|
||||
"utilitiespath": "",
|
||||
"logpath": "",
|
||||
|
||||
@@ -1,355 +0,0 @@
|
||||
# install-shopdbagent.ps1
|
||||
# ShopDB Endpoint Agent installer for the test endpoint. Run from an ELEVATED
|
||||
# (admin) PowerShell on the endpoint. Provisions the registry secrets, copies
|
||||
# the agent files to Program Files, runs a one-shot smoke test, then installs
|
||||
# and starts the Windows service.
|
||||
#
|
||||
# Secrets are provisioned to HKLM\SOFTWARE\GE\ShopDB. On locked-down hosts
|
||||
# (CyberArk/EPM) the agent may be blocked from READING that hive, so the same
|
||||
# values are also stamped into the installed appsettings.json as a fallback.
|
||||
#
|
||||
# The agent and tray are published as single-file self-contained executables,
|
||||
# so copying the two .exe files plus appsettings.json is the complete payload.
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
# ShopDB server root URL for this instance. REQUIRED - there is no default,
|
||||
# because a default here means one site's hostname shipped to every site
|
||||
# (ADR-015) and, worse, a working install pointed at somebody else's server.
|
||||
#
|
||||
# https is required. The collector key travels in the X-API-Key header and
|
||||
# every telemetry payload follows it, so plain http puts both on the wire in
|
||||
# clear. Pass -AllowInsecureHttp only for a local test instance.
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string] $BaseUrl,
|
||||
|
||||
# Endpoints-only collector key: this instance's .env COLLECTOR_API_KEY_ENDPOINTS.
|
||||
# REQUIRED and never defaulted - a key in source is a key in git history, and
|
||||
# this one is also the command channel's kill switch.
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string] $CollectorKey,
|
||||
|
||||
# Permit an http:// BaseUrl. For a local test instance only.
|
||||
[switch] $AllowInsecureHttp,
|
||||
|
||||
|
||||
[string] $InstallDir = "C:\Program Files\ShopDB\EndpointAgent",
|
||||
[string] $ServiceName = "ShopDbEndpointAgent",
|
||||
|
||||
# Interactive tray app that shows the system-tray icon and the on-demand
|
||||
# "Send Status now" action. Separate process from the SYSTEM service (a
|
||||
# service cannot draw a tray icon in a user session). Pass -NoTray to skip.
|
||||
[string] $TrayExeName = "shopdb-endpoint-tray.exe",
|
||||
[switch] $NoTray,
|
||||
|
||||
# Add a Microsoft Defender path exclusion for the install directory. The
|
||||
# agent and tray are unsigned single-file self-contained executables, which
|
||||
# Defender heuristics (and the tray's GDI/user32 icon P/Invoke) can flag as
|
||||
# a false positive and quarantine, killing the tray before its icon appears.
|
||||
# Pass -NoDefenderExclusion to skip (e.g. where policy manages exclusions
|
||||
# centrally or the binaries are code-signed).
|
||||
[switch] $NoDefenderExclusion
|
||||
)
|
||||
|
||||
# Refuse cleartext unless it was asked for explicitly. The key is a bearer
|
||||
# credential for the whole fleet's telemetry and command channel.
|
||||
if ($BaseUrl -notmatch '^https://' -and -not $AllowInsecureHttp) {
|
||||
throw "BaseUrl must be https:// (got '$BaseUrl'). Pass -AllowInsecureHttp for a local test instance."
|
||||
}
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$sourceDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
|
||||
function Assert-Admin {
|
||||
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
|
||||
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
|
||||
throw "This script must be run from an elevated (Run as administrator) PowerShell."
|
||||
}
|
||||
}
|
||||
|
||||
Assert-Admin
|
||||
|
||||
if ([string]::IsNullOrWhiteSpace($BaseUrl)) {
|
||||
throw "BaseUrl is not set. Pass -BaseUrl 'http://<host>:<port>' when running the script."
|
||||
}
|
||||
|
||||
# Write HKLM values through the explicit 64-bit registry view. If this script
|
||||
# runs under 32-bit PowerShell (some management tooling launches x86), the
|
||||
# PSDrive HKLM: silently redirects writes into WOW6432Node. The 64-bit agent
|
||||
# service and 64-bit Explorer (the logon Run key) then read the 64-bit view and
|
||||
# see nothing, so config "goes missing" and the tray never autostarts. Pinning
|
||||
# the view removes that dependency on the launching shell's bitness.
|
||||
function Set-Hklm64Value {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string] $SubKey,
|
||||
[Parameter(Mandatory)] [string] $Name,
|
||||
[Parameter(Mandatory)] [string] $Value
|
||||
)
|
||||
$base = [Microsoft.Win32.RegistryKey]::OpenBaseKey(
|
||||
[Microsoft.Win32.RegistryHive]::LocalMachine,
|
||||
[Microsoft.Win32.RegistryView]::Registry64)
|
||||
try {
|
||||
$key = $base.CreateSubKey($SubKey)
|
||||
try {
|
||||
$key.SetValue($Name, $Value, [Microsoft.Win32.RegistryValueKind]::String)
|
||||
}
|
||||
finally { $key.Dispose() }
|
||||
}
|
||||
finally { $base.Dispose() }
|
||||
}
|
||||
|
||||
# Tear down any previously installed agent BEFORE touching files. The agent and
|
||||
# tray are single-file self-contained exes; while the old service or tray is
|
||||
# running it holds those .exe files open, so Copy-Item over them fails with a
|
||||
# file-in-use error. Stopping the service, deleting it, and killing any lingering
|
||||
# agent/tray processes up front makes the install a clean overwrite. Best-effort
|
||||
# throughout: a missing service or process is the normal fresh-install case.
|
||||
function Uninstall-PreviousAgent {
|
||||
param(
|
||||
[Parameter(Mandatory)] [string] $Service,
|
||||
[Parameter(Mandatory)] [string] $TrayExe,
|
||||
[Parameter(Mandatory)] [string] $TargetDir
|
||||
)
|
||||
|
||||
$existing = Get-Service -Name $Service -ErrorAction SilentlyContinue
|
||||
if ($existing) {
|
||||
Write-Host " stopping and removing existing service $Service ..."
|
||||
if ($existing.Status -ne "Stopped") {
|
||||
try { Stop-Service -Name $Service -Force -ErrorAction Stop }
|
||||
catch { Write-Warning " could not stop service: $($_.Exception.Message)" }
|
||||
}
|
||||
# sc.exe delete only marks the service for deletion; the SCM finishes the
|
||||
# removal once every handle closes. Give it a moment so the subsequent
|
||||
# sc.exe create does not hit a "marked for deletion" (1072) error.
|
||||
sc.exe delete $Service | Out-Null
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
|
||||
# Kill any lingering worker or tray processes that would keep the exes
|
||||
# locked. Match by the installed exe base names; the service host process is
|
||||
# shopdbagent, the tray is the TrayExe base name.
|
||||
$trayBase = [System.IO.Path]::GetFileNameWithoutExtension($TrayExe)
|
||||
foreach ($name in @("shopdbagent", $trayBase)) {
|
||||
$procs = Get-Process -Name $name -ErrorAction SilentlyContinue
|
||||
foreach ($p in $procs) {
|
||||
Write-Host " killing lingering process $($p.ProcessName) (pid $($p.Id)) ..."
|
||||
try { Stop-Process -Id $p.Id -Force -ErrorAction Stop }
|
||||
catch { Write-Warning " could not kill $($p.ProcessName): $($_.Exception.Message)" }
|
||||
}
|
||||
}
|
||||
# Also kill anything still running out of the target install directory,
|
||||
# regardless of exe name. A tray from an older build (different exe name) or
|
||||
# one launched from a stale path would otherwise survive the name-based sweep
|
||||
# above, keep its exe locked, and leave a duplicate icon once step 5 starts
|
||||
# the fresh tray. Match on the executable path so we only touch our own dir.
|
||||
$targetPrefix = $TargetDir.TrimEnd('\') + '\'
|
||||
foreach ($p in (Get-Process -ErrorAction SilentlyContinue)) {
|
||||
$path = $null
|
||||
try { $path = $p.Path } catch { $path = $null }
|
||||
if ($path -and $path.StartsWith($targetPrefix, [System.StringComparison]::OrdinalIgnoreCase)) {
|
||||
Write-Host " killing process from install dir $($p.ProcessName) (pid $($p.Id)) ..."
|
||||
try { Stop-Process -Id $p.Id -Force -ErrorAction Stop }
|
||||
catch { Write-Warning " could not kill $($p.ProcessName): $($_.Exception.Message)" }
|
||||
}
|
||||
}
|
||||
# Brief settle so file handles are released before the copy overwrites them.
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
|
||||
Write-Host "== ShopDB Endpoint Agent install ==" -ForegroundColor Cyan
|
||||
Write-Host "BaseUrl : $BaseUrl"
|
||||
Write-Host "InstallDir : $InstallDir"
|
||||
Write-Host "ServiceName : $ServiceName"
|
||||
|
||||
# 0. Clear the Mark-of-the-Web on the SOURCE payload first. The operator
|
||||
# typically extracts installer.zip into Downloads, so every file carries a
|
||||
# Zone.Identifier ADS. SmartScreen reads that and blocks the unsigned exes with
|
||||
# "Windows protected your PC" the first time they run - including the [3/5]
|
||||
# smoke test below, which launches shopdbagent.exe out of $sourceDir before the
|
||||
# installed copy exists. Unblocking the source folder up front keeps the smoke
|
||||
# test (and any manual run from the extract folder) clean; the installed copies
|
||||
# get unblocked again in step 2a after they land in Program Files. Best-effort:
|
||||
# a file with no MotW is a no-op.
|
||||
Write-Host "`n[0a] Clearing Mark-of-the-Web on the extracted installer files ..."
|
||||
Get-ChildItem -Path $sourceDir -File | ForEach-Object {
|
||||
try { Unblock-File -Path $_.FullName -ErrorAction Stop }
|
||||
catch { Write-Warning " could not unblock $($_.Name): $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
# 0. Uninstall any previous agent first, so the file copy is a clean overwrite
|
||||
# and does not fail on exes locked by a running service or tray.
|
||||
Write-Host "`n[0/5] Removing any previously installed agent ..."
|
||||
Uninstall-PreviousAgent -Service $ServiceName -TrayExe $TrayExeName -TargetDir $InstallDir
|
||||
|
||||
# 1. Provision registry secrets (64-bit view, see Set-Hklm64Value).
|
||||
$regPath = "HKLM:\SOFTWARE\GE\ShopDB"
|
||||
Write-Host "`n[1/5] Provisioning registry $regPath (Registry64) ..."
|
||||
Set-Hklm64Value -SubKey "SOFTWARE\GE\ShopDB" -Name "BaseUrl" -Value $BaseUrl
|
||||
Set-Hklm64Value -SubKey "SOFTWARE\GE\ShopDB" -Name "CollectorKey" -Value $CollectorKey
|
||||
|
||||
# 2. Copy agent files to the install directory. The agent and tray are
|
||||
# single-file self-contained executables, so the .exe plus appsettings.json is
|
||||
# the whole payload; there are no side-by-side runtime DLLs to copy.
|
||||
Write-Host "[2/5] Copying agent files to $InstallDir ..."
|
||||
New-Item -ItemType Directory -Force -Path $InstallDir | Out-Null
|
||||
$agentSource = Join-Path $sourceDir "shopdbagent.exe"
|
||||
if (-not (Test-Path $agentSource)) {
|
||||
throw "shopdbagent.exe not found in $sourceDir. Unzip the full installer package and run from that folder."
|
||||
}
|
||||
Copy-Item -Path $agentSource -Destination $InstallDir -Force
|
||||
$appsettings = Join-Path $sourceDir "appsettings.json"
|
||||
$installedSettings = Join-Path $InstallDir "appsettings.json"
|
||||
if (Test-Path $appsettings) {
|
||||
Copy-Item -Path $appsettings -Destination $InstallDir -Force
|
||||
}
|
||||
|
||||
# Copy the interactive tray app next to the agent so its same-dir resolution
|
||||
# finds shopdbagent.exe for the "Send Status now" action. Optional: absent on a
|
||||
# service-only (headless) build.
|
||||
$trayInstalled = $null
|
||||
if (-not $NoTray) {
|
||||
$traySource = Join-Path $sourceDir $TrayExeName
|
||||
if (Test-Path $traySource) {
|
||||
Copy-Item -Path $traySource -Destination $InstallDir -Force
|
||||
$trayInstalled = Join-Path $InstallDir $TrayExeName
|
||||
Write-Host " copied tray app $TrayExeName"
|
||||
}
|
||||
else {
|
||||
Write-Warning " tray app $TrayExeName not found in $sourceDir; skipping tray (service still installs)"
|
||||
}
|
||||
}
|
||||
|
||||
# Fallback for locked-down endpoints: some hosts (CyberArk/EPM) gate the HKLM
|
||||
# hive so the agent cannot READ the values the registry step just wrote. Stamp
|
||||
# BaseUrl/CollectorKey into the installed appsettings.json too. The agent reads
|
||||
# appsettings first, then lets the registry override it, so this is a safe
|
||||
# no-op where the registry read works and the rescue where it does not.
|
||||
if (Test-Path $installedSettings) {
|
||||
try {
|
||||
$json = Get-Content $installedSettings -Raw | ConvertFrom-Json
|
||||
$json.baseurl = $BaseUrl
|
||||
$json.apikey = $CollectorKey
|
||||
$json | ConvertTo-Json -Depth 10 | Set-Content -Path $installedSettings -Encoding UTF8
|
||||
Write-Host " stamped BaseUrl/CollectorKey into appsettings.json (registry-read fallback)"
|
||||
}
|
||||
catch {
|
||||
Write-Warning " could not stamp appsettings.json: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
$exePath = Join-Path $InstallDir "shopdbagent.exe"
|
||||
|
||||
# 2a. Strip the Mark-of-the-Web from the copied files. Windows tags files
|
||||
# extracted from a downloaded zip with a Zone.Identifier alternate data stream;
|
||||
# SmartScreen reads that and blocks the unsigned exe with "Windows protected
|
||||
# your PC" the first time it runs, and the operator sees only a "Don't run"
|
||||
# button. Unblock-File removes the ADS so the smoke test and the tray launch
|
||||
# start clean. Best-effort: a file with no MotW is a no-op.
|
||||
Write-Host "[2a/5] Clearing Mark-of-the-Web on installed files ..."
|
||||
Get-ChildItem -Path $InstallDir -File | ForEach-Object {
|
||||
try { Unblock-File -Path $_.FullName -ErrorAction Stop }
|
||||
catch { Write-Warning " could not unblock $($_.Name): $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
# 2b. Defender path exclusion for the install dir. Unsigned single-file
|
||||
# self-contained binaries (and the tray's icon P/Invoke) can trip Defender
|
||||
# heuristics and get quarantined, which is why the tray dies before showing an
|
||||
# icon. Excluding the fixed install directory is the least-privilege fix short
|
||||
# of code-signing. Best-effort: Set-MpPreference is absent on servers without
|
||||
# Defender and can be blocked by tamper protection; a failure is not fatal.
|
||||
if (-not $NoDefenderExclusion) {
|
||||
Write-Host "[2b/5] Adding Defender path exclusion for $InstallDir ..."
|
||||
if (Get-Command Add-MpPreference -ErrorAction SilentlyContinue) {
|
||||
try {
|
||||
Add-MpPreference -ExclusionPath $InstallDir -ErrorAction Stop
|
||||
Write-Host " excluded $InstallDir from Defender scanning."
|
||||
}
|
||||
catch {
|
||||
Write-Warning " could not add Defender exclusion (tamper protection or policy?): $($_.Exception.Message)"
|
||||
Write-Warning " if the tray is quarantined, code-sign the binaries or add the exclusion via policy."
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-Host " Defender cmdlets not present; skipping exclusion."
|
||||
}
|
||||
}
|
||||
|
||||
# 3. One-shot smoke test (exact production collection path, then exits).
|
||||
Write-Host "[3/5] Running one-shot smoke test (--once) ..."
|
||||
$proc = Start-Process -FilePath $exePath -ArgumentList "--once" -WorkingDirectory $InstallDir -Wait -PassThru -NoNewWindow
|
||||
Write-Host " smoke test exit code: $($proc.ExitCode)"
|
||||
|
||||
$logDir = Join-Path $env:ProgramData "ShopDB\EndpointAgent\logs"
|
||||
$logFile = Join-Path $logDir ("agent-{0}.log" -f (Get-Date -Format "yyyyMMdd"))
|
||||
if (Test-Path $logFile) {
|
||||
Write-Host " --- last log lines ---" -ForegroundColor DarkGray
|
||||
Get-Content $logFile -Tail 20 | ForEach-Object { Write-Host " $_" -ForegroundColor DarkGray }
|
||||
}
|
||||
if ($proc.ExitCode -ne 0) {
|
||||
throw "Smoke test failed (exit $($proc.ExitCode)). Check $logFile before installing the service."
|
||||
}
|
||||
|
||||
# 4. Install the Windows service. Any previous instance was already stopped,
|
||||
# deleted, and its processes killed in step 0, so this is a clean create. Guard
|
||||
# against the rare case where the SCM has not finished a pending deletion yet.
|
||||
Write-Host "[4/5] Installing service $ServiceName ..."
|
||||
$existing = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
|
||||
if ($existing) {
|
||||
Write-Host " service still present (pending deletion); removing again ..."
|
||||
if ($existing.Status -ne "Stopped") { Stop-Service -Name $ServiceName -Force }
|
||||
sc.exe delete $ServiceName | Out-Null
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
$binPath = '"' + $exePath + '"'
|
||||
sc.exe create $ServiceName binPath= $binPath start= auto DisplayName= "ShopDB Endpoint Agent" | Out-Null
|
||||
sc.exe description $ServiceName "Collects endpoint telemetry for ShopDB and runs allowlisted WebUI commands." | Out-Null
|
||||
sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/5000 | Out-Null
|
||||
|
||||
Start-Service -Name $ServiceName
|
||||
Write-Host " service started."
|
||||
|
||||
# 5. Register the interactive tray app to launch at user logon (per-machine).
|
||||
Write-Host "[5/5] Registering tray app for logon ..."
|
||||
if ($trayInstalled -and (Test-Path $trayInstalled)) {
|
||||
# Register in the 64-bit Run key so 64-bit Explorer fires it at logon. A
|
||||
# 32-bit write would land in WOW6432Node\...\Run, which Explorer ignores.
|
||||
Set-Hklm64Value -SubKey "SOFTWARE\Microsoft\Windows\CurrentVersion\Run" `
|
||||
-Name "ShopDbEndpointTray" -Value ('"' + $trayInstalled + '"')
|
||||
Write-Host " tray app registered to start at logon (Registry64): $trayInstalled"
|
||||
|
||||
# Start it now so the icon appears without waiting for the next logon. The
|
||||
# Run key only fires at logon, so a fresh install would otherwise show no
|
||||
# tray until the operator signs out and back in.
|
||||
#
|
||||
# The installer runs elevated, so a direct Start-Process would launch the
|
||||
# tray ELEVATED too. That is wrong for a tray app (it should run at the
|
||||
# user's medium integrity) and, on EPM-managed hosts (CyberArk), elevating an
|
||||
# unsigned exe triggers a broker prompt that comes back as Win32 1223
|
||||
# "The operation was canceled by the user" - which is exactly what we hit.
|
||||
# Launching through explorer.exe hands the exe to the already-running user
|
||||
# shell, which spawns it de-elevated in the interactive session, sidestepping
|
||||
# the elevation broker entirely.
|
||||
$interactiveUser = $env:USERNAME
|
||||
if ($interactiveUser -and $interactiveUser -ne "SYSTEM") {
|
||||
try {
|
||||
# explorer.exe <path> re-parents the launch under the user shell at
|
||||
# medium integrity; it returns immediately regardless of the tray's
|
||||
# own lifetime, so there is nothing to wait on here.
|
||||
Start-Process -FilePath "explorer.exe" -ArgumentList "`"$trayInstalled`"" | Out-Null
|
||||
Write-Host " tray app started for the current session (de-elevated via explorer)."
|
||||
}
|
||||
catch {
|
||||
Write-Warning " could not start tray now: $($_.Exception.Message); it will start at next logon."
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-Host " no interactive user session; tray will start at next logon."
|
||||
}
|
||||
}
|
||||
else {
|
||||
Write-Host " tray skipped (headless install)."
|
||||
}
|
||||
|
||||
Write-Host "`nInstall complete." -ForegroundColor Green
|
||||
Write-Host "Service '$ServiceName' is running and set to auto-start."
|
||||
Write-Host "Logs: $logDir"
|
||||
76
agent/test-fingerprint.ps1
Normal file
@@ -0,0 +1,76 @@
|
||||
# Test fingerprint computation matches Python backend
|
||||
# Expected: SHA256("Google Chrome|117.0.5938.132|Google LLC")
|
||||
|
||||
$code = @"
|
||||
using System;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
||||
public class FingerprintTester
|
||||
{
|
||||
public static string ComputeFingerprint(string displayName, string displayVersion, string publisher)
|
||||
{
|
||||
var normalized = $"{displayName}|{displayVersion}|{publisher}";
|
||||
var bytes = Encoding.UTF8.GetBytes(normalized);
|
||||
var hash = SHA256.HashData(bytes);
|
||||
return Convert.ToHexString(hash).ToLowerInvariant();
|
||||
}
|
||||
}
|
||||
"@
|
||||
|
||||
Add-Type -TypeDefinition $code
|
||||
|
||||
# Test vectors from backend
|
||||
$tests = @(
|
||||
@{
|
||||
Name = "Google Chrome"
|
||||
Version = "117.0.5938.132"
|
||||
Publisher = "Google LLC"
|
||||
Description = "Standard application"
|
||||
},
|
||||
@{
|
||||
Name = "Microsoft Office"
|
||||
Version = "16.0.16827.20166"
|
||||
Publisher = "Microsoft Corporation"
|
||||
Description = "Office suite"
|
||||
},
|
||||
@{
|
||||
Name = "Test App No Version"
|
||||
Version = ""
|
||||
Publisher = "Test Publisher"
|
||||
Description = "Empty version string"
|
||||
},
|
||||
@{
|
||||
Name = "Test App Empty"
|
||||
Version = ""
|
||||
Publisher = ""
|
||||
Description = "Empty strings"
|
||||
}
|
||||
)
|
||||
|
||||
Write-Host "C# Fingerprint Algorithm Test" -ForegroundColor Cyan
|
||||
Write-Host "==============================" -ForegroundColor Cyan
|
||||
Write-Host ""
|
||||
|
||||
foreach ($test in $tests) {
|
||||
$fp = [FingerprintTester]::ComputeFingerprint($test.Name, $test.Version, $test.Publisher)
|
||||
Write-Host "Test: $($test.Description)" -ForegroundColor Yellow
|
||||
Write-Host " Input: '$($test.Name)|$($test.Version)|$($test.Publisher)'"
|
||||
Write-Host " Fingerprint: $fp" -ForegroundColor Green
|
||||
Write-Host " Length: $($fp.Length) chars" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
}
|
||||
|
||||
# Verify format
|
||||
$testFp = [FingerprintTester]::ComputeFingerprint("Test", "1.0", "Publisher")
|
||||
if ($testFp.Length -eq 64) {
|
||||
Write-Host "Format check: PASS (64 characters)" -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host "Format check: FAIL (expected 64, got $($testFp.Length))" -ForegroundColor Red
|
||||
}
|
||||
|
||||
if ($testFp -match '^[0-9a-f]{64}$') {
|
||||
Write-Host "Hex check: PASS (lowercase hexadecimal)" -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host "Hex check: FAIL (invalid characters)" -ForegroundColor Red
|
||||
}
|
||||
119
agent/utilities/shopdb-dellupdates.ps1
Normal file
@@ -0,0 +1,119 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
ShopDB agent utility: reports pending Dell Command | Update items.
|
||||
|
||||
.DESCRIPTION
|
||||
Discovered and executed by PowerShellUtilityRunner on the telemetry cadence.
|
||||
Emits a single JSON object on stdout, which the agent attaches to the report
|
||||
as `dellupdates`. Never throws and never writes anything but JSON: a failure
|
||||
here must not break a telemetry run, so every error path still emits a valid
|
||||
object with a `status` explaining why the list is empty.
|
||||
|
||||
THIS SCRIPT DOES NOT SCAN. It only reads the log a previous scan wrote. The
|
||||
scan itself is fired by the \ShopDB-DellCommandScan scheduled task, which is
|
||||
what pins dcu-cli.exe's -outputLog to the path below. Keeping the two apart
|
||||
is deliberate: a telemetry collector that shelled out to dcu-cli would block
|
||||
the cadence for minutes and hammer Dell's servers every 15 minutes.
|
||||
|
||||
OUTPUT CONTRACT (keep in sync with plugins/endpoints/plugin.py):
|
||||
status unknown | ok | error - "unknown" means no scan has ever run,
|
||||
which is NOT the same as "ok" with an empty list (known clean).
|
||||
scannedat ISO-8601 UTC timestamp of the scan log's last write, or null.
|
||||
count total pending updates found.
|
||||
updates up to $MaxUpdates entries of { name, severity, version }.
|
||||
truncated true when count exceeded $MaxUpdates.
|
||||
|
||||
Dell's own long description text is deliberately NOT carried: it is
|
||||
sentence-length marketing prose per update, which would add roughly an order
|
||||
of magnitude to every report for no operational value. Name plus severity is
|
||||
what an operator decides on.
|
||||
#>
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# Cap the list rather than the payload. A machine that is years behind can have
|
||||
# dozens of pending updates, and an unbounded list would bloat every report on
|
||||
# the cadence. Mirrors the MaxReportedConnections precedent in CommandExecutor:
|
||||
# a truncated answer is more useful than a rejected payload.
|
||||
$MaxUpdates = 25
|
||||
|
||||
# Must match the -outputLog path set in the \ShopDB-DellCommandScan task
|
||||
# definition. ShopDB-initiated scans write HERE rather than to Dell's default
|
||||
# location, which is what keeps them distinguishable from the corporate-managed
|
||||
# DCU tasks that also run on these machines.
|
||||
$ScanLog = Join-Path $env:ProgramData 'ShopDB\EndpointAgent\dellcommand\scan.xml'
|
||||
|
||||
function Write-Result {
|
||||
param($Status, $ScannedAt, $Count, $Updates, $Truncated)
|
||||
# Compress so the payload carries no pretty-printing whitespace. Depth 3
|
||||
# covers the nested updates array.
|
||||
[pscustomobject]@{
|
||||
status = $Status
|
||||
scannedat = $ScannedAt
|
||||
count = $Count
|
||||
updates = @($Updates)
|
||||
truncated = [bool]$Truncated
|
||||
} | ConvertTo-Json -Depth 3 -Compress
|
||||
}
|
||||
|
||||
try {
|
||||
if (-not (Test-Path -LiteralPath $ScanLog)) {
|
||||
# No scan has run yet. This is the normal state for a freshly adopted
|
||||
# endpoint, not an error, but it must stay distinct from "scanned and
|
||||
# found nothing" or the UI would claim a machine is up to date when
|
||||
# nobody has ever checked.
|
||||
Write-Result -Status 'unknown' -ScannedAt $null -Count 0 -Updates @() -Truncated $false
|
||||
return
|
||||
}
|
||||
|
||||
$logFile = Get-Item -LiteralPath $ScanLog
|
||||
$scannedAt = $logFile.LastWriteTimeUtc.ToString('o')
|
||||
|
||||
[xml]$report = Get-Content -LiteralPath $ScanLog -Raw
|
||||
|
||||
# dcu-cli's scan report shape has varied across DCU versions, so select the
|
||||
# update nodes by name anywhere in the tree rather than by a fixed path.
|
||||
$nodes = @($report.SelectNodes('//update'))
|
||||
if ($nodes.Count -eq 0) {
|
||||
$nodes = @($report.SelectNodes('//Update'))
|
||||
}
|
||||
|
||||
$updates = foreach ($node in $nodes) {
|
||||
# Attribute casing also varies by version; take whichever is present and
|
||||
# fall back rather than emitting a null-filled row.
|
||||
$name = $node.name
|
||||
if (-not $name) { $name = $node.Name }
|
||||
if (-not $name) { $name = $node.display }
|
||||
if (-not $name) { continue }
|
||||
|
||||
$severity = $node.urgency
|
||||
if (-not $severity) { $severity = $node.Urgency }
|
||||
if (-not $severity) { $severity = $node.criticality }
|
||||
|
||||
$version = $node.version
|
||||
if (-not $version) { $version = $node.Version }
|
||||
|
||||
[pscustomobject]@{
|
||||
name = [string]$name
|
||||
severity = if ($severity) { [string]$severity } else { 'unknown' }
|
||||
version = if ($version) { [string]$version } else { $null }
|
||||
}
|
||||
}
|
||||
|
||||
$updates = @($updates)
|
||||
$count = $updates.Count
|
||||
$truncated = $count -gt $MaxUpdates
|
||||
if ($truncated) {
|
||||
$updates = $updates[0..($MaxUpdates - 1)]
|
||||
}
|
||||
|
||||
Write-Result -Status 'ok' -ScannedAt $scannedAt -Count $count `
|
||||
-Updates $updates -Truncated $truncated
|
||||
}
|
||||
catch {
|
||||
# Emit a valid object rather than failing: the runner would log a non-JSON
|
||||
# utility and drop it, which looks identical to "never ran" on the server.
|
||||
# An explicit error status tells an operator the scan log exists but could
|
||||
# not be read, which is a different problem from no scan at all.
|
||||
Write-Result -Status 'error' -ScannedAt $null -Count 0 -Updates @() -Truncated $false
|
||||
}
|
||||
124
deploy/ci/buildkit-remote.yaml
Normal file
@@ -0,0 +1,124 @@
|
||||
# Persistent, non-privileged BuildKit for the publish-images workflow.
|
||||
#
|
||||
# WHY this exists: the arc-runner-set runner pods have the docker/buildx CLIs
|
||||
# but no Docker daemon and cannot run a privileged DinD sidecar under cluster
|
||||
# policy. So image builds are delegated to this standalone buildkitd Service.
|
||||
# The workflow (.github/workflows/publish-images.yml) points Buildx at it with
|
||||
# driver: remote
|
||||
# endpoint: ${{ secrets.BUILDKIT_ENDPOINT }}
|
||||
# where BUILDKIT_ENDPOINT is set to:
|
||||
# tcp://buildkit.buildkit.svc.cluster.local:1234
|
||||
#
|
||||
# This runs rootless buildkitd, so it needs NO privileged: true. It only needs
|
||||
# a few unconfined-ish knobs the rootless image documents. A persistent
|
||||
# Deployment (not a per-job container) means the build cache in the PVC survives
|
||||
# across runs, so the heavy frontend npm build is only paid in full once.
|
||||
#
|
||||
# Apply once with `kubectl apply -f deploy/ci/buildkit-remote.yaml`. This
|
||||
# changes cluster infra only. It does NOT modify any application code or any
|
||||
# branch of the repo. A site that builds its own image with plain `docker
|
||||
# build` or `docker compose build` needs none of this.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: buildkit
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: buildkit
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app: buildkit
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: buildkit
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: buildkit
|
||||
annotations:
|
||||
# Required by rootless buildkitd to unshare user/network namespaces
|
||||
# without full privilege. These are the values from the upstream
|
||||
# rootless example; they are far weaker than privileged: true.
|
||||
container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
|
||||
spec:
|
||||
containers:
|
||||
- name: buildkitd
|
||||
image: moby/buildkit:v0.16.0-rootless
|
||||
args:
|
||||
- --addr
|
||||
- unix:///run/user/1000/buildkit/buildkitd.sock
|
||||
- --addr
|
||||
- tcp://0.0.0.0:1234
|
||||
# TLS is terminated in-cluster; the Service is only reachable from
|
||||
# runner pods on the internal network. If you want mTLS between the
|
||||
# runner and buildkitd, mount certs here and add --tlscacert etc,
|
||||
# then set the endpoint secret to the tcp:// address with certs.
|
||||
- --oci-worker-no-process-sandbox
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
- buildctl
|
||||
- debug
|
||||
- workers
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 30
|
||||
livenessProbe:
|
||||
exec:
|
||||
command:
|
||||
- buildctl
|
||||
- debug
|
||||
- workers
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 30
|
||||
securityContext:
|
||||
# Rootless: no privilege escalation, runs as uid 1000. The seccomp
|
||||
# and AppArmor unconfined settings below are what rootless needs in
|
||||
# place of privileged: true.
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
allowPrivilegeEscalation: false
|
||||
seccompProfile:
|
||||
type: Unconfined
|
||||
ports:
|
||||
- containerPort: 1234
|
||||
name: buildkit
|
||||
volumeMounts:
|
||||
- name: buildkit-cache
|
||||
mountPath: /home/user/.local/share/buildkit
|
||||
volumes:
|
||||
- name: buildkit-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: buildkit-cache
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: buildkit-cache
|
||||
namespace: buildkit
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
# Layer + build cache for the app image. Bump if the cache fills.
|
||||
storage: 20Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: buildkit
|
||||
namespace: buildkit
|
||||
spec:
|
||||
selector:
|
||||
app: buildkit
|
||||
ports:
|
||||
- name: buildkit
|
||||
port: 1234
|
||||
targetPort: 1234
|
||||
# ClusterIP only: reachable from runner pods, not exposed outside the cluster.
|
||||
type: ClusterIP
|
||||
@@ -3,22 +3,17 @@
|
||||
"plugins": [
|
||||
"backups",
|
||||
"computers",
|
||||
"employees",
|
||||
"endpoints",
|
||||
"geenforce",
|
||||
"guides",
|
||||
"knowledgebase",
|
||||
"machines",
|
||||
"measuringtools",
|
||||
"network",
|
||||
"notifications",
|
||||
"printedparts",
|
||||
"printers",
|
||||
"slides",
|
||||
"tools",
|
||||
"usb",
|
||||
"warranty"
|
||||
],
|
||||
"locked": [],
|
||||
"_comment": "The profile the released Windows installer is built from. Every bundled plugin that carries a manifest, so ONE exe serves any site: the wizard offers all of them and the operator ticks what that site uses. Plugins left unticked are never installed, and 'flask plugin prune-schema' drops their tables at provisioning (ADR-014). 'applications' is deliberately absent - it is manifest-less core and always ships. Build with: deploy/windows/installer/build-installer.sh deploy/site-profile-universal.json <repo>. Use site-profile.example.json instead only when a site genuinely needs a lean build; see ADR-013."
|
||||
"_comment": "Every plugin this branch bundles, which is what a receiving site installs by default. It is the profile docs/DEPLOY.md provisions from: 'flask plugin apply-profile deploy/site-profile-universal.json'. A plugin named here that is not on disk fails the command loudly, so this file and plugins/ must agree - they did not on 2026-09-19, when this still named five plugins that had been left on the site line, and provisioning a fresh site failed with PluginError. 'applications' is deliberately absent: it is manifest-less core and always ships. A site that wants fewer than all twelve copies site-profile.example.json and prunes; 'flask plugin prune-schema' drops the tables of anything left out (ADR-014, ADR-013)."
|
||||
}
|
||||
|
||||
6
deploy/windows/installer/.gitignore
vendored
@@ -1,6 +0,0 @@
|
||||
# Build output and staged payload - regenerable, and ~220MB.
|
||||
bundle/
|
||||
Output/
|
||||
# Generated by build-installer.sh from the staged bundle.
|
||||
plugins.iss
|
||||
version.iss
|
||||
@@ -1,247 +0,0 @@
|
||||
# Windows installer
|
||||
|
||||
Builds a single self-contained `.exe` that installs ShopDB-Flask on an
|
||||
**air-gapped** Windows Server. Nothing here ever touches the network at install
|
||||
time: Python, the wheels, the SPA and (optionally) MySQL all ship inside it.
|
||||
|
||||
Lives with the application on purpose. The installer depends on app internals -
|
||||
`flask plugin` verbs, `site-profile.json`, `MOUNT_PATH`, the plugin registry -
|
||||
so a separate repo would drift out of step with the thing it installs.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `shopdb-preflight.ps1` | Stage 1. Read-only. Changes nothing, reports what this server is missing. |
|
||||
| `shopdb-install.ps1` | Stages 0 and 2-5 plus `uninstall`. All the actual work. |
|
||||
| `shopdb-admin.ps1` | Operator console installed alongside the app: status, restart, logs, backup, plugins. |
|
||||
| `ShopDBFlask.iss` | Inno Setup wizard. A thin wrapper - it collects input and runs the stages. |
|
||||
| `build-installer.sh` | Stages the bundle from a site profile. |
|
||||
| `make-branding.py` | Generates wizard artwork and the icon from `frontend/public/*.svg`. |
|
||||
| `*.bmp`, `shopdb.ico` | Generated artwork, committed so a Windows build box needs no Python. |
|
||||
| `build-installer.ps1` | The same build, natively on Windows. No Bash needed. |
|
||||
| `bundle-lock.json` | The exact third-party payload this installer ships. Reviewed by commit. |
|
||||
| `bundle-lock.ps1` | Creates and checks that lock. Also runs on the target server. |
|
||||
| `refresh-bundle-lock.ps1` | Regenerates the lock, after showing what changed. |
|
||||
| `verify_bundle_lock.py` | The same check for the Bash builder, so Linux needs no pwsh. |
|
||||
|
||||
## Building
|
||||
|
||||
Two builders, same result. Use whichever machine you are on - `build-installer.ps1`
|
||||
does the whole job natively so a Windows work PC needs no Bash.
|
||||
|
||||
```bash
|
||||
# Linux / WSL
|
||||
./build-installer.sh ../../site-profile.example.json
|
||||
```
|
||||
|
||||
```powershell
|
||||
# Windows
|
||||
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
|
||||
```
|
||||
|
||||
Both stage the app tree, build the SPA twice, write `plugins.iss`, copy the
|
||||
installer scripts **from this directory**, and then verify the third-party
|
||||
payload against `bundle-lock.json`. They exit non-zero if it does not match.
|
||||
|
||||
The payload itself is added by hand, and no longer needs Windows to produce:
|
||||
|
||||
```
|
||||
bundle/wheels/ pip download -r requirements.txt --only-binary=:all: \
|
||||
--platform win_amd64 --python-version 314 \
|
||||
--implementation cp --abi cp314 -d wheels
|
||||
bundle/python/ python-3.14.x-amd64.exe
|
||||
bundle/httpplatformhandler/ httpPlatformHandler_amd64.msi
|
||||
bundle/urlrewrite/ rewrite_amd64.msi (client-IP rule; see below)
|
||||
bundle/vcredist/ VC_redist.x64.exe (MySQL will not install without it)
|
||||
bundle/mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)
|
||||
bundle/mysqlclient/ mysql.exe, mysqldump.exe + libcrypto/libssl (backups)
|
||||
```
|
||||
|
||||
Then compile on Windows:
|
||||
|
||||
```
|
||||
iscc ShopDBFlask.iss
|
||||
```
|
||||
|
||||
**Inno Setup 6.6.0 or newer.** The wizard uses the built-in `windows11` custom
|
||||
style, which earlier versions reject. The script fails at compile time with that
|
||||
sentence rather than with a bare "WizardStyle is invalid".
|
||||
|
||||
The wheelhouse is **cp314-locked**. A different Python minor version means a
|
||||
different wheelhouse; the installer will not use a Python it did not install.
|
||||
|
||||
## What is pinned, and where
|
||||
|
||||
Three layers, because no one of them covers the whole problem.
|
||||
|
||||
| Layer | Covers | Enforced |
|
||||
|---|---|---|
|
||||
| `requirements.txt` sha256 per package | every wheel is genuinely what upstream published | `pip --require-hashes` at install; aborts on mismatch |
|
||||
| `bundle-lock.json` | the EXACT payload: wheels, Python installer, MSIs | both builders, and again on the server before anything runs |
|
||||
| git | the application tree | code review |
|
||||
|
||||
`--require-hashes` alone is not enough. pip lists every artifact of a pinned
|
||||
version - `cffi 2.1.0` has 100 hashes - so it proves the wheel is genuine, not
|
||||
that it is the wheel this bundle was built and tested with. It also ignores
|
||||
extra files in the wheelhouse, and says nothing about the Python installer or
|
||||
the MSIs, all of which run as SYSTEM on the target server.
|
||||
|
||||
So `bundle-lock.json` records an exact file set with a sha256 and a byte size
|
||||
each, and verification is **set equality**: a missing file, an unexpected extra
|
||||
file, or changed content all fail. There is no install-time override.
|
||||
|
||||
The lockfile is `--universal`, so one file serves Linux (dev, Docker, CI) and the
|
||||
Windows wheelhouse. A Linux-only resolve had silently omitted `colorama`, a
|
||||
win32-only dependency of `click` - which in hash-checking mode is a hard error
|
||||
rather than a quiet omission.
|
||||
|
||||
### Verifying the installer itself
|
||||
|
||||
`bundle-lock.json` is **inside** the thing it describes, so it proves the payload
|
||||
was not altered between build and install - not that the `.exe` you received is
|
||||
the one that was built. That needs something out of band. Two options, in order
|
||||
of preference:
|
||||
|
||||
1. **Authenticode-sign the `.exe`** with a GE code-signing certificate. Windows
|
||||
then shows a real publisher instead of "Unknown", which is also what stops an
|
||||
operator learning to click through the SmartScreen warning.
|
||||
2. **Publish a sha256 per release** through a different channel than the file
|
||||
itself, and have the receiving site check it:
|
||||
`Get-FileHash ShopDBFlask_Installer_*.exe -Algorithm SHA256`
|
||||
|
||||
Neither is wired up yet. Until one is, an installer is only as trustworthy as
|
||||
the share it arrived on.
|
||||
|
||||
### Changing what ships
|
||||
|
||||
```powershell
|
||||
.\refresh-bundle-lock.ps1 # show what changed, write nothing
|
||||
.\refresh-bundle-lock.ps1 -Yes # write it
|
||||
```
|
||||
|
||||
Then **commit `bundle-lock.json`**. That commit is the review - it is the only
|
||||
place a change to what runs as SYSTEM on a customer's server becomes visible to
|
||||
a human. `refresh-bundle-lock.ps1` refuses to overwrite an existing lock until
|
||||
you have seen the diff, for that reason.
|
||||
|
||||
To stage a bundle before its lock exists: `ALLOW_UNLOCKED=1` (Bash) or
|
||||
`-AllowUnlocked` (PowerShell). Bundles built that way must not be shipped.
|
||||
|
||||
Verifying a live server, months later and offline:
|
||||
|
||||
```powershell
|
||||
shopdb-admin.ps1 verify
|
||||
```
|
||||
|
||||
## Servers this installer did not build
|
||||
|
||||
It is built for greenfield: its own Python, its own venv, its own IIS objects.
|
||||
Its upgrade path assumes the thing being upgraded came out of a previous run.
|
||||
Two guards keep it from damaging a server that was deployed by hand.
|
||||
|
||||
**Python minor version.** An existing venv is reused, which is right for a repair
|
||||
or an upgrade. It is wrong when the venv belongs to a different Python - the
|
||||
wheelhouse is tagged for one minor version, so pip would die at the first
|
||||
compiled package, *after* Python was installed and the app tree replaced. The
|
||||
installer compares the two up front and stops with both version numbers.
|
||||
|
||||
**IIS objects.** Switching deployment method removes the other method's artifact,
|
||||
which is correct when the installer owns both and dangerous when it does not: a
|
||||
wrong `-MountAlias` would delete a live mount with no prompt. It now refuses
|
||||
unless there is a version stamp proving it made the install, or you pass
|
||||
`-AdoptExisting`. The refusal lists exactly what it would have removed.
|
||||
|
||||
An existing `web.config` is never overwritten in either case.
|
||||
|
||||
For the West Jefferson production server specifically, this is a **migration, not
|
||||
an upgrade** - prod runs Python 3.13 against a hand-built deployment, so it needs
|
||||
a deliberate window, a database backup, and web.config reconciled by hand.
|
||||
|
||||
## Bill of materials
|
||||
|
||||
Every build stages a CycloneDX 1.6 SBOM at `sbom.cdx.json`, inside the
|
||||
application tree, so it installs onto the server with the app. Both ecosystems,
|
||||
in one document:
|
||||
|
||||
- **Python** - every pin in `requirements.txt`, with the sha256 the installer
|
||||
enforces. Environment markers are ignored: a `sys_platform == 'win32'`
|
||||
dependency still installs on the target.
|
||||
- **npm** - every package in `frontend/package-lock.json`. Build-only packages
|
||||
are marked `scope: excluded` rather than dropped, so "not here" is
|
||||
distinguishable from "not looked for".
|
||||
|
||||
It ships to the server because an air-gapped site cannot be scanned from
|
||||
anywhere else. When a CVE lands, the answer is already on the box:
|
||||
|
||||
```powershell
|
||||
shopdb-admin.ps1 verify # counts, and which bundle this is
|
||||
shopdb-admin.ps1 verify -Path leaflet # is that component here, at what version
|
||||
```
|
||||
|
||||
Generated by `scripts/generate_sbom.py` from files that are already pinned and
|
||||
committed, so it is a translation rather than a scan - no network, no extra
|
||||
toolchain on the build box, and byte-identical output for the same inputs. It is
|
||||
deliberately not in `bundle-lock.json`: its provenance is git, not the payload.
|
||||
|
||||
## Client IP addresses
|
||||
|
||||
IIS does not set `X-Forwarded-For` on its own, and HttpPlatformHandler connects
|
||||
from loopback. Without a rule, **every client reads as 127.0.0.1** - so the
|
||||
GE-Enforce IP allowlist, the dashboard's visitor-location lookup and per-host
|
||||
login rate limiting all stop working, silently.
|
||||
|
||||
The wizard asks, because the two answers are mutually exclusive:
|
||||
|
||||
- **Clients connect directly** (`-ClientIpSource direct`, the default) - installs
|
||||
URL Rewrite from the bundle and sets `X-Forwarded-For` from `REMOTE_ADDR`.
|
||||
Overwriting the header is what stops a client spoofing its own.
|
||||
- **A proxy sits in front** (`-ClientIpSource proxy`) - leaves the rule off.
|
||||
Behind ARR or a load balancer `REMOTE_ADDR` is the *proxy*, so applying the
|
||||
rule would discard the real client IP.
|
||||
|
||||
An existing `web.config` is never overwritten - it is the one file on a server
|
||||
that legitimately carries hand-edits. The installer reports what it found
|
||||
instead.
|
||||
|
||||
## Deployment methods
|
||||
|
||||
Chosen in the wizard, and the bundle carries a SPA build for each because Vite
|
||||
compiles the base path in - it cannot be switched at install time.
|
||||
|
||||
- **Its own site** on a port (default 8090).
|
||||
- **Subpath** under an existing site, e.g. `http://<server-fqdn>/shopdb/`. Needs
|
||||
no new DNS record. Three things must agree - the IIS application alias,
|
||||
`MOUNT_PATH` in `.env`, and the SPA's build-time base - so the alias is fixed
|
||||
per bundle (`SUBPATH_ALIAS`, default `shopdb`) and the installer refuses if the
|
||||
bundle's build does not match what was asked for.
|
||||
|
||||
Switching between methods removes the other one's IIS artifact and reconciles
|
||||
`MOUNT_PATH` and `CORS_ORIGINS`, so a server never ends up with both.
|
||||
|
||||
## Upgrades
|
||||
|
||||
Run a newer installer over an existing install. It:
|
||||
|
||||
- backs the database up first, **verifies** the dump is complete, and refuses to
|
||||
migrate if it cannot;
|
||||
- restores from that backup if migrations fail, and reports honestly that DDL
|
||||
the failed migration committed cannot be undone;
|
||||
- refuses to run a bundle older than what is installed;
|
||||
- keeps `.env` unless new credentials are supplied, and copies it aside first;
|
||||
- stops the app pool before replacing files, then starts it again.
|
||||
|
||||
Whether a run is an upgrade is decided by probing the **target database**, not by
|
||||
whether the app directory exists - a rebuilt server pointed at an existing
|
||||
database is an upgrade, and treating it as fresh would drop tables.
|
||||
|
||||
## Testing notes
|
||||
|
||||
Verified end to end on Windows Server 2025 against both a bundled MySQL 8.4 LTS and
|
||||
an existing MySQL 5.6: fresh install, upgrade, re-run idempotency, failure and
|
||||
rollback, uninstall, and both deployment methods including switching between
|
||||
them.
|
||||
|
||||
**Not yet verified:** a fully air-gapped run with the network disabled at the
|
||||
hypervisor, and any load in a real browser (all HTTP checks so far used curl,
|
||||
which sends no `Origin` header - so `CORS_ORIGINS` is untested in anger).
|
||||
@@ -1,301 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Stage a lean per-site installer bundle on Windows, and verify its third-party
|
||||
payload against bundle-lock.json.
|
||||
|
||||
.DESCRIPTION
|
||||
The Windows equivalent of build-installer.sh, for a work PC with no Bash. It
|
||||
does the whole job natively - it does NOT shell out to build-site.sh - so the
|
||||
only tools needed are Python, Node and (to compile) Inno Setup.
|
||||
|
||||
What it does, in order:
|
||||
|
||||
1. Resolves the site's plugin closure from its profile.
|
||||
2. Builds the SPA twice, because Vite compiles the base path in and it
|
||||
therefore cannot be chosen at install time: once for /<alias> (subpath
|
||||
deployment) and once for / (its own site).
|
||||
3. Stages the backend tree: core, the chosen plugins only, and the runtime
|
||||
files a deployable tree needs.
|
||||
4. Writes plugins.iss so the wizard's plugin page matches the payload.
|
||||
5. Copies the installer scripts from THIS directory.
|
||||
6. Verifies wheels\, python\ and the MSIs against bundle-lock.json, and
|
||||
FAILS if the bundle is not exactly what the lock describes.
|
||||
|
||||
Step 6 is the point of the script. A bundle that does not match its lock is
|
||||
not shipped, and "the wheelhouse was missing" is caught here rather than by
|
||||
an operator halfway through installing on a server with no network.
|
||||
|
||||
.PARAMETER Profile
|
||||
Path to the site profile (see deploy\site-profile.example.json).
|
||||
|
||||
.PARAMETER RepoRoot
|
||||
The repository. Defaults to four levels up from this script, which is correct
|
||||
for a normal checkout.
|
||||
|
||||
.PARAMETER SkipFrontend
|
||||
Reuse the SPA builds already staged in the bundle. For iterating on the
|
||||
installer itself, where two Vite builds per run is most of the wall clock.
|
||||
Never use it for a bundle you intend to ship.
|
||||
|
||||
.PARAMETER AllowUnlocked
|
||||
Stage the bundle and report payload problems WITHOUT failing. For assembling
|
||||
a bundle before its lock exists. A bundle built this way must not be shipped;
|
||||
run refresh-bundle-lock.ps1, commit the lock, then build again without this.
|
||||
|
||||
.EXAMPLE
|
||||
.\build-installer.ps1 -Profile ..\..\site-profile.example.json
|
||||
.\build-installer.ps1 -Profile C:\sites\wjf.json -SkipFrontend
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $Profile,
|
||||
[string] $RepoRoot = (Resolve-Path (Join-Path $PSScriptRoot '..\..\..')).Path,
|
||||
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
|
||||
[string] $SubpathAlias = 'shopdb',
|
||||
[switch] $SkipFrontend,
|
||||
[switch] $AllowUnlocked
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
|
||||
|
||||
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
|
||||
function Step { param($m) Write-Host ''; Write-Host "==> $m" -ForegroundColor Cyan }
|
||||
function Die { param($m, $fix = '') Write-Host ''; Write-Host " $m" -ForegroundColor Red
|
||||
if ($fix) { Write-Host " $fix" -ForegroundColor Yellow }; exit 1 }
|
||||
|
||||
function Invoke-Tool {
|
||||
# Runs a build tool and stops on a non-zero exit. npm writes progress to
|
||||
# stderr on SUCCESS, so stderr alone must never be treated as failure.
|
||||
param([string] $Exe, [string[]] $Arguments, [string] $WorkDir, [string] $What)
|
||||
Push-Location $WorkDir
|
||||
try {
|
||||
$prev = $ErrorActionPreference
|
||||
$ErrorActionPreference = 'Continue'
|
||||
& $Exe @Arguments 2>&1 | ForEach-Object { Say " $_" 'DarkGray' }
|
||||
$code = $LASTEXITCODE
|
||||
$ErrorActionPreference = $prev
|
||||
if ($code -ne 0) { Die "$What failed (exit $code)" }
|
||||
} finally { Pop-Location }
|
||||
}
|
||||
|
||||
if (-not (Test-Path $Profile)) { Die "profile not found: $Profile" }
|
||||
if (-not (Test-Path $RepoRoot)) { Die "repo not found: $RepoRoot" }
|
||||
$Profile = (Resolve-Path $Profile).Path
|
||||
$RepoRoot = (Resolve-Path $RepoRoot).Path
|
||||
$AppOut = Join-Path $BundleRoot 'app'
|
||||
|
||||
$python = (Get-Command python -ErrorAction SilentlyContinue)
|
||||
if (-not $python) { $python = Get-Command py -ErrorAction SilentlyContinue }
|
||||
if (-not $python) { Die 'Python not found on PATH' 'Install Python 3.14 and re-open the shell.' }
|
||||
$npm = Get-Command npm.cmd -ErrorAction SilentlyContinue
|
||||
if (-not $npm -and -not $SkipFrontend) { Die 'npm not found on PATH' 'Install Node, or pass -SkipFrontend to reuse the staged SPA.' }
|
||||
|
||||
Say ''
|
||||
Say " repo : $RepoRoot"
|
||||
Say " profile : $Profile"
|
||||
Say " bundle : $BundleRoot"
|
||||
|
||||
# --- 1. plugin closure ------------------------------------------------------
|
||||
# Same resolver the Linux builder uses, so both produce the same set from one
|
||||
# profile instead of two implementations of the closure rules.
|
||||
Step 'Resolving plugin closure'
|
||||
$closure = (& $python.Source (Join-Path $RepoRoot 'scripts\resolve_plugin_closure.py') $Profile $RepoRoot)
|
||||
if ($LASTEXITCODE -ne 0 -or -not $closure) { Die 'could not resolve the plugin closure from the profile' }
|
||||
$closure = $closure.Trim()
|
||||
Say " $closure" 'White'
|
||||
|
||||
# --- 2. frontend ------------------------------------------------------------
|
||||
$frontend = Join-Path $RepoRoot 'frontend'
|
||||
$subStaged = Join-Path $BundleRoot 'spa-subpath'
|
||||
$rootStaged= Join-Path $BundleRoot 'spa-root'
|
||||
|
||||
if ($SkipFrontend) {
|
||||
if (-not (Test-Path $subStaged) -or -not (Test-Path $rootStaged)) {
|
||||
Die '-SkipFrontend was passed but no SPA build is staged' 'Run once without it.'
|
||||
}
|
||||
Say ''
|
||||
Say ' Reusing the staged SPA builds (-SkipFrontend). NOT shippable if the frontend changed.' 'Yellow'
|
||||
} else {
|
||||
Step "Building the SPA for /$SubpathAlias/"
|
||||
$env:SITE_PLUGINS = $closure
|
||||
$env:VITE_BASE_PATH = "/$SubpathAlias/"
|
||||
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'subpath frontend build' }
|
||||
finally { Remove-Item Env:\VITE_BASE_PATH -ErrorAction SilentlyContinue }
|
||||
Remove-Item $subStaged -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Copy-Item (Join-Path $frontend 'dist') $subStaged -Recurse -Force
|
||||
# The alias is fixed at BUILD time; the installer reads this and refuses to
|
||||
# publish under a different one rather than serving a page that cannot load
|
||||
# its own assets.
|
||||
Set-Content -Path (Join-Path $subStaged '.alias') -Value $SubpathAlias -Encoding ASCII
|
||||
|
||||
# Root build LAST, so frontend\dist is left in the state a developer expects.
|
||||
Step 'Building the SPA for /'
|
||||
try { Invoke-Tool $npm.Source @('run','build','--silent') $frontend 'root frontend build' }
|
||||
finally { Remove-Item Env:\SITE_PLUGINS -ErrorAction SilentlyContinue }
|
||||
Remove-Item $rootStaged -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Copy-Item (Join-Path $frontend 'dist') $rootStaged -Recurse -Force
|
||||
}
|
||||
|
||||
# --- 3. backend tree --------------------------------------------------------
|
||||
Step "Staging the application tree"
|
||||
Remove-Item $AppOut -Recurse -Force -ErrorAction SilentlyContinue
|
||||
New-Item -ItemType Directory -Path (Join-Path $AppOut 'plugins') -Force | Out-Null
|
||||
|
||||
Copy-Item (Join-Path $RepoRoot 'shopdb') $AppOut -Recurse -Force
|
||||
foreach ($name in $closure.Split(',')) {
|
||||
$src = Join-Path $RepoRoot ('plugins\' + $name.Trim())
|
||||
if (-not (Test-Path $src)) { Die "plugin in the closure is not on disk: $name" }
|
||||
Copy-Item $src (Join-Path $AppOut 'plugins') -Recurse -Force
|
||||
}
|
||||
|
||||
# Runtime files a deployable tree needs beyond the Python packages. Without
|
||||
# these the tree imports but cannot be run or migrated.
|
||||
foreach ($f in @('wsgi.py', 'requirements.txt')) {
|
||||
Copy-Item (Join-Path $RepoRoot $f) $AppOut -Force
|
||||
}
|
||||
Copy-Item (Join-Path $RepoRoot 'migrations') $AppOut -Recurse -Force
|
||||
# ONLY web.config, not all of deploy\. The bundle is staged INSIDE deploy\, so
|
||||
# copying the whole tree would recurse into its own output, and the rest of
|
||||
# deploy\ is installer source that has no business on an application server.
|
||||
# shopdb-install.ps1 reads it from exactly this path.
|
||||
$cfgSrc = Join-Path $RepoRoot 'deploy\windows\web.config'
|
||||
if (Test-Path $cfgSrc) {
|
||||
New-Item -ItemType Directory -Path (Join-Path $AppOut 'deploy\windows') -Force | Out-Null
|
||||
Copy-Item $cfgSrc (Join-Path $AppOut 'deploy\windows') -Force
|
||||
}
|
||||
|
||||
# A CycloneDX SBOM of everything this tree depends on, Python and npm together.
|
||||
# Staged INTO the tree so it installs onto the server with the application: an
|
||||
# air-gapped site cannot be scanned remotely, so the only way to answer "are we
|
||||
# exposed to this CVE, and where" is for the answer to be sitting on the box.
|
||||
Step 'Generating SBOM'
|
||||
& $python.Source (Join-Path $RepoRoot 'scripts\generate_sbom.py') $RepoRoot `
|
||||
-o (Join-Path $AppOut 'sbom.cdx.json') | ForEach-Object { Say " $_" 'White' }
|
||||
if ($LASTEXITCODE -ne 0) { Die 'SBOM generation failed' }
|
||||
|
||||
# Docs the running site serves, plus the runbooks an air-gapped server has no
|
||||
# other way to reach. Without openapi.json and llms.txt the self-hosted /api/docs
|
||||
# page is broken on every installed server.
|
||||
Step 'Staging docs'
|
||||
$docsOut = Join-Path $AppOut 'docs'
|
||||
New-Item -ItemType Directory -Path $docsOut -Force | Out-Null
|
||||
foreach ($doc in @('openapi.json', 'llms.txt', 'api-inventory.json',
|
||||
'INSTALL-WINDOWS.md', 'OPERATE-WINDOWS.md',
|
||||
'BACKUP-RESTORE.md', 'UPGRADE.md')) {
|
||||
$src = Join-Path $RepoRoot ('docs\' + $doc)
|
||||
if (Test-Path $src) { Copy-Item $src $docsOut -Force; Say " $doc" }
|
||||
}
|
||||
|
||||
# Stage the profile INTO the tree: `flask plugin apply-profile` at provisioning
|
||||
# reads the same profile the tree was staged from, so the installed plugin set
|
||||
# and the shipped plugin code cannot drift.
|
||||
Copy-Item $Profile (Join-Path $AppOut 'site-profile.json') -Force
|
||||
|
||||
# The installer expects frontend\dist and frontend\dist-subpath.
|
||||
$feOut = Join-Path $AppOut 'frontend'
|
||||
New-Item -ItemType Directory -Path $feOut -Force | Out-Null
|
||||
Copy-Item $rootStaged (Join-Path $feOut 'dist') -Recurse -Force
|
||||
Copy-Item $subStaged (Join-Path $feOut 'dist-subpath') -Recurse -Force
|
||||
|
||||
Get-ChildItem $AppOut -Recurse -Directory -Filter '__pycache__' -ErrorAction SilentlyContinue |
|
||||
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Get-ChildItem $AppOut -Recurse -File -Filter '*.pyc' -ErrorAction SilentlyContinue |
|
||||
Remove-Item -Force -ErrorAction SilentlyContinue
|
||||
|
||||
# --- 4. plugins.iss ---------------------------------------------------------
|
||||
Step 'Writing plugins.iss'
|
||||
$shipped = (Get-ChildItem (Join-Path $AppOut 'plugins') -Directory | Select-Object -ExpandProperty Name) -join ','
|
||||
$aliasBuilt = ''
|
||||
$aliasFile = Join-Path $feOut 'dist-subpath\.alias'
|
||||
if (Test-Path $aliasFile) { $aliasBuilt = (Get-Content $aliasFile -TotalCount 1).Trim() }
|
||||
@"
|
||||
; GENERATED by build-installer.ps1 - do not edit.
|
||||
; The plugins present in bundle\app\plugins at build time.
|
||||
#define AvailablePlugins "$shipped"
|
||||
; The alias the subpath SPA was built for, or empty if this bundle has no
|
||||
; subpath build - in which case the wizard must not offer that option.
|
||||
#define SubpathAlias "$aliasBuilt"
|
||||
"@ | Set-Content -Path (Join-Path $PSScriptRoot 'plugins.iss') -Encoding ASCII
|
||||
Say " $shipped" 'White'
|
||||
|
||||
# The product version, read from the code rather than restated in the .iss.
|
||||
Step 'Writing version.iss'
|
||||
$initText = Get-Content (Join-Path $RepoRoot 'shopdb\__init__.py') -Raw
|
||||
if ($initText -notmatch "(?m)^__version__\s*=\s*'([^']+)'") { Die 'could not read __version__ from shopdb\__init__.py' }
|
||||
$appVersion = $Matches[1]
|
||||
@"
|
||||
; GENERATED by build-installer.ps1 from shopdb/__init__.py - do not edit.
|
||||
#define AppVersion "$appVersion"
|
||||
"@ | Set-Content -Path (Join-Path $PSScriptRoot 'version.iss') -Encoding ASCII
|
||||
Say " $appVersion" 'White'
|
||||
|
||||
# --- 5. installer scripts ---------------------------------------------------
|
||||
# From THIS directory, which is the reviewed copy under version control. They
|
||||
# used to be copied from a downloads folder, so the logic that shipped was not
|
||||
# the logic that was committed and the build only worked on one machine.
|
||||
Step 'Copying installer scripts'
|
||||
foreach ($f in @('shopdb-install.ps1', 'shopdb-preflight.ps1', 'bundle-lock.ps1')) {
|
||||
$src = Join-Path $PSScriptRoot $f
|
||||
if (-not (Test-Path $src)) { Die "installer script missing from the repo: $f" }
|
||||
Copy-Item $src $BundleRoot -Force
|
||||
Say " $f"
|
||||
}
|
||||
|
||||
# --- 6. payload verification ------------------------------------------------
|
||||
Step 'Verifying the third-party payload against bundle-lock.json'
|
||||
$lockPath = Join-Path $PSScriptRoot 'bundle-lock.json'
|
||||
$lock = Read-BundleLock $lockPath
|
||||
if (-not $lock) {
|
||||
$msg = "no bundle-lock.json at $lockPath"
|
||||
if ($AllowUnlocked) { Say " $msg - continuing because -AllowUnlocked was passed" 'Yellow' }
|
||||
else {
|
||||
Die $msg @'
|
||||
Add the wheels and installers to the bundle, then:
|
||||
.\refresh-bundle-lock.ps1 (review the list)
|
||||
.\refresh-bundle-lock.ps1 -Yes (write it)
|
||||
and commit bundle-lock.json. Pass -AllowUnlocked to stage a bundle without one -
|
||||
it must not be shipped.
|
||||
'@
|
||||
}
|
||||
} else {
|
||||
$problems = Test-BundleLock -BundleRoot $BundleRoot -Lock $lock
|
||||
if ($problems.Count -eq 0) {
|
||||
Say (" payload matches the lock ({0}, {1})" -f `
|
||||
(Get-JsonProperty $lock 'pythontag' 'unknown'), (Get-JsonProperty $lock 'platform' 'unknown')) 'Green'
|
||||
# Ships WITH the bundle: the installer re-checks the payload on the
|
||||
# target server before running any of it, so tampering between build and
|
||||
# install is caught too.
|
||||
Copy-Item $lockPath $BundleRoot -Force
|
||||
} else {
|
||||
Say ''
|
||||
foreach ($p in $problems) { Say " $p" 'Red' }
|
||||
Say ''
|
||||
if ($AllowUnlocked) {
|
||||
Say ' -AllowUnlocked: continuing anyway. DO NOT SHIP this bundle.' 'Yellow'
|
||||
} else {
|
||||
Die ("{0} payload problem(s) - the bundle is not what the lock describes" -f $problems.Count) @'
|
||||
Either the payload is wrong (fix the bundle) or it changed on purpose (run
|
||||
refresh-bundle-lock.ps1, read the diff, and commit the new lock).
|
||||
'@
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- summary ----------------------------------------------------------------
|
||||
Write-Host ''
|
||||
Say " Bundle staged at: $BundleRoot" 'White'
|
||||
foreach ($d in @('app', 'wheels', 'python', 'httpplatformhandler', 'urlrewrite', 'vcredist', 'mysqlclient', 'mysql')) {
|
||||
$p = Join-Path $BundleRoot $d
|
||||
if (Test-Path $p) {
|
||||
$mb = ((Get-ChildItem $p -Recurse -File | Measure-Object Length -Sum).Sum / 1MB)
|
||||
Say (" {0,-22} {1,8:N1} MB" -f $d, $mb)
|
||||
} else {
|
||||
Say (" {0,-22} {1}" -f $d, 'absent') 'DarkGray'
|
||||
}
|
||||
}
|
||||
Write-Host ''
|
||||
Say ' Compile with: iscc ShopDBFlask.iss' 'White'
|
||||
Say ' (Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)' 'DarkGray'
|
||||
Write-Host ''
|
||||
@@ -1,141 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Stage a lean per-site bundle next to ShopDBFlask.iss, ready for Inno Setup.
|
||||
#
|
||||
# The bundle is built FOR ONE SITE from its plugin profile (ADR-013): plugins the
|
||||
# site did not choose are absent from the payload entirely. Build one installer
|
||||
# per site, not one universal installer.
|
||||
#
|
||||
# Usage: build-installer.sh <site-profile.json> [repo-path]
|
||||
#
|
||||
# The wheelhouse cannot be built here. Wheels are cp314 win_amd64 and must be
|
||||
# produced ON Windows with the matching Python:
|
||||
# pip download -r requirements.txt -d wheels --only-binary=:all:
|
||||
# Copy that wheels\ directory in before compiling.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROFILE="${1:?usage: build-installer.sh <site-profile.json> [repo-path]}"
|
||||
REPO="${2:-$HOME/projects/shopdb-flask}"
|
||||
BUNDLE="$HERE/bundle"
|
||||
|
||||
[ -f "$PROFILE" ] || { echo "profile not found: $PROFILE"; exit 1; }
|
||||
[ -d "$REPO" ] || { echo "repo not found: $REPO"; exit 1; }
|
||||
|
||||
echo "==> Staging lean app tree from $PROFILE"
|
||||
rm -rf "$BUNDLE/app"
|
||||
bash "$REPO/scripts/build-site.sh" "$PROFILE" "$BUNDLE/app"
|
||||
|
||||
# build-site.sh emits the SPA as frontend-dist; the installer's web.config and
|
||||
# static route expect frontend\dist.
|
||||
if [ -d "$BUNDLE/app/frontend-dist" ]; then
|
||||
mkdir -p "$BUNDLE/app/frontend"
|
||||
rm -rf "$BUNDLE/app/frontend/dist"
|
||||
mv "$BUNDLE/app/frontend-dist" "$BUNDLE/app/frontend/dist"
|
||||
fi
|
||||
# The /shopdb-based build, used when the operator picks the subpath deployment.
|
||||
if [ -d "$BUNDLE/app/frontend-dist-subpath" ]; then
|
||||
rm -rf "$BUNDLE/app/frontend/dist-subpath"
|
||||
mv "$BUNDLE/app/frontend-dist-subpath" "$BUNDLE/app/frontend/dist-subpath"
|
||||
fi
|
||||
|
||||
# Tell the .iss which plugins this bundle actually carries, so the wizard's
|
||||
# plugin page always matches the payload instead of a hand-maintained list.
|
||||
echo "==> Writing plugins.iss"
|
||||
PLUGINS=$(ls "$BUNDLE/app/plugins" 2>/dev/null | tr '\n' ',' | sed 's/,$//')
|
||||
# The subpath SPA is built with its base path compiled in, so whether the wizard
|
||||
# can OFFER a subpath install is a property of the bundle, not a runtime choice.
|
||||
SUBPATH_ALIAS_BUILT=""
|
||||
if [ -f "$BUNDLE/app/frontend/dist-subpath/.alias" ]; then
|
||||
SUBPATH_ALIAS_BUILT="$(cat "$BUNDLE/app/frontend/dist-subpath/.alias")"
|
||||
fi
|
||||
|
||||
cat > "$HERE/plugins.iss" <<EOF
|
||||
; GENERATED by build-installer.sh - do not edit.
|
||||
; The plugins present in bundle\\app\\plugins at build time.
|
||||
#define AvailablePlugins "$PLUGINS"
|
||||
; The alias the subpath SPA was built for, or empty if this bundle has no
|
||||
; subpath build - in which case the wizard must not offer that option.
|
||||
#define SubpathAlias "$SUBPATH_ALIAS_BUILT"
|
||||
EOF
|
||||
echo " $PLUGINS"
|
||||
|
||||
# The product version, read from the code rather than restated here. A hardcoded
|
||||
# AppVersion in the .iss had drifted two minor versions from shopdb/__init__.py.
|
||||
echo "==> Writing version.iss"
|
||||
APPVERSION=$(sed -n "s/^__version__ = '\\(.*\\)'/\\1/p" "$REPO/shopdb/__init__.py" | head -1)
|
||||
[ -n "$APPVERSION" ] || { echo "could not read __version__ from shopdb/__init__.py"; exit 1; }
|
||||
cat > "$HERE/version.iss" <<EOF
|
||||
; GENERATED by build-installer.sh from shopdb/__init__.py - do not edit.
|
||||
#define AppVersion "$APPVERSION"
|
||||
EOF
|
||||
echo " $APPVERSION"
|
||||
|
||||
# From THIS directory, which is the reviewed copy under version control. These
|
||||
# used to be copied from $HOME/Downloads, so the installer logic that shipped was
|
||||
# not the logic that was committed, and the build only worked on one machine.
|
||||
echo "==> Copying installer scripts"
|
||||
mkdir -p "$BUNDLE"
|
||||
for f in shopdb-install.ps1 shopdb-preflight.ps1 bundle-lock.ps1; do
|
||||
[ -f "$HERE/$f" ] || { echo "installer script missing from the repo: $f"; exit 1; }
|
||||
cp "$HERE/$f" "$BUNDLE/"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Bundle staged at: $BUNDLE"
|
||||
for d in app wheels python httpplatformhandler urlrewrite vcredist mysqlclient mysql; do
|
||||
if [ -d "$BUNDLE/$d" ]; then
|
||||
printf ' %-20s %s\n' "$d" "$(du -sh "$BUNDLE/$d" | cut -f1)"
|
||||
else
|
||||
printf ' %-20s absent\n' "$d"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
echo " plugins shipped: $(ls "$BUNDLE/app/plugins" 2>/dev/null | tr '\n' ' ')"
|
||||
|
||||
# --- payload verification ---------------------------------------------------
|
||||
# The third-party payload is the part git does not record: the wheels, the Python
|
||||
# installer and the MSIs that run as SYSTEM on the target server. It must be
|
||||
# EXACTLY what bundle-lock.json describes - no missing file, no stale extra wheel
|
||||
# left over from a previous build, no changed content - or this is not a bundle
|
||||
# anyone reviewed. Previously a missing wheelhouse printed MISSING and the script
|
||||
# still exited 0, so an empty bundle compiled into a shippable installer and the
|
||||
# failure surfaced on an air-gapped server with no way to fix it.
|
||||
#
|
||||
# ALLOW_UNLOCKED=1 downgrades this to a warning, for assembling a bundle before
|
||||
# its lock exists. A bundle built that way must not be shipped.
|
||||
echo ""
|
||||
echo "==> Verifying the third-party payload against bundle-lock.json"
|
||||
if python3 "$HERE/verify_bundle_lock.py" "$BUNDLE" "$HERE/bundle-lock.json"; then
|
||||
echo " payload matches the lock"
|
||||
# Ships WITH the bundle: the installer re-checks the payload on the target
|
||||
# server before running any of it, so tampering between build and install is
|
||||
# caught too.
|
||||
cp "$HERE/bundle-lock.json" "$BUNDLE/"
|
||||
elif [ "${ALLOW_UNLOCKED:-0}" = "1" ]; then
|
||||
echo ""
|
||||
echo " ALLOW_UNLOCKED=1: continuing anyway. DO NOT SHIP this bundle."
|
||||
else
|
||||
echo ""
|
||||
echo " The bundle is not what the lock describes."
|
||||
echo ""
|
||||
echo " Add the missing pieces by hand:"
|
||||
echo " wheels/ pip download -r requirements.txt --only-binary=:all: \\"
|
||||
echo " --platform win_amd64 --python-version 314 \\"
|
||||
echo " --implementation cp --abi cp314 -d wheels"
|
||||
echo " python/ python-3.14.x-amd64.exe"
|
||||
echo " httpplatformhandler/ httpPlatformHandler_amd64.msi"
|
||||
echo " urlrewrite/ rewrite_amd64.msi (client-IP rule; see README)"
|
||||
echo " vcredist/ VC_redist.x64.exe (MySQL requires it)"
|
||||
echo " mysql/ mysql-8.4.x-winx64.msi (bundled-database option only)"
|
||||
echo ""
|
||||
echo " If the payload changed ON PURPOSE, regenerate and COMMIT the lock:"
|
||||
echo " pwsh ./refresh-bundle-lock.ps1 # review the diff"
|
||||
echo " pwsh ./refresh-bundle-lock.ps1 -Yes # write it"
|
||||
echo ""
|
||||
echo " To stage a bundle before its lock exists: ALLOW_UNLOCKED=1 $0 ..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Then compile on Windows: iscc ShopDBFlask.iss"
|
||||
echo "(Inno Setup 6.6.0 or newer - the wizard uses the windows11 custom style.)"
|
||||
@@ -1,240 +0,0 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"generated": "2026-08-04T23:11:22Z",
|
||||
"pythontag": "cp314",
|
||||
"platform": "win_amd64",
|
||||
"note": "Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.",
|
||||
"payloads": {
|
||||
"httpplatformhandler": {
|
||||
"files": {
|
||||
"httpPlatformHandler_amd64.msi": {
|
||||
"sha256": "90f8d4905a0ab4f2c95223b3c79e2807a0b74507747d240e43c4302e8db4b5ef",
|
||||
"size": 557056
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"python": {
|
||||
"files": {
|
||||
"python-3.14.6-amd64.exe": {
|
||||
"sha256": "14b3e9a710a3fcf0bd9b55ab6b60412bd91227563f813fc49040cabc0209e0bd",
|
||||
"size": 30774112
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"vcredist": {
|
||||
"files": {
|
||||
"VC_redist.x64.exe": {
|
||||
"sha256": "cc0ff0eb1dc3f5188ae6300faef32bf5beeba4bdd6e8e445a9184072096b713b",
|
||||
"size": 25635768
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"mysql": {
|
||||
"files": {
|
||||
"mysql-8.4.6-winx64.msi": {
|
||||
"sha256": "868885b2e221409f0170729bb30b2b2ce83440a1a4c735d19b3c225b51045762",
|
||||
"size": 135081984
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"wheels": {
|
||||
"files": {
|
||||
"email_validator-2.3.0-py3-none-any.whl": {
|
||||
"sha256": "80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4",
|
||||
"size": 35604
|
||||
},
|
||||
"cachelib-0.13.0-py3-none-any.whl": {
|
||||
"sha256": "8c8019e53b6302967d4e8329a504acf75e7bc46130291d30188a6e4e58162516",
|
||||
"size": 20914
|
||||
},
|
||||
"flask_caching-2.4.0-py3-none-any.whl": {
|
||||
"sha256": "d15b8135f055c4f28f6f7dbcf8d36a3de4af1224def975ae6e0b43cbfa684486",
|
||||
"size": 28727
|
||||
},
|
||||
"typing_extensions-4.15.0-py3-none-any.whl": {
|
||||
"sha256": "f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548",
|
||||
"size": 44614
|
||||
},
|
||||
"greenlet-3.5.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "3bc59be3945ae9750b9e7d45067d01ae3fe90ea5f9ade99239dabdd6e28a5033",
|
||||
"size": 239835
|
||||
},
|
||||
"charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f",
|
||||
"size": 159634
|
||||
},
|
||||
"colorama-0.4.6-py2.py3-none-any.whl": {
|
||||
"sha256": "4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6",
|
||||
"size": 25335
|
||||
},
|
||||
"Flask_Migrate-4.1.0-py3-none-any.whl": {
|
||||
"sha256": "24d8051af161782e0743af1b04a152d007bad9772b2bca67b7ec1e8ceeb3910d",
|
||||
"size": 21237
|
||||
},
|
||||
"markupsafe-3.0.3-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581",
|
||||
"size": 15341
|
||||
},
|
||||
"flask_marshmallow-1.5.0-py3-none-any.whl": {
|
||||
"sha256": "99951c77e5654111ed733811c6dc9310bfb4c3688c78a9e76f80b5ae0b2279a6",
|
||||
"size": 12161
|
||||
},
|
||||
"tzdata-2026.3-py2.py3-none-any.whl": {
|
||||
"sha256": "dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931",
|
||||
"size": 348168
|
||||
},
|
||||
"python_dotenv-1.2.2-py3-none-any.whl": {
|
||||
"sha256": "1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a",
|
||||
"size": 22101
|
||||
},
|
||||
"dnspython-2.8.0-py3-none-any.whl": {
|
||||
"sha256": "01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af",
|
||||
"size": 331094
|
||||
},
|
||||
"pymysql-1.1.3-py3-none-any.whl": {
|
||||
"sha256": "8164ba62c552f6105f3b11753352d0f16b90d1703ba67d81923d5a8a5d1c5289",
|
||||
"size": 45356
|
||||
},
|
||||
"packaging-26.3-py3-none-any.whl": {
|
||||
"sha256": "d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c",
|
||||
"size": 129956
|
||||
},
|
||||
"flask_jwt_extended-4.7.3-py2.py3-none-any.whl": {
|
||||
"sha256": "905ac807b52b5409bc9244dbcca434968c13ca6f9d91bffe7d4cb71e0e6231cb",
|
||||
"size": 22698
|
||||
},
|
||||
"urllib3-2.7.0-py3-none-any.whl": {
|
||||
"sha256": "9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897",
|
||||
"size": 131087
|
||||
},
|
||||
"marshmallow-4.3.0-py3-none-any.whl": {
|
||||
"sha256": "46c4fe6984707e3cbd485dfebbf0a59874f58d695aad05c1668d15e8c6e13b46",
|
||||
"size": 49148
|
||||
},
|
||||
"waitress-3.0.2-py3-none-any.whl": {
|
||||
"sha256": "c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e",
|
||||
"size": 56232
|
||||
},
|
||||
"idna-3.13-py3-none-any.whl": {
|
||||
"sha256": "892ea0cde124a99ce773decba204c5552b69c3c67ffd5f232eb7696135bc8bb3",
|
||||
"size": 68629
|
||||
},
|
||||
"flask-3.1.3-py3-none-any.whl": {
|
||||
"sha256": "f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c",
|
||||
"size": 103424
|
||||
},
|
||||
"werkzeug-3.1.8-py3-none-any.whl": {
|
||||
"sha256": "63a77fb8892bf28ebc3178683445222aa500e48ebad5ec77b0ad80f8726b1f50",
|
||||
"size": 226459
|
||||
},
|
||||
"flask_sqlalchemy-3.1.1-py3-none-any.whl": {
|
||||
"sha256": "4ba4be7f419dc72f4efd8802d69974803c37259dd42f3913b0dcf75c9447e0a0",
|
||||
"size": 25125
|
||||
},
|
||||
"marshmallow_sqlalchemy-1.5.0-py3-none-any.whl": {
|
||||
"sha256": "3865232672f3dd38c4d5e4e85fdedce76904200742c3594948a2d11d0af93258",
|
||||
"size": 16582
|
||||
},
|
||||
"alembic-1.18.4-py3-none-any.whl": {
|
||||
"sha256": "a5ed4adcf6d8a4cb575f3d759f071b03cd6e5c7618eb796cb52497be25bfe19a",
|
||||
"size": 263893
|
||||
},
|
||||
"tabulate-0.10.0-py3-none-any.whl": {
|
||||
"sha256": "f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3",
|
||||
"size": 39814
|
||||
},
|
||||
"cffi-2.1.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "1b96bfe2c4bd825681b7d311ad6d9b7280a091f43e8f63da5729638083cd3bfb",
|
||||
"size": 187937
|
||||
},
|
||||
"jinja2-3.1.6-py3-none-any.whl": {
|
||||
"sha256": "85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67",
|
||||
"size": 134899
|
||||
},
|
||||
"cryptography-50.0.0-cp311-abi3-win_amd64.whl": {
|
||||
"sha256": "bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30",
|
||||
"size": 3840395
|
||||
},
|
||||
"certifi-2026.4.22-py3-none-any.whl": {
|
||||
"sha256": "3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a",
|
||||
"size": 135707
|
||||
},
|
||||
"sqlalchemy-2.0.49-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "77641d299179c37b89cf2343ca9972c88bb6eef0d5fc504a2f86afd15cd5adf5",
|
||||
"size": 2144204
|
||||
},
|
||||
"blinker-1.9.0-py3-none-any.whl": {
|
||||
"sha256": "ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc",
|
||||
"size": 8458
|
||||
},
|
||||
"pycparser-3.0-py3-none-any.whl": {
|
||||
"sha256": "b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992",
|
||||
"size": 48172
|
||||
},
|
||||
"flask_cors-6.0.2-py3-none-any.whl": {
|
||||
"sha256": "e57544d415dfd7da89a9564e1e3a9e515042df76e12130641ca6f3f2f03b699a",
|
||||
"size": 13257
|
||||
},
|
||||
"requests-2.33.1-py3-none-any.whl": {
|
||||
"sha256": "4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a",
|
||||
"size": 64947
|
||||
},
|
||||
"pyjwt-2.12.1-py3-none-any.whl": {
|
||||
"sha256": "28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c",
|
||||
"size": 29726
|
||||
},
|
||||
"click-8.3.3-py3-none-any.whl": {
|
||||
"sha256": "a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613",
|
||||
"size": 110502
|
||||
},
|
||||
"mysql_connector_python-9.7.0-cp314-cp314-win_amd64.whl": {
|
||||
"sha256": "5a5abbc152bc28cb2e64a04605ecd9941eff6b0dc5f9528cb84adb873e9a1e49",
|
||||
"size": 18197576
|
||||
},
|
||||
"mako-1.3.12-py3-none-any.whl": {
|
||||
"sha256": "8f61569480282dbf557145ce441e4ba888be453c30989f879f0d652e39f53ea9",
|
||||
"size": 78521
|
||||
},
|
||||
"itsdangerous-2.2.0-py3-none-any.whl": {
|
||||
"sha256": "c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef",
|
||||
"size": 16234
|
||||
}
|
||||
},
|
||||
"required": true
|
||||
},
|
||||
"urlrewrite": {
|
||||
"files": {
|
||||
"rewrite_amd64_en-US.msi": {
|
||||
"sha256": "37342ff2f585f263f34f48e9de59eb1051d61015a8e967dbde4075716230a32a",
|
||||
"size": 6078464
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
},
|
||||
"mysqlclient": {
|
||||
"files": {
|
||||
"libcrypto-3-x64.dll": {
|
||||
"sha256": "e84ec61fc1b07a3c899ae39c1dc3f38591cdd310c043a388e7ab40a6828aa297",
|
||||
"size": 5187216
|
||||
},
|
||||
"mysqldump.exe": {
|
||||
"sha256": "329410b7a8ae3d68d38e6f5b781610770854d3f11b975060e231dae62c20dc46",
|
||||
"size": 7142528
|
||||
},
|
||||
"mysql.exe": {
|
||||
"sha256": "bd6880253c1853fc17b1e19129d6e5d9f9c86a2dccf2c7c9e6b878a10baa9770",
|
||||
"size": 7177336
|
||||
},
|
||||
"libssl-3-x64.dll": {
|
||||
"sha256": "e24fcd6c8ec31a14c97f520b5b960ed360c55ddf0935040eff297e511697633e",
|
||||
"size": 788616
|
||||
}
|
||||
},
|
||||
"required": false
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,261 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Hash manifest for the installer's third-party payload: create it, and check a
|
||||
bundle against it.
|
||||
|
||||
.DESCRIPTION
|
||||
Dot-source this. It defines two functions and runs nothing on its own:
|
||||
|
||||
New-BundleLock hash a staged bundle and return the lock object
|
||||
Test-BundleLock compare a staged bundle against a lock, return problems
|
||||
|
||||
WHAT IT COVERS, and why pip's own hash checking is not enough.
|
||||
|
||||
requirements.txt carries a sha256 for every wheel, so pip refuses an artifact
|
||||
upstream did not publish. Three gaps remain, and all three are what actually
|
||||
goes wrong with a hand-assembled offline bundle:
|
||||
|
||||
1. pip lists EVERY artifact of a pinned version - cffi 2.1.0 alone has 100
|
||||
hashes. It proves the wheel is genuine, not that it is the wheel this
|
||||
bundle was built and tested with.
|
||||
2. pip ignores extra files in the wheelhouse. A stale wheel left behind by
|
||||
a previous build sits there unnoticed until a resolve picks it up.
|
||||
3. pip says nothing about the rest of the payload - the Python installer,
|
||||
the HttpPlatformHandler MSI, URL Rewrite, MySQL. Those are executables
|
||||
that run as SYSTEM on the target server and were, until this file, the
|
||||
only unverified thing the installer would run.
|
||||
|
||||
So the lock records an exact file set with a sha256 and a byte size each, and
|
||||
verification is SET EQUALITY: a missing file, an unexpected extra file, or a
|
||||
changed file all fail. Nothing is skipped and nothing is "close enough".
|
||||
|
||||
The app tree is deliberately NOT covered. It is built from the repository on
|
||||
every run and changes with every commit; hashing it would make the lock churn
|
||||
constantly and train everyone to regenerate it without reading it. Git is the
|
||||
record for the app tree. This file is the record for everything that comes
|
||||
from outside the repository.
|
||||
|
||||
.NOTES
|
||||
Stock Windows PowerShell 5.1, and also runs under pwsh on Linux so the Bash
|
||||
builder can call the same checker.
|
||||
#>
|
||||
|
||||
# Payload directories under the bundle root. 'required' means the installer
|
||||
# cannot work without it; the optional ones are per-deployment choices, and an
|
||||
# absent optional directory is fine. A PRESENT directory is always checked in
|
||||
# full, optional or not.
|
||||
$script:BundlePayloads = @(
|
||||
@{ Name = 'wheels'; Required = $true; What = 'Python wheels for the offline install' }
|
||||
@{ Name = 'python'; Required = $true; What = 'the Python installer' }
|
||||
@{ Name = 'httpplatformhandler'; Required = $true; What = 'the IIS module that launches waitress' }
|
||||
@{ Name = 'urlrewrite'; Required = $false; What = 'IIS URL Rewrite, for the client-IP rule' }
|
||||
@{ Name = 'mysqlclient'; Required = $false; What = 'mysql/mysqldump, for backups against a remote database' }
|
||||
@{ Name = 'vcredist'; Required = $false; What = 'the Visual C++ runtime MySQL requires' }
|
||||
@{ Name = 'mysql'; Required = $false; What = 'MySQL, for the bundled-database option' }
|
||||
)
|
||||
|
||||
function Get-JsonProperty {
|
||||
# shopdb-install.ps1 runs under Set-StrictMode 2.0, where reading a property
|
||||
# that does not exist on a PSCustomObject THROWS instead of returning $null.
|
||||
# A truncated or hand-edited bundle-lock.json would therefore blow up with
|
||||
# "Property 'payloads' cannot be found" rather than saying what is wrong with
|
||||
# the lock. Every read of parsed JSON goes through here.
|
||||
param($Object, [string] $Name, $Default = $null)
|
||||
if ($null -eq $Object) { return $Default }
|
||||
$prop = $Object.PSObject.Properties[$Name]
|
||||
if ($null -eq $prop) { return $Default }
|
||||
return $prop.Value
|
||||
}
|
||||
|
||||
function Get-FileDigest {
|
||||
param([string] $Path)
|
||||
$sha = [System.Security.Cryptography.SHA256]::Create()
|
||||
$stream = [System.IO.File]::OpenRead($Path)
|
||||
try { return (-join ($sha.ComputeHash($stream) | ForEach-Object { $_.ToString('x2') })) }
|
||||
finally { $stream.Dispose(); $sha.Dispose() }
|
||||
}
|
||||
|
||||
function Get-PayloadFiles {
|
||||
<#
|
||||
Every file in the directory, keyed by its path RELATIVE to that directory
|
||||
with forward slashes, so a lock generated on Windows reads the same from
|
||||
the Bash builder.
|
||||
|
||||
The root comes from Get-Item, NOT Resolve-Path, and the prefix is checked
|
||||
before it is trimmed. Both matter, and a real install proved it:
|
||||
|
||||
Inno extracts the bundle under C:\Users\ADMINI~1\AppData\Local\Temp\... -
|
||||
an 8.3 SHORT path. Resolve-Path kept that short form while Get-ChildItem
|
||||
returned the long one (Administrator), so the root was five characters
|
||||
shorter than the prefix it was slicing off. Every relative path came out
|
||||
mangled - 'wheels/heels/flask.whl' - and the verifier reported all 96 files
|
||||
as simultaneously missing and unexpected. The payload was fine; the
|
||||
comparison was not.
|
||||
|
||||
Get-Item and Get-ChildItem go through the same provider, so their path
|
||||
forms agree. The StartsWith guard means that if they ever disagree again
|
||||
this fails loudly instead of inventing paths.
|
||||
#>
|
||||
param([string] $Dir)
|
||||
$out = @{}
|
||||
if (-not (Test-Path $Dir)) { return $out }
|
||||
$rootItem = Get-Item -LiteralPath $Dir
|
||||
$root = $rootItem.FullName.TrimEnd('\', '/')
|
||||
foreach ($f in (Get-ChildItem -LiteralPath $rootItem.FullName -Recurse -File)) {
|
||||
if (-not $f.FullName.StartsWith($root, [System.StringComparison]::OrdinalIgnoreCase)) {
|
||||
throw ("cannot place '{0}' beneath '{1}' - path forms disagree (8.3 short name?)" -f $f.FullName, $root)
|
||||
}
|
||||
$rel = $f.FullName.Substring($root.Length).TrimStart('\', '/').Replace('\', '/')
|
||||
$out[$rel] = @{ sha256 = (Get-FileDigest $f.FullName); size = $f.Length }
|
||||
}
|
||||
return $out
|
||||
}
|
||||
|
||||
function New-BundleLock {
|
||||
<#
|
||||
Hash a staged bundle. The caller writes the result to bundle-lock.json;
|
||||
this returns the object so a caller can diff it against the committed lock
|
||||
before overwriting anything.
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $BundleRoot,
|
||||
[string] $PythonTag = 'cp314',
|
||||
[string] $Platform = 'win_amd64'
|
||||
)
|
||||
$payloads = @{}
|
||||
foreach ($p in $script:BundlePayloads) {
|
||||
$dir = Join-Path $BundleRoot $p.Name
|
||||
if (-not (Test-Path $dir)) {
|
||||
if ($p.Required) { throw ("payload directory is missing: {0} ({1})" -f $p.Name, $p.What) }
|
||||
continue
|
||||
}
|
||||
$files = Get-PayloadFiles $dir
|
||||
if ($files.Count -eq 0 -and $p.Required) {
|
||||
throw ("payload directory is empty: {0} ({1})" -f $p.Name, $p.What)
|
||||
}
|
||||
$payloads[$p.Name] = @{ required = $p.Required; files = $files }
|
||||
}
|
||||
return [ordered]@{
|
||||
schema = 1
|
||||
generated = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')
|
||||
pythontag = $PythonTag
|
||||
platform = $Platform
|
||||
note = 'Exact third-party payload of the installer bundle. Regenerate with refresh-bundle-lock.ps1 and COMMIT the change as a reviewed dependency bump.'
|
||||
payloads = $payloads
|
||||
}
|
||||
}
|
||||
|
||||
function Test-BundleLock {
|
||||
<#
|
||||
Compare a staged bundle against a lock. Returns an array of problem
|
||||
strings - EMPTY means the bundle is exactly what the lock describes.
|
||||
|
||||
Returning problems rather than throwing is deliberate: an operator fixing a
|
||||
wheelhouse wants the whole list at once, not one failure per rebuild.
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string] $BundleRoot,
|
||||
[Parameter(Mandatory = $true)] $Lock
|
||||
)
|
||||
$problems = @()
|
||||
$payloads = Get-JsonProperty $Lock 'payloads'
|
||||
# ,@(...) for the same reason as the final return: a bare one-element array
|
||||
# unrolls to a string, and the caller's .Count then measures the wrong thing.
|
||||
if ($null -eq $payloads) { return ,@('bundle-lock.json has no "payloads" section') }
|
||||
|
||||
foreach ($p in $script:BundlePayloads) {
|
||||
$name = $p.Name
|
||||
$dir = Join-Path $BundleRoot $name
|
||||
$present = Test-Path $dir
|
||||
$entry = Get-JsonProperty $payloads $name
|
||||
$locked = ($null -ne $entry)
|
||||
|
||||
if (-not $locked) {
|
||||
# Not in the lock at all. An unlocked directory that exists is a
|
||||
# payload nobody reviewed, which is exactly what this is here to stop.
|
||||
if ($present) { $problems += "$name/ is present but is not in bundle-lock.json - regenerate the lock" }
|
||||
elseif ($p.Required) { $problems += "$name/ is required but is in neither the bundle nor the lock" }
|
||||
continue
|
||||
}
|
||||
if (-not $present) {
|
||||
if ($p.Required -or (Get-JsonProperty $entry 'required' $false)) {
|
||||
$problems += "$name/ is in the lock but missing from the bundle ($($p.What))"
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
$expected = @{}
|
||||
$lockedFiles = Get-JsonProperty $entry 'files'
|
||||
if ($null -ne $lockedFiles) {
|
||||
foreach ($prop in $lockedFiles.PSObject.Properties) { $expected[$prop.Name] = $prop.Value }
|
||||
}
|
||||
$actual = Get-PayloadFiles $dir
|
||||
|
||||
# Sorted so the report reads the same way twice, and so it matches the
|
||||
# order verify_bundle_lock.py produces.
|
||||
foreach ($rel in ($expected.Keys | Sort-Object)) {
|
||||
if (-not $actual.ContainsKey($rel)) { $problems += "$name/$rel is in the lock but missing from the bundle"; continue }
|
||||
if ($actual[$rel].sha256 -ne $expected[$rel].sha256) {
|
||||
$problems += "$name/$rel does NOT match the lock (expected sha256 $($expected[$rel].sha256.Substring(0,12))..., got $($actual[$rel].sha256.Substring(0,12))...)"
|
||||
} elseif ([int64] $actual[$rel].size -ne [int64] $expected[$rel].size) {
|
||||
# Cannot happen for a matching sha256, so it means the lock itself
|
||||
# was hand-edited. Say so rather than passing it.
|
||||
$problems += "$name/$rel size disagrees with the lock - the lock has been edited by hand"
|
||||
}
|
||||
}
|
||||
foreach ($rel in ($actual.Keys | Sort-Object)) {
|
||||
if (-not $expected.ContainsKey($rel)) { $problems += "$name/$rel is in the bundle but NOT in the lock (unexpected extra file)" }
|
||||
}
|
||||
}
|
||||
$problems += Test-WheelhouseCoversRequirements -BundleRoot $BundleRoot
|
||||
# The comma keeps this an ARRAY through the return. Without it PowerShell
|
||||
# unrolls an empty result to $null and a single result to a bare string, and
|
||||
# every caller that asks for .Count then behaves differently depending on how
|
||||
# many problems there happen to be.
|
||||
return ,$problems
|
||||
}
|
||||
|
||||
function Test-WheelhouseCoversRequirements {
|
||||
<#
|
||||
The lock records what IS in the wheelhouse, not what the application NEEDS.
|
||||
Without this an incomplete wheelhouse gets locked, blessed, and shipped,
|
||||
and the install fails on an air-gapped server.
|
||||
|
||||
Not hypothetical: assembling the wheelhouse anywhere other than Windows
|
||||
silently omits colorama, a win32-only dependency of click, because pip
|
||||
evaluates environment markers against the machine doing the downloading
|
||||
rather than the machine being targeted. Markers are therefore IGNORED here
|
||||
- a requirement guarded by sys_platform == 'win32' is precisely the one
|
||||
that has to be present.
|
||||
#>
|
||||
param([Parameter(Mandatory = $true)] [string] $BundleRoot)
|
||||
$wheels = Join-Path $BundleRoot 'wheels'
|
||||
$reqs = Join-Path $BundleRoot 'app\requirements.txt'
|
||||
if (-not (Test-Path $wheels) -or -not (Test-Path $reqs)) { return @() }
|
||||
|
||||
$have = @(Get-ChildItem $wheels -File -ErrorAction SilentlyContinue | ForEach-Object { $_.Name.ToLower() })
|
||||
$problems = @()
|
||||
$pins = @{}
|
||||
foreach ($line in (Get-Content $reqs)) {
|
||||
$trimmed = $line.Trim()
|
||||
if (-not $trimmed -or $trimmed.StartsWith('#')) { continue }
|
||||
if ($trimmed -match '^([A-Za-z0-9._-]+)==([^\s;\\]+)') {
|
||||
# PEP 427 wheel filename form: runs of non-alphanumerics become one _.
|
||||
$pins[([regex]::Replace($Matches[1], '[^A-Za-z0-9.]+', '_')).ToLower()] = $Matches[2]
|
||||
}
|
||||
}
|
||||
foreach ($name in ($pins.Keys | Sort-Object)) {
|
||||
$prefix = "$name-$($pins[$name])-"
|
||||
if (-not ($have | Where-Object { $_.StartsWith($prefix) })) {
|
||||
$problems += ("wheels/ has no wheel for {0}=={1}, which requirements.txt pins " +
|
||||
"(a marked-out dependency still installs on Windows)") -f $name, $pins[$name]
|
||||
}
|
||||
}
|
||||
return $problems
|
||||
}
|
||||
|
||||
function Read-BundleLock {
|
||||
param([Parameter(Mandatory = $true)] [string] $Path)
|
||||
if (-not (Test-Path $Path)) { return $null }
|
||||
return (Get-Content $Path -Raw | ConvertFrom-Json)
|
||||
}
|
||||
@@ -1,159 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate Inno Setup wizard artwork from the app's own brand assets.
|
||||
|
||||
Everything here is derived from frontend/public/*.svg so the installer and the
|
||||
running application are visibly the same product. Nothing is redrawn by hand.
|
||||
|
||||
Inno stretches artwork to fit and does not resample well, so render at the exact
|
||||
sizes it asks for and supply the 125%/250% variants for high-DPI displays.
|
||||
|
||||
WizardImageFile 164x314, 192x386, 384x772
|
||||
WizardSmallImageFile 55x55, 64x64, 138x138
|
||||
SetupIconFile .ico with 16/24/32/48/64/128/256
|
||||
|
||||
Usage: python3 make-branding.py [output-dir]
|
||||
"""
|
||||
import io
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import cairosvg
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
ASSETS = Path.home() / "projects/shopdb-flask/frontend/public"
|
||||
OUT = Path(sys.argv[1] if len(sys.argv) > 1 else Path(__file__).parent)
|
||||
|
||||
# Sampled from the application's own palette so the installer does not look like
|
||||
# a different product wearing the same badge.
|
||||
NAVY = (10, 34, 74) # deep base
|
||||
BLUE = (16, 74, 150) # GE blue
|
||||
CYAN = (0, 158, 224) # accent
|
||||
WHITE = (255, 255, 255)
|
||||
MUTED = (176, 197, 226)
|
||||
|
||||
|
||||
def render_svg(name, width=None, height=None):
|
||||
png = cairosvg.svg2png(url=str(ASSETS / name), output_width=width, output_height=height)
|
||||
return Image.open(io.BytesIO(png)).convert("RGBA")
|
||||
|
||||
|
||||
def recolour(img, colour):
|
||||
"""Replace RGB while keeping the alpha mask. Source marks are dark-on-light;
|
||||
on a dark panel they must be inverted or they disappear."""
|
||||
solid = Image.new("RGBA", img.size, colour + (255,))
|
||||
solid.putalpha(img.getchannel("A"))
|
||||
return solid
|
||||
|
||||
|
||||
def font(size, bold=False):
|
||||
for path in (
|
||||
f"/usr/share/fonts/truetype/dejavu/DejaVuSans{'-Bold' if bold else ''}.ttf",
|
||||
f"/usr/share/fonts/truetype/liberation/LiberationSans{'-Bold' if bold else '-Regular'}.ttf",
|
||||
):
|
||||
if Path(path).exists():
|
||||
return ImageFont.truetype(path, size)
|
||||
return ImageFont.load_default()
|
||||
|
||||
|
||||
def vertical_gradient(size, top, bottom):
|
||||
w, h = size
|
||||
img = Image.new("RGB", size)
|
||||
draw = ImageDraw.Draw(img)
|
||||
for y in range(h):
|
||||
t = y / max(1, h - 1)
|
||||
# Ease the ramp so the middle does not look flat.
|
||||
t = t * t * (3 - 2 * t)
|
||||
draw.line(
|
||||
[(0, y), (w, y)],
|
||||
fill=tuple(int(top[i] + (bottom[i] - top[i]) * t) for i in range(3)),
|
||||
)
|
||||
return img
|
||||
|
||||
|
||||
def banner(w, h):
|
||||
img = vertical_gradient((w, h), BLUE, NAVY)
|
||||
draw = ImageDraw.Draw(img)
|
||||
k = w / 164.0 # scale factor from the 100% design
|
||||
|
||||
# Faint diagonal wash: stops the flat area under the text reading as empty.
|
||||
glow = Image.new("RGBA", (w, h), (0, 0, 0, 0))
|
||||
gd = ImageDraw.Draw(glow)
|
||||
gd.polygon([(0, int(h * 0.52)), (w, int(h * 0.30)), (w, h), (0, h)],
|
||||
fill=(255, 255, 255, 10))
|
||||
img = Image.alpha_composite(img.convert("RGBA"), glow).convert("RGB")
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
margin = int(22 * k)
|
||||
|
||||
# GE Aerospace wordmark at the top, above the product name - the corporate
|
||||
# mark leads, the product sits under it. (Previously the bare monogram was
|
||||
# here and the wordmark was stranded at the bottom.)
|
||||
mark_w = w - (margin * 2)
|
||||
mark = render_svg("ge-aerospace-logo.svg", mark_w, int(mark_w * 32 / 138))
|
||||
mark = recolour(mark, WHITE)
|
||||
img.paste(mark, (margin, int(34 * k)), mark)
|
||||
|
||||
# Product name, directly beneath it.
|
||||
y = int(34 * k) + mark.height + int(30 * k)
|
||||
draw.text((margin, y), "ShopDB", font=font(int(26 * k), bold=True), fill=WHITE)
|
||||
y += int(31 * k)
|
||||
|
||||
# Hairline rule, then the descriptor. Cheap way to look considered.
|
||||
draw.rectangle([margin, y, margin + int(30 * k), y + max(1, int(2 * k))], fill=CYAN)
|
||||
y += int(14 * k)
|
||||
for line in ("Asset management", "for the shop floor"):
|
||||
draw.text((margin, y), line, font=font(int(10.5 * k)), fill=MUTED)
|
||||
y += int(15 * k)
|
||||
|
||||
# Accent bar flush to the bottom edge.
|
||||
bar = max(2, int(4 * k))
|
||||
draw.rectangle([0, h - bar, w, h], fill=CYAN)
|
||||
return img
|
||||
|
||||
|
||||
def small(size):
|
||||
"""Header mark on every page after the welcome page. White plate so it sits
|
||||
correctly on the wizard's own header, in light or dark mode."""
|
||||
img = Image.new("RGB", (size, size), WHITE)
|
||||
m = int(size * 0.80)
|
||||
mono = recolour(render_svg("ge-monogram.svg", m, m), BLUE)
|
||||
off = (size - m) // 2
|
||||
img.paste(mono, (off, off), mono)
|
||||
return img
|
||||
|
||||
|
||||
def icon(path):
|
||||
"""Installer icon. Rounded navy tile with the monogram, so it reads at 16px
|
||||
instead of turning into mush."""
|
||||
base = 256
|
||||
img = Image.new("RGBA", (base, base), (0, 0, 0, 0))
|
||||
d = ImageDraw.Draw(img)
|
||||
d.rounded_rectangle([0, 0, base - 1, base - 1], radius=int(base * 0.22), fill=BLUE + (255,))
|
||||
d.rounded_rectangle([0, 0, base - 1, int(base * 0.5)], radius=int(base * 0.22),
|
||||
fill=(30, 96, 175, 255))
|
||||
d.rounded_rectangle([0, int(base * 0.3), base - 1, base - 1], radius=int(base * 0.22),
|
||||
fill=BLUE + (255,))
|
||||
m = int(base * 0.62)
|
||||
mono = recolour(render_svg("ge-monogram.svg", m, m), WHITE)
|
||||
img.paste(mono, ((base - m) // 2, (base - m) // 2), mono)
|
||||
img.save(path, sizes=[(16, 16), (24, 24), (32, 32), (48, 48), (64, 64),
|
||||
(128, 128), (256, 256)])
|
||||
|
||||
|
||||
def main():
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
made = []
|
||||
for w, h, name in [(164, 314, "wizard-image.bmp"),
|
||||
(192, 386, "wizard-image@125.bmp"),
|
||||
(384, 772, "wizard-image@250.bmp")]:
|
||||
banner(w, h).save(OUT / name, "BMP"); made.append((name, f"{w}x{h}"))
|
||||
for s, name in [(55, "wizard-small.bmp"), (64, "wizard-small@125.bmp"),
|
||||
(138, "wizard-small@250.bmp")]:
|
||||
small(s).save(OUT / name, "BMP"); made.append((name, f"{s}x{s}"))
|
||||
icon(OUT / "shopdb.ico"); made.append(("shopdb.ico", "multi-size"))
|
||||
for name, dims in made:
|
||||
print(f" {name:<26} {dims:<10} {(OUT / name).stat().st_size // 1024} KB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,110 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Regenerate bundle-lock.json from the staged bundle, after showing what would
|
||||
change.
|
||||
|
||||
.DESCRIPTION
|
||||
Run this on the machine that assembled the wheelhouse, once the bundle holds
|
||||
the payload you intend to ship:
|
||||
|
||||
bundle\wheels\ the wheels, built with the matching Python
|
||||
bundle\python\ the Python installer
|
||||
bundle\httpplatformhandler\ the IIS module MSI
|
||||
bundle\urlrewrite\ URL Rewrite MSI (optional)
|
||||
bundle\mysql\ MySQL MSI (optional)
|
||||
|
||||
Then COMMIT the resulting bundle-lock.json. That commit is the review: it is
|
||||
the only place a change to what runs as SYSTEM on a customer's server becomes
|
||||
visible to a human. A lock regenerated and committed without reading the diff
|
||||
provides nothing, so this refuses to overwrite an existing lock until you
|
||||
have seen the change and passed -Yes.
|
||||
|
||||
.EXAMPLE
|
||||
.\refresh-bundle-lock.ps1 # show the diff, write nothing
|
||||
.\refresh-bundle-lock.ps1 -Yes # write it
|
||||
|
||||
.NOTES
|
||||
Building the wheelhouse itself no longer requires Windows. From any machine:
|
||||
pip download -r requirements.txt -d wheels --only-binary=:all: `
|
||||
--platform win_amd64 --python-version 314 --implementation cp --abi cp314
|
||||
Do it wherever you like; this script records what came out.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string] $BundleRoot = (Join-Path $PSScriptRoot 'bundle'),
|
||||
[string] $LockPath = (Join-Path $PSScriptRoot 'bundle-lock.json'),
|
||||
[string] $PythonTag = 'cp314',
|
||||
[string] $Platform = 'win_amd64',
|
||||
[switch] $Yes
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
. (Join-Path $PSScriptRoot 'bundle-lock.ps1')
|
||||
|
||||
function Say { param($m, $c = 'Gray') Write-Host $m -ForegroundColor $c }
|
||||
|
||||
if (-not (Test-Path $BundleRoot)) {
|
||||
Say "bundle not found: $BundleRoot" 'Red'
|
||||
Say 'Stage it first with build-installer.ps1 (or build-installer.sh), then add' 'Yellow'
|
||||
Say 'the wheels and installers by hand.' 'Yellow'
|
||||
exit 1
|
||||
}
|
||||
|
||||
Say ''
|
||||
Say " Hashing $BundleRoot" 'Cyan'
|
||||
$new = New-BundleLock -BundleRoot $BundleRoot -PythonTag $PythonTag -Platform $Platform
|
||||
|
||||
foreach ($name in $new.payloads.Keys) {
|
||||
Say (" {0,-22} {1,4} files" -f $name, $new.payloads[$name].files.Count)
|
||||
}
|
||||
|
||||
$old = Read-BundleLock $LockPath
|
||||
if (-not $old) {
|
||||
Say ''
|
||||
Say ' No existing lock - this will be the first one.' 'Yellow'
|
||||
} else {
|
||||
# Diff by file, per payload, so the operator sees exactly which artifacts
|
||||
# changed rather than "the lock is different".
|
||||
Say ''
|
||||
Say ' Changes against the committed lock:' 'Cyan'
|
||||
$changes = 0
|
||||
foreach ($name in $new.payloads.Keys) {
|
||||
$oldFiles = @{}
|
||||
if ($old.payloads.PSObject.Properties.Name -contains $name) {
|
||||
foreach ($p in $old.payloads.$name.files.PSObject.Properties) { $oldFiles[$p.Name] = $p.Value.sha256 }
|
||||
}
|
||||
$newFiles = $new.payloads[$name].files
|
||||
foreach ($rel in ($newFiles.Keys | Sort-Object)) {
|
||||
if (-not $oldFiles.ContainsKey($rel)) { Say " + $name/$rel" 'Green'; $changes++ }
|
||||
elseif ($oldFiles[$rel] -ne $newFiles[$rel].sha256) { Say " ~ $name/$rel (content changed)" 'Yellow'; $changes++ }
|
||||
}
|
||||
foreach ($rel in ($oldFiles.Keys | Sort-Object)) {
|
||||
if (-not $newFiles.ContainsKey($rel)) { Say " - $name/$rel" 'Red'; $changes++ }
|
||||
}
|
||||
}
|
||||
foreach ($p in $old.payloads.PSObject.Properties.Name) {
|
||||
if (-not $new.payloads.Contains($p)) { Say " - $p/ (whole payload gone)" 'Red'; $changes++ }
|
||||
}
|
||||
if ($changes -eq 0) {
|
||||
Say ' none - the bundle already matches the lock' 'Green'
|
||||
exit 0
|
||||
}
|
||||
Say ''
|
||||
Say (" {0} change(s)." -f $changes) 'White'
|
||||
}
|
||||
|
||||
if (-not $Yes) {
|
||||
Say ''
|
||||
Say ' Nothing written. Read the list above, then re-run with -Yes.' 'Yellow'
|
||||
Say ' Commit the resulting bundle-lock.json - that commit IS the review.' 'Yellow'
|
||||
exit 2
|
||||
}
|
||||
|
||||
# ConvertTo-Json defaults to a depth of 2, which silently flattens the per-file
|
||||
# entries into "System.Collections.Hashtable" strings and produces a lock that
|
||||
# verifies against nothing.
|
||||
$new | ConvertTo-Json -Depth 8 | Set-Content -Path $LockPath -Encoding UTF8
|
||||
Say ''
|
||||
Say " Written: $LockPath" 'Green'
|
||||
Say ' Commit it.' 'Green'
|
||||
@@ -1,562 +0,0 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
ShopDB-Flask installer - Stage 1: read-only preflight.
|
||||
|
||||
.DESCRIPTION
|
||||
Discovers everything the installer needs to know about this box and reports
|
||||
it. Makes NO changes: no installs, no config edits, no service restarts.
|
||||
Safe to run on a production server.
|
||||
|
||||
Written for stock Windows PowerShell 5.1 (Windows Server ships it). No
|
||||
pwsh-only syntax, no external modules, no network access.
|
||||
|
||||
.PARAMETER SitePort
|
||||
The port the ShopDB site will listen on. Default 8090 (the runbook's example;
|
||||
the classic ASP site keeps 8080).
|
||||
|
||||
.PARAMETER AppRoot
|
||||
Intended install directory. Default C:\shopdb-flask.
|
||||
|
||||
.PARAMETER Json
|
||||
Emit machine-readable JSON instead of the human report. Later installer
|
||||
stages consume this.
|
||||
|
||||
.EXAMPLE
|
||||
powershell -ExecutionPolicy Bypass -File .\shopdb-preflight.ps1
|
||||
powershell -ExecutionPolicy Bypass -File .\shopdb-preflight.ps1 -Json > preflight.json
|
||||
|
||||
.NOTES
|
||||
Exit 0 = no blocking problems. Exit 1 = at least one FAIL.
|
||||
#>
|
||||
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int] $SitePort = 8090,
|
||||
[string] $AppRoot = 'C:\shopdb-flask',
|
||||
# Needed so the port check can tell OUR site apart from a stranger's.
|
||||
[string] $SiteName = 'shopdb-flask',
|
||||
[switch] $Json,
|
||||
# Machine-readable output for a GUI caller: one record per line,
|
||||
# STATUS|AREA|CHECK|DETAIL|FIX
|
||||
# The console rendering below aligns columns with padding spaces, which only
|
||||
# works in a fixed-width font at console width. A GUI must do its own layout,
|
||||
# so give it DATA and let it decide - do not make it parse a formatted table.
|
||||
# (-Json exists too, but Inno's Pascal Script has no JSON parser.)
|
||||
[switch] $Delimited
|
||||
)
|
||||
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# --- result collection -------------------------------------------------------
|
||||
# Every check appends one record. Status is PASS / WARN / FAIL / INFO.
|
||||
$script:Results = New-Object System.Collections.ArrayList
|
||||
$script:IisPresent = $false
|
||||
|
||||
function Add-Result {
|
||||
param(
|
||||
[string] $Area,
|
||||
[string] $Check,
|
||||
[ValidateSet('PASS','WARN','FAIL','INFO','SKIP')] [string] $Status,
|
||||
[string] $Detail,
|
||||
[string] $Fix = ''
|
||||
)
|
||||
$null = $script:Results.Add([PSCustomObject]@{
|
||||
Area = $Area
|
||||
Check = $Check
|
||||
Status = $Status
|
||||
Detail = $Detail
|
||||
Fix = $Fix
|
||||
})
|
||||
}
|
||||
|
||||
# Wrap a check so one failure cannot abort the whole run. On an unfamiliar box
|
||||
# an unexpected exception is itself a finding, not a crash.
|
||||
function Invoke-Check {
|
||||
param([string] $Area, [string] $Check, [scriptblock] $Body)
|
||||
try { & $Body }
|
||||
catch {
|
||||
Add-Result $Area $Check 'WARN' "check could not run: $($_.Exception.Message)" `
|
||||
'Report this output; the installer needs to handle this box shape.'
|
||||
}
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 1. Operator context
|
||||
# =============================================================================
|
||||
|
||||
Invoke-Check 'System' 'Elevation' {
|
||||
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$adm = (New-Object Security.Principal.WindowsPrincipal($id)).IsInRole(
|
||||
[Security.Principal.WindowsBuiltInRole]::Administrator)
|
||||
if ($adm) { Add-Result 'System' 'Elevation' 'PASS' 'running as Administrator' }
|
||||
else {
|
||||
Add-Result 'System' 'Elevation' 'FAIL' 'not elevated' `
|
||||
'Re-run PowerShell as Administrator. IIS and service changes require it.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'System' 'Windows version' {
|
||||
$os = Get-CimInstance Win32_OperatingSystem
|
||||
$name = $os.Caption
|
||||
$ver = $os.Version
|
||||
# ProductType: 1 = workstation, 2 = domain controller, 3 = server
|
||||
$isServer = ($os.ProductType -ne 1)
|
||||
$detail = "$name (build $ver), $(if ($isServer) {'Server'} else {'Client'})"
|
||||
|
||||
$supported = $false
|
||||
if ($isServer -and [version]$ver -ge [version]'10.0.17763') { $supported = $true } # 2019+
|
||||
if (-not $isServer -and [version]$ver -ge [version]'10.0.19045') { $supported = $true } # Win10 22H2+
|
||||
|
||||
if ($supported) { Add-Result 'System' 'Windows version' 'PASS' $detail }
|
||||
else {
|
||||
Add-Result 'System' 'Windows version' 'FAIL' $detail `
|
||||
'Supported: Windows Server 2019/2022+, or Windows 10 22H2 / 11 Pro+.'
|
||||
}
|
||||
|
||||
# Client SKUs must be Pro/Enterprise/Education for IIS.
|
||||
if (-not $isServer -and $name -match 'Home') {
|
||||
Add-Result 'System' 'Windows edition' 'FAIL' 'Windows Home edition' `
|
||||
'IIS is not available on Home editions. Pro or higher is required.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'System' 'Architecture' {
|
||||
if ([Environment]::Is64BitOperatingSystem) {
|
||||
Add-Result 'System' 'Architecture' 'PASS' '64-bit'
|
||||
} else {
|
||||
Add-Result 'System' 'Architecture' 'FAIL' '32-bit' `
|
||||
'The bundled Python and wheels are 64-bit (win_amd64) only.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'System' 'PowerShell version' {
|
||||
$v = $PSVersionTable.PSVersion
|
||||
Add-Result 'System' 'PowerShell version' 'INFO' "$v"
|
||||
if ($v.Major -lt 5) {
|
||||
Add-Result 'System' 'PowerShell version' 'FAIL' "$v" `
|
||||
'PowerShell 5.1 or later is required.'
|
||||
}
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 2. Disk and ports
|
||||
# =============================================================================
|
||||
|
||||
Invoke-Check 'Disk' 'Free space' {
|
||||
$drive = (Split-Path -Qualifier $AppRoot)
|
||||
$d = Get-CimInstance Win32_LogicalDisk -Filter "DeviceID='$drive'"
|
||||
if ($null -eq $d) {
|
||||
Add-Result 'Disk' 'Free space' 'FAIL' "drive $drive not found" `
|
||||
"Choose an -AppRoot on an existing volume."
|
||||
return
|
||||
}
|
||||
$freeGB = [math]::Round($d.FreeSpace / 1GB, 1)
|
||||
$detail = "$freeGB GB free on $drive"
|
||||
if ($freeGB -ge 5) { Add-Result 'Disk' 'Free space' 'PASS' $detail }
|
||||
else { Add-Result 'Disk' 'Free space' 'FAIL' $detail 'At least 5 GB is required.' }
|
||||
}
|
||||
|
||||
Invoke-Check 'Disk' 'AppRoot' {
|
||||
if (Test-Path $AppRoot) {
|
||||
$existing = @(Get-ChildItem $AppRoot -Force -ErrorAction SilentlyContinue)
|
||||
if ($existing.Count -gt 0) {
|
||||
$hasEnv = Test-Path (Join-Path $AppRoot '.env')
|
||||
if ($hasEnv) {
|
||||
# An existing install is the NORMAL state for an upgrade. Reporting
|
||||
# it as a warning makes a routine update look like a problem.
|
||||
$ver = ''
|
||||
$vf = Join-Path $AppRoot '.installed-version'
|
||||
if (Test-Path $vf) { $ver = ' version ' + (Get-Content $vf -TotalCount 1).Trim() }
|
||||
Add-Result 'Disk' 'AppRoot' 'INFO' `
|
||||
"existing ShopDB-Flask install found$ver - it will be upgraded in place, and your settings and database are kept"
|
||||
} else {
|
||||
Add-Result 'Disk' 'AppRoot' 'WARN' "$AppRoot exists and is not empty" `
|
||||
'Confirm this directory is safe to install into.'
|
||||
}
|
||||
} else { Add-Result 'Disk' 'AppRoot' 'PASS' "$AppRoot exists and is empty" }
|
||||
} else { Add-Result 'Disk' 'AppRoot' 'PASS' "$AppRoot does not exist yet" }
|
||||
}
|
||||
|
||||
function Test-PortFree {
|
||||
param([int] $Port)
|
||||
# Get-NetTCPConnection is the reliable listener check on Server 2012R2+.
|
||||
try {
|
||||
$listening = @(Get-NetTCPConnection -State Listen -LocalPort $Port -ErrorAction SilentlyContinue)
|
||||
return ($listening.Count -eq 0)
|
||||
} catch {
|
||||
# Fall back to a bind attempt if the cmdlet is unavailable.
|
||||
try {
|
||||
$l = New-Object System.Net.Sockets.TcpListener([System.Net.IPAddress]::Any, $Port)
|
||||
$l.Start(); $l.Stop(); return $true
|
||||
} catch { return $false }
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'Network' 'Site port' {
|
||||
if (Test-PortFree $SitePort) {
|
||||
Add-Result 'Network' 'Site port' 'PASS' "TCP $SitePort is free"
|
||||
} else {
|
||||
$owner = ''
|
||||
try {
|
||||
$c = Get-NetTCPConnection -State Listen -LocalPort $SitePort -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||
if ($c) { $owner = " (pid $($c.OwningProcess): $((Get-Process -Id $c.OwningProcess -ErrorAction SilentlyContinue).ProcessName))" }
|
||||
} catch { }
|
||||
# Is the listener OUR OWN site? On a reinstall or upgrade the port is held
|
||||
# by the very application being upgraded, and blocking on that makes the
|
||||
# installer refuse to update anything it previously installed.
|
||||
$ours = $false
|
||||
try {
|
||||
Import-Module WebAdministration -ErrorAction SilentlyContinue
|
||||
$site = Get-Website -Name $SiteName -ErrorAction SilentlyContinue
|
||||
if ($site) {
|
||||
foreach ($b in $site.bindings.Collection) {
|
||||
# ${} is required: "$SitePort:" parses as a DRIVE-qualified variable.
|
||||
if ($b.bindingInformation -match ":${SitePort}:") { $ours = $true }
|
||||
}
|
||||
}
|
||||
} catch { }
|
||||
|
||||
if ($ours) {
|
||||
Add-Result 'Network' 'Site port' 'INFO' `
|
||||
"TCP $SitePort is used by the existing $SiteName site - this will be upgraded in place"
|
||||
} else {
|
||||
# WARN, not FAIL. This check runs before the operator has reached the
|
||||
# Address page, so it is testing the DEFAULT port, not necessarily
|
||||
# the one they intend to use. Blocking here would refuse an install
|
||||
# over a conflict the very next page lets them resolve.
|
||||
Add-Result 'Network' 'Site port' 'WARN' "TCP $SitePort is in use$owner" `
|
||||
"Pick a different port on the Address page later in this wizard, or stop whatever is holding it."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 3. IIS
|
||||
# =============================================================================
|
||||
|
||||
Invoke-Check 'IIS' 'Installed' {
|
||||
$svc = Get-Service -Name W3SVC -ErrorAction SilentlyContinue
|
||||
$script:IisPresent = ($null -ne $svc)
|
||||
if ($null -eq $svc) {
|
||||
Add-Result 'IIS' 'Installed' 'FAIL' 'W3SVC service not found' `
|
||||
'Install IIS. Server: Install-WindowsFeature Web-Server -IncludeManagementTools. Client: enable Internet Information Services in Windows Features.'
|
||||
return
|
||||
}
|
||||
Add-Result 'IIS' 'Installed' 'PASS' "W3SVC present, status $($svc.Status)"
|
||||
if ($svc.Status -ne 'Running') {
|
||||
Add-Result 'IIS' 'Running' 'WARN' "W3SVC is $($svc.Status)" 'Start-Service W3SVC'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'IIS' 'WebAdministration module' {
|
||||
$m = Get-Module -ListAvailable -Name WebAdministration
|
||||
if ($m) { Add-Result 'IIS' 'WebAdministration module' 'PASS' 'available' }
|
||||
else {
|
||||
Add-Result 'IIS' 'WebAdministration module' 'FAIL' 'not available' `
|
||||
'Install the IIS management tools (Web-Mgmt-Console / IIS Management Scripts and Tools).'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'IIS' 'HttpPlatformHandler' {
|
||||
# The handler registers itself as a global module. Check the module list.
|
||||
$appcmd = Join-Path $env:windir 'system32\inetsrv\appcmd.exe'
|
||||
if (-not (Test-Path $appcmd)) {
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'SKIP' 'appcmd.exe not present (IIS not installed)' `
|
||||
'Re-run this preflight after installing IIS.'
|
||||
return
|
||||
}
|
||||
$modules = & $appcmd list module 2>$null
|
||||
if ($modules -match 'httpPlatformHandler') {
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'PASS' 'installed'
|
||||
} else {
|
||||
# NOT a blocker: the MSI is in the bundle and stage 4 installs it. This
|
||||
# was a FAIL, which - once the preflight page started blocking on any
|
||||
# failure - stopped the wizard dead over something the installer was
|
||||
# about to do by itself, with no way forward but to go and install it by
|
||||
# hand. Nothing the installer SUPPLIES may be a blocker.
|
||||
Add-Result 'IIS' 'HttpPlatformHandler' 'INFO' 'not installed yet' `
|
||||
'The installer installs it from the bundle. No action needed.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'IIS' 'Locked config sections' {
|
||||
# The authoritative source is applicationHost.config. `appcmd list config
|
||||
# /section:X` prints the section CONTENTS, not its lock state, so grepping
|
||||
# that output silently reports every section as unlocked.
|
||||
$cfg = Join-Path $env:windir 'system32\inetsrv\config\applicationHost.config'
|
||||
if (-not (Test-Path $cfg)) {
|
||||
Add-Result 'IIS' 'Locked config sections' 'SKIP' 'applicationHost.config not found'
|
||||
return
|
||||
}
|
||||
foreach ($name in @('handlers','httpPlatform')) {
|
||||
$line = Select-String -Path $cfg -Pattern ('<section name="' + $name + '"') |
|
||||
Select-Object -First 1
|
||||
if ($null -eq $line) {
|
||||
# httpPlatform is registered by the HttpPlatformHandler MSI. Before
|
||||
# that, `appcmd unlock config /section:system.webServer/httpPlatform`
|
||||
# fails with "Unknown config section".
|
||||
Add-Result 'IIS' "Section $name" 'SKIP' 'section not registered yet' `
|
||||
'Install HttpPlatformHandler FIRST; only then can this section be unlocked.'
|
||||
} elseif ($line.Line -match 'overrideModeDefault="Deny"') {
|
||||
Add-Result 'IIS' "Section $name" 'WARN' 'locked (overrideModeDefault="Deny")' `
|
||||
"Installer must run: appcmd unlock config /section:system.webServer/$name (else IIS 500.19)"
|
||||
} else {
|
||||
Add-Result 'IIS' "Section $name" 'PASS' 'not locked'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'IIS' 'URL Rewrite module' {
|
||||
$appcmd = Join-Path $env:windir 'system32\inetsrv\appcmd.exe'
|
||||
if (-not (Test-Path $appcmd)) {
|
||||
Add-Result 'IIS' 'URL Rewrite module' 'SKIP' 'IIS not installed; cannot check'
|
||||
return
|
||||
}
|
||||
$modules = & $appcmd list module 2>$null
|
||||
if ($modules -match 'RewriteModule') {
|
||||
Add-Result 'IIS' 'URL Rewrite module' 'PASS' 'installed (the X-Forwarded-For rule can be enabled)'
|
||||
} else {
|
||||
# Not a FAIL: it is only needed for -ClientIpSource direct, and the
|
||||
# installer carries the MSI and installs it itself. Worth reporting
|
||||
# because without the rule IIS sends no X-Forwarded-For at all, so every
|
||||
# client reads as 127.0.0.1 and the GE-Enforce IP allowlist, the
|
||||
# visitor-location lookup and per-host login rate limiting go quiet.
|
||||
Add-Result 'IIS' 'URL Rewrite module' 'INFO' 'not installed' `
|
||||
'The installer installs it from the bundle when -ClientIpSource is direct. Behind a reverse proxy that already sets X-Forwarded-For, use -ClientIpSource proxy and leave it out.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'IIS' 'Existing sites' {
|
||||
if (-not $script:IisPresent) {
|
||||
Add-Result 'IIS' 'Existing sites' 'SKIP' 'IIS not installed'
|
||||
return
|
||||
}
|
||||
try {
|
||||
Import-Module WebAdministration -ErrorAction Stop
|
||||
$sites = @(Get-Website)
|
||||
if ($sites.Count -eq 0) { Add-Result 'IIS' 'Existing sites' 'INFO' 'none' ; return }
|
||||
$desc = ($sites | ForEach-Object {
|
||||
$b = ($_.bindings.Collection | ForEach-Object { $_.bindingInformation }) -join ','
|
||||
"$($_.Name) [$($_.State)] $b"
|
||||
}) -join '; '
|
||||
Add-Result 'IIS' 'Existing sites' 'INFO' $desc
|
||||
# Adoption sites typically run the classic ASP shopdb here already.
|
||||
if ($desc -match '8080') {
|
||||
Add-Result 'IIS' 'Classic ASP site' 'INFO' 'a site is bound on 8080 (likely the classic ASP shopdb)' `
|
||||
'Install ShopDB as a separate site on its own port; do not disturb this one.'
|
||||
}
|
||||
} catch {
|
||||
Add-Result 'IIS' 'Existing sites' 'WARN' "could not enumerate: $($_.Exception.Message)" ''
|
||||
}
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 4. MySQL (detect BEFORE offering bundled vs existing)
|
||||
# =============================================================================
|
||||
|
||||
Invoke-Check 'MySQL' 'Service' {
|
||||
$svcs = @(Get-Service -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^MySQL' -or $_.DisplayName -match 'MySQL' })
|
||||
if ($svcs.Count -eq 0) {
|
||||
Add-Result 'MySQL' 'Service' 'INFO' 'no MySQL service found' `
|
||||
'Bundled MySQL 8.4 LTS is the appropriate choice on this box.'
|
||||
return
|
||||
}
|
||||
foreach ($s in $svcs) {
|
||||
Add-Result 'MySQL' 'Service' 'WARN' "$($s.Name) ($($s.DisplayName)) is $($s.Status)" `
|
||||
'MySQL already present. Default to the EXISTING-server option; installing bundled MySQL will collide on port 3306.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'MySQL' 'Port 3306' {
|
||||
if (Test-PortFree 3306) {
|
||||
Add-Result 'MySQL' 'Port 3306' 'INFO' 'nothing listening on 3306'
|
||||
} else {
|
||||
Add-Result 'MySQL' 'Port 3306' 'WARN' 'something is listening on 3306' `
|
||||
'Bundled MySQL cannot use the default port. Use the existing server, or pick another port.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'MySQL' 'Backup client' {
|
||||
# mysqldump is what takes the mandatory pre-upgrade backup. Without it every
|
||||
# upgrade skips the backup - and skips it AFTER the application pool has been
|
||||
# stopped and the tree replaced, so the site is down and there is nothing to
|
||||
# restore from. A site whose database is on another server typically has no
|
||||
# MySQL client installed here at all, which is exactly the case that needs it.
|
||||
$names = @('mysqldump.exe')
|
||||
$found = ''
|
||||
foreach ($root in @((Join-Path $PSScriptRoot 'mysqlclient'),
|
||||
(Join-Path $AppRoot 'mysqlclient'),
|
||||
'C:\MySQL84', 'C:\Program Files\MySQL', 'C:\mysql56\bin',
|
||||
'C:\Program Files (x86)\MySQL')) {
|
||||
if (-not (Test-Path $root)) { continue }
|
||||
$hit = Get-ChildItem $root -Filter $names[0] -Recurse -ErrorAction SilentlyContinue |
|
||||
Select-Object -First 1
|
||||
if ($hit) { $found = $hit.FullName; break }
|
||||
}
|
||||
if ($found) {
|
||||
Add-Result 'MySQL' 'Backup client' 'PASS' "mysqldump found ($found)"
|
||||
} else {
|
||||
Add-Result 'MySQL' 'Backup client' 'WARN' 'mysqldump not found on this server' `
|
||||
'Needed for the automatic pre-upgrade backup and for "shopdb-admin.ps1 backup". A first install works without it; upgrades will not be protected. It is not on this server yet - the installer stages its own copy, so this normally resolves itself during installation.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'MySQL' 'Version and config' {
|
||||
# Find mysqld.exe via the service binary path; read the version and locate my.ini.
|
||||
$svc = Get-CimInstance Win32_Service -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.PathName -match 'mysqld' } | Select-Object -First 1
|
||||
if ($null -eq $svc) { return }
|
||||
|
||||
$path = $svc.PathName
|
||||
$exe = ''
|
||||
if ($path -match '"([^"]+mysqld[^"]*)"') { $exe = $matches[1] }
|
||||
elseif ($path -match '(\S+mysqld\S*)') { $exe = $matches[1] }
|
||||
|
||||
$ver = ''
|
||||
if ($exe -and (Test-Path $exe)) {
|
||||
try { $ver = (& $exe --version 2>$null | Out-String).Trim() } catch { }
|
||||
}
|
||||
if ($ver) { Add-Result 'MySQL' 'Version' 'INFO' $ver }
|
||||
|
||||
# my.ini path is passed as --defaults-file in the service command line.
|
||||
$ini = ''
|
||||
if ($path -match '--defaults-file="?([^"]+\.ini)"?') { $ini = $matches[1] }
|
||||
if ($ini -and (Test-Path $ini)) {
|
||||
Add-Result 'MySQL' 'Config file' 'INFO' $ini
|
||||
|
||||
# MySQL 5.6 needs three flags or `flask db upgrade` dies with error 1071.
|
||||
$is56 = ($ver -match '\b5\.6\.')
|
||||
if ($is56) {
|
||||
$content = Get-Content $ini -Raw
|
||||
$need = @('innodb_file_per_table','innodb_file_format','innodb_large_prefix')
|
||||
$missing = @()
|
||||
foreach ($k in $need) { if ($content -notmatch $k) { $missing += $k } }
|
||||
if ($missing.Count -eq 0) {
|
||||
# Present in the FILE is not the same as ACTIVE. MySQL must be
|
||||
# restarted for them to take effect, and the app's own
|
||||
# `flask db-utils preflight` queries the live server - trust that.
|
||||
Add-Result 'MySQL' '5.6 index flags' 'WARN' 'all three present in my.ini' `
|
||||
'Present in the file only. They take effect after a MySQL RESTART, which interrupts the classic ASP app. Confirm with SHOW VARIABLES or flask db-utils preflight.'
|
||||
} else {
|
||||
# WARN, not FAIL. This inspects the LOCAL MySQL, which may not be the
|
||||
# database the operator is about to install against - a bundled 8.4,
|
||||
# or a remote server. Blocking the wizard here refused an install
|
||||
# over a server that had nothing to do with it. Stage 3 runs
|
||||
# 'flask db-utils preflight' against the database actually chosen,
|
||||
# which is the check that can genuinely block.
|
||||
Add-Result 'MySQL' '5.6 index flags' 'WARN' ("missing: " + ($missing -join ', ')) `
|
||||
"Add to [mysqld] in $ini and restart MySQL, or 'flask db upgrade' fails with error 1071. NOTE: restarting interrupts the classic ASP app."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# 5. Python (detect, but the installer uses its OWN bundled interpreter)
|
||||
# =============================================================================
|
||||
|
||||
Invoke-Check 'Python' 'On PATH' {
|
||||
$cmd = Get-Command python -ErrorAction SilentlyContinue
|
||||
if ($null -eq $cmd) {
|
||||
Add-Result 'Python' 'On PATH' 'INFO' 'no python on PATH' `
|
||||
'Expected. The installer supplies its own interpreter.'
|
||||
return
|
||||
}
|
||||
$v = ''
|
||||
try { $v = (& $cmd.Source --version 2>&1 | Out-String).Trim() } catch { }
|
||||
$detail = "$v at $($cmd.Source)"
|
||||
|
||||
# A per-user install under %LOCALAPPDATA% is unreadable by the IIS app-pool
|
||||
# identity. That produces a 500 with an empty HttpPlatform log.
|
||||
if ($cmd.Source -like "$env:LOCALAPPDATA*") {
|
||||
Add-Result 'Python' 'On PATH' 'WARN' "$detail (PER-USER install)" `
|
||||
'The IIS app-pool identity cannot read %LOCALAPPDATA%. The installer must install Python for ALL USERS and use absolute paths.'
|
||||
} elseif ($cmd.Source -like '*WindowsApps*') {
|
||||
Add-Result 'Python' 'On PATH' 'WARN' "$detail (Microsoft Store)" `
|
||||
'Store Python misbehaves under service identities. The installer will not use it.'
|
||||
} else {
|
||||
Add-Result 'Python' 'On PATH' 'INFO' $detail `
|
||||
'Not used by the installer, but a manual `flask` command later would resolve to this interpreter.'
|
||||
}
|
||||
}
|
||||
|
||||
Invoke-Check 'Python' 'Registered installs' {
|
||||
$found = @()
|
||||
foreach ($hive in @('HKLM:\SOFTWARE\Python\PythonCore','HKCU:\SOFTWARE\Python\PythonCore')) {
|
||||
if (Test-Path $hive) {
|
||||
foreach ($k in Get-ChildItem $hive -ErrorAction SilentlyContinue) {
|
||||
$ip = Join-Path $k.PSPath 'InstallPath'
|
||||
if (Test-Path $ip) {
|
||||
$loc = (Get-ItemProperty $ip -ErrorAction SilentlyContinue).'(default)'
|
||||
$scope = if ($hive -like 'HKLM*') { 'all-users' } else { 'per-user' }
|
||||
$found += "$($k.PSChildName) ($scope) $loc"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if ($found.Count -eq 0) { Add-Result 'Python' 'Registered installs' 'INFO' 'none' }
|
||||
else { Add-Result 'Python' 'Registered installs' 'INFO' ($found -join '; ') }
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Report
|
||||
# =============================================================================
|
||||
|
||||
$fails = @($script:Results | Where-Object { $_.Status -eq 'FAIL' })
|
||||
$warns = @($script:Results | Where-Object { $_.Status -eq 'WARN' })
|
||||
$skips = @($script:Results | Where-Object { $_.Status -eq 'SKIP' })
|
||||
|
||||
if ($Delimited) {
|
||||
# Data only. No padding, no colour, no alignment - the caller lays it out.
|
||||
# Pipes are stripped from field values so the record can be split naively.
|
||||
foreach ($r in $script:Results) {
|
||||
$fix = ''
|
||||
if ($r.Fix) { $fix = $r.Fix }
|
||||
$fields = @($r.Status, $r.Area, $r.Check, $r.Detail, $fix) | ForEach-Object {
|
||||
([string]$_) -replace '\|', '/' -replace '\s*\r?\n\s*', ' '
|
||||
}
|
||||
Write-Output ($fields -join '|')
|
||||
}
|
||||
} elseif ($Json) {
|
||||
[PSCustomObject]@{
|
||||
Timestamp = (Get-Date).ToString('s')
|
||||
Computer = $env:COMPUTERNAME
|
||||
SitePort = $SitePort
|
||||
AppRoot = $AppRoot
|
||||
Failures = $fails.Count
|
||||
Warnings = $warns.Count
|
||||
Skipped = $skips.Count
|
||||
Results = $script:Results
|
||||
} | ConvertTo-Json -Depth 5
|
||||
} else {
|
||||
Write-Host ''
|
||||
Write-Host 'ShopDB-Flask preflight' -ForegroundColor Cyan
|
||||
Write-Host (" host {0} site port {1} approot {2}" -f $env:COMPUTERNAME, $SitePort, $AppRoot)
|
||||
Write-Host ''
|
||||
$area = ''
|
||||
foreach ($r in $script:Results) {
|
||||
if ($r.Area -ne $area) { $area = $r.Area; Write-Host "[$area]" -ForegroundColor White }
|
||||
$colour = 'Gray'
|
||||
if ($r.Status -eq 'PASS') { $colour = 'Green' }
|
||||
if ($r.Status -eq 'WARN') { $colour = 'Yellow' }
|
||||
if ($r.Status -eq 'FAIL') { $colour = 'Red' }
|
||||
if ($r.Status -eq 'SKIP') { $colour = 'DarkGray' }
|
||||
Write-Host (" {0,-5} {1,-28} {2}" -f $r.Status, $r.Check, $r.Detail) -ForegroundColor $colour
|
||||
if ($r.Fix -and $r.Status -ne 'PASS' -and $r.Status -ne 'INFO' -and $r.Status -ne 'SKIP') {
|
||||
Write-Host (" -> {0}" -f $r.Fix) -ForegroundColor DarkGray
|
||||
}
|
||||
}
|
||||
Write-Host ''
|
||||
if ($fails.Count -eq 0) {
|
||||
Write-Host "No blocking problems. $($warns.Count) warning(s), $($skips.Count) skipped." -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host "$($fails.Count) blocking problem(s), $($warns.Count) warning(s), $($skips.Count) skipped." -ForegroundColor Red
|
||||
}
|
||||
if ($skips.Count -gt 0) {
|
||||
Write-Host " Skipped checks were NOT verified. Re-run once their prerequisite is installed." -ForegroundColor DarkGray
|
||||
}
|
||||
Write-Host ''
|
||||
}
|
||||
|
||||
if ($fails.Count -gt 0) { exit 1 } else { exit 0 }
|
||||
|
Before Width: | Height: | Size: 52 KiB |
|
Before Width: | Height: | Size: 822 B |
|
Before Width: | Height: | Size: 822 B |
|
Before Width: | Height: | Size: 822 B |
@@ -1,130 +0,0 @@
|
||||
# Tests the installer's durable install record.
|
||||
#
|
||||
# The record answers one question: "is this a re-run of MY install?". Getting it
|
||||
# wrong is destructive in one direction - answering "first provisioning" for a
|
||||
# server holding real data lets prune-schema --force loose on it - and a dead
|
||||
# end in the other, which is what it was built to fix.
|
||||
#
|
||||
# Run directly: pwsh -File test-install-state.ps1
|
||||
# pytest runs it through tests/test_installer_state.py wherever pwsh exists.
|
||||
|
||||
Set-StrictMode -Version 2.0
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$Here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||
$Installer = Join-Path (Split-Path -Parent $Here) 'shopdb-install.ps1'
|
||||
if (-not (Test-Path $Installer)) { throw "installer not found at $Installer" }
|
||||
|
||||
$Sandbox = Join-Path ([System.IO.Path]::GetTempPath()) ('shopdb-state-' + [System.Guid]::NewGuid().ToString('N'))
|
||||
|
||||
# Load ONLY the state block, so the rest of the installer does not run. Bounded
|
||||
# by the same markers the installer uses, and it fails loudly if either moves
|
||||
# rather than silently testing nothing.
|
||||
$lines = Get-Content $Installer
|
||||
$start = ($lines | Select-String -Pattern '^\$script:StateFile ' | Select-Object -First 1)
|
||||
$end = ($lines | Select-String -Pattern '^function Invoke-Native' | Select-Object -First 1)
|
||||
if (-not $start -or -not $end) { throw 'could not locate the install-record block in shopdb-install.ps1' }
|
||||
$block = $lines[($start.LineNumber - 1)..($end.LineNumber - 2)] -join "`n"
|
||||
if ($block -notmatch 'function Test-FirstProvisioning') { throw 'extracted block does not contain the state functions' }
|
||||
|
||||
# Stand-ins for the surrounding script.
|
||||
$script:Created = New-Object System.Collections.ArrayList
|
||||
function Write-Log { param($m, $l = 'INFO') }
|
||||
function Protect-File { param([string] $Path, [switch] $Directory) }
|
||||
function Test-SamePath {
|
||||
param([string] $A, [string] $B)
|
||||
if (-not $A) { return -not $B }
|
||||
return ($A.TrimEnd('\', '/').ToLowerInvariant() -eq $B.TrimEnd('\', '/').ToLowerInvariant())
|
||||
}
|
||||
if (-not $env:ProgramData) { $env:ProgramData = [System.IO.Path]::GetTempPath() }
|
||||
|
||||
Invoke-Expression $block
|
||||
|
||||
$script:Failures = 0
|
||||
function Check {
|
||||
param([string] $Name, $Expected, $Actual)
|
||||
if ($Expected -eq $Actual) {
|
||||
Write-Host (" PASS " + $Name)
|
||||
} else {
|
||||
Write-Host (" FAIL {0}: expected [{1}], got [{2}]" -f $Name, $Expected, $Actual)
|
||||
$script:Failures++
|
||||
}
|
||||
}
|
||||
|
||||
function Reset-Case {
|
||||
param([switch] $Stamped)
|
||||
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
|
||||
New-Item -ItemType Directory -Path $Sandbox -Force | Out-Null
|
||||
$script:AppRoot = $Sandbox
|
||||
$script:StateFile = Join-Path $Sandbox 'install-state.json'
|
||||
$script:InstallState = $null
|
||||
if ($Stamped) { Set-Content (Join-Path $Sandbox '.installed-version') '0.7.0' }
|
||||
}
|
||||
|
||||
Write-Host 'brand new server'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
Check 'first provisioning' $true (Test-FirstProvisioning)
|
||||
Check 'record written' $true (Test-Path $script:StateFile)
|
||||
|
||||
Write-Host 'retry of a failed first install'
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'still first provisioning' $true (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'genuine upgrade after a completed install'
|
||||
Complete-InstallState
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'no longer first provisioning' $false (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'install predating the record - must take the safe side'
|
||||
Reset-Case -Stamped
|
||||
Initialize-InstallState
|
||||
Check 'treated as established' $false (Test-FirstProvisioning)
|
||||
|
||||
Write-Host 'record naming a different directory'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
$other = Get-Content $script:StateFile -Raw | ConvertFrom-Json
|
||||
$other.approot = 'C:\somewhere-else'
|
||||
($other | ConvertTo-Json -Depth 6) | Set-Content $script:StateFile
|
||||
$script:InstallState = $null
|
||||
Check 'foreign record rejected' $null (Get-InstallState)
|
||||
|
||||
Write-Host 'corrupt record'
|
||||
Reset-Case
|
||||
Set-Content $script:StateFile '{ this is not json'
|
||||
$script:InstallState = $null
|
||||
Check 'corrupt record rejected' $null (Get-InstallState)
|
||||
Reset-Case
|
||||
Set-Content $script:StateFile '{ this is not json'
|
||||
Initialize-InstallState
|
||||
Check 'recovers and rewrites' $true ($null -ne $script:InstallState)
|
||||
|
||||
Write-Host 'created items survive a crash'
|
||||
Reset-Case
|
||||
Initialize-InstallState
|
||||
Track 'site' 'shopdb-flask'
|
||||
Track 'apppool' 'shopdbflask'
|
||||
Track 'site' 'shopdb-flask'
|
||||
$script:InstallState = $null
|
||||
Initialize-InstallState
|
||||
Check 'site remembered' 'shopdb-flask' ((Get-CreatedItems 'site') -join ',')
|
||||
Check 'apppool remembered' 'shopdbflask' ((Get-CreatedItems 'apppool') -join ',')
|
||||
# A zero-length array returned from a function unrolls to $null, and $null.Count
|
||||
# is fatal under StrictMode. This is the check that catches losing the comma.
|
||||
Check 'unknown kind is an empty array' 0 ((Get-CreatedItems 'firewall')).Count
|
||||
|
||||
Write-Host 'no record at all'
|
||||
$script:InstallState = $null
|
||||
Check 'empty list, not null' 0 ((Get-CreatedItems 'site')).Count
|
||||
Check 'not first provisioning' $false (Test-FirstProvisioning)
|
||||
|
||||
if (Test-Path $Sandbox) { Remove-Item $Sandbox -Recurse -Force }
|
||||
if ($script:Failures -gt 0) {
|
||||
Write-Host ("{0} check(s) failed" -f $script:Failures)
|
||||
exit 1
|
||||
}
|
||||
Write-Host 'all checks passed'
|
||||
exit 0
|
||||
@@ -1,171 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check a staged installer bundle against bundle-lock.json.
|
||||
|
||||
Prints one line per problem and exits non-zero if there are any. Exits 0 only
|
||||
when the bundle's third-party payload is EXACTLY what the lock describes: no
|
||||
missing file, no unexpected extra file, no changed content.
|
||||
|
||||
Why this exists alongside bundle-lock.ps1, which does the same job:
|
||||
|
||||
- bundle-lock.ps1 is canonical. It runs at INSTALL time on the target server,
|
||||
where PowerShell is the only thing guaranteed to be present - Python is not
|
||||
installed until stage 2, and verifying the payload after running part of it
|
||||
would defeat the purpose.
|
||||
- This file lets the Linux builder (build-installer.sh) do the same check
|
||||
without adding pwsh as a build dependency.
|
||||
|
||||
The two are kept honest by tests/test_bundle_lock.py, which runs BOTH against
|
||||
the same fixtures and fails if they disagree.
|
||||
|
||||
Usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# Must match $script:BundlePayloads in bundle-lock.ps1.
|
||||
PAYLOADS = [
|
||||
('wheels', True, 'Python wheels for the offline install'),
|
||||
('python', True, 'the Python installer'),
|
||||
('httpplatformhandler', True, 'the IIS module that launches waitress'),
|
||||
('urlrewrite', False, 'IIS URL Rewrite, for the client-IP rule'),
|
||||
('mysqlclient', False, 'mysql/mysqldump, for backups against a remote database'),
|
||||
('vcredist', False, 'the Visual C++ runtime MySQL requires'),
|
||||
('mysql', False, 'MySQL, for the bundled-database option'),
|
||||
]
|
||||
|
||||
|
||||
def digest(path):
|
||||
sha = hashlib.sha256()
|
||||
with open(path, 'rb') as fh:
|
||||
for chunk in iter(lambda: fh.read(1024 * 1024), b''):
|
||||
sha.update(chunk)
|
||||
return sha.hexdigest()
|
||||
|
||||
|
||||
def payload_files(directory):
|
||||
"""Every file under the directory, keyed by forward-slashed relative path."""
|
||||
found = {}
|
||||
if not os.path.isdir(directory):
|
||||
return found
|
||||
for root, _dirs, files in os.walk(directory):
|
||||
for name in files:
|
||||
full = os.path.join(root, name)
|
||||
rel = os.path.relpath(full, directory).replace(os.sep, '/')
|
||||
found[rel] = {'sha256': digest(full), 'size': os.path.getsize(full)}
|
||||
return found
|
||||
|
||||
|
||||
def normalize(name):
|
||||
"""PEP 427 wheel filename form: runs of non-alphanumerics become one _."""
|
||||
return re.sub(r'[^A-Za-z0-9.]+', '_', name).lower()
|
||||
|
||||
|
||||
def requirement_pins(requirements_path):
|
||||
"""Every 'name==version' pinned in a lockfile, including marked-out ones.
|
||||
|
||||
Markers are deliberately IGNORED. A requirement guarded by
|
||||
sys_platform == 'win32' is exactly the case that must be present, because the
|
||||
target is Windows and the wheelhouse is usually assembled somewhere else.
|
||||
"""
|
||||
pins = {}
|
||||
with open(requirements_path) as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line or line.startswith('#'):
|
||||
continue
|
||||
match = re.match(r'^([A-Za-z0-9._-]+)==([^\s;\\]+)', line)
|
||||
if match:
|
||||
pins[normalize(match.group(1))] = match.group(2)
|
||||
return pins
|
||||
|
||||
|
||||
def check_wheelhouse_covers_requirements(bundle_root):
|
||||
"""The lock records what IS in the wheelhouse, not what the app NEEDS.
|
||||
|
||||
Without this, an incomplete wheelhouse gets locked and blessed, and the
|
||||
install fails on an air-gapped server. That is not hypothetical: assembling
|
||||
the wheelhouse on Linux silently omits colorama, a win32-only dependency of
|
||||
click, because pip evaluates environment markers against the machine doing
|
||||
the downloading rather than the machine being targeted.
|
||||
"""
|
||||
wheels = os.path.join(bundle_root, 'wheels')
|
||||
requirements = os.path.join(bundle_root, 'app', 'requirements.txt')
|
||||
if not os.path.isdir(wheels) or not os.path.exists(requirements):
|
||||
return []
|
||||
have = os.listdir(wheels)
|
||||
problems = []
|
||||
for name, version in sorted(requirement_pins(requirements).items()):
|
||||
prefix = '%s-%s-' % (name, version)
|
||||
if not any(f.lower().startswith(prefix) for f in have):
|
||||
problems.append(
|
||||
'wheels/ has no wheel for %s==%s, which requirements.txt pins '
|
||||
'(a marked-out dependency still installs on Windows)' % (name, version))
|
||||
return problems
|
||||
|
||||
|
||||
def verify(bundle_root, lock):
|
||||
problems = []
|
||||
locked = lock.get('payloads')
|
||||
if not locked:
|
||||
return ['bundle-lock.json has no "payloads" section']
|
||||
|
||||
for name, required, what in PAYLOADS:
|
||||
directory = os.path.join(bundle_root, name)
|
||||
present = os.path.isdir(directory)
|
||||
if name not in locked:
|
||||
if present:
|
||||
problems.append(
|
||||
'%s/ is present but is not in bundle-lock.json - regenerate the lock' % name)
|
||||
elif required:
|
||||
problems.append(
|
||||
'%s/ is required but is in neither the bundle nor the lock' % name)
|
||||
continue
|
||||
if not present:
|
||||
if required or locked[name].get('required'):
|
||||
problems.append('%s/ is in the lock but missing from the bundle (%s)' % (name, what))
|
||||
continue
|
||||
|
||||
expected = locked[name].get('files', {})
|
||||
actual = payload_files(directory)
|
||||
for rel, want in sorted(expected.items()):
|
||||
got = actual.get(rel)
|
||||
if got is None:
|
||||
problems.append('%s/%s is in the lock but missing from the bundle' % (name, rel))
|
||||
elif got['sha256'] != want['sha256']:
|
||||
problems.append(
|
||||
'%s/%s does NOT match the lock (expected sha256 %s..., got %s...)'
|
||||
% (name, rel, want['sha256'][:12], got['sha256'][:12]))
|
||||
elif int(got['size']) != int(want['size']):
|
||||
# Impossible for a matching sha256, so the lock was hand-edited.
|
||||
problems.append(
|
||||
'%s/%s size disagrees with the lock - the lock has been edited by hand'
|
||||
% (name, rel))
|
||||
for rel in sorted(actual):
|
||||
if rel not in expected:
|
||||
problems.append(
|
||||
'%s/%s is in the bundle but NOT in the lock (unexpected extra file)'
|
||||
% (name, rel))
|
||||
problems.extend(check_wheelhouse_covers_requirements(bundle_root))
|
||||
return problems
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 3:
|
||||
sys.exit('usage: verify_bundle_lock.py <bundle-root> <bundle-lock.json>')
|
||||
bundle_root, lock_path = sys.argv[1], sys.argv[2]
|
||||
if not os.path.exists(lock_path):
|
||||
print('no bundle-lock.json at %s' % lock_path)
|
||||
return 1
|
||||
with open(lock_path) as fh:
|
||||
lock = json.load(fh)
|
||||
problems = verify(bundle_root, lock)
|
||||
for problem in problems:
|
||||
print(problem)
|
||||
return 1 if problems else 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
|
Before Width: | Height: | Size: 151 KiB |
|
Before Width: | Height: | Size: 217 KiB |
|
Before Width: | Height: | Size: 869 KiB |
|
Before Width: | Height: | Size: 9.1 KiB |
|
Before Width: | Height: | Size: 12 KiB |
|
Before Width: | Height: | Size: 56 KiB |
@@ -1,422 +0,0 @@
|
||||
"""Collect everything needed to diagnose a ShopDB-Flask stage 5 failure.
|
||||
|
||||
Stage 5 is the smoke test: the installer asks IIS for the site and expects 200.
|
||||
When it does not get one, the cause is always in one of four places, and this
|
||||
script reads all four in one pass so the answer arrives in a single round trip:
|
||||
|
||||
1. What IIS actually answers, and with which status code and error page.
|
||||
2. Whether the config sections httpPlatformHandler needs are unlocked.
|
||||
3. Whether the app-pool identity can read the app and run its venv.
|
||||
4. Whether the app itself imports and starts.
|
||||
|
||||
Run it ON THE SERVER, as Administrator:
|
||||
|
||||
C:\\Python314\\python.exe shopdb-diagnose.py
|
||||
|
||||
It writes shopdb-diagnose-<timestamp>.txt next to itself and prints the path.
|
||||
Send that file back.
|
||||
|
||||
SECRETS: the report never contains them. Values from .env (database password,
|
||||
SECRET_KEY, JWT_SECRET_KEY) are read first, then scrubbed out of every section
|
||||
of the report before it is written, including command output and tracebacks
|
||||
that might quote them.
|
||||
|
||||
Standard library only, so it runs on the bundled runtime or any system Python.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
|
||||
APP_ROOT = os.environ.get('SHOPDB_APPROOT', r'C:\shopdb-flask')
|
||||
ALIAS = 'shopdb'
|
||||
TIMEOUT = 25
|
||||
|
||||
# Filled from .env, then scrubbed from the whole report.
|
||||
SECRETS = []
|
||||
|
||||
WINDIR = os.environ.get('WINDIR', r'C:\Windows')
|
||||
# Sysnative gives a 32-bit process the real 64-bit System32. Harmless on 64-bit.
|
||||
APPCMD_CANDIDATES = [
|
||||
os.path.join(WINDIR, 'Sysnative', 'inetsrv', 'appcmd.exe'),
|
||||
os.path.join(WINDIR, 'System32', 'inetsrv', 'appcmd.exe'),
|
||||
]
|
||||
|
||||
|
||||
def find_appcmd():
|
||||
for path in APPCMD_CANDIDATES:
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
class Report(object):
|
||||
def __init__(self):
|
||||
self.chunks = []
|
||||
|
||||
def head(self, title):
|
||||
self.chunks.append('\n' + '=' * 72 + '\n' + title + '\n' + '=' * 72)
|
||||
|
||||
def line(self, text=''):
|
||||
self.chunks.append(str(text))
|
||||
|
||||
def block(self, title, body):
|
||||
self.chunks.append('\n--- %s ---' % title)
|
||||
if body is None or str(body).strip() == '':
|
||||
self.chunks.append('(no output)')
|
||||
else:
|
||||
self.chunks.append(str(body).rstrip())
|
||||
|
||||
def text(self):
|
||||
return '\n'.join(self.chunks) + '\n'
|
||||
|
||||
|
||||
def run(cmd, timeout=60):
|
||||
"""Run a command, return combined output. Never raises."""
|
||||
try:
|
||||
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, shell=False)
|
||||
out, _ = proc.communicate(timeout=timeout)
|
||||
text = out.decode('utf-8', 'replace') if out else ''
|
||||
return '[exit %s]\n%s' % (proc.returncode, text)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
proc.kill()
|
||||
except Exception:
|
||||
pass
|
||||
return '[TIMED OUT after %ss]' % timeout
|
||||
except Exception as exc:
|
||||
return '[could not run: %s]' % exc
|
||||
|
||||
|
||||
def powershell(script, timeout=90):
|
||||
exe = os.path.join(WINDIR, 'Sysnative', 'WindowsPowerShell', 'v1.0', 'powershell.exe')
|
||||
if not os.path.isfile(exe):
|
||||
exe = 'powershell.exe'
|
||||
return run([exe, '-NoProfile', '-ExecutionPolicy', 'Bypass', '-Command', script],
|
||||
timeout=timeout)
|
||||
|
||||
|
||||
def load_secrets():
|
||||
"""Read .env so its secret VALUES can be scrubbed from the report."""
|
||||
env_path = os.path.join(APP_ROOT, '.env')
|
||||
found = {}
|
||||
if not os.path.isfile(env_path):
|
||||
return found, None
|
||||
try:
|
||||
with open(env_path, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
raw = handle.read()
|
||||
except Exception as exc:
|
||||
return found, '[could not read .env: %s]' % exc
|
||||
|
||||
for line in raw.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith('#') or '=' not in line:
|
||||
continue
|
||||
key, value = line.split('=', 1)
|
||||
key, value = key.strip(), value.strip().strip('"').strip("'")
|
||||
found[key] = value
|
||||
if not value:
|
||||
continue
|
||||
upper = key.upper()
|
||||
if 'SECRET' in upper or 'PASSWORD' in upper or 'TOKEN' in upper or 'KEY' in upper:
|
||||
SECRETS.append(value)
|
||||
if upper == 'DATABASE_URL':
|
||||
# mysql+pymysql://user:PASSWORD@host/db -- the password only.
|
||||
match = re.match(r'^[^:]+://([^:@/]+):([^@]+)@', value)
|
||||
if match:
|
||||
SECRETS.append(match.group(2))
|
||||
return found, raw
|
||||
|
||||
|
||||
def scrub(text):
|
||||
"""Remove every known secret value from the report."""
|
||||
for secret in SECRETS:
|
||||
if secret and len(secret) >= 4:
|
||||
text = text.replace(secret, '<REDACTED>')
|
||||
# Catch a password inside any URL that did not come from .env.
|
||||
text = re.sub(r'(://[^:@/\s]+:)[^@\s]+(@)', r'\1<REDACTED>\2', text)
|
||||
return text
|
||||
|
||||
|
||||
def http_probe(url):
|
||||
"""Fetch a URL, returning status, headers and body even for an error page."""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
started = time.time()
|
||||
try:
|
||||
request = urllib.request.Request(url, headers={'User-Agent': 'shopdb-diagnose'})
|
||||
with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
|
||||
body = response.read(4000).decode('utf-8', 'replace')
|
||||
return {'status': response.status, 'reason': response.reason,
|
||||
'headers': dict(response.headers), 'body': body,
|
||||
'seconds': time.time() - started}
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = ''
|
||||
try:
|
||||
body = exc.read(4000).decode('utf-8', 'replace')
|
||||
except Exception:
|
||||
pass
|
||||
return {'status': exc.code, 'reason': exc.reason,
|
||||
'headers': dict(exc.headers or {}), 'body': body,
|
||||
'seconds': time.time() - started}
|
||||
except Exception as exc:
|
||||
return {'status': None, 'reason': '%s: %s' % (type(exc).__name__, exc),
|
||||
'headers': {}, 'body': '', 'seconds': time.time() - started}
|
||||
|
||||
|
||||
def summarise_iis_error(body):
|
||||
"""Pull the meaningful bits out of an IIS error page."""
|
||||
if not body:
|
||||
return None
|
||||
import html
|
||||
|
||||
# style and script blocks first: their contents survive plain tag stripping
|
||||
# and drag CSS into the summary.
|
||||
flat = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', body)
|
||||
flat = re.sub(r'(?s)<!--.*?-->', ' ', flat)
|
||||
flat = re.sub(r'(?s)<[^>]*>', ' ', flat)
|
||||
# Any unterminated tag left by the 4000-byte body truncation.
|
||||
flat = re.sub(r'(?s)<[^>]*$', ' ', flat)
|
||||
flat = html.unescape(flat)
|
||||
flat = re.sub(r'\s+', ' ', flat).strip()
|
||||
|
||||
hints = []
|
||||
for pattern in (r'\b\d{3}\.\d+\b', r'0x[0-9a-fA-F]{8}',
|
||||
r'Error Code[^.]{0,80}', r'Config (?:Error|File)[^.]{0,120}',
|
||||
r'Requested URL[^.]{0,120}', r'Physical Path[^.]{0,120}'):
|
||||
for match in re.findall(pattern, flat):
|
||||
cleaned = re.sub(r'\s+', ' ', match).strip(' :-')
|
||||
if cleaned and cleaned not in hints:
|
||||
hints.append(cleaned)
|
||||
return {'flat': flat[:1200], 'hints': hints}
|
||||
|
||||
|
||||
def main():
|
||||
report = Report()
|
||||
stamp = datetime.now().strftime('%Y%m%d-%H%M%S')
|
||||
|
||||
env_values, env_raw = load_secrets()
|
||||
|
||||
report.line('ShopDB-Flask stage 5 diagnostic')
|
||||
report.line('generated %s' % datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
|
||||
report.line('host %s' % socket.gethostname())
|
||||
report.line('app root %s' % APP_ROOT)
|
||||
report.line('python %s' % sys.version.replace('\n', ' '))
|
||||
report.line('process is %d-bit' % (64 if sys.maxsize > 2 ** 32 else 32))
|
||||
|
||||
# ---------------------------------------------------------------- 1. HTTP
|
||||
report.head('1. WHAT IIS ANSWERS')
|
||||
report.line('This is the single most important section. The status code names')
|
||||
report.line('the fault: 500.19 = config locked, 502.3/503 = the app did not')
|
||||
report.line('start, 404 = application or handler mapping missing.')
|
||||
|
||||
hostname = socket.gethostname()
|
||||
targets = [
|
||||
'http://localhost/%s/' % ALIAS,
|
||||
'http://127.0.0.1/%s/' % ALIAS,
|
||||
'http://[::1]/%s/' % ALIAS,
|
||||
'http://%s/%s/' % (hostname, ALIAS),
|
||||
'http://localhost/',
|
||||
]
|
||||
for url in targets:
|
||||
result = http_probe(url)
|
||||
report.line('\n%s' % url)
|
||||
report.line(' status : %s %s' % (result['status'], result['reason']))
|
||||
report.line(' time : %.1fs' % result['seconds'])
|
||||
server = result['headers'].get('Server')
|
||||
if server:
|
||||
report.line(' server : %s' % server)
|
||||
summary = summarise_iis_error(result['body'])
|
||||
if summary and summary['hints']:
|
||||
report.line(' hints : %s' % ' | '.join(summary['hints'][:8]))
|
||||
if summary and summary['flat']:
|
||||
report.line(' body : %s' % summary['flat'][:600])
|
||||
|
||||
# localhost resolving to ::1 first has bitten this install before.
|
||||
report.block('name resolution for localhost', run(
|
||||
['nslookup', 'localhost'], timeout=20))
|
||||
|
||||
# ------------------------------------------------------------- 2. IIS state
|
||||
report.head('2. IIS STATE')
|
||||
appcmd = find_appcmd()
|
||||
if not appcmd:
|
||||
report.line('appcmd.exe NOT FOUND - is the IIS role installed?')
|
||||
else:
|
||||
report.line('appcmd: %s' % appcmd)
|
||||
report.block('sites', run([appcmd, 'list', 'sites']))
|
||||
report.block('applications', run([appcmd, 'list', 'apps']))
|
||||
report.block('app pools', run([appcmd, 'list', 'apppools']))
|
||||
report.block('worker processes (empty means nothing is running)',
|
||||
run([appcmd, 'list', 'wp']))
|
||||
report.block('modules: httpPlatformHandler present?',
|
||||
run([appcmd, 'list', 'modules']))
|
||||
# overrideMode tells us whether the unlock actually took effect.
|
||||
#
|
||||
# allowedServerVariables is in this list because it caused a 500.52 that
|
||||
# the first two sections could not explain: it is Deny by default, so an
|
||||
# <allowedServerVariables> block in the app's web.config is rejected
|
||||
# before httpPlatformHandler runs. Checking only the sections we unlock
|
||||
# would have missed the one we do not.
|
||||
for section in ('system.webServer/handlers',
|
||||
'system.webServer/httpPlatform',
|
||||
'system.webServer/rewrite/allowedServerVariables',
|
||||
'system.webServer/rewrite/rules'):
|
||||
report.block('lock state of %s' % section,
|
||||
run([appcmd, 'list', 'config', '/section:%s' % section,
|
||||
'/text:*']))
|
||||
|
||||
report.block('W3SVC / WAS services', powershell(
|
||||
"Get-Service W3SVC,WAS | Format-Table Name,Status,StartType -AutoSize | Out-String"))
|
||||
report.block('listeners on port 80', powershell(
|
||||
"Get-NetTCPConnection -LocalPort 80 -State Listen -EA SilentlyContinue | "
|
||||
"Format-Table LocalAddress,LocalPort,OwningProcess -AutoSize | Out-String"))
|
||||
report.block('app pool detail', powershell(
|
||||
"Import-Module WebAdministration -EA SilentlyContinue; "
|
||||
"Get-Item IIS:\\AppPools\\shopdbflask -EA SilentlyContinue | "
|
||||
"Select-Object name,state,managedRuntimeVersion,enable32BitAppOnWin64,"
|
||||
"@{n='identity';e={$_.processModel.identityType}} | Format-List | Out-String"))
|
||||
|
||||
# --------------------------------------------------------- 3. app + config
|
||||
report.head('3. APPLICATION AND CONFIG')
|
||||
|
||||
web_config = os.path.join(APP_ROOT, 'web.config')
|
||||
if os.path.isfile(web_config):
|
||||
try:
|
||||
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
report.block('web.config', handle.read())
|
||||
except Exception as exc:
|
||||
report.block('web.config', '[could not read: %s]' % exc)
|
||||
else:
|
||||
report.block('web.config', 'MISSING at %s' % web_config)
|
||||
|
||||
if env_raw is None:
|
||||
report.block('.env', 'MISSING at %s' % os.path.join(APP_ROOT, '.env'))
|
||||
else:
|
||||
# Keys and non-secret values only. Secret values are scrubbed anyway.
|
||||
lines = []
|
||||
for key in sorted(env_values):
|
||||
upper = key.upper()
|
||||
secretish = ('SECRET' in upper or 'PASSWORD' in upper
|
||||
or 'TOKEN' in upper or 'KEY' in upper
|
||||
or upper == 'DATABASE_URL')
|
||||
if secretish:
|
||||
lines.append('%s = <set, %d chars>' % (key, len(env_values[key])))
|
||||
else:
|
||||
lines.append('%s = %s' % (key, env_values[key]))
|
||||
report.block('.env (secret values withheld)', '\n'.join(lines))
|
||||
|
||||
venv_python = os.path.join(APP_ROOT, 'venv', 'Scripts', 'python.exe')
|
||||
report.line('\nvenv python exists: %s' % os.path.isfile(venv_python))
|
||||
if os.path.isfile(venv_python):
|
||||
# The exact failure stage 3 used to hit. Proves the app imports.
|
||||
report.block('venv: import shopdb', run(
|
||||
[venv_python, '-c',
|
||||
'import shopdb; print("import OK"); '
|
||||
'app = shopdb.create_app(); print("create_app OK")'], timeout=120))
|
||||
report.block('venv: waitress present', run(
|
||||
[venv_python, '-c', 'import waitress; print(waitress.__version__)'],
|
||||
timeout=60))
|
||||
|
||||
# What httpPlatformHandler is told to launch, and whether it exists.
|
||||
if os.path.isfile(web_config):
|
||||
try:
|
||||
with open(web_config, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
raw = handle.read()
|
||||
match = re.search(r'processPath\s*=\s*"([^"]+)"', raw)
|
||||
args = re.search(r'arguments\s*=\s*"([^"]*)"', raw)
|
||||
if match:
|
||||
path = os.path.expandvars(match.group(1))
|
||||
report.line('\nhttpPlatform processPath : %s' % match.group(1))
|
||||
report.line(' resolved : %s' % path)
|
||||
report.line(' exists : %s' % os.path.isfile(path))
|
||||
if args:
|
||||
report.line('httpPlatform arguments : %s' % args.group(1))
|
||||
except Exception as exc:
|
||||
report.line('[could not parse web.config: %s]' % exc)
|
||||
|
||||
# ------------------------------------------------------------- 4. app logs
|
||||
report.head('4. APPLICATION LOGS')
|
||||
log_dir = os.path.join(APP_ROOT, 'logs')
|
||||
if not os.path.isdir(log_dir):
|
||||
report.line('MISSING: %s' % log_dir)
|
||||
else:
|
||||
entries = []
|
||||
for name in sorted(os.listdir(log_dir)):
|
||||
full = os.path.join(log_dir, name)
|
||||
try:
|
||||
entries.append((os.path.getmtime(full), full, name,
|
||||
os.path.getsize(full)))
|
||||
except OSError:
|
||||
pass
|
||||
if not entries:
|
||||
report.line('%s is EMPTY.' % log_dir)
|
||||
report.line('No stdout log at all means httpPlatformHandler never')
|
||||
report.line('launched python - look at the pool identity and ACLs.')
|
||||
for _, full, name, size in sorted(entries, reverse=True)[:5]:
|
||||
if size == 0:
|
||||
report.block('%s (0 bytes)' % name,
|
||||
'EMPTY - python was launched but wrote nothing.')
|
||||
continue
|
||||
try:
|
||||
with open(full, 'r', encoding='utf-8', errors='replace') as handle:
|
||||
tail = handle.readlines()[-60:]
|
||||
report.block('%s (%d bytes, last 60 lines)' % (name, size),
|
||||
''.join(tail))
|
||||
except Exception as exc:
|
||||
report.block(name, '[could not read: %s]' % exc)
|
||||
|
||||
# ---------------------------------------------------------------- 5. ACLs
|
||||
report.head('5. PERMISSIONS')
|
||||
report.line('The pool runs as "IIS AppPool\\shopdbflask". It needs RX on the')
|
||||
report.line('tree, Modify on logs and instance, and Read on .env.')
|
||||
for target in (APP_ROOT, log_dir, os.path.join(APP_ROOT, '.env'),
|
||||
os.path.join(APP_ROOT, 'venv', 'Scripts')):
|
||||
if os.path.exists(target):
|
||||
report.block('icacls %s' % target, run(['icacls', target], timeout=40))
|
||||
|
||||
# ------------------------------------------------------------ 6. event log
|
||||
report.head('6. EVENT LOG')
|
||||
report.block('recent application errors', powershell(
|
||||
"Get-WinEvent -FilterHashtable @{LogName='Application';"
|
||||
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
|
||||
"Where-Object { $_.ProviderName -match 'HttpPlatform|IIS|W3SVC|WAS|\\.NET' "
|
||||
"-or $_.LevelDisplayName -eq 'Error' } | Select-Object -First 25 "
|
||||
"TimeCreated,ProviderName,LevelDisplayName,Message | Format-List | Out-String",
|
||||
timeout=180))
|
||||
report.block('system log: WAS / W3SVC', powershell(
|
||||
"Get-WinEvent -FilterHashtable @{LogName='System';"
|
||||
"StartTime=(Get-Date).AddHours(-6)} -EA SilentlyContinue | "
|
||||
"Where-Object { $_.ProviderName -match 'WAS|W3SVC|HTTP' } | "
|
||||
"Select-Object -First 20 TimeCreated,ProviderName,LevelDisplayName,Message | "
|
||||
"Format-List | Out-String", timeout=180))
|
||||
|
||||
# ------------------------------------------------------------------ write
|
||||
body = scrub(report.text())
|
||||
out_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
'shopdb-diagnose-%s.txt' % stamp)
|
||||
try:
|
||||
with open(out_path, 'w', encoding='utf-8') as handle:
|
||||
handle.write(body)
|
||||
except Exception:
|
||||
out_path = os.path.join(os.environ.get('TEMP', r'C:\Windows\Temp'),
|
||||
'shopdb-diagnose-%s.txt' % stamp)
|
||||
with open(out_path, 'w', encoding='utf-8') as handle:
|
||||
handle.write(body)
|
||||
|
||||
print('')
|
||||
print('Report written to:')
|
||||
print(' %s' % out_path)
|
||||
print('')
|
||||
print('%d secret value(s) were scrubbed from it.' % len(SECRETS))
|
||||
print('Send that file back.')
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
@@ -1,140 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!--
|
||||
IIS site config for shopdb-flask via HttpPlatformHandler.
|
||||
|
||||
IIS launches waitress (a Windows-friendly WSGI server; gunicorn does NOT run
|
||||
on Windows) and forwards requests to it on a private loopback port that IIS
|
||||
assigns via %HTTP_PLATFORM_PORT%. One process serves both /api and the built
|
||||
Vue SPA (frontend/dist), so no separate static site is needed.
|
||||
|
||||
Prerequisites on the box:
|
||||
- HttpPlatformHandler IIS module installed
|
||||
(https://www.iis.net/downloads/microsoft/httpplatformhandler)
|
||||
- URL Rewrite module installed (only for the optional X-Forwarded-For rule)
|
||||
- Python 3.14 + a venv at APP_ROOT\venv with requirements.txt + waitress
|
||||
- Secrets live in APP_ROOT\.env (wsgi.py load_dotenv() reads it). Keep them
|
||||
OUT of this file. Lock .env ACLs to the app pool identity + admins.
|
||||
|
||||
Replace APP_ROOT (C:\shopdb-flask below) with the real deploy path. The IIS
|
||||
site's physical path MUST be APP_ROOT (where wsgi.py lives).
|
||||
-->
|
||||
<configuration>
|
||||
<system.webServer>
|
||||
|
||||
<handlers>
|
||||
<add name="httpplatformhandler" path="*" verb="*"
|
||||
modules="httpPlatformHandler" resourceType="Unspecified" />
|
||||
</handlers>
|
||||
|
||||
<httpPlatform
|
||||
processPath="C:\shopdb-flask\venv\Scripts\waitress-serve.exe"
|
||||
arguments="--port=%HTTP_PLATFORM_PORT% --host=127.0.0.1 --threads=8 --trusted-proxy=127.0.0.1 --trusted-proxy-headers=x-forwarded-for --trusted-proxy-count=1 wsgi:app"
|
||||
stdoutLogEnabled="true"
|
||||
stdoutLogFile="C:\shopdb-flask\logs\httpplatform"
|
||||
startupTimeLimit="120"
|
||||
startupRetryCount="3">
|
||||
<environmentVariables>
|
||||
<!-- FLASK_ENV MUST be production here or wsgi.py defaults to the dev
|
||||
config (SQL echo, debug, wrong DB URL). Real secrets go in .env. -->
|
||||
<environmentVariable name="FLASK_ENV" value="production" />
|
||||
<environmentVariable name="PYTHONPATH" value="C:\shopdb-flask" />
|
||||
<!-- Subpath method only: when this web.config sits in an IIS
|
||||
Application (e.g. /ops) under an existing site instead of its own
|
||||
site, tell the app its mount path. Must match the alias the
|
||||
Application was created with AND the VITE_BASE_PATH the frontend
|
||||
was built with ('/ops/'). Omit for the own-site method.
|
||||
<environmentVariable name="MOUNT_PATH" value="/ops" />
|
||||
-->
|
||||
</environmentVariables>
|
||||
</httpPlatform>
|
||||
|
||||
<!--
|
||||
Forward the real client IP, so the audit log, the kiosk visitor-location
|
||||
feature (IP -> business unit), the GE-Enforce IP allowlist and per-host
|
||||
login rate limiting all see the caller rather than the loopback address
|
||||
HttpPlatformHandler connects from.
|
||||
|
||||
IIS does not set X-Forwarded-For on its own. Without the rule below there
|
||||
is no such header at all, and every client looks like 127.0.0.1 - so the
|
||||
allowlist and the visitor-location lookup silently stop working.
|
||||
|
||||
ONLY CORRECT WHEN IIS IS DIRECTLY EXPOSED. It overwrites the header with
|
||||
REMOTE_ADDR, which is what stops a client spoofing its own X-Forwarded-For.
|
||||
Behind a reverse proxy (ARR, a load balancer) REMOTE_ADDR is the PROXY, so
|
||||
this rule would destroy the real client IP - there, leave it disabled and
|
||||
let the proxy set the header.
|
||||
|
||||
It ships DISABLED because it needs the URL Rewrite module; enabled without
|
||||
it, IIS returns HTTP 500.19 ("configuration section not well-formed").
|
||||
|
||||
There is deliberately NO <allowedServerVariables> block below. Setting a
|
||||
server variable requires that variable to be allowed, but the section
|
||||
system.webServer/rewrite/allowedServerVariables ships with
|
||||
overrideModeDefault="Deny", so declaring it in an application's own
|
||||
web.config is refused outright: IIS answered 500.52 with error 0x80070021,
|
||||
"this configuration section cannot be used at this path", BEFORE it ever
|
||||
reached httpPlatformHandler - so python was never launched and the stdout
|
||||
log stayed empty, which looks like an application fault and is not one.
|
||||
|
||||
The installer instead allows the single variable at server level, which
|
||||
grants exactly HTTP_X_FORWARDED_FOR rather than unlocking the section and
|
||||
letting every site on the machine declare arbitrary server variables.
|
||||
|
||||
The installer handles both: -ClientIpSource direct installs URL Rewrite
|
||||
from the bundle, allows the variable, and enables this; -ClientIpSource
|
||||
proxy leaves it alone. By hand: install URL Rewrite, run
|
||||
appcmd set config /section:system.webServer/rewrite/allowedServerVariables ^
|
||||
/+"[name='HTTP_X_FORWARDED_FOR']" /commit:apphost
|
||||
then delete the two marker lines below.
|
||||
-->
|
||||
<!-- SHOPDB-CLIENTIP-BEGIN
|
||||
<rewrite>
|
||||
<rules>
|
||||
<rule name="Set X-Forwarded-For" stopProcessing="false">
|
||||
<match url=".*" />
|
||||
<serverVariables>
|
||||
<set name="HTTP_X_FORWARDED_FOR" value="{REMOTE_ADDR}" />
|
||||
</serverVariables>
|
||||
<action type="None" />
|
||||
</rule>
|
||||
</rules>
|
||||
</rewrite>
|
||||
SHOPDB-CLIENTIP-END -->
|
||||
|
||||
</system.webServer>
|
||||
|
||||
<!--
|
||||
Installer downloads: serve /installers/* as IIS static files instead of
|
||||
forwarding them to Flask. The handler above is path="*", so without this a
|
||||
request for /installers/Foo.exe goes to waitress, which has no such route
|
||||
(SPA fallback), and large binaries would stream through a Python thread.
|
||||
|
||||
This <location> clears the httpPlatformHandler for that one subpath and puts
|
||||
the static file handler back, so IIS serves the bytes directly (kernel-mode,
|
||||
range/resume, no Python thread held).
|
||||
|
||||
Requires a physical folder at APP_ROOT\installers (the site's physical path
|
||||
is APP_ROOT). Drop the installer binaries there, e.g. robocopy them from the
|
||||
classic wwwroot\installers. The stored installpath 'installers/Foo.exe' then
|
||||
resolves to <mount>/installers/Foo.exe (e.g. /shopdb/installers/Foo.exe).
|
||||
|
||||
.exe/.msi are given an explicit MIME map; if the parent site has a Request
|
||||
Filtering rule that denies executable extensions, also allow them there.
|
||||
-->
|
||||
<location path="installers">
|
||||
<system.webServer>
|
||||
<handlers>
|
||||
<clear />
|
||||
<add name="StaticFile" path="*" verb="*"
|
||||
modules="StaticFileModule" resourceType="File"
|
||||
requireAccess="Read" />
|
||||
</handlers>
|
||||
<staticContent>
|
||||
<remove fileExtension=".exe" />
|
||||
<mimeMap fileExtension=".exe" mimeType="application/octet-stream" />
|
||||
<remove fileExtension=".msi" />
|
||||
<mimeMap fileExtension=".msi" mimeType="application/octet-stream" />
|
||||
</staticContent>
|
||||
</system.webServer>
|
||||
</location>
|
||||
</configuration>
|
||||
@@ -3,7 +3,7 @@
|
||||
# For a site with NO internet. Nothing is built or pulled here: the images are
|
||||
# built on a connected box (scripts/build-offline-bundle.ps1), shipped as a
|
||||
# tarball, and `docker load`ed at the site. This file only RUNS pre-loaded
|
||||
# images. See docs/DEPLOY-AIRGAP.md for the full runbook.
|
||||
# images. See the DEPLOY-AIRGAP page in the project wiki for the full runbook.
|
||||
#
|
||||
# Differences from docker-compose.yml (the connected/build template):
|
||||
# - api uses `image:` (a loaded image), never `build: .` (build needs the net).
|
||||
@@ -32,9 +32,9 @@ x-app-env: &app-env
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
ZABBIX_URL: ${ZABBIX_URL:-}
|
||||
ZABBIX_TOKEN: ${ZABBIX_TOKEN:-}
|
||||
# Channel label for a distributable installer. Empty on the stable (main)
|
||||
# bundle, 'DEV' on the edge (dev-be) bundle, which lights up the sidebar
|
||||
# badge. Sourced from .env, which build-channel-installer.ps1 pre-fills.
|
||||
# Channel label for a distributable installer. Empty on the stable bundle,
|
||||
# 'DEV' on the edge bundle, which lights up the sidebar badge. Sourced from
|
||||
# .env, which build-channel-installer.ps1 pre-fills.
|
||||
SHOPDB_ENV_LABEL: ${SHOPDB_ENV_LABEL:-}
|
||||
|
||||
services:
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Channel overlay for side-by-side stable / bleeding-edge stacks.
|
||||
#
|
||||
# The base docker-compose.yml is channel-agnostic. This overlay adds only the
|
||||
# pieces that differ between a stable (main) and an edge (dev-be) stack so the
|
||||
# pieces that differ between a stable and an edge stack so the
|
||||
# two can run at once on one dev workstation without colliding:
|
||||
#
|
||||
# - SHOPDB_ENV_LABEL is passed into the api container. When non-empty the
|
||||
|
||||
112
docker-compose.prebuilt.yml
Normal file
@@ -0,0 +1,112 @@
|
||||
# Docker Compose using pre-built frontend
|
||||
# Build frontend on host first: cd frontend && npm run build
|
||||
# Then: docker compose -f docker-compose.prebuilt.yml build
|
||||
|
||||
services:
|
||||
db:
|
||||
image: mysql:8.4
|
||||
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
|
||||
ports:
|
||||
- "127.0.0.1:3306:3306"
|
||||
environment:
|
||||
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
|
||||
MYSQL_DATABASE: shopdb_flask
|
||||
MYSQL_USER: shopdb
|
||||
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
|
||||
volumes:
|
||||
- db_data:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
|
||||
# Schema, plugin bootstrap and seed data - runs once before API starts
|
||||
migrate:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.prebuilt
|
||||
restart: "no"
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- FLASK_ENV=${FLASK_ENV:-production}
|
||||
- DATABASE_URL=mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask?charset=utf8mb4
|
||||
- SECRET_KEY=${SECRET_KEY}
|
||||
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
- CORS_ORIGINS=${CORS_ORIGINS}
|
||||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- >
|
||||
flask db upgrade &&
|
||||
/app/scripts/bootstrap-plugins.sh &&
|
||||
flask plugin upgrade-all &&
|
||||
flask seed permissions &&
|
||||
flask seed settings &&
|
||||
flask seed reference-data
|
||||
volumes:
|
||||
- ./plugins:/app/plugins:ro
|
||||
- ./scripts:/app/scripts:ro
|
||||
- instance_data:/app/instance
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.prebuilt
|
||||
ports:
|
||||
- "${API_PORT:-5001}:5001"
|
||||
environment:
|
||||
- DATABASE_URL=mysql+pymysql://shopdb:${MYSQL_PASSWORD}@db:3306/shopdb_flask
|
||||
- SECRET_KEY=${SECRET_KEY}
|
||||
- JWT_SECRET_KEY=${JWT_SECRET_KEY}
|
||||
- FLASK_ENV=${FLASK_ENV:-production}
|
||||
- FLASK_DEBUG=${FLASK_DEBUG:-0}
|
||||
- CORS_ORIGINS=${CORS_ORIGINS}
|
||||
- SHOPDB_ENV_LABEL=${SHOPDB_ENV_LABEL}
|
||||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||||
- ZABBIX_URL=${ZABBIX_URL:-}
|
||||
- ZABBIX_TOKEN=${ZABBIX_TOKEN:-}
|
||||
- COLLECTOR_API_KEY=${COLLECTOR_API_KEY:-}
|
||||
- COLLECTOR_API_KEY_ENDPOINTS=${COLLECTOR_API_KEY_ENDPOINTS:-}
|
||||
volumes:
|
||||
- instance_data:/app/instance
|
||||
- ./plugins:/app/plugins:ro
|
||||
- ./scripts:/app/scripts:ro
|
||||
- ./certs:/app/certs:ro
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- gunicorn
|
||||
- --bind
|
||||
- 0.0.0.0:5001
|
||||
- --worker-class
|
||||
- gevent
|
||||
- --workers
|
||||
- "4"
|
||||
- --worker-connections
|
||||
- "2000"
|
||||
- --timeout
|
||||
- "60"
|
||||
- --access-logfile
|
||||
- "-"
|
||||
- --access-logformat
|
||||
- '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
|
||||
- --error-logfile
|
||||
- "-"
|
||||
- --certfile
|
||||
- /app/certs/server.crt
|
||||
- --keyfile
|
||||
- /app/certs/server.key
|
||||
- wsgi:app
|
||||
|
||||
volumes:
|
||||
db_data:
|
||||
instance_data:
|
||||
@@ -63,6 +63,11 @@ services:
|
||||
ZABBIX_TOKEN: ${ZABBIX_TOKEN:-}
|
||||
COLLECTOR_API_KEY: ${COLLECTOR_API_KEY:-}
|
||||
COLLECTOR_API_KEY_ENDPOINTS: ${COLLECTOR_API_KEY_ENDPOINTS:-}
|
||||
# This file is the source-build stack. Released channels ship through
|
||||
# the pull/airgap compose files, not this one. So default the badge label
|
||||
# to DEV here - a raw source build is by definition not a release - while
|
||||
# still allowing an override from .env.
|
||||
SHOPDB_ENV_LABEL: ${SHOPDB_ENV_LABEL:-DEV}
|
||||
ports:
|
||||
- "${API_PORT:-5001}:5001"
|
||||
# Terminate TLS in gunicorn with the mounted self-signed cert so the browser
|
||||
@@ -74,10 +79,26 @@ services:
|
||||
- gunicorn
|
||||
- --bind
|
||||
- 0.0.0.0:5001
|
||||
# Threaded workers, not the blocking sync default. An agent that fails
|
||||
# the TLS handshake retries in a loop, and with 4 sync workers a handful
|
||||
# of those pinned every worker: real UI requests queued 20+ seconds while
|
||||
# the CPU sat idle. A stuck connection now holds one THREAD, and 32
|
||||
# (4 x 8) have to be exhausted before a person waits.
|
||||
#
|
||||
# Found in production at the other site (their commit db75ca7). Their
|
||||
# server, our agent - so the same client behaviour will reach any site
|
||||
# running the endpoint agent, and the fix belongs in the product rather
|
||||
# than on one branch.
|
||||
- --worker-class
|
||||
- gthread
|
||||
- --workers
|
||||
- "4"
|
||||
- --threads
|
||||
- "8"
|
||||
# 30, not 60: a hung connection frees its thread in half the time, which
|
||||
# is what decides whether a retry storm is absorbed or queues.
|
||||
- --timeout
|
||||
- "60"
|
||||
- "30"
|
||||
- --access-logfile
|
||||
- "-"
|
||||
- --error-logfile
|
||||
|
||||
291
docs/ADOPTING-AT-ANOTHER-SITE.md
Normal file
@@ -0,0 +1,291 @@
|
||||
<!-- docmeta
|
||||
purpose: howto
|
||||
audience: adopter
|
||||
status: current
|
||||
summary: Point the asset reporter and EventSaver at your own ShopDB, by Intune or by hand.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Adopting the fleet tools at another site
|
||||
|
||||
Two things run on shop-floor PCs and talk to a ShopDB instance: the **asset
|
||||
reporter**, which posts what a PC is to the collector API, and **EventSaver**, a
|
||||
screensaver that shows a slideshow served by ShopDB.
|
||||
|
||||
Neither is site-specific by design. What is site-specific is the URL, the
|
||||
credential, and which PCs it runs on - and all three are inputs, not code.
|
||||
|
||||
**Getting them onto a PC is your site's job, not ShopDB's.** ShopDB serves the
|
||||
data; something else lays the files down and keeps them there. This page covers
|
||||
the two delivery paths that do not assume a tool you may not have: Microsoft
|
||||
Intune, and doing it by hand. If you already run a configuration-enforcement
|
||||
tool on the floor, the end state below is what it has to produce.
|
||||
|
||||
Read [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md) first if you have not: it is one
|
||||
page and it is what makes the rest of this make sense.
|
||||
|
||||
## Where these files come from
|
||||
|
||||
| Artifact | Where it is |
|
||||
|---|---|
|
||||
| `Report-AssetToShopDB.ps1` | `plugins/computers/client/`, so it versions with the collector contract it implements. It names no site: the server comes from `HKLM:\SOFTWARE\GE\ShopDB` `BaseUrl` or `-ApiUrl` |
|
||||
| `Report-PrintersToShopDB.ps1` | `plugins/printers/client/`. Optional - it records what print queues a bay actually has |
|
||||
| EventSaver (`EventSaver.cs`, `EventSaver.ini`) | `plugins/slides/client/`. One C# file that builds with the in-box .NET Framework compiler, so a site can rebuild it rather than trust a binary. The compiled `EventSaver.scr` is a release asset, not a file in the repository. Neither file names a site |
|
||||
|
||||
---
|
||||
|
||||
## Part 1: the asset reporter
|
||||
|
||||
### What it does
|
||||
|
||||
Runs as SYSTEM and POSTs what the PC is - hostname, BIOS serial, PC type,
|
||||
logged-in user, make and model, OS, last boot, every routable NIC, the machine
|
||||
number where one is configured - to
|
||||
`POST https://<your-shopdb>/api/collector/computers`. The payload contract is
|
||||
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md). A field your site does not collect
|
||||
is simply absent; the server upserts on `hostname` and leaves the rest alone.
|
||||
|
||||
### What you must provide
|
||||
|
||||
| Input | Where it comes from |
|
||||
|---|---|
|
||||
| Server URL | `-ApiUrl`, or `HKLM:\SOFTWARE\GE\ShopDB` value `BaseUrl`. A script downloaded from Settings has your URL already in it |
|
||||
| A credential | `HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey` for the shared key, or the PC's own credential once it has enrolled. Never stamped into the script. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
|
||||
| Your routable ranges | A **server** setting, `collector_corporate_ranges`. The bay reports every NIC and takes no view; the server picks which one is the PC's address |
|
||||
|
||||
The reporter still accepts `-AllowedRanges` and reads a `CollectorRanges`
|
||||
registry value, and **ignores both**, logging a note when either is set. Choosing
|
||||
the NIC on the bay was wrong - a bay cannot know a site's addressing - so it
|
||||
moved to the server. The parameters remain only so an existing scheduled task
|
||||
does not fail on an unknown argument. Set the server setting.
|
||||
|
||||
**Shortcut: download it pre-configured.** Settings > Computers > Asset reporter
|
||||
generates the script with your server's URL already in the parameter defaults,
|
||||
and shows its SHA-256. The credential is deliberately not included.
|
||||
|
||||
### The credential to start with
|
||||
|
||||
Mint a managed personal access token scoped to `collector.ingest` and nothing
|
||||
else, so a token recovered off a shop-floor PC cannot read your asset register.
|
||||
Plan to move off it onto per-PC credentials.
|
||||
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole story and the only page that
|
||||
describes it.
|
||||
|
||||
### Deploying it
|
||||
|
||||
The three paths below are alternatives - pick one. All of them end with the same
|
||||
two facts on the PC: the script exists somewhere it can run from, and the
|
||||
credential is in the registry.
|
||||
|
||||
#### Option A: Microsoft Intune
|
||||
|
||||
**A1. The key, as a platform script.** Devices > Scripts and remediations >
|
||||
Platform scripts > Add > Windows 10 and later. Run as SYSTEM, do not run in the
|
||||
64-bit context only.
|
||||
|
||||
```powershell
|
||||
# Set-ShopdbCollectorKey.ps1 (Intune platform script, runs once per device)
|
||||
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
|
||||
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
|
||||
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
|
||||
exit 0
|
||||
```
|
||||
|
||||
Putting a token in an Intune script body means anyone who can read the Intune
|
||||
configuration can read the token. That is an argument for the `collector.ingest`
|
||||
scope, not against Intune: the blast radius of that token is one API. Restrict
|
||||
the registry key's ACL to SYSTEM and Administrators so an interactive user on the
|
||||
bay cannot read it.
|
||||
|
||||
**A2. The reporter, as a remediation on a schedule.** Devices > Scripts and
|
||||
remediations > Remediations. A remediation is the natural fit because it re-runs
|
||||
on a schedule, which is exactly what reporting is.
|
||||
|
||||
*Detection script* - forces the remediation to run every cycle:
|
||||
|
||||
```powershell
|
||||
# Always "not compliant": reporting is not a state to converge on, it is an
|
||||
# event that should happen on every schedule tick.
|
||||
exit 1
|
||||
```
|
||||
|
||||
*Remediation script* - the reporter itself, with your URL:
|
||||
|
||||
```powershell
|
||||
& "$PSScriptRoot\Report-AssetToShopDB.ps1" `
|
||||
-ApiUrl 'https://shopdb.example.net/api/collector/computers'
|
||||
exit 0
|
||||
```
|
||||
|
||||
Assign it with a daily schedule. Run as SYSTEM, 64-bit.
|
||||
|
||||
**A3. Intune with DSC (Machine Configuration).** If your estate is already
|
||||
governed by Azure Machine Configuration, express the *credential* as
|
||||
configuration and leave the *reporting* to a scheduled task, because a report is
|
||||
an event and DSC converges state:
|
||||
|
||||
```powershell
|
||||
Configuration ShopdbCollector
|
||||
{
|
||||
Import-DscResource -ModuleName PSDesiredStateConfiguration
|
||||
|
||||
Node localhost
|
||||
{
|
||||
Registry CollectorKey
|
||||
{
|
||||
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
|
||||
ValueName = 'CollectorKey'
|
||||
ValueData = 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
|
||||
ValueType = 'String'
|
||||
Ensure = 'Present'
|
||||
}
|
||||
|
||||
Registry CollectorUrl
|
||||
{
|
||||
Key = 'HKEY_LOCAL_MACHINE\SOFTWARE\GE\ShopDB'
|
||||
ValueName = 'BaseUrl'
|
||||
ValueData = 'https://shopdb.example.net'
|
||||
ValueType = 'String'
|
||||
Ensure = 'Present'
|
||||
}
|
||||
|
||||
# ...and a Script resource whose SetScript registers the scheduled task
|
||||
# below (option B has the three lines). Pass it no -ApiUrl: BaseUrl
|
||||
# above is where the script reads the server from, and one source beats
|
||||
# two that can disagree.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Do not add a `CollectorRanges` resource. That value is read and ignored; the
|
||||
ranges are a server setting now.
|
||||
|
||||
#### Option B: manual
|
||||
|
||||
For a pilot, a single bay, or an estate with no management plane:
|
||||
|
||||
```powershell
|
||||
# As administrator, once per PC.
|
||||
New-Item -Path 'C:\ProgramData\ShopDB' -ItemType Directory -Force | Out-Null
|
||||
Copy-Item .\Report-AssetToShopDB.ps1 'C:\ProgramData\ShopDB\'
|
||||
|
||||
New-Item -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Force | Out-Null
|
||||
Set-ItemProperty -Path 'HKLM:\SOFTWARE\GE\ShopDB' -Name CollectorKey `
|
||||
-Value 'shopdb_pat_REPLACE_WITH_YOUR_TOKEN'
|
||||
|
||||
$action = New-ScheduledTaskAction -Execute 'powershell.exe' -Argument @'
|
||||
-NoProfile -ExecutionPolicy Bypass -File "C:\ProgramData\ShopDB\Report-AssetToShopDB.ps1" -ApiUrl "https://shopdb.example.net/api/collector/computers"
|
||||
'@
|
||||
$trigger = New-ScheduledTaskTrigger -Daily -At 7am
|
||||
Register-ScheduledTask -TaskName 'ShopDB asset report' -Action $action -Trigger $trigger `
|
||||
-User 'SYSTEM' -RunLevel Highest -Force
|
||||
|
||||
Start-ScheduledTask -TaskName 'ShopDB asset report'
|
||||
```
|
||||
|
||||
Lock down `C:\ProgramData\ShopDB` if you put it there: a script a user can edit,
|
||||
run by SYSTEM on a schedule, is a local privilege escalation. Break inheritance
|
||||
and grant Administrators and SYSTEM only.
|
||||
|
||||
### Verifying, and what to do when it fails
|
||||
|
||||
Open Computers in the web UI and find the hostname: it should be there with its
|
||||
serial, type and a last-reported timestamp within one schedule tick. The same
|
||||
answer over the API needs a logged-in session, because the read side is
|
||||
permissioned (`computers.view`) even though the ingest side is not:
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>' `
|
||||
-Headers @{ Authorization = "Bearer $jwt" }
|
||||
```
|
||||
|
||||
The collector key does NOT work here - it grants `collector.ingest` and nothing
|
||||
else, which is the point of it. The script also logs to `C:\Logs\Shopfloor`.
|
||||
Every failure mode and its fix is the table in
|
||||
[COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md).
|
||||
|
||||
---
|
||||
|
||||
## Part 2: EventSaver
|
||||
|
||||
### What you have to decide
|
||||
|
||||
[EVENTSAVER](EVENTSAVER.md) is what EventSaver is, how it picks slides and how to
|
||||
diagnose it. Read it first; adopting it adds only three decisions:
|
||||
|
||||
1. **Your feed URL**, in `EventSaver.ini`:
|
||||
`url=https://<your-shopdb>/api/slides/feed?surface=shopfloor`.
|
||||
2. **Ship the ini with the binary.** There is no compiled-in fallback, so a
|
||||
missing ini is a blank screen. Check any `.scr` you were handed rather than
|
||||
built: an early build still has another site's file server baked in, which is
|
||||
exactly why the fallback was removed (ADR-015).
|
||||
3. **Which PCs get it.** A screensaver over a live display is a defect. Exclude
|
||||
kiosks, lobby displays and metrology bays.
|
||||
|
||||
If your configuration tool enforces the ini by hash, your own hand edits on a
|
||||
bay revert - that is the feature working. Edit the copy your tool deploys from,
|
||||
and update the hash it detects on:
|
||||
|
||||
```powershell
|
||||
(Get-FileHash .\configs\EventSaver.ini -Algorithm SHA256).Hash
|
||||
```
|
||||
|
||||
### Deploying it
|
||||
|
||||
#### Option A: Intune
|
||||
|
||||
Package the three files as a Win32 app (`EventSaver.scr`, `EventSaver.ini`, and
|
||||
the enable script) with this install command:
|
||||
|
||||
```powershell
|
||||
# install.ps1 for the Win32 app
|
||||
Copy-Item .\EventSaver.scr 'C:\Windows\System32\EventSaver.scr' -Force
|
||||
Copy-Item .\EventSaver.ini 'C:\Windows\System32\EventSaver.ini' -Force
|
||||
exit 0
|
||||
```
|
||||
|
||||
Detection rule: file `C:\Windows\System32\EventSaver.scr` exists **and** matches
|
||||
the expected SHA-256, so a changed binary reinstalls.
|
||||
|
||||
Enable it per user with a configuration profile (Settings catalog > Screen
|
||||
saver) or a user-context script setting the three values under option B. With
|
||||
Machine Configuration/DSC, the two files are `File` resources - but DSC runs as
|
||||
SYSTEM, so the per-user setting needs the logged-on user's hive or a
|
||||
default-user template, which is usually more trouble than a login script.
|
||||
|
||||
#### Option B: manual
|
||||
|
||||
```powershell
|
||||
Copy-Item .\EventSaver.scr 'C:\Windows\System32\' -Force
|
||||
Copy-Item .\EventSaver.ini 'C:\Windows\System32\' -Force # with YOUR url=
|
||||
|
||||
# Per user, in the user's own session:
|
||||
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name SCRNSAVE.EXE -Value 'C:\Windows\System32\EventSaver.scr'
|
||||
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveActive -Value '1'
|
||||
Set-ItemProperty 'HKCU:\Control Panel\Desktop' -Name ScreenSaveTimeOut -Value '480'
|
||||
```
|
||||
|
||||
Those three values are the whole "enable" step in every option above: a
|
||||
screensaver is a per-user setting, so whatever deploys the binary still has to
|
||||
reach the user hive.
|
||||
|
||||
### Verifying
|
||||
|
||||
Run `EventSaver.scr /c` on the PC - it reports which source it resolved and
|
||||
where it is caching - then open that feed URL in a browser on the same PC. The
|
||||
symptom table in [EVENTSAVER](EVENTSAVER.md) reads the answers.
|
||||
|
||||
---
|
||||
|
||||
## Which path to choose
|
||||
|
||||
| You have | Use |
|
||||
|---|---|
|
||||
| Intune across the estate | Option A. A remediation for the reporter, a Win32 app for EventSaver. |
|
||||
| A configuration-enforcement tool on the floor | Its own idiom, producing the end state option B describes: the script on disk, the credential in the registry, a task as SYSTEM. |
|
||||
| Neither, or a pilot of a few bays | Option B, then move to A once it earns its place. |
|
||||
|
||||
These are not exclusive, and on a real floor they usually are not. Where two
|
||||
tools both converge the same PC, decide which owns which layer before deploying
|
||||
both: the deciding question is which one you would look at first to answer "why
|
||||
is this bay wrong".
|
||||
88
docs/API-REFERENCE.md
Normal file
@@ -0,0 +1,88 @@
|
||||
<!-- docmeta
|
||||
purpose: reference
|
||||
audience: integrator, developer
|
||||
status: current
|
||||
summary: Where the generated API spec is, what llms.txt is for, and the surface and auth orientation a spec cannot give.
|
||||
loadwith: none
|
||||
-->
|
||||
|
||||
# API reference
|
||||
|
||||
**The reference is generated and served by the app. This page is a pointer to
|
||||
it, plus the two things a specification cannot tell you.**
|
||||
|
||||
There used to be a hand-written endpoint list here. It competed with the
|
||||
generated spec, lost, and went stale in between - so it was cut rather than
|
||||
maintained. Everything it listed is in the spec, derived from the code: every
|
||||
operation with its auth, its query parameters, its response shape and
|
||||
harvested examples. Counts are in [PROJECT-MAP](PROJECT-MAP.md), which is
|
||||
generated; nothing of that kind belongs on this page.
|
||||
|
||||
## Where the generated reference is
|
||||
|
||||
| Reach it at | What it is |
|
||||
|---|---|
|
||||
| `GET /api/docs` | The reference, rendered with Redoc. The bundle is vendored in the image, so it renders with no network - which is the only reason it is usable on an air-gapped site. |
|
||||
| `GET /api/docs/openapi.json` | The raw OpenAPI 3.1 spec. Also tracked as `docs/openapi.json`, so it can be read from a checkout without running anything; regenerate with `scripts/gen_openapi.py` after any API change. |
|
||||
| `GET /api/docs/llms.txt` | The short version, for a machine caller. Also [llms.txt](llms.txt) in this corpus. |
|
||||
|
||||
[llms.txt](llms.txt) is not a second spec. It is the orientation a caller needs
|
||||
before the spec is useful: the response envelope, how authentication is
|
||||
obtained and sent, the pagination and filtering conventions, and the handful of
|
||||
recipes people get wrong. Read it first, then look operations up in the spec.
|
||||
|
||||
## Which surface owns what
|
||||
|
||||
The spec lists every operation; it does not say which contract governs one.
|
||||
There are three surfaces, and the third is everything else.
|
||||
|
||||
| Surface | Who calls it | Where its semantics live |
|
||||
|---|---|---|
|
||||
| **Fleet and client contracts** | Shop-floor PCs, the endpoint agent, kiosks, displays and printer installers - machines, not browsers | [COLLECTOR-INTEGRATION](COLLECTOR-INTEGRATION.md), [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md), [COLLECTOR-AUTH](COLLECTOR-AUTH.md), [PRINTER-INSTALLER](PRINTER-INSTALLER.md), [EVENTSAVER](EVENTSAVER.md) |
|
||||
| **Bulk import** | A script loading a site's data. There is no separate import surface: it drives the same endpoints the UI does, with a header that preserves the source rows' timestamps | [IMPORT-API](IMPORT-API.md) |
|
||||
| **Core UI API** | The Vue frontend, and anything else holding a token | [CONTRACT-STABILITY](CONTRACT-STABILITY.md) for what is safe to depend on |
|
||||
|
||||
## How auth works at a glance
|
||||
|
||||
Four credentials, and which one an operation takes follows from its surface.
|
||||
The spec states it per operation; this is the shape behind it.
|
||||
|
||||
- **JWT** - a login token or a Personal Access Token. The core UI API.
|
||||
- **Scoped service token** - `X-API-Key` or `Authorization: Bearer`, minted and
|
||||
revoked per purpose (`collector.ingest`, `collector.ingest.endpoints`). A
|
||||
token can be bound to specific resources and is refused anything outside
|
||||
them. This is what the fleet uses.
|
||||
- **A per-PC credential** - an enrolled endpoint holds its own secret, and
|
||||
revoking one endpoint costs nothing to the rest. See [COLLECTOR-AUTH](COLLECTOR-AUTH.md)
|
||||
and [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md).
|
||||
- **A kiosk device token, or an address inside the kiosk's allowlist** - a
|
||||
credential for a screen, not authorization for a person. It says which
|
||||
cabinet is asking; the badge on the request says who is standing at it. Both
|
||||
kiosks are off by default and answer 404, not 403, when they are off.
|
||||
|
||||
Two rules cover almost everything else:
|
||||
|
||||
- **Every read is behind a permission.** Reading is not free, and a site
|
||||
decides what a signed-out visitor may see by granting the seeded `guest` role
|
||||
the read permissions it chooses - in the same role grid it uses for staff.
|
||||
- **Every mutation needs a JWT and a permission or role.** None are public.
|
||||
|
||||
## What is reachable with no credential
|
||||
|
||||
This is the inventory a firewall or a deployment reviewer asks for, and it is
|
||||
the one question the spec answers badly - it is a question about the whole
|
||||
surface, not about any one operation.
|
||||
|
||||
Do not read it off a page somebody maintains. It is enumerated, with a reason
|
||||
per entry, in `tests/test_core/test_public_read_surface.py`, which fails the
|
||||
build when a new public read appears without being added to the list. That file
|
||||
is the answer, and it is the answer because it cannot go stale without turning
|
||||
CI red.
|
||||
|
||||
## See also
|
||||
|
||||
- [CONFIG](CONFIG.md) - the environment variables and settings behind these
|
||||
endpoints, including the public settings allowlist.
|
||||
- [CONTRACT-STABILITY](CONTRACT-STABILITY.md) - what a sister site may depend
|
||||
on, and how tightly to pin.
|
||||
- [DEPLOY](DEPLOY.md) - the site-exposure surface a deploy reviewer needs.
|
||||
158
docs/ASSET-COMPOSITION.md
Normal file
@@ -0,0 +1,158 @@
|
||||
<!-- docmeta
|
||||
purpose: explanation
|
||||
audience: developer, adopter
|
||||
status: current
|
||||
summary: When several devices share one identifier, file each under a parent asset - when to, and what it buys.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Composition: several devices under one parent
|
||||
|
||||
When more than one physical device answers to a single identifier, model each
|
||||
device as its own asset and file it under a parent. This page is the pattern:
|
||||
when to reach for it, what you get for free, and how to make the collector do it
|
||||
without writing code. The mechanism itself is not new - `partof` and its
|
||||
propagation rails are the ADR-001 platform contract; the recipe is.
|
||||
|
||||
## The problem it solves
|
||||
|
||||
A machine number is supposed to identify one thing. Sometimes it does not.
|
||||
|
||||
At the reference site, several Telesis part markers serve one operation number:
|
||||
0613, 0615 and PRTMK01 each have more than one. Their configurations differ, most
|
||||
often by COM port. Treating the operation as the device collapsed them into a
|
||||
single record, and the damage was quiet:
|
||||
|
||||
- Their backups overwrote each other, so the history read as one device
|
||||
flip-flopping between configurations that were really two devices.
|
||||
- No question about an individual device could be asked at all - how many there
|
||||
are, which port one is on, which one failed.
|
||||
- Every PC driving one contested the single `controls` link to the operation,
|
||||
so whichever reported last appeared to own it.
|
||||
|
||||
None of that announced itself. It surfaced as duplicate backup rows, weeks
|
||||
later.
|
||||
|
||||
## When to use it
|
||||
|
||||
Reach for composition when ALL of these hold:
|
||||
|
||||
- Several physical devices share one identifier, and that is CORRECT - not a
|
||||
numbering mistake. Two PCs carrying the same machine number by accident is a
|
||||
fault to fix on the PC, not a model to build. `flask relationships
|
||||
check-shared-machines` tells the two apart.
|
||||
- The devices are individually interesting: they fail, get replaced, carry
|
||||
their own configuration or calibration.
|
||||
- Something already identifies each device. One device per PC is the easy case,
|
||||
because the PC names it.
|
||||
|
||||
Do NOT reach for it when the parent is simply a location. A room holding six
|
||||
printers wants a Location, not a parent asset.
|
||||
|
||||
## What you get
|
||||
|
||||
Filing a device `partof` a parent buys behaviour already built:
|
||||
|
||||
- **Control propagates.** `controls` propagates through `partof` (ADR-001, and
|
||||
seeded by `flask seed reference-data`). A PC controlling a device therefore
|
||||
controls its parent by inheritance, so the PC does NOT need - and must not
|
||||
have - a direct link to the parent. That is what stops several devices
|
||||
contesting a link only one can hold.
|
||||
- **Position inherits.** Map-position resolution walks `partof` first, so a
|
||||
device with no coordinates of its own shows at its parent's position.
|
||||
- **History separates.** Anything keyed on the asset - backup revisions,
|
||||
relationships, notes, audit - is per device instead of merged.
|
||||
|
||||
## Making the collector do it
|
||||
|
||||
A PC that drives a subordinate device declares it in
|
||||
`plugins/computers/pctypemap.py`:
|
||||
|
||||
```python
|
||||
SUBORDINATE_DEVICE_MAP = {
|
||||
'gea-shopfloor-partmarker': {
|
||||
'assettype': 'machine', # core AssetType for the device
|
||||
'typename': 'Part Marker', # type within that vocabulary
|
||||
'description': 'Telesis part marker',
|
||||
'suffix': 'PARTMARKER', # device asset number = <PC>-<suffix>
|
||||
'partof': True, # file under the reported machine number
|
||||
'label': 'collector:partmarker',
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
`partof: True` is the switch that matters. It files the device under the
|
||||
operation the PC reports AND stops that PC claiming the operation directly.
|
||||
Leave it False for a device that does not share an identifier - a CMM is a
|
||||
subordinate device too, but no two CMMs answer to one number, so it links to
|
||||
its PC and nothing more.
|
||||
|
||||
`label` is the relationship origin marker. Only rows carrying it are archived
|
||||
by a collector push, so links made by hand are never touched. **Do not change
|
||||
an existing label**: those exact strings are in production databases.
|
||||
|
||||
A site adds or retargets an entry without a code change, per ADR-015:
|
||||
|
||||
```
|
||||
Setting: subordinatedevice_<pctype> category: pctypemapping
|
||||
Value: {"assettype": "machine", "typename": "Marking Laser",
|
||||
"suffix": "LASER", "partof": true, "label": "collector:partmarker"}
|
||||
```
|
||||
|
||||
A malformed override falls back to the built-in default rather than failing the
|
||||
collector push - a bad setting must not stop a bay reporting its inventory.
|
||||
|
||||
## Making a backup kind follow the device
|
||||
|
||||
A backup posted by a PC that drives a device belongs to the DEVICE, not the
|
||||
parent. `plugins/backups/services/registry.py` resolves this through
|
||||
`markerforsource`: it finds the PC by the reported `sourcehostname`, follows the
|
||||
PC's active device link, and files against what it finds. It falls back to the
|
||||
machine number whenever the device cannot be resolved - no hostname on the
|
||||
payload, a lean build without the computers plugin, or a PC that has not
|
||||
reported yet - because filing under the parent beats rejecting a backup.
|
||||
|
||||
Two things that matter if you add a kind:
|
||||
|
||||
- **`sourcehostname` is load-bearing**, not informational. Without it a backup
|
||||
files against the parent and merges two devices' histories.
|
||||
- **A revision chain is (asset, kind, source hostname)**, so two devices on one
|
||||
parent keep separate chains even before they become separate assets.
|
||||
|
||||
## Finding the next one
|
||||
|
||||
```
|
||||
flask relationships check-shared-machines
|
||||
```
|
||||
|
||||
Read-only. Lists every machine number claimed by more than one PC and separates
|
||||
the two cases by whether child assets exist:
|
||||
|
||||
```
|
||||
0615 11 PCs, 11 child asset(s) - modelled
|
||||
2026 2 PCs, NO child assets - PCONE, PCTWO
|
||||
```
|
||||
|
||||
The first is composition working. The second is two PCs carrying one number -
|
||||
fix that on the PC.
|
||||
|
||||
## The parent is not disposable
|
||||
|
||||
Once devices are filed under it, the parent is doing a real job even though it
|
||||
may look empty: it is the thing that says those devices belong together.
|
||||
Deactivating it (`isactive = 0`, which is what "delete" does in the UI) breaks
|
||||
filing, because the resolver requires an ACTIVE parent - every subsequent
|
||||
report warns and files nothing.
|
||||
|
||||
A hard SQL `DELETE` is worse: `backuprevisions.assetid` is `ON DELETE CASCADE`,
|
||||
so it destroys history still attached to the parent.
|
||||
|
||||
If the parent looks wrong in a list - an operation number appearing among
|
||||
machines - that is a CLASSIFICATION question, not a deletion one.
|
||||
|
||||
## See also
|
||||
|
||||
- ADR-001 - relationship types, propagation rails, position inheritance
|
||||
- ADR-015 - why the device map is settings-overridable
|
||||
- [COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) - the collector contract
|
||||
and the part-marker case end to end
|
||||
52
docs/BACKUP-KINDS.md
Normal file
@@ -0,0 +1,52 @@
|
||||
<!-- docmeta
|
||||
purpose: reference
|
||||
audience: developer
|
||||
status: current
|
||||
summary: How to add a backup kind to the backups plugin - the BackupKind members and what a new kind inherits for free.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Adding a fleet config backup kind
|
||||
|
||||
The recipe for adding a kind to the `backups` plugin - the per-PC CONFIG
|
||||
backups the collector takes off shopfloor machines (NTLARS settings,
|
||||
part-marker configs). Backing up the ShopDB database itself is
|
||||
[BACKUP-RESTORE.md](BACKUP-RESTORE.md); why a Backup badge reads green or red
|
||||
is [BACKUP-STALENESS.md](BACKUP-STALENESS.md), and you do not need it here.
|
||||
|
||||
---
|
||||
|
||||
A kind is one class in `plugins/backups/services/registry.py`, added to
|
||||
`REGISTRY` at the bottom of that file:
|
||||
|
||||
```python
|
||||
REGISTRY = {k.key: k for k in (NtlarsKind(), PartMarkerKind())}
|
||||
```
|
||||
|
||||
Subclass `BackupKind` and override what applies. `NtlarsKind` is the fullest
|
||||
example (parseable, renderable, has an info panel); `PartMarkerKind` is the
|
||||
lean one.
|
||||
|
||||
| Member | What it is |
|
||||
|--------|------------|
|
||||
| `key` | Wire value. This is what lands in `backuprevisions.backupkind` and what the fleet table shows on the badge. Keep it short and lowercase. |
|
||||
| `displayname` | Human label for the UI. |
|
||||
| `storagebackend` | `'shopdb'` (bytes live in the DB, deduped on a semantic hash) or `'share'` (file lives on the SMB share, deduped on a raw byte hash). |
|
||||
| `assettypes` | Which core asset types this kind can attach to; `['*']` for any. |
|
||||
| `emptytext` | Text when an asset has no revisions of this kind. `None` HIDES the panel, which is the right default - a kind applies to an asset TYPE, but whether a given machine ever has that backup is per-machine. A part-marker panel on all 144 machines is noise. |
|
||||
| `parse(raw)` | Opaque kinds return `None`; parseable kinds return a projection dict. |
|
||||
| `formats()` / `render(...)` | Download formats, for `shopdb` kinds. |
|
||||
| `resolveassetid(payload)` | Map a collector payload to the asset the backup belongs to. |
|
||||
| `infopanel()` / `buildinfo(...)` | Optional at-a-glance card (ADR-010). Declared by the KIND, not hardcoded in the plugin, so a successor technology ships its own card by adding a class. |
|
||||
| `sharedir(...)` | Conventional UNC directory for a `share` kind. Advisory - the authoritative path is whatever the collector reported, because the PC is what actually wrote the file. |
|
||||
|
||||
### What you do NOT have to touch
|
||||
|
||||
The **Backups stopped** dashboard card needs no change. It reads
|
||||
`backuprevisions` generically: newest revision per chain, whatever the kind, and
|
||||
shows `backupkind` as the detail line. A new kind inherits the card with no
|
||||
work. The staleness rule needs no change either - it is time-based and
|
||||
kind-agnostic ([BACKUP-STALENESS.md](BACKUP-STALENESS.md)).
|
||||
|
||||
Making a kind file against the DEVICE rather than the parent asset it hangs
|
||||
under is [ASSET-COMPOSITION.md](ASSET-COMPOSITION.md).
|
||||
202
docs/BACKUP-RESTORE.md
Normal file
@@ -0,0 +1,202 @@
|
||||
<!-- docmeta
|
||||
purpose: howto
|
||||
audience: operator, adopter
|
||||
status: current
|
||||
summary: Back up and restore a ShopDB instance - the database plus instance/ - on Docker or external MySQL.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Backup and restore the ShopDB database
|
||||
|
||||
This is the ShopDB instance itself. For the per-PC CONFIG backups the collector
|
||||
takes off shopfloor machines, see [BACKUP-KINDS.md](BACKUP-KINDS.md).
|
||||
|
||||
Each site owns its own data (single-tenant, ADR-004), so backups are the site's
|
||||
responsibility. A complete backup is two parts, and **a database-only backup is
|
||||
not a backup**: restoring the database without the matching `instance/`
|
||||
directory leaves the app pointing at floor plans and logos that no longer exist.
|
||||
|
||||
| Item | Location | Why |
|
||||
|------|----------|-----|
|
||||
| Database | MySQL `shopdb_flask` | All asset, user, audit and settings data. |
|
||||
| `instance/branding/` | `instance/` dir | Uploaded logos and favicon. |
|
||||
| `instance/modelimages/` | `instance/` dir | Uploaded vendor-model photos. |
|
||||
| `instance/` floor plans | `instance/` dir | Uploaded map blueprints. |
|
||||
| `instance/plugins.json` | `instance/` dir | Which plugins this site enabled. |
|
||||
| `.env` | repo root (offline, secured) | Secrets needed to bring the stack back up. Store separately from the data backup, in a secrets manager. |
|
||||
|
||||
## The shipped Docker backup overlay
|
||||
|
||||
`docker-compose.backup.yml` is an opt-in per-site overlay that automates all of
|
||||
the above. Turn it on with:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.backup.yml \
|
||||
--profile backup up -d
|
||||
```
|
||||
|
||||
It adds three cooperating services: `backup-store`, a resident restic REST
|
||||
server holding every snapshot in its own volume, deliberately NOT published to
|
||||
the host so backups are reachable only over the compose network; `backup-watcher`,
|
||||
which actuates the "Back up now" sentinel the api drops, because the api
|
||||
container cannot run docker itself; and `backup`, the same one-shot run manually
|
||||
(`run --rm backup`) for sites that prefer a host scheduler.
|
||||
|
||||
Each run captures one recoverable set: `db.sql.gz` (a non-blocking
|
||||
`mysqldump --single-transaction`), a restic snapshot of `instance/`, and
|
||||
`manifest.json` recording the core and plugin Alembic revisions plus the image
|
||||
tag at capture time. That manifest is what makes a restore a CHECKED operation:
|
||||
`scripts/backup/restore.sh` refuses to load a set onto code whose schema is
|
||||
older than the set's, rather than leaving the site half-broken with rows
|
||||
referencing columns the live schema has not applied. It does NOT roll back code
|
||||
- restore the code first, run `flask db upgrade` and `flask plugin upgrade-all`,
|
||||
then load the data.
|
||||
|
||||
Retention is restic-native: `--keep-daily 5 --keep-weekly 4 --keep-monthly 6`.
|
||||
`RESTIC_REPOSITORY` defaults to the bundled store and can point at S3, Azure
|
||||
Blob or a mounted share instead; credentials come from the site's env or secret
|
||||
store, never committed.
|
||||
|
||||
## Backing up by hand
|
||||
|
||||
### Database (Docker)
|
||||
|
||||
```bash
|
||||
docker compose exec -T db mysqldump \
|
||||
-u root -p"${MYSQL_ROOT_PASSWORD}" \
|
||||
--single-transaction --routines --triggers \
|
||||
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
`--single-transaction` gives a consistent dump without locking the tables (InnoDB).
|
||||
|
||||
### Database (external MySQL, no container)
|
||||
|
||||
```bash
|
||||
mysqldump -h <host> -u <user> -p \
|
||||
--single-transaction --routines --triggers \
|
||||
shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
|
||||
```
|
||||
|
||||
### instance directory
|
||||
|
||||
```bash
|
||||
tar czf instance-$(date +%F).tar.gz instance/
|
||||
```
|
||||
|
||||
Recommended cadence: nightly database dump to offsite storage, 14-day
|
||||
retention; `instance/` captured on the same schedule (and always right before an
|
||||
upgrade). Verify a restore quarterly.
|
||||
|
||||
## Restore
|
||||
|
||||
Restoring replaces the current database contents. Do it into a known-empty or a
|
||||
throwaway target first if you are unsure.
|
||||
|
||||
### Step 1: Bring up the stack (or a fresh one)
|
||||
|
||||
```bash
|
||||
cp .env.example .env # or restore your saved .env
|
||||
# ensure MYSQL_* and DATABASE_URL match the dump's database name (shopdb_flask)
|
||||
docker compose up -d db
|
||||
```
|
||||
|
||||
Wait for the `db` container to report healthy (`docker compose ps`).
|
||||
|
||||
### Step 2: Load the database dump
|
||||
|
||||
```bash
|
||||
gunzip -c shopdb-2026-07-10.sql.gz | \
|
||||
docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask
|
||||
```
|
||||
|
||||
For an external MySQL:
|
||||
|
||||
```bash
|
||||
gunzip -c shopdb-2026-07-10.sql.gz | mysql -h <host> -u <user> -p shopdb_flask
|
||||
```
|
||||
|
||||
If the target database does not exist yet, create it as utf8mb4 first (matching
|
||||
the schema charset):
|
||||
|
||||
```sql
|
||||
CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
```
|
||||
|
||||
### Step 3: Restore the instance directory
|
||||
|
||||
**On Docker, `instance/` is a named volume, not the host directory.** The api
|
||||
service mounts `instance_data:/app/instance`, so extracting the tarball into
|
||||
`./instance` on the host restores nothing the application can see. The data has
|
||||
to go into the volume:
|
||||
|
||||
```bash
|
||||
tar xzf instance-2026-07-10.tar.gz # unpacks ./instance on the host
|
||||
docker compose up -d api # creates the container and volume
|
||||
docker compose cp instance/. api:/app/instance # copy into the volume
|
||||
docker compose restart api
|
||||
```
|
||||
|
||||
Check it landed before going on:
|
||||
|
||||
```bash
|
||||
docker compose exec api ls /app/instance
|
||||
```
|
||||
|
||||
`plugins.json` must be there. Without it the site comes back with its plugins
|
||||
disabled and database rows pointing at uploaded files that are no longer
|
||||
present.
|
||||
|
||||
A site that has replaced the named volume with its own bind mount
|
||||
(`- ./instance:/app/instance`) restores by unpacking the tarball on the host and
|
||||
nothing else. Check `docker-compose.yml` rather than assuming which one you have.
|
||||
|
||||
For a non-docker deploy, unpack the tarball over the application directory:
|
||||
`instance/` is a plain directory there.
|
||||
|
||||
### Step 4: Reconcile migrations
|
||||
|
||||
```bash
|
||||
docker compose exec api flask db upgrade
|
||||
docker compose exec api flask plugin upgrade-all
|
||||
```
|
||||
|
||||
For a non-docker deploy, the same two commands without the prefix.
|
||||
|
||||
`flask db upgrade` is a safety net: if the dump predates the current code, this
|
||||
applies only the core Alembic chain. `flask plugin upgrade-all` then applies any
|
||||
newer per-plugin migrations (each bundled plugin owns its own chain, ADR-008);
|
||||
without it, plugin-owned tables stay un-migrated. If the dump is at the same
|
||||
version both are no-ops.
|
||||
|
||||
### Step 5: Verify
|
||||
|
||||
- Log in with a known account.
|
||||
- Confirm the floor map renders (branding and map blueprints resolve from
|
||||
`instance/`).
|
||||
- Spot-check a few asset records and the audit log.
|
||||
- `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .`
|
||||
should return a `VALIDATION_ERROR`, not a 500.
|
||||
|
||||
## What is lost if `.env` is lost
|
||||
|
||||
`.env` is not in the database dump, and rebuilding it is not simply retyping it:
|
||||
|
||||
| Value | If it is lost |
|
||||
|---|---|
|
||||
| `SECRET_KEY`, `JWT_SECRET_KEY` | Every issued token and session becomes invalid. Users log in again; managed API tokens must be reissued, which means every collector client needs its key replaced. Recoverable, but it is a fleet-wide job. |
|
||||
| `DATABASE_URL` password | Recoverable: reset the MySQL user's password and write the new one in. |
|
||||
| `MYSQL_ROOT_PASSWORD` | Recoverable through MySQL's own reset procedure, which requires stopping the server. |
|
||||
| `RESTIC_PASSWORD` | NOT recoverable. Every restic snapshot in the repository is encrypted with it; without it the backups are unreadable by anyone, including you. |
|
||||
| `ZABBIX_TOKEN` and similar integration tokens | Reissue at the far end. Nothing else breaks. |
|
||||
|
||||
So back it up with the database, to somewhere as protected as the dump. A dump
|
||||
without its `.env` restores the data and locks everyone out of it.
|
||||
|
||||
## See also
|
||||
|
||||
- [BACKUP-KINDS.md](BACKUP-KINDS.md) - the other backups: per-PC config
|
||||
snapshots taken by the collector. A different subject entirely.
|
||||
- [DEPLOY.md](DEPLOY.md) - first-time deploy
|
||||
- [UPGRADE.md](UPGRADE.md) - upgrade procedure (back up first)
|
||||
- [CONFIG.md](CONFIG.md) - environment variables and Setting keys
|
||||
104
docs/BACKUP-STALENESS.md
Normal file
@@ -0,0 +1,104 @@
|
||||
<!-- docmeta
|
||||
purpose: explanation
|
||||
audience: developer, operator
|
||||
status: current
|
||||
summary: Why the Backups-stopped card lists a chain, why it stays quiet about one, and why staleness is measured from lastseenat.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# How the backups card decides good vs stale
|
||||
|
||||
The `backups` plugin's health verdict, as shown on the dashboard's **Backups
|
||||
stopped** card. Read it when the card looks wrong, or before changing the rule.
|
||||
Adding a new kind needs none of this - see [BACKUP-KINDS.md](BACKUP-KINDS.md).
|
||||
|
||||
## The trap this is built around
|
||||
|
||||
**A revision is only written when the config CHANGES.** Dedup compares against
|
||||
the latest revision for the chain `(asset, kind, sourcehostname)`, so a machine
|
||||
whose config has been stable for six months has a six-month-old newest revision
|
||||
and is perfectly healthy. The question worth answering is not "when was the last
|
||||
backup taken" - it is "is this still being checked".
|
||||
|
||||
So the timestamp the card uses is `backuprevisions.lastseenat`: the last time
|
||||
the collector CONFIRMED this config, whether or not anything changed. It moves
|
||||
on every successful collection; the revision does not.
|
||||
|
||||
It is a card of exceptions rather than a column of raw dates because a raw date
|
||||
read as neglect. At the default `backups_intervalhours` of **24** the collector
|
||||
only attempts once a day, so a day-old confirmation IS the healthy steady state -
|
||||
and the date made the reader do arithmetic against a setting they would have to
|
||||
go and look up.
|
||||
|
||||
## What it does now
|
||||
|
||||
`GET /api/backups/dashboard/stale` returns one row per chain that has gone
|
||||
quiet, worst first:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `assetid`, `assetnumber` | The asset the backup belongs to. |
|
||||
| `backupkind` | Which kind stopped. Chains fail independently, so the kind is part of the identity, not a summary. |
|
||||
| `sourcehostname` | The PC that was doing the collecting. |
|
||||
| `lastseenat` | When this chain was last CONFIRMED still current (ISO). |
|
||||
| `quietdays` | How long it has been quiet, which is what the card sorts on. |
|
||||
|
||||
`lastseenat` is reported rather than the capture time because they answer
|
||||
different questions and one cannot stand for both: the first proves the check is
|
||||
running, the second only says what is stored. Collapsing them was what made a
|
||||
healthy machine look neglected.
|
||||
|
||||
## The chain is the unit, not the host
|
||||
|
||||
A marker bay runs both an NTLARS backup and a part-marker backup. Reporting one
|
||||
verdict per host would let a healthy kind mask a dead one: the part-marker
|
||||
capture could stop for a month and the host would look fine on the strength of
|
||||
its NTLARS row. So the card is keyed on the CHAIN - asset, kind and source PC
|
||||
together - because that is the unit that fails independently.
|
||||
|
||||
## Silence is not a verdict
|
||||
|
||||
A chain whose newest revision has never been confirmed - `lastseenat` NULL,
|
||||
which means a backfilled row or one written before the column existed - is not
|
||||
stale, it is UNMEASURABLE, and it is skipped rather than listed. Substituting
|
||||
`collectedat` for it looked right and was wrong: on a real fleet it reported
|
||||
every machine whose config had simply been stable for months as a stopped
|
||||
backup, so the whole board lit up and said the site was broken when it was fine.
|
||||
A chain becomes measurable the first time its PC posts, and the card says
|
||||
nothing about it until then.
|
||||
|
||||
The card is equally silent about an asset that has NEVER been backed up. Whether
|
||||
one SHOULD be is a question the asset record cannot answer - most machines carry
|
||||
no NTLARS at all - and guessing would list a hundred healthy assets, which is
|
||||
the noise that makes a board worth ignoring.
|
||||
|
||||
Known consequence: a device retired from a PC leaves its last revision behind,
|
||||
so that chain stays listed until the revision is pruned. That is the honest
|
||||
failure direction - it complains about a backup that genuinely is not running
|
||||
any more, rather than going quiet about one.
|
||||
|
||||
The threshold is the backups plugin's own `backups_staledays` setting (default
|
||||
**3**), read through `plugins.backups.services.staleness.staledays()` rather
|
||||
than re-derived, so there is ONE definition of stale. `0` disables the card.
|
||||
|
||||
## Why time-based rather than per-kind
|
||||
|
||||
Every kind answers the same question the same way: something confirmed this
|
||||
recently, or it did not. Making the rule per-kind would mean each new kind has
|
||||
to define health before it can appear, for no gain. If a kind ever needs its own
|
||||
window - a weekly backup that should not be judged on a 3-day rule - add a
|
||||
threshold override on `BackupKind` and have the rule prefer it; the
|
||||
skip-the-unmeasurable contract stays as it is.
|
||||
|
||||
## Where the code lives
|
||||
|
||||
- `plugins/backups/services/staleness.py` - `staledays()` (the shared
|
||||
threshold), `latestperchain()` (newest revision per chain), `stalechains()`
|
||||
(the verdict)
|
||||
- `plugins/backups/api/routes.py` - `dashboard_stale`, a thin route over that
|
||||
rule so it stays testable without an auth layer in the way
|
||||
- `plugins/backups/plugin.py` - the `backups-stale` dashboard widget
|
||||
declaration, including the `backups_staledays` setting
|
||||
|
||||
Adding a kind is [BACKUP-KINDS.md](BACKUP-KINDS.md); backing up the ShopDB
|
||||
database itself is [BACKUP-RESTORE.md](BACKUP-RESTORE.md), a different subject.
|
||||
292
docs/COLLECTOR-AUTH.md
Normal file
@@ -0,0 +1,292 @@
|
||||
<!-- docmeta
|
||||
purpose: reference
|
||||
audience: adopter, integrator, developer
|
||||
status: current
|
||||
summary: The two ways a shop-floor PC authenticates to the collector API - its own per-PC credential, or the legacy shared key.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Collector authentication
|
||||
|
||||
**This page is the one authoritative description of how a fleet client proves
|
||||
who it is.** Every other page links here rather than restating it.
|
||||
|
||||
There are two credentials, and which one a PC uses is a property of that PC, not
|
||||
of the site:
|
||||
|
||||
| | Per-PC credential | Shared collector key |
|
||||
|---|---|---|
|
||||
| Status | **current** | **legacy - still works, do not build on it** |
|
||||
| Header | `X-Endpoint-Credential` | `X-API-Key` (or `Authorization: Bearer`) |
|
||||
| Identifies | this one PC | any holder of the key |
|
||||
| Issued by | enrollment, once per PC | an admin, once per fleet |
|
||||
| Revocable | per PC | only fleet-wide |
|
||||
| Blast radius if it leaks | one PC's report | every PC's report |
|
||||
|
||||
A PC that has enrolled sends its credential and nothing else. A PC that has not
|
||||
enrolled sends the shared key, exactly as the whole fleet used to. **Never both
|
||||
on one request.**
|
||||
|
||||
Per-PC credentials need the `endpoints` plugin on the server and an endpoint
|
||||
agent (or reporter script) that knows how to enroll. A site running older
|
||||
clients is on the shared key and keeps working; that is the point of keeping
|
||||
both.
|
||||
|
||||
> **Both halves are in this tree.** `plugins/endpoints/api/agentauth.py` resolves
|
||||
> the credential and decides the shared-key fall-through,
|
||||
> `plugins/endpoints/api/enrollment.py` carries enroll, approve, reject, revoke,
|
||||
> allow-reenroll and the migration waves,
|
||||
> `plugins/endpoints/models/credential.py` holds the three tables, and
|
||||
> `shopdb/core/api/collector.py` reads the header on the generic collector
|
||||
> route. The C# agent sends it
|
||||
> (`agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs`) and both shipped
|
||||
> PowerShell reporters fall back to the shared key on their own. The decision
|
||||
> record is [ADR-019](adr/ADR-019-per-pc-endpoint-credentials.md); the wire
|
||||
> contract is `plugins/endpoints/AGENT-CONTRACT.md` section 9, and the fleet
|
||||
> rollout order is `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Which endpoints take which
|
||||
|
||||
| Endpoint | Per-PC credential | Shared key |
|
||||
|---|---|---|
|
||||
| `POST /api/collector/computers` | yes | yes |
|
||||
| `POST /api/collector/printers` | yes | yes |
|
||||
| `POST /api/collector/endpoints` (and `/api/endpoints/collect`) | yes | yes |
|
||||
| `POST /api/collector/<other plugin>` | only if that plugin's schema declares `acceptsagentcredential` | yes |
|
||||
| `POST /api/collector/pc`, `/apps`, `/heartbeat`, `/bulk` | no | yes, see section 5 |
|
||||
| `GET /api/collector/status` | no | yes |
|
||||
| `GET /api/collector/_schemas` | no | no - JWT only, it is for humans |
|
||||
|
||||
A plugin's own unattended routes are outside this table. They authenticate with
|
||||
a scoped managed service token, or with a source-IP allowlist, and they accept
|
||||
neither a per-PC credential nor a collector key. That is the distinction people
|
||||
get wrong: the collector credential says WHICH PC is writing a report, a service
|
||||
token says WHAT an integration is allowed to do.
|
||||
|
||||
## 2. How the server decides
|
||||
|
||||
For a collector endpoint whose plugin schema declares `acceptsagentcredential`:
|
||||
|
||||
1. `X-Endpoint-Credential` present? Resolve it to an asset. Unknown or revoked
|
||||
is `401 CREDENTIAL_INVALID`. The asset the credential resolves to is the asset
|
||||
the request acts for, whatever hostname the body carries.
|
||||
2. Otherwise fall through to shared-key auth (section 4).
|
||||
3. If the shared key authenticated but the named PC **has enrolled**, the
|
||||
request is refused `401 SHARED_KEY_REFUSED`. A PC with its own secret must
|
||||
use it; a shared-key call in its name is an older agent or somebody else.
|
||||
4. A payload that names a different asset than the credential is
|
||||
`403 ASSET_MISMATCH`.
|
||||
|
||||
A PC that has NEVER enrolled has no credential row, is never refused by step 3,
|
||||
and keeps reporting on the shared key indefinitely. That is what carries an
|
||||
un-migrated fleet.
|
||||
|
||||
### Error codes
|
||||
|
||||
| HTTP | Code | Meaning | What the client does |
|
||||
|---|---|---|---|
|
||||
| 401 | `CREDENTIAL_INVALID` | unknown or revoked credential | delete the stored credential, retry once with the shared key, wake enrollment |
|
||||
| 401 | `SHARED_KEY_REFUSED` | shared key sent for an enrolled PC, or the shared key is switched off | use the credential; if there is none, enroll |
|
||||
| 403 | `ASSET_MISMATCH` | the credential's asset is not the one the request names | log and stop. Do not retry the same request |
|
||||
| 401 | `Invalid API key` | the shared key is wrong or missing | fix the key |
|
||||
| 500 | `Collector API key not configured` | server-side: no env key set and no managed token presented | a SERVER config gap, not a client problem |
|
||||
|
||||
`ASSET_MISMATCH` also covers a hostname that is on no computer record at all.
|
||||
The code and status are the same because nothing changes for the client, but the
|
||||
message says which of the two it is and names the hostname. An admin fixes it by
|
||||
setting the hostname on the enrolled computer.
|
||||
|
||||
## 3. Per-PC credentials (current)
|
||||
|
||||
### Shape and storage
|
||||
|
||||
A credential is `sdbe_` plus 43 base64url characters (32 random bytes). The
|
||||
server stores only its SHA-256 plus the first 12 characters for display; the full
|
||||
secret exists exactly once, in the enrollment response.
|
||||
|
||||
On the PC it lives in
|
||||
`%ProgramData%\ShopDB\EndpointAgent\credential\credential.bin`, DPAPI
|
||||
`LocalMachine`, in a folder ACL'd to SYSTEM and Administrators only. The file
|
||||
records the `baseurl` it was issued for, and the credential must never be sent
|
||||
anywhere else. `enroll.json` beside it holds the enrollment nonce in plain text.
|
||||
|
||||
### Enrolling
|
||||
|
||||
`POST {baseurl}/api/endpoints/enroll` with `hostname` and `serialnumber`
|
||||
(required), plus an optional `deviceid`, an optional `agentversion`, and a
|
||||
persisted `enrollnonce` of 32 to 128 characters from `A-Za-z0-9_-`.
|
||||
|
||||
The route never refuses a call for its authentication. A PC installed without the
|
||||
shared key still records a request an admin can approve. The shared key only
|
||||
decides whether a credential is issued on the spot.
|
||||
|
||||
Because it takes no auth and writes rows, it carries the two bounds an open door
|
||||
needs: a per-IP fixed window of 300 calls a minute, sized for a whole shop floor
|
||||
behind one address rather than for one PC, and a cap on how many PENDING rows
|
||||
may exist at once (`plugin.endpoints.maxpendingenrollments`, default 500). The
|
||||
cap turns away only a hostname and serial never seen before, and it drops
|
||||
abandoned rows before it refuses a real PC, so neither bound can lock out a
|
||||
machine already in the queue.
|
||||
|
||||
| Status | Body | What it means |
|
||||
|---|---|---|
|
||||
| 201 | `{"status":"enrolled","credential":"sdbe_...","assetid":123}` | issued. Store it and switch headers; no restart |
|
||||
| 202 | `{"status":"pending","requestid":45,"reason":"serial mismatch"}` | queued for an admin. Retry on a backoff |
|
||||
| 403 | `ENROLL_NONCE_MISMATCH` | the approved request belongs to a different nonce. An admin must reject and re-approve |
|
||||
|
||||
A credential is auto-issued only when ALL of these hold: the shared key
|
||||
authenticated the call, `plugin.endpoints.autoenroll` is on, the hostname matches
|
||||
exactly one computer, the serial on record matches (trimmed, case-insensitive),
|
||||
and the asset has no active credential. Otherwise the request is pending with one
|
||||
of: `unknown hostname`, `duplicate hostname`, `no serial on record`,
|
||||
`serial mismatch`, `already enrolled`, `awaiting approval`, `nonce changed`,
|
||||
`rejected`, `reenroll blocked`.
|
||||
|
||||
There is one request row per (hostname, serial number); repeated calls update it.
|
||||
The nonce is what an approval binds to, so a second machine claiming the same
|
||||
hostname and serial cannot collect the approved credential. A REJECTED row binds
|
||||
to the identity instead of to the nonce: any nonce from that hostname and serial
|
||||
stays rejected, so a reinstall cannot argue its way back into the queue, and
|
||||
approving the request is the only way out of a rejection.
|
||||
|
||||
### Revoking, rotating, rolling back
|
||||
|
||||
Revoking sets `revokedat` on the asset's one credential row. Re-issuing updates
|
||||
that same row, so an asset never carries two credentials.
|
||||
|
||||
By default a revoked PC **falls back to the shared key**. That is deliberate: a
|
||||
revoke is also how an admin rotates a credential, and how a PC is rolled back to
|
||||
an older agent that only knows the shared key. Denying by default would silence
|
||||
that PC instead of rolling it back.
|
||||
|
||||
- `blockreenroll` on the revoke stops the PC enrolling again until an admin calls
|
||||
`allow-reenroll`. That is what makes a revoke stick for a compromised PC.
|
||||
- `plugin.endpoints.revokeddeniessharedkey` (default **off**) makes a revoke a
|
||||
real eviction: the PC reports nothing until it enrolls again. Turn it on only
|
||||
once the fleet is off the shared key, because it also closes the rollback path.
|
||||
|
||||
Admin API, for reference: `GET /api/endpoints/enrollment` (needs
|
||||
`endpoints.fleet`), and the audited admin-role writes
|
||||
`.../requests/<id>/approve`, `.../requests/<id>/reject`,
|
||||
`.../assets/<assetid>/revoke`, `.../assets/<assetid>/allow-reenroll`.
|
||||
|
||||
### Server pins
|
||||
|
||||
A pinned agent accepts the TLS handshake if and only if the leaf certificate's
|
||||
SubjectPublicKeyInfo SHA-256 is in its pin set; with no pins it requires a clean
|
||||
Windows chain validation.
|
||||
|
||||
`HKLM:\SOFTWARE\GE\ShopDB` `ServerPins` (REG_MULTI_SZ, either registry view) is
|
||||
authoritative whenever the value exists, **even when empty** - an empty value
|
||||
means "registry authoritative, no pins". Only when it is absent from both views
|
||||
do the `serverpins` entries in the agent's `appsettings.json` apply. The agent
|
||||
logs which source is in effect at startup and on every change, and re-reads the
|
||||
set within about a minute with no restart.
|
||||
|
||||
The Server Certificate settings page shows the pin; the client kit has
|
||||
`Get-ShopDBServerPin.ps1` to compute one and `Set-ShopDBServerPins.ps1` to write
|
||||
the set (`-None` for an empty authoritative value, `-Clear` to delete it).
|
||||
|
||||
## 4. The shared collector key (legacy)
|
||||
|
||||
### What counts as one
|
||||
|
||||
- A **managed API token** (`shopdb_pat_...`) scoped to ONLY `collector.ingest`,
|
||||
or to `collector.ingest.<pluginname>` for one collector. This is the form to
|
||||
use. It can be minted, rotated and revoked from Settings > API Tokens, and its
|
||||
use shows in `lastusedat` and the audit log.
|
||||
- An **environment key**: `COLLECTOR_API_KEY_<PLUGINNAME>` (per plugin,
|
||||
uppercased) or `COLLECTOR_API_KEY` (shared fallback). Resolution is per-plugin
|
||||
first, then shared. These stay supported as a bootstrap and legacy path.
|
||||
|
||||
An unscoped token, or one scoped to some other permission, is NOT a collector
|
||||
credential and is rejected. `collector.ingest` gates no normal route and is
|
||||
denied on every role-gated endpoint, so a scoped token that leaks can submit
|
||||
collector payloads and do nothing else - not even with an admin owner, because
|
||||
scoping suspends the admin bypass.
|
||||
|
||||
### Header only, fail-closed
|
||||
|
||||
The key travels in `X-API-Key` (Bearer also accepted for a managed token).
|
||||
**The `?api_key=<key>` querystring form was removed on every collector
|
||||
endpoint** - querystring keys leak into access logs, proxy history and browser
|
||||
history. A caller still putting it in the URL gets 401.
|
||||
|
||||
If neither an env key nor a managed token is available server-side, every request
|
||||
is refused. An unconfigured server never silently accepts unauthenticated data.
|
||||
|
||||
### Minting one
|
||||
|
||||
1. Settings > API Tokens > New Token.
|
||||
2. Check **Restrict permissions**, tick only **Submit collector payloads (fleet
|
||||
reporting)** (`collector.ingest`). Name it, optionally set an expiry, Create.
|
||||
3. Copy the `shopdb_pat_...` secret, shown once, and deliver it to the fleet
|
||||
(section 6).
|
||||
|
||||
If you prefer a non-admin owner, create a low-privilege user whose role holds
|
||||
only `collector.ingest` and let it mint its own token; the scope ceiling then
|
||||
caps anything it mints. The admin-minted route is simpler and equally contained.
|
||||
|
||||
### Rotating one
|
||||
|
||||
1. Mint the replacement.
|
||||
2. Deliver it (section 6) - one value per site.
|
||||
3. Watch the new token's `lastusedat` climb and the old one's go stale.
|
||||
4. Revoke the old token. Revocation is immediate.
|
||||
|
||||
The endpoint agent's key has its own kill switch: Settings > Endpoint agent key
|
||||
generates a key scoped to `collector.ingest.endpoints`, revokes any previous one
|
||||
in the same request, and switches off the `.env` fallback for endpoints
|
||||
(`plugin.endpoints.envkeyfallback`). Revoking it stops all endpoint-agent traffic
|
||||
without touching the computers or printers collectors.
|
||||
|
||||
### Switching it off
|
||||
|
||||
`plugin.endpoints.sharedkeyallowed` (default on) gates the shared key for the
|
||||
endpoints collector. Turning it off is the last step of a migration, not the
|
||||
first: every PC that has not enrolled stops reporting the moment it flips.
|
||||
|
||||
## 5. The four legacy core routes
|
||||
|
||||
`POST /api/collector/pc`, `/apps`, `/heartbeat` and `/bulk` predate both the
|
||||
generic collector contract and per-PC credentials. They take their target
|
||||
hostname from the body and rewrite that PC's logged-in user, boot time, serial
|
||||
number and installed apps, so the shared key is an **any-host write key** on
|
||||
them and no per-PC credential narrows it. `plugin.endpoints.sharedkeyallowed`
|
||||
does not reach them.
|
||||
|
||||
`collector.legacysharedkeyrefused` (default **false**, today's behaviour) closes
|
||||
them to the shared key; a collector-scoped managed token still works. Every
|
||||
shared-key use on these routes is logged whether the gate is on or off, so read
|
||||
the "Legacy collector route used with the shared key" warnings for a while first
|
||||
- whatever still calls them stops working the moment it is on. Neither shipped
|
||||
reporter is affected; they post to `/api/collector/computers` and
|
||||
`/api/collector/printers`.
|
||||
|
||||
Per ADR-006 the legacy `/pc` path is deprecated and slated for removal before
|
||||
v1.0. New integrations target `POST /api/collector/<plugin>`.
|
||||
|
||||
## 6. Delivering the shared key to a PC
|
||||
|
||||
Do NOT hardcode a key in a script that lives on a file share: a literal key there
|
||||
is published to everyone with share read. In order of preference:
|
||||
|
||||
1. **`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`**, written at imaging
|
||||
time, by DSC, or by an Intune platform script. This is
|
||||
where both shipped reporters look. Restrict the key's ACL to SYSTEM and
|
||||
Administrators so an interactive or kiosk user cannot read it.
|
||||
2. **A per-site config value** the machine account already reads -
|
||||
`collectorApiKey` in `C:\Enrollment\site-config.json`. Rotation is one value
|
||||
per site, no re-image and no script redeploy.
|
||||
3. **A machine-local file** staged at imaging
|
||||
(`C:\Enrollment\collector-api-key.txt`), ACL'd to SYSTEM. Tighter blast radius
|
||||
- the key never sits on the share at all - but rotation means touching every
|
||||
PC.
|
||||
|
||||
Never put a token in a scheduled task's `-Argument` string: task XML is
|
||||
world-readable. Never put one in a manifest: manifests sync broadly.
|
||||
|
||||
A per-PC credential needs none of this. It is issued to the PC by enrollment and
|
||||
never leaves it.
|
||||
147
docs/COLLECTOR-CLIENT-SETUP.md
Normal file
@@ -0,0 +1,147 @@
|
||||
<!-- docmeta
|
||||
purpose: howto
|
||||
audience: operator, adopter
|
||||
status: current
|
||||
summary: Get the shipped asset and printer reporters running on a fleet, prove it, and fix it when a bay returns an error.
|
||||
loadwith: COLLECTOR-AUTH.md
|
||||
-->
|
||||
|
||||
# Running the collector reporters on a fleet
|
||||
|
||||
Follow this to put PC reporting on a fleet that does not have it yet, or to move
|
||||
an existing fleet off a classic-ASP reporter. The payload contract is
|
||||
[COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) and the credential choice is
|
||||
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); you should not need either open to work
|
||||
through this page.
|
||||
|
||||
Adopting the whole tool set at a plant that has none of it is a wider job -
|
||||
[ADOPTING-AT-ANOTHER-SITE](ADOPTING-AT-ANOTHER-SITE.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Use the shipped reporters
|
||||
|
||||
Do not write your own. Two scripts ship in this repository and version with the
|
||||
contract they implement:
|
||||
|
||||
| Script | Reports | Posts to |
|
||||
|---|---|---|
|
||||
| `plugins/computers/client/Report-AssetToShopDB.ps1` | hostname, BIOS serial, pc-type, logged-in user, make/model, OS, last boot, machine number, device id, every routable NIC | `POST /api/collector/computers` |
|
||||
| `plugins/printers/client/Report-PrintersToShopDB.ps1` | every real print queue on the host | `POST /api/collector/printers` |
|
||||
|
||||
Both run as SYSTEM, always exit 0 so "last run result" stays clean, and log to
|
||||
`C:\Logs\Shopfloor\`. Neither names a site: ADR-015 makes a site name in product
|
||||
code a defect, and these ship everywhere.
|
||||
|
||||
Each reads its configuration in this order:
|
||||
|
||||
1. Its own parameters (`-ApiUrl`, `-ApiKey`), which a scheduled task's arguments
|
||||
can supply for testing.
|
||||
2. `HKLM:\SOFTWARE\GE\ShopDB` - `BaseUrl` for the server and `CollectorKey` for
|
||||
the shared key. This is the secret store every shipped reporter reads, and
|
||||
whatever images or configures the bay is what provisions it.
|
||||
|
||||
With neither, the script logs the reason and exits without posting.
|
||||
|
||||
**Which credential it sends** is decided on the PC, not in configuration: if the
|
||||
PC holds a per-PC credential it is sent as `X-Endpoint-Credential` and the shared
|
||||
key is not sent at all; otherwise the shared key goes as `X-API-Key`. If the
|
||||
server refuses the credential as invalid or revoked, the reporter retries the
|
||||
same POST once with the shared key, which is what makes a revoke a rollback
|
||||
rather than an outage.
|
||||
|
||||
**Shortcut for a single site:** Settings > Computers > Asset reporter generates
|
||||
`Report-AssetToShopDB.ps1` with your server's URL already in the parameter
|
||||
defaults, and shows its SHA-256. The key is deliberately not included.
|
||||
|
||||
## 2. Prepare the server
|
||||
|
||||
1. Enable the `computers` plugin (and `printers`, if you want observed queues).
|
||||
2. Mint the credential the fleet will start on - a managed token scoped to
|
||||
`collector.ingest`. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 4 has the
|
||||
steps. An un-enrolled fleet needs this even if you intend to move to per-PC
|
||||
credentials, because enrollment itself auto-issues only to a call the shared
|
||||
key authenticated.
|
||||
3. Set `collector_corporate_ranges` to this site's routable CIDRs. Leave it blank
|
||||
and the server falls back to the NIC the bay says carries the default route,
|
||||
which is a guess on any bay with two NICs.
|
||||
4. Map your imaging pc-types. Unmapped values only warn, but every report warns
|
||||
until you do.
|
||||
|
||||
## 3. Deliver the key to the PCs
|
||||
|
||||
`HKLM:\SOFTWARE\GE\ShopDB` value `CollectorKey`, ACL'd to SYSTEM and
|
||||
Administrators. How it gets there depends on what you run - an imaging task,
|
||||
Intune, DSC or by hand - and the options with their trade-offs are in
|
||||
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) section 6.
|
||||
|
||||
Two rules regardless: never bake the key into a script that lives on a file
|
||||
share, and never put it in a manifest or a scheduled task's arguments.
|
||||
|
||||
## 4. Schedule the reporter
|
||||
|
||||
Reporting is an EVENT on a schedule, not a state to converge on. Two placements,
|
||||
pick one:
|
||||
|
||||
- **As a scheduled task**, registered at imaging time, by Intune, or by hand.
|
||||
SYSTEM, RunLevel Highest, at startup plus an interval.
|
||||
- **As an entry in whatever enforces configuration on your floor**, run off a
|
||||
share every cycle with no local copy. Order the key-provisioning step BEFORE
|
||||
it so the key is present the same cycle the first report fires.
|
||||
|
||||
Either way, run it ALONGSIDE whatever reports today during rollout. Do not
|
||||
remove the old path until the new one is proven.
|
||||
|
||||
## 5. Stage the rollout
|
||||
|
||||
1. Leave the existing reporting in place. Provision `CollectorKey` and enable the
|
||||
plugins.
|
||||
2. Add the reporter on a handful of pilot bays, one per pc-type. It cannot break
|
||||
enforcement: it never throws and always exits 0.
|
||||
3. Verify on the server. Pilot PCs appear or update, `action` is
|
||||
`created`/`updated`, and every warning is understood - map any unmapped
|
||||
`pctype`, add missing OS strings to the `operatingsystems` vocabulary.
|
||||
Cross-check the client log's `OK` lines against the audit log.
|
||||
4. Roll it out fleet-wide. From the `common` scope, every pc-type gets it.
|
||||
5. Once ShopDB is the system of record and has survived a full imaging cycle,
|
||||
retire the old reporter.
|
||||
6. Migrate to per-PC credentials, then switch the shared key off. That is its
|
||||
own rollout and deliberately comes last: read
|
||||
[COLLECTOR-AUTH](COLLECTOR-AUTH.md) for what a PC does to enroll and what an
|
||||
admin approves, and `plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`
|
||||
for the order the shared key has to be cleared out of the places it lives.
|
||||
Both shipped reporters already send a credential when one is present and fall
|
||||
back to the shared key when it is not, so a PC that has not enrolled keeps
|
||||
reporting and nothing has to be redeployed to start.
|
||||
|
||||
## 6. Verify one PC
|
||||
|
||||
```powershell
|
||||
Invoke-RestMethod -Uri 'https://shopdb.example.net/api/computers/by-hostname/<HOSTNAME>'
|
||||
```
|
||||
|
||||
The PC should exist with its serial, type and last-reported timestamp. The
|
||||
client log is `C:\Logs\Shopfloor\report-asset-YYYYMMDD.log` (printer queues:
|
||||
`report-printers-YYYYMMDD.log`); it records the POST line and the server's
|
||||
response verbatim, including which credential it used.
|
||||
|
||||
## 7. When a bay does not report
|
||||
|
||||
| Symptom | HTTP | Cause / fix |
|
||||
|---|---|---|
|
||||
| `Invalid API key` | 401 | The `X-API-Key` header is missing or wrong. Confirm the client key matches `COLLECTOR_API_KEY_COMPUTERS` (or the shared `COLLECTOR_API_KEY`) or is the managed token you minted. Check the client actually read it. The querystring form no longer works. |
|
||||
| `Collector API key not configured` | 500 | Fail-closed: no env key set server-side and the caller presented no managed token. A SERVER config gap, not a client problem. |
|
||||
| `Endpoint credential is unknown or revoked` | 401 | The PC's credential was revoked, or the database was restored past its issue. The reporter falls back to the shared key on its own; let the PC enroll again to restore it. |
|
||||
| `The shared key is not accepted for this endpoint` | 401 | This PC has enrolled, so it must use its own credential - or the shared key is switched off site-wide. Check that `credential.bin` is present and readable by SYSTEM. |
|
||||
| `The request names a different endpoint than the credential` | 403 | The hostname in the payload is on a different asset than the credential, or on no computer record at all. The message says which. Do not retry; fix the record. |
|
||||
| `No collector registered for plugin computers` | 404 | The plugin is disabled or not loaded on that instance. Enable it. |
|
||||
| `hostname is required` / `No data provided` | 400 | Empty body, non-JSON body, or missing identity field. Check `-ContentType 'application/json'`. |
|
||||
| `Internal error processing collector payload` | 500 | Generic by design - the server does not leak the cause. The real error is in the server log (`Collector upsert failed for computers`). |
|
||||
| `queues is required; send an empty array for a host with no queues` | 400 | The printers payload omitted `queues` entirely. Absent and empty are not the same thing here. |
|
||||
| A bay's observed queues all vanished | 200 | It reported `queues: []`, which legitimately clears the host. Check the client log for `posting NOTHING` (a failed enumeration correctly sends nothing) versus an empty set. If the bay really has queues, fix the enumeration on that host. |
|
||||
| Observed queues show as `unknown` | 200 | The queue matched no printer asset by port address or name. Add the printer to ShopDB with its IP and re-read - matching happens at read time, so the bay does not need to report again. |
|
||||
| `warnings` present but `action` is created/updated | 200 | Soft issues only; the row WAS written. Common: unmapped `pctype`, unknown `osname`, unknown app name, `pcsubtype` not stored. No action needed unless the warning matters to you. |
|
||||
| The wrong IP is recorded as primary | 200 | `collector_corporate_ranges` is blank or does not cover this bay. The bay reports every NIC and takes no view; set the ranges. |
|
||||
|
||||
Client-side logs are under `C:\Logs\Shopfloor\`. Server-side, the Flask app log
|
||||
carries upsert failures and per-plugin schema failures.
|
||||
85
docs/COLLECTOR-INTEGRATION.md
Normal file
@@ -0,0 +1,85 @@
|
||||
<!-- docmeta
|
||||
purpose: explanation
|
||||
audience: operator, adopter, integrator, developer
|
||||
status: current
|
||||
summary: What the collector is, what it guarantees, and which of the three collector pages to open next.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# The collector
|
||||
|
||||
The collector is how the shop-floor fleet pushes what it knows about itself into
|
||||
ShopDB. It replaced the classic-ASP `api.asp?action=updateCompleteAsset` path.
|
||||
|
||||
This page is the shape and the guarantees. It is short on purpose: the detail
|
||||
lives in three pages below, and you almost never need more than one of them.
|
||||
|
||||
## Which page you want
|
||||
|
||||
| You are | Open |
|
||||
|---|---|
|
||||
| Rolling reporting out to a fleet, or a bay is returning an error | [COLLECTOR-CLIENT-SETUP](COLLECTOR-CLIENT-SETUP.md) |
|
||||
| Writing or changing a client, or answering "what does this field do" | [COLLECTOR-PAYLOADS](COLLECTOR-PAYLOADS.md) |
|
||||
| Deciding how a PC proves who it is, minting or revoking a credential | [COLLECTOR-AUTH](COLLECTOR-AUTH.md) |
|
||||
| Asking why the contract is shaped this way | [ADR-006](adr/ADR-006-collector-contract.md) |
|
||||
|
||||
Where all of this sits relative to EventSaver, the backup collectors and the
|
||||
kiosks is [FLEET-ARCHITECTURE](FLEET-ARCHITECTURE.md).
|
||||
|
||||
## The shape
|
||||
|
||||
One dynamic route, `POST /api/collector/<plugin>`, serves every enabled plugin
|
||||
that returns a collector schema. A plugin declares its identity field and its
|
||||
accepted fields; the dispatcher handles auth, parsing, the audit row and the
|
||||
error envelope, and the plugin handles the upsert. Adding a collector is adding a
|
||||
schema, not adding a route.
|
||||
|
||||
Two plugins use it on a normal fleet:
|
||||
|
||||
- `computers` - what this PC is. This is the record everything else hangs off.
|
||||
- `printers` - what print queues this PC actually has.
|
||||
|
||||
The `endpoints` plugin adds a third, much larger payload for sites running the
|
||||
endpoint agent.
|
||||
|
||||
## What it guarantees
|
||||
|
||||
**Idempotent.** The same report twice changes nothing. A PC is matched by its
|
||||
identity field, so re-imaging a bay updates its record instead of duplicating it,
|
||||
and existing asset relationships survive.
|
||||
|
||||
**Patch-style, where it says so.** On the computers payload an omitted field
|
||||
leaves the stored value alone, so a report from a bay that could not read its
|
||||
BIOS never blanks a serial number. The printers payload is deliberately the
|
||||
opposite: the reported set REPLACES the host's queues, because "what does this
|
||||
bay have right now" is not a question about history.
|
||||
|
||||
**Warnings, not failures, for soft problems.** An unmapped pc-type, an unknown OS
|
||||
string or an unknown application name is reported back in `warnings` and the row
|
||||
is still written. The alternative - refusing the report - loses the whole PC over
|
||||
one unrecognised string.
|
||||
|
||||
**Observed is never assigned.** Nothing a PC reports becomes a statement of what
|
||||
it SHOULD have. A bay that installed the wrong printer cannot make itself right
|
||||
by reporting it; turning observed state into assigned state is always a person's
|
||||
decision.
|
||||
|
||||
**The report says what the PC is, not where it is on the network.** The bay
|
||||
reports every NIC and takes no view of which matters. Only the server knows the
|
||||
site's addressing, so only the server picks.
|
||||
|
||||
**Fail-closed on authentication.** A server with no credential configured refuses
|
||||
every request rather than silently accepting unauthenticated data. Credentials go
|
||||
in a header, never a querystring, so they stay out of access logs and proxy
|
||||
history.
|
||||
|
||||
## The one thing that changed recently
|
||||
|
||||
A PC used to authenticate with a key the whole fleet shared. It now
|
||||
authenticates as ITSELF, with a per-PC credential issued by enrollment and
|
||||
revocable one PC at a time. The shared key still works for PCs that have not
|
||||
enrolled, and is refused for any PC that has.
|
||||
|
||||
If you are adopting this product today, plan on per-PC credentials and treat the
|
||||
shared key as the bootstrap. [COLLECTOR-AUTH](COLLECTOR-AUTH.md) is the whole
|
||||
story.
|
||||
399
docs/COLLECTOR-PAYLOADS.md
Normal file
@@ -0,0 +1,399 @@
|
||||
<!-- docmeta
|
||||
purpose: reference
|
||||
audience: integrator, developer
|
||||
status: current
|
||||
summary: Field-by-field contract for the computers and printers collector payloads, and what the server does with each.
|
||||
loadwith: COLLECTOR-AUTH.md
|
||||
-->
|
||||
|
||||
# Collector payload contract
|
||||
|
||||
What a fleet client may send to `POST /api/collector/<plugin>` and what the
|
||||
server does with each field. Authentication is a separate subject and lives in
|
||||
[COLLECTOR-AUTH](COLLECTOR-AUTH.md); the rationale for the contract shape is
|
||||
[ADR-006](adr/ADR-006-collector-contract.md).
|
||||
|
||||
Source of truth is `get_collector_schema` in each plugin
|
||||
(`plugins/computers/plugin.py`, `plugins/printers/plugin.py`). If you change a
|
||||
payload, change it there and re-check the table here.
|
||||
|
||||
---
|
||||
|
||||
## 1. The generic endpoint
|
||||
|
||||
One dynamic route serves every enabled plugin that returns a collector schema.
|
||||
For the PC fleet that is `POST /api/collector/computers`.
|
||||
|
||||
Request flow inside `generic_collect`:
|
||||
|
||||
1. Look up the plugin's schema. Unknown or disabled plugin -> `404 No collector
|
||||
registered for plugin <plugin>`.
|
||||
2. Authenticate (COLLECTOR-AUTH).
|
||||
3. Parse the JSON body. Missing or non-object -> `400 No data provided`.
|
||||
4. Resolve the identity field the schema names (`hostname` for both collectors
|
||||
below). Missing or blank -> `400 <identityfield> is required`.
|
||||
5. Idempotent upsert: the plugin's `apply_collector_payload` finds the existing
|
||||
asset by identity and updates it, or inserts a new one. The same identity on a
|
||||
later submission updates the same row - re-imaging a PC does not duplicate it,
|
||||
and existing asset relationships are preserved.
|
||||
6. Audit: every accepted submission writes an `AuditLog` row (`action`
|
||||
created/updated, entity `Collector`, `details = collector:<plugin>
|
||||
action=<action>`), then commits.
|
||||
|
||||
### Response
|
||||
|
||||
HTTP 200, in the standard envelope, with the collector result under `data`:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"status": "ok",
|
||||
"action": "created",
|
||||
"assetid": 12345,
|
||||
"identityvalue": "SHOPPC2335",
|
||||
"warnings": ["unknown operating system: Windows 11 Enterprise 23H2 (build 22631)"]
|
||||
},
|
||||
"message": "computers collector created",
|
||||
"meta": { "timestamp": "...", "requestid": "..." }
|
||||
}
|
||||
```
|
||||
|
||||
`action` is `created`, `updated` or `noop`. `warnings` is a list of soft problems
|
||||
(unmapped pc-type, unknown OS, unknown app, un-stored pcsubtype) that did NOT
|
||||
fail the request - the row was still written.
|
||||
|
||||
Errors use the same envelope with `status: "error"` and the detail under
|
||||
`data.error`:
|
||||
|
||||
```json
|
||||
{ "status": "error", "data": { "error": { "code": "UNAUTHORIZED", "message": "Invalid API key" } }, "meta": { } }
|
||||
```
|
||||
|
||||
A plugin raises `ValueError` for controlled validation failures and its message
|
||||
is returned verbatim (400). Any other exception is caught, logged server-side,
|
||||
the transaction rolled back, and the caller gets a generic `500 Internal error
|
||||
processing collector payload` with no internal detail - by design. Check the
|
||||
server log to see what actually failed.
|
||||
|
||||
### Schema discovery
|
||||
|
||||
`GET /api/collector/_schemas` returns the schema for every enabled plugin. It is
|
||||
JWT-protected, not key-protected: it is for humans and tooling introspecting what
|
||||
payloads are accepted, not for the headless collectors themselves.
|
||||
|
||||
---
|
||||
|
||||
## 2. Computers payload
|
||||
|
||||
Identity field `hostname`. Every other field is optional and patch-style: an
|
||||
omitted field leaves the existing value untouched, so a bare report never blanks
|
||||
a column.
|
||||
|
||||
| Payload field | Type | Server behaviour (`apply_collector_payload`) |
|
||||
|---|---|---|
|
||||
| `hostname` (required) | string | Identity. Matches `Computer.hostname` (case-insensitive), then falls back to `Asset.assetnumber`. New asset created if no match. |
|
||||
| `machinenumber` | string | Identifies the MACHINE this PC drives, never the PC. A new PC always takes `assetnumber = hostname`, and an existing PC's `assetnumber` is left alone. The number resolves a machine asset and builds a PC -> machine `controls` relationship; an unknown number warns rather than creating a machine. The placeholder `9999` and empty string link nothing. See section 3. |
|
||||
| `deviceid` | string | Identifies the DEVICE hanging off this PC, read from `C:\Enrollment\asset-id.txt`. It names the asset and nothing else - it does NOT say what kind of device it is, because the pc-type already does - so the same file serves a part marker, a Keyence, a Genspect or anything declared later. SEPARATE from `machinenumber` on purpose: "which bay is this" and "which device is this" are different facts, and `machinenumber` is what a site's own bay-gated configuration keys off, so naming a device there would silently stop every bay-gated rule from matching. An unknown value warns rather than minting; a value resolving to the wrong KIND of asset is refused with the asset named. See section 4. |
|
||||
| `measuringtoolid` | string | The 0.12.0 name for the same thing, read from `C:\Enrollment\measuringtool-id.txt`. Still accepted as an alias so a bay already staged with that file keeps reporting; `deviceid` wins when both arrive. Prefer `asset-id.txt` for anything new. |
|
||||
| `pctype` | string | `gea-shopfloor-*` imaging type -> `Computer.computertypeid` via the configurable `pctypemap` settings. Unmapped value -> warning, not error. See section 5. |
|
||||
| `pcsubtype` | string | Accepted but not stored yet -> warning. |
|
||||
| `serialnumber` | string | `Asset.serialnumber`. |
|
||||
| `loggedinuser` | string | `Computer.loggedinuser`. (`currentuser` is also accepted as a legacy alias.) |
|
||||
| `lastboottime` | ISO-8601 datetime | `Computer.lastboottime`. Unparseable -> warning. |
|
||||
| `lastcheckin` | ISO-8601 datetime | Accepted (heartbeat semantics). `lastreporteddate` is set server-side on every call regardless. |
|
||||
| `interfaces` | array of `{ipaddress, macaddress, adaptername, isdefaultroute}` | EVERY connected hardware NIC with a routable IPv4, in no meaningful order. One `Communication` row per NIC. See "Which NIC is the PC's address" below. |
|
||||
| `ipaddress` | string | The one-address form, kept for bays on an older collector. A payload carrying both wins on `interfaces`. |
|
||||
| `vendorname` | string | `Computer.vendorid`. Vendor row auto-created if missing (free vocab). |
|
||||
| `modelnumber` | string | `Computer.modelnumberid`, scoped to the vendor when known. Model row auto-created if missing. |
|
||||
| `osname` | string | `Computer.osid`. Controlled vocab: looked up in `operatingsystems`, NOT auto-created. Unknown value -> warning (row still written, `osid` left unset). |
|
||||
| `installedsoftware` | array of `{name, version}` | `ComputerInstalledApp` rows for applications shopdb already tracks. Unknown app name -> warning, skipped. |
|
||||
| `accessprotocols` | array of strings | Remote-access protocols this PC actually exposes, by catalog name (`VNC`, `WinRM`, `RDP`). PRESENCE of the key drives the sync: reported protocols are activated, catalogued ones not reported are deactivated. Omit the key entirely to leave existing rows alone - most came from the legacy `isvnc`/`iswinrm` migration. |
|
||||
|
||||
`defaultprinter` and `printers` are **retired**. See section 6.
|
||||
|
||||
### Which NIC is the PC's address
|
||||
|
||||
The bay reports every NIC and takes no view. The server picks, in this order:
|
||||
|
||||
1. The first reported address inside a CIDR in the `collector_corporate_ranges`
|
||||
setting.
|
||||
2. Else the NIC the bay flagged `isdefaultroute`.
|
||||
3. Else the first reported.
|
||||
|
||||
Step 3 is a guess, and configuring the ranges is what makes it stop being one.
|
||||
Choosing on the bay was the wrong place for it: a bay cannot know a site's
|
||||
addressing, so a CMM with a fast link to its controller reported the machine-side
|
||||
address as the PC's and never sent the corporate one at all.
|
||||
|
||||
Rows are keyed on the MAC where the bay sent one, because an address moves
|
||||
between adapters and a MAC does not, and fall back to the address for a bay that
|
||||
reports none. Exactly one row is flagged primary, and only among the rows this
|
||||
report touched. A NIC the bay did NOT report is left alone: a NIC that is down at
|
||||
check-in is not evidence the record is wrong, and hand-entered detail lives on
|
||||
those same rows.
|
||||
|
||||
## 3. PC -> machine links, and part markers
|
||||
|
||||
The reported `machinenumber` says which machine the PC drives. It is not the
|
||||
PC's own identifier, and writing it as one is a defect this contract used to
|
||||
describe: `assets.assetnumber` is uniquely indexed and the machine already holds
|
||||
that value, so the insert failed and the bay got a 500 on every report, forever.
|
||||
A PC takes its hostname as its asset number and keeps it.
|
||||
|
||||
- Link: PC -> machine as `controls`, tagged
|
||||
`assetrelationships.label = 'collector:machine'`. Only tagged rows are ever
|
||||
archived by a push, so a link made by hand is never touched.
|
||||
- Unknown number: warns and links nothing. A machine is never invented, because
|
||||
a mistyped number would create equipment nobody can account for.
|
||||
- A second PC reporting a machine another PC holds is a CLAIM, not a handover.
|
||||
A PC imaged for a machine carries that number before it reaches the floor, so
|
||||
the PC currently running the machine keeps the link while it is still alive -
|
||||
reporting within 24 hours (`MACHINE_CLAIM_QUIET_HOURS`) and still `In Use`.
|
||||
The challenger is recorded as a dormant link, which also marks the claim as
|
||||
already announced so it does not alert every cycle.
|
||||
- Handover completes on its own once the incumbent has been quiet past that
|
||||
window, or immediately when someone moves it off `In Use` - which is the
|
||||
one-step way to force a swap at the moment it happens. The old link is
|
||||
archived, never deleted, so "which PC ran this machine in June" stays
|
||||
answerable. The retired PC's STATUS is deliberately not changed: the collector
|
||||
cannot tell shelved from broken from re-imaged.
|
||||
- Alerts for handover and contested claims are gated on the
|
||||
`computers_machinelink_alerts` setting, which ships OFF. A site may run
|
||||
several PCs on one machine number legitimately, and there the alerts fire on
|
||||
correct data. Warnings ride in the collector response either way.
|
||||
|
||||
A PC reporting `pctype = gea-shopfloor-partmarker` is handled differently,
|
||||
because several markers can serve one machine number and an operation holds any
|
||||
number of them. Such a PC gets its own Part Marker machine asset
|
||||
(`<HOSTNAME>-PARTMARKER`), `controls` it, and the marker is filed `partof` the
|
||||
operation the PC reported. Both links are tagged `collector:partmarker`. The PC
|
||||
does NOT claim the operation directly: `controls` propagates through `partof`,
|
||||
so control still follows from controlling the marker, without two markers
|
||||
contesting a link only one can hold. Backups from a marker PC file against the
|
||||
marker rather than the operation.
|
||||
|
||||
That is one instance of a general pattern - several devices under one parent.
|
||||
[ASSET-COMPOSITION](ASSET-COMPOSITION.md) covers when to use it, what
|
||||
propagation buys you, and how to declare a new device type without a code change.
|
||||
|
||||
## 4. PC -> device links (`asset-id.txt`)
|
||||
|
||||
Some pc-types mean the PC drives an attached DEVICE: a part marker, or a
|
||||
metrology instrument such as a CMM, Keyence, Genspect or wax-trace unit.
|
||||
`SUBORDINATE_DEVICE_MAP` in `plugins/computers/pctypemap.py` declares which, and
|
||||
a site can add its own through a `subordinatedevice_<pctype>` setting without a
|
||||
code change.
|
||||
|
||||
**On this line only the part-marker case does anything.** The four metrology
|
||||
pc-types declare `assettype: measuring_tool`, and the plugin that owned that
|
||||
asset class is not bundled here (ADR-018: metrology instruments are machines).
|
||||
The collector detects the absence, skips the link, and says so in the response
|
||||
warnings - it does not fail the report, and it does not mint anything. File a
|
||||
CMM as a machine and link it by hand.
|
||||
|
||||
`C:\Enrollment\asset-id.txt` names that device: one line, the device's
|
||||
`assetnumber`. It does NOT record what kind of device it is - the pc-type already
|
||||
does - so a bay declared later needs no new file and no client change.
|
||||
|
||||
Resolution runs most-stable-identity first, and ADOPTS before it mints:
|
||||
|
||||
1. **`deviceid`** - the asset named in `asset-id.txt`.
|
||||
2. A prior collector link from this PC, reactivated.
|
||||
3. An existing device this PC already controls that the collector did NOT
|
||||
create - a legacy `MT-####` row, or one somebody made by hand. Adopting
|
||||
stamps the label so it is ours from then on.
|
||||
4. Mint, only when none of the above matched.
|
||||
|
||||
**Why the file exists, and why it is step 1.** Every other identity here is
|
||||
derived from the PC. Step 2 looks for a link from THIS PC asset; step 4 mints
|
||||
`<PC number>-<SUFFIX>`, and a PC's asset number is its hostname. All of them
|
||||
survive a re-image and NONE of them survives a PC SWAP: a new hostname is a new
|
||||
PC asset with no prior link and a predicted number that has never existed, so
|
||||
the same physical device gets a second record while the first keeps its config
|
||||
and backup history under a dead PC's name. That is how 43 legacy `MT-####`
|
||||
tools ended up shadowed by minted twins, three records deep in places. Treat
|
||||
anything minted as a placeholder to be reconciled.
|
||||
|
||||
Two guards on the file's contents:
|
||||
|
||||
- An **unresolvable** value warns and links nothing, rather than inventing a
|
||||
phantom device nobody can account for.
|
||||
- A value resolving to the **wrong kind** of asset is refused, naming the asset
|
||||
it hit. A machine number pasted into `asset-id.txt` would otherwise file that
|
||||
MACHINE under a device label the collector also owns - a link that reads as an
|
||||
instrument, or as a marker, everywhere downstream.
|
||||
|
||||
A part marker additionally stays `partof` the operation from `machinenumber`,
|
||||
because several markers serve one operation. Naming the marker in `asset-id.txt`
|
||||
changes which marker asset the PC controls; it does not change which operation
|
||||
that marker files under.
|
||||
|
||||
## 5. pc-type mapping (configurable per site)
|
||||
|
||||
`pctype` (e.g. `gea-shopfloor-cmm`) maps to a shopdb Computer Type through
|
||||
`pctypemap_<pxetype>` settings. The built-in defaults in
|
||||
`plugins/computers/pctypemap.py` are seeded on plugin install and the collector
|
||||
reads them, so existing enrollment keeps working; add a setting of that name to
|
||||
map an imaging type of your own. Unmapped pc-types produce a warning, not a
|
||||
failure, because a bay must not stop reporting its inventory over a naming
|
||||
question.
|
||||
|
||||
## 6. PC -> printer relationships are NOT set here (retired)
|
||||
|
||||
**A collector report has never been allowed to set a printer assignment, and as
|
||||
of 2026-08-24 it no longer can.** `defaultprinter` and `printers` used to be
|
||||
accepted on the computers payload and written as `defaultprinter`/`connectedto`
|
||||
relationship rows. That contradicted "observed is not assigned" below, and this
|
||||
contract carried both claims at once while the code followed the wrong one.
|
||||
|
||||
If you are still sending either key: **your push still succeeds** - they are
|
||||
ignored, and the response carries a warning naming them. To report what a PC
|
||||
actually has, use `POST /api/collector/printers`; to turn that into an
|
||||
assignment, a person calls
|
||||
`POST /api/printers/assignments/seed-from-observed/<assetid>` after reviewing the
|
||||
comparison.
|
||||
|
||||
## 7. Printers payload (observed print queues)
|
||||
|
||||
`POST /api/collector/printers`. ShopDB has always known what a bay SHOULD have -
|
||||
`GET /api/printers/for-host/<hostname>`, applied by `Set-ShopdbPrinters.ps1` -
|
||||
and has never known what it actually has. This payload is that missing half.
|
||||
Same dispatcher, same auth, same audit row; only the payload and the plugin
|
||||
differ.
|
||||
|
||||
- Schema + apply: `get_collector_schema` / `apply_collector_payload` in
|
||||
`plugins/printers/plugin.py`.
|
||||
- Client: `plugins/printers/client/Report-PrintersToShopDB.ps1` (SYSTEM context,
|
||||
`Type=PS1` / DetectionMethod `Always` manifest entry, logs to
|
||||
`C:\Logs\Shopfloor\report-printers-YYYYMMDD.log`, always exits 0).
|
||||
- Storage: `printerobservedqueues`, one row per observed queue, owned by the
|
||||
printers plugin. A separate table from the assignment on purpose.
|
||||
|
||||
The identity field is `hostname`. Unlike the computers payload this one is NOT
|
||||
patch-style: `queues` is required, and the reported set replaces everything
|
||||
previously recorded for that host.
|
||||
|
||||
| Payload field | Type | Server behaviour |
|
||||
|---|---|---|
|
||||
| `hostname` (required) | string | Identity of the report. `COMPUTERNAME` or its FQDN; matched case-insensitively. Rows are keyed by the NAME, so a bay that reports before its PC record exists still records everything - the report resolves to an asset the moment that record appears. An unknown hostname is a warning, never an error. |
|
||||
| `queues` (required) | array of objects | Every real print queue on the host. This REPLACES the host's previous set. An empty array is a valid report meaning "this bay has no queues" and clears the host's rows. A missing key is rejected (400), because one client bug that dropped the field would otherwise erase the fleet's observed state host by host. |
|
||||
| `queues[].queuename` (required) | string | Windows printer name. A queue with no name is skipped with a warning; a repeated name within one report is dropped with a warning (Windows cannot hold two queues of one name). |
|
||||
| `queues[].drivername` | string | Driver name verbatim, as the INF spells it. Compared against the assignment's driver to detect drift. |
|
||||
| `queues[].portname` | string | Windows port name. Also tried as a match key, since a port created outside the client is usually named after the host address. |
|
||||
| `queues[].portaddress` | string | `PrinterHostAddress` of a TCP/IP port - an IP or FQDN. The PRIMARY key for matching a queue to a printer asset. Omit it for a non-TCP port (USB, WSD, redirected); such a queue is still worth reporting and matches on name alone. |
|
||||
| `queues[].isdefault` | boolean | True on the one queue that is the user's default. If several arrive true, the first is kept and the rest are cleared with a warning - a bay has exactly one default, and keeping both would leave a seed picking one at random. |
|
||||
| `queues[].isshared` | boolean | True when the queue is shared off this PC. Recorded, not acted on. |
|
||||
| `observedat` | ISO-8601 datetime | Accepted and ignored, as is any other client timestamp. The server stamps `observedat` at ingest, one value for the whole report, so a bay with a wrong clock cannot report itself fresh or stale. |
|
||||
|
||||
Response `data` is the standard collector shape: `action` is always `updated`
|
||||
(this endpoint replaces rows and creates no asset - calling an identical
|
||||
re-report `noop` would hide that the bay is still checking in), `assetid` is the
|
||||
resolved PC or `null`, `extra.queuecount` is how many rows were stored, and
|
||||
`warnings` carries the soft problems above.
|
||||
|
||||
```
|
||||
POST /api/collector/printers
|
||||
X-Endpoint-Credential: sdbe_...
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"hostname": "WORKSTATION01",
|
||||
"queues": [
|
||||
{
|
||||
"queuename": "Bay Label Printer",
|
||||
"drivername": "Generic / Text Only",
|
||||
"portname": "IP_192.0.2.40",
|
||||
"portaddress": "192.0.2.40",
|
||||
"isdefault": true
|
||||
},
|
||||
{
|
||||
"queuename": "Office Laser",
|
||||
"drivername": "HP Universal Printing PCL 6",
|
||||
"portname": "IP_192.0.2.41",
|
||||
"portaddress": "192.0.2.41",
|
||||
"isdefault": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### The latest report replaces the previous one
|
||||
|
||||
This is CURRENT STATE, not history. Each report deletes every row previously
|
||||
recorded for that hostname and writes the reported set in its place, inside the
|
||||
dispatcher's transaction. Nothing accumulates, "what does this bay have" is
|
||||
never a question about time, and a queue removed from a bay disappears from
|
||||
ShopDB on the next cycle without anyone tidying up.
|
||||
|
||||
That has one sharp edge, and it belongs to the client: an empty `queues` array
|
||||
is a legitimate report that wipes the host's rows. A client whose enumeration
|
||||
FAILED must therefore send nothing at all rather than an empty list. Reporting
|
||||
nothing loses one cycle; reporting `[]` after a WMI hiccup deletes real state
|
||||
and reads as a bay that lost its printers. `Report-PrintersToShopDB.ps1` tracks
|
||||
this with an `$enumerated` flag and exits without posting when both the cmdlet
|
||||
and the WMI fallback failed.
|
||||
|
||||
### Observed is not assigned
|
||||
|
||||
Stated plainly, because it is the whole point of keeping two tables:
|
||||
|
||||
**A collector report NEVER becomes an assignment.** Nothing on this path writes
|
||||
a `usesprinter` or `defaultprinter` row. The moment a drifted bay's observed
|
||||
state is treated as correct, enforcement stops meaning anything - a bay that
|
||||
installed the wrong printer would make itself right simply by reporting it.
|
||||
|
||||
Observed state becomes assigned state only when a person asks for it, through
|
||||
`POST /api/printers/assignments/seed-from-observed/<assetid>` (requires
|
||||
`printers.edit`), after reviewing the comparison. That route refuses rather than
|
||||
guesses: it will not seed from an ambiguous host, will not seed queues that
|
||||
match no printer unless told to, and will not write an empty set.
|
||||
|
||||
### Matching, and what "unknown" means
|
||||
|
||||
Nothing is matched at ingest - the raw observed strings are stored as reported,
|
||||
and resolution to a printer asset happens at READ time, so a printer added to
|
||||
ShopDB tomorrow matches yesterday's report without the bay reporting again.
|
||||
|
||||
At read time each queue is matched by PORT ADDRESS first (an IP or FQDN names
|
||||
one device unambiguously), then by queue name. There is no fuzzier fallback: a
|
||||
queue that matches nothing is reported as `unknown` rather than guessed, because
|
||||
a wrong match seeds a wrong assignment, which is worse than no assignment.
|
||||
|
||||
### Reading it back
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `GET /api/printers/observed/<hostname>` | What the host last reported, every queue classified against the assignment it resolves to (`matching`, `drifted`, `extra`, `missing`, `unknown`), with `observedat`, a summary, and a `seedcandidate` preview. Read-only. Needs `printers.view`. |
|
||||
| `POST /api/printers/assignments/seed-from-observed/<assetid>` | The one path from observed to assigned, human-triggered. Needs `printers.edit`. Normally posted against the MACHINE, so the assignment survives a reimage. |
|
||||
|
||||
Server-side the printers key resolves per-plugin first: set
|
||||
`COLLECTOR_API_KEY_PRINTERS` if you scope env keys per collector, or rely on the
|
||||
shared `COLLECTOR_API_KEY`. A site that has scoped the computers key per-plugin
|
||||
and set no printers key gets 401 here until one of the two is in place. A
|
||||
per-PC credential and a `collector.ingest` managed token both work here exactly
|
||||
as they do for computers.
|
||||
|
||||
## 8. Porting from classic `api.asp`
|
||||
|
||||
The fleet's classic-ASP reporter posts form fields to
|
||||
`api.asp?action=updateCompleteAsset`. Map them to the collector JSON as follows:
|
||||
|
||||
| Classic `updateCompleteAsset` form field | Collector field |
|
||||
|---|---|
|
||||
| `hostname` | `hostname` |
|
||||
| `machineNo` | `machinenumber` |
|
||||
| `deviceId` | `deviceid` |
|
||||
| `pcType` | `pctype` |
|
||||
| `serialNumber` | `serialnumber` |
|
||||
| `loggedInUser` | `loggedinuser` |
|
||||
| `lastBootUpTime` / `lastBootTime` | `lastboottime` |
|
||||
| `manufacturer` | `vendorname` |
|
||||
| `model` | `modelnumber` |
|
||||
| `osVersion` | `osname` |
|
||||
| `installedApps` | `installedsoftware` |
|
||||
|
||||
Not carried over (no current home in the computers schema): warranty fields, DNC
|
||||
config, VNC/WinRM flags as booleans (report them through `accessprotocols`
|
||||
instead). The classic reporter's full `networkInterfaces` array maps onto
|
||||
`interfaces`.
|
||||
352
docs/CONFIG.md
Normal file
@@ -0,0 +1,352 @@
|
||||
<!-- docmeta
|
||||
purpose: reference
|
||||
audience: operator, adopter, developer
|
||||
status: current
|
||||
summary: Every environment variable and Settings key, what it defaults to and which one to reach for.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Configuration reference
|
||||
|
||||
**To find one setting: search this page for its exact key.** Every key appears
|
||||
in exactly one table. Nothing here has to be read in order.
|
||||
|
||||
Two places, and the split is deliberate:
|
||||
|
||||
- **Environment variables** hold **secrets and deploy-time wiring**: database
|
||||
credentials, signing keys, CORS origins, ports, API keys. Read once at boot by
|
||||
`shopdb/config.py`, which is authoritative - where this page and that file
|
||||
disagree, the file wins.
|
||||
- **The Settings table** holds **site preferences**: branding, ServiceNow links,
|
||||
search toggles, facility identity. Seeded by `flask seed settings`, edited in
|
||||
the UI under Settings or the setup wizard, changeable at runtime with no
|
||||
restart. `build_default_settings` in `shopdb/core/api/settings.py` is
|
||||
authoritative for the defaults.
|
||||
|
||||
Rule of thumb: if leaking it would be a security incident, it is an environment
|
||||
variable. Never put a secret in the Settings table.
|
||||
|
||||
**Part 1, environment variables:** [Flask core](#flask-core) |
|
||||
[Database](#database) | [Auth rate limiting](#authentication-rate-limiting) |
|
||||
[Collector ingest](#collector-ingest-adr-006) | [Zabbix](#zabbix-printer-supply-monitoring) |
|
||||
[Employee directory DB](#employee-directory-database-optional-read-only) |
|
||||
[docker-compose only](#docker-compose-only)
|
||||
|
||||
**Part 2, Settings keys:** [site](#site) (facility name, timezone, base URL,
|
||||
retention) | [branding](#branding) | [printing](#printing) (QR and label
|
||||
targets) | [map](#map) (superseded) | [integrations](#integrations)
|
||||
(ServiceNow, Zabbix, Dell warranty) | [email](#email) | [audit](#audit) |
|
||||
[auth](#auth) (SAML placeholders) | [tools](#tools) (Secure Send) |
|
||||
[identifiers](#identifiers-dynamic) | [search](#search-dynamic) |
|
||||
[Custom fields](#custom-fields)
|
||||
|
||||
---
|
||||
|
||||
## Part 1: Environment variables
|
||||
|
||||
Copy `.env.example` to `.env` and fill in values. In `production`
|
||||
(`FLASK_ENV=production`), `ProductionConfig.validate()` refuses to boot if
|
||||
`SECRET_KEY`, `JWT_SECRET_KEY`, `DATABASE_URL`, or `CORS_ORIGINS` are missing or
|
||||
set to the dev defaults.
|
||||
|
||||
### Flask core
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `FLASK_APP` | No | `wsgi.py` | Entry point for the `flask` CLI. |
|
||||
| `FLASK_ENV` | Yes | `development` | `production` for live sites (triggers `validate()`). Other values: `development`, `testing`. |
|
||||
| `SECRET_KEY` | Yes (prod) | dev default | Flask session/signing key. Generate: `python -c "import secrets; print(secrets.token_urlsafe(64))"`. |
|
||||
| `JWT_SECRET_KEY` | Yes (prod) | dev default | JWT signing key. Different value from `SECRET_KEY`. |
|
||||
| `JWT_ACCESS_TOKEN_EXPIRES` | No | `3600` | Access-token TTL in seconds. |
|
||||
| `JWT_REFRESH_TOKEN_EXPIRES` | No | `2592000` | Refresh-token TTL in seconds (30 days). |
|
||||
| `CORS_ORIGINS` | Yes (prod) | `http://localhost:5173` | Comma-separated explicit origins. Wildcard `*` is rejected in production. |
|
||||
| `LOG_LEVEL` | No | `INFO` | Logging verbosity. |
|
||||
|
||||
### Database
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `DATABASE_URL` | Yes (prod) | dev localhost URL | `mysql+pymysql://<user>:<pass>@<host>:<port>/<db>?charset=utf8mb4`. Keep `?charset=utf8mb4`. |
|
||||
|
||||
### Authentication rate limiting
|
||||
|
||||
IP-based fixed-window limit on the login endpoint, defense-in-depth atop the
|
||||
per-account lockout. Uses the cache extension, which is per-process, so the
|
||||
limit is approximate across multiple gunicorn workers.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `AUTH_RATELIMIT_ENABLED` | No | `True` | Set `False` to disable (TestingConfig disables it). |
|
||||
| `AUTH_RATELIMIT_MAX` | No | `30` | Max login attempts per source IP per window before 429. |
|
||||
| `AUTH_RATELIMIT_WINDOW_SECONDS` | No | `300` | Window length in seconds. |
|
||||
|
||||
### Collector ingest (ADR-006)
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `COLLECTOR_API_KEY` | No | (empty) | Shared key for `/api/collector/*`. Endpoint fails closed (denies) when unset and no managed token is presented. Sent as the `X-API-Key` header. |
|
||||
| `COLLECTOR_API_KEY_<PLUGIN>` | No | (empty) | Per-plugin override, e.g. `COLLECTOR_API_KEY_COMPUTERS`. Checked before the shared key. |
|
||||
|
||||
These env keys are the bootstrap/legacy fallback. A managed API token scoped to
|
||||
`collector.ingest` is preferred and is minted, rotated and revoked from
|
||||
Settings > API Tokens; see
|
||||
[COLLECTOR-INTEGRATION.md](COLLECTOR-INTEGRATION.md) (Managed collector tokens).
|
||||
|
||||
### Zabbix (printer supply monitoring)
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `ZABBIX_ENABLED` | No | `false` | Enable the Zabbix integration. |
|
||||
| `ZABBIX_URL` | No | (empty) | Zabbix API URL. |
|
||||
| `ZABBIX_TOKEN` | No | (empty) | Zabbix API bearer token. |
|
||||
|
||||
These are the boot-time wiring; the Settings entries (`zabbix_enabled`,
|
||||
`zabbix_url`, `zabbix_token`) are preferred for runtime changes.
|
||||
|
||||
### Employee directory database (optional, read-only)
|
||||
|
||||
Separate HR lookup DB, read-only. This is where staff names come from: global
|
||||
search resolves the employee domain through it, and the audit log uses it to
|
||||
show a name instead of a bare SSO. Nothing here is required - leave it unset and
|
||||
both surfaces fall back to SSOs. There is no safe default for the password; an
|
||||
unset one fails loud rather than trying a guessed credential.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `EMPLOYEE_DB_HOST` | No | `localhost` | HR DB host. |
|
||||
| `EMPLOYEE_DB_USER` | No | (empty) | HR DB user. |
|
||||
| `EMPLOYEE_DB_PASSWORD` | No | (empty) | HR DB password. No safe default. |
|
||||
| `EMPLOYEE_DB_NAME` | No | `wjf_employees` | HR DB name. |
|
||||
|
||||
### docker-compose only
|
||||
|
||||
Read by `docker-compose.yml`, not by the Flask app directly.
|
||||
|
||||
| Variable | Required | Default | Notes |
|
||||
|----------|----------|---------|-------|
|
||||
| `MYSQL_ROOT_PASSWORD` | Yes | (none) | Root password for the bundled MySQL container. |
|
||||
| `MYSQL_PASSWORD` | Yes | (none) | App-user password; must match the `DATABASE_URL` password. |
|
||||
| `MYSQL_PORT` | No | `3306` | Host port for MySQL. Bound to `127.0.0.1` only. |
|
||||
| `API_PORT` | No | `5001` | Host port for the API container. |
|
||||
|
||||
The opt-in backup overlay adds `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`,
|
||||
`SHOPDB_SITE` and optional cloud credentials - see
|
||||
[BACKUP-RESTORE.md](BACKUP-RESTORE.md).
|
||||
|
||||
---
|
||||
|
||||
## Part 2: Settings table keys
|
||||
|
||||
Seeded by `flask seed settings` (idempotent; re-running adds anything missing).
|
||||
Values are stored as strings and typed by `valuetype`. Secrets in this table
|
||||
(anything whose key contains `password`, `token`, or `secret`) are masked when
|
||||
read back through the API.
|
||||
|
||||
### site
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `setup_complete` | `false` | Set true once the first-run wizard finishes; gates the `/setup` route. |
|
||||
| `employee_directory_mode` | `selfhosted` | `external` reads a separate HR database (see `EMPLOYEE_DB_*`). No plugin on this line keeps a self-hosted directory, so `selfhosted` resolves no names - set `external`, or accept bare SSOs. |
|
||||
| `audit_retentiondays` | `0` | Delete audit entries older than this many days. 0 keeps everything forever. Saving a period deletes nothing by itself: purge from this page or run `flask audit purge --apply`. |
|
||||
| `site_base_url` | (empty) | Public base URL for QR codes and absolute links. Scheme + host, optionally including the subpath a site is served under (`https://host/shopdb`). Blank = use the browsing origin. |
|
||||
| `facility_name` | (empty) | Facility name in the dashboard header. Blank = frontend falls back to `ShopDB`. |
|
||||
| `pc_access_domain` | `device.geaerospace.net` | Domain appended to a PC hostname for remote-access links. Blank = hostname as-is. |
|
||||
| `employeeid_pattern` | `^\d{9}$` | Regex a search term must match to be treated as an employee id. Invalid regex falls back to the default and never 500s. |
|
||||
| `printer_hostname_template` | `Printer-{ip}.printer.geaerospace.net` | Printer hostname template. `{ip}` is the dash-separated IP address. |
|
||||
| `contact_email_domain` | `geaerospace.com` | Email domain appended to a support contact's SSO to build email (`sso@domain`) and Teams-chat links. Blank hides the contact action buttons. |
|
||||
| `dualpath_single_machine` | `true` | Treat a Dualpath pair (a dual-bay machine with one controller) as a single machine in the machines list, dashboard/report counts, and the floor map (the secondary bay is hidden). The data model always keeps both bay records; detail pages stay per-bay with a sibling banner. `false` lists and counts both bays separately. |
|
||||
| `site_timezone` | `America/New_York` | IANA timezone for the site. Notification times are stored UTC and converted for display by `frontend/src/utils/datetime.js`, so they are entered and shown in THIS zone, not the viewer's browser zone; daily-reset notification expiry (`expirymode=dailytime`) is computed here too. Public-readable so kiosks can resolve it. |
|
||||
|
||||
### branding
|
||||
|
||||
Blank values fall back to the shipped GE default asset, so an un-reconfigured
|
||||
install still renders. Upload replacements at Settings > Branding, which saves
|
||||
them under `instance/branding/`.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `site_logo` | `/ge-aerospace-logo.svg` | Header and login-page logo. |
|
||||
| `qr_logo` | `/ge-monogram.svg` | Logo composited in printer QR labels. Blank = no overlay. |
|
||||
| `badge_logo` | `/ge-aerospace-logo.svg` | Logo on the machine badge print page. |
|
||||
| `site_favicon` | (empty) | Browser tab favicon. Blank = shipped `/favicon.svg`. |
|
||||
| `brand_primary_color` | (empty) | Primary brand color as a CSS color value (maps to `--primary`). Blank = built-in theme color. |
|
||||
| `brand_primary_dark_color` | (empty) | Primary hover/active color (maps to `--primary-dark`). Blank = auto-derived by darkening the primary color ~15%. |
|
||||
| `brand_accent_color` | (empty) | Accent color for secondary buttons and badges (maps to `--secondary`). Blank = built-in theme color. |
|
||||
| `brand_sidebar_color` | (empty) | Sidebar background color (maps to `--sidebar-bg`). Blank = built-in theme color. |
|
||||
|
||||
### printing
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `qr_target_printer` | (empty) | Custom URL template for printer QR labels. Blank = link to the printer page on this instance. Placeholders: `{printerid}`, `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{ip}`, `{hostname}`. |
|
||||
| `qr_target_machine` | (empty) | Custom URL template for machine labels. Blank = link to the machine page. Placeholders: `{assetid}`, `{assetnumber}`, `{serialnumber}`, `{name}`, `{pluginid}`. |
|
||||
| `qr_target_computer` | (empty) | Custom URL template for computer labels. Blank = link to the computer page. Same placeholders as machine. |
|
||||
| `qr_target_network_device` | (empty) | Custom URL template for network-device labels. Blank = link to the device page. Same placeholders as machine. |
|
||||
| `label_default_style` | `card` | Default asset-label layout when a label first opens: `card` (badge with image and identity) or `plain` (just the code and a caption). |
|
||||
| `label_default_codetype` | `qr` | Default asset-label code type when a label first opens: `qr` or `barcode` (CODE128). |
|
||||
| `label_default_encodes_machine` | `assetnumber` | What a machine label encodes by default. |
|
||||
| `label_default_encodes_computer` | `assetpage` | What a computer label encodes by default. |
|
||||
| `label_default_encodes_printer` | `assetpage` | What a printer label encodes by default. |
|
||||
| `label_default_encodes_network_device` | `assetpage` | What a network-device label encodes by default. |
|
||||
|
||||
Values for those four `label_default_encodes_*` keys: `assetpage`,
|
||||
`assetnumber`, `serialnumber`, or `custom` (the matching `qr_target_<type>`
|
||||
template). All of it is overridable on the label page itself.
|
||||
|
||||
There is no `measuring_tool` label type on this branch. Metrology instruments
|
||||
are machines (ADR-018), so use the machine label keys, and the measuring-tools
|
||||
plugin is not bundled here - `shopdb/core/api/settings.py` declares four
|
||||
`qr_target_*` and four `label_default_encodes_*` keys to match.
|
||||
|
||||
| Route | What it does |
|
||||
|---|---|
|
||||
| `/print/asset-label/<assettype>/<id>` | One label. Public, like the other `/print/*` pages. `<assettype>` is `machine`, `computer`, `printer` or `network_device`; `<id>` is the asset's plugin id. |
|
||||
| `/print/asset-label-batch/<assettype>` | A multi-selection of one type onto ULINE sheets: 6-up 3 in x 3 in, or a dense 72-up mini-label, with a start-cell offset for reusing partial sheets. Reached from the "Print Labels" button on each asset list page. |
|
||||
|
||||
### map
|
||||
|
||||
**SUPERSEDED as of 0.11.0. Editing these four keys changes nothing on the map.**
|
||||
A blueprint and its pixel dimensions are properties of a LEVEL now (ADR-017).
|
||||
They are retained only so a downgrade finds the blueprint it had; the levels
|
||||
migration copied their values onto the first level. Manage plans in
|
||||
**Settings > Buildings and levels** - see [FLOOR-MAP.md](FLOOR-MAP.md).
|
||||
|
||||
| Key | Default | Superseded by |
|
||||
|-----|---------|-------|
|
||||
| `map_blueprint_light` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintlight` |
|
||||
| `map_blueprint_dark` | `/static/images/floorplan-placeholder.svg` | `maplevels.blueprintdark` |
|
||||
| `map_width` | `3300` | `maplevels.mapwidth` |
|
||||
| `map_height` | `2550` | `maplevels.mapheight` |
|
||||
|
||||
### integrations
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `servicenow_enabled` | `true` | Enable ServiceNow ticket recognition and links. Disabled = tickets render as plain text. |
|
||||
| `servicenow_search_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. |
|
||||
| `servicenow_ticket_prefixes` | `GEINC,GECHG,GERIT,GESCT` | Comma-separated prefixes recognized as ServiceNow tickets. |
|
||||
| `servicenow_incident_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct incident URL if your instance has one. |
|
||||
| `servicenow_change_url` | geaerospaceqa.service-now.com global-search template | `{ticket}` is substituted. Replace with a direct change URL if your instance has one. |
|
||||
| `zabbix_enabled` | `false` | Enable Zabbix for printer supply monitoring. |
|
||||
| `zabbix_url` | (empty) | Zabbix API URL. |
|
||||
| `zabbix_token` | (empty) | Zabbix API token (masked). |
|
||||
| `warranty_dell_enabled` | `false` | Enable Dell warranty (service-tag) lookups. |
|
||||
| `warranty_dell_clientid` | (empty) | Dell TechDirect API client id. |
|
||||
| `warranty_dell_clientsecret` | (empty) | Dell TechDirect API client secret (masked). |
|
||||
| `warranty_dell_tokenurl` | (empty) | Dell OAuth token URL. Blank = Dell default. |
|
||||
| `warranty_dell_apiurl` | (empty) | Dell warranty API URL. Blank = Dell default. |
|
||||
|
||||
### email
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `smtp_enabled` | `false` | Enable email notifications and alerts. |
|
||||
| `smtp_host` | (empty) | SMTP server hostname. |
|
||||
| `smtp_port` | `587` | SMTP port (587 TLS, 465 SSL, 25 plain). |
|
||||
| `smtp_username` | (empty) | SMTP auth username. |
|
||||
| `smtp_password` | (empty) | SMTP auth password (masked). |
|
||||
| `smtp_use_tls` | `true` | Use TLS for the SMTP connection. |
|
||||
| `smtp_from_address` | (empty) | From address for outgoing email. |
|
||||
| `smtp_from_name` | `ShopDB` | From name for outgoing email. |
|
||||
| `alert_recipients` | (empty) | Default alert/report recipients (comma-separated). |
|
||||
|
||||
The mailer (`shopdb/utils/mailer.py`) reads these settings-first, falling back to
|
||||
matching `SMTP_*` environment variables only when at least one of them is set.
|
||||
When `smtp_enabled` is false or `smtp_host` is blank, every send is a graceful
|
||||
no-op that logs a warning, so an unconfigured site never crashes. The password
|
||||
is never logged.
|
||||
|
||||
Three flows send mail, and **there is no scheduler in this app** - sending is
|
||||
always on-demand:
|
||||
|
||||
- **Welcome email**, on `POST /api/users`. Best-effort: the account is created
|
||||
even if the send fails (the response carries a `warning`). The new account is
|
||||
flagged `mustchangepassword`, so the frontend forces `/change-password` before
|
||||
the app; changing it clears the flag and resets lockout counters. Opt out with
|
||||
`sendwelcome: false` in the create body.
|
||||
- **Test email**, `POST /api/settings/test-email` (`settings.edit`), behind the
|
||||
"Send Test Email" button. A real SMTP error is surfaced with the password
|
||||
scrubbed out.
|
||||
- **Alerts and report delivery**, `POST /api/reports/email` (`reports.export`),
|
||||
taking `{subject, columns, rows, intro?, to?}` and mailing the rows as an HTML
|
||||
table. Recipients default to `alert_recipients`. To automate a recurring send,
|
||||
point an external cron job at this endpoint with a PAT scoped to
|
||||
`reports.export`.
|
||||
|
||||
### audit
|
||||
|
||||
Retention is `audit_retentiondays`, in the [site](#site) section. There is only
|
||||
one such key; anything writing `audit_retention_days` is writing to nothing.
|
||||
|
||||
### auth
|
||||
|
||||
**SAML is not implemented yet.** These keys exist so a site can be configured
|
||||
ahead of the feature, and the settings page says so on screen. Nothing reads
|
||||
them: there is no SAML login route and no assertion is accepted, so sign-in is
|
||||
local accounts only whatever is set here. In particular `saml_allow_local_login`
|
||||
does NOT disable password login - the toggle is locked on until there is another
|
||||
way in, because a control that appears to close the local door while it stays
|
||||
open is worse than no control.
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `saml_enabled` | `false` | Reserved. Enable SAML SSO once it ships. |
|
||||
| `saml_idp_metadata_url` | (empty) | Reserved. SAML IdP metadata URL. |
|
||||
| `saml_entity_id` | (empty) | Reserved. SAML SP entity id. |
|
||||
| `saml_acs_url` | (empty) | Reserved. SAML Assertion Consumer Service URL. |
|
||||
| `saml_allow_local_login` | `true` | Reserved, and locked on. Local login cannot be turned off while it is the only way in. |
|
||||
| `saml_auto_create_users` | `true` | Reserved. Auto-create users on first SAML login. |
|
||||
| `saml_admin_group` | (empty) | Reserved. SAML group name that grants the admin role. |
|
||||
|
||||
**Personal API tokens (PATs) have no configuration here**, and
|
||||
`JWT_ACCESS_TOKEN_EXPIRES` does not apply to them. They are created and revoked
|
||||
from Settings > API Tokens; the token model is documented once, in
|
||||
[IMPORT-API.md](IMPORT-API.md#1-authentication).
|
||||
|
||||
### tools
|
||||
|
||||
| Key | Default | Notes |
|
||||
|-----|---------|-------|
|
||||
| `securesend_allow_guests` | `false` | Let people with no account create Secure Send links. Public (readable before login) so the create page can show a guest the form instead of a sign-in wall. Off by default: a guest secret has no creator, so the audit trail can say when and from where but never who. Receiving a secret never needs an account either way. |
|
||||
|
||||
[SECURE-SEND.md](SECURE-SEND.md) is authoritative for the rest of it - guest
|
||||
rate limiting, why the read fails closed, and why the size limit is not a tuning
|
||||
knob.
|
||||
|
||||
### identifiers (dynamic)
|
||||
|
||||
One boolean key per asset identifier per asset type, keyed
|
||||
`identifier_<name>_<assettype>_enabled` (default `true`). Admins choose which
|
||||
optional identifiers show on which asset types. See ADR-001. The exact set is
|
||||
generated from `IDENTIFIER_LABELS` x `IDENTIFIER_ASSETTYPES` in
|
||||
`shopdb/core/api/settings.py`.
|
||||
|
||||
### search (dynamic)
|
||||
|
||||
One boolean key per search domain, keyed `search_<type>_enabled` (default
|
||||
`true`). Toggles whether a domain appears in global search results. The set is
|
||||
generated from `SEARCH_DOMAINS` in `shopdb/core/api/settings.py`.
|
||||
|
||||
Search terms are matched word-wise: a multi-word query returns rows containing
|
||||
EVERY word, each word anywhere in the searched fields, in any order ("CSF Roles"
|
||||
matches a row with "CSF" and "Roles" in different columns). Quoting does not
|
||||
force a contiguous phrase.
|
||||
|
||||
## Custom fields
|
||||
|
||||
Site-defined extra attributes per asset type (Settings > Custom Fields, table
|
||||
`customfields`). Each field has a `searchable` flag (default off). When on, the
|
||||
field's stored values are matched by global search and a hit routes to the
|
||||
owning asset's detail page. The asset's `search_<type>_enabled` domain toggle
|
||||
still applies, so a custom-field hit on a computer only shows when the computer
|
||||
search domain is enabled. Inactive or non-searchable fields are never matched.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [DEPLOY.md](DEPLOY.md) - per-site deployment runbook
|
||||
- [UPGRADE.md](UPGRADE.md) - upgrading an existing site
|
||||
- [BACKUP-RESTORE.md](BACKUP-RESTORE.md) - backup and restore
|
||||
- `GET /api/docs` - what endpoints exist and their auth, generated from the code
|
||||
149
docs/CONTRACT-STABILITY.md
Normal file
@@ -0,0 +1,149 @@
|
||||
<!-- docmeta
|
||||
purpose: explanation
|
||||
audience: adopter, integrator
|
||||
status: current
|
||||
summary: How stable each part of the plugin contract is today, and how tightly a sister site should pin.
|
||||
loadwith: none
|
||||
-->
|
||||
|
||||
# Path to contract 1.0
|
||||
|
||||
This page is an honest read of how stable the plugin contract is today, for a
|
||||
sister site deciding how much to build on it. It is derived from the ADRs and
|
||||
the live code, not aspiration. It does NOT define the contract - what each hook
|
||||
takes and returns is in [PLUGIN-HOOKS.md](PLUGIN-HOOKS.md), and the versioning
|
||||
rules are in [ADR-002](adr/ADR-002-plugin-versioning.md). What it adds is a
|
||||
judgement about how much weight each part will bear.
|
||||
|
||||
## Current version
|
||||
|
||||
The plugin contract version is declared in `shopdb/__init__.py` as
|
||||
`__contract_version__`, and the current value is in
|
||||
[PROJECT-MAP.md](PROJECT-MAP.md), which is generated. It is pre-1.0, which under
|
||||
semver means any 0.x minor bump is allowed to break the contract, and this
|
||||
project has used that latitude - see 0.19.0 below.
|
||||
|
||||
The product release version (`__version__`) is a separate series with its own
|
||||
bump rules; see [ADR-007](adr/ADR-007-product-versioning-and-releases.md).
|
||||
Do not pin against it for compatibility - pin against `__contract_version__`.
|
||||
|
||||
### 0.x history
|
||||
|
||||
Recorded in the comment block in `shopdb/__init__.py`:
|
||||
|
||||
| Version | Change | Kind |
|
||||
|---------|--------|------|
|
||||
| 0.3.0 | `shopdb.api` expanded to the full plugin import surface (db, cache, model bases, core models, response + pagination helpers, `employee_connection`) so plugins stop importing internal core paths | additive (minor) |
|
||||
| 0.4.0 | Removed the never-implemented `get_searchable_fields` hook (search is a core concern over the asset model) and wired `get_dashboard_widgets` to a real consumer (`/api/dashboard/widgets`) | pre-1.0 contract reduction |
|
||||
| 0.6.0 | Added the `get_reports` hook, consumed by `GET /api/reports` to merge plugin report cards into the Reports hub | additive optional hook (minor) |
|
||||
| 0.7.0 | Added the four ADR-010 frontend-contribution hooks (`get_settings_cards`, `get_asset_panels`, `get_map_overlays`, `get_asset_presentation`), consumed by the `GET /api/pluginui/*` endpoints | additive optional hooks (minor) |
|
||||
| 0.9.0 | Exposed the dualpath pair-resolution helpers on `shopdb.api` for the machines plugin | additive surface (minor) |
|
||||
| 0.10.0 | Added the `get_permissions` hook so plugins declare their own RBAC permissions; the catalog is resolved dynamically from core + enabled plugins | additive optional hook (minor) |
|
||||
| 0.11.0 | Added `service_token_authorized(scope)` to `shopdb.api` so a plugin's unattended endpoints can authorize a scoped managed service token without importing core token internals | additive surface (minor) |
|
||||
| 0.12.0 | Added the mailer helpers (`send_email`, `send_alert`) to `shopdb.api` | additive surface (minor) |
|
||||
| 0.13.0 | Added the `User` model to the `shopdb.api` surface | additive surface (minor) |
|
||||
| 0.14.0 | Added `send_webhook` to the `shopdb.api` surface | additive surface (minor) |
|
||||
| 0.15.0 | Added `authorized_service_token` / the `SupportTeam` model to the `shopdb.api` surface | additive surface (minor) |
|
||||
| 0.16.0 | Added the `get_settings_defaults` hook so a plugin declares the Setting rows it owns; the framework seeds them at install, at enable, and on `flask plugin upgrade-all`, files a first-time write under the declared category, and honours `public: True` for pages that render before login | additive optional hook (minor) |
|
||||
| 0.17.0 | Added `DashboardDefault` to the `shopdb.api` surface, so a plugin can resolve a display without reaching into core | additive surface (minor) |
|
||||
| 0.18.0 | Added `DISPLAY_ROLES`, `DISPLAY_ROLE_PATHS` and `normalize_display_role`, and finally exported `DashboardDefault`, which 0.17.0 imported but left out of `__all__`. The role vocabulary became the kiosk's own - `Dashboard`, `Lobby`, `3DPrintRoom` - so a plugin holding its own copy of that map reads core's instead of drifting from it | additive surface (minor) |
|
||||
| 0.19.0 | **BREAKING.** `get_dashboard_widgets` returns DATA AND SHAPE, not a component name. The old shape (`name` + `component` + `size`) named a Vue component per widget, which cannot survive a lean build - a plugin's component may never be staged into the frontend bundle (ADR-013) - and five plugins were declaring widgets that pointed at components nobody had written. A card now declares `id` / `title` / `endpoint` / `render` / `severity` / `permission` / `empty` / `position`; see PLUGIN-HOOKS.md. Also added `send_upload` so a plugin serving user-supplied bytes gets the headers that keep an SVG from running as script | **contract change (minor, pre-1.0)** |
|
||||
| 0.20.0 | The `Asset` model gained `levelid` (which floor plan its `mapx`/`mapy` are pixels of) and `mapverifiedat`; `Location` gained `levelid` too, and `resolve_asset_position` now returns the `levelid` belonging to whichever source supplied the coordinates. A plugin that writes a map position MUST write the level with it - a position without one renders as "level unknown" rather than being drawn on the default drawing, because a marker on the wrong floor plan looks correct (ADR-017) | additive surface (minor) |
|
||||
| 0.21.0 | Added the `get_guest_grantable_permissions` hook: a plugin names the WRITE permissions a site may grant to the seeded `guest` role, which is who anyone without a token now holds. Reads (`.view`/`.read`) are grantable without a plugin saying anything; anything else is refused. Also exposed `require_any_permission` on `shopdb.api` for a read two different rights should reach | additive optional hook + surface (minor) |
|
||||
| 0.22.0 | Added the `register_import_hook` surface: a plugin owns a VIRTUAL COLUMN on an importable CSV table - a header the bulk importer accepts but never writes - plus a post-write row hook that turns the value into a plugin concept. The computers plugin uses it for `machineassetnumber` on `computers.csv`, which builds the PC->machine controls link, so a site writes the machine number in the PC's own row instead of hand-building a separate relationships sheet. Core never learns what the column means. `RowWarning` is exported with it so a hook can report a soft failure without aborting the import | additive surface (minor) |
|
||||
| 0.23.0 | Added `mint_service_token`, `list_service_tokens` and `revoke_service_tokens` so a plugin can provision, list and revoke a single-scope managed service token without importing core token internals. The endpoints plugin uses them for a UI-managed, regenerable agent key scoped to `collector.ingest.endpoints` alone - narrow on purpose, so rotating or revoking it cannot disturb the computers collector, and revocation is an instant kill switch on agent ingest | additive surface (minor) |
|
||||
| 0.24.0 | Added `collector_authorized(pluginname)` so a plugin can gate an agent-facing route that lives OUTSIDE the generic collector blueprint on the same per-plugin key and kill switch the ingest route uses. The endpoints command channel needs it: the agent long-polls `/api/endpoints/commands/next` and posts results back, and those are agent callbacks rather than JWT routes. Without it a plugin would reach into core collector internals to authorize them | additive surface (minor) |
|
||||
| 0.25.0 | Added `caller_holds(permission)`, which answers whether the current caller holds a permission WITHOUT refusing the request - the complement to `require_permission` for a route that serves everyone but should only offer what the caller can act on. The endpoints services panel is the case: it emitted Start/Stop row actions to every reader, because the emission never consulted the caller. Clicking them was always refused server-side, so nothing was exploitable, but offering a read-only viewer controls over a shop-floor PC is the wrong signal, and it becomes a real problem once a site grants that panel to guests. Guests are included in the answer, since a signed-out visitor holds whatever the guest role was granted | additive surface (minor) |
|
||||
| 0.26.0 | Dashboard cards may declare `shortname`. The dashboard merged its per-source cards into one worst-first queue, and a card's `title` is written as a heading ("Endpoints with crashes or shutdowns") which as a row badge ran wider than the hostname it qualified. Optional, defaults to the title, so a plugin that does not declare one is unaffected | additive field (minor) |
|
||||
| 0.27.0 | Added `Script` so a plugin can read the ShopDB-hosted script library that backs endpoint `run_script` delivery without importing core model internals. The endpoints plugin resolves a delivered script by name and serves its body by digest to the agent | additive surface (minor) |
|
||||
| 0.29.0 | Added an optional `group` to `get_asset_panels`, naming which detail-page tab a panel belongs to: `details` (the default) / `monitors` / `software` / `activity`. The detail pages are tabbed, and a page that placed panels by knowing their ids would defeat the hook. Additive: an absent or unrecognised group resolves to `details` server-side, so every existing plugin keeps working with no change | additive surface (minor) |
|
||||
| 0.28.0 | Added `client_ip` and `strip_port` so a plugin does not hand-roll caller-IP extraction. Two plugins had, and both read the first `X-Forwarded-For` hop, which the caller supplies - so a rate limit keyed on it was defeated by rotating the header. The helper resolves from `remote_addr`, which ProxyFix has already filled from the header when `TRUSTED_PROXY_COUNT` says a proxy is really in front | additive surface (minor) |
|
||||
| 0.31.0 | The asset-panel vocabulary a plugin may declare grew four optional things: two more tab groups, `serviceprocess` and `reports`, beside the 0.29.0 set; a `format: 'datetime'` on a column or field, so a panel stops shipping pre-formatted date strings the browser cannot re-localise; a `map.body` key, so a keyvalue field can name where its value lives in a nested payload; and a `tabs` section that carries `columns`/`rows`/`empty` and renders as a table, so one card can hold two tables that share no key (volume capacity and physical drive health are the case). All optional and all resolved server-side when absent, so a plugin declaring none of them is unaffected | additive surface (minor) |
|
||||
|
||||
The source comment block documents 0.3.0, 0.4.0, 0.6.0, 0.7.0, 0.9.0, 0.10.0,
|
||||
0.11.0, 0.16.0, 0.18.0 and 0.19.0; 0.12.0 through 0.15.0 and 0.17.0 are recorded
|
||||
in this table only.
|
||||
|
||||
**0.19.0 is the one to read before pinning.** It is the only entry in this
|
||||
series that took something away, and it shipped before it was recorded here -
|
||||
which is precisely the reason to pin tight rather than to trust that a minor
|
||||
bump is safe. Earlier points
|
||||
(0.1.x / 0.2.x) predate that recorded rationale; `PluginMeta`'s fallback
|
||||
`core_version` default of `>=0.2.0,<1.0.0` is the only remaining trace of the
|
||||
0.2 baseline.
|
||||
|
||||
## Settled surface
|
||||
|
||||
These are unlikely to break before 1.0. A plugin can depend on them with
|
||||
reasonable confidence; a breaking change to any would be a major bump and would
|
||||
land with a new or amended ADR. What each one does is in
|
||||
[PLUGIN-HOOKS.md](PLUGIN-HOOKS.md); this is only the stability judgement.
|
||||
|
||||
- The required hooks and the manifest schema: `meta` / `PluginMeta`,
|
||||
`get_blueprint`, `get_models`.
|
||||
- Plugin plumbing: `init_app`, `get_cli_commands`, `get_services`, the settings
|
||||
helpers (`get_setting` / `set_setting`), and the four lifecycle hooks.
|
||||
- The UI-contribution hooks: `get_navigation_items`, `get_dashboard_widgets`,
|
||||
`get_reports`, `get_permissions`, and the ADR-010 set (`get_settings_cards`,
|
||||
`get_asset_panels`, `get_map_overlays`, `get_asset_presentation`).
|
||||
- The collector pair, `get_collector_schema` + `apply_collector_payload`, per
|
||||
[ADR-006](adr/ADR-006-collector-contract.md).
|
||||
- The setup-wizard declarations: `get_provisioning_note`, `get_config_schema`.
|
||||
- The `shopdb.api` import surface itself: adding a name is minor, removing one
|
||||
is major.
|
||||
|
||||
The asset model itself - `Asset`, `AssetType`, `AssetStatus`,
|
||||
`AssetRelationship`, and the shared reference models - is the platform contract
|
||||
locked in [ADR-001](adr/ADR-001-asset-as-platform-contract.md).
|
||||
|
||||
## Expected churn before 1.0
|
||||
|
||||
Known-unstable areas. Building on these means expecting rework.
|
||||
|
||||
| Area | Status | Reference |
|
||||
|------|--------|-----------|
|
||||
| Frontend renderers (residual) | The four data-only hooks and their `/api/pluginui/*` consumers are settled (0.7.0). The generic core renderers are landing incrementally: the settings-cards rail/landing renderer ships with 0.7.0; the asset-panel, map-overlay, and search-presentation renderers are wired opt-in per the ADR adoption plan. Real component-backed panels (bespoke charts, custom overlays) remain deferred to Option C (build-time glob discovery). | [ADR-010](adr/ADR-010-frontend-plugin-hooks.md) (ACCEPTED) |
|
||||
| Per-plugin migrations | Brand new. The per-plugin Alembic engine exists and every bundled plugin now carries a chain, but the pattern has one release of production mileage, not years. | [ADR-008](adr/ADR-008-plugin-migration-ownership.md) (2026-07-10) |
|
||||
| Pip distribution | Deferred to v2. External plugins install by clone / submodule / symlink; there is no entry-point discovery and no automatic update path yet. | [ADR-003](adr/ADR-003-plugin-distribution.md) |
|
||||
|
||||
## Bump rules
|
||||
|
||||
From [ADR-002](adr/ADR-002-plugin-versioning.md), applied to `__contract_version__`:
|
||||
|
||||
| Bump | Trigger |
|
||||
|------|---------|
|
||||
| major | Breaking change to the `BasePlugin` ABC, the `PluginMeta` schema, or any model in the platform contract (`Asset`, `AssetType`, `AssetStatus`, `AssetRelationship`, `Vendor`, `Location`, `BusinessUnit`, `Model`, `OperatingSystem`). Removing a name from `shopdb.api` is major. |
|
||||
| minor | Additive change: a new optional hook, a new field on a contract model with a default, a new name added to `shopdb.api`. |
|
||||
| patch | Bug fix with no change to the contract surface. |
|
||||
|
||||
### Deprecation policy
|
||||
|
||||
ADR-002 defines the bump classification above but is **silent** on any
|
||||
deprecation window or notice period for pre-1.0 removals. In practice a removal
|
||||
is simply a major (or, pre-1.0, a breaking minor) bump: the hook or name is
|
||||
gone, and the loader fails loud in dev or excludes the mismatched plugin in prod
|
||||
(see the load-time table in [PLUGIN-EXTERNAL-REPO.md](PLUGIN-EXTERNAL-REPO.md)).
|
||||
Because the ADR says nothing about a grace period, this document proposes none;
|
||||
the honest guidance for sister sites is the mitigation that already exists - pin
|
||||
a tight `core_version` range and re-test before widening it.
|
||||
|
||||
## Criteria for declaring 1.0
|
||||
|
||||
Provisional. This is the maintainer's working list, not a committed checklist,
|
||||
and it will move. ADR-002's own open question ("When does the framework declare
|
||||
1.0.0?") ties 1.0 to the `Machine` retirement from ADR-001 and the framework
|
||||
being "ready for sister sites"; the items below expand that intent.
|
||||
|
||||
- Frontend hook contract defined via its own ADR (asset-detail panels, map
|
||||
markers, search results), closing the biggest churn item above.
|
||||
- `Machine` / legacy-model retirement complete per ADR-001, so the asset model
|
||||
is the only contract.
|
||||
- At least two external plugins running in production at a second site - the
|
||||
bar ADR-003 already sets before pip distribution and sister-site readiness
|
||||
are considered justified.
|
||||
- The per-plugin migration pattern (ADR-008) proven across real upgrades, not
|
||||
just fresh installs.
|
||||
- No contract bump needed for N consecutive product releases (N to be fixed
|
||||
when the list is firmed up), showing the surface has actually settled.
|
||||
|
||||
Until then: pre-1.0, pin tight, re-test on every framework bump.
|
||||
139
docs/CSV-IMPORT.md
Normal file
@@ -0,0 +1,139 @@
|
||||
<!-- docmeta
|
||||
purpose: howto
|
||||
audience: adopter, operator
|
||||
status: current
|
||||
summary: Load a new site's starting data from generated CSV templates, with no developer and no source database.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Loading a site's data from spreadsheets
|
||||
|
||||
**Four import pages.** Spreadsheet and no developer: this page. The HTTP wire
|
||||
contract: [IMPORT-API](IMPORT-API.md). How to shape a per-site loader around
|
||||
it: [IMPORT-ADOPTION](IMPORT-ADOPTION.md). Source is the classic-ASP shopdb:
|
||||
[IMPORT-LEGACY-ASP](IMPORT-LEGACY-ASP.md) on top of the contract.
|
||||
|
||||
---
|
||||
|
||||
## The short version
|
||||
|
||||
```bash
|
||||
cd C:\shopdb-flask # or your install directory
|
||||
venv\Scripts\flask csv templates --out csv-templates
|
||||
```
|
||||
|
||||
Fill in the templates. Then:
|
||||
|
||||
```bash
|
||||
venv\Scripts\flask csv import --dir csv-templates
|
||||
```
|
||||
|
||||
That **checks only** and changes nothing. It tells you what it would create and
|
||||
what is wrong. When you are happy:
|
||||
|
||||
```bash
|
||||
venv\Scripts\flask csv import --dir csv-templates --commit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Write names, not numbers
|
||||
|
||||
This is the part that makes the difference. Every column that points at another
|
||||
table accepts the **name** of the thing:
|
||||
|
||||
```
|
||||
assetnumber,name,assettypeid,statusid,locationid
|
||||
CMM-01,Zeiss Contura,Machine,Active,Gage Lab
|
||||
MILL-07,Haas VF-2,Machine,Active,Bay 3
|
||||
```
|
||||
|
||||
`assettypeid` gets `Machine`. `locationid` gets `Gage Lab`. You never
|
||||
have to import a file, read back the numbers it generated, and paste them into
|
||||
the next one.
|
||||
|
||||
The column keeps its database name so it matches the rest of the system, but
|
||||
the value is whatever you actually know. Numeric ids still work if you have
|
||||
them - useful when re-importing something this system exported.
|
||||
|
||||
Names resolve **across files in the same run**, so `assets.csv` can reference a
|
||||
location that only exists because `locations.csv` was loaded moments earlier.
|
||||
|
||||
## Nothing is half-imported
|
||||
|
||||
Every row is checked before anything is written. If one row is wrong, nothing is
|
||||
written at all - you fix the file and run it again. A mistake on line 400 never
|
||||
leaves 399 rows loaded.
|
||||
|
||||
## Running it twice is safe
|
||||
|
||||
Each file is matched on a natural key - `assetnumber` for assets, `locationname`
|
||||
for locations, and so on. Re-importing an edited file **updates** those rows
|
||||
rather than creating second copies. Correcting a spreadsheet and re-running is
|
||||
the expected workflow, not a mistake.
|
||||
|
||||
## What the errors look like
|
||||
|
||||
```
|
||||
assets: 0 new, 0 updated, 1 problem(s)
|
||||
line 4, column 'locationid': nothing in locations is named 'Bay 9'
|
||||
- add it to locations.csv, or import that file first
|
||||
```
|
||||
|
||||
Line, column, value, and what to do. Not a foreign key constraint violation.
|
||||
|
||||
---
|
||||
|
||||
## What you can import
|
||||
|
||||
Fourteen tables. `flask csv templates` writes one file per table and `--dir`
|
||||
imports them in dependency order, which is the order the generated folder lists
|
||||
them in: assetstatuses, assettypes, locationtypes, modeltypes, computertypes,
|
||||
machinetypes, businessunits, locations, vendors, models, operatingsystems,
|
||||
assets, computers, machines. The authoritative list, with each table's natural
|
||||
key, is `IMPORTABLE` in `shopdb/core/services/csvimport.py`.
|
||||
|
||||
You only need the files you have; skip any you do not care about. Use `--file`
|
||||
with `--table` for a single file out of order.
|
||||
|
||||
**User accounts are deliberately not importable.** Passwords do not belong in a
|
||||
spreadsheet, in either direction. Create the first administrator through the
|
||||
first-run page and the rest in the application.
|
||||
|
||||
## The templates are generated, not maintained
|
||||
|
||||
`flask csv templates` builds them from the live database schema each time, so
|
||||
every column offered exists, every required one is marked, and every foreign key
|
||||
says which file it refers to. Do not hand-maintain a template set: the one that
|
||||
was tried had invented columns on seven of eleven tables and named a table that
|
||||
does not exist, while looking entirely plausible. A test fails the build if the
|
||||
generated set ever drifts from the schema.
|
||||
|
||||
## Editing the files
|
||||
|
||||
- **UTF-8**, no BOM. Excel: "CSV UTF-8 (Comma delimited)".
|
||||
- Lines starting with `#` are ignored, so the notes at the top of each template
|
||||
can stay where they are.
|
||||
- **The example rows are not comments. Delete them.** Each template carries a
|
||||
worked example set as LIVE rows, so that a freshly downloaded folder imports
|
||||
cleanly as a demo. Import it with your own data still in it and you get
|
||||
`PC-001`, `CNC-014`, `Dell` and `Bay 3` in your database. The template says so
|
||||
in its own header.
|
||||
- Booleans are `1` or `0`.
|
||||
- Dates are `YYYY-MM-DD` (`2026-08-04`). `YYYY-MM-DD HH:MM:SS` also works, as do
|
||||
`DD/MM/YYYY` and `MM/DD/YYYY`.
|
||||
- Leave a cell **empty** for "no value". Not `NULL`, not `N/A`.
|
||||
- Quote anything containing a comma: `"Bay 3, North"`.
|
||||
|
||||
## If it will not run
|
||||
|
||||
**"the 'assets' table does not exist in this database"** - the schema has not
|
||||
been created. Run `flask db upgrade` first, and check `DATABASE_URL` points at
|
||||
the site you meant.
|
||||
|
||||
**"unknown column(s): ..."** - a column that does not exist, usually from an
|
||||
older template. Regenerate with `flask csv templates`; the message lists what
|
||||
the table does accept.
|
||||
|
||||
**"required column(s) missing: ..."** - a column that must be present has been
|
||||
deleted from the header. Regenerate and copy your data across.
|
||||
108
docs/DEPENDENCY-SECURITY.md
Normal file
@@ -0,0 +1,108 @@
|
||||
<!-- docmeta
|
||||
purpose: howto
|
||||
audience: developer
|
||||
status: current
|
||||
summary: Answer "are we exposed to this advisory" from the shipped SBOM, and what can and cannot fail a build.
|
||||
loadwith:
|
||||
-->
|
||||
|
||||
# Dependency security
|
||||
|
||||
Two scripts, one question each.
|
||||
|
||||
| question | script | output |
|
||||
|---|---|---|
|
||||
| what are we running | `scripts/generate_sbom.py` | `sbom.cdx.json`, CycloneDX 1.6, shipped with every release |
|
||||
| are we exposed | `scripts/scan_sbom.py` | a report, and an exit code |
|
||||
|
||||
The SBOM was already generated and installed onto every server. Nothing read
|
||||
it, so "are we exposed to this advisory" was still a person going and looking.
|
||||
The scanner closes that.
|
||||
|
||||
## Running it
|
||||
|
||||
```bash
|
||||
venv/bin/python scripts/scan_sbom.py # report, exit 0
|
||||
venv/bin/python scripts/scan_sbom.py --gate # exit 1 on a shipped finding
|
||||
venv/bin/python scripts/scan_sbom.py --json out.json # machine-readable too
|
||||
```
|
||||
|
||||
With no `--sbom`, it generates one first, so there is never a stale input.
|
||||
|
||||
It needs outbound HTTPS to `api.osv.dev`. **Run it where the repository lives,
|
||||
not on a customer's server** - a deployed instance is frequently air-gapped and
|
||||
has nothing to query. That is also why the SBOM ships as a file: it is the
|
||||
artefact a security team scans on their own terms.
|
||||
|
||||
## Why OSV
|
||||
|
||||
It takes a purl and returns the advisories that apply. No account, no API key,
|
||||
no agent installed anywhere, no vulnerability database to keep current. The
|
||||
SBOM is already the input it wants, so this is a lookup against what shipped
|
||||
rather than a second scan of the tree that could disagree with it.
|
||||
|
||||
## Shipped and build-only are not the same finding
|
||||
|
||||
Two thirds of the components in the SBOM are build tooling - bundlers, test
|
||||
runners, their transitive dependencies. They never reach a server, and the SBOM
|
||||
already marks them `scope: excluded`.
|
||||
|
||||
Both are reported. **Only shipped components can fail a build.**
|
||||
|
||||
This is deliberate. At the time of writing, the highest-severity match in the
|
||||
whole tree is a CRITICAL against the test runner, which is not installed
|
||||
anywhere a release reaches. A gate that fails on that teaches everybody to pass
|
||||
`--no-verify` within a month, and then it catches nothing at all.
|
||||
|
||||
## Severity is taken, not computed
|
||||
|
||||
Where an advisory records a plain word - GHSA gives LOW, MODERATE, HIGH,
|
||||
CRITICAL - that word is used. Where there is only a CVSS vector, the report says
|
||||
UNKNOWN rather than parsing the vector into a score it might get wrong.
|
||||
|
||||
UNKNOWN is shown and never gates. A guessed score is worse than an honest gap,
|
||||
because somebody would act on it.
|
||||
|
||||
## The allowlist is a record, not a mute button
|
||||
|
||||
`scripts/vulnerability-allowlist.json` holds advisories that were reviewed and
|
||||
accepted. An entry carries the advisory id, the reason, who accepted it, the
|
||||
date, and what would invalidate the reasoning.
|
||||
|
||||
A good reason names the precondition that is absent. The file has held exactly
|
||||
one entry so far, and how it ended is the point.
|
||||
|
||||
A PyJWT advisory rated HIGH - algorithm confusion, forged HS256 tokens - needs a
|
||||
verifier configured with both symmetric and asymmetric algorithms AND a raw JSON
|
||||
JWK passed as the key. This application configures neither: there is no
|
||||
`JWT_ALGORITHM` or `JWT_DECODE_ALGORITHMS` anywhere in the tree, so
|
||||
flask-jwt-extended runs HS256-only against a string secret. That was written
|
||||
down, with `invalidatedby` recording what would make it untrue - SSO issuing
|
||||
tokens verified against a JWK.
|
||||
|
||||
Then PyJWT was upgraded, the advisory stopped matching, and the entry was
|
||||
deleted. The allowlist is empty again, which is the state to aim for.
|
||||
|
||||
`invalidatedby` is the field that earns its keep. A reason stops being true when
|
||||
the code that made it true changes, and nobody re-reads an allowlist looking for
|
||||
that. Write down what would break it.
|
||||
|
||||
Upgrading is almost always cheaper than defending a decision to skip it. Do not
|
||||
add an entry to make a build pass.
|
||||
|
||||
## What the report tells you to do
|
||||
|
||||
The report ends with a per-component upgrade rollup: the smallest upgrade that
|
||||
clears every advisory against that component, and how many it clears.
|
||||
|
||||
"Smallest" is doing real work there. An advisory patched on several release
|
||||
branches lists a fix per branch - nanoid's is fixed in both 3.3.18 and 5.1.16 -
|
||||
and taking the highest recommends a major upgrade when a patch on the current
|
||||
branch would do. Where the package is transitive that major may not even be
|
||||
reachable: nanoid arrives under postcss, pinned `^3.3.11`, a range 5.x cannot
|
||||
satisfy. So the target is the lowest published fix above the version in use.
|
||||
Fixes on older branches are ignored rather than offered as a downgrade.
|
||||
|
||||
A target on a different major version is still labelled `MAJOR`, but now that
|
||||
label means something: no fix exists on the current branch, so the upgrade is
|
||||
genuinely a different job - it can be ESM-only, or drop an API.
|
||||