Build the plugin set a real site runs, and write down what shipped
The lean-build job proved the delete-a-plugin guarantee with two plugins and four hand-picked marker components. It now builds a real twelve-plugin set, and the assertions follow a rule instead of a hunch: one component from EVERY omitted plugin must be absent, one from EVERY chosen plugin must be present, and the backend tree is checked both ways. An omitted set that grows without its check growing is how a leak ships. CHANGELOG had not been touched across the whole range, so the largest security-behaviour change in the product - per-PC endpoint credentials, the shared-key refusal for enrolled PCs, server pinning, agent 1.20.1 and tray 1.7.1 - had no release record. An [Unreleased] section now carries it, written from the commits rather than from memory. CLAUDE.md and the agent quick-reference stated that ADR-016 bans storing credentials in the registry. It does not, and this fleet depends on HKLM for its collector key and server pins: the next person following that text removes them and silently breaks reporting and pinning on every un-enrolled PC. Corrected, along with two other statements that were false on this branch - the database is MySQL, not PostgreSQL, and column names are lowercase concatenated, not snake_case, which is the opposite of what the gate enforces. The wiki drafts are updated against the code as it stands, each old finding given a verdict with a file and line rather than left as an assertion from a week ago, and a new page records the ADR conformance result - including that the ADR set itself is not on this branch, so nothing here enforces it except the naming and style script. They are also scrubbed of the things that must not reach a published wiki: the site name, the internal git server, developer home paths and the assistant tooling. They now pass the repository's own publishability patterns, which is what the scrub is for.
This commit is contained in:
cproudlock
@@ -1,8 +1,37 @@
|
||||
# Maintenance Footprint Baseline - 2026-09-11
|
||||
|
||||
**Total annual burden:** 936 hours/year
|
||||
**Risk level:** 8 critical + 10 high operational risks
|
||||
**Total annual burden:** 936 hours/year (estimate, see the note below)
|
||||
**Risk level:** originally stated as 8 critical + 10 high operational risks
|
||||
**Optimization potential:** 147-205 hours/year savings with automation
|
||||
**Last checked against code:** 2026-09-18, branch gh-wj at 35d4fcd41
|
||||
|
||||
## About the numbers (added 2026-09-18)
|
||||
|
||||
Every hour figure on this page is an estimate with no recorded measurement
|
||||
behind it. No time log, ticket export or deploy log is cited anywhere, and none
|
||||
was found in the repository. Treat the hours as an ordering of where effort
|
||||
goes, not as data. The same applies to the row counts and growth rates in the
|
||||
Database Management section: no query output is cited and none could be taken
|
||||
from a repository checkout.
|
||||
|
||||
What CAN be checked from the repository is which of the recommendations have
|
||||
since been built. Several have. The sections below are annotated
|
||||
accordingly, dated 2026-09-18, and the stale risk-matrix rows are corrected.
|
||||
|
||||
## What the tree actually contains (2026-09-18)
|
||||
|
||||
| Thing | Count | How counted |
|
||||
|-------|-------|-------------|
|
||||
| Plugins | 18 directories, 17 with a `manifest.json` | `ls -d plugins/*/`; `applications` carries no manifest |
|
||||
| Plugin migration revisions | 70 across 17 chains | `ls plugins/*/migrations/versions/*.py` |
|
||||
| Largest single chain | `endpoints`, 24 revisions | one root, one head, no branch |
|
||||
| Core migration revisions | 58 | `ls migrations/versions/*.py` |
|
||||
| Test files | 223 | `find tests -name "test_*.py"` |
|
||||
| Top-level scripts | 29 (.sh/.py/.ps1/.mjs), plus `scripts/backup`, `scripts/dev`, `scripts/migration`, `scripts/loadtest`, `scripts/site_imports` | `ls scripts/` |
|
||||
| Automation skills | 13 | `ls <assistant-config>/skills/*.md` minus README |
|
||||
|
||||
The "13 automation skills" claim later on this page is the one headline figure
|
||||
that checks out exactly.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
@@ -29,6 +58,19 @@ Baseline assessment of ShopDB operational burden across air-gapped production en
|
||||
- Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated)
|
||||
- Status: PLANNED for Q4 2026
|
||||
|
||||
**2026-09-18 update: half built, and the half that is built is manual.**
|
||||
Retention exists as a service and a CLI, not as a cron job and not as a stored
|
||||
procedure. `shopdb/core/services/auditretention.py` provides `configureddays()`,
|
||||
`stats()` and `purge()`; `flask audit stats` (`shopdb/cli/__init__.py:1891`)
|
||||
reports size against the configured window and `flask audit purge`
|
||||
(`:1911`) deletes, with a dry-run preview first and `--days` to override for one
|
||||
run. The window is the `audit_retentiondays` setting
|
||||
(`shopdb/core/api/settings.py:1049`), and its default is **`'0'`, which means
|
||||
keep everything forever**: nothing is deleted until an operator sets a value.
|
||||
There is no scheduler in the application and no cron entry in the repository, so
|
||||
the purge runs when somebody runs it. The remaining work is a value in Settings
|
||||
> Site & Facility and a host cron line, not code.
|
||||
|
||||
**Endpoint events cleanup:** 26 hrs/year (HIGH RISK)
|
||||
- Current: 450K rows, 127MB, no retention
|
||||
- Growth: Process events every 30 seconds, app launches, USB connects
|
||||
@@ -38,10 +80,32 @@ Baseline assessment of ShopDB operational burden across air-gapped production en
|
||||
- Implementation: Similar cron job as audit log
|
||||
- Savings: 26 hrs/year (fully automated)
|
||||
|
||||
**2026-09-18 update: partly built, per table rather than globally.** Operator
|
||||
reports carry a retention window defaulting to 7 days
|
||||
(`plugins/endpoints/api/routes.py:3619-3620`,
|
||||
`endpoints.operatorreport.retentiondays`, 0 or less meaning keep forever), and
|
||||
endpoint health transitions are trimmed to a high-water mark on write, keeping
|
||||
every unresolved row regardless of age
|
||||
(`plugins/endpoints/services/health.py:326`). The row counts and the 127MB
|
||||
figure above were never sourced and are not verifiable from a checkout.
|
||||
|
||||
**Database backups:** 78 hrs/year (MEDIUM RISK)
|
||||
- Current: Manual mysqldump weekly, stored on same server
|
||||
- Current (as stated 2026-09-11): Manual mysqldump weekly, stored on same server
|
||||
- Risk: No offsite copies, no restore testing, backup growth unbounded
|
||||
- Backup size: 2.1GB compressed, growing 15% per quarter
|
||||
- Backup size: 2.1GB compressed, growing 15% per quarter (unsourced)
|
||||
|
||||
**2026-09-18 update: STALE. This is built, and it is what the recommendation
|
||||
below asks for.** `scripts/backup/` holds `backup.sh`, `restore.sh`, `status.sh`,
|
||||
`watch.sh`, `simulate-restore.sh` and `test-restore-datarev.sh`, plus a
|
||||
`Dockerfile.backup` and a compose file. Retention is enforced by restic:
|
||||
`scripts/backup/backup.sh:134-141` runs `restic forget --keep-daily 5
|
||||
--keep-weekly 4 --keep-monthly 6 --prune`. Destinations are configured in
|
||||
Settings > Backups and include local, SMB and AWS S3, so offsite is a setting
|
||||
rather than a manual copy. Each backup set carries `db.sql.gz`, the `instance/`
|
||||
tree and a `manifest.json` of schema revisions, and restore validates schema
|
||||
compatibility before it runs. A restore-simulation harness is in the tree.
|
||||
Backup credential redaction landed in the same commit as audit retention
|
||||
(`scripts/redact_backup_secrets.py`).
|
||||
|
||||
**Recommended:** Automated backup with rotation
|
||||
- Daily incremental, weekly full, monthly archive
|
||||
@@ -69,7 +133,7 @@ Baseline assessment of ShopDB operational burden across air-gapped production en
|
||||
|
||||
**Air-gap coordination:** 312 hrs/year (CRITICAL BURDEN)
|
||||
- Current: Manual bundle creation, USB transfer, deploy in 3 environments
|
||||
- Environments: dev PC (gitea) <-> work PC (GitHub) <-> VM (prod)
|
||||
- Environments: dev PC (the internal git server) <-> work PC (GitHub) <-> VM (prod)
|
||||
- Steps per deploy: 14 manual steps, 45-90 minutes each
|
||||
- Frequency: 2-3 deploys/week = 104-156 deploys/year
|
||||
|
||||
@@ -111,6 +175,13 @@ logging:
|
||||
```
|
||||
- Savings: 52 hrs/year (fully automated, from manual cleanup)
|
||||
|
||||
**2026-09-18 update: STILL OPEN, and still the cheapest item on this page.**
|
||||
`grep -c logging docker-compose.yml` returns 0: there is no `logging:` block on
|
||||
any service. Three services do carry `restart: unless-stopped`
|
||||
(`docker-compose.yml:24`, `:104`, `:120`) and three carry `healthcheck:` blocks
|
||||
(`:38`, `:112`, `:219`), so the container lifecycle is managed; log size is not.
|
||||
The 500MB/week growth figure was never sourced.
|
||||
|
||||
**Image rebuilds:** 52 hrs/year
|
||||
- Current: Manual rebuild when dependencies change
|
||||
- Frequency: ~20 rebuilds/year (Python deps, base image updates)
|
||||
@@ -197,9 +268,57 @@ logging:
|
||||
| Docker log disk fill | HIGH | High (3-6 mo) | Container crashes | Log rotation Q4 2026 |
|
||||
| Deployment coordination errors | HIGH | Medium (2-3/year) | Failed deploys, downtime | Automation scripts implemented |
|
||||
| Migration conflicts (ADR-008) | HIGH | Low | Plugin blocked | Migration-check skill |
|
||||
| Agent certificate validation disabled | CRITICAL | Low | MITM attack | Per-endpoint cert pinning Q4 2026 |
|
||||
| No login rate limiting | HIGH | Medium | Brute force attack | Rate limiting Q1 2027 |
|
||||
| Unauthenticated collector endpoints | CRITICAL | N/A | FIXED 2026-09-11 | Device token auth |
|
||||
| Agent certificate validation disabled | CORRECTED 2026-09-18 | n/a | n/a | Built. See note below |
|
||||
| No login rate limiting | CORRECTED 2026-09-18 | n/a | n/a | Built. See note below |
|
||||
| Unauthenticated collector endpoints | CORRECTED 2026-09-18 | n/a | n/a | Per-PC endpoint credentials, not device tokens. See note below |
|
||||
| Docker log disk fill | HIGH | unmeasured | Container crashes | STILL OPEN, no `logging:` block in docker-compose.yml |
|
||||
|
||||
### Risk matrix corrections (2026-09-18)
|
||||
|
||||
**Agent certificate validation.** Not disabled. `AgentHttpFactory` is the only
|
||||
`HttpClient` construction site in the agent, every channel runs
|
||||
`ServerPinSet.Validate`, and `Validate` has no path returning true for an
|
||||
unmatched pin (`agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs:291-331`). A
|
||||
failed registry re-read returns null and keeps the existing pins rather than
|
||||
silently unpinning (`:352-386`). The three SYSTEM PowerShell reporters refuse to
|
||||
send at all when `ServerPins` is set and the check cannot be enabled
|
||||
(`plugins/computers/client/Report-AssetToShopDB.ps1:228-247`, and the same block
|
||||
in `Report-PrintersToShopDB.ps1` and `upgrade-shopdbagent.ps1:436-447`). The one
|
||||
`ServerCertificateValidationCallback = {$true}` left in the tree is
|
||||
`test-vm-connectivity.ps1:44`, a developer connectivity probe.
|
||||
|
||||
**Login rate limiting.** Built, and not "Q1 2027":
|
||||
`shopdb/core/api/auth.py:35` `_login_ratelimited()`, keyed per host at `:52`,
|
||||
applied at `:80`, with `tests/test_core/test_auth_ratelimit.py` over it. Note
|
||||
the counter lives in per-process `SimpleCache`, so at four gunicorn workers the
|
||||
effective budget is four times the configured maximum.
|
||||
|
||||
**Collector authentication.** The mechanism is a per-PC endpoint credential,
|
||||
not a device token and not ADR-016. An enrolled PC sends
|
||||
`X-Endpoint-Credential` (`plugins/endpoints/api/agentauth.py:1-18`); the server
|
||||
stores only a sha256 hash plus a 12-character display prefix
|
||||
(`plugins/endpoints/models/credential.py`); once a PC holds an active
|
||||
credential the shared key is refused for it
|
||||
(`plugins/endpoints/api/agentauth.py:129-134`). ADR-016's own mechanism -
|
||||
`credentials.fetch`, `GET /api/collector/credentials`, Fernet at rest - is
|
||||
**not built**: grepping `shopdb/` and `plugins/` for `credentials.fetch`,
|
||||
`collector/credentials` and `Fernet` returns zero hits, matching ADR-016's own
|
||||
"Implementation status, 2026-08-14". The rollout runbook is
|
||||
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.
|
||||
|
||||
One boundary the runbook does not state: turning the shared key off does not
|
||||
close the four legacy core collector routes. `/pc`
|
||||
(`shopdb/core/api/collector.py:349`), `/apps` (`:387`), `/heartbeat` (`:459`)
|
||||
and `/bulk` (`:494`) are decorated `@require_api_key`, which at `:154-164`
|
||||
checks only `COLLECTOR_API_KEY` or a `collector.ingest`-scoped token and never
|
||||
consults `agent_shared_key_refused`. Anything still holding the env shared key
|
||||
can rewrite logged-in user, last boot time, serial number and installed-app
|
||||
inventory for any hostname. Regenerating the key in the UI
|
||||
(`plugins/endpoints/api/routes.py:1864-1884`) mints a new managed token and
|
||||
clears the env fallback flag but does not change `COLLECTOR_API_KEY`, and
|
||||
`_env_fallback_disabled(None)` returns False (`collector.py:60-74`), so the
|
||||
legacy routes keep honouring the env key. Rotating `COLLECTOR_API_KEY` in
|
||||
`.env` is the step that closes it.
|
||||
|
||||
## Automation Savings Summary
|
||||
|
||||
@@ -313,11 +432,23 @@ rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/
|
||||
|
||||
### 5. check-naming-and-style.sh Optimization (DONE)
|
||||
|
||||
**Effort:** 2 hours
|
||||
**Savings:** Script runtime 5x faster (10s -> 2s)
|
||||
**Status:** Committed 2026-09-11 (commit 4943e182c3)
|
||||
**Status:** In the tree and on by default. `USE_GIT_LS_FILES` defaults to `1`
|
||||
(`scripts/check-naming-and-style.sh:24`) and the script takes the
|
||||
`git ls-files -z` path whenever it is inside a work tree (`:26`, `:159`, `:183`,
|
||||
`:218`), filtering out `migrations/versions/`, `frontend/dist/`, `staticdocs/`
|
||||
and `.plugins-staged/`. Setting `USE_GIT_LS_FILES=0` falls back to recursive
|
||||
grep with `--exclude-dir` for non-git checkouts. The whole script exits 0 on
|
||||
gh-wj at 35d4fcd41.
|
||||
|
||||
**Impact:** 10-15 minutes saved per commit = ~26 hrs/year for team
|
||||
**The 5x / 10s-to-2s claim is unsourced** and no timing was recorded for this
|
||||
page. The savings line below it follows from it and is equally unsourced.
|
||||
|
||||
**Coverage note added 2026-09-18:** the root-anchored `^migrations/versions/`
|
||||
filter does not exclude `plugins/*/migrations/versions/`, so plugin migrations
|
||||
ARE scanned. They are not caught by the column rule, though: that rule greps
|
||||
`^\s+[a-z]+_[a-z_]+\s*=\s*db\.Column` (`:236`), which only matches a model
|
||||
attribute assignment, while migrations write `sa.Column('name', ...)`. The
|
||||
column name is created in the one file the naming gate cannot read.
|
||||
|
||||
### 6. Pre-commit Hook Integration
|
||||
|
||||
@@ -349,17 +480,36 @@ pytest tests/ -x || exit 1
|
||||
|
||||
## Long-Term Improvements (Planned)
|
||||
|
||||
### 1. Agent Delta Updates (Q1 2027)
|
||||
### 1. Agent Delta Updates - BUILT (2026-09-18 update)
|
||||
|
||||
**Current:** Agent POSTs full inventory (11KB) every 30 seconds
|
||||
**Proposed:** POST only deltas (added/removed apps, IP changes) = ~500 bytes
|
||||
**Original text:** "Agent POSTs full inventory (11KB) every 30 seconds.
|
||||
Proposed: POST only deltas." Both halves are now wrong.
|
||||
|
||||
**Savings:**
|
||||
- 99% reduction in network traffic (17GB/day -> 170MB/day for 500 endpoints)
|
||||
- 99% reduction in database writes (only changed data)
|
||||
- Performance: Collector API load reduced from 1.44M requests/day to ~14K writes/day
|
||||
**The cadence was never 30 seconds.** The agent default is 900 seconds with up
|
||||
to 60 seconds of jitter: `agent/ShopDbEndpointAgent/appsettings.json`
|
||||
(`"cadenceseconds": 900`, `"jitterseconds": 60`) and
|
||||
`agent/ShopDbEndpointAgent.Core/AgentConfig.cs:14` and `:31`. The 17GB/day and
|
||||
1.44M-requests/day figures were derived from the wrong interval and should not
|
||||
be quoted.
|
||||
|
||||
**Effort:** 40 hours (agent changes + server changes + testing)
|
||||
**The delta is implemented, on the server side.** The agent still reports the
|
||||
full installed-software list each cycle, but each entry carries a SHA-256
|
||||
fingerprint of `displayname|displayversion|publisher`, and
|
||||
`_replace_applications` (`plugins/endpoints/plugin.py:1122-1140`) compares
|
||||
fingerprint sets instead of delete-all-insert-all: adds for new fingerprints,
|
||||
deletes for departed ones, unchanged rows untouched (`:1277-1289`). The
|
||||
docstring states the effect as roughly 100 database operations per check-in
|
||||
down to about 2. It is guarded: fingerprint format validation, a delete-count
|
||||
check that warns rather than aborting, a memory cap above 5000 apps, a legacy
|
||||
fall-back for agents that send no fingerprints, and collision reconciliation
|
||||
inside a savepoint (`:1318`, `:1387`). The unique `(assetid, appfingerprint)`
|
||||
index was added NULL-filled by migration
|
||||
`plugins/endpoints/migrations/versions/0020_*`, so index creation cannot fail on
|
||||
a populated production table.
|
||||
|
||||
Remaining network-side work, if anyone wants it, is making the agent send only
|
||||
the changed entries rather than the full list. The database win is already
|
||||
taken.
|
||||
|
||||
### 2. Automated Staging Tests (Q4 2026)
|
||||
|
||||
@@ -369,14 +519,17 @@ pytest tests/ -x || exit 1
|
||||
**Savings:** 50-100 hrs/year (from deployment coordination)
|
||||
**Effort:** 20 hours (test suite + CI integration)
|
||||
|
||||
### 3. Per-Endpoint Certificate Pinning (Q4 2026)
|
||||
### 3. Per-Endpoint Certificate Pinning - BUILT (2026-09-18 update)
|
||||
|
||||
**Current:** Certificate validation disabled (security risk)
|
||||
**Proposed:** Each endpoint validates server cert against pinned thumbprint
|
||||
|
||||
**Savings:** Eliminates CRITICAL security risk
|
||||
**Effort:** 40 hours (agent changes + deployment script + mass agent update)
|
||||
**Status:** Planned (see Security-Audit-2026-09-11.md)
|
||||
Pins are SPKI SHA-256, validated as 64 lowercase hex on the way in, enforced on
|
||||
every agent channel, and owned by the registry rather than by `appsettings.json`
|
||||
(`plugins/endpoints/client/Set-ShopDBServerPins.ps1` writes
|
||||
`HKLM:\SOFTWARE\GE\ShopDB` `ServerPins`; the precedence is stated in
|
||||
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`, section "The registry
|
||||
wins"). A CA-flagged certificate is pinned but deliberately NOT imported into
|
||||
`LocalMachine\Root`, and the build warns the admin through an
|
||||
`X-ShopDB-Build-Warning` header. See the risk-matrix correction above for the
|
||||
enforcement call sites.
|
||||
|
||||
## Monitoring & Alerting
|
||||
|
||||
@@ -399,8 +552,11 @@ pytest tests/ -x || exit 1
|
||||
## References
|
||||
|
||||
- Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned
|
||||
- Architectural Findings 2026-09-11: 22 findings (6 critical, 6 high) from Gortex analysis
|
||||
- Automation Skills: 13 skills created in `.claude/skills/` directory
|
||||
- Deployment Documentation: `docs/deployment/` and `.claude/setup/` guides
|
||||
- Architectural Findings 2026-09-11: re-checked 2026-09-18; most of the 22
|
||||
findings are refuted or stale, see that page for the verdict on each
|
||||
- ADR-Conformance-2026-09-18: which ADRs were checked in the 2026-09-18 review
|
||||
and which hold
|
||||
- Automation Skills: 13 skills created in the assistant skills directory
|
||||
- Deployment Documentation: `docs/deployment/` and the assistant setup guides
|
||||
- ADR-004: Deployment Topology (air-gapped architecture)
|
||||
- ADR-008: Plugin Migration Ownership (linear chain requirement)
|
||||
|
||||
Reference in New Issue
Block a user