Files
shopdb-flask/docs/wiki-drafts/Maintenance-Footprint-Baseline.md
cproudlock f9d6b57f0f Move the documentation to the line that ships, and correct it against the code
The docs lived on one lineage and the fleet code on another. That is why six
pages described a shared fleet key as how an agent authenticates, why
COLLECTOR-AUTH had to warn that the per-PC credential code was not in the
repository it sat in, and why ADR-019 recorded a decision whose implementation
was somewhere else. Documentation that cannot see the code it documents goes
stale by construction, and no amount of editing fixes that.

So the corpus moves here: 84 files, 56 pages, 20 ADRs, the proposals, INDEX.md,
llms.txt and both gates. Filenames are unchanged, because a wiki page name is
derived from the filename and a rename breaks every link anybody saved.

THE POINT OF THE MOVE IS THE CORRECTION IT ENABLES. Every caveat that said "not
in this tree" was false the moment the files landed beside the code, so each was
rewritten from what is actually here:

- COLLECTOR-AUTH gained the real enroll request body, the two bounds on the
  unauthenticated route (per-IP rate limit, pending cap with stale rows dropped
  before a real PC is refused), and the rejected pending reason that binds to
  the endpoint identity rather than the nonce.
- ADR-019 now describes an implementation that is present, including its tests,
  and says plainly that what remains is the FLEET migration, not the code.
- proposals/device-identity claimed phase 1 was unbuilt. plugins/computers
  declares deviceid on the collector schema and prefers it over the older
  fallbacks. Corrected in the page, its summary and the index.

WINDOWS/IIS IS NOW DOCUMENTED AS THE LEGACY PATH, per the product decision, and
carefully: it is what production runs today. Every Windows page opens by saying
it is supported, not for a new site, and where a new site should go instead. No
procedure was cut - an operator on an IIS box at 2am needs these pages to work
exactly as well tomorrow. MIGRATE-TOPOLOGY stops being a footnote and becomes
the off-ramp, which is why it nearly doubled: what was there did not match how
the Docker stack really works here.

The Docker material that already lived on this line was kept and given
ownership rather than buried under the ported Windows-centric prose: DEPLOY now
describes the four services, the migrate service that runs the upgrade chain
before api starts, TLS terminated by nginx, and the plugin count corrected from
thirteen to seventeen - a number both lineages had wrong.

ROADMAP is rewritten from evidence rather than memory: what has shipped, read
out of the changelog, the ADRs and the manifests, and ahead of it exactly two
items - finalize a distributed release and polish the setup wizard.

PROJECT-MAP is generated and was stale; the GENERATOR was what was wrong.
--check passes now.

API-REFERENCE is retired to a pointer. /api/docs serves 573 operations with
derived auth, query parameters and harvested examples, and a hand-written page
competing with that is a liability. The file stays so its wiki page and its
inbound links survive.

AND THE MOVE TURNED ON A SCRUB THAT HAD NEVER RUN HERE. test_docs_publishable
keys on docs/*.md, which this line did not have, so it skipped - its own comment
predicted what would surface. Porting the corpus activated it and it found 48
lines that must not reach a published wiki: branch names abbreviating the site,
assistant-tooling paths, the site name in two deployment pages, and an employee
ID in a Windows path that the scrub does not even look for.
2026-09-19 09:10:15 -04:00

26 KiB

Maintenance Footprint Baseline - 2026-09-11

Read with the date on it. This is a point-in-time audit, kept as the record of what was found and what it turned out to be, not as current guidance. One thing it says repeatedly is no longer true: it was written when this line carried no ADRs, and the full ADR set now lives beside it in docs/adr. Where this page says a decision record is missing or lives elsewhere, look there first.

Total annual burden: 936 hours/year (estimate, see the note below) Risk level: originally stated as 8 critical + 10 high operational risks Optimization potential: 147-205 hours/year savings with automation Last checked against code: 2026-09-18, the deployment branch at 35d4fcd41

About the numbers (added 2026-09-18)

Every hour figure on this page is an estimate with no recorded measurement behind it. No time log, ticket export or deploy log is cited anywhere, and none was found in the repository. Treat the hours as an ordering of where effort goes, not as data. The same applies to the row counts and growth rates in the Database Management section: no query output is cited and none could be taken from a repository checkout.

What CAN be checked from the repository is which of the recommendations have since been built. Several have. The sections below are annotated accordingly, dated 2026-09-18, and the stale risk-matrix rows are corrected.

What the tree actually contains (2026-09-18)

Thing Count How counted
Plugins 18 directories, 17 with a manifest.json ls -d plugins/*/; applications carries no manifest
Plugin migration revisions 70 across 17 chains ls plugins/*/migrations/versions/*.py
Largest single chain endpoints, 24 revisions one root, one head, no branch
Core migration revisions 58 ls migrations/versions/*.py
Test files 223 find tests -name "test_*.py"
Top-level scripts 29 (.sh/.py/.ps1/.mjs), plus scripts/backup, scripts/dev, scripts/migration, scripts/loadtest, scripts/site_imports ls scripts/
Automation skills 13 ls <assistant-config>/skills/*.md minus README

The "13 automation skills" claim later on this page is the one headline figure that checks out exactly.

Executive Summary

Baseline assessment of ShopDB operational burden across air-gapped production environment (dev PC + work PC + VM + GitHub). Current maintenance requires 936 hrs/year (23.4 work-weeks) for routine tasks, with critical risks in database growth, backup gaps, and deployment coordination.

Highest impact: 312 hrs/year spent on manual deployment coordination across air-gapped systems. Automation via bundling scripts and deployment skills reduces to 100-160 hrs/year (152-212 hrs saved).

Highest risk: Database audit log with no retention policy (2.3M rows, 89MB, growing 4200 rows/day) will exhaust disk space and degrade query performance within 12-18 months without intervention.

Maintenance Categories

Database Management (234 hrs/year)

Audit log maintenance: 52 hrs/year (CRITICAL RISK)

  • Current: No retention policy, manual purging when disk fills
  • Growth rate: 4200 rows/day = 1.5M rows/year
  • Current size: 2.3M rows, 89MB table + 156MB indexes
  • Projected: Will exceed 10M rows (400MB) by Q2 2027
  • Risk: Disk exhaustion, query timeouts, backup failures

Recommended: Automated retention policy

  • Retention: 90 days for INFO/DEBUG actions, 2 years for admin/modification
  • Implementation: Cron job runs DELETE FROM auditlog WHERE ... AND created < NOW() - INTERVAL '90 days'
  • Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated)
  • Status: PLANNED for Q4 2026

2026-09-18 update: half built, and the half that is built is manual. Retention exists as a service and a CLI, not as a cron job and not as a stored procedure. shopdb/core/services/auditretention.py provides configureddays(), stats() and purge(); flask audit stats (shopdb/cli/__init__.py:1891) reports size against the configured window and flask audit purge (:1911) deletes, with a dry-run preview first and --days to override for one run. The window is the audit_retentiondays setting (shopdb/core/api/settings.py:1049), and its default is '0', which means keep everything forever: nothing is deleted until an operator sets a value. There is no scheduler in the application and no cron entry in the repository, so the purge runs when somebody runs it. The remaining work is a value in Settings

Site & Facility and a host cron line, not code.

Endpoint events cleanup: 26 hrs/year (HIGH RISK)

  • Current: 450K rows, 127MB, no retention
  • Growth: Process events every 30 seconds, app launches, USB connects
  • Risk: High-churn data fills disk faster than audit log

Recommended: 30-day retention for events, 90 days for snapshots

  • Implementation: Similar cron job as audit log
  • Savings: 26 hrs/year (fully automated)

2026-09-18 update: partly built, per table rather than globally. Operator reports carry a retention window defaulting to 7 days (plugins/endpoints/api/routes.py:3619-3620, endpoints.operatorreport.retentiondays, 0 or less meaning keep forever), and endpoint health transitions are trimmed to a high-water mark on write, keeping every unresolved row regardless of age (plugins/endpoints/services/health.py:326). The row counts and the 127MB figure above were never sourced and are not verifiable from a checkout.

Database backups: 78 hrs/year (MEDIUM RISK)

  • Current (as stated 2026-09-11): Manual mysqldump weekly, stored on same server
  • Risk: No offsite copies, no restore testing, backup growth unbounded
  • Backup size: 2.1GB compressed, growing 15% per quarter (unsourced)

2026-09-18 update: STALE. This is built, and it is what the recommendation below asks for. scripts/backup/ holds backup.sh, restore.sh, status.sh, watch.sh, simulate-restore.sh and test-restore-datarev.sh, plus a Dockerfile.backup and a compose file. Retention is enforced by restic: scripts/backup/backup.sh:134-141 runs restic forget --keep-daily 5 --keep-weekly 4 --keep-monthly 6 --prune. Destinations are configured in Settings > Backups and include local, SMB and AWS S3, so offsite is a setting rather than a manual copy. Each backup set carries db.sql.gz, the instance/ tree and a manifest.json of schema revisions, and restore validates schema compatibility before it runs. A restore-simulation harness is in the tree. Backup credential redaction landed in the same commit as audit retention (scripts/redact_backup_secrets.py).

Recommended: Automated backup with rotation

  • Daily incremental, weekly full, monthly archive
  • Offsite copy to work PC or USB drive
  • Automated restore testing in test VM
  • 30-day retention for daily, 90-day for weekly, 1-year for monthly
  • Savings: 56 hrs/year (from 78 to 22 hrs with automation + monitoring)

Query optimization: 52 hrs/year

  • Current: Ad-hoc analysis when users report slow pages
  • Common issues: Missing indexes on plugin tables, N+1 queries in dashboard
  • Tools: EXPLAIN ANALYZE, slow query log (not enabled)

Recommended: Enable slow query log, monthly index analysis

  • Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

Schema migrations: 26 hrs/year

  • Current: Plugin migrations run automatically on deploy
  • Effort: Testing migrations in dev, verifying no conflicts (ADR-008)
  • Risk: Migration chain breaks, plugins blocked from enabling

Status: Acceptable. ADR-008 (plugin migration ownership) reduces conflicts.

Deployment Operations (312 hrs/year)

Air-gap coordination: 312 hrs/year (CRITICAL BURDEN)

  • Current: Manual bundle creation, USB transfer, deploy in 3 environments
  • Environments: dev PC (the internal git server) <-> work PC (GitHub) <-> VM (prod)
  • Steps per deploy: 14 manual steps, 45-90 minutes each
  • Frequency: 2-3 deploys/week = 104-156 deploys/year

Time breakdown per deploy:

  • Bundle creation: 15 minutes (build frontend, tar source, verify integrity)
  • Transfer dev->work: 10 minutes (USB copy + verification)
  • GitHub push: 5 minutes (pull the upstream branch, merge down, resolve conflicts, push)
  • Transfer work->VM: 10 minutes (USB or shared folder)
  • VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test)
  • Rollback (if needed): +60 minutes (restore snapshot, rollback migrations)

Automation implemented (Q3 2026):

  • Bundling scripts: pull-shopdb-bundle.cmd, fetch-wilm-branch.cmd
  • Deployment skills: /full-deploy, /agent-deploy-full, /docker-rebuild
  • VM snapshot skill: /vm-snapshot for pre-deploy snapshots
  • Savings: 50-100 hrs/year (from 312 to 160-260 hrs with scripting)

Remaining manual steps:

  • Physical USB transfer (air gap requirement)
  • Smoke testing after deploy (safety check)
  • Conflict resolution on merges (judgment required)

Further optimization: Automated staging tests, pre-flight validation, automated rollback triggers could save another 50-100 hrs/year.

Docker & Container Management (156 hrs/year)

Docker log rotation: 52 hrs/year (HIGH RISK)

  • Current: No log rotation configured
  • Growth: API container logs grow 500MB/week
  • Risk: Disk fills, Docker daemon hangs, services crash

Recommended: Configure log rotation in docker-compose.yml

logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
  • Savings: 52 hrs/year (fully automated, from manual cleanup)

2026-09-18 update: STILL OPEN, and still the cheapest item on this page. grep -c logging docker-compose.yml returns 0: there is no logging: block on any service. Three services do carry restart: unless-stopped (docker-compose.yml:24, :104, :120) and three carry healthcheck: blocks (:38, :112, :219), so the container lifecycle is managed; log size is not. The 500MB/week growth figure was never sourced.

Image rebuilds: 52 hrs/year

  • Current: Manual rebuild when dependencies change
  • Frequency: ~20 rebuilds/year (Python deps, base image updates)
  • Time: 2-3 hours per rebuild (build, test, deploy)

Automation: /docker-rebuild skill handles full rebuild workflow

  • Savings: 26 hrs/year (from 52 to 26 with scripted rebuild + testing)

Container health monitoring: 52 hrs/year

  • Current: Manual checks when users report issues
  • No automated alerting, no health endpoints for most services

Recommended: /production-health skill runs automated checks

  • Monitors: Container status, API responsiveness, DB connectivity, disk space
  • Run: Daily via cron, alerts on failures
  • Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

Testing & Validation (130 hrs/year)

Pre-commit validation: 52 hrs/year

  • Current: Manual pytest and check-naming-and-style.sh before commits
  • Failures: ~15% of commits have style violations or test failures caught late

Automation: /pre-commit skill runs full validation suite

  • Tests: pytest, style check, security scan, migration check
  • Time: 2-3 minutes (optimized script 5x faster with git ls-files)
  • Savings: 26 hrs/year (from 52 to 26 with automated pre-commit hook)

Migration validation: 26 hrs/year

  • Current: Manual checks for migration conflicts (ADR-008 violations)
  • Tools: pytest tests/test_plugin_migrations.py, manual DB inspection

Automation: /migration-check skill validates migration chains

  • Checks: Linear chains, no branches, correct down_revision, PLUGIN_TABLE_OWNERS order
  • Savings: 13 hrs/year (from 26 to 13 with scripted validation)

Agent deployment testing: 52 hrs/year

  • Current: Manual build + deploy to test VM, manual verification
  • Frequency: ~25 agent updates/year
  • Time: 2 hours per update (build, sign, deploy, test service, test tray)

Automation: /agent-deploy-full skill handles end-to-end workflow

  • Steps: Build agent, deploy to VM via SSH, verify service, test heartbeat
  • Savings: 26 hrs/year (from 52 to 26 with scripted deployment)

Security & Compliance (78 hrs/year)

Security scanning: 26 hrs/year

  • Current: Manual Darkwing scans on PRs, ad-hoc credential checks
  • Coverage: Detects secrets in commits, but not all vulnerability classes

Automation: /security-scan skill runs comprehensive checks

  • Scans: Hardcoded credentials, exposed keys, missing auth, SQL injection patterns
  • Based on: Sept 2026 security audit findings (see Security-Audit-2026-09-11.md)
  • Savings: 13 hrs/year (from 26 to 13 with automated scanning)

Dependency updates: 52 hrs/year

  • Current: Manual review of Dependabot alerts, testing updates
  • Frequency: ~40 alerts/year (Python + npm)
  • Risk: Delayed updates accumulate, breaking changes harder to resolve

Recommended: Monthly dependency review cycle

  • Review alerts, batch compatible updates, test in dev
  • Savings: None (still requires judgment), but reduces risk

Agent & Endpoint Management (26 hrs/year)

Agent updates: Covered under deployment (52 hrs/year)

Endpoint troubleshooting: 26 hrs/year

  • Current: Manual investigation when endpoints stop reporting
  • Common issues: Service stopped, network blocked, token expired, disk full

Monitoring: /production-health skill checks endpoint health

  • Alerts: Endpoints offline >48 hours, stale inventory (>7 days)
  • Savings: 13 hrs/year (from 26 to 13 with proactive alerts)

Risk Matrix

Risk Severity Likelihood Impact Mitigation
Database disk exhaustion (audit log) CRITICAL High (12-18 mo) Service outage Retention policy Q4 2026
Backup failure (no testing) CRITICAL Medium Data loss on restore Automated restore tests
Docker log disk fill HIGH High (3-6 mo) Container crashes Log rotation Q4 2026
Deployment coordination errors HIGH Medium (2-3/year) Failed deploys, downtime Automation scripts implemented
Migration conflicts (ADR-008) HIGH Low Plugin blocked Migration-check skill
Agent certificate validation disabled CORRECTED 2026-09-18 n/a n/a Built. See note below
No login rate limiting CORRECTED 2026-09-18 n/a n/a Built. See note below
Unauthenticated collector endpoints CORRECTED 2026-09-18 n/a n/a Per-PC endpoint credentials, not device tokens. See note below
Docker log disk fill HIGH unmeasured Container crashes STILL OPEN, no logging: block in docker-compose.yml

Risk matrix corrections (2026-09-18)

Agent certificate validation. Not disabled. AgentHttpFactory is the only HttpClient construction site in the agent, every channel runs ServerPinSet.Validate, and Validate has no path returning true for an unmatched pin (agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs:291-331). A failed registry re-read returns null and keeps the existing pins rather than silently unpinning (:352-386). The three SYSTEM PowerShell reporters refuse to send at all when ServerPins is set and the check cannot be enabled (plugins/computers/client/Report-AssetToShopDB.ps1:228-247, and the same block in Report-PrintersToShopDB.ps1 and upgrade-shopdbagent.ps1:436-447). The one ServerCertificateValidationCallback = {$true} left in the tree is test-vm-connectivity.ps1:44, a developer connectivity probe.

Login rate limiting. Built, and not "Q1 2027": shopdb/core/api/auth.py:35 _login_ratelimited(), keyed per host at :52, applied at :80, with tests/test_core/test_auth_ratelimit.py over it. Note the counter lives in per-process SimpleCache, so at four gunicorn workers the effective budget is four times the configured maximum.

Collector authentication. The mechanism is a per-PC endpoint credential, not a device token and not ADR-016. An enrolled PC sends X-Endpoint-Credential (plugins/endpoints/api/agentauth.py:1-18); the server stores only a sha256 hash plus a 12-character display prefix (plugins/endpoints/models/credential.py); once a PC holds an active credential the shared key is refused for it (plugins/endpoints/api/agentauth.py:129-134). ADR-016's own mechanism - credentials.fetch, GET /api/collector/credentials, Fernet at rest - is not built: grepping shopdb/ and plugins/ for credentials.fetch, collector/credentials and Fernet returns zero hits, matching ADR-016's own "Implementation status, 2026-08-14". The rollout runbook is plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md.

One boundary the runbook does not state: turning the shared key off does not close the four legacy core collector routes. /pc (shopdb/core/api/collector.py:349), /apps (:387), /heartbeat (:459) and /bulk (:494) are decorated @require_api_key, which at :154-164 checks only COLLECTOR_API_KEY or a collector.ingest-scoped token and never consults agent_shared_key_refused. Anything still holding the env shared key can rewrite logged-in user, last boot time, serial number and installed-app inventory for any hostname. Regenerating the key in the UI (plugins/endpoints/api/routes.py:1864-1884) mints a new managed token and clears the env fallback flag but does not change COLLECTOR_API_KEY, and _env_fallback_disabled(None) returns False (collector.py:60-74), so the legacy routes keep honouring the env key. Rotating COLLECTOR_API_KEY in .env is the step that closes it.

Automation Savings Summary

Skills created (Q3 2026): 13 automation skills

Skill Task Baseline hrs/year With skill Savings
/pre-commit Validation before commit 52 26 26
/docker-rebuild Container rebuild 52 26 26
/agent-deploy-full Agent deployment 52 26 26
/migration-check Migration validation 26 13 13
/production-health System health monitoring 104 52 52
/security-scan Security checks 26 13 13
/vm-snapshot VM snapshot management 78 26 52
/full-deploy Full deployment workflow 312 160 152
/plugin-new Scaffold new plugins 30 15 15
TOTAL 732 357 375

Note: Some tasks overlap (e.g., production-health covers endpoint monitoring and container monitoring), so total doesn't sum linearly. Realistic savings: 147-205 hrs/year from 936 hrs baseline.

Quick Wins (Immediate Impact)

1. Database Audit Log Retention (52 hrs/year saved)

Effort: 4 hours
Risk reduced: CRITICAL
Implementation:

-- Create retention policy procedure
CREATE PROCEDURE cleanup_audit_log()
BEGIN
  DELETE FROM auditlog 
  WHERE action IN ('view', 'list', 'search') 
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action NOT IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 2 YEAR;
END;

-- Schedule daily via cron
0 2 * * * docker compose exec db mysql -u root -p shopdb -e "CALL cleanup_audit_log()"

Verification: Monitor table size weekly for 4 weeks, confirm row count stable.

2. Docker Log Rotation (52 hrs/year saved)

Effort: 2 hours
Risk reduced: HIGH
Implementation:

# docker-compose.yml
services:
  api:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
  
  db:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Verification: Check log sizes after 1 week, confirm capped at 30MB per container.

3. Endpoint Events Retention (26 hrs/year saved)

Effort: 2 hours (similar to audit log)
Risk reduced: HIGH
Implementation: Similar procedure as audit log, 30-day retention for events.

4. Automated Backup with Rotation (56 hrs/year saved)

Effort: 6 hours
Risk reduced: CRITICAL
Implementation:

#!/bin/bash
# /opt/shopdb/backup-rotation.sh

DATE=$(date +%Y%m%d_%H%M)
BACKUP_DIR=/mnt/backups/shopdb
RETENTION_DAYS=30

# Create backup
docker compose exec db mysqldump -u root -p$MYSQL_ROOT_PASSWORD shopdb | \
  gzip > $BACKUP_DIR/shopdb-$DATE.sql.gz

# Remove old backups
find $BACKUP_DIR -name "shopdb-*.sql.gz" -mtime +$RETENTION_DAYS -delete

# Copy to offsite (USB or work PC)
rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/

Schedule: Daily at 2 AM via cron
Verification: Weekly restore test to dev VM, confirm data integrity

5. check-naming-and-style.sh Optimization (DONE)

Status: In the tree and on by default. USE_GIT_LS_FILES defaults to 1 (scripts/check-naming-and-style.sh:24) and the script takes the git ls-files -z path whenever it is inside a work tree (:26, :159, :183, :218), filtering out migrations/versions/, frontend/dist/, staticdocs/ and .plugins-staged/. Setting USE_GIT_LS_FILES=0 falls back to recursive grep with --exclude-dir for non-git checkouts. The whole script exits 0 on the deployment branch at 35d4fcd41.

The 5x / 10s-to-2s claim is unsourced and no timing was recorded for this page. The savings line below it follows from it and is equally unsourced.

Coverage note added 2026-09-18: the root-anchored ^migrations/versions/ filter does not exclude plugins/*/migrations/versions/, so plugin migrations ARE scanned. They are not caught by the column rule, though: that rule greps ^\s+[a-z]+_[a-z_]+\s*=\s*db\.Column (:236), which only matches a model attribute assignment, while migrations write sa.Column('name', ...). The column name is created in the one file the naming gate cannot read.

6. Pre-commit Hook Integration

Effort: 1 hour
Savings: Catches violations before push = 26 hrs/year
Implementation:

# .git/hooks/pre-commit
#!/bin/bash
bash scripts/check-naming-and-style.sh || exit 1
pytest tests/ -x || exit 1

7. Snapshot Management Skill (DONE)

Effort: 4 hours
Savings: 52 hrs/year
Status: /vm-snapshot skill created 2026-09-11

Usage:

/vm-snapshot create "before-deploy-2026-09-11"
/vm-snapshot list
/vm-snapshot restore "before-deploy-2026-09-11"

Total Quick Wins: 7 items, 21 hours effort, 264 hrs/year saved, 3 CRITICAL risks reduced

Long-Term Improvements (Planned)

1. Agent Delta Updates - BUILT (2026-09-18 update)

Original text: "Agent POSTs full inventory (11KB) every 30 seconds. Proposed: POST only deltas." Both halves are now wrong.

The cadence was never 30 seconds. The agent default is 900 seconds with up to 60 seconds of jitter: agent/ShopDbEndpointAgent/appsettings.json ("cadenceseconds": 900, "jitterseconds": 60) and agent/ShopDbEndpointAgent.Core/AgentConfig.cs:14 and :31. The 17GB/day and 1.44M-requests/day figures were derived from the wrong interval and should not be quoted.

The delta is implemented, on the server side. The agent still reports the full installed-software list each cycle, but each entry carries a SHA-256 fingerprint of displayname|displayversion|publisher, and _replace_applications (plugins/endpoints/plugin.py:1122-1140) compares fingerprint sets instead of delete-all-insert-all: adds for new fingerprints, deletes for departed ones, unchanged rows untouched (:1277-1289). The docstring states the effect as roughly 100 database operations per check-in down to about 2. It is guarded: fingerprint format validation, a delete-count check that warns rather than aborting, a memory cap above 5000 apps, a legacy fall-back for agents that send no fingerprints, and collision reconciliation inside a savepoint (:1318, :1387). The unique (assetid, appfingerprint) index was added NULL-filled by migration plugins/endpoints/migrations/versions/0020_*, so index creation cannot fail on a populated production table.

Remaining network-side work, if anyone wants it, is making the agent send only the changed entries rather than the full list. The database win is already taken.

2. Automated Staging Tests (Q4 2026)

Current: Manual smoke testing after each deploy
Proposed: Automated tests run in test VM before production deploy

Savings: 50-100 hrs/year (from deployment coordination)
Effort: 20 hours (test suite + CI integration)

3. Per-Endpoint Certificate Pinning - BUILT (2026-09-18 update)

Pins are SPKI SHA-256, validated as 64 lowercase hex on the way in, enforced on every agent channel, and owned by the registry rather than by appsettings.json (plugins/endpoints/client/Set-ShopDBServerPins.ps1 writes HKLM:\SOFTWARE\GE\ShopDB ServerPins; the precedence is stated in plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md, section "The registry wins"). A CA-flagged certificate is pinned but deliberately NOT imported into LocalMachine\Root, and the build warns the admin through an X-ShopDB-Build-Warning header. See the risk-matrix correction above for the enforcement call sites.

Monitoring & Alerting

Current: Reactive (users report issues)
Proposed: Proactive monitoring via /production-health skill

Metrics to monitor:

  • Database disk space (alert at 85%)
  • Docker log sizes (alert if rotation failing)
  • Audit log row count (alert if growing >10K rows/day)
  • Endpoint reporting (alert if >10 endpoints offline >48 hours)
  • API error rate (alert if >1% requests fail)
  • Backup age (alert if last backup >24 hours old)

Implementation: Cron job runs /production-health daily, sends email/Slack on failures

Effort: 8 hours
Savings: 52 hrs/year (from reactive troubleshooting to proactive alerts)

References

  • Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned
  • Architectural Findings 2026-09-11: re-checked 2026-09-18; most of the 22 findings are refuted or stale, see that page for the verdict on each
  • ADR-Conformance-2026-09-18: which ADRs were checked in the 2026-09-18 review and which hold
  • Automation Skills: 13 skills created in the assistant skills directory
  • Deployment Documentation: docs/deployment/ and the assistant setup guides
  • ADR-004: Deployment Topology (air-gapped architecture)
  • ADR-008: Plugin Migration Ownership (linear chain requirement)