shopdb-flask/docs/wiki-drafts/Maintenance-Footprint-Baseline.md

<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: Point-in-time measurement of what maintaining this codebase costs, annotated where the tree moved on.
loadwith: none
-->
# Maintenance Footprint Baseline - 2026-09-11

> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.


**Total annual burden:** 936 hours/year (estimate, see the note below)
**Risk level:** originally stated as 8 critical + 10 high operational risks
**Optimization potential:** 147-205 hours/year savings with automation
**Last checked against code:** 2026-09-18, the deployment branch at 35d4fcd41

## About the numbers (added 2026-09-18)

Every hour figure on this page is an estimate with no recorded measurement
behind it. No time log, ticket export or deploy log is cited anywhere, and none
was found in the repository. Treat the hours as an ordering of where effort
goes, not as data. The same applies to the row counts and growth rates in the
Database Management section: no query output is cited and none could be taken
from a repository checkout.

What CAN be checked from the repository is which of the recommendations have
since been built. Several have. The sections below are annotated
accordingly, dated 2026-09-18, and the stale risk-matrix rows are corrected.

## What the tree actually contains (2026-09-18)

| Thing | Count | How counted |
|-------|-------|-------------|
| Plugins | 18 directories, 17 with a `manifest.json` | `ls -d plugins/*/`; `applications` carries no manifest |
| Plugin migration revisions | 70 across 17 chains | `ls plugins/*/migrations/versions/*.py` |
| Largest single chain | `endpoints`, 24 revisions | one root, one head, no branch |
| Core migration revisions | 58 | `ls migrations/versions/*.py` |
| Test files | 223 | `find tests -name "test_*.py"` |
| Top-level scripts | 29 (.sh/.py/.ps1/.mjs), plus `scripts/backup`, `scripts/dev`, `scripts/migration`, `scripts/loadtest`, `scripts/site_imports` | `ls scripts/` |
| Automation skills | 13 | `ls <assistant-config>/skills/*.md` minus README |

The "13 automation skills" claim later on this page is the one headline figure
that checks out exactly.

## Executive Summary

Baseline assessment of ShopDB operational burden across air-gapped production environment (dev PC + work PC + VM + GitHub). Current maintenance requires 936 hrs/year (23.4 work-weeks) for routine tasks, with critical risks in database growth, backup gaps, and deployment coordination.

**Highest impact:** 312 hrs/year spent on manual deployment coordination across air-gapped systems. Automation via bundling scripts and deployment skills reduces to 100-160 hrs/year (152-212 hrs saved).

**Highest risk:** Database audit log with no retention policy (2.3M rows, 89MB, growing 4200 rows/day) will exhaust disk space and degrade query performance within 12-18 months without intervention.

## Maintenance Categories

### Database Management (234 hrs/year)

**Audit log maintenance:** 52 hrs/year (CRITICAL RISK)
- Current: No retention policy, manual purging when disk fills
- Growth rate: 4200 rows/day = 1.5M rows/year
- Current size: 2.3M rows, 89MB table + 156MB indexes
- Projected: Will exceed 10M rows (400MB) by Q2 2027
- Risk: Disk exhaustion, query timeouts, backup failures

**Recommended:** Automated retention policy
- Retention: 90 days for INFO/DEBUG actions, 2 years for admin/modification
- Implementation: Cron job runs `DELETE FROM auditlog WHERE ... AND created < NOW() - INTERVAL '90 days'`
- Savings: 52 hrs/year (from 52 hrs to ~0 hrs, fully automated)
- Status: PLANNED for Q4 2026

**2026-09-18 update: half built, and the half that is built is manual.**
Retention exists as a service and a CLI, not as a cron job and not as a stored
procedure. `shopdb/core/services/auditretention.py` provides `configureddays()`,
`stats()` and `purge()`; `flask audit stats` (`shopdb/cli/__init__.py:1891`)
reports size against the configured window and `flask audit purge`
(`:1911`) deletes, with a dry-run preview first and `--days` to override for one
run. The window is the `audit_retentiondays` setting
(`shopdb/core/api/settings.py:1049`), and its default is **`'0'`, which means
keep everything forever**: nothing is deleted until an operator sets a value.
There is no scheduler in the application and no cron entry in the repository, so
the purge runs when somebody runs it. The remaining work is a value in Settings
> Site & Facility and a host cron line, not code.

**Endpoint events cleanup:** 26 hrs/year (HIGH RISK)
- Current: 450K rows, 127MB, no retention
- Growth: Process events every 30 seconds, app launches, USB connects
- Risk: High-churn data fills disk faster than audit log

**Recommended:** 30-day retention for events, 90 days for snapshots
- Implementation: Similar cron job as audit log
- Savings: 26 hrs/year (fully automated)

**2026-09-18 update: partly built, per table rather than globally.** Operator
reports carry a retention window defaulting to 7 days
(`plugins/endpoints/api/routes.py:3619-3620`,
`endpoints.operatorreport.retentiondays`, 0 or less meaning keep forever), and
endpoint health transitions are trimmed to a high-water mark on write, keeping
every unresolved row regardless of age
(`plugins/endpoints/services/health.py:326`). The row counts and the 127MB
figure above were never sourced and are not verifiable from a checkout.

**Database backups:** 78 hrs/year (MEDIUM RISK)
- Current (as stated 2026-09-11): Manual mysqldump weekly, stored on same server
- Risk: No offsite copies, no restore testing, backup growth unbounded
- Backup size: 2.1GB compressed, growing 15% per quarter (unsourced)

**2026-09-18 update: STALE. This is built, and it is what the recommendation
below asks for.** `scripts/backup/` holds `backup.sh`, `restore.sh`, `status.sh`,
`watch.sh`, `simulate-restore.sh` and `test-restore-datarev.sh`, plus a
`Dockerfile.backup` and a compose file. Retention is enforced by restic:
`scripts/backup/backup.sh:134-141` runs `restic forget --keep-daily 5
--keep-weekly 4 --keep-monthly 6 --prune`. Destinations are configured in
Settings > Backups and include local, SMB and AWS S3, so offsite is a setting
rather than a manual copy. Each backup set carries `db.sql.gz`, the `instance/`
tree and a `manifest.json` of schema revisions, and restore validates schema
compatibility before it runs. A restore-simulation harness is in the tree.
Backup credential redaction landed in the same commit as audit retention
(`scripts/redact_backup_secrets.py`).

**Recommended:** Automated backup with rotation
- Daily incremental, weekly full, monthly archive
- Offsite copy to work PC or USB drive
- Automated restore testing in test VM
- 30-day retention for daily, 90-day for weekly, 1-year for monthly
- Savings: 56 hrs/year (from 78 to 22 hrs with automation + monitoring)

**Query optimization:** 52 hrs/year
- Current: Ad-hoc analysis when users report slow pages
- Common issues: Missing indexes on plugin tables, N+1 queries in dashboard
- Tools: `EXPLAIN ANALYZE`, slow query log (not enabled)

**Recommended:** Enable slow query log, monthly index analysis
- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

**Schema migrations:** 26 hrs/year
- Current: Plugin migrations run automatically on deploy
- Effort: Testing migrations in dev, verifying no conflicts (ADR-008)
- Risk: Migration chain breaks, plugins blocked from enabling

**Status:** Acceptable. ADR-008 (plugin migration ownership) reduces conflicts.

### Deployment Operations (312 hrs/year)

**Air-gap coordination:** 312 hrs/year (CRITICAL BURDEN)
- Current: Manual bundle creation, USB transfer, deploy in 3 environments
- Environments: dev PC (the internal git server) <-> work PC (GitHub) <-> VM (prod)
- Steps per deploy: 14 manual steps, 45-90 minutes each
- Frequency: 2-3 deploys/week = 104-156 deploys/year

**Time breakdown per deploy:**
- Bundle creation: 15 minutes (build frontend, tar source, verify integrity)
- Transfer dev->work: 10 minutes (USB copy + verification)
- GitHub push: 5 minutes (pull the upstream branch, merge down, resolve conflicts, push)
- Transfer work->VM: 10 minutes (USB or shared folder)
- VM deployment: 30-45 minutes (extract, migrate DB, restart services, smoke test)
- Rollback (if needed): +60 minutes (restore snapshot, rollback migrations)

**Automation implemented (Q3 2026):**
- Bundling scripts: `pull-shopdb-bundle.cmd`, `fetch-wilm-branch.cmd`
- Deployment skills: `/full-deploy`, `/agent-deploy-full`, `/docker-rebuild`
- VM snapshot skill: `/vm-snapshot` for pre-deploy snapshots
- Savings: 50-100 hrs/year (from 312 to 160-260 hrs with scripting)

**Remaining manual steps:**
- Physical USB transfer (air gap requirement)
- Smoke testing after deploy (safety check)
- Conflict resolution on merges (judgment required)

**Further optimization:** Automated staging tests, pre-flight validation, automated rollback triggers could save another 50-100 hrs/year.

### Docker & Container Management (156 hrs/year)

**Docker log rotation:** 52 hrs/year (HIGH RISK)
- Current: No log rotation configured
- Growth: API container logs grow 500MB/week
- Risk: Disk fills, Docker daemon hangs, services crash

**Recommended:** Configure log rotation in docker-compose.yml
```yaml
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
```
- Savings: 52 hrs/year (fully automated, from manual cleanup)

**2026-09-18 update: STILL OPEN, and still the cheapest item on this page.**
`grep -c logging docker-compose.yml` returns 0: there is no `logging:` block on
any service. Three services do carry `restart: unless-stopped`
(`docker-compose.yml:24`, `:104`, `:120`) and three carry `healthcheck:` blocks
(`:38`, `:112`, `:219`), so the container lifecycle is managed; log size is not.
The 500MB/week growth figure was never sourced.

**Image rebuilds:** 52 hrs/year
- Current: Manual rebuild when dependencies change
- Frequency: ~20 rebuilds/year (Python deps, base image updates)
- Time: 2-3 hours per rebuild (build, test, deploy)

**Automation:** `/docker-rebuild` skill handles full rebuild workflow
- Savings: 26 hrs/year (from 52 to 26 with scripted rebuild + testing)

**Container health monitoring:** 52 hrs/year
- Current: Manual checks when users report issues
- No automated alerting, no health endpoints for most services

**Recommended:** `/production-health` skill runs automated checks
- Monitors: Container status, API responsiveness, DB connectivity, disk space
- Run: Daily via cron, alerts on failures
- Savings: 26 hrs/year (from 52 to 26 with proactive monitoring)

### Testing & Validation (130 hrs/year)

**Pre-commit validation:** 52 hrs/year
- Current: Manual `pytest` and `check-naming-and-style.sh` before commits
- Failures: ~15% of commits have style violations or test failures caught late

**Automation:** `/pre-commit` skill runs full validation suite
- Tests: pytest, style check, security scan, migration check
- Time: 2-3 minutes (optimized script 5x faster with git ls-files)
- Savings: 26 hrs/year (from 52 to 26 with automated pre-commit hook)

**Migration validation:** 26 hrs/year
- Current: Manual checks for migration conflicts (ADR-008 violations)
- Tools: `pytest tests/test_plugin_migrations.py`, manual DB inspection

**Automation:** `/migration-check` skill validates migration chains
- Checks: Linear chains, no branches, correct down_revision, PLUGIN_TABLE_OWNERS order
- Savings: 13 hrs/year (from 26 to 13 with scripted validation)

**Agent deployment testing:** 52 hrs/year
- Current: Manual build + deploy to test VM, manual verification
- Frequency: ~25 agent updates/year
- Time: 2 hours per update (build, sign, deploy, test service, test tray)

**Automation:** `/agent-deploy-full` skill handles end-to-end workflow
- Steps: Build agent, deploy to VM via SSH, verify service, test heartbeat
- Savings: 26 hrs/year (from 52 to 26 with scripted deployment)

### Security & Compliance (78 hrs/year)

**Security scanning:** 26 hrs/year
- Current: Manual Darkwing scans on PRs, ad-hoc credential checks
- Coverage: Detects secrets in commits, but not all vulnerability classes

**Automation:** `/security-scan` skill runs comprehensive checks
- Scans: Hardcoded credentials, exposed keys, missing auth, SQL injection patterns
- Based on: Sept 2026 security audit findings (see Security-Audit-2026-09-11.md)
- Savings: 13 hrs/year (from 26 to 13 with automated scanning)

**Dependency updates:** 52 hrs/year
- Current: Manual review of Dependabot alerts, testing updates
- Frequency: ~40 alerts/year (Python + npm)
- Risk: Delayed updates accumulate, breaking changes harder to resolve

**Recommended:** Monthly dependency review cycle
- Review alerts, batch compatible updates, test in dev
- Savings: None (still requires judgment), but reduces risk

### Agent & Endpoint Management (26 hrs/year)

**Agent updates:** Covered under deployment (52 hrs/year)

**Endpoint troubleshooting:** 26 hrs/year
- Current: Manual investigation when endpoints stop reporting
- Common issues: Service stopped, network blocked, token expired, disk full

**Monitoring:** `/production-health` skill checks endpoint health
- Alerts: Endpoints offline >48 hours, stale inventory (>7 days)
- Savings: 13 hrs/year (from 26 to 13 with proactive alerts)

## Risk Matrix

| Risk | Severity | Likelihood | Impact | Mitigation |
|------|----------|------------|--------|------------|
| Database disk exhaustion (audit log) | CRITICAL | High (12-18 mo) | Service outage | Retention policy Q4 2026 |
| Backup failure (no testing) | CRITICAL | Medium | Data loss on restore | Automated restore tests |
| Docker log disk fill | HIGH | High (3-6 mo) | Container crashes | Log rotation Q4 2026 |
| Deployment coordination errors | HIGH | Medium (2-3/year) | Failed deploys, downtime | Automation scripts implemented |
| Migration conflicts (ADR-008) | HIGH | Low | Plugin blocked | Migration-check skill |
| Agent certificate validation disabled | CORRECTED 2026-09-18 | n/a | n/a | Built. See note below |
| No login rate limiting | CORRECTED 2026-09-18 | n/a | n/a | Built. See note below |
| Unauthenticated collector endpoints | CORRECTED 2026-09-18 | n/a | n/a | Per-PC endpoint credentials, not device tokens. See note below |
| Docker log disk fill | HIGH | unmeasured | Container crashes | STILL OPEN, no `logging:` block in docker-compose.yml |

### Risk matrix corrections (2026-09-18)

**Agent certificate validation.** Not disabled. `AgentHttpFactory` is the only
`HttpClient` construction site in the agent, every channel runs
`ServerPinSet.Validate`, and `Validate` has no path returning true for an
unmatched pin (`agent/ShopDbEndpointAgent.Core/AgentHttpFactory.cs:291-331`). A
failed registry re-read returns null and keeps the existing pins rather than
silently unpinning (`:352-386`). The three SYSTEM PowerShell reporters refuse to
send at all when `ServerPins` is set and the check cannot be enabled
(`plugins/computers/client/Report-AssetToShopDB.ps1:228-247`, and the same block
in `Report-PrintersToShopDB.ps1` and `upgrade-shopdbagent.ps1:436-447`). The one
`ServerCertificateValidationCallback = {$true}` left in the tree is
`test-vm-connectivity.ps1:44`, a developer connectivity probe.

**Login rate limiting.** Built, and not "Q1 2027":
`shopdb/core/api/auth.py:35` `_login_ratelimited()`, keyed per host at `:52`,
applied at `:80`, with `tests/test_core/test_auth_ratelimit.py` over it. Note
the counter lives in per-process `SimpleCache`, so at four gunicorn workers the
effective budget is four times the configured maximum.

**Collector authentication.** The mechanism is a per-PC endpoint credential,
not a device token and not ADR-016. An enrolled PC sends
`X-Endpoint-Credential` (`plugins/endpoints/api/agentauth.py:1-18`); the server
stores only a sha256 hash plus a 12-character display prefix
(`plugins/endpoints/models/credential.py`); once a PC holds an active
credential the shared key is refused for it
(`plugins/endpoints/api/agentauth.py:129-134`). ADR-016's own mechanism -
`credentials.fetch`, `GET /api/collector/credentials`, Fernet at rest - is
**not built**: grepping `shopdb/` and `plugins/` for `credentials.fetch`,
`collector/credentials` and `Fernet` returns zero hits, matching ADR-016's own
"Implementation status, 2026-08-14". The rollout runbook is
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`.

One boundary the runbook does not state: turning the shared key off does not
close the four legacy core collector routes. `/pc`
(`shopdb/core/api/collector.py:349`), `/apps` (`:387`), `/heartbeat` (`:459`)
and `/bulk` (`:494`) are decorated `@require_api_key`, which at `:154-164`
checks only `COLLECTOR_API_KEY` or a `collector.ingest`-scoped token and never
consults `agent_shared_key_refused`. Anything still holding the env shared key
can rewrite logged-in user, last boot time, serial number and installed-app
inventory for any hostname. Regenerating the key in the UI
(`plugins/endpoints/api/routes.py:1864-1884`) mints a new managed token and
clears the env fallback flag but does not change `COLLECTOR_API_KEY`, and
`_env_fallback_disabled(None)` returns False (`collector.py:60-74`), so the
legacy routes keep honouring the env key. Rotating `COLLECTOR_API_KEY` in
`.env` is the step that closes it.

## Automation Savings Summary

**Skills created (Q3 2026):** 13 automation skills

| Skill | Task | Baseline hrs/year | With skill | Savings |
|-------|------|-------------------|------------|---------|
| `/pre-commit` | Validation before commit | 52 | 26 | 26 |
| `/docker-rebuild` | Container rebuild | 52 | 26 | 26 |
| `/agent-deploy-full` | Agent deployment | 52 | 26 | 26 |
| `/migration-check` | Migration validation | 26 | 13 | 13 |
| `/production-health` | System health monitoring | 104 | 52 | 52 |
| `/security-scan` | Security checks | 26 | 13 | 13 |
| `/vm-snapshot` | VM snapshot management | 78 | 26 | 52 |
| `/full-deploy` | Full deployment workflow | 312 | 160 | 152 |
| `/plugin-new` | Scaffold new plugins | 30 | 15 | 15 |
| **TOTAL** | | **732** | **357** | **375** |

**Note:** Some tasks overlap (e.g., production-health covers endpoint monitoring and container monitoring), so total doesn't sum linearly. Realistic savings: **147-205 hrs/year** from 936 hrs baseline.

## Quick Wins (Immediate Impact)

### 1. Database Audit Log Retention (52 hrs/year saved)

**Effort:** 4 hours  
**Risk reduced:** CRITICAL  
**Implementation:**

```sql
-- Create retention policy procedure
CREATE PROCEDURE cleanup_audit_log()
BEGIN
  DELETE FROM auditlog 
  WHERE action IN ('view', 'list', 'search') 
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action NOT IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 90 DAY;
  
  DELETE FROM auditlog
  WHERE action IN ('create', 'update', 'delete', 'admin')
    AND created < NOW() - INTERVAL 2 YEAR;
END;

-- Schedule daily via cron
0 2 * * * docker compose exec db mysql -u root -p shopdb -e "CALL cleanup_audit_log()"
```

**Verification:** Monitor table size weekly for 4 weeks, confirm row count stable.

### 2. Docker Log Rotation (52 hrs/year saved)

**Effort:** 2 hours  
**Risk reduced:** HIGH  
**Implementation:**

```yaml
# docker-compose.yml
services:
  api:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
  
  db:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
```

**Verification:** Check log sizes after 1 week, confirm capped at 30MB per container.

### 3. Endpoint Events Retention (26 hrs/year saved)

**Effort:** 2 hours (similar to audit log)  
**Risk reduced:** HIGH  
**Implementation:** Similar procedure as audit log, 30-day retention for events.

### 4. Automated Backup with Rotation (56 hrs/year saved)

**Effort:** 6 hours  
**Risk reduced:** CRITICAL  
**Implementation:**

```bash
#!/bin/bash
# /opt/shopdb/backup-rotation.sh

DATE=$(date +%Y%m%d_%H%M)
BACKUP_DIR=/mnt/backups/shopdb
RETENTION_DAYS=30

# Create backup
docker compose exec db mysqldump -u root -p$MYSQL_ROOT_PASSWORD shopdb | \
  gzip > $BACKUP_DIR/shopdb-$DATE.sql.gz

# Remove old backups
find $BACKUP_DIR -name "shopdb-*.sql.gz" -mtime +$RETENTION_DAYS -delete

# Copy to offsite (USB or work PC)
rsync -av $BACKUP_DIR/ /mnt/offsite-backups/shopdb/
```

**Schedule:** Daily at 2 AM via cron  
**Verification:** Weekly restore test to dev VM, confirm data integrity

### 5. check-naming-and-style.sh Optimization (DONE)

**Status:** In the tree and on by default. `USE_GIT_LS_FILES` defaults to `1`
(`scripts/check-naming-and-style.sh:24`) and the script takes the
`git ls-files -z` path whenever it is inside a work tree (`:26`, `:159`, `:183`,
`:218`), filtering out `migrations/versions/`, `frontend/dist/`, `staticdocs/`
and `.plugins-staged/`. Setting `USE_GIT_LS_FILES=0` falls back to recursive
grep with `--exclude-dir` for non-git checkouts. The whole script exits 0 on
the deployment branch at 35d4fcd41.

**The 5x / 10s-to-2s claim is unsourced** and no timing was recorded for this
page. The savings line below it follows from it and is equally unsourced.

**Coverage note added 2026-09-18:** the root-anchored `^migrations/versions/`
filter does not exclude `plugins/*/migrations/versions/`, so plugin migrations
ARE scanned. They are not caught by the column rule, though: that rule greps
`^\s+[a-z]+_[a-z_]+\s*=\s*db\.Column` (`:236`), which only matches a model
attribute assignment, while migrations write `sa.Column('name', ...)`. The
column name is created in the one file the naming gate cannot read.

### 6. Pre-commit Hook Integration

**Effort:** 1 hour  
**Savings:** Catches violations before push = 26 hrs/year  
**Implementation:**

```bash
# .git/hooks/pre-commit
#!/bin/bash
bash scripts/check-naming-and-style.sh || exit 1
pytest tests/ -x || exit 1
```

### 7. Snapshot Management Skill (DONE)

**Effort:** 4 hours  
**Savings:** 52 hrs/year  
**Status:** `/vm-snapshot` skill created 2026-09-11

**Usage:**
```bash
/vm-snapshot create "before-deploy-2026-09-11"
/vm-snapshot list
/vm-snapshot restore "before-deploy-2026-09-11"
```

**Total Quick Wins:** 7 items, 21 hours effort, **264 hrs/year saved**, 3 CRITICAL risks reduced

## Long-Term Improvements (Planned)

### 1. Agent Delta Updates - BUILT (2026-09-18 update)

**Original text:** "Agent POSTs full inventory (11KB) every 30 seconds.
Proposed: POST only deltas." Both halves are now wrong.

**The cadence was never 30 seconds.** The agent default is 900 seconds with up
to 60 seconds of jitter: `agent/ShopDbEndpointAgent/appsettings.json`
(`"cadenceseconds": 900`, `"jitterseconds": 60`) and
`agent/ShopDbEndpointAgent.Core/AgentConfig.cs:14` and `:31`. The 17GB/day and
1.44M-requests/day figures were derived from the wrong interval and should not
be quoted.

**The delta is implemented, on the server side.** The agent still reports the
full installed-software list each cycle, but each entry carries a SHA-256
fingerprint of `displayname|displayversion|publisher`, and
`_replace_applications` (`plugins/endpoints/plugin.py:1122-1140`) compares
fingerprint sets instead of delete-all-insert-all: adds for new fingerprints,
deletes for departed ones, unchanged rows untouched (`:1277-1289`). The
docstring states the effect as roughly 100 database operations per check-in
down to about 2. It is guarded: fingerprint format validation, a delete-count
check that warns rather than aborting, a memory cap above 5000 apps, a legacy
fall-back for agents that send no fingerprints, and collision reconciliation
inside a savepoint (`:1318`, `:1387`). The unique `(assetid, appfingerprint)`
index was added NULL-filled by migration
`plugins/endpoints/migrations/versions/0020_*`, so index creation cannot fail on
a populated production table.

Remaining network-side work, if anyone wants it, is making the agent send only
the changed entries rather than the full list. The database win is already
taken.

### 2. Automated Staging Tests (Q4 2026)

**Current:** Manual smoke testing after each deploy  
**Proposed:** Automated tests run in test VM before production deploy

**Savings:** 50-100 hrs/year (from deployment coordination)  
**Effort:** 20 hours (test suite + CI integration)

### 3. Per-Endpoint Certificate Pinning - BUILT (2026-09-18 update)

Pins are SPKI SHA-256, validated as 64 lowercase hex on the way in, enforced on
every agent channel, and owned by the registry rather than by `appsettings.json`
(`plugins/endpoints/client/Set-ShopDBServerPins.ps1` writes
`HKLM:\SOFTWARE\GE\ShopDB` `ServerPins`; the precedence is stated in
`plugins/endpoints/docs/PER-PC-CREDENTIALS-ROLLOUT.md`, section "The registry
wins"). A CA-flagged certificate is pinned but deliberately NOT imported into
`LocalMachine\Root`, and the build warns the admin through an
`X-ShopDB-Build-Warning` header. See the risk-matrix correction above for the
enforcement call sites.

## Monitoring & Alerting

**Current:** Reactive (users report issues)  
**Proposed:** Proactive monitoring via `/production-health` skill

**Metrics to monitor:**
- Database disk space (alert at 85%)
- Docker log sizes (alert if rotation failing)
- Audit log row count (alert if growing >10K rows/day)
- Endpoint reporting (alert if >10 endpoints offline >48 hours)
- API error rate (alert if >1% requests fail)
- Backup age (alert if last backup >24 hours old)

**Implementation:** Cron job runs `/production-health` daily, sends email/Slack on failures

**Effort:** 8 hours  
**Savings:** 52 hrs/year (from reactive troubleshooting to proactive alerts)

## References

- Security Audit 2026-09-11: 4 critical + 6 high vulnerabilities, some fixed, some planned
- Architectural Findings 2026-09-11: re-checked 2026-09-18; most of the 22
  findings are refuted or stale, see that page for the verdict on each
- ADR-Conformance-2026-09-18: which ADRs were checked in the 2026-09-18 review
  and which hold
- Automation Skills: 13 skills created in the assistant skills directory
- Deployment Documentation: `docs/deployment/` and the assistant setup guides
- ADR-004: Deployment Topology (air-gapped architecture)
- ADR-008: Plugin Migration Ownership (linear chain requirement)