Make main the lean product a site receives

main was 16 days and 259 commits behind the line the product is actually built on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It was neither current nor curated - just stale. This brings it forward and, at the same time, makes it mean something: main is what a receiving site installs, and nothing else. WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty - plus applications, which is manifest-less core and always ships. The only manifest dependency inside that set is endpoints on computers, and both are in it, so the closure is self-contained. WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb stay on the site line. So do the Windows installer tree, the VM rehearsal helpers, the agent tooling config, one site's data loader, the internal audit and findings records, and the planner spreadsheet. 1,135 tracked files against 1,438. A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped: deploy/site-profile-universal.json still named all seventeen plugins. It is the file docs/DEPLOY.md tells a site to provision from, so a fresh install would have failed at `flask plugin apply-profile` with PluginError - the first thing a receiving site does, broken by the branch meant to serve them. It now names the twelve that exist, and its comment says the two must agree. agent/ was never refreshed. Whoever populated this branch copied shopdb/, plugins/, frontend/ and tests/ from the site line and left the agent tree at its September state: 22 C# and PowerShell sources missing, including AgentCredentials, EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the site line had deleted still present. The old tree is internally consistent, so it compiles, and nothing in CI builds it - every gate was green over a server that speaks enrollment and per-PC credentials to a bundled agent source that cannot. Resynced whole. DECISIONS THIS ENCODES, rather than defers: - The plugin-authoring lab builds printedparts, whose manifest depends on employees, so it could not stay. The lab remains on the site line and main ships no build-along tutorial. That is a real loss for an adopting developer and is recorded here rather than discovered later. - Recognition, Training and Recertification notifications are no longer seeded. Each splits one notification into a card per employee and renders a photo, which needs a plugin this branch does not bundle. Seeding them would offer an author a type that cannot resolve anybody. A site that installs employees gets them from that plugin's own seed. - Windows/IIS is documented as the legacy path throughout, and the README leads with Docker, because that is what a receiving site gets. The API description is regenerated for THIS branch - 347 paths and 483 operations, down from 415 and 573 - so the published contract describes what is here rather than what the site line serves. docs/CONFIG.md lost the two measuring-tool label keys the code no longer declares, which two lanes had decided opposite ways. Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run rather than skip, and every index line count is derived from the file it names.
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions
--- a/docs/BACKUP-RESTORE.md
+++ b/docs/BACKUP-RESTORE.md
@@ -0,0 +1,202 @@
+<!-- docmeta
+purpose: howto
+audience: operator, adopter
+status: current
+summary: Back up and restore a ShopDB instance - the database plus instance/ - on Docker or external MySQL.
+loadwith:
+-->
+
+# Backup and restore the ShopDB database
+
+This is the ShopDB instance itself. For the per-PC CONFIG backups the collector
+takes off shopfloor machines, see [BACKUP-KINDS.md](BACKUP-KINDS.md).
+
+Each site owns its own data (single-tenant, ADR-004), so backups are the site's
+responsibility. A complete backup is two parts, and **a database-only backup is
+not a backup**: restoring the database without the matching `instance/`
+directory leaves the app pointing at floor plans and logos that no longer exist.
+
+| Item | Location | Why |
+|------|----------|-----|
+| Database | MySQL `shopdb_flask` | All asset, user, audit and settings data. |
+| `instance/branding/` | `instance/` dir | Uploaded logos and favicon. |
+| `instance/modelimages/` | `instance/` dir | Uploaded vendor-model photos. |
+| `instance/` floor plans | `instance/` dir | Uploaded map blueprints. |
+| `instance/plugins.json` | `instance/` dir | Which plugins this site enabled. |
+| `.env` | repo root (offline, secured) | Secrets needed to bring the stack back up. Store separately from the data backup, in a secrets manager. |
+
+## The shipped Docker backup overlay
+
+`docker-compose.backup.yml` is an opt-in per-site overlay that automates all of
+the above. Turn it on with:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.backup.yml \
+  --profile backup up -d
+```
+
+It adds three cooperating services: `backup-store`, a resident restic REST
+server holding every snapshot in its own volume, deliberately NOT published to
+the host so backups are reachable only over the compose network; `backup-watcher`,
+which actuates the "Back up now" sentinel the api drops, because the api
+container cannot run docker itself; and `backup`, the same one-shot run manually
+(`run --rm backup`) for sites that prefer a host scheduler.
+
+Each run captures one recoverable set: `db.sql.gz` (a non-blocking
+`mysqldump --single-transaction`), a restic snapshot of `instance/`, and
+`manifest.json` recording the core and plugin Alembic revisions plus the image
+tag at capture time. That manifest is what makes a restore a CHECKED operation:
+`scripts/backup/restore.sh` refuses to load a set onto code whose schema is
+older than the set's, rather than leaving the site half-broken with rows
+referencing columns the live schema has not applied. It does NOT roll back code
+- restore the code first, run `flask db upgrade` and `flask plugin upgrade-all`,
+then load the data.
+
+Retention is restic-native: `--keep-daily 5 --keep-weekly 4 --keep-monthly 6`.
+`RESTIC_REPOSITORY` defaults to the bundled store and can point at S3, Azure
+Blob or a mounted share instead; credentials come from the site's env or secret
+store, never committed.
+
+## Backing up by hand
+
+### Database (Docker)
+
+```bash
+docker compose exec -T db mysqldump \
+  -u root -p"${MYSQL_ROOT_PASSWORD}" \
+  --single-transaction --routines --triggers \
+  shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
+```
+
+`--single-transaction` gives a consistent dump without locking the tables (InnoDB).
+
+### Database (external MySQL, no container)
+
+```bash
+mysqldump -h <host> -u <user> -p \
+  --single-transaction --routines --triggers \
+  shopdb_flask | gzip > shopdb-$(date +%F).sql.gz
+```
+
+### instance directory
+
+```bash
+tar czf instance-$(date +%F).tar.gz instance/
+```
+
+Recommended cadence: nightly database dump to offsite storage, 14-day
+retention; `instance/` captured on the same schedule (and always right before an
+upgrade). Verify a restore quarterly.
+
+## Restore
+
+Restoring replaces the current database contents. Do it into a known-empty or a
+throwaway target first if you are unsure.
+
+### Step 1: Bring up the stack (or a fresh one)
+
+```bash
+cp .env.example .env   # or restore your saved .env
+# ensure MYSQL_* and DATABASE_URL match the dump's database name (shopdb_flask)
+docker compose up -d db
+```
+
+Wait for the `db` container to report healthy (`docker compose ps`).
+
+### Step 2: Load the database dump
+
+```bash
+gunzip -c shopdb-2026-07-10.sql.gz | \
+  docker compose exec -T db mysql -u root -p"${MYSQL_ROOT_PASSWORD}" shopdb_flask
+```
+
+For an external MySQL:
+
+```bash
+gunzip -c shopdb-2026-07-10.sql.gz | mysql -h <host> -u <user> -p shopdb_flask
+```
+
+If the target database does not exist yet, create it as utf8mb4 first (matching
+the schema charset):
+
+```sql
+CREATE DATABASE shopdb_flask CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+```
+
+### Step 3: Restore the instance directory
+
+**On Docker, `instance/` is a named volume, not the host directory.** The api
+service mounts `instance_data:/app/instance`, so extracting the tarball into
+`./instance` on the host restores nothing the application can see. The data has
+to go into the volume:
+
+```bash
+tar xzf instance-2026-07-10.tar.gz             # unpacks ./instance on the host
+docker compose up -d api                       # creates the container and volume
+docker compose cp instance/. api:/app/instance # copy into the volume
+docker compose restart api
+```
+
+Check it landed before going on:
+
+```bash
+docker compose exec api ls /app/instance
+```
+
+`plugins.json` must be there. Without it the site comes back with its plugins
+disabled and database rows pointing at uploaded files that are no longer
+present.
+
+A site that has replaced the named volume with its own bind mount
+(`- ./instance:/app/instance`) restores by unpacking the tarball on the host and
+nothing else. Check `docker-compose.yml` rather than assuming which one you have.
+
+For a non-docker deploy, unpack the tarball over the application directory:
+`instance/` is a plain directory there.
+
+### Step 4: Reconcile migrations
+
+```bash
+docker compose exec api flask db upgrade
+docker compose exec api flask plugin upgrade-all
+```
+
+For a non-docker deploy, the same two commands without the prefix.
+
+`flask db upgrade` is a safety net: if the dump predates the current code, this
+applies only the core Alembic chain. `flask plugin upgrade-all` then applies any
+newer per-plugin migrations (each bundled plugin owns its own chain, ADR-008);
+without it, plugin-owned tables stay un-migrated. If the dump is at the same
+version both are no-ops.
+
+### Step 5: Verify
+
+- Log in with a known account.
+- Confirm the floor map renders (branding and map blueprints resolve from
+  `instance/`).
+- Spot-check a few asset records and the audit log.
+- `curl -s -X POST -H "Content-Type: application/json" -d '{}' http://localhost:5001/api/auth/login | jq .`
+  should return a `VALIDATION_ERROR`, not a 500.
+
+## What is lost if `.env` is lost
+
+`.env` is not in the database dump, and rebuilding it is not simply retyping it:
+
+| Value | If it is lost |
+|---|---|
+| `SECRET_KEY`, `JWT_SECRET_KEY` | Every issued token and session becomes invalid. Users log in again; managed API tokens must be reissued, which means every collector client needs its key replaced. Recoverable, but it is a fleet-wide job. |
+| `DATABASE_URL` password | Recoverable: reset the MySQL user's password and write the new one in. |
+| `MYSQL_ROOT_PASSWORD` | Recoverable through MySQL's own reset procedure, which requires stopping the server. |
+| `RESTIC_PASSWORD` | NOT recoverable. Every restic snapshot in the repository is encrypted with it; without it the backups are unreadable by anyone, including you. |
+| `ZABBIX_TOKEN` and similar integration tokens | Reissue at the far end. Nothing else breaks. |
+
+So back it up with the database, to somewhere as protected as the dump. A dump
+without its `.env` restores the data and locks everyone out of it.
+
+## See also
+
+- [BACKUP-KINDS.md](BACKUP-KINDS.md) - the other backups: per-PC config
+  snapshots taken by the collector. A different subject entirely.
+- [DEPLOY.md](DEPLOY.md) - first-time deploy
+- [UPGRADE.md](UPGRADE.md) - upgrade procedure (back up first)
+- [CONFIG.md](CONFIG.md) - environment variables and Setting keys