docs: add project health review, wiki update plan, API reference (Fable review)

This commit is contained in:
cproudlock
2026-07-30 07:51:12 -04:00
parent ecf4ef6edd
commit 8575837d8e
3 changed files with 3558 additions and 0 deletions

3311
docs/API-REFERENCE.md Normal file

File diff suppressed because it is too large Load Diff

56
docs/PROJECT-REVIEW.md Normal file
View File

@@ -0,0 +1,56 @@
# ShopDB Flask - Project Health Review
As of HEAD `ecf4ef6` (2026-07-30), product `__version__ 0.7.0`, contract `__contract_version__ 0.15.0` (verified in `shopdb/__init__.py`).
## 1. Executive Summary
**Overall: healthy engineering, drifting focus.** The stated product vision ("plugin system is the product," CLAUDE.md) is delivered: all 7 refactor phases are done, 13 bundled plugins are contract-compliant, the per-plugin migration regime (ADR-008) is live and exercised (geenforce is at `geenforce0002blobs`, proving the post-cutover chain works in anger), and CI enforces naming, contract, and real-MySQL migration idempotency. Test count grew 1077 -> 1159 since the last CLAUDE.md snapshot.
The concern is not quality but trajectory. CLAUDE.md and ROADMAP.md both name the last big milestone before 1.0 as "legacy-ASP data import + production pilot." The loader is built and VM-validated (16 stages, `scripts/site_imports/wjf/`), but the prod run has not happened, and ~35 of the last 60 commits went to the GE-Enforce HTTPS cutover instead. That work is legitimate and high-value, but it is feature/fleet work on one plugin, and it has accumulated two process debts that violate the project's own discipline: the cutover playbook (`docs/geenforce-api-cutover.md`) is the only dirty file in the repo and is **untracked**, and the `prod-patch-geenforce` robocopy fast-path can leave prod ahead of git.
**Verdict: on track against standards, behind against goals.** The 1.0 gate has four items; only one is arguably done, and the roadmap doc does not know it.
## 2. Standards Compliance
| Rule | Status | Evidence |
|---|---|---|
| Naming (tables/columns/vars, CONTRIBUTING.md) | **MET** | `scripts/check-naming-and-style.sh` present and executable; dedicated `naming` job in `.gitea/workflows/ci.yml`. One borderline: `asset.py:175` documents a derived API key `location_name` with an underscore - not a DB-mirrored column so likely legal, but worth a glance since "response keys match column names exactly" is the spirit of the rule. |
| Plugin contract (manifest.json, BasePlugin, `shopdb.api` only) | **MET** | 13 `plugins/*/manifest.json` verified; contract test suites in `tests/`; contract bumped correctly to 0.15.0 for the geenforce resource-scope fetch tokens (75386d2), per ADR-002 discipline. |
| Migration ownership (ADR-008) | **MET** | `PLUGIN_TABLE_OWNERS` registry tested by `tests/test_plugin_migrations.py` (`EXPECTED_HEAD_REVISION` lines 47-51); `migrations-mysql` CI job does fresh utf8mb4 MySQL 8 upgrade + all plugin chains + second-upgrade-no-op assertion. The geenforce `0002blobs` revision shows the per-plugin chain is being used as designed, not just anchored. |
| Versioning/release discipline (ADR-007) | **AT RISK** | Tags through v0.7.0 exist and contract bumps are disciplined, but the documentation half of the procedure has drifted - see Gaps 3. |
| ADRs canonical, new priorities get an ADR | **AT RISK** | 14 ADRs present. But lean per-site builds are half-shipped (ADR-014 ACCEPTED and implemented; `default_enabled: false` on 5 plugins) while ADR-013, which defines the catalog/tiers/signed-artifact model those builds imply, is still PROPOSED. The GE-Enforce HTTPS cutover itself - a major architectural shift off the SMB share - lives in an untracked doc, not an ADR or ADR-012 amendment. |
| Style (plain ASCII, no emojis, comment discipline) | **MET** | Enforced by the same pre-commit hook + CI naming job. |
| Everything in git / repo as source of truth | **VIOLATED** | `docs/geenforce-api-cutover.md` untracked (only dirty file, verified `git status`); `prod-patch-geenforce` robocopy path acknowledged in-doc as leaving prod ahead of git. |
## 3. Roadmap Status
**Done:** Phases 0-6 (contract lock through multi-site distribution, tags v0.5.0-v0.7.0). Legacy import machinery complete: `docs/IMPORT-API.md` contract, 16-stage wjf loader VM-validated. Air-gapped deploy kit (6534590).
**1.0 must-haves (ROADMAP.md), honestly scored:**
1. *Asset model fully wired* - **appears DONE but unrecorded.** `Asset.mapx` (`shopdb/core/models/asset.py:121`), `inheritsposition` (`relationship.py:132`), and propagation logic (`relationship.py`, `core/api/assets.py`, `cli/__init__.py`) are all in code. ROADMAP still lists this as outstanding. Verify the ADR-001 contract tests cover it, then strike it.
2. *Equipment data migration one-shot* - **NOT DONE.** `scripts/migration/` contains only `fix_legacy_schema.sql`, `one-offs/`, and a README. No equipment script.
3. *Printers legacy-table cleanup* - **NOT DONE.** Recent printers commits (0d40780..c075658) are installer/feature work, not retirement.
4. *External plugin UI packaging* - **NOT DONE**, and gate criterion 3 (one external plugin built end-to-end) has no evidence.
**In-flight:** GE-Enforce HTTPS cutover dominates (~35/60 recent commits). Per the cutover doc's own section 12: only displays/kiosks are on the API; cmm/collections/keyence/genspect/heattreat/partmarker/common fleet still enforce from the SFLD SMB share; loggedinuser resolution unwired; registry cleanup pending; 3DPrintRoom route is a placeholder. Secondary streams: printers install-batch, applications notes, server reclassification, TV dashboard.
**Pace/scope health:** Velocity is high and test coverage tracks the work (17 of ~29 plugin test files are geenforce). But the project has been at 0.7.0 with "prod pilot is the last big milestone" as the stated goal since mid-July, while shipping ~185 commits of plugin-feature work. That is a real product being used - good - but the 1.0 gate is not moving, and a half-migrated fleet (API for displays, SMB for everything else) is the worst place to pause the cutover.
## 4. Gaps and Risks
1. **Untracked cutover playbook** (`docs/geenforce-api-cutover.md`). The single most valuable in-flight document is one `rm` away from gone, and invisible to any other machine or contributor.
2. **Prod-ahead-of-git debt.** The `prod-patch-geenforce` fast-path means production behavior may not be reproducible from any commit. This directly undermines ADR-012's "engine is source of truth" and the release discipline of ADR-007.
3. **Documentation drift, three concrete instances (all verified):** ROADMAP.md header says contract 0.13.0 (actual 0.15.0); CLAUDE.md says 1077 tests (actual 1159 collected) and claims a "lean-build" CI job that does not exist in `.gitea/workflows/ci.yml` (jobs: backend, naming, frontend, migrations-mysql - lean coverage is folded into pytest via `tests/test_lean_build_guards.py`). Also `.github/workflows/ci.yml` differs from the gitea workflow - one of them is stale.
4. **Split-brain fleet enforcement.** Displays/kiosks on the API, the rest of the fleet on the SMB share, with staged-but-unpushed manifest fixes elsewhere (MTConnect v1 stranding). Two delivery mechanisms means two failure modes and doubles the audit surface until the cutover finishes.
5. **1.0 gate criterion 4 unproven:** `docs/DEPLOY.md` has not been validated by an actual fresh-host prod deploy. The air-gapped kit exists; the pilot does not.
6. **ADR-013 limbo:** lean builds shipped under ADR-014 while the catalog/signing model that makes external distribution safe remains PROPOSED. Fine short-term, but gate criterion 3 (external plugin) will force the question.
## 5. Prioritized Recommendations
1. **Commit `docs/geenforce-api-cutover.md` today.** Zero-cost, eliminates the worst single-point-of-loss risk.
2. **Reconcile prod-patched geenforce files back into git** and gate or retire the robocopy fast-path. Until prod == some tag, ADR-007 is fiction for this plugin.
3. **One doc-sync pass (30 min):** ROADMAP header to 0.15.0, CLAUDE.md test count and CI job list, strike must-have (a) if contract tests confirm the Asset wiring, delete or sync the stale `.github` workflow.
4. **Finish the cutover or park it cleanly.** Either drive the remaining fleet groups onto the API per section 12, or write down the frozen state as an ADR-012 amendment so the split-brain period is a documented decision, not drift.
5. **Schedule the prodscratch import run and prod pilot.** This is the actual 1.0 milestone and everything is built for it; it validates DEPLOY.md (gate 4) for free.
6. **Pair the equipment one-shot migration with printers retirement** (must-haves b and c) - they are coordinated by design; doing them together avoids touching the legacy tables twice.
7. **Decide ADR-013** before building the external-plugin end-to-end proof (gate 3); the geenforce client work is the natural seed for that external plugin.

191
docs/WIKI-UPDATE-PLAN.md Normal file
View File

@@ -0,0 +1,191 @@
# WIKI UPDATE PLAN - shopdb-flask docs (as of HEAD ecf4ef6, 2026-07-30)
Priority order: items 1-4 are stale-on-shipped-features (fix first), 5-6 are new pages, 7-10 are minor rows/notes, 11 is structural.
---
## 1. docs/geenforce-api-cutover.md - UPDATE (and COMMIT - it is untracked)
**Action: update + `git add`.** This is the most valuable in-flight doc and the only dirty file in the repo. Committing it is step zero of this plan.
**Changes:**
a) Section 5, subsection "The display-type.txt dispatcher pattern" (line ~338): it predates commit b22701a. The dispatcher is no longer file-first; it is server-first. Rewrite the subsection opening to:
> ### The display dispatcher: server-resolved role, file fallback
>
> The inline dispatcher payload (built by `plugins/geenforce/seed_display_scope.py`) resolves what the display should show in two steps:
>
> 1. **Server (authoritative):** `GET $KioskBaseUrl/api/dashboarddefaults/display-role?fqdn=<own-fqdn>`. This is a PUBLIC endpoint (no token). The server matches the FQDN against the `dashboarddefaults` table (IP fallback) and returns `{role, path, businessunitid, businessunit}`. Roles: `dashboard`, `lobby`, `partskiosk`. Changing a display's job is now a server-side edit; no touch on the PC.
> 2. **Fallback (offline, or unmapped):** the local `C:\Enrollment\display-type.txt` value against the `DISPLAY_TYPE_TARGETS` map baked into the script. If neither resolves, the dispatcher logs and configures nothing.
>
> The FQDN is derived from the hostname plus `DisplayFqdnDomain` (registry) or the built-in default domain. `DetectionMethod = Always`, but the script is idempotent: it rewrites the all-users Startup shortcut only when the resolved target changed.
b) Section 12 (Open items): add a line that the robocopy `prod-patch-geenforce` reconciliation debt now includes this very doc being untracked - remove that line after commit.
c) Add a "See also" block near the top linking `GE-ENFORCE-DISPLAY.md`, `GE-ENFORCE-CLIENT.md`, `GE-ENFORCE-DEPLOY.md`, and (new) `API-REFERENCE.md`.
---
## 2. docs/GE-ENFORCE-DISPLAY.md - UPDATE
**Action: update.** One stale claim plus one missing section.
**Changes:**
a) The "display-type -> target map" section (lines ~30, ~61-75) presents `display-type.txt` as THE role source. Retitle the section to **"Role resolution: server first, display-type.txt fallback"** and insert before the table:
> The dispatcher first asks the server: `GET /api/dashboarddefaults/display-role?fqdn=<fqdn>` (public, unauthenticated). A row in `dashboarddefaults` keyed by the display's FQDN (IP fallback) wins and returns the role and frontend path directly. Only when the server is unreachable or has no mapping does the dispatcher fall back to the local `display-type.txt` map below. To repurpose a display, edit its `dashboarddefaults` row; the change takes effect on the next enforce cycle.
Keep the existing table, but relabel its caption "fallback map (local file)".
b) Add a new short section **"Dashboard-defaults FQDN keying"** (this is the home for missing-doc item [D]):
> ### Dashboard-defaults FQDN keying
>
> `dashboarddefaults` rows were historically keyed by IP. Migration `7d31_dashboarddefault_fqdn` added an `fqdn` column; resolution is now FQDN-first with IP as fallback (`_resolve_default` in `shopdb/core/api/dashboarddefaults.py`). FQDNs are stored lowercase. This survives DHCP churn on kiosk subnets. `POST /api/dashboarddefaults` accepts `fqdn`, `ipaddress`, `displayrole` (`dashboard`|`lobby`|`partskiosk`), `displaypath`, `businessunitid`. Two public read endpoints consume it: `/api/dashboarddefaults/display-role` (dispatcher) and `/api/dashboarddefaults/visitor-location` (lobby business-unit lookup). `plugins/geenforce/seed_display_scope.py` builds the client-side lookup (line ~130).
---
## 3. docs/GE-ENFORCE.md - UPDATE
**Action: update lines 288-291.** The sentence "Until that cutover, the client only REPORTS; the manifest still comes from the share via Export to Share (4.2)" is false for the displays cohort. Replace with:
> The cutover from share-sourced to shopdb-sourced manifests is per PC type. The **displays/kiosks cohort has cut over**: share-less display PCs fetch their manifest and payloads entirely over HTTPS (see `docs/geenforce-api-cutover.md` and `docs/GE-ENFORCE-DISPLAY.md`). All other fleet PC types (cmm, collections, keyence, genspect, heattreat, partmarker, nocollections, common) still enforce from the SFLD SMB share via Export to Share (4.2) and only REPORT to shopdb. The playbook for moving the next PC type is `geenforce-api-cutover.md` section 11.
---
## 4. docs/PILOT-DEPLOY.md - UPDATE
**Action: update.** Two changes:
a) Line 88 asset-count table predates the servers-to-network reclassify [F]. The writer must re-run the counts on the current prodscratch after the reclassify script; do not hand-edit numbers. Replace the row with re-measured values and a footnote:
> Counts taken AFTER `scripts/reclassify_servers_to_network.py --commit`. Servers imported as computers are re-pointed to network devices, so the computer count drops and network rises by the same amount versus a raw import.
b) Add a new numbered step to the import/verify flow, immediately after the import stages complete and before the parallel-validation counts:
> ### Reclassify servers into network devices
>
> The classic DB stored servers as PCs, so the import lands them as `computer` assets. Re-point them in place:
>
> ```
> DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py # dry run, prints matches
> DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py --commit # apply
> # match on an exact computer type instead of the SVR- name prefix:
> ... --type "Server" --commit
> ```
>
> The assetid does not change: communications, relationships, map position, and audit history carry over. Only the extension row is swapped (computers -> networkdevices) and the asset type flipped; reclassified devices get the `Server` networkdevicetype. Run the dry run, eyeball the list, then commit. Re-running is safe (already-moved assets no longer match).
---
## 5. NEW: docs/PRINTER-INSTALLER.md - CREATE
**Action: create.** Missing-doc item [B]. This is a consumed client contract (GE-ENFORCE-DISPLAY.md:82 lists "printer map" as a fleet manifest entry) with zero coverage. Model it on COLLECTOR-INTEGRATION.md (contract doc for an external client). Draft:
> # Printer installer map and install endpoints
>
> Replaces the classic apiprinters.asp / apipcdefaultprinter.asp / installprinter.asp contract. Shopfloor 2.0 PCs cannot run unsigned .bat maps, so a signed installer EXE (and the public web map page) drives installs from three endpoints in the printers plugin (`plugins/printers/api/asset_routes.py`). All three are `@jwt_required(optional=True)`: anonymous fleet clients work, a logged-in browser also works.
>
> ## The public map page
>
> `PrinterInstallerMap` is a public (no-login) frontend page: the floor map with printer hotspots at `mapx`/`mapy`. The user clicks printers, the page requests an install batch. The PC's default printer is preselected via `pc-default`.
>
> ## GET /api/printers/install-list
>
> Flat, unpaginated list of active NETWORK printers (must have a hostname or a non-USB IP; USB-only printers are excluded). Fields per row: `printerid`, `name`, `machinenumber`, `windowsname`, `sharename`, `hostname`, `ipaddress`, `vendorname`, `modelnumber`, `installpath`, `iscsf`, `locationname`, `mapx`, `mapy`.
> `?format=text` returns a pipe-delimited line per printer (fixed field order: printerid|windowsname|vendorname|modelnumber|hostname|ipaddress|mapx|mapy) so the Inno/Pascal installer does a split() instead of parsing JSON.
>
> ## GET /api/printers/pc-default?machine=NNNN
>
> The PC's default printer by machine (asset) number, persisted at PXE enrollment. Resolved via the `defaultprinter` asset relationship (PC asset -> printer asset). Returns `{printerid, windowsname}`, or `{}` when unknown/none. `?format=text` returns one pipe line, or an EMPTY body for none.
>
> ## GET /api/printers/install-batch?printerids=1,2,3
>
> Returns a self-deleting Windows .bat installing the selected printers, grouped like classic installprinter.asp:
> - HP / Xerox: one universal `PrinterInstaller.exe /PRINTER="a,b,c"` call
> - printers with a `.exe` installpath: PowerShell Invoke-WebRequest download (caller's Windows creds, site base URL + IIS `/installers` folder) then run `/SILENT`
> - no installpath or `.zip`: listed as manual install
> `printerids` is required (comma-separated, non-numeric tokens ignored).
>
> ## Fleet wiring
>
> The `common` scope's `printer map` manifest entry (see GE-ENFORCE-DISPLAY.md) lays down the signed installer. Install name preference: `windowsname`, else `sharename`, else asset name/number.
Cross-link it from PLUGINS.md printers row (item 7) and GE-ENFORCE-DISPLAY.md:82.
---
## 6. docs/GE-ENFORCE-DEPLOY.md - UPDATE (minor)
**Action: update.** Add one cross-link paragraph near the ShopdbUrl/ApiToken registry contract section:
> For PC types that have cut over to HTTPS manifest delivery (currently displays/kiosks), the full server-side setup, auth model (IP allowlist vs ApiToken), and per-PC-type cutover playbook live in `geenforce-api-cutover.md`. This doc covers what gets laid on the PC; that doc covers where the manifest comes from.
---
## 7. docs/PLUGINS.md - UPDATE (two rows)
**Action: update.**
Line 13, printers row, new text:
> | `printers` | Network and shop-floor printers | Public installer map page + fleet install contract (`/api/printers/install-list`, `/pc-default`, `/install-batch`, see PRINTER-INSTALLER.md). Optional Zabbix integration for supply tracking. Legacy `PrinterData` retiring per ADR-001. |
Line 22, slides row, new text:
> | `slides` | Slides for the lobby display and shop-floor screensaver | Upload / reorder / delete per surface. Management gated on the `slides.manage` permission, grantable to non-admin curators; display routes are public. |
Also add a short **"Plugin permissions"** note after the table (first place any plugin permission is documented):
> Plugins may register their own permissions (e.g. `slides.manage`). Admins implicitly hold them; grant them to specific roles/users to delegate curation without admin. Each plugin's registered permissions appear in its `plugin.py` `get_permissions()`.
---
## 8. docs/IMPORT-API.md - UPDATE (one row)
**Action: update line 237.** Replace the dashboarddefaults row:
> | `dashboarddefaults` | `POST /api/dashboarddefaults` | `fqdn` (preferred key, stored lowercase), `ipaddress` (fallback key), `displayrole` (`dashboard`/`lobby`/`partskiosk`), `displaypath`, `businessunitid` (remapped), `description` | `fqdn`, else `ipaddress` |
Add a one-line note under the table: "Resolution at runtime is FQDN-first with IP fallback (migration `7d31_dashboarddefault_fqdn`); import both when the legacy source has them."
---
## 9. docs/CONFIG.md - UPDATE (one note)
**Action: update.** In the "search (dynamic)" section (line ~316), append:
> Search terms are matched word-wise: a multi-word query returns rows containing EVERY word, each word anywhere in the searched fields, in any order ("CSF Roles" matches a row with "CSF" and "Roles" in different columns). Quoting does not force a contiguous phrase.
No full doc for [E]; this note plus a CHANGELOG line covers it.
---
## 10. docs/IMPORT-ADOPTION.md - UPDATE (optional, one line)
**Action: update.** In the per-site loader section, add: "Post-import fixups that re-point existing assets (example: `scripts/reclassify_servers_to_network.py`, servers imported as PCs moved to network devices in place) belong in the site loader's verify stage, not in the stable API layer."
---
## 11. NEW: docs/API-REFERENCE.md - CREATE (proposal)
**Where:** `docs/API-REFERENCE.md`, wiki-bound alongside the rest.
**Scope:** an index, not a spec. One table per audience, each row = endpoint, auth, one-line purpose, link to the owning contract doc. Sections:
1. **Fleet/client contracts (unauthenticated or token)** - geenforce fetch/report + `/payload/<sha>` (-> GE-ENFORCE-CLIENT.md, geenforce-api-cutover.md), collector (-> COLLECTOR-INTEGRATION.md), printers install trio (-> PRINTER-INSTALLER.md), `dashboarddefaults/display-role` + `visitor-location` (-> GE-ENFORCE-DISPLAY.md).
2. **Import API** - pointer to IMPORT-API.md, do not duplicate.
3. **Core UI API** - one line: JWT-authenticated, versioned by plugin contract (0.15.0), see CONTRACT-STABILITY.md; enumerate only the public/optional-auth endpoints since those are the site-exposure surface a firewall reviewer asks about.
**Linked from:** docs/adr/README.md sibling index if one exists, PLUGINS.md header, DEPLOY.md security section (public-endpoint inventory is exactly what a deploy reviewer needs), and each contract doc's "See also". Rule to state at top: detailed request/response shapes live in the contract docs; this page only answers "what endpoints exist, who calls them, what auth".
---
## Out of scope for the wiki but flagged to the maintainer
- CLAUDE.md drift (test count 1077 vs 1159, phantom "lean-build job", 2026-07-13 "Current state" missing the entire HTTPS-cutover arc) is repo-doc, not wiki, but should ride the same commit.
- No retirements: every existing page stays. ADR-005 must NOT be edited for [F]; ADRs are immutable and its "reclassification" is equipment/measuringtools only. The reclassify script is operational, documented in PILOT-DEPLOY.md (item 4).
Key source files for the writer: `/home/camp/projects/shopdb-flask/plugins/printers/api/asset_routes.py` (install endpoints, lines 311/432/568), `/home/camp/projects/shopdb-flask/shopdb/core/api/dashboarddefaults.py` (display-role, line 72), `/home/camp/projects/shopdb-flask/plugins/geenforce/seed_display_scope.py` (dispatcher, line ~117-151), `/home/camp/projects/shopdb-flask/scripts/reclassify_servers_to_network.py` (usage in module docstring), `/home/camp/projects/shopdb-flask/shopdb/core/api/search.py` (`_word_match`, lines 27-42), `/home/camp/projects/shopdb-flask/plugins/slides/plugin.py` (slides.manage, line 66).