diff --git a/CHANGELOG.md b/CHANGELOG.md index 228ac33..79e10b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,6 +66,106 @@ ADR-007 and ADR-002. components nobody wrote. Cards are ordered by severity, hide themselves when there is nothing to report, gate on a permission, and each fetches independently so one broken endpoint cannot blank the board. +- Tech Tools: a section for the small conversions and code generation that + otherwise happen in a browser tab on somebody's phone. Starts with barcode and + QR codes laid out for the label stock actually in the printers. +- Kiosks report what they are. The dispatcher already read + `C:\Enrollment\display-type.txt` to pick its target; now it tells ShopDB, so + the Asset column on an enforcement report shows a reported value instead of + one inferred from a mapping table nobody filled in. +- Enforcement reports carry what the PC is, where it is, and whether it is + backed up: the host links to its own page, the Asset column names the machine + or instrument the PC controls, a map pin shows the location on hover, and the + backup column judges the chain rather than printing its last date. +- `flask relationships audit`, read-only, listing links that cannot both be + true, plus an inverse guard that refuses to create one. +- An employees endpoint that says WHY a name did not resolve, rather than + returning nothing and leaving the caller to guess. +- The display client module updates itself through the manifest, so a client + change no longer means hands on every kiosk. +- `docs/PROJECT-MAP.md`, generated by `scripts/gen_project_map.py`: versions, + the plugin inventory, every Alembic chain head, the ADR index and the size of + the codebase, derived from the code. The hand-written equivalents in CLAUDE.md + had drifted by two contract releases and 400 tests. + +### Changed + +- Plugin contract **0.19.0**: `get_dashboard_widgets` returns data and shape, + not a component name. The old shape named a Vue component per widget, which + cannot survive a lean build (ADR-013) and had five plugins pointing at + components nobody wrote. The change itself shipped earlier without a bump and + with `BasePlugin` and PLUGIN-HOOKS.md still teaching the dead shape; this + records it and corrects both. +- Plugin contract **0.17.0**: `DashboardDefault` is part of the `shopdb.api` + surface, so a plugin can resolve a display without reaching into core. +- Plugin contract **0.18.0**: one name per display role, and it is the kiosk's + own (`Dashboard`, `Lobby`, `3DPrintRoom`). Server and dispatcher had three + vocabularies for the same three kiosks, and a display could report a role core + could not store. `normalize_display_role` resolves case and the retired + spelling; an unknown value is kept verbatim so a typo stays visible. + Contract versions are their own series (ADR-002) and belong in this file; + 0.17.0 and 0.18.0 shipped without an entry, which is what this note corrects. +- Publishing a GE-Enforce manifest is refused when PCs reporting for that scope + run an older client library than the manifest needs, naming the hosts. A minor + version bump that NARROWS behaviour is not backward compatible: an old library + reads a gated entry as ungated and installs every version it cannot detect. +- Shadow mode runs inside the enforce cycle instead of on its own schedule. The + share is mounted only for the length of a cycle, so a task on its own clock + woke to a missing drive and reported 0 installed / 0 skipped / 0 failed, which + is indistinguishable from a healthy no-op. +- A GE-Enforce report is judged stale on both clocks, the server's and the + client's asserted check-in, so a PC cannot mask real silence with a future + timestamp. A PC that has gone quiet stops reading as healthy. + +### Fixed + +- **Every plugin migration chain re-ran its head on each MySQL deploy.** Core's + Alembic env got a `connection.commit()`; the per-plugin template did not, so + the last migration of every run lost its version stamp while its DDL survived. + Invisible for as long as every head happened to be idempotent - and two were + not. The backups head cleared a column for every row, and the geenforce head + added a column unconditionally and failed on the second attempt. Both are now + bounded and guarded, and the template commits. +- A part-marker PC reported 200 on the cycle that created its marker and 500 on + every cycle after. Both of the links it makes were looked up by label, so a row + made by hand or by the legacy import was invisible and the insert violated the + uniqueness of (source, target, type); and the marker's asset number is derived + from the PC, so it could already be taken. +- A CMM reported the same way: the instrument IS the reported bay, so adoption + was relabelling the row the machine sync owns and finding it by that label. +- A reported identifier is matched exactly rather than with `ilike`, so `_` and + `%` in a file on a shopfloor PC are no longer wildcards. `MT-600_` had been + adopting MT-6001, and a bare `%` adopted whatever active asset came first. +- Measuring tools are adopted before they are minted. Minting derived the asset + number from the HOSTNAME, so a permanent instrument took the identity of + whichever PC drove it that week, and the collector could not see a tool it had + not created - which left legacy instruments shadowed by minted twins. +- An unanchored rsync exclude in the export was matching at any depth, so a + whole plugin was silently dropped from the published payload. Root excludes + are anchored and the export now aborts if a plugin manifest fails to survive. +- Backup revisions are ordered in Python, not with `ORDER BY ... NULLS LAST`, + which SQLite accepts and MySQL rejects. + +### Security + +- The Windows installer breaks ACL inheritance on the roots it creates. A + directory made directly under `C:\` inherits an inherit-only Modify grant for + Authenticated Users, so `C:\shopdb-flask`, `C:\Python314` and `C:\MySQL84` + were writable by any logged-on user. Two files inside them make that a local + privilege escalation: `shopdb-admin.ps1`, which self-elevates, and + `instance/config.py`, which the app executes unconditionally. The MySQL data + directory is locked down as well, since it holds the password hashes. A Python + installation this installer did not create is reported rather than seized. +- Site-specific values no longer ship in product code (ADR-015). The backups + share root defaults to blank instead of one site's file server, the shadow + client no longer overwrites a site's own ShopDB URL in the registry on every + cycle, the kiosk dispatcher takes its URL from the `site_base_url` setting and + refuses rather than guessing, and one site's hostname, FQDN and internal + subnets are out of the documentation, the generated API specs and the UI + placeholders. The ADR-015 scanner now covers PowerShell, the installer, the + seeds and generated JSON, case-insensitively, and FAILS the build; deliberate + organisation-wide defaults are declared with an `ADR-015-OK` marker. Both + publication gates carry the site patterns. ## [0.9.0] - 2026-08-11 diff --git a/CLAUDE.md b/CLAUDE.md index 04daa04..455fff8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,145 +1,131 @@ # ShopDB Flask Project -Modern rewrite of the classic-ASP shopdb. Built as a framework so sister GE Aerospace sites can adopt it. Plugin system is the product. +Modern rewrite of the classic-ASP shopdb. Built as a framework so sister GE +Aerospace sites can adopt it. The plugin system is the product. + +**This site is not the only site.** Anything that names West Jefferson - a +hostname, a share path, a subnet, a machine-number format, a label prefix - is a +defect unless it is a setting with a neutral default, a site-namespaced +directory, or seed data. That is ADR-015, and it is the rule most often broken +in the places tooling cannot see: PowerShell, the installer, JSON, the imaging +share. + +## Facts that move live in the map + +`docs/PROJECT-MAP.md` is GENERATED. It carries the product and contract +versions, the plugin inventory, every Alembic chain head, the ADR index with +status, and the size of the codebase. Read it instead of asking; regenerate it +rather than correcting it by hand: + +```bash +venv/bin/python scripts/gen_project_map.py # write +venv/bin/python scripts/gen_project_map.py --check # CI form +``` + +Hand-written version numbers in this file were wrong by two contract releases +and 400 tests before the map existed. Nothing that the generator can derive +should be typed here again. + +## Where to look + +| question | read | +|---|---| +| what a name may be called | `CONTRIBUTING.md` | +| why the architecture is like that | `docs/adr/` (index in the map) | +| what a plugin may touch | `docs/PLUGIN-HOOKS.md` | +| what the collector sends | `docs/COLLECTOR-INTEGRATION.md`, ADR-006 | +| how a site is deployed | `docs/INSTALL-WINDOWS.md`, `docs/OPERATE-WINDOWS.md` | +| how code reaches GitHub | `tools/export-github.sh` | + +## Standards are gates, not prose + +Compliance is checked, not remembered. Run these rather than re-reading the +rules: + +```bash +bash scripts/check-naming-and-style.sh # naming, style, site literals (ADR-015) +venv/bin/python -m pytest -q # includes the docs publishability gate +venv/bin/python -m pytest tests/test_plugin_migrations.py # chain heads registry +venv/bin/python scripts/gen_project_map.py --check +``` + +A new plugin table means updating `PLUGIN_TABLE_OWNERS` and +`EXPECTED_HEAD_REVISION` in `tests/test_plugin_migrations.py`. ## Database -- **Active database:** `shopdb_flask` (MySQL, asset-based schema) -- **Legacy database:** `shopdb` (Classic ASP schema, used only for one-time data import via `scripts/import_from_mysql.py`) -- **Connection:** `.env` file. See `.env.example`. +- **Active:** `shopdb_flask` (MySQL, asset-based schema) +- **Legacy:** `shopdb` (classic ASP schema; one-time import only) +- **Connection:** `.env`, see `.env.example` -Architecture decisions live in `docs/adr/`. Read those before making schema or contract changes. +Core migrations run with `flask db upgrade`; every plugin owns its own chain +(ADR-008) and runs with `flask plugin upgrade-all`. **A deploy needs both** - +skipping the second is how a new column reaches production as a 1054 error. -- ADR-001: Asset model is the platform contract (Machine retires) - ACCEPTED -- ADR-002: Plugin contract versioning (semver) - ACCEPTED -- ADR-003: Plugin distribution model (in-tree bundled + filesystem-based external) - ACCEPTED -- ADR-004: Deployment topology (per-site instances, not multi-tenant) - ACCEPTED -- ADR-005: Equipment vs measuringtools plugin scope - ACCEPTED -- ADR-006: Plugin collector contract pattern - ACCEPTED -- ADR-007: Product versioning and releases - ACCEPTED -- ADR-008: Plugin migration ownership (per-plugin chains) - ACCEPTED -- ADR-009: Frontend plugin route gating - ACCEPTED -- ADR-010: Frontend plugin hook contract - ACCEPTED -- ADR-011: Machines rename + modeltypes retyping - ACCEPTED -- ADR-012: GE-Enforce manifest ownership in shopdb - ACCEPTED -- ADR-013: Plugin catalog, curated shelf, and lean per-site builds - PROPOSED -- ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, prune not-installed plugin tables) - ACCEPTED -- ADR-015: Where a site's own data is allowed to live (setting with a neutral default, site-namespaced dir, or seed) - ACCEPTED -- ADR-016: Credential delivery to the fleet (ShopDB resolves targeting, encrypted at rest, dedicated fetch scope) - ACCEPTED +Tests run on SQLite, production is MySQL. Dialect drift is not caught by the +suite: `ORDER BY ... NULLS LAST` parses on SQLite and is rejected by MySQL. Sort +in Python when in doubt. ## Coding convention -`CONTRIBUTING.md` defines naming rules (DB tables, columns, Python, JS, Vue, API). Pre-commit hook at `scripts/check-naming-and-style.sh` enforces them. Read `CONTRIBUTING.md` before naming any new identifier. - -## Current state (as of 2026-07-13) - -Refactor phases 0-5 landed; phase 6 (multi-site distribution readiness) largely complete; the last big milestone is the legacy-data import + a production pilot. - -### Phases done - -- **Phase 0**: 6 ADRs accepted, naming convention v1, pre-commit style hook -- **Phase 1**: 8 smoke tests + 7 production-config tests, Flask-SQLAlchemy 3 fixtures, uv lockfile, hardened ProductionConfig -- **Phase 2**: contract surface defined (`__contract_version__`, `BasePlugin` hooks, `docs/PLUGIN-HOOKS.md`), 51 contract tests -- **Phase 3**: manifest-first loader, fail-loud/isolate policy, contract-version range checking, auto-register core blueprints, `shopdb.api` namespace, `BasePlugin.get_setting/set_setting` helpers -- **Phase 4**: `flask plugin new ` CLI, scaffold templates, 14 canary tests, `docs/PLUGIN-QUICKSTART.md` -- **Phase 5**: ADRs moved to `docs/adr/`, Alembic baseline migration, per-site deploy artifacts (`Dockerfile`, `docker-compose.yml`, `docs/DEPLOY.md`) - -### Active state - -- 1159 tests, naming/style check green, Gitea Actions CI (backend + naming + frontend build + a lean-build job + a migrations-mysql job that runs the real fresh upgrade on utf8mb4 MySQL 8) -- GE-Enforce HTTPS cutover: the displays/kiosks cohort now fetches manifest + inline payloads entirely over HTTPS (share-less); the `gea-shopfloor-display` scope is authored in code (`plugins/geenforce/seed_display_scope.py`) and published via `seed_display_scope(publish=True)`. Other fleet PC types still enforce from the SMB share and only report. See `docs/geenforce-api-cutover.md`. -- `__contract_version__` at 0.16.0 (0.12.0 mailer, 0.13.0 User/Role, 0.14.0 send_webhook, 0.15.0 authorized_service_token, 0.16.0 get_settings_defaults) (product `__version__` 0.7.0, tags v0.5.0/v0.6.0/v0.7.0 - distinct series, ADR-007) -- 13 bundled plugins all satisfy contract: computers, employees, geenforce, knowledgebase, machines, measuringtools, network, notifications, printedparts, printers, slides, usb, warranty -- Core Alembic chain: baseline `68b3947ae14f` -> head `7d26_settings_description_text` (33 core migrations). Each plugin owns its own chain (ADR-008); deploy runs `flask db upgrade` then `flask plugin upgrade-all`. Reproducible + idempotent from empty (env.py relaxes session sql_mode so the chain runs on strict MySQL 8). -- Lean per-site builds (ADR-013 + ADR-014): `scripts/build-site.sh` (backend) + `SITE_PLUGINS` via `scripts/stage-frontend.mjs` (frontend) ship only chosen plugins; `flask plugin prune-schema` drops non-installed plugins' tables at provisioning. Sidebar nav / settings / Displays all gate on staged routes. Manifest-less `plugins//frontend/` dirs (e.g. `applications`) are core and always ship. -- Windows sites install from a single air-gapped installer `.exe` built per site from its plugin profile (`deploy/windows/installer/`, built by `build-installer.sh` or `build-installer.ps1`). Operator docs: `docs/INSTALL-WINDOWS.md` + `docs/OPERATE-WINDOWS.md` - these are canonical for a NEW site. `docs/INSTALL-WINDOWS-IIS.md` and `docs/DEPLOY-WINDOWS-IIS.md` are the MANUAL procedure, kept for hand-built servers only. The installer verifies its third-party payload against `bundle-lock.json` and installs wheels with `pip --require-hashes`; every build stages a CycloneDX SBOM (`sbom.cdx.json`) onto the server. -- Legacy import: `docs/IMPORT-API.md` is the schema-agnostic import contract; `docs/IMPORT-ADOPTION.md` + `docs/PILOT-DEPLOY.md` cover adopting a site; `scripts/site_imports/wjf/` is the West Jefferson reference loader (all 16 stages, validated end-to-end including on a Windows + MySQL 8 VM). -- API is migration-complete: an admin PAT + docs/IMPORT-API.md let a script import the whole legacy DB (X-Import-Mode preserves timestamps). -- Pre-1.0 framework; sister sites should pin tight `core_version` ranges until contract reaches 1.0 - -### Deferred - -- Equipment data migration (one-shot script for legacy ASP shopdb -> assets). Per ADR-001, only `category='Equipment' AND machinenumber IS NOT NULL` migrates. Skill `migrating-asset-schema` documents the pattern; the actual one-shot script lives in `scripts/migration/` when run. -- Printers retirement: legacy `PrinterData` model + frontend changes. Coordinated with the equipment data migration. -- (DONE 2026-07-11) `measuringtools` plugin (ADR-005) is built and bundled; docs/PLUGIN-GUIDE.md narrates its construction as the plugin tutorial. -- (DONE) Frontend plugin hook contract (ADR-010): get_settings_cards / get_asset_panels / get_map_overlays / get_asset_presentation shipped; generic renderers for panels/overlays land incrementally. -- (DONE) Per-plugin Alembic chains (ADR-008): every bundled plugin carries its own chain; no plugin uses db.create_all(). -- Legacy ASP data import against the renamed schema (unblocked; run via docs/IMPORT-API.md) + a production pilot deployment. - -## Quick start - -```bash -# Start dev environment -~/start-dev-env.sh - -# Activate venv and install deps -cd /home/camp/projects/shopdb-flask -source venv/bin/activate -pip install -r requirements.txt - -# Configure environment -cp .env.example .env -# Edit .env with DB credentials, JWT secrets - -# Create / update database tables via the Alembic chain -flask db upgrade - -# Seed RBAC, default settings, and reference data (all idempotent) -flask seed permissions -flask seed settings -flask seed reference-data - -# Restart services -pm2 restart shopdb-flask-api shopdb-flask-ui -``` - -## Service URLs - -- Flask API: http://localhost:5001 -- Flask UI: http://localhost:5173 -- Legacy ASP (data source for one-time import): http://192.168.122.151:8080 +`CONTRIBUTING.md` is authoritative: lowercase concatenated DB columns, snake_case +pure-code identifiers, a closed acronym list, banned shorthand. The pre-commit +hook `scripts/check-naming-and-style.sh` enforces what it can. ## Plugin structure ``` -plugins/ - / - __init__.py - plugin.py # BasePlugin implementation - manifest.json # Plugin metadata (name, version, dependencies, api_prefix) - models/ - __init__.py # Export all models - .py # SQLAlchemy models - api/ - __init__.py - routes.py # Flask Blueprint - services/ # Optional, business logic - schemas/ # Optional, marshmallow schemas - migrations/ # Optional, plugin-specific Alembic migrations +plugins// + manifest.json # metadata, single source of truth (ADR-002) + plugin.py # BasePlugin implementation + models/ # __init__.py exports every model + api/routes.py # Flask blueprint + migrations/ # this plugin's own Alembic chain + services/ schemas/ frontend/ client/ # as needed ``` -Each plugin must have: - -- `models/__init__.py` exports all models -- `plugin.py` extends `BasePlugin` -- `manifest.json` with metadata (single source of truth per ADR-002) -- No direct imports from core code (use the contract surface defined in ADR-001) +No plugin imports core directly - everything goes through the `shopdb.api` +contract surface. A manifest-less directory under `plugins/` is core frontend +surface and always ships. ## Key files -- `shopdb/__init__.py` - app factory, blueprint registration -- `shopdb/plugins/base.py` - BasePlugin ABC, PluginMeta dataclass -- `shopdb/plugins/loader.py` - filesystem discovery, dependency-aware loading -- `shopdb/core/api/assets.py` - example of optional plugin imports -- `frontend/src/router/index.js` - frontend routing -- `frontend/src/components/AppSidebar.vue` - navigation menu -- `docs/adr/` - architecture decision records +- `shopdb/__init__.py` - app factory, version constants +- `shopdb/plugins/base.py` - BasePlugin, PluginMeta +- `shopdb/plugins/loader.py` - discovery and dependency-aware loading +- `shopdb/plugins/alembic_template.py` - the env every plugin chain runs through +- `frontend/src/router/index.js`, `frontend/src/components/AppSidebar.vue` +- `tools/export-github.sh` - scrub and publication gate -## Migration notes +## State -- `migrations/DATA_MIGRATION_GUIDE.md` - one-time import from legacy ASP shopdb -- `migrations/MIGRATE_USB_DEVICES_FROM_EQUIPMENT.md` - USB device migration from equipment table -- `migrations/FIX_LOCATIONONLY_EQUIPMENT_TYPES.md` - LocationOnly equipment type fix -- `migrations/PRODUCTION_MIGRATION_GUIDE.md` - production import methods -- `migrations/rename_underscore_columns.sql` - one-time rename of snake_case columns to lowercase concatenated (per CONTRIBUTING.md) -- `migrations/versions/` - the core Alembic chain (baseline `68b3947ae14f` -> head `7d26_settings_description_text`). Run `flask db upgrade` to apply. +Refactor phases 0-5 landed; phase 6 (multi-site distribution readiness) largely +complete. The last milestone before 1.0 is unchanged and still open: the +legacy-data import plus a production pilot (`scripts/site_imports/wjf/` is the +reference loader, validated end to end). + +Pre-1.0 framework. Sister sites pin tight `core_version` ranges until the +contract reaches 1.0. + +### Deferred + +- Equipment data migration (legacy ASP -> assets). Per ADR-001 only + `category='Equipment' AND machinenumber IS NOT NULL` migrates. +- Printers retirement: legacy `PrinterData` model, coordinated with the above. + +## Quick start + +```bash +source venv/bin/activate +pip install -r requirements.txt +cp .env.example .env # DB credentials, JWT secrets + +flask db upgrade # core chain +flask plugin upgrade-all # every plugin chain +flask seed permissions && flask seed settings && flask seed reference-data + +pm2 restart shopdb-flask-api shopdb-flask-ui +``` + +Dev API on :5001, dev UI on :5173. `venv/bin/python tools/shot.py /route` logs +in and screenshots a page. diff --git a/docs/PROJECT-MAP.md b/docs/PROJECT-MAP.md new file mode 100644 index 0000000..89f57cb --- /dev/null +++ b/docs/PROJECT-MAP.md @@ -0,0 +1,89 @@ + + +# Project map + +Derived from the code on every regeneration. If something here is wrong, the +code changed and the map was not regenerated - fix it by running the generator, +never by editing this file. + +## Versions + +| series | value | governed by | +|---|---|---| +| product `__version__` | `0.9.0` | ADR-007 | +| plugin contract `__contract_version__` | `0.19.0` | ADR-002 | + +They move independently. A contract bump is not a release. + +## Alembic chains (ADR-008) + +Core runs with `flask db upgrade`; every plugin chain runs +with `flask plugin upgrade-all`. Both are needed on a deploy. + +| chain | head | +|---|---| +| core | `7d32_displayrole_kiosk_vocabulary` | +| backups | `backups0003clearlastseen` | +| computers | `computers0001anchor` | +| employees | `employees0002photo` | +| geenforce | `geenforce0004minlib` | +| knowledgebase | `knowledgebase0001anchor` | +| machines | `machines0002rename` | +| measuringtools | `measuringtools0001baseline` | +| network | `network0003prefix` | +| notifications | `notifications0005boardorder` | +| printedparts | `printedparts0004txnrev` | +| printers | `printers0002supplyalerts` | +| slides | `slides0001anchor` | +| usb | `usb0002dropmachineid` | +| warranty | `warranty0002proof` | + +## Bundled plugins (15) + +| plugin | version | core_version | owns migrations | +|---|---|---|---| +| backups | 1.0.0 | >=0.16.0,<1.0.0 | yes | +| computers | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| employees | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| geenforce | 0.1.0 | >=0.18.0,<1.0.0 | yes | +| knowledgebase | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| machines | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| measuringtools | 1.0.0 | >=0.6.0,<1.0.0 | yes | +| network | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| notifications | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| printedparts | 0.1.0 | >=0.16.0,<1.0.0 | yes | +| printers | 1.0.0 | >=0.16.0,<1.0.0 | yes | +| slides | 2.0.0 | >=0.2.0,<1.0.0 | yes | +| tools | 1.0.0 | >=0.16.0,<1.0.0 | no | +| usb | 1.0.0 | >=0.1.0,<1.0.0 | yes | +| warranty | 1.0.0 | >=0.2.0,<1.0.0 | yes | + +Manifest-less directories under `plugins/` are core frontend surface and always ship: `applications`. + +## Architecture decisions + +| ADR | title | status | +|---|---|---| +| ADR-001-asset-as-platform-contract.md | ADR-001: Asset model is the platform contract | ACCEPTED | +| ADR-002-plugin-versioning.md | ADR-002: Plugin contract versioning | ACCEPTED | +| ADR-003-plugin-distribution.md | ADR-003: Plugin distribution model | ACCEPTED | +| ADR-004-deployment-topology.md | ADR-004: Deployment topology (per-site instances) | ACCEPTED | +| ADR-005-equipment-vs-measuringtools.md | ADR-005: Equipment plugin scope vs measuringtools plugin | ACCEPTED | +| ADR-006-collector-contract.md | ADR-006: Plugin collector contract pattern | ACCEPTED | +| ADR-007-product-versioning-and-releases.md | ADR-007: Product versioning and releases | ACCEPTED | +| ADR-008-plugin-migration-ownership.md | ADR-008: Plugin migration ownership (per-plugin chains from the cutover) | ACCEPTED | +| ADR-009-frontend-plugin-gating.md | ADR-009: Frontend plugin gating | ACCEPTED | +| ADR-010-frontend-plugin-hooks.md | ADR-010: Frontend plugin hook contract | ACCEPTED | +| ADR-011-machines-rename.md | ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes | ACCEPTED | +| ADR-012-geenforce-manifest-ownership.md | ADR-012: GE-Enforce manifest ownership in shopdb | ACCEPTED | +| ADR-013-plugin-catalog-and-lean-builds.md | ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds | PROPOSED | +| ADR-014-schema-lean-per-site.md | ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables) | ACCEPTED | +| ADR-015-site-specific-configuration.md | ADR-015: Where a site's own data is allowed to live | ACCEPTED | +| ADR-016-credential-delivery.md | ADR-016: Credential delivery to the fleet | ACCEPTED | + +## Size + +- test functions defined: **1036** (parametrised cases collect higher) +- documented API paths: **245** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`) + diff --git a/scripts/gen_project_map.py b/scripts/gen_project_map.py new file mode 100644 index 0000000..6bda25f --- /dev/null +++ b/scripts/gen_project_map.py @@ -0,0 +1,244 @@ +#!/usr/bin/env python3 +"""Generate docs/PROJECT-MAP.md: the facts about this repo that go stale. + +Versions, the plugin inventory, every Alembic chain head, the ADR index and the +endpoint count were all written by hand in CLAUDE.md and in the docs. Written +facts about a moving codebase are wrong within a fortnight - CLAUDE.md claimed +contract 0.16.0 and 1159 tests while the code was at 0.18.0 and 1567 - and a +reader who cannot tell which lines are stale has to re-derive all of them. + +So they are derived here instead, from the code, and CLAUDE.md points at the +result. Anything in this file that cannot be read from the repository does not +belong in it. + +Usage: + venv/bin/python scripts/gen_project_map.py # write docs/PROJECT-MAP.md + venv/bin/python scripts/gen_project_map.py --check # fail if it is out of date + venv/bin/python scripts/gen_project_map.py --stdout # print, write nothing + +--check is the CI form: it regenerates in memory and compares, so a plugin added +without regenerating the map fails the build instead of quietly aging. +""" + +import argparse +import json +import re +import sys +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +OUTPUT = REPO / 'docs' / 'PROJECT-MAP.md' + +HEADER = """ + +# Project map + +Derived from the code on every regeneration. If something here is wrong, the +code changed and the map was not regenerated - fix it by running the generator, +never by editing this file. +""" + + +def _read(path): + return path.read_text(encoding='utf-8', errors='replace') + + +def versions(): + """Product and plugin-contract versions, which are separate series (ADR-007, + ADR-002) and are routinely quoted as if they were one.""" + text = _read(REPO / 'shopdb' / '__init__.py') + found = {} + for name in ('__version__', '__contract_version__'): + match = re.search(r"^%s\s*=\s*['\"]([^'\"]+)['\"]" % name, text, re.M) + found[name] = match.group(1) if match else '?' + return found + + +def chainhead(versionsdir): + """Head revision of an Alembic chain: the revision nothing else revises. + + Read from the files rather than from a database, so the map is generatable + on a checkout with no MySQL and cannot report what one particular server + happens to have applied. + """ + if not versionsdir.is_dir(): + return None + revisions, parents = set(), set() + for path in sorted(versionsdir.glob('*.py')): + text = _read(path) + rev = re.search(r"^revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M) + down = re.search(r"^down_revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M) + if rev: + revisions.add(rev.group(1)) + if down: + parents.add(down.group(1)) + heads = sorted(revisions - parents) + if not heads: + return None + # More than one head means the chain has branched, which breaks `upgrade`. + # Reporting it is the point; hiding it behind a [0] is how it stays unnoticed. + return ' + '.join(heads) + + +def plugins(): + rows = [] + for manifest in sorted((REPO / 'plugins').glob('*/manifest.json')): + try: + data = json.loads(_read(manifest)) + except ValueError as exc: + rows.append({'name': manifest.parent.name, 'version': 'UNREADABLE', + 'core': str(exc)[:60], 'head': None}) + continue + rows.append({ + 'name': data.get('name', manifest.parent.name), + 'version': data.get('version', '?'), + 'core': data.get('core_version', ''), + 'head': chainhead(manifest.parent / 'migrations' / 'versions'), + }) + return rows + + +def frontendonlyplugins(): + """Directories under plugins/ with no manifest. They are core frontend + surface that always ships (applications, for one), and they look like + missing plugins to anyone counting directories.""" + names = [] + for path in sorted((REPO / 'plugins').iterdir()): + # Dot-directories are build scratch (plugins/.staging), not surface. + if path.is_dir() and not (path / 'manifest.json').exists() \ + and not path.name.startswith(('__', '.')): + names.append(path.name) + return names + + +def adrs(): + rows = [] + for path in sorted((REPO / 'docs' / 'adr').glob('ADR-*.md')): + text = _read(path) + title = next((line.lstrip('# ').strip() + for line in text.splitlines() if line.startswith('# ')), + path.stem) + # Both spellings are in use: '- Status: ACCEPTED' and + # '- **Status:** ACCEPTED'. Matching only one reported every ADR as '?', + # which reads as "nobody decided" rather than "the regex is wrong". + status = re.search(r'^[-*\s]*(?:\*\*)?Status(?:\*\*)?\s*:?\s*(?:\*\*)?\s*(.+)$', + text, re.M | re.I) + clean = re.sub(r'[*_`]', '', status.group(1)).strip() if status else '?' + rows.append({'file': path.name, 'title': title, 'status': clean[:40]}) + return rows + + +def endpointcount(): + """From the generated spec, not from a live app: this script must stay + importable without a database or an app context.""" + spec = REPO / 'docs' / 'openapi.json' + if not spec.is_file(): + return None + try: + return len(json.loads(_read(spec)).get('paths', {})) + except ValueError: + return None + + +def testcount(): + """Test FUNCTIONS, counted by reading the files. + + Deliberately not `pytest --collect-only`: that needs the venv and the whole + import graph, and a map generator that only runs in a fully installed + environment is one that stops being run. Parametrised cases make the real + collected number higher; this is a floor, and it is labelled as one. + """ + total = 0 + for path in (REPO / 'tests').rglob('test_*.py'): + total += len(re.findall(r'^\s*def test_', _read(path), re.M)) + return total + + +def render(): + lines = [HEADER] + version = versions() + + lines.append('## Versions\n') + lines.append('| series | value | governed by |') + lines.append('|---|---|---|') + lines.append('| product `__version__` | `%s` | ADR-007 |' + % version['__version__']) + lines.append('| plugin contract `__contract_version__` | `%s` | ADR-002 |' + % version['__contract_version__']) + lines.append('') + lines.append('They move independently. A contract bump is not a release.\n') + + lines.append('## Alembic chains (ADR-008)\n') + lines.append('Core runs with `flask db upgrade`; every plugin chain runs') + lines.append('with `flask plugin upgrade-all`. Both are needed on a deploy.\n') + lines.append('| chain | head |') + lines.append('|---|---|') + lines.append('| core | `%s` |' % (chainhead(REPO / 'migrations' / 'versions') + or 'none')) + rows = plugins() + for row in rows: + if row['head']: + lines.append('| %s | `%s` |' % (row['name'], row['head'])) + lines.append('') + + lines.append('## Bundled plugins (%d)\n' % len(rows)) + lines.append('| plugin | version | core_version | owns migrations |') + lines.append('|---|---|---|---|') + for row in rows: + lines.append('| %s | %s | %s | %s |' % ( + row['name'], row['version'], row['core'] or '-', + 'yes' if row['head'] else 'no')) + lines.append('') + extra = frontendonlyplugins() + if extra: + lines.append('Manifest-less directories under `plugins/` are core ' + 'frontend surface and always ship: %s.\n' + % ', '.join('`%s`' % name for name in extra)) + + lines.append('## Architecture decisions\n') + lines.append('| ADR | title | status |') + lines.append('|---|---|---|') + for row in adrs(): + lines.append('| %s | %s | %s |' % (row['file'], row['title'], row['status'])) + lines.append('') + + lines.append('## Size\n') + endpoints = endpointcount() + lines.append('- test functions defined: **%d** (parametrised cases collect ' + 'higher)' % testcount()) + lines.append('- documented API paths: **%s** (`docs/openapi.json`, ' + 'regenerate with `scripts/gen_openapi.py`)' + % ('%d' % endpoints if endpoints is not None else 'unknown')) + lines.append('') + return '\n'.join(lines) + '\n' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--check', action='store_true', + help='exit 1 if the committed map is out of date') + parser.add_argument('--stdout', action='store_true', + help='print the map instead of writing it') + args = parser.parse_args() + + content = render() + if args.stdout: + sys.stdout.write(content) + return 0 + if args.check: + current = _read(OUTPUT) if OUTPUT.is_file() else '' + if current != content: + sys.stderr.write( + 'docs/PROJECT-MAP.md is out of date.\n' + 'Regenerate: venv/bin/python scripts/gen_project_map.py\n') + return 1 + print('docs/PROJECT-MAP.md is current') + return 0 + OUTPUT.write_text(content, encoding='utf-8') + print('wrote %s' % OUTPUT.relative_to(REPO)) + return 0 + + +if __name__ == '__main__': + raise SystemExit(main())