ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Generate the facts that go stale, and record the fortnight in the changelog

Browse Source 
CLAUDE.md is read at the start of every session and claimed contract 0.16.0 and
1159 tests while the code was at 0.18.0 and 1567, along with a plugin count and
a chain head that were both wrong. Hand-written facts about a moving codebase
are wrong within a fortnight, and a reader who cannot tell which lines are stale
has to re-derive all of them.

So they are derived. scripts/gen_project_map.py reads the versions, the plugin
inventory, every Alembic chain head, the ADR index with status and the size of
the codebase out of the code, and --check fails a build where the committed map
no longer matches. CLAUDE.md now points at it, leads with the multi-site rule,
and lists the gates to run instead of restating the conventions they enforce.

The changelog's Unreleased section covered 8 of the 46 commits since 0.9.0 and
had no Changed, Fixed or Security sections at all. It now carries the whole
fortnight, including both contract bumps - which had never been recorded even
though ADR-002 makes contract versions their own series.
This commit is contained in:
cproudlock
2026-08-14 13:47:48 -04:00
parent 035419fa51
commit 8f9805d964
4 changed files with 543 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
CHANGELOG.md
View File

@@ -66,6 +66,106 @@ ADR-007 and ADR-002.
components nobody wrote. Cards are ordered by severity, hide themselves when
there is nothing to report, gate on a permission, and each fetches
independently so one broken endpoint cannot blank the board.
- Tech Tools: a section for the small conversions and code generation that
otherwise happen in a browser tab on somebody's phone. Starts with barcode and
QR codes laid out for the label stock actually in the printers.
- Kiosks report what they are. The dispatcher already read
`C:\Enrollment\display-type.txt` to pick its target; now it tells ShopDB, so
the Asset column on an enforcement report shows a reported value instead of
one inferred from a mapping table nobody filled in.
- Enforcement reports carry what the PC is, where it is, and whether it is
backed up: the host links to its own page, the Asset column names the machine
or instrument the PC controls, a map pin shows the location on hover, and the
backup column judges the chain rather than printing its last date.
- `flask relationships audit`, read-only, listing links that cannot both be
true, plus an inverse guard that refuses to create one.
- An employees endpoint that says WHY a name did not resolve, rather than
returning nothing and leaving the caller to guess.
- The display client module updates itself through the manifest, so a client
change no longer means hands on every kiosk.
- `docs/PROJECT-MAP.md`, generated by `scripts/gen_project_map.py`: versions,
the plugin inventory, every Alembic chain head, the ADR index and the size of
the codebase, derived from the code. The hand-written equivalents in CLAUDE.md
had drifted by two contract releases and 400 tests.
### Changed
- Plugin contract **0.19.0**: `get_dashboard_widgets` returns data and shape,
not a component name. The old shape named a Vue component per widget, which
cannot survive a lean build (ADR-013) and had five plugins pointing at
components nobody wrote. The change itself shipped earlier without a bump and
with `BasePlugin` and PLUGIN-HOOKS.md still teaching the dead shape; this
records it and corrects both.
- Plugin contract **0.17.0**: `DashboardDefault` is part of the `shopdb.api`
surface, so a plugin can resolve a display without reaching into core.
- Plugin contract **0.18.0**: one name per display role, and it is the kiosk's
own (`Dashboard`, `Lobby`, `3DPrintRoom`). Server and dispatcher had three
vocabularies for the same three kiosks, and a display could report a role core
could not store. `normalize_display_role` resolves case and the retired
spelling; an unknown value is kept verbatim so a typo stays visible.
Contract versions are their own series (ADR-002) and belong in this file;
0.17.0 and 0.18.0 shipped without an entry, which is what this note corrects.
- Publishing a GE-Enforce manifest is refused when PCs reporting for that scope
run an older client library than the manifest needs, naming the hosts. A minor
version bump that NARROWS behaviour is not backward compatible: an old library
reads a gated entry as ungated and installs every version it cannot detect.
- Shadow mode runs inside the enforce cycle instead of on its own schedule. The
share is mounted only for the length of a cycle, so a task on its own clock
woke to a missing drive and reported 0 installed / 0 skipped / 0 failed, which
is indistinguishable from a healthy no-op.
- A GE-Enforce report is judged stale on both clocks, the server's and the
client's asserted check-in, so a PC cannot mask real silence with a future
timestamp. A PC that has gone quiet stops reading as healthy.
### Fixed
- **Every plugin migration chain re-ran its head on each MySQL deploy.** Core's
Alembic env got a `connection.commit()`; the per-plugin template did not, so
the last migration of every run lost its version stamp while its DDL survived.
Invisible for as long as every head happened to be idempotent - and two were
not. The backups head cleared a column for every row, and the geenforce head
added a column unconditionally and failed on the second attempt. Both are now
bounded and guarded, and the template commits.
- A part-marker PC reported 200 on the cycle that created its marker and 500 on
every cycle after. Both of the links it makes were looked up by label, so a row
made by hand or by the legacy import was invisible and the insert violated the
uniqueness of (source, target, type); and the marker's asset number is derived
from the PC, so it could already be taken.
- A CMM reported the same way: the instrument IS the reported bay, so adoption
was relabelling the row the machine sync owns and finding it by that label.
- A reported identifier is matched exactly rather than with `ilike`, so `_` and
`%` in a file on a shopfloor PC are no longer wildcards. `MT-600_` had been
adopting MT-6001, and a bare `%` adopted whatever active asset came first.
- Measuring tools are adopted before they are minted. Minting derived the asset
number from the HOSTNAME, so a permanent instrument took the identity of
whichever PC drove it that week, and the collector could not see a tool it had
not created - which left legacy instruments shadowed by minted twins.
- An unanchored rsync exclude in the export was matching at any depth, so a
whole plugin was silently dropped from the published payload. Root excludes
are anchored and the export now aborts if a plugin manifest fails to survive.
- Backup revisions are ordered in Python, not with `ORDER BY ... NULLS LAST`,
which SQLite accepts and MySQL rejects.
### Security
- The Windows installer breaks ACL inheritance on the roots it creates. A
directory made directly under `C:\` inherits an inherit-only Modify grant for
Authenticated Users, so `C:\shopdb-flask`, `C:\Python314` and `C:\MySQL84`
were writable by any logged-on user. Two files inside them make that a local
privilege escalation: `shopdb-admin.ps1`, which self-elevates, and
`instance/config.py`, which the app executes unconditionally. The MySQL data
directory is locked down as well, since it holds the password hashes. A Python
installation this installer did not create is reported rather than seized.
- Site-specific values no longer ship in product code (ADR-015). The backups
share root defaults to blank instead of one site's file server, the shadow
client no longer overwrites a site's own ShopDB URL in the registry on every
cycle, the kiosk dispatcher takes its URL from the `site_base_url` setting and
refuses rather than guessing, and one site's hostname, FQDN and internal
subnets are out of the documentation, the generated API specs and the UI
placeholders. The ADR-015 scanner now covers PowerShell, the installer, the
seeds and generated JSON, case-insensitively, and FAILS the build; deliberate
organisation-wide defaults are declared with an `ADR-015-OK` marker. Both
publication gates carry the site patterns.
## [0.9.0] - 2026-08-11

234
CLAUDE.md
View File

@@ -1,145 +1,131 @@
# ShopDB Flask Project
Modern rewrite of the classic-ASP shopdb. Built as a framework so sister GE Aerospace sites can adopt it. Plugin system is the product.
Modern rewrite of the classic-ASP shopdb. Built as a framework so sister GE
Aerospace sites can adopt it. The plugin system is the product.
**This site is not the only site.** Anything that names West Jefferson - a
hostname, a share path, a subnet, a machine-number format, a label prefix - is a
defect unless it is a setting with a neutral default, a site-namespaced
directory, or seed data. That is ADR-015, and it is the rule most often broken
in the places tooling cannot see: PowerShell, the installer, JSON, the imaging
share.
## Facts that move live in the map
`docs/PROJECT-MAP.md` is GENERATED. It carries the product and contract
versions, the plugin inventory, every Alembic chain head, the ADR index with
status, and the size of the codebase. Read it instead of asking; regenerate it
rather than correcting it by hand:
```bash
venv/bin/python scripts/gen_project_map.py # write
venv/bin/python scripts/gen_project_map.py --check # CI form
```
Hand-written version numbers in this file were wrong by two contract releases
and 400 tests before the map existed. Nothing that the generator can derive
should be typed here again.
## Where to look
| question | read |
|---|---|
| what a name may be called | `CONTRIBUTING.md` |
| why the architecture is like that | `docs/adr/` (index in the map) |
| what a plugin may touch | `docs/PLUGIN-HOOKS.md` |
| what the collector sends | `docs/COLLECTOR-INTEGRATION.md`, ADR-006 |
| how a site is deployed | `docs/INSTALL-WINDOWS.md`, `docs/OPERATE-WINDOWS.md` |
| how code reaches GitHub | `tools/export-github.sh` |
## Standards are gates, not prose
Compliance is checked, not remembered. Run these rather than re-reading the
rules:
```bash
bash scripts/check-naming-and-style.sh # naming, style, site literals (ADR-015)
venv/bin/python -m pytest -q # includes the docs publishability gate
venv/bin/python -m pytest tests/test_plugin_migrations.py # chain heads registry
venv/bin/python scripts/gen_project_map.py --check
```
A new plugin table means updating `PLUGIN_TABLE_OWNERS` and
`EXPECTED_HEAD_REVISION` in `tests/test_plugin_migrations.py`.
## Database
- **Active database:** `shopdb_flask` (MySQL, asset-based schema)
- **Legacy database:** `shopdb` (Classic ASP schema, used only for one-time data import via `scripts/import_from_mysql.py`)
- **Connection:** `.env` file. See `.env.example`.
- **Active:** `shopdb_flask` (MySQL, asset-based schema)
- **Legacy:** `shopdb` (classic ASP schema; one-time import only)
- **Connection:** `.env`, see `.env.example`
Architecture decisions live in `docs/adr/`. Read those before making schema or contract changes.
Core migrations run with `flask db upgrade`; every plugin owns its own chain
(ADR-008) and runs with `flask plugin upgrade-all`. **A deploy needs both** -
skipping the second is how a new column reaches production as a 1054 error.
- ADR-001: Asset model is the platform contract (Machine retires) - ACCEPTED
- ADR-002: Plugin contract versioning (semver) - ACCEPTED
- ADR-003: Plugin distribution model (in-tree bundled + filesystem-based external) - ACCEPTED
- ADR-004: Deployment topology (per-site instances, not multi-tenant) - ACCEPTED
- ADR-005: Equipment vs measuringtools plugin scope - ACCEPTED
- ADR-006: Plugin collector contract pattern - ACCEPTED
- ADR-007: Product versioning and releases - ACCEPTED
- ADR-008: Plugin migration ownership (per-plugin chains) - ACCEPTED
- ADR-009: Frontend plugin route gating - ACCEPTED
- ADR-010: Frontend plugin hook contract - ACCEPTED
- ADR-011: Machines rename + modeltypes retyping - ACCEPTED
- ADR-012: GE-Enforce manifest ownership in shopdb - ACCEPTED
- ADR-013: Plugin catalog, curated shelf, and lean per-site builds - PROPOSED
- ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, prune not-installed plugin tables) - ACCEPTED
- ADR-015: Where a site's own data is allowed to live (setting with a neutral default, site-namespaced dir, or seed) - ACCEPTED
- ADR-016: Credential delivery to the fleet (ShopDB resolves targeting, encrypted at rest, dedicated fetch scope) - ACCEPTED
Tests run on SQLite, production is MySQL. Dialect drift is not caught by the
suite: `ORDER BY ... NULLS LAST` parses on SQLite and is rejected by MySQL. Sort
in Python when in doubt.
## Coding convention
`CONTRIBUTING.md` defines naming rules (DB tables, columns, Python, JS, Vue, API). Pre-commit hook at `scripts/check-naming-and-style.sh` enforces them. Read `CONTRIBUTING.md` before naming any new identifier.
## Current state (as of 2026-07-13)
Refactor phases 0-5 landed; phase 6 (multi-site distribution readiness) largely complete; the last big milestone is the legacy-data import + a production pilot.
### Phases done
- **Phase 0**: 6 ADRs accepted, naming convention v1, pre-commit style hook
- **Phase 1**: 8 smoke tests + 7 production-config tests, Flask-SQLAlchemy 3 fixtures, uv lockfile, hardened ProductionConfig
- **Phase 2**: contract surface defined (`__contract_version__`, `BasePlugin` hooks, `docs/PLUGIN-HOOKS.md`), 51 contract tests
- **Phase 3**: manifest-first loader, fail-loud/isolate policy, contract-version range checking, auto-register core blueprints, `shopdb.api` namespace, `BasePlugin.get_setting/set_setting` helpers
- **Phase 4**: `flask plugin new <name>` CLI, scaffold templates, 14 canary tests, `docs/PLUGIN-QUICKSTART.md`
- **Phase 5**: ADRs moved to `docs/adr/`, Alembic baseline migration, per-site deploy artifacts (`Dockerfile`, `docker-compose.yml`, `docs/DEPLOY.md`)
### Active state
- 1159 tests, naming/style check green, Gitea Actions CI (backend + naming + frontend build + a lean-build job + a migrations-mysql job that runs the real fresh upgrade on utf8mb4 MySQL 8)
- GE-Enforce HTTPS cutover: the displays/kiosks cohort now fetches manifest + inline payloads entirely over HTTPS (share-less); the `gea-shopfloor-display` scope is authored in code (`plugins/geenforce/seed_display_scope.py`) and published via `seed_display_scope(publish=True)`. Other fleet PC types still enforce from the SMB share and only report. See `docs/geenforce-api-cutover.md`.
- `__contract_version__` at 0.16.0 (0.12.0 mailer, 0.13.0 User/Role, 0.14.0 send_webhook, 0.15.0 authorized_service_token, 0.16.0 get_settings_defaults) (product `__version__` 0.7.0, tags v0.5.0/v0.6.0/v0.7.0 - distinct series, ADR-007)
- 13 bundled plugins all satisfy contract: computers, employees, geenforce, knowledgebase, machines, measuringtools, network, notifications, printedparts, printers, slides, usb, warranty
- Core Alembic chain: baseline `68b3947ae14f` -> head `7d26_settings_description_text` (33 core migrations). Each plugin owns its own chain (ADR-008); deploy runs `flask db upgrade` then `flask plugin upgrade-all`. Reproducible + idempotent from empty (env.py relaxes session sql_mode so the chain runs on strict MySQL 8).
- Lean per-site builds (ADR-013 + ADR-014): `scripts/build-site.sh` (backend) + `SITE_PLUGINS` via `scripts/stage-frontend.mjs` (frontend) ship only chosen plugins; `flask plugin prune-schema` drops non-installed plugins' tables at provisioning. Sidebar nav / settings / Displays all gate on staged routes. Manifest-less `plugins/<name>/frontend/` dirs (e.g. `applications`) are core and always ship.
- Windows sites install from a single air-gapped installer `.exe` built per site from its plugin profile (`deploy/windows/installer/`, built by `build-installer.sh` or `build-installer.ps1`). Operator docs: `docs/INSTALL-WINDOWS.md` + `docs/OPERATE-WINDOWS.md` - these are canonical for a NEW site. `docs/INSTALL-WINDOWS-IIS.md` and `docs/DEPLOY-WINDOWS-IIS.md` are the MANUAL procedure, kept for hand-built servers only. The installer verifies its third-party payload against `bundle-lock.json` and installs wheels with `pip --require-hashes`; every build stages a CycloneDX SBOM (`sbom.cdx.json`) onto the server.
- Legacy import: `docs/IMPORT-API.md` is the schema-agnostic import contract; `docs/IMPORT-ADOPTION.md` + `docs/PILOT-DEPLOY.md` cover adopting a site; `scripts/site_imports/wjf/` is the West Jefferson reference loader (all 16 stages, validated end-to-end including on a Windows + MySQL 8 VM).
- API is migration-complete: an admin PAT + docs/IMPORT-API.md let a script import the whole legacy DB (X-Import-Mode preserves timestamps).
- Pre-1.0 framework; sister sites should pin tight `core_version` ranges until contract reaches 1.0
### Deferred
- Equipment data migration (one-shot script for legacy ASP shopdb -> assets). Per ADR-001, only `category='Equipment' AND machinenumber IS NOT NULL` migrates. Skill `migrating-asset-schema` documents the pattern; the actual one-shot script lives in `scripts/migration/` when run.
- Printers retirement: legacy `PrinterData` model + frontend changes. Coordinated with the equipment data migration.
- (DONE 2026-07-11) `measuringtools` plugin (ADR-005) is built and bundled; docs/PLUGIN-GUIDE.md narrates its construction as the plugin tutorial.
- (DONE) Frontend plugin hook contract (ADR-010): get_settings_cards / get_asset_panels / get_map_overlays / get_asset_presentation shipped; generic renderers for panels/overlays land incrementally.
- (DONE) Per-plugin Alembic chains (ADR-008): every bundled plugin carries its own chain; no plugin uses db.create_all().
- Legacy ASP data import against the renamed schema (unblocked; run via docs/IMPORT-API.md) + a production pilot deployment.
## Quick start
```bash
# Start dev environment
~/start-dev-env.sh
# Activate venv and install deps
cd /home/camp/projects/shopdb-flask
source venv/bin/activate
pip install -r requirements.txt
# Configure environment
cp .env.example .env
# Edit .env with DB credentials, JWT secrets
# Create / update database tables via the Alembic chain
flask db upgrade
# Seed RBAC, default settings, and reference data (all idempotent)
flask seed permissions
flask seed settings
flask seed reference-data
# Restart services
pm2 restart shopdb-flask-api shopdb-flask-ui
```
## Service URLs
- Flask API: http://localhost:5001
- Flask UI: http://localhost:5173
- Legacy ASP (data source for one-time import): http://192.168.122.151:8080
`CONTRIBUTING.md` is authoritative: lowercase concatenated DB columns, snake_case
pure-code identifiers, a closed acronym list, banned shorthand. The pre-commit
hook `scripts/check-naming-and-style.sh` enforces what it can.
## Plugin structure
```
plugins/
<plugin_name>/
__init__.py
plugin.py # BasePlugin implementation
manifest.json # Plugin metadata (name, version, dependencies, api_prefix)
models/
__init__.py # Export all models
<model>.py # SQLAlchemy models
api/
__init__.py
routes.py # Flask Blueprint
services/ # Optional, business logic
schemas/ # Optional, marshmallow schemas
migrations/ # Optional, plugin-specific Alembic migrations
plugins/<name>/
manifest.json # metadata, single source of truth (ADR-002)
plugin.py # BasePlugin implementation
models/ # __init__.py exports every model
api/routes.py # Flask blueprint
migrations/ # this plugin's own Alembic chain
services/ schemas/ frontend/ client/ # as needed
```
Each plugin must have:
- `models/__init__.py` exports all models
- `plugin.py` extends `BasePlugin`
- `manifest.json` with metadata (single source of truth per ADR-002)
- No direct imports from core code (use the contract surface defined in ADR-001)
No plugin imports core directly - everything goes through the `shopdb.api`
contract surface. A manifest-less directory under `plugins/` is core frontend
surface and always ships.
## Key files
- `shopdb/__init__.py` - app factory, blueprint registration
- `shopdb/plugins/base.py` - BasePlugin ABC, PluginMeta dataclass
- `shopdb/plugins/loader.py` - filesystem discovery, dependency-aware loading
- `shopdb/core/api/assets.py` - example of optional plugin imports
- `frontend/src/router/index.js` - frontend routing
- `frontend/src/components/AppSidebar.vue` - navigation menu
- `docs/adr/` - architecture decision records
- `shopdb/__init__.py` - app factory, version constants
- `shopdb/plugins/base.py` - BasePlugin, PluginMeta
- `shopdb/plugins/loader.py` - discovery and dependency-aware loading
- `shopdb/plugins/alembic_template.py` - the env every plugin chain runs through
- `frontend/src/router/index.js`, `frontend/src/components/AppSidebar.vue`
- `tools/export-github.sh` - scrub and publication gate
## Migration notes
## State
- `migrations/DATA_MIGRATION_GUIDE.md` - one-time import from legacy ASP shopdb
- `migrations/MIGRATE_USB_DEVICES_FROM_EQUIPMENT.md` - USB device migration from equipment table
- `migrations/FIX_LOCATIONONLY_EQUIPMENT_TYPES.md` - LocationOnly equipment type fix
- `migrations/PRODUCTION_MIGRATION_GUIDE.md` - production import methods
- `migrations/rename_underscore_columns.sql` - one-time rename of snake_case columns to lowercase concatenated (per CONTRIBUTING.md)
- `migrations/versions/` - the core Alembic chain (baseline `68b3947ae14f` -> head `7d26_settings_description_text`). Run `flask db upgrade` to apply.
Refactor phases 0-5 landed; phase 6 (multi-site distribution readiness) largely
complete. The last milestone before 1.0 is unchanged and still open: the
legacy-data import plus a production pilot (`scripts/site_imports/wjf/` is the
reference loader, validated end to end).
Pre-1.0 framework. Sister sites pin tight `core_version` ranges until the
contract reaches 1.0.
### Deferred
- Equipment data migration (legacy ASP -> assets). Per ADR-001 only
`category='Equipment' AND machinenumber IS NOT NULL` migrates.
- Printers retirement: legacy `PrinterData` model, coordinated with the above.
## Quick start
```bash
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # DB credentials, JWT secrets
flask db upgrade # core chain
flask plugin upgrade-all # every plugin chain
flask seed permissions && flask seed settings && flask seed reference-data
pm2 restart shopdb-flask-api shopdb-flask-ui
```
Dev API on :5001, dev UI on :5173. `venv/bin/python tools/shot.py /route` logs
in and screenshots a page.

89
docs/PROJECT-MAP.md Normal file
View File

@@ -0,0 +1,89 @@
<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
Regenerate with: venv/bin/python scripts/gen_project_map.py -->
# Project map
Derived from the code on every regeneration. If something here is wrong, the
code changed and the map was not regenerated - fix it by running the generator,
never by editing this file.
## Versions
| series | value | governed by |
|---|---|---|
| product `__version__` | `0.9.0` | ADR-007 |
| plugin contract `__contract_version__` | `0.19.0` | ADR-002 |
They move independently. A contract bump is not a release.
## Alembic chains (ADR-008)
Core runs with `flask db upgrade`; every plugin chain runs
with `flask plugin upgrade-all`. Both are needed on a deploy.
| chain | head |
|---|---|
| core | `7d32_displayrole_kiosk_vocabulary` |
| backups | `backups0003clearlastseen` |
| computers | `computers0001anchor` |
| employees | `employees0002photo` |
| geenforce | `geenforce0004minlib` |
| knowledgebase | `knowledgebase0001anchor` |
| machines | `machines0002rename` |
| measuringtools | `measuringtools0001baseline` |
| network | `network0003prefix` |
| notifications | `notifications0005boardorder` |
| printedparts | `printedparts0004txnrev` |
| printers | `printers0002supplyalerts` |
| slides | `slides0001anchor` |
| usb | `usb0002dropmachineid` |
| warranty | `warranty0002proof` |
## Bundled plugins (15)
| plugin | version | core_version | owns migrations |
|---|---|---|---|
| backups | 1.0.0 | >=0.16.0,<1.0.0 | yes |
| computers | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| employees | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| geenforce | 0.1.0 | >=0.18.0,<1.0.0 | yes |
| knowledgebase | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| machines | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| measuringtools | 1.0.0 | >=0.6.0,<1.0.0 | yes |
| network | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| notifications | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| printedparts | 0.1.0 | >=0.16.0,<1.0.0 | yes |
| printers | 1.0.0 | >=0.16.0,<1.0.0 | yes |
| slides | 2.0.0 | >=0.2.0,<1.0.0 | yes |
| tools | 1.0.0 | >=0.16.0,<1.0.0 | no |
| usb | 1.0.0 | >=0.1.0,<1.0.0 | yes |
| warranty | 1.0.0 | >=0.2.0,<1.0.0 | yes |
Manifest-less directories under `plugins/` are core frontend surface and always ship: `applications`.
## Architecture decisions
| ADR | title | status |
|---|---|---|
| ADR-001-asset-as-platform-contract.md | ADR-001: Asset model is the platform contract | ACCEPTED |
| ADR-002-plugin-versioning.md | ADR-002: Plugin contract versioning | ACCEPTED |
| ADR-003-plugin-distribution.md | ADR-003: Plugin distribution model | ACCEPTED |
| ADR-004-deployment-topology.md | ADR-004: Deployment topology (per-site instances) | ACCEPTED |
| ADR-005-equipment-vs-measuringtools.md | ADR-005: Equipment plugin scope vs measuringtools plugin | ACCEPTED |
| ADR-006-collector-contract.md | ADR-006: Plugin collector contract pattern | ACCEPTED |
| ADR-007-product-versioning-and-releases.md | ADR-007: Product versioning and releases | ACCEPTED |
| ADR-008-plugin-migration-ownership.md | ADR-008: Plugin migration ownership (per-plugin chains from the cutover) | ACCEPTED |
| ADR-009-frontend-plugin-gating.md | ADR-009: Frontend plugin gating | ACCEPTED |
| ADR-010-frontend-plugin-hooks.md | ADR-010: Frontend plugin hook contract | ACCEPTED |
| ADR-011-machines-rename.md | ADR-011: Rename the equipment domain to machines; retype the models catalog with modeltypes | ACCEPTED |
| ADR-012-geenforce-manifest-ownership.md | ADR-012: GE-Enforce manifest ownership in shopdb | ACCEPTED |
| ADR-013-plugin-catalog-and-lean-builds.md | ADR-013: Plugin Catalog, Curated Shelf, and Lean Per-Site Builds | PROPOSED |
| ADR-014-schema-lean-per-site.md | ADR-014: Schema-lean per-site builds (retire cross-plugin FKs, lift plugin tables) | ACCEPTED |
| ADR-015-site-specific-configuration.md | ADR-015: Where a site's own data is allowed to live | ACCEPTED |
| ADR-016-credential-delivery.md | ADR-016: Credential delivery to the fleet | ACCEPTED |
## Size
- test functions defined: **1036** (parametrised cases collect higher)
- documented API paths: **245** (`docs/openapi.json`, regenerate with `scripts/gen_openapi.py`)

244
scripts/gen_project_map.py Normal file
View File

@@ -0,0 +1,244 @@
#!/usr/bin/env python3
"""Generate docs/PROJECT-MAP.md: the facts about this repo that go stale.
Versions, the plugin inventory, every Alembic chain head, the ADR index and the
endpoint count were all written by hand in CLAUDE.md and in the docs. Written
facts about a moving codebase are wrong within a fortnight - CLAUDE.md claimed
contract 0.16.0 and 1159 tests while the code was at 0.18.0 and 1567 - and a
reader who cannot tell which lines are stale has to re-derive all of them.
So they are derived here instead, from the code, and CLAUDE.md points at the
result. Anything in this file that cannot be read from the repository does not
belong in it.
Usage:
venv/bin/python scripts/gen_project_map.py # write docs/PROJECT-MAP.md
venv/bin/python scripts/gen_project_map.py --check # fail if it is out of date
venv/bin/python scripts/gen_project_map.py --stdout # print, write nothing
--check is the CI form: it regenerates in memory and compares, so a plugin added
without regenerating the map fails the build instead of quietly aging.
"""
import argparse
import json
import re
import sys
from pathlib import Path
REPO = Path(__file__).resolve().parents[1]
OUTPUT = REPO / 'docs' / 'PROJECT-MAP.md'
HEADER = """<!-- GENERATED by scripts/gen_project_map.py - do not edit by hand.
Regenerate with: venv/bin/python scripts/gen_project_map.py -->
# Project map
Derived from the code on every regeneration. If something here is wrong, the
code changed and the map was not regenerated - fix it by running the generator,
never by editing this file.
"""
def _read(path):
return path.read_text(encoding='utf-8', errors='replace')
def versions():
"""Product and plugin-contract versions, which are separate series (ADR-007,
ADR-002) and are routinely quoted as if they were one."""
text = _read(REPO / 'shopdb' / '__init__.py')
found = {}
for name in ('__version__', '__contract_version__'):
match = re.search(r"^%s\s*=\s*['\"]([^'\"]+)['\"]" % name, text, re.M)
found[name] = match.group(1) if match else '?'
return found
def chainhead(versionsdir):
"""Head revision of an Alembic chain: the revision nothing else revises.
Read from the files rather than from a database, so the map is generatable
on a checkout with no MySQL and cannot report what one particular server
happens to have applied.
"""
if not versionsdir.is_dir():
return None
revisions, parents = set(), set()
for path in sorted(versionsdir.glob('*.py')):
text = _read(path)
rev = re.search(r"^revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M)
down = re.search(r"^down_revision\s*=\s*['\"]([^'\"]+)['\"]", text, re.M)
if rev:
revisions.add(rev.group(1))
if down:
parents.add(down.group(1))
heads = sorted(revisions - parents)
if not heads:
return None
# More than one head means the chain has branched, which breaks `upgrade`.
# Reporting it is the point; hiding it behind a [0] is how it stays unnoticed.
return ' + '.join(heads)
def plugins():
rows = []
for manifest in sorted((REPO / 'plugins').glob('*/manifest.json')):
try:
data = json.loads(_read(manifest))
except ValueError as exc:
rows.append({'name': manifest.parent.name, 'version': 'UNREADABLE',
'core': str(exc)[:60], 'head': None})
continue
rows.append({
'name': data.get('name', manifest.parent.name),
'version': data.get('version', '?'),
'core': data.get('core_version', ''),
'head': chainhead(manifest.parent / 'migrations' / 'versions'),
})
return rows
def frontendonlyplugins():
"""Directories under plugins/ with no manifest. They are core frontend
surface that always ships (applications, for one), and they look like
missing plugins to anyone counting directories."""
names = []
for path in sorted((REPO / 'plugins').iterdir()):
# Dot-directories are build scratch (plugins/.staging), not surface.
if path.is_dir() and not (path / 'manifest.json').exists() \
and not path.name.startswith(('__', '.')):
names.append(path.name)
return names
def adrs():
rows = []
for path in sorted((REPO / 'docs' / 'adr').glob('ADR-*.md')):
text = _read(path)
title = next((line.lstrip('# ').strip()
for line in text.splitlines() if line.startswith('# ')),
path.stem)
# Both spellings are in use: '- Status: ACCEPTED' and
# '- **Status:** ACCEPTED'. Matching only one reported every ADR as '?',
# which reads as "nobody decided" rather than "the regex is wrong".
status = re.search(r'^[-*\s]*(?:\*\*)?Status(?:\*\*)?\s*:?\s*(?:\*\*)?\s*(.+)$',
text, re.M | re.I)
clean = re.sub(r'[*_`]', '', status.group(1)).strip() if status else '?'
rows.append({'file': path.name, 'title': title, 'status': clean[:40]})
return rows
def endpointcount():
"""From the generated spec, not from a live app: this script must stay
importable without a database or an app context."""
spec = REPO / 'docs' / 'openapi.json'
if not spec.is_file():
return None
try:
return len(json.loads(_read(spec)).get('paths', {}))
except ValueError:
return None
def testcount():
"""Test FUNCTIONS, counted by reading the files.
Deliberately not `pytest --collect-only`: that needs the venv and the whole
import graph, and a map generator that only runs in a fully installed
environment is one that stops being run. Parametrised cases make the real
collected number higher; this is a floor, and it is labelled as one.
"""
total = 0
for path in (REPO / 'tests').rglob('test_*.py'):
total += len(re.findall(r'^\s*def test_', _read(path), re.M))
return total
def render():
lines = [HEADER]
version = versions()
lines.append('## Versions\n')
lines.append('| series | value | governed by |')
lines.append('|---|---|---|')
lines.append('| product `__version__` | `%s` | ADR-007 |'
% version['__version__'])
lines.append('| plugin contract `__contract_version__` | `%s` | ADR-002 |'
% version['__contract_version__'])
lines.append('')
lines.append('They move independently. A contract bump is not a release.\n')
lines.append('## Alembic chains (ADR-008)\n')
lines.append('Core runs with `flask db upgrade`; every plugin chain runs')
lines.append('with `flask plugin upgrade-all`. Both are needed on a deploy.\n')
lines.append('| chain | head |')
lines.append('|---|---|')
lines.append('| core | `%s` |' % (chainhead(REPO / 'migrations' / 'versions')
or 'none'))
rows = plugins()
for row in rows:
if row['head']:
lines.append('| %s | `%s` |' % (row['name'], row['head']))
lines.append('')
lines.append('## Bundled plugins (%d)\n' % len(rows))
lines.append('| plugin | version | core_version | owns migrations |')
lines.append('|---|---|---|---|')
for row in rows:
lines.append('| %s | %s | %s | %s |' % (
row['name'], row['version'], row['core'] or '-',
'yes' if row['head'] else 'no'))
lines.append('')
extra = frontendonlyplugins()
if extra:
lines.append('Manifest-less directories under `plugins/` are core '
'frontend surface and always ship: %s.\n'
% ', '.join('`%s`' % name for name in extra))
lines.append('## Architecture decisions\n')
lines.append('| ADR | title | status |')
lines.append('|---|---|---|')
for row in adrs():
lines.append('| %s | %s | %s |' % (row['file'], row['title'], row['status']))
lines.append('')
lines.append('## Size\n')
endpoints = endpointcount()
lines.append('- test functions defined: **%d** (parametrised cases collect '
'higher)' % testcount())
lines.append('- documented API paths: **%s** (`docs/openapi.json`, '
'regenerate with `scripts/gen_openapi.py`)'
% ('%d' % endpoints if endpoints is not None else 'unknown'))
lines.append('')
return '\n'.join(lines) + '\n'
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--check', action='store_true',
help='exit 1 if the committed map is out of date')
parser.add_argument('--stdout', action='store_true',
help='print the map instead of writing it')
args = parser.parse_args()
content = render()
if args.stdout:
sys.stdout.write(content)
return 0
if args.check:
current = _read(OUTPUT) if OUTPUT.is_file() else ''
if current != content:
sys.stderr.write(
'docs/PROJECT-MAP.md is out of date.\n'
'Regenerate: venv/bin/python scripts/gen_project_map.py\n')
return 1
print('docs/PROJECT-MAP.md is current')
return 0
OUTPUT.write_text(content, encoding='utf-8')
print('wrote %s' % OUTPUT.relative_to(REPO))
return 0
if __name__ == '__main__':
raise SystemExit(main())