Six commits from wilm-import: the squashed import, route documentation, the
channel remap onto our tag-and-trunk model, the project map, parallel test
support, and the four smaller review findings.
Merged rather than cherry-picked: the published mirror rewrites hashes, so the
commits do not apply one at a time against our history even though the squashed
diff applies clean.
Scope: product
# Conflicts:
# docs/PROJECT-MAP.md
The channel work assumed the branch model it was written against: upstream,
main is a release branch and dev-be is the trunk. Neither holds here. main IS
our trunk - 212 commits past v0.11.3 - and a release is a tag (ADR-007). dev-be
does not exist at all, so the edge half would never have fired and the launchers
would have cloned a missing branch.
Taken literally it would have published our unreleased trunk to every site as
:stable, which is the opposite of what the word promises to someone who has no
way to check.
Remapped throughout:
tag v* -> :stable the thing that was actually released
main -> :edge the trunk, DEV badge lit
publish-images.yml keys the channel on github.ref_type = tag rather than on a
branch name, so the mapping cannot silently drift back. Start-ShopDBDev.ps1,
dev-bootstrap.ps1 and build-channel-installer.ps1 resolve stable to the newest
v* tag via git ls-remote --sort=-v:refname, warning and falling back to main if
a repo has no tags yet. DEV-BUILD-TEST.md follows.
The push: trigger is commented out. Automatic publishing waits on someone
confirming the GHCR package exists under this owner and deciding who may pull
it; workflow_dispatch works meanwhile. Turning it on is one uncomment.
All three scripts parse clean under pwsh, and the tag-selection expression was
run against this repo rather than eyeballed. Per the standing rule, the runtime
paths still want a pass in the win11 VM before anyone leans on them.
Scope: product
GE Aerospace now expects a Darkwing check on every repo. Two files rather than
one, because they answer different questions and only one of them is ours.
scan-darkwing-portable.yml is copied byte-for-byte from the ge-aero/darkwing
README so it keeps matching the sanctioned integration. Manual dispatch only.
scan-darkwing-pr.yml runs on pull requests and is ours to change. It exists
because the README workflow cannot simply be given a pull_request trigger: it
resolves its target as `inputs.target_ref || 'main'`, and a pull_request event
carries no inputs, so the expression falls through to 'main' and the job scans
the default branch instead of the code under review. That is a check that is
green because it never looked. This one resolves the PR head SHA explicitly.
Neither file gates a merge. Darkwing exits 0 whatever it finds, and its results
vary between runs on identical code, so a passing job means the scan ran and
nothing more. The SARIF-parsing step that would turn a finding into a failure is
written but commented out, to be switched on once the false-positive rate on
this codebase is known.
Scope: product
Everything on origin/wilm that is not on the published main, applied as one
change because the published history has rewritten hashes: per-commit
cherry-pick conflicts against our main even though the squashed diff applies
clean.
Contents: endpoint monitor catalog + policy attachment + warn-only drift
reconciler, operator issue reports from the tray, agent launch-path allowlist,
and the GHCR channel-image distribution path for dev workstations.
NOT READY FOR main. Open items:
- 10 new routes are absent from docs/api-inventory.json, so
test_api_inventory_parity fails. That gate SKIPS on the published mirror
(the test ships, its data file is scrubbed), which is why wilm never saw it.
- dev-bootstrap.ps1 and Start-ShopDBDev.ps1 hardcode the ge-aero GitHub org.
- plugins/endpoints/plugin.py carries a whitespace-only re-indent.
Scope: product
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
Every job on GitHub is warning:
Node.js 20 is deprecated. The following actions target Node.js 20 but are
being forced to run on Node.js 24: actions/checkout@v4, actions/setup-node@v4
Working today because the runners shim it, and a hard failure the day that shim
goes. checkout@v5, setup-node@v5 and setup-python@v6 are the releases built for
Node 24, so the warning goes away rather than being suppressed.
Nine references across five jobs. The comment above them says why, so nobody
pins them back to the v4 generation on the assumption it is the safer number.
The org's actions policy allows GitHub-owned actions with no SHA pinning
required, so the tag change needs no allow-list update.
Two faults with one cause: docs/ is excluded from publication because the prose
becomes the wiki, and two things under docs/ are not prose.
shopdb/core/api/docs.py reads docs/openapi.json AT RUNTIME to serve /api/docs.
Excluding it meant every site installed from the published repo shipped the
ReDoc page, loaded it, and got {"error":"openapi.json not generated"}. The spec
now publishes - it is a product file that happens to live under docs/. The prose
still goes to the wiki and nowhere else, and docs/api-inventory.json stays out,
because nothing reads it at runtime.
The second fault is the reason this was worth chasing. Three gate modules skip
themselves when the file they read is absent, which is correct in a published
checkout and silently wrong on GitHub, which runs against exactly that checkout.
The modules skipped, CI reported green, and an outside branch added nine
undocumented routes and passed. A gate that stops running is worse than no gate,
because nobody looks at it again.
scripts/check-gates-ran.sh asserts each gate ran, with two lists: the ones that
must run everywhere, and the ones needing files publication deliberately leaves
behind - required here, allowed to be absent there. A `gates` job runs it in CI.
Proven in both directions: green in each checkout, and red when the file a gate
reads is removed.
Also purge instance/plugins.json from the publication tree. It was never
published - it is untracked there - but an old copy had sat in that tree for
months, protected from --delete by its own exclude, and it made the published
repo fail its own suite locally: 13 plugins enabled where the code expects 16.
A publication tree that cannot pass its own tests cannot be verified before
shipping. That is the exact trap the comment at the top of the script warns
about, which had claimed a second victim.
Also: concurrency on the workflow, so a second push supersedes the first rather
than queueing behind it holding a self-hosted runner.
REQ-D: restore waitress and tzdata to requirements.in. They existed ONLY in the
generated requirements.txt (hand-added in bf9e60e), so the next
`uv pip compile` would have silently removed the WSGI server and the IANA
timezone database from every Windows install.
REQ-E: split production and development requirements. requirements.txt was
installing pytest, pytest-cov, pytest-flask, coverage, iniconfig and pluggy onto
production servers. Verified on a real Windows Server box before this change.
CI, scripts/test-external-plugin.sh and the dev docs now use requirements-dev.txt.
REQ-F: standardise on Python 3.14. The repo declared four different versions
(Dockerfile 3.12, DEPLOY-WINDOWS-IIS 3.12, INSTALL-WINDOWS-IIS 3.13, CI 3.13,
plus README, web.config and PLUGIN-EXTERNAL-REPO). 3.14 is in active bugfix
support until ~Apr 2027 and supported to Oct 2030; 3.13 entered security-only in
Apr 2026. All four compiled dependencies publish win_amd64 wheels for 3.14
(cryptography via an abi3 wheel), verified by building an offline wheelhouse and
installing it on Windows Server 2025.
REQ-G: state MySQL 8.0 as the standard for new installs; 5.7+/5.6 remain
supported on an existing server.
Lockfiles regenerated with uv pip compile. Production deps 44 -> 38.
The org IP allow list blocks GitHub-hosted runner IPs (checkout 403), so
point all jobs at the self-hosted arc-runner-set. Drop the rsync dependency
in build-site.sh (cp + bytecode prune; the ARC runner image has no rsync)
and remove the migrations-mysql job - ARC/Kubernetes has no service
containers, so that MySQL 8 coverage stays on the internal CI.
A frontend dir under plugins/ with no manifest.json is a CORE feature, not a
per-site plugin - applications is one (backend is shopdb/core/api/applications.py,
nav is advertised as core in dashboard.py). stage-frontend.mjs treated it like
a plugin and dropped it under SITE_PLUGINS, so a lean site showed the core
Applications nav item but had no route for it -> blank page. Now manifest-less
frontends always stage regardless of SITE_PLUGINS; SITE_PLUGINS selection applies
only to real plugins. CI lean-build job asserts ApplicationsList ships in a lean
bundle. Found while testing a live machines+printers lean site.
New CI job builds a lean site (machines + printers) via build-site.sh and
asserts omitted-plugin code (PartsKiosk, ManifestEditor, USBLabelBatch,
KnowledgeBaseDetail) is absent from the bundle while chosen-plugin code is
present, and that only chosen plugin dirs stage into the backend. Locks the
lean-build guarantee so a future change cannot silently pull an unchosen plugin
into a per-site build.
The dualpath_single_machine setting description is 257 chars but
settings.description was varchar(255). On strict MySQL 8 an over-length
insert is a hard error 1406 (Data too long), so `flask seed settings`
failed on a fresh install; older/relaxed MySQL truncated silently and
hid it. Widen the column to TEXT (matches value, already TEXT) via core
migration 7d26.
CI only ran `flask db upgrade` + plugin install, never the seeders, so it
missed this. Add a seed step to the migrations-mysql job so a seeded row
that violates a column constraint fails CI on strict MySQL 8 instead of
shipping.
pymysql needs the cryptography package to speak MySQL 8's default
caching_sha2_password, so 'flask db upgrade' against a stock MySQL 8
failed with 'cryptography package is required'. Make it a real
dependency (dev, prod, CI all connect cleanly) and drop the CI
native-auth workaround that stood in for it.
Prod runs Python 3.13.7, not the originally planned 3.12. Align the
stack: CI both jobs 3.12->3.13, the IIS install runbook and the dev
setup guide to 3.13 (winget Python.Python.3.13). NOTE for whoever
maintains the offline kit: its wheels are still cp312 and must be
regenerated as cp313 before the next air-gapped deploy.
From the Fable/Opus documentation audit (8 confirmed + verified
lab-drift the run's session limit had cut short):
- HIGH: the lab's kiosk _kiosk_find_item block showed the pre-stage-17
row-id resolver as current; replace with the shipped gagelabtag /
numeric-tail resolver, fix the stale 'resolved by row id' prose and
the 'stage-7 code is corrected' note.
- MED: the badge _external_lookup block used dict-only row access that
breaks on a tuple cursor; use the tuple-or-dict form shipped. Split
'&&' command chains (fail in PowerShell 5.1) in the lab.
- LOW/link: the Windows note's [DEVELOPMENT-SETUP] link dropped the .md
and 404'd in four docs; fix. Correct the stage-6a->16a comment and
the lab-stage tag range (..16 -> ..17).
- Leaks: drop /home/camp path from ADR-006, the internal gitea host
from PLUGINS.md.
- Windows: add an mklink junction note for the external-plugin symlink
dev loop.
- CI: prime root to mysql_native_password so pymysql connects to the
MySQL 8 service without the cryptography package (and its kit wheel).
The CI workflow comment named the internal server, and
PLUGIN-EXTERNAL-REPO carried internal gitea clone URLs (it becomes a
public wiki page). Point both at the GitHub home / a generic CI
mention so the publication scrub gate passes and the wiki does not
expose internal infrastructure.
GitHub had no CI, so naming/tests/build were unenforced on the public
mirror. Add .github/workflows/ci.yml mirroring the internal pipeline:
backend pytest, the naming gate, frontend vitest+build, and the
migrations-mysql job that proves a fresh flask db upgrade + every
plugin chain on utf8mb4 MySQL 8 is idempotent. Flip the dev-setup CI
note to reflect it. Add an identical Windows/VS Code convention note to
the four developer docs (venv\Scripts vs venv/bin, $env: vs export,
pointer to DEVELOPMENT-SETUP).
