Import the wilm agent store and WINC adoption work, with four fixes

Eleven commits from the sister site: a reversible Maximo/WINC machine-adoption
import, a versioned agent binary store with upload and rollback, remote-BuildKit
CI, tray version reporting, and UI fixes. Applied as a patch again - the
published mirror rewrites hashes, so the branches have unrelated histories and
git refuses to merge them.

Reviewed by three agents before merging. Four things fixed here.

THE PUBLISH WORKFLOW WAS DEAD. The new remote-BuildKit step in
publish-images.yml sits at column 0 instead of indented under steps:, so the
whole file is invalid YAML and GitHub would refuse it. Their entire GHCR
distribution channel - which Start-ShopDBDev.ps1 and the compose files consume -
was broken at the source and clearly never run. Indented; all five workflows now
parse.

THE IMPORT'S APPROVAL CHECK COULD NOT FAIL. commit() built the plan and the
approval packet from the same upload and then asked the writer whether they
matched, so packet_matches_plan compared a hash against itself. The documented
contract - "a human approves a specific, frozen plan, and the writer can prove
nothing drifted" - did not exist in the web path: an admin could preview file A
and commit file B, gated only by the needs-create group NAMES they ticked. The
preview's planhash is now required at commit and checked against a freshly
rebuilt plan. Their tests committed without previewing, which is what let this
through; they go through the real flow now, and three new tests pin it - all
confirmed to fail with the check stubbed out.

endpoints.agentpackage QUIETLY GREW. It used to mean "wrap the staged exe in a
ZIP"; with the upload routes it also means "choose the executable every future
package installs and runs as SYSTEM". Before this, staging needed filesystem
access to the server. The permission description still said only "generate and
download a package", so an admin granting it could not see what they were
granting. Reworded.

NOTHING RECORDED WHAT WAS STAGED, OR BY WHOM. The manifest held version, size
and timestamp; re-uploading a version replaces the bytes in place under the same
label. A sha256 of the stored file is now recorded, and the upload is
audit-logged with it - after an incident there was previously no answer to who
put that binary on the fleet.

Also documented the nine new routes in docs/api-inventory.json (that gate cannot
run on the mirror, so it falls to us at promotion time again), regenerated the
spec and the project map, and kept our own docs where they conflicted: docs/ is
scrubbed from the mirror, so their DEV-BUILD-TEST.md still had the dev-be
channel mapping we corrected, and their PLUGIN-HOOKS.md was a new 45-line file
rather than our 737-line canonical one. Their journal section is appended to
ours.

Contract version comes across as 0.30.0 for a genuinely additive surface.

Suite 2369, frontend 324, gate green.

Scope: product
This commit is contained in:
cproudlock
2026-09-03 07:54:14 -04:00
parent 7f6aaf2e64
commit 13be0ef90f
53 changed files with 5064 additions and 95 deletions

View File

@@ -99,7 +99,8 @@ EXPECTED_HEAD_REVISION['warranty'] = 'warranty0002proof'
# the operatorreports table for tray issue reports.
# 0010 renames the shipped business policy off the originating site's
# designator on installs that already stamped 0007 (ADR-015).
EXPECTED_HEAD_REVISION['endpoints'] = 'endpoints0010renamepolicy'
# 0011 adds the trayversion cache column to endpointstates.
EXPECTED_HEAD_REVISION['endpoints'] = 'endpoints0011trayversion'
# Plugins built after the cutover: their 0001 baseline really creates tables the
# core chain never owned.

View File

@@ -0,0 +1,57 @@
"""Unit tests for the versioned agent binary store helpers.
Covers the pure, context-free helpers: version validation, PE magic and PE
file-version parsing. The staging/retention paths that touch the Flask instance
directory are exercised through the API tests; here we keep to the logic that
needs no app context.
The PE version parser walks the resource directory to the outer image's
RT_VERSION resource (not the first VS_FIXEDFILEINFO signature in the bytes),
because a self-contained single-file .NET exe bundles many framework assemblies
that each carry their own version block. Building a full PE with a resource
directory inline is impractical here, so the parser's correctness is verified
against real agent binaries in the API/integration layer; these unit tests pin
the surrounding validators and the malformed-input contract (never raises,
returns None).
"""
from plugins.endpoints.api import agentstore
def test_is_valid_version_accepts_dotted_numeric():
assert agentstore.is_valid_version('1')
assert agentstore.is_valid_version('1.6.3')
assert agentstore.is_valid_version('1.6.3.0')
def test_is_valid_version_rejects_bad_input():
assert not agentstore.is_valid_version('')
assert not agentstore.is_valid_version('v1.6')
assert not agentstore.is_valid_version('1.6.3.0.1')
assert not agentstore.is_valid_version('../etc')
assert not agentstore.is_valid_version('latest')
def test_is_valid_binary_key():
assert agentstore.is_valid_binary_key('agent')
assert agentstore.is_valid_binary_key('tray')
assert not agentstore.is_valid_binary_key('other')
def test_looks_like_pe():
assert agentstore.looks_like_pe(b'MZ\x90\x00')
assert not agentstore.looks_like_pe(b'PK\x03\x04')
assert not agentstore.looks_like_pe(b'')
def test_read_pe_file_version_none_on_non_pe():
assert agentstore.read_pe_file_version(b'not a pe file') is None
def test_read_pe_file_version_none_on_truncated_pe():
# MZ header but nothing else valid: must return None, never raise.
assert agentstore.read_pe_file_version(b'MZ' + b'\x00' * 200) is None
def test_read_pe_file_version_none_on_empty():
assert agentstore.read_pe_file_version(b'') is None

View File

@@ -0,0 +1,270 @@
"""WINC machine-adoption import routes: preview / commit / undo.
WINC is not a canonical-CSV source adapter (that path is covered by
test_import_sources). It updates existing machines only, needs explicit
acknowledgment before creating a business unit, and every commit is journalled
so it can be undone. These tests drive the real routes end to end against the
in-memory schema: seed a machine asset, upload an export that matches it, and
assert what preview reports, what commit writes, and that undo puts it back.
"""
import io
import pytest
def _seed_machine(db, assetnumber='0083A Debur'):
"""Create one machine asset the export can match on, plus a J3 location.
Returns (assetid, locationid). The workstation key of '0083A Debur' is
'0083A', which the export row 'W0083A' reduces to, so the two crosswalk.
"""
from shopdb.core.models import AssetType, AssetStatus, Location
from shopdb.core.models.asset import Asset
from plugins.machines.models.machine import Machine
assettype = AssetType(assettype='machine')
status = AssetStatus(status='In Use')
location = Location(locationname='J3')
db.session.add_all([assettype, status, location])
db.session.flush()
asset = Asset(
assetnumber=assetnumber,
name='Old Name',
assettypeid=assettype.assettypeid,
statusid=status.statusid,
)
db.session.add(asset)
db.session.flush()
db.session.add(Machine(assetid=asset.assetid))
db.session.commit()
return asset.assetid, location.locationid
# An export whose single in-scope row matches the seeded machine. 'WM-J3'
# resolves to the J3 location; 'lg parts' aliases to the 'Large Parts' business
# unit (present or needs-create depending on the test).
EXPORT_CSV = (
'assetnum,description,location,eq1\r\n'
'W0083A,Renamed Machine,WM-J3,lg parts\r\n'
'FACILITIES-1,Not a machine,,\r\n'
)
def _upload(client, headers, url, csv_text=EXPORT_CSV, extra=None):
data = {'files': (io.BytesIO(csv_text.encode('utf-8')), 'winc.csv')}
if extra:
data.update(extra)
return client.post(url, data=data, content_type='multipart/form-data',
headers=headers)
def _commit(client, headers, csv_text=EXPORT_CSV, extra=None, planhash=None):
"""Preview, then commit the plan the preview returned.
This is the real flow, and the tests have to use it: the commit route now
requires the planhash from the preview. Committing without previewing was
what made the approval meaningless - the writer compared a freshly built
packet against the plan it was built from, so its drift check could not
fail and an admin could approve one plan and apply another.
Pass planhash explicitly to simulate approving one plan and committing a
different one.
"""
if planhash is None:
preview = _upload(client, headers, PREVIEW, csv_text=csv_text)
planhash = preview.get_json()['data']['planhash']
payload = dict(extra or {})
payload['planhash'] = planhash
return _upload(client, headers, COMMIT, csv_text=csv_text, extra=payload)
PREVIEW = '/api/settings/asset-import/winc/preview'
COMMIT = '/api/settings/asset-import/winc/commit'
UNDO = '/api/settings/asset-import/winc/undo'
BATCHES = '/api/settings/asset-import/winc/batches'
def test_preview_reports_plan_without_writing(client, db, auth_headers):
_seed_machine(db)
response = _upload(client, auth_headers, PREVIEW)
assert response.status_code == 200, response.get_json()
report = response.get_json()['data']
# One workstation row in scope, the facilities row filtered out.
assert report['committed'] is False
assert report['rowcount'] == 1
assert report['summary']['in_scope'] == 1
assert report['summary']['non_workstation_rows'] == 1
# 'Large Parts' does not exist yet, so it is flagged for acknowledgment.
assert report['needscreate']['businessunit'] == {'Large Parts': 1}
assert report['blocking'] is False
# A preview must not have touched the asset.
from shopdb.core.models.asset import Asset
assert Asset.query.first().name == 'Old Name'
def test_preview_rejects_non_winc_upload(client, db, auth_headers):
_seed_machine(db)
response = _upload(client, auth_headers, PREVIEW,
csv_text='foo,bar\r\n1,2\r\n')
assert response.status_code == 400
assert 'assetnum' in response.get_json()['data']['error']['message']
def test_commit_requires_acknowledgment_for_needs_create(client, db,
auth_headers):
_seed_machine(db)
# No acknowledge field: the needs-create business unit blocks the write.
response = _commit(client, auth_headers)
assert response.status_code == 400
message = response.get_json()['data']['error']['message'].lower()
assert 'acknowledge' in message
from shopdb.core.models.asset import Asset
from shopdb.core.models import BusinessUnit
assert Asset.query.first().name == 'Old Name'
assert BusinessUnit.query.count() == 0
def test_commit_applies_plan_and_creates_business_unit(client, db,
auth_headers):
assetid, locationid = _seed_machine(db)
response = _commit(client, auth_headers, extra={'acknowledge': 'businessunit'})
assert response.status_code == 200, response.get_json()
report = response.get_json()['data']
assert report['committed'] is True
assert report['batchid']
from shopdb.core.models.asset import Asset
from shopdb.core.models import BusinessUnit
asset = db.session.get(Asset, assetid)
assert asset.name == 'Renamed Machine'
assert asset.locationid == locationid
created = BusinessUnit.query.filter_by(businessunit='Large Parts').one()
assert asset.businessunitid == created.businessunitid
def test_commit_uses_existing_business_unit_when_present(client, db,
auth_headers):
from shopdb.core.models import BusinessUnit
existing = BusinessUnit(businessunit='Large Parts')
db.session.add(existing)
db.session.commit()
existing_id = existing.businessunitid
assetid, _ = _seed_machine(db)
# Already resolved, so no needs-create and no acknowledgment needed.
response = _commit(client, auth_headers)
assert response.status_code == 200, response.get_json()
from shopdb.core.models.asset import Asset
asset = db.session.get(Asset, assetid)
assert asset.businessunitid == existing_id
# No duplicate business unit was created.
assert BusinessUnit.query.filter_by(businessunit='Large Parts').count() == 1
def test_undo_reverses_a_committed_batch(client, db, auth_headers):
assetid, _ = _seed_machine(db)
committed = _commit(client, auth_headers, extra={'acknowledge': 'businessunit'})
batchid = committed.get_json()['data']['batchid']
from shopdb.core.models.asset import Asset
from shopdb.core.models import BusinessUnit
assert db.session.get(Asset, assetid).name == 'Renamed Machine'
response = client.post(UNDO, json={'batchid': batchid}, headers=auth_headers)
assert response.status_code == 200, response.get_json()
result = response.get_json()['data']
assert result['batchid'] == batchid
assert result['reverted'] > 0
assert result['conflicts'] == []
db.session.expire_all()
asset = db.session.get(Asset, assetid)
assert asset.name == 'Old Name'
assert asset.locationid is None
assert asset.businessunitid is None
# The importer-created business unit was hard-deleted on undo.
assert BusinessUnit.query.filter_by(businessunit='Large Parts').count() == 0
def test_undo_requires_batchid(client, db, auth_headers):
response = client.post(UNDO, json={}, headers=auth_headers)
assert response.status_code == 400
assert 'batchid' in response.get_json()['data']['error']['message'].lower()
def test_batches_lists_committed_then_undone(client, db, auth_headers):
_seed_machine(db)
committed = _commit(client, auth_headers, extra={'acknowledge': 'businessunit'})
batchid = committed.get_json()['data']['batchid']
listed = client.get(BATCHES, headers=auth_headers).get_json()['data']
batch = next(b for b in listed['batches'] if b['batchid'] == batchid)
assert batch['status'] == 'applied'
assert batch['kind'] == 'winc-machine-adoption'
client.post(UNDO, json={'batchid': batchid}, headers=auth_headers)
listed = client.get(BATCHES, headers=auth_headers).get_json()['data']
batch = next(b for b in listed['batches'] if b['batchid'] == batchid)
assert batch['status'] == 'undone'
assert batch['undoneat'] is not None
def test_winc_routes_require_admin(client, db, member_headers):
_seed_machine(db)
for url in (PREVIEW, COMMIT):
response = _upload(client, member_headers, url)
assert response.status_code in (401, 403)
assert client.get(BATCHES, headers=member_headers).status_code in (401, 403)
# --- an approved plan has to be the plan that gets applied --------------------
#
# The writer's drift check compared a packet against the plan it was built from,
# so it could never fail. "Approved" meant only that the admin ticked the right
# needs-create group NAMES - the plan itself could be anything.
def test_commit_refuses_a_plan_that_was_not_the_one_previewed(client, db,
auth_headers):
_seed_machine(db)
response = _commit(client, auth_headers,
extra={'acknowledge': 'businessunit'},
planhash='a' * 64)
assert response.status_code == 400
assert 'previewed' in response.get_json()['data']['error']['message']
def test_commit_refuses_without_a_planhash_at_all(client, db, auth_headers):
"""Committing without previewing is what made the approval vacuous."""
_seed_machine(db)
response = _upload(client, auth_headers, COMMIT,
extra={'acknowledge': 'businessunit'})
assert response.status_code == 400
assert 'planhash' in response.get_json()['data']['error']['message']
def test_a_preview_of_one_file_does_not_approve_another(client, db,
auth_headers):
"""The concrete attack: preview file A, commit file B."""
from shopdb.core.models import Asset
_seed_machine(db)
other_csv = (
'assetnum,description,location,eq1\r\n'
'W0083A,Something Else Entirely,WM-J3,lg parts\r\n'
)
approved = _upload(client, auth_headers, PREVIEW).get_json()['data']['planhash']
response = _upload(client, auth_headers, COMMIT, csv_text=other_csv,
extra={'acknowledge': 'businessunit',
'planhash': approved})
assert response.status_code == 400
assert Asset.query.filter_by(name='Something Else Entirely').first() is None