Make main the lean product a site receives

main was 16 days and 259 commits behind the line the product is actually built
on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It
was neither current nor curated - just stale. This brings it forward and, at the
same time, makes it mean something: main is what a receiving site installs, and
nothing else.

WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides,
knowledgebase, machines, network, notifications, printers, slides, tools,
warranty - plus applications, which is manifest-less core and always ships. The
only manifest dependency inside that set is endpoints on computers, and both are
in it, so the closure is self-contained.

WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb
stay on the site line. So do the Windows installer tree, the VM rehearsal
helpers, the agent tooling config, one site's data loader, the internal audit
and findings records, and the planner spreadsheet. 1,135 tracked files against
1,438.

A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped:

deploy/site-profile-universal.json still named all seventeen plugins. It is the
file docs/DEPLOY.md tells a site to provision from, so a fresh install would
have failed at `flask plugin apply-profile` with PluginError - the first thing a
receiving site does, broken by the branch meant to serve them. It now names the
twelve that exist, and its comment says the two must agree.

agent/ was never refreshed. Whoever populated this branch copied shopdb/,
plugins/, frontend/ and tests/ from the site line and left the agent tree at its
September state: 22 C# and PowerShell sources missing, including AgentCredentials,
EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the
site line had deleted still present. The old tree is internally consistent, so it
compiles, and nothing in CI builds it - every gate was green over a server that
speaks enrollment and per-PC credentials to a bundled agent source that cannot.
Resynced whole.

DECISIONS THIS ENCODES, rather than defers:

- The plugin-authoring lab builds printedparts, whose manifest depends on
  employees, so it could not stay. The lab remains on the site line and main
  ships no build-along tutorial. That is a real loss for an adopting developer
  and is recorded here rather than discovered later.
- Recognition, Training and Recertification notifications are no longer seeded.
  Each splits one notification into a card per employee and renders a photo,
  which needs a plugin this branch does not bundle. Seeding them would offer an
  author a type that cannot resolve anybody. A site that installs employees gets
  them from that plugin's own seed.
- Windows/IIS is documented as the legacy path throughout, and the README leads
  with Docker, because that is what a receiving site gets.

The API description is regenerated for THIS branch - 347 paths and 483
operations, down from 415 and 573 - so the published contract describes what is
here rather than what the site line serves. docs/CONFIG.md lost the two
measuring-tool label keys the code no longer declares, which two lanes had
decided opposite ways.

Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run
rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
cproudlock
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions

View File

@@ -13,8 +13,9 @@ so the catalog matches what a deployed agent already watches:
process agent.exe -> MTConnect
service MTConnect UDC Agent -> MTConnect
Shipped rows are product-owned (source='shipped'): the CRUD API refuses to edit
or delete them. Launch paths mirror the agent's launchpaths block; collector and
Shipped rows (source='shipped') keep their kind and displayname - that pair is
the identity the monitor policies match on - but a site may edit their paths and
active flag. Launch paths mirror the agent's launchpaths block; collector and
log paths are left null until site paths are supplied.
Idempotent: creation checks the live schema first so a db.create_all() site

View File

@@ -1,12 +1,22 @@
"""Ship the shopfloor app monitors as a read-only Site Policy.
"""Ship the shopfloor app monitors as a Site Policy bundle.
SUPERSEDED BY 0019. This migration is kept because installs have stamped it, but
what it seeds no longer survives an upgrade: 0019 splits this single bundle into
one Configuration Policy per monitor and REMOVES the shipped Site Policy, so
sites compose their own. Read this for history; do not extend it.
Two claims below were true when written and are now false, corrected here rather
than left to mislead: the seeded rows are NOT read-only (0019 admits 'shipped'
at the policy CRUD guards), and no seeder reconciles them - the idempotency
check at the top of upgrade() returns early if the policy exists, which is what
keeps a site's edits safe across upgrades.
Wraps the four shipped endpointmonitors (eDNC, Universal Data Collector,
MTConnect process, MTConnect UDC Agent service) in a product-owned policy bundle
so the agent stops carrying them as hardcoded appsettings.json defaults and reads
its watch-list from the server instead.
MTConnect process, MTConnect UDC Agent service) in a policy bundle so the agent
stops carrying them as hardcoded appsettings.json defaults and reads its
watch-list from the server instead.
Seeds two computers-plugin rows, both source='shipped' (read-only in the UI, the
CRUD API refuses to edit or delete them):
Seeds two computers-plugin rows, both source='shipped':
ConfigurationPolicy "Shopfloor App Monitors"
scope {"type":"true"} (every endpoint), one action per shipped monitor,

View File

@@ -0,0 +1,80 @@
"""Dedup curated Windows events on the Windows event record id.
Adds endpointevents.eventrecordid and a unique index over
(assetid, eventchannel, eventrecordid).
Why: the agent used to re-read the newest window of every curated channel on
each cadence tick, and this ingest appended those rows without a dedup check, so
one real event was stored once per report for as long as it stayed in that
window - up to 96 identical rows a day. The agent now carries a per-channel
watermark keyed on the Windows record id, which is monotonic and unique within a
channel on a machine; this index is the server-side guarantee that survives an
agent reinstall, a restored image, or two overlapping reports, none of which
preserve the agent's watermark.
Nullable on purpose: an agent older than this field reports no record id, and
MySQL permits repeated NULLs inside a unique index, so those rows keep the
previous append-only behavior instead of failing ingest.
Existing rows are left alone. They all predate the column and so carry NULL,
which cannot collide with the new index. Purging the historical duplicates is a
separate, deliberate data cleanup - not something a schema migration should do
irreversibly.
Idempotent: both the column-add and the index-create check the live schema
first, so a db.create_all() site and a re-run both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0012eventrecordid'
down_revision = 'endpoints0011trayversion'
branch_labels = None
depends_on = None
INDEX_NAME = 'idx_endpointevents_assetrecord'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def _has_index(table_name, index_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(index['name'] == index_name
for index in inspector.get_indexes(table_name))
def upgrade():
if not _has_column('endpointevents', 'eventrecordid'):
op.add_column(
'endpointevents',
sa.Column('eventrecordid', sa.BigInteger(), nullable=True))
if not _has_index('endpointevents', INDEX_NAME):
op.create_index(
INDEX_NAME,
'endpointevents',
['assetid', 'eventchannel', 'eventrecordid'],
unique=True)
def downgrade():
if _has_index('endpointevents', INDEX_NAME):
op.drop_index(INDEX_NAME, table_name='endpointevents')
if _has_column('endpointevents', 'eventrecordid'):
op.drop_column('endpointevents', 'eventrecordid')

View File

@@ -0,0 +1,93 @@
"""Persist the hardware, OS, and session inventory the agent already reports.
Adds to endpointstates: disksjson, biosjson, memoryjson, processorjson,
osinfojson, usercontextjson, domain, and pendingreboot.
Why now: the agent has been sending disks, bios, memory, processor, osinfo and
domain on every report for several versions. The ingest logged them as unknown
payload keys and threw them away, which was the intended staging step - the
shapes are now proven against real endpoint traffic, so they get columns.
usercontext and pendingreboot are new in this same change.
Shape follows the existing servicesjson/portsjson convention: newest snapshot
per endpoint as MEDIUMTEXT, not one row per sample. The agent resends the full
structure every cycle and operators ask "what is in this PC now", not "what was
in it in March". History would be pure write amplification on a fleet-sized
table.
domain and pendingreboot are real columns rather than JSON because they are
single values the fleet view filters, groups, and alerts on; burying a scalar in
a blob makes it unqueryable.
All columns are nullable. Every existing row predates them, and there is no
correct backfill value - a machine whose reboot state has never been reported is
NOT the same as one confirmed to have no reboot pending, and inventing False
would silently create a fleet of "healthy" endpoints.
Idempotent: each column-add checks the live schema first, so a db.create_all()
site and a re-run both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0013inventorycolumns'
down_revision = 'endpoints0012eventrecordid'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
# MEDIUMTEXT. An installed-application or disk list on a busy engineering
# workstation comfortably exceeds the 64KB TEXT ceiling, and a silent truncation
# there would corrupt the JSON rather than merely shorten it.
_SNAPSHOT_COLUMNS = (
'disksjson',
'biosjson',
'memoryjson',
'processorjson',
'osinfojson',
'usercontextjson',
)
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
for column_name in _SNAPSHOT_COLUMNS:
if not _has_column(TABLE_NAME, column_name):
op.add_column(
TABLE_NAME,
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
if not _has_column(TABLE_NAME, 'domain'):
op.add_column(
TABLE_NAME,
sa.Column('domain', sa.String(length=255), nullable=True))
if not _has_column(TABLE_NAME, 'pendingreboot'):
op.add_column(
TABLE_NAME,
sa.Column('pendingreboot', sa.Boolean(), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, 'pendingreboot'):
op.drop_column(TABLE_NAME, 'pendingreboot')
if _has_column(TABLE_NAME, 'domain'):
op.drop_column(TABLE_NAME, 'domain')
for column_name in reversed(_SNAPSHOT_COLUMNS):
if _has_column(TABLE_NAME, column_name):
op.drop_column(TABLE_NAME, column_name)

View File

@@ -0,0 +1,56 @@
"""Add diskhealthjson to endpointstates.
Per-drive health, kept separate from disksjson. That column answers "how full
is this drive"; this one answers "is this drive dying". Merging them would mean
a capacity collection failure and a health collection failure could not be told
apart, and the two come from different WMI sources with different failure modes.
JSON rather than scalar columns because a machine has a variable number of
drives and each carries its own verdict, and because the agent reports the same
drive from two sources - MSFT_PhysicalDisk for NVMe and Win32_DiskDrive for
ATA/SATA - which the server keeps side by side rather than collapsing.
MEDIUMTEXT for consistency with the other snapshot columns on this table.
Nullable with no backfill. A drive whose health has never been reported is not
a healthy drive, and defaulting to an empty list would read as "no disks have
problems" on every endpoint that predates this column.
Idempotent: the column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0014diskhealth'
down_revision = 'endpoints0013inventorycolumns'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
COLUMN_NAME = 'diskhealthjson'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
if not _has_column(TABLE_NAME, COLUMN_NAME):
op.add_column(
TABLE_NAME,
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, COLUMN_NAME):
op.drop_column(TABLE_NAME, COLUMN_NAME)

View File

@@ -0,0 +1,78 @@
"""Add rebootreasonsjson, bootinfojson and defenderjson to endpointstates.
Phase 2 system-side telemetry. All three are collect-and-persist; none is
surfaced in the UI by this change.
rebootreasonsjson is NOT redundant with the existing pendingreboot column. That
column is a single boolean OR-ed from three registry signals, so it cannot say
which one fired, and the difference is operationally significant: a
PendingFileRenameOperations entry means files are staged for replacement on the
next boot, so the machine is mid-servicing, whereas a lone Component Based
Servicing or Windows Update key can sit set for weeks on a box nobody has
rebooted. Same boolean, very different urgency.
bootinfojson exists because uptime lies under fast startup - a machine reports
days of uptime after what the operator experienced as a shutdown, and a "reboot"
that was really a hybrid resume does not clear the driver or kernel state a
support call is usually trying to clear.
defenderjson is stored deliberately unsurfaced. Defender is known to produce
false positives in this environment, so wiring it to a card now would train
operators to ignore that card; persisting it starts the history accumulating for
whenever someone is ready to interpret it.
JSON rather than scalars: reboot reasons is a variable-length list, and the boot
and Defender payloads are small related groups that are read together and would
otherwise be six sparse columns.
MEDIUMTEXT for consistency with the other snapshot columns on this table.
Nullable with no backfill. Null means never reported, which is deliberately
distinct from an empty list (checked, nothing pending) - defaulting the reboot
reasons to [] would assert every pre-existing endpoint was confirmed clean.
Idempotent: each column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0015systemtelemetry'
down_revision = 'endpoints0014diskhealth'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
_COLUMNS = (
'rebootreasonsjson',
'bootinfojson',
'defenderjson',
)
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
for column_name in _COLUMNS:
if not _has_column(TABLE_NAME, column_name):
op.add_column(
TABLE_NAME,
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
def downgrade():
for column_name in reversed(_COLUMNS):
if _has_column(TABLE_NAME, column_name):
op.drop_column(TABLE_NAME, column_name)

View File

@@ -0,0 +1,51 @@
"""Add dellupdatesjson to endpointstates.
Stores the pending Dell Command | Update summary reported by the
shopdb-dellupdates agent utility. MEDIUMTEXT for consistency with the other
snapshot columns on this table.
Nullable with no backfill. Null means the endpoint has never reported a Dell
scan result, which is deliberately distinct from a stored payload whose status
is 'ok' with an empty list (scanned, nothing pending). The status card renders
those two states differently, so defaulting this to an empty summary would
assert every pre-existing endpoint had been scanned and found clean.
Idempotent: the column-add checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0016dellupdates'
down_revision = 'endpoints0015systemtelemetry'
branch_labels = None
depends_on = None
TABLE_NAME = 'endpointstates'
COLUMN_NAME = 'dellupdatesjson'
def _has_column(table_name, column_name):
from sqlalchemy import inspect
inspector = inspect(op.get_bind())
if not inspector.has_table(table_name):
return False
return any(col['name'] == column_name
for col in inspector.get_columns(table_name))
def upgrade():
if not _has_column(TABLE_NAME, COLUMN_NAME):
op.add_column(
TABLE_NAME,
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
def downgrade():
if _has_column(TABLE_NAME, COLUMN_NAME):
op.drop_column(TABLE_NAME, COLUMN_NAME)

View File

@@ -0,0 +1,133 @@
"""Add endpoint health rollup tables.
Three tables behind the Fleet Health column:
endpointhealthstates one cached verdict per endpoint, read directly by
the Fleet list instead of recomputing a 48h
fleet-wide event aggregate on every page load
endpointhealthtransitions capped history, written on CHANGE not on sample
endpointmonitorimpacts per-PC-type operational impact for a monitor
No backfill. An endpoint with no row has simply not checked in since this
shipped, and the first check-in writes one. Backfilling would have to invent a
firstseenat for every existing condition, and a fabricated "broken since" is
worse than an absent one - an operator reading it has no way to tell it apart
from an observed timestamp.
endpointmonitorimpacts ships EMPTY on purpose. Absence of a row means the
built-in default impact applies, so an endpoint with no policies behaves
exactly as it did before this migration. Seeding defaults here would freeze
today's guesses into rows nobody could distinguish from deliberate choices.
Idempotent: every create checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0017healthrollup'
down_revision = 'endpoints0016dellupdates'
branch_labels = None
depends_on = None
HEALTHSTATES = 'endpointhealthstates'
TRANSITIONS = 'endpointhealthtransitions'
IMPACTS = 'endpointmonitorimpacts'
def _has_table(table_name):
from sqlalchemy import inspect
return inspect(op.get_bind()).has_table(table_name)
def upgrade():
if not _has_table(HEALTHSTATES):
op.create_table(
HEALTHSTATES,
sa.Column('endpointhealthstateid', sa.Integer(), nullable=False),
sa.Column('assetid', sa.Integer(), nullable=False),
sa.Column('level', sa.String(length=16), nullable=False,
server_default='unknown'),
sa.Column('label', sa.String(length=255), nullable=True),
sa.Column('impactclass', sa.String(length=16), nullable=True),
sa.Column('sourcekind', sa.String(length=16), nullable=False,
server_default='none'),
sa.Column('sourceref', sa.String(length=64), nullable=True),
sa.Column('firstseenat', sa.DateTime(), nullable=True),
sa.Column('lastconfirmedat', sa.DateTime(), nullable=True),
sa.Column('lastevaluatedat', sa.DateTime(), nullable=False),
sa.Column('flapcount', sa.Integer(), nullable=False,
server_default='0'),
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
ondelete='CASCADE'),
sa.PrimaryKeyConstraint('endpointhealthstateid'),
# One live verdict per endpoint. Unique rather than indexed: two
# rows for one asset is not a state this table should represent.
sa.UniqueConstraint('assetid',
name='uq_endpointhealthstates_assetid'),
)
op.create_index('idx_endpointhealthstates_asset', HEALTHSTATES,
['assetid'])
op.create_index('idx_endpointhealthstates_impactclass', HEALTHSTATES,
['impactclass'])
if not _has_table(TRANSITIONS):
op.create_table(
TRANSITIONS,
sa.Column('endpointhealthtransitionid', sa.Integer(),
nullable=False),
sa.Column('assetid', sa.Integer(), nullable=False),
sa.Column('level', sa.String(length=16), nullable=False),
sa.Column('label', sa.String(length=255), nullable=True),
sa.Column('impactclass', sa.String(length=16), nullable=True),
sa.Column('sourcekind', sa.String(length=16), nullable=False,
server_default='none'),
sa.Column('sourceref', sa.String(length=64), nullable=True),
sa.Column('openedat', sa.DateTime(), nullable=False),
sa.Column('resolvedat', sa.DateTime(), nullable=True),
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
ondelete='CASCADE'),
sa.PrimaryKeyConstraint('endpointhealthtransitionid'),
)
# Retention reads newest-first per asset; the resolved index serves the
# "never evict an unresolved row" filter without scanning history.
op.create_index('idx_endpointhealthtransitions_assetopened',
TRANSITIONS, ['assetid', 'openedat'])
op.create_index('idx_endpointhealthtransitions_assetresolved',
TRANSITIONS, ['assetid', 'resolvedat'])
if not _has_table(IMPACTS):
op.create_table(
IMPACTS,
sa.Column('endpointmonitorimpactid', sa.Integer(), nullable=False),
sa.Column('endpointmonitorid', sa.Integer(), nullable=False),
# Plain integer, not a ForeignKey: computertypes belongs to the
# computers plugin and owns its own migration chain. A cross-plugin
# FK would couple the two chains' ordering for no benefit the
# unique constraint does not already provide.
sa.Column('computertypeid', sa.Integer(), nullable=False),
sa.Column('impactclass', sa.String(length=16), nullable=False,
server_default='degraded'),
sa.Column('createdat', sa.DateTime(), nullable=False),
sa.ForeignKeyConstraint(['endpointmonitorid'],
['endpointmonitors.endpointmonitorid'],
ondelete='CASCADE'),
sa.PrimaryKeyConstraint('endpointmonitorimpactid'),
sa.UniqueConstraint('endpointmonitorid', 'computertypeid',
name='uq_endpointmonitorimpacts_monitortype'),
)
op.create_index('idx_endpointmonitorimpacts_computertypeid', IMPACTS,
['computertypeid'])
def downgrade():
# Dropped newest-first so the FK to endpointmonitors goes before anything
# that might reference it. All three are caches or mappings - nothing here
# is a system of record, so a downgrade loses no telemetry.
for table_name in (IMPACTS, TRANSITIONS, HEALTHSTATES):
if _has_table(table_name):
op.drop_table(table_name)

View File

@@ -0,0 +1,87 @@
"""Add endpointlocks (admin-only endpoint quarantine).
One table behind the endpoint lock: an admin-set refusal of operator commands
on a single endpoint, independent of policy and permission.
Deliberately separate from the computers plugin's policyactionholds. That table
suppresses ONE desired-state action on the policy-resolution axis and expires on
a timer; this one refuses NON-ADMIN CALLERS on the whole endpoint and does not
expire at all. Two concepts, two tables, two enforcement points - collapsing
them would recreate the axis-mixing the surrounding refactor exists to undo.
No expiresat column, and that is the design rather than an omission. A lock
stands for a physical-world condition (machine mid-repair, validated part
running, box that must not reboot during an audit) and those do not end on a
schedule. A TTL would silently unlock a machine someone still has their hands
inside.
Ships EMPTY. No endpoint has ever been locked before this migration, so there is
nothing to backfill and every endpoint keeps behaving exactly as it did.
Idempotent: the create checks the live schema first, so a re-run and a
db.create_all() site both stay clean on strict MySQL 8.
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = 'endpoints0018endpointlocks'
down_revision = 'endpoints0017healthrollup'
branch_labels = None
depends_on = None
LOCKS = 'endpointlocks'
def _has_table(table_name):
from sqlalchemy import inspect
return inspect(op.get_bind()).has_table(table_name)
def upgrade():
if not _has_table(LOCKS):
op.create_table(
LOCKS,
sa.Column('endpointlockid', sa.Integer(), nullable=False),
sa.Column('assetid', sa.Integer(), nullable=False),
sa.Column('reason', sa.Text(), nullable=False),
sa.Column('lockedbyuserid', sa.Integer(), nullable=True),
sa.Column('lockedbyusername', sa.String(length=100), nullable=True),
sa.Column('lockedat', sa.DateTime(), nullable=False),
# Null while the lock stands. Setting it is what releases the lock,
# so there is no separate active flag able to drift out of step.
sa.Column('releasedat', sa.DateTime(), nullable=True),
sa.Column('releasedbyuserid', sa.Integer(), nullable=True),
sa.Column('releasedbyusername', sa.String(length=100),
nullable=True),
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
ondelete='CASCADE'),
# SET NULL, not CASCADE: deleting the admin who locked a machine
# must not delete the record that it was locked. The username
# snapshot columns survive to say who it was.
sa.ForeignKeyConstraint(['lockedbyuserid'], ['users.userid'],
ondelete='SET NULL'),
sa.ForeignKeyConstraint(['releasedbyuserid'], ['users.userid'],
ondelete='SET NULL'),
sa.PrimaryKeyConstraint('endpointlockid'),
)
# Serves the hot path: "is there an active lock on this asset", asked on
# every enqueue and every endpoint detail load. NOT unique on assetid -
# released rows accumulate per endpoint as history, and a unique
# constraint would make the second lock in a machine's life fail. The
# one-active-lock invariant is enforced on write instead.
op.create_index('idx_endpointlocks_assetactive', LOCKS,
['assetid', 'releasedat'])
def downgrade():
# Safe to drop: the table is a system of record for quarantine state, but a
# downgrade past this revision removes the enforcement that reads it, so
# leaving orphaned rows behind would be worse - they would silently reapply
# if the revision were ever re-applied.
if _has_table(LOCKS):
op.drop_index('idx_endpointlocks_assetactive', table_name=LOCKS)
op.drop_table(LOCKS)

View File

@@ -0,0 +1,320 @@
"""Split the shipped monitor bundle into one policy per monitor.
0007 seeded a single Configuration Policy "Shopfloor App Monitors" carrying all
four shipped monitor actions, wrapped in a shipped Site (business) Policy of the
same name. That bundle was all-or-nothing: a site that runs eDNC but no
MTConnect could not detach the half it does not have, because the whole row was
product-owned and read-only.
This migration replaces it with one Configuration Policy per shipped monitor:
Monitor: eDNC (process)
Monitor: Universal Data Collector (process)
Monitor: MTConnect (process)
Monitor: MTConnect (service)
Each carries exactly one 'monitor:<endpointmonitorid>' action, mode='gate',
scope {"type":"true"} - no PC type attribution. The name carries the kind
because displayname alone is not unique: MTConnect ships as both a process and
a service, and configurationpolicies.name is UNIQUE.
source stays 'shipped'. Provenance is still true - the product did ship these -
and the UI keeps showing where they came from. What changes is that the CRUD
guards now admit 'shipped' for edit/delete, so a site can retune or drop one
without losing the record of its origin.
The shipped SITE policy is REMOVED and not replaced. Composition is a site
decision: which monitors apply to which endpoints, and at what priority, is
exactly the judgement a plant makes about its own floor. Shipping one meant
every install inherited "apply all four to every endpoint" as a starting point
that could not be edited. Sites now build their own from the per-monitor pieces.
Removing it is the behaviour change to be aware of on upgrade: monitors that
were attached through that bundle stop being attached until a site policy
composes them. It is deliberate and it is the point of the split - see the
"off by default" direction in the 2026-09-08 plan - but it is not silent, since
the per-monitor policies are listed and ready to compose.
Idempotent on the policy names, so a re-run neither duplicates the split rows
nor resurrects the bundle. Guards on the policy tables being present and no-ops
if they are not, matching 0007.
Revision ID: endpoints0019splitpolicies
Revises: endpoints0018endpointlocks
"""
from alembic import op
import sqlalchemy as sa
from datetime import datetime
# Alembic stores this in alembic_version.version_num, a VARCHAR(32). A longer
# id passes on SQLite and fails on strict MySQL with "Data too long".
revision = 'endpoints0019splitpolicies'
down_revision = 'endpoints0018endpointlocks'
branch_labels = None
depends_on = None
BUNDLE_NAME = 'Shopfloor App Monitors'
# The shipped monitors, by identity (kind, displayname) - never by hardcoded id,
# so this survives id drift across installs exactly as 0007 does.
_SHIPPED_MONITORS = (
('process', 'eDNC'),
('process', 'Universal Data Collector'),
('process', 'MTConnect'),
('service', 'MTConnect'),
)
_REQUIRED = {'endpointmonitors', 'configurationpolicies',
'configurationpolicyactions', 'businesspolicies',
'businesspolicyitems'}
def _tables(bind):
from sqlalchemy import inspect
return set(inspect(bind).get_table_names())
def _policy_name(kind, displayname):
return f'Monitor: {displayname} ({kind})'
def _policy_description(kind, displayname):
noun = 'process' if kind == 'process' else 'service'
return (f'Product-shipped watch-list entry for the {displayname} {noun}. '
f'Attach it to a Site Policy to apply it to endpoints.')
def _config_policy_id(bind, name):
return bind.execute(
sa.text('SELECT configurationpolicyid FROM configurationpolicies '
'WHERE name = :name'),
{'name': name}).scalar()
def _drop_config_policy(bind, name):
"""Delete a configuration policy, its actions, and any items citing it."""
policyid = _config_policy_id(bind, name)
if policyid is None:
return
# Clear children explicitly rather than lean on ON DELETE CASCADE, which
# sqlite does not enforce by default.
bind.execute(
sa.text('DELETE FROM businesspolicyitems '
'WHERE configurationpolicyid = :id'), {'id': policyid})
bind.execute(
sa.text('DELETE FROM configurationpolicyactions '
'WHERE configurationpolicyid = :id'), {'id': policyid})
bind.execute(
sa.text('DELETE FROM configurationpolicies '
'WHERE configurationpolicyid = :id'), {'id': policyid})
def _drop_business_policy(bind, name):
policyid = bind.execute(
sa.text('SELECT businesspolicyid FROM businesspolicies '
'WHERE name = :name AND source = :source'),
{'name': name, 'source': 'shipped'}).scalar()
if policyid is None:
return
bind.execute(
sa.text('DELETE FROM businesspolicyitems WHERE businesspolicyid = :id'),
{'id': policyid})
bind.execute(
sa.text('DELETE FROM businesspolicies WHERE businesspolicyid = :id'),
{'id': policyid})
def _monitor_id(bind, kind, displayname):
return bind.execute(
sa.text('SELECT endpointmonitorid FROM endpointmonitors '
'WHERE kind = :kind AND displayname = :displayname'),
{'kind': kind, 'displayname': displayname}).scalar()
def upgrade():
bind = op.get_bind()
if not _REQUIRED.issubset(_tables(bind)):
return
now = datetime.utcnow()
config_policies = sa.table(
'configurationpolicies',
sa.column('name', sa.String),
sa.column('description', sa.Text),
sa.column('scopejson', sa.Text),
sa.column('source', sa.String),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
actions = sa.table(
'configurationpolicyactions',
sa.column('configurationpolicyid', sa.Integer),
sa.column('action', sa.String),
sa.column('mode', sa.String),
sa.column('desiredstate', sa.Text),
sa.column('params', sa.Text),
sa.column('intervalminutes', sa.Integer),
sa.column('sortorder', sa.Integer),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
for kind, displayname in _SHIPPED_MONITORS:
name = _policy_name(kind, displayname)
if _config_policy_id(bind, name) is not None:
continue
monitorid = _monitor_id(bind, kind, displayname)
if monitorid is None:
# A site that never had this monitor gets no policy for it.
continue
op.bulk_insert(config_policies, [{
'name': name,
'description': _policy_description(kind, displayname),
'scopejson': '{"type": "true"}',
'source': 'shipped',
'createddate': now,
'modifieddate': now,
'isactive': True,
}])
op.bulk_insert(actions, [{
'configurationpolicyid': _config_policy_id(bind, name),
'action': f'monitor:{monitorid}',
'mode': 'gate',
'desiredstate': None,
'params': None,
'intervalminutes': None,
'sortorder': 0,
'createddate': now,
'modifieddate': now,
'isactive': True,
}])
# Retire the monolith. The business policy goes first so its item rows are
# gone before the configuration policy it cited is removed.
_drop_business_policy(bind, BUNDLE_NAME)
_drop_config_policy(bind, BUNDLE_NAME)
def downgrade():
"""Reinstate the single shipped bundle and drop the per-monitor policies.
Rebuilt from the same monitor identities rather than from anything the
upgrade stashed, so it lands in the state 0007 produces regardless of what
a site did to the split rows in between.
"""
bind = op.get_bind()
if not _REQUIRED.issubset(_tables(bind)):
return
now = datetime.utcnow()
monitor_ids = []
for kind, displayname in _SHIPPED_MONITORS:
monitorid = _monitor_id(bind, kind, displayname)
if monitorid is not None:
monitor_ids.append(monitorid)
for kind, displayname in _SHIPPED_MONITORS:
_drop_config_policy(bind, _policy_name(kind, displayname))
if not monitor_ids or _config_policy_id(bind, BUNDLE_NAME) is not None:
return
config_policies = sa.table(
'configurationpolicies',
sa.column('name', sa.String),
sa.column('description', sa.Text),
sa.column('scopejson', sa.Text),
sa.column('source', sa.String),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
op.bulk_insert(config_policies, [{
'name': BUNDLE_NAME,
'description': 'Product-shipped watch-list for the shopfloor apps '
'(eDNC, Universal Data Collector, MTConnect). Read-only.',
'scopejson': '{"type": "true"}',
'source': 'shipped',
'createddate': now,
'modifieddate': now,
'isactive': True,
}])
config_policy_id = _config_policy_id(bind, BUNDLE_NAME)
actions = sa.table(
'configurationpolicyactions',
sa.column('configurationpolicyid', sa.Integer),
sa.column('action', sa.String),
sa.column('mode', sa.String),
sa.column('desiredstate', sa.Text),
sa.column('params', sa.Text),
sa.column('intervalminutes', sa.Integer),
sa.column('sortorder', sa.Integer),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
op.bulk_insert(actions, [{
'configurationpolicyid': config_policy_id,
'action': f'monitor:{monitorid}',
'mode': 'gate',
'desiredstate': None,
'params': None,
'intervalminutes': None,
'sortorder': index,
'createddate': now,
'modifieddate': now,
'isactive': True,
} for index, monitorid in enumerate(monitor_ids)])
business_policies = sa.table(
'businesspolicies',
sa.column('name', sa.String),
sa.column('description', sa.Text),
sa.column('priority', sa.Integer),
sa.column('source', sa.String),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
op.bulk_insert(business_policies, [{
'name': BUNDLE_NAME,
'description': 'Product-shipped Site Policy that applies the shopfloor '
'app monitors to every endpoint. Read-only.',
'priority': 100,
'source': 'shipped',
'createddate': now,
'modifieddate': now,
'isactive': True,
}])
business_policy_id = bind.execute(
sa.text('SELECT businesspolicyid FROM businesspolicies '
'WHERE name = :name'), {'name': BUNDLE_NAME}).scalar()
items = sa.table(
'businesspolicyitems',
sa.column('businesspolicyid', sa.Integer),
sa.column('configurationpolicyid', sa.Integer),
sa.column('sortorder', sa.Integer),
sa.column('createddate', sa.DateTime),
sa.column('modifieddate', sa.DateTime),
sa.column('isactive', sa.Boolean),
)
op.bulk_insert(items, [{
'businesspolicyid': business_policy_id,
'configurationpolicyid': config_policy_id,
'sortorder': 0,
'createddate': now,
'modifieddate': now,
'isactive': True,
}])

View File

@@ -0,0 +1,90 @@
"""Add fingerprint column for delta-based application updates
Revision ID: endpoints0020appfingerprints
Revises: endpoints0019splitpolicies
Create Date: 2026-09-14
Performance optimization: Add appfingerprint column to endpointapplications
table to enable delta-based updates instead of delete-all-insert-all pattern.
Impact:
- Before: 200 endpoints x 50 apps x 4 reports/hour = 40,000 DELETEs + 40,000 INSERTs/hour
- After: Only changed apps are touched (~2% per day) = 40 DELETEs + 40 INSERTs/hour
- Reduction: 99.9% fewer writes (1.92M writes/day -> 1,920 writes/day)
The fingerprint is SHA256(displayname|displayversion|publisher) computed by
the agent and sent in the collector payload. Nullable during rollout - legacy
agents without fingerprint support fall back to delete-all-insert-all behavior.
DEPLOYMENT SEQUENCE (CRITICAL):
1. Deploy this migration: flask db upgrade
2. Run backfill script: python scripts/backfill_app_fingerprints.py
3. Deploy updated plugin.py (delta logic code)
4. Deploy updated C# agent (fingerprint computation)
ROLLBACK PROCEDURE:
1. Deploy old plugin.py FIRST (remove delta logic)
2. THEN run: flask db downgrade -1
3. Verify: SELECT COUNT(*) FROM endpointapplications (should be unchanged)
WARNING: Do NOT downgrade migration before reverting code - will cause AttributeError!
SAFETY FEATURES:
- Row-level locking prevents concurrent update races
- Fingerprint format validation (64 hex chars)
- Delete count verification detects silent failures
- Memory limits prevent pathological cases (>5000 apps)
- Metrics logged for monitoring
Idempotent: a database that already has the column (create_all built it from
the model, or an earlier build of this revision added it as app_fingerprint)
is left alone. 0023 renames the old spelling.
"""
from alembic import op
import sqlalchemy as sa
revision = 'endpoints0020appfingerprints'
down_revision = 'endpoints0019splitpolicies'
branch_labels = None
depends_on = None
plugin_name = 'endpoints'
TABLE = 'endpointapplications'
INDEX = 'idx_endpointapplications_fingerprint'
def _columns():
from sqlalchemy import inspect
return {c['name'] for c in inspect(op.get_bind()).get_columns(TABLE)}
def _indexes():
from sqlalchemy import inspect
return {ix['name'] for ix in inspect(op.get_bind()).get_indexes(TABLE)}
def upgrade():
columns = _columns()
# Old spelling present: leave it for 0023 to rename.
if 'appfingerprint' in columns or 'app_fingerprint' in columns:
return
op.add_column(TABLE, sa.Column('appfingerprint', sa.String(64),
nullable=True))
if INDEX not in _indexes():
# Unique per asset: the delta logic keys rows on it.
op.create_index(INDEX, TABLE, ['assetid', 'appfingerprint'],
unique=True)
def downgrade():
if INDEX in _indexes():
op.drop_index(INDEX, table_name=TABLE)
columns = _columns()
for name in ('appfingerprint', 'app_fingerprint'):
if name in columns:
with op.batch_alter_table(TABLE) as batch:
batch.drop_column(name)

View File

@@ -0,0 +1,26 @@
"""Retired: client certificate columns on assets.
Revision ID: endpoints0021clientcerts
Revises: endpoints0020appfingerprints
Create Date: 2026-09-14
This revision once added client certificate columns to the core assets table.
Client certificate authentication was replaced by per-PC endpoint credentials
(0023), so the revision is kept only so databases already stamped with it
still resolve. It does nothing; 0023 drops the columns wherever an earlier
build of this revision created them.
"""
revision = 'endpoints0021clientcerts'
down_revision = 'endpoints0020appfingerprints'
branch_labels = None
depends_on = None
plugin_name = 'endpoints'
def upgrade():
pass
def downgrade():
pass

View File

@@ -0,0 +1,24 @@
"""Retired: client certificate revocation columns on assets.
Revision ID: endpoints0022certrevocation
Revises: endpoints0021clientcerts
Create Date: 2026-09-14
Kept only so databases already stamped with this revision still resolve. It
does nothing; 0023 drops the columns wherever an earlier build of this
revision created them.
"""
revision = 'endpoints0022certrevocation'
down_revision = 'endpoints0021clientcerts'
branch_labels = None
depends_on = None
plugin_name = 'endpoints'
def upgrade():
pass
def downgrade():
pass

View File

@@ -0,0 +1,181 @@
"""Per-PC endpoint credentials; remove the client certificate leftovers.
Revision ID: endpoints0023credentials
Revises: endpoints0022certrevocation
Create Date: 2026-09-15
Three things, each guarded against the live schema so the revision is safe on
a database built by create_all, on one that ran the earlier client certificate
builds of 0021/0022, and on a re-run:
1. Drop the client certificate columns (and their index) from the core assets
table where an earlier build of 0021/0022 created them.
2. Rename endpointapplications.app_fingerprint to appfingerprint where an
earlier build of 0020 used the underscore spelling.
3. Create endpointcredentials (with the reenrollblocked flag and its reason),
endpointenrollrequests and endpointmigrationwaves.
MySQL 5.6 safe: no JSON columns, and the enroll request lookup index uses
prefix lengths so the key stays under 767 bytes on utf8mb4.
Downgrade drops the three tables. The certificate columns are not restored:
the code that read them is gone.
"""
from alembic import op
import sqlalchemy as sa
revision = 'endpoints0023credentials'
down_revision = 'endpoints0022certrevocation'
branch_labels = None
depends_on = None
plugin_name = 'endpoints'
CREDENTIALS = 'endpointcredentials'
REQUESTS = 'endpointenrollrequests'
WAVES = 'endpointmigrationwaves'
_CERT_COLUMNS = ('client_cert_fingerprint', 'cert_issued_at', 'cert_expires_at',
'cert_revoked_at', 'cert_revocation_reason')
_APPS = 'endpointapplications'
_FINGERPRINT_INDEX = 'idx_endpointapplications_fingerprint'
def _inspector():
from sqlalchemy import inspect
return inspect(op.get_bind())
def _has_table(name):
return _inspector().has_table(name)
def _columns(table):
return {c['name'] for c in _inspector().get_columns(table)}
def _indexes(table):
return _inspector().get_indexes(table)
def _drop_certificate_columns():
if not _has_table('assets'):
return
present = _columns('assets') & set(_CERT_COLUMNS)
if not present:
return
# Any index over a certificate column goes first: 0021 named its own,
# create_all names by convention, and SQLite refuses to drop an indexed column.
for index in _indexes('assets'):
if set(index.get('column_names') or ()) & present:
op.drop_index(index['name'], table_name='assets')
for column in _CERT_COLUMNS:
if column in present:
op.drop_column('assets', column)
def _rename_fingerprint():
if not _has_table(_APPS):
return
columns = _columns(_APPS)
if 'app_fingerprint' not in columns:
return
if 'appfingerprint' in columns:
return
for index in _indexes(_APPS):
if 'app_fingerprint' in (index.get('column_names') or ()):
op.drop_index(index['name'], table_name=_APPS)
op.alter_column(_APPS, 'app_fingerprint', new_column_name='appfingerprint',
existing_type=sa.String(64), existing_nullable=True)
op.create_index(_FINGERPRINT_INDEX, _APPS, ['assetid', 'appfingerprint'],
unique=True)
def upgrade():
_drop_certificate_columns()
_rename_fingerprint()
if not _has_table(CREDENTIALS):
op.create_table(
CREDENTIALS,
sa.Column('credentialid', sa.Integer(), nullable=False),
sa.Column('assetid', sa.Integer(), nullable=False),
sa.Column('credentialhash', sa.CHAR(length=64), nullable=False),
sa.Column('credentialprefix', sa.String(length=12), nullable=False),
sa.Column('enrolledvia', sa.String(length=16), nullable=False),
sa.Column('issuedat', sa.DateTime(), nullable=False),
sa.Column('lastusedat', sa.DateTime(), nullable=True),
sa.Column('revokedat', sa.DateTime(), nullable=True),
sa.Column('revokedby', sa.String(length=64), nullable=True),
sa.Column('reenrollblocked', sa.Boolean(), nullable=False,
server_default=sa.false()),
sa.Column('reenrollblockedreason', sa.String(length=64),
nullable=True),
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
ondelete='CASCADE'),
sa.PrimaryKeyConstraint('credentialid'),
sa.UniqueConstraint('assetid'),
sa.UniqueConstraint('credentialhash'),
)
else:
# A test database that ran an earlier build of this revision.
columns = _columns(CREDENTIALS)
if 'reenrollblocked' not in columns:
op.add_column(CREDENTIALS, sa.Column(
'reenrollblocked', sa.Boolean(), nullable=False,
server_default=sa.false()))
if 'reenrollblockedreason' not in columns:
op.add_column(CREDENTIALS, sa.Column(
'reenrollblockedreason', sa.String(length=64), nullable=True))
if not _has_table(REQUESTS):
op.create_table(
REQUESTS,
sa.Column('requestid', sa.Integer(), nullable=False),
sa.Column('hostname', sa.String(length=255), nullable=False),
sa.Column('serialnumber', sa.String(length=128), nullable=False),
sa.Column('deviceid', sa.String(length=64), nullable=True),
sa.Column('enrollnonce', sa.String(length=128), nullable=False),
sa.Column('agentversion', sa.String(length=32), nullable=True),
sa.Column('status', sa.String(length=16), nullable=False),
sa.Column('reason', sa.String(length=64), nullable=True),
sa.Column('matchedassetid', sa.Integer(), nullable=True),
sa.Column('sourceip', sa.String(length=45), nullable=True),
sa.Column('requestedat', sa.DateTime(), nullable=False),
sa.Column('lastseenat', sa.DateTime(), nullable=False),
sa.Column('decidedat', sa.DateTime(), nullable=True),
sa.Column('decidedby', sa.String(length=64), nullable=True),
sa.ForeignKeyConstraint(['matchedassetid'], ['assets.assetid'],
ondelete='SET NULL'),
sa.PrimaryKeyConstraint('requestid'),
)
existing = {tuple(ix.get('column_names') or ()) for ix in _indexes(REQUESTS)}
if ('hostname', 'serialnumber') not in existing:
op.create_index('idx_endpointenrollrequests_hostserial', REQUESTS,
['hostname', 'serialnumber'],
mysql_length={'hostname': 128, 'serialnumber': 63})
if not _has_table(WAVES):
op.create_table(
WAVES,
sa.Column('waveid', sa.Integer(), nullable=False),
sa.Column('createdat', sa.DateTime(), nullable=False),
sa.Column('createdby', sa.String(length=64), nullable=True),
sa.Column('assetids', sa.Text(), nullable=False),
sa.Column('notes', sa.String(length=255), nullable=True),
sa.PrimaryKeyConstraint('waveid'),
)
def downgrade():
if _has_table(WAVES):
op.drop_table(WAVES)
if _has_table(REQUESTS):
for index in _indexes(REQUESTS):
if index['name'] == 'idx_endpointenrollrequests_hostserial':
op.drop_index(index['name'], table_name=REQUESTS)
op.drop_table(REQUESTS)
if _has_table(CREDENTIALS):
op.drop_table(CREDENTIALS)

View File

@@ -0,0 +1,65 @@
"""One-line bootstrap install tokens.
Revision ID: endpoints0024bootstraptokens
Revises: endpoints0023credentials
Create Date: 2026-09-15
Creates endpointbootstraptokens: the short-lived, admin-minted authorizations
that let a tech fetch the bootstrap script and the keyless agent package with a
single command. Only the sha256 of the secret is stored.
Guarded so the revision is safe on a database built by create_all and on a
re-run. MySQL 5.6 safe: no JSON columns, and the only index is the unique key
on a fixed-length CHAR(64), well under the 767-byte limit on utf8mb4.
Downgrade drops the table.
"""
from alembic import op
import sqlalchemy as sa
revision = 'endpoints0024bootstraptokens'
down_revision = 'endpoints0023credentials'
branch_labels = None
depends_on = None
plugin_name = 'endpoints'
TOKENS = 'endpointbootstraptokens'
def _inspector():
from sqlalchemy import inspect
return inspect(op.get_bind())
def _has_table(name):
return _inspector().has_table(name)
def upgrade():
if _has_table(TOKENS):
return
op.create_table(
TOKENS,
sa.Column('tokenid', sa.Integer(), nullable=False),
sa.Column('tokenhash', sa.CHAR(length=64), nullable=False),
sa.Column('tokenprefix', sa.String(length=12), nullable=False),
sa.Column('createdat', sa.DateTime(), nullable=False),
sa.Column('createdby', sa.String(length=64), nullable=True),
sa.Column('expiresat', sa.DateTime(), nullable=False),
sa.Column('maxuses', sa.Integer(), nullable=False),
sa.Column('usecount', sa.Integer(), nullable=False,
server_default='0'),
sa.Column('revokedat', sa.DateTime(), nullable=True),
sa.Column('lastusedat', sa.DateTime(), nullable=True),
sa.Column('lastusedip', sa.String(length=45), nullable=True),
sa.PrimaryKeyConstraint('tokenid'),
sa.UniqueConstraint('tokenhash'),
)
def downgrade():
if _has_table(TOKENS):
op.drop_table(TOKENS)