Make main the lean product a site receives
main was 16 days and 259 commits behind the line the product is actually built on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It was neither current nor curated - just stale. This brings it forward and, at the same time, makes it mean something: main is what a receiving site installs, and nothing else. WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty - plus applications, which is manifest-less core and always ships. The only manifest dependency inside that set is endpoints on computers, and both are in it, so the closure is self-contained. WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb stay on the site line. So do the Windows installer tree, the VM rehearsal helpers, the agent tooling config, one site's data loader, the internal audit and findings records, and the planner spreadsheet. 1,135 tracked files against 1,438. A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped: deploy/site-profile-universal.json still named all seventeen plugins. It is the file docs/DEPLOY.md tells a site to provision from, so a fresh install would have failed at `flask plugin apply-profile` with PluginError - the first thing a receiving site does, broken by the branch meant to serve them. It now names the twelve that exist, and its comment says the two must agree. agent/ was never refreshed. Whoever populated this branch copied shopdb/, plugins/, frontend/ and tests/ from the site line and left the agent tree at its September state: 22 C# and PowerShell sources missing, including AgentCredentials, EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the site line had deleted still present. The old tree is internally consistent, so it compiles, and nothing in CI builds it - every gate was green over a server that speaks enrollment and per-PC credentials to a bundled agent source that cannot. Resynced whole. DECISIONS THIS ENCODES, rather than defers: - The plugin-authoring lab builds printedparts, whose manifest depends on employees, so it could not stay. The lab remains on the site line and main ships no build-along tutorial. That is a real loss for an adopting developer and is recorded here rather than discovered later. - Recognition, Training and Recertification notifications are no longer seeded. Each splits one notification into a card per employee and renders a photo, which needs a plugin this branch does not bundle. Seeding them would offer an author a type that cannot resolve anybody. A site that installs employees gets them from that plugin's own seed. - Windows/IIS is documented as the legacy path throughout, and the README leads with Docker, because that is what a receiving site gets. The API description is regenerated for THIS branch - 347 paths and 483 operations, down from 415 and 573 - so the published contract describes what is here rather than what the site line serves. docs/CONFIG.md lost the two measuring-tool label keys the code no longer declares, which two lanes had decided opposite ways. Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
@@ -13,8 +13,9 @@ so the catalog matches what a deployed agent already watches:
|
||||
process agent.exe -> MTConnect
|
||||
service MTConnect UDC Agent -> MTConnect
|
||||
|
||||
Shipped rows are product-owned (source='shipped'): the CRUD API refuses to edit
|
||||
or delete them. Launch paths mirror the agent's launchpaths block; collector and
|
||||
Shipped rows (source='shipped') keep their kind and displayname - that pair is
|
||||
the identity the monitor policies match on - but a site may edit their paths and
|
||||
active flag. Launch paths mirror the agent's launchpaths block; collector and
|
||||
log paths are left null until site paths are supplied.
|
||||
|
||||
Idempotent: creation checks the live schema first so a db.create_all() site
|
||||
|
||||
@@ -1,12 +1,22 @@
|
||||
"""Ship the shopfloor app monitors as a read-only Site Policy.
|
||||
"""Ship the shopfloor app monitors as a Site Policy bundle.
|
||||
|
||||
SUPERSEDED BY 0019. This migration is kept because installs have stamped it, but
|
||||
what it seeds no longer survives an upgrade: 0019 splits this single bundle into
|
||||
one Configuration Policy per monitor and REMOVES the shipped Site Policy, so
|
||||
sites compose their own. Read this for history; do not extend it.
|
||||
|
||||
Two claims below were true when written and are now false, corrected here rather
|
||||
than left to mislead: the seeded rows are NOT read-only (0019 admits 'shipped'
|
||||
at the policy CRUD guards), and no seeder reconciles them - the idempotency
|
||||
check at the top of upgrade() returns early if the policy exists, which is what
|
||||
keeps a site's edits safe across upgrades.
|
||||
|
||||
Wraps the four shipped endpointmonitors (eDNC, Universal Data Collector,
|
||||
MTConnect process, MTConnect UDC Agent service) in a product-owned policy bundle
|
||||
so the agent stops carrying them as hardcoded appsettings.json defaults and reads
|
||||
its watch-list from the server instead.
|
||||
MTConnect process, MTConnect UDC Agent service) in a policy bundle so the agent
|
||||
stops carrying them as hardcoded appsettings.json defaults and reads its
|
||||
watch-list from the server instead.
|
||||
|
||||
Seeds two computers-plugin rows, both source='shipped' (read-only in the UI, the
|
||||
CRUD API refuses to edit or delete them):
|
||||
Seeds two computers-plugin rows, both source='shipped':
|
||||
|
||||
ConfigurationPolicy "Shopfloor App Monitors"
|
||||
scope {"type":"true"} (every endpoint), one action per shipped monitor,
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Dedup curated Windows events on the Windows event record id.
|
||||
|
||||
Adds endpointevents.eventrecordid and a unique index over
|
||||
(assetid, eventchannel, eventrecordid).
|
||||
|
||||
Why: the agent used to re-read the newest window of every curated channel on
|
||||
each cadence tick, and this ingest appended those rows without a dedup check, so
|
||||
one real event was stored once per report for as long as it stayed in that
|
||||
window - up to 96 identical rows a day. The agent now carries a per-channel
|
||||
watermark keyed on the Windows record id, which is monotonic and unique within a
|
||||
channel on a machine; this index is the server-side guarantee that survives an
|
||||
agent reinstall, a restored image, or two overlapping reports, none of which
|
||||
preserve the agent's watermark.
|
||||
|
||||
Nullable on purpose: an agent older than this field reports no record id, and
|
||||
MySQL permits repeated NULLs inside a unique index, so those rows keep the
|
||||
previous append-only behavior instead of failing ingest.
|
||||
|
||||
Existing rows are left alone. They all predate the column and so carry NULL,
|
||||
which cannot collide with the new index. Purging the historical duplicates is a
|
||||
separate, deliberate data cleanup - not something a schema migration should do
|
||||
irreversibly.
|
||||
|
||||
Idempotent: both the column-add and the index-create check the live schema
|
||||
first, so a db.create_all() site and a re-run both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0012eventrecordid'
|
||||
down_revision = 'endpoints0011trayversion'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
INDEX_NAME = 'idx_endpointevents_assetrecord'
|
||||
|
||||
|
||||
def _has_column(table_name, column_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(col['name'] == column_name
|
||||
for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def _has_index(table_name, index_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(index['name'] == index_name
|
||||
for index in inspector.get_indexes(table_name))
|
||||
|
||||
|
||||
def upgrade():
|
||||
if not _has_column('endpointevents', 'eventrecordid'):
|
||||
op.add_column(
|
||||
'endpointevents',
|
||||
sa.Column('eventrecordid', sa.BigInteger(), nullable=True))
|
||||
|
||||
if not _has_index('endpointevents', INDEX_NAME):
|
||||
op.create_index(
|
||||
INDEX_NAME,
|
||||
'endpointevents',
|
||||
['assetid', 'eventchannel', 'eventrecordid'],
|
||||
unique=True)
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_index('endpointevents', INDEX_NAME):
|
||||
op.drop_index(INDEX_NAME, table_name='endpointevents')
|
||||
|
||||
if _has_column('endpointevents', 'eventrecordid'):
|
||||
op.drop_column('endpointevents', 'eventrecordid')
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Persist the hardware, OS, and session inventory the agent already reports.
|
||||
|
||||
Adds to endpointstates: disksjson, biosjson, memoryjson, processorjson,
|
||||
osinfojson, usercontextjson, domain, and pendingreboot.
|
||||
|
||||
Why now: the agent has been sending disks, bios, memory, processor, osinfo and
|
||||
domain on every report for several versions. The ingest logged them as unknown
|
||||
payload keys and threw them away, which was the intended staging step - the
|
||||
shapes are now proven against real endpoint traffic, so they get columns.
|
||||
usercontext and pendingreboot are new in this same change.
|
||||
|
||||
Shape follows the existing servicesjson/portsjson convention: newest snapshot
|
||||
per endpoint as MEDIUMTEXT, not one row per sample. The agent resends the full
|
||||
structure every cycle and operators ask "what is in this PC now", not "what was
|
||||
in it in March". History would be pure write amplification on a fleet-sized
|
||||
table.
|
||||
|
||||
domain and pendingreboot are real columns rather than JSON because they are
|
||||
single values the fleet view filters, groups, and alerts on; burying a scalar in
|
||||
a blob makes it unqueryable.
|
||||
|
||||
All columns are nullable. Every existing row predates them, and there is no
|
||||
correct backfill value - a machine whose reboot state has never been reported is
|
||||
NOT the same as one confirmed to have no reboot pending, and inventing False
|
||||
would silently create a fleet of "healthy" endpoints.
|
||||
|
||||
Idempotent: each column-add checks the live schema first, so a db.create_all()
|
||||
site and a re-run both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0013inventorycolumns'
|
||||
down_revision = 'endpoints0012eventrecordid'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
TABLE_NAME = 'endpointstates'
|
||||
|
||||
# MEDIUMTEXT. An installed-application or disk list on a busy engineering
|
||||
# workstation comfortably exceeds the 64KB TEXT ceiling, and a silent truncation
|
||||
# there would corrupt the JSON rather than merely shorten it.
|
||||
_SNAPSHOT_COLUMNS = (
|
||||
'disksjson',
|
||||
'biosjson',
|
||||
'memoryjson',
|
||||
'processorjson',
|
||||
'osinfojson',
|
||||
'usercontextjson',
|
||||
)
|
||||
|
||||
|
||||
def _has_column(table_name, column_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(col['name'] == column_name
|
||||
for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade():
|
||||
for column_name in _SNAPSHOT_COLUMNS:
|
||||
if not _has_column(TABLE_NAME, column_name):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
|
||||
|
||||
if not _has_column(TABLE_NAME, 'domain'):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column('domain', sa.String(length=255), nullable=True))
|
||||
|
||||
if not _has_column(TABLE_NAME, 'pendingreboot'):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column('pendingreboot', sa.Boolean(), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_column(TABLE_NAME, 'pendingreboot'):
|
||||
op.drop_column(TABLE_NAME, 'pendingreboot')
|
||||
|
||||
if _has_column(TABLE_NAME, 'domain'):
|
||||
op.drop_column(TABLE_NAME, 'domain')
|
||||
|
||||
for column_name in reversed(_SNAPSHOT_COLUMNS):
|
||||
if _has_column(TABLE_NAME, column_name):
|
||||
op.drop_column(TABLE_NAME, column_name)
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Add diskhealthjson to endpointstates.
|
||||
|
||||
Per-drive health, kept separate from disksjson. That column answers "how full
|
||||
is this drive"; this one answers "is this drive dying". Merging them would mean
|
||||
a capacity collection failure and a health collection failure could not be told
|
||||
apart, and the two come from different WMI sources with different failure modes.
|
||||
|
||||
JSON rather than scalar columns because a machine has a variable number of
|
||||
drives and each carries its own verdict, and because the agent reports the same
|
||||
drive from two sources - MSFT_PhysicalDisk for NVMe and Win32_DiskDrive for
|
||||
ATA/SATA - which the server keeps side by side rather than collapsing.
|
||||
|
||||
MEDIUMTEXT for consistency with the other snapshot columns on this table.
|
||||
|
||||
Nullable with no backfill. A drive whose health has never been reported is not
|
||||
a healthy drive, and defaulting to an empty list would read as "no disks have
|
||||
problems" on every endpoint that predates this column.
|
||||
|
||||
Idempotent: the column-add checks the live schema first, so a re-run and a
|
||||
db.create_all() site both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0014diskhealth'
|
||||
down_revision = 'endpoints0013inventorycolumns'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
TABLE_NAME = 'endpointstates'
|
||||
COLUMN_NAME = 'diskhealthjson'
|
||||
|
||||
|
||||
def _has_column(table_name, column_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(col['name'] == column_name
|
||||
for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade():
|
||||
if not _has_column(TABLE_NAME, COLUMN_NAME):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_column(TABLE_NAME, COLUMN_NAME):
|
||||
op.drop_column(TABLE_NAME, COLUMN_NAME)
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Add rebootreasonsjson, bootinfojson and defenderjson to endpointstates.
|
||||
|
||||
Phase 2 system-side telemetry. All three are collect-and-persist; none is
|
||||
surfaced in the UI by this change.
|
||||
|
||||
rebootreasonsjson is NOT redundant with the existing pendingreboot column. That
|
||||
column is a single boolean OR-ed from three registry signals, so it cannot say
|
||||
which one fired, and the difference is operationally significant: a
|
||||
PendingFileRenameOperations entry means files are staged for replacement on the
|
||||
next boot, so the machine is mid-servicing, whereas a lone Component Based
|
||||
Servicing or Windows Update key can sit set for weeks on a box nobody has
|
||||
rebooted. Same boolean, very different urgency.
|
||||
|
||||
bootinfojson exists because uptime lies under fast startup - a machine reports
|
||||
days of uptime after what the operator experienced as a shutdown, and a "reboot"
|
||||
that was really a hybrid resume does not clear the driver or kernel state a
|
||||
support call is usually trying to clear.
|
||||
|
||||
defenderjson is stored deliberately unsurfaced. Defender is known to produce
|
||||
false positives in this environment, so wiring it to a card now would train
|
||||
operators to ignore that card; persisting it starts the history accumulating for
|
||||
whenever someone is ready to interpret it.
|
||||
|
||||
JSON rather than scalars: reboot reasons is a variable-length list, and the boot
|
||||
and Defender payloads are small related groups that are read together and would
|
||||
otherwise be six sparse columns.
|
||||
|
||||
MEDIUMTEXT for consistency with the other snapshot columns on this table.
|
||||
|
||||
Nullable with no backfill. Null means never reported, which is deliberately
|
||||
distinct from an empty list (checked, nothing pending) - defaulting the reboot
|
||||
reasons to [] would assert every pre-existing endpoint was confirmed clean.
|
||||
|
||||
Idempotent: each column-add checks the live schema first, so a re-run and a
|
||||
db.create_all() site both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0015systemtelemetry'
|
||||
down_revision = 'endpoints0014diskhealth'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
TABLE_NAME = 'endpointstates'
|
||||
|
||||
_COLUMNS = (
|
||||
'rebootreasonsjson',
|
||||
'bootinfojson',
|
||||
'defenderjson',
|
||||
)
|
||||
|
||||
|
||||
def _has_column(table_name, column_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(col['name'] == column_name
|
||||
for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade():
|
||||
for column_name in _COLUMNS:
|
||||
if not _has_column(TABLE_NAME, column_name):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column(column_name, sa.Text(length=16777215), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
for column_name in reversed(_COLUMNS):
|
||||
if _has_column(TABLE_NAME, column_name):
|
||||
op.drop_column(TABLE_NAME, column_name)
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Add dellupdatesjson to endpointstates.
|
||||
|
||||
Stores the pending Dell Command | Update summary reported by the
|
||||
shopdb-dellupdates agent utility. MEDIUMTEXT for consistency with the other
|
||||
snapshot columns on this table.
|
||||
|
||||
Nullable with no backfill. Null means the endpoint has never reported a Dell
|
||||
scan result, which is deliberately distinct from a stored payload whose status
|
||||
is 'ok' with an empty list (scanned, nothing pending). The status card renders
|
||||
those two states differently, so defaulting this to an empty summary would
|
||||
assert every pre-existing endpoint had been scanned and found clean.
|
||||
|
||||
Idempotent: the column-add checks the live schema first, so a re-run and a
|
||||
db.create_all() site both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0016dellupdates'
|
||||
down_revision = 'endpoints0015systemtelemetry'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
TABLE_NAME = 'endpointstates'
|
||||
|
||||
COLUMN_NAME = 'dellupdatesjson'
|
||||
|
||||
|
||||
def _has_column(table_name, column_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
inspector = inspect(op.get_bind())
|
||||
if not inspector.has_table(table_name):
|
||||
return False
|
||||
return any(col['name'] == column_name
|
||||
for col in inspector.get_columns(table_name))
|
||||
|
||||
|
||||
def upgrade():
|
||||
if not _has_column(TABLE_NAME, COLUMN_NAME):
|
||||
op.add_column(
|
||||
TABLE_NAME,
|
||||
sa.Column(COLUMN_NAME, sa.Text(length=16777215), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_column(TABLE_NAME, COLUMN_NAME):
|
||||
op.drop_column(TABLE_NAME, COLUMN_NAME)
|
||||
@@ -0,0 +1,133 @@
|
||||
"""Add endpoint health rollup tables.
|
||||
|
||||
Three tables behind the Fleet Health column:
|
||||
|
||||
endpointhealthstates one cached verdict per endpoint, read directly by
|
||||
the Fleet list instead of recomputing a 48h
|
||||
fleet-wide event aggregate on every page load
|
||||
endpointhealthtransitions capped history, written on CHANGE not on sample
|
||||
endpointmonitorimpacts per-PC-type operational impact for a monitor
|
||||
|
||||
No backfill. An endpoint with no row has simply not checked in since this
|
||||
shipped, and the first check-in writes one. Backfilling would have to invent a
|
||||
firstseenat for every existing condition, and a fabricated "broken since" is
|
||||
worse than an absent one - an operator reading it has no way to tell it apart
|
||||
from an observed timestamp.
|
||||
|
||||
endpointmonitorimpacts ships EMPTY on purpose. Absence of a row means the
|
||||
built-in default impact applies, so an endpoint with no policies behaves
|
||||
exactly as it did before this migration. Seeding defaults here would freeze
|
||||
today's guesses into rows nobody could distinguish from deliberate choices.
|
||||
|
||||
Idempotent: every create checks the live schema first, so a re-run and a
|
||||
db.create_all() site both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0017healthrollup'
|
||||
down_revision = 'endpoints0016dellupdates'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
HEALTHSTATES = 'endpointhealthstates'
|
||||
TRANSITIONS = 'endpointhealthtransitions'
|
||||
IMPACTS = 'endpointmonitorimpacts'
|
||||
|
||||
|
||||
def _has_table(table_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return inspect(op.get_bind()).has_table(table_name)
|
||||
|
||||
|
||||
def upgrade():
|
||||
if not _has_table(HEALTHSTATES):
|
||||
op.create_table(
|
||||
HEALTHSTATES,
|
||||
sa.Column('endpointhealthstateid', sa.Integer(), nullable=False),
|
||||
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||
sa.Column('level', sa.String(length=16), nullable=False,
|
||||
server_default='unknown'),
|
||||
sa.Column('label', sa.String(length=255), nullable=True),
|
||||
sa.Column('impactclass', sa.String(length=16), nullable=True),
|
||||
sa.Column('sourcekind', sa.String(length=16), nullable=False,
|
||||
server_default='none'),
|
||||
sa.Column('sourceref', sa.String(length=64), nullable=True),
|
||||
sa.Column('firstseenat', sa.DateTime(), nullable=True),
|
||||
sa.Column('lastconfirmedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('lastevaluatedat', sa.DateTime(), nullable=False),
|
||||
sa.Column('flapcount', sa.Integer(), nullable=False,
|
||||
server_default='0'),
|
||||
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||
ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('endpointhealthstateid'),
|
||||
# One live verdict per endpoint. Unique rather than indexed: two
|
||||
# rows for one asset is not a state this table should represent.
|
||||
sa.UniqueConstraint('assetid',
|
||||
name='uq_endpointhealthstates_assetid'),
|
||||
)
|
||||
op.create_index('idx_endpointhealthstates_asset', HEALTHSTATES,
|
||||
['assetid'])
|
||||
op.create_index('idx_endpointhealthstates_impactclass', HEALTHSTATES,
|
||||
['impactclass'])
|
||||
|
||||
if not _has_table(TRANSITIONS):
|
||||
op.create_table(
|
||||
TRANSITIONS,
|
||||
sa.Column('endpointhealthtransitionid', sa.Integer(),
|
||||
nullable=False),
|
||||
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||
sa.Column('level', sa.String(length=16), nullable=False),
|
||||
sa.Column('label', sa.String(length=255), nullable=True),
|
||||
sa.Column('impactclass', sa.String(length=16), nullable=True),
|
||||
sa.Column('sourcekind', sa.String(length=16), nullable=False,
|
||||
server_default='none'),
|
||||
sa.Column('sourceref', sa.String(length=64), nullable=True),
|
||||
sa.Column('openedat', sa.DateTime(), nullable=False),
|
||||
sa.Column('resolvedat', sa.DateTime(), nullable=True),
|
||||
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||
ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('endpointhealthtransitionid'),
|
||||
)
|
||||
# Retention reads newest-first per asset; the resolved index serves the
|
||||
# "never evict an unresolved row" filter without scanning history.
|
||||
op.create_index('idx_endpointhealthtransitions_assetopened',
|
||||
TRANSITIONS, ['assetid', 'openedat'])
|
||||
op.create_index('idx_endpointhealthtransitions_assetresolved',
|
||||
TRANSITIONS, ['assetid', 'resolvedat'])
|
||||
|
||||
if not _has_table(IMPACTS):
|
||||
op.create_table(
|
||||
IMPACTS,
|
||||
sa.Column('endpointmonitorimpactid', sa.Integer(), nullable=False),
|
||||
sa.Column('endpointmonitorid', sa.Integer(), nullable=False),
|
||||
# Plain integer, not a ForeignKey: computertypes belongs to the
|
||||
# computers plugin and owns its own migration chain. A cross-plugin
|
||||
# FK would couple the two chains' ordering for no benefit the
|
||||
# unique constraint does not already provide.
|
||||
sa.Column('computertypeid', sa.Integer(), nullable=False),
|
||||
sa.Column('impactclass', sa.String(length=16), nullable=False,
|
||||
server_default='degraded'),
|
||||
sa.Column('createdat', sa.DateTime(), nullable=False),
|
||||
sa.ForeignKeyConstraint(['endpointmonitorid'],
|
||||
['endpointmonitors.endpointmonitorid'],
|
||||
ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('endpointmonitorimpactid'),
|
||||
sa.UniqueConstraint('endpointmonitorid', 'computertypeid',
|
||||
name='uq_endpointmonitorimpacts_monitortype'),
|
||||
)
|
||||
op.create_index('idx_endpointmonitorimpacts_computertypeid', IMPACTS,
|
||||
['computertypeid'])
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Dropped newest-first so the FK to endpointmonitors goes before anything
|
||||
# that might reference it. All three are caches or mappings - nothing here
|
||||
# is a system of record, so a downgrade loses no telemetry.
|
||||
for table_name in (IMPACTS, TRANSITIONS, HEALTHSTATES):
|
||||
if _has_table(table_name):
|
||||
op.drop_table(table_name)
|
||||
87
plugins/endpoints/migrations/versions/0018_endpoint_locks.py
Normal file
87
plugins/endpoints/migrations/versions/0018_endpoint_locks.py
Normal file
@@ -0,0 +1,87 @@
|
||||
"""Add endpointlocks (admin-only endpoint quarantine).
|
||||
|
||||
One table behind the endpoint lock: an admin-set refusal of operator commands
|
||||
on a single endpoint, independent of policy and permission.
|
||||
|
||||
Deliberately separate from the computers plugin's policyactionholds. That table
|
||||
suppresses ONE desired-state action on the policy-resolution axis and expires on
|
||||
a timer; this one refuses NON-ADMIN CALLERS on the whole endpoint and does not
|
||||
expire at all. Two concepts, two tables, two enforcement points - collapsing
|
||||
them would recreate the axis-mixing the surrounding refactor exists to undo.
|
||||
|
||||
No expiresat column, and that is the design rather than an omission. A lock
|
||||
stands for a physical-world condition (machine mid-repair, validated part
|
||||
running, box that must not reboot during an audit) and those do not end on a
|
||||
schedule. A TTL would silently unlock a machine someone still has their hands
|
||||
inside.
|
||||
|
||||
Ships EMPTY. No endpoint has ever been locked before this migration, so there is
|
||||
nothing to backfill and every endpoint keeps behaving exactly as it did.
|
||||
|
||||
Idempotent: the create checks the live schema first, so a re-run and a
|
||||
db.create_all() site both stay clean on strict MySQL 8.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'endpoints0018endpointlocks'
|
||||
down_revision = 'endpoints0017healthrollup'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
LOCKS = 'endpointlocks'
|
||||
|
||||
|
||||
def _has_table(table_name):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return inspect(op.get_bind()).has_table(table_name)
|
||||
|
||||
|
||||
def upgrade():
|
||||
if not _has_table(LOCKS):
|
||||
op.create_table(
|
||||
LOCKS,
|
||||
sa.Column('endpointlockid', sa.Integer(), nullable=False),
|
||||
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||
sa.Column('reason', sa.Text(), nullable=False),
|
||||
sa.Column('lockedbyuserid', sa.Integer(), nullable=True),
|
||||
sa.Column('lockedbyusername', sa.String(length=100), nullable=True),
|
||||
sa.Column('lockedat', sa.DateTime(), nullable=False),
|
||||
# Null while the lock stands. Setting it is what releases the lock,
|
||||
# so there is no separate active flag able to drift out of step.
|
||||
sa.Column('releasedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('releasedbyuserid', sa.Integer(), nullable=True),
|
||||
sa.Column('releasedbyusername', sa.String(length=100),
|
||||
nullable=True),
|
||||
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||
ondelete='CASCADE'),
|
||||
# SET NULL, not CASCADE: deleting the admin who locked a machine
|
||||
# must not delete the record that it was locked. The username
|
||||
# snapshot columns survive to say who it was.
|
||||
sa.ForeignKeyConstraint(['lockedbyuserid'], ['users.userid'],
|
||||
ondelete='SET NULL'),
|
||||
sa.ForeignKeyConstraint(['releasedbyuserid'], ['users.userid'],
|
||||
ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('endpointlockid'),
|
||||
)
|
||||
# Serves the hot path: "is there an active lock on this asset", asked on
|
||||
# every enqueue and every endpoint detail load. NOT unique on assetid -
|
||||
# released rows accumulate per endpoint as history, and a unique
|
||||
# constraint would make the second lock in a machine's life fail. The
|
||||
# one-active-lock invariant is enforced on write instead.
|
||||
op.create_index('idx_endpointlocks_assetactive', LOCKS,
|
||||
['assetid', 'releasedat'])
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Safe to drop: the table is a system of record for quarantine state, but a
|
||||
# downgrade past this revision removes the enforcement that reads it, so
|
||||
# leaving orphaned rows behind would be worse - they would silently reapply
|
||||
# if the revision were ever re-applied.
|
||||
if _has_table(LOCKS):
|
||||
op.drop_index('idx_endpointlocks_assetactive', table_name=LOCKS)
|
||||
op.drop_table(LOCKS)
|
||||
@@ -0,0 +1,320 @@
|
||||
"""Split the shipped monitor bundle into one policy per monitor.
|
||||
|
||||
0007 seeded a single Configuration Policy "Shopfloor App Monitors" carrying all
|
||||
four shipped monitor actions, wrapped in a shipped Site (business) Policy of the
|
||||
same name. That bundle was all-or-nothing: a site that runs eDNC but no
|
||||
MTConnect could not detach the half it does not have, because the whole row was
|
||||
product-owned and read-only.
|
||||
|
||||
This migration replaces it with one Configuration Policy per shipped monitor:
|
||||
|
||||
Monitor: eDNC (process)
|
||||
Monitor: Universal Data Collector (process)
|
||||
Monitor: MTConnect (process)
|
||||
Monitor: MTConnect (service)
|
||||
|
||||
Each carries exactly one 'monitor:<endpointmonitorid>' action, mode='gate',
|
||||
scope {"type":"true"} - no PC type attribution. The name carries the kind
|
||||
because displayname alone is not unique: MTConnect ships as both a process and
|
||||
a service, and configurationpolicies.name is UNIQUE.
|
||||
|
||||
source stays 'shipped'. Provenance is still true - the product did ship these -
|
||||
and the UI keeps showing where they came from. What changes is that the CRUD
|
||||
guards now admit 'shipped' for edit/delete, so a site can retune or drop one
|
||||
without losing the record of its origin.
|
||||
|
||||
The shipped SITE policy is REMOVED and not replaced. Composition is a site
|
||||
decision: which monitors apply to which endpoints, and at what priority, is
|
||||
exactly the judgement a plant makes about its own floor. Shipping one meant
|
||||
every install inherited "apply all four to every endpoint" as a starting point
|
||||
that could not be edited. Sites now build their own from the per-monitor pieces.
|
||||
|
||||
Removing it is the behaviour change to be aware of on upgrade: monitors that
|
||||
were attached through that bundle stop being attached until a site policy
|
||||
composes them. It is deliberate and it is the point of the split - see the
|
||||
"off by default" direction in the 2026-09-08 plan - but it is not silent, since
|
||||
the per-monitor policies are listed and ready to compose.
|
||||
|
||||
Idempotent on the policy names, so a re-run neither duplicates the split rows
|
||||
nor resurrects the bundle. Guards on the policy tables being present and no-ops
|
||||
if they are not, matching 0007.
|
||||
|
||||
Revision ID: endpoints0019splitpolicies
|
||||
Revises: endpoints0018endpointlocks
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
# Alembic stores this in alembic_version.version_num, a VARCHAR(32). A longer
|
||||
# id passes on SQLite and fails on strict MySQL with "Data too long".
|
||||
revision = 'endpoints0019splitpolicies'
|
||||
down_revision = 'endpoints0018endpointlocks'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
BUNDLE_NAME = 'Shopfloor App Monitors'
|
||||
|
||||
# The shipped monitors, by identity (kind, displayname) - never by hardcoded id,
|
||||
# so this survives id drift across installs exactly as 0007 does.
|
||||
_SHIPPED_MONITORS = (
|
||||
('process', 'eDNC'),
|
||||
('process', 'Universal Data Collector'),
|
||||
('process', 'MTConnect'),
|
||||
('service', 'MTConnect'),
|
||||
)
|
||||
|
||||
_REQUIRED = {'endpointmonitors', 'configurationpolicies',
|
||||
'configurationpolicyactions', 'businesspolicies',
|
||||
'businesspolicyitems'}
|
||||
|
||||
|
||||
def _tables(bind):
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return set(inspect(bind).get_table_names())
|
||||
|
||||
|
||||
def _policy_name(kind, displayname):
|
||||
return f'Monitor: {displayname} ({kind})'
|
||||
|
||||
|
||||
def _policy_description(kind, displayname):
|
||||
noun = 'process' if kind == 'process' else 'service'
|
||||
return (f'Product-shipped watch-list entry for the {displayname} {noun}. '
|
||||
f'Attach it to a Site Policy to apply it to endpoints.')
|
||||
|
||||
|
||||
def _config_policy_id(bind, name):
|
||||
return bind.execute(
|
||||
sa.text('SELECT configurationpolicyid FROM configurationpolicies '
|
||||
'WHERE name = :name'),
|
||||
{'name': name}).scalar()
|
||||
|
||||
|
||||
def _drop_config_policy(bind, name):
|
||||
"""Delete a configuration policy, its actions, and any items citing it."""
|
||||
policyid = _config_policy_id(bind, name)
|
||||
if policyid is None:
|
||||
return
|
||||
# Clear children explicitly rather than lean on ON DELETE CASCADE, which
|
||||
# sqlite does not enforce by default.
|
||||
bind.execute(
|
||||
sa.text('DELETE FROM businesspolicyitems '
|
||||
'WHERE configurationpolicyid = :id'), {'id': policyid})
|
||||
bind.execute(
|
||||
sa.text('DELETE FROM configurationpolicyactions '
|
||||
'WHERE configurationpolicyid = :id'), {'id': policyid})
|
||||
bind.execute(
|
||||
sa.text('DELETE FROM configurationpolicies '
|
||||
'WHERE configurationpolicyid = :id'), {'id': policyid})
|
||||
|
||||
|
||||
def _drop_business_policy(bind, name):
|
||||
policyid = bind.execute(
|
||||
sa.text('SELECT businesspolicyid FROM businesspolicies '
|
||||
'WHERE name = :name AND source = :source'),
|
||||
{'name': name, 'source': 'shipped'}).scalar()
|
||||
if policyid is None:
|
||||
return
|
||||
bind.execute(
|
||||
sa.text('DELETE FROM businesspolicyitems WHERE businesspolicyid = :id'),
|
||||
{'id': policyid})
|
||||
bind.execute(
|
||||
sa.text('DELETE FROM businesspolicies WHERE businesspolicyid = :id'),
|
||||
{'id': policyid})
|
||||
|
||||
|
||||
def _monitor_id(bind, kind, displayname):
|
||||
return bind.execute(
|
||||
sa.text('SELECT endpointmonitorid FROM endpointmonitors '
|
||||
'WHERE kind = :kind AND displayname = :displayname'),
|
||||
{'kind': kind, 'displayname': displayname}).scalar()
|
||||
|
||||
|
||||
def upgrade():
|
||||
bind = op.get_bind()
|
||||
if not _REQUIRED.issubset(_tables(bind)):
|
||||
return
|
||||
|
||||
now = datetime.utcnow()
|
||||
|
||||
config_policies = sa.table(
|
||||
'configurationpolicies',
|
||||
sa.column('name', sa.String),
|
||||
sa.column('description', sa.Text),
|
||||
sa.column('scopejson', sa.Text),
|
||||
sa.column('source', sa.String),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
actions = sa.table(
|
||||
'configurationpolicyactions',
|
||||
sa.column('configurationpolicyid', sa.Integer),
|
||||
sa.column('action', sa.String),
|
||||
sa.column('mode', sa.String),
|
||||
sa.column('desiredstate', sa.Text),
|
||||
sa.column('params', sa.Text),
|
||||
sa.column('intervalminutes', sa.Integer),
|
||||
sa.column('sortorder', sa.Integer),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
|
||||
for kind, displayname in _SHIPPED_MONITORS:
|
||||
name = _policy_name(kind, displayname)
|
||||
if _config_policy_id(bind, name) is not None:
|
||||
continue
|
||||
monitorid = _monitor_id(bind, kind, displayname)
|
||||
if monitorid is None:
|
||||
# A site that never had this monitor gets no policy for it.
|
||||
continue
|
||||
|
||||
op.bulk_insert(config_policies, [{
|
||||
'name': name,
|
||||
'description': _policy_description(kind, displayname),
|
||||
'scopejson': '{"type": "true"}',
|
||||
'source': 'shipped',
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
}])
|
||||
op.bulk_insert(actions, [{
|
||||
'configurationpolicyid': _config_policy_id(bind, name),
|
||||
'action': f'monitor:{monitorid}',
|
||||
'mode': 'gate',
|
||||
'desiredstate': None,
|
||||
'params': None,
|
||||
'intervalminutes': None,
|
||||
'sortorder': 0,
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
}])
|
||||
|
||||
# Retire the monolith. The business policy goes first so its item rows are
|
||||
# gone before the configuration policy it cited is removed.
|
||||
_drop_business_policy(bind, BUNDLE_NAME)
|
||||
_drop_config_policy(bind, BUNDLE_NAME)
|
||||
|
||||
|
||||
def downgrade():
|
||||
"""Reinstate the single shipped bundle and drop the per-monitor policies.
|
||||
|
||||
Rebuilt from the same monitor identities rather than from anything the
|
||||
upgrade stashed, so it lands in the state 0007 produces regardless of what
|
||||
a site did to the split rows in between.
|
||||
"""
|
||||
bind = op.get_bind()
|
||||
if not _REQUIRED.issubset(_tables(bind)):
|
||||
return
|
||||
|
||||
now = datetime.utcnow()
|
||||
|
||||
monitor_ids = []
|
||||
for kind, displayname in _SHIPPED_MONITORS:
|
||||
monitorid = _monitor_id(bind, kind, displayname)
|
||||
if monitorid is not None:
|
||||
monitor_ids.append(monitorid)
|
||||
|
||||
for kind, displayname in _SHIPPED_MONITORS:
|
||||
_drop_config_policy(bind, _policy_name(kind, displayname))
|
||||
|
||||
if not monitor_ids or _config_policy_id(bind, BUNDLE_NAME) is not None:
|
||||
return
|
||||
|
||||
config_policies = sa.table(
|
||||
'configurationpolicies',
|
||||
sa.column('name', sa.String),
|
||||
sa.column('description', sa.Text),
|
||||
sa.column('scopejson', sa.Text),
|
||||
sa.column('source', sa.String),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
op.bulk_insert(config_policies, [{
|
||||
'name': BUNDLE_NAME,
|
||||
'description': 'Product-shipped watch-list for the shopfloor apps '
|
||||
'(eDNC, Universal Data Collector, MTConnect). Read-only.',
|
||||
'scopejson': '{"type": "true"}',
|
||||
'source': 'shipped',
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
}])
|
||||
config_policy_id = _config_policy_id(bind, BUNDLE_NAME)
|
||||
|
||||
actions = sa.table(
|
||||
'configurationpolicyactions',
|
||||
sa.column('configurationpolicyid', sa.Integer),
|
||||
sa.column('action', sa.String),
|
||||
sa.column('mode', sa.String),
|
||||
sa.column('desiredstate', sa.Text),
|
||||
sa.column('params', sa.Text),
|
||||
sa.column('intervalminutes', sa.Integer),
|
||||
sa.column('sortorder', sa.Integer),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
op.bulk_insert(actions, [{
|
||||
'configurationpolicyid': config_policy_id,
|
||||
'action': f'monitor:{monitorid}',
|
||||
'mode': 'gate',
|
||||
'desiredstate': None,
|
||||
'params': None,
|
||||
'intervalminutes': None,
|
||||
'sortorder': index,
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
} for index, monitorid in enumerate(monitor_ids)])
|
||||
|
||||
business_policies = sa.table(
|
||||
'businesspolicies',
|
||||
sa.column('name', sa.String),
|
||||
sa.column('description', sa.Text),
|
||||
sa.column('priority', sa.Integer),
|
||||
sa.column('source', sa.String),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
op.bulk_insert(business_policies, [{
|
||||
'name': BUNDLE_NAME,
|
||||
'description': 'Product-shipped Site Policy that applies the shopfloor '
|
||||
'app monitors to every endpoint. Read-only.',
|
||||
'priority': 100,
|
||||
'source': 'shipped',
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
}])
|
||||
business_policy_id = bind.execute(
|
||||
sa.text('SELECT businesspolicyid FROM businesspolicies '
|
||||
'WHERE name = :name'), {'name': BUNDLE_NAME}).scalar()
|
||||
|
||||
items = sa.table(
|
||||
'businesspolicyitems',
|
||||
sa.column('businesspolicyid', sa.Integer),
|
||||
sa.column('configurationpolicyid', sa.Integer),
|
||||
sa.column('sortorder', sa.Integer),
|
||||
sa.column('createddate', sa.DateTime),
|
||||
sa.column('modifieddate', sa.DateTime),
|
||||
sa.column('isactive', sa.Boolean),
|
||||
)
|
||||
op.bulk_insert(items, [{
|
||||
'businesspolicyid': business_policy_id,
|
||||
'configurationpolicyid': config_policy_id,
|
||||
'sortorder': 0,
|
||||
'createddate': now,
|
||||
'modifieddate': now,
|
||||
'isactive': True,
|
||||
}])
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Add fingerprint column for delta-based application updates
|
||||
|
||||
Revision ID: endpoints0020appfingerprints
|
||||
Revises: endpoints0019splitpolicies
|
||||
Create Date: 2026-09-14
|
||||
|
||||
Performance optimization: Add appfingerprint column to endpointapplications
|
||||
table to enable delta-based updates instead of delete-all-insert-all pattern.
|
||||
|
||||
Impact:
|
||||
- Before: 200 endpoints x 50 apps x 4 reports/hour = 40,000 DELETEs + 40,000 INSERTs/hour
|
||||
- After: Only changed apps are touched (~2% per day) = 40 DELETEs + 40 INSERTs/hour
|
||||
- Reduction: 99.9% fewer writes (1.92M writes/day -> 1,920 writes/day)
|
||||
|
||||
The fingerprint is SHA256(displayname|displayversion|publisher) computed by
|
||||
the agent and sent in the collector payload. Nullable during rollout - legacy
|
||||
agents without fingerprint support fall back to delete-all-insert-all behavior.
|
||||
|
||||
DEPLOYMENT SEQUENCE (CRITICAL):
|
||||
1. Deploy this migration: flask db upgrade
|
||||
2. Run backfill script: python scripts/backfill_app_fingerprints.py
|
||||
3. Deploy updated plugin.py (delta logic code)
|
||||
4. Deploy updated C# agent (fingerprint computation)
|
||||
|
||||
ROLLBACK PROCEDURE:
|
||||
1. Deploy old plugin.py FIRST (remove delta logic)
|
||||
2. THEN run: flask db downgrade -1
|
||||
3. Verify: SELECT COUNT(*) FROM endpointapplications (should be unchanged)
|
||||
|
||||
WARNING: Do NOT downgrade migration before reverting code - will cause AttributeError!
|
||||
|
||||
SAFETY FEATURES:
|
||||
- Row-level locking prevents concurrent update races
|
||||
- Fingerprint format validation (64 hex chars)
|
||||
- Delete count verification detects silent failures
|
||||
- Memory limits prevent pathological cases (>5000 apps)
|
||||
- Metrics logged for monitoring
|
||||
|
||||
Idempotent: a database that already has the column (create_all built it from
|
||||
the model, or an earlier build of this revision added it as app_fingerprint)
|
||||
is left alone. 0023 renames the old spelling.
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = 'endpoints0020appfingerprints'
|
||||
down_revision = 'endpoints0019splitpolicies'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
plugin_name = 'endpoints'
|
||||
|
||||
TABLE = 'endpointapplications'
|
||||
INDEX = 'idx_endpointapplications_fingerprint'
|
||||
|
||||
|
||||
def _columns():
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return {c['name'] for c in inspect(op.get_bind()).get_columns(TABLE)}
|
||||
|
||||
|
||||
def _indexes():
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return {ix['name'] for ix in inspect(op.get_bind()).get_indexes(TABLE)}
|
||||
|
||||
|
||||
def upgrade():
|
||||
columns = _columns()
|
||||
# Old spelling present: leave it for 0023 to rename.
|
||||
if 'appfingerprint' in columns or 'app_fingerprint' in columns:
|
||||
return
|
||||
op.add_column(TABLE, sa.Column('appfingerprint', sa.String(64),
|
||||
nullable=True))
|
||||
if INDEX not in _indexes():
|
||||
# Unique per asset: the delta logic keys rows on it.
|
||||
op.create_index(INDEX, TABLE, ['assetid', 'appfingerprint'],
|
||||
unique=True)
|
||||
|
||||
|
||||
def downgrade():
|
||||
if INDEX in _indexes():
|
||||
op.drop_index(INDEX, table_name=TABLE)
|
||||
columns = _columns()
|
||||
for name in ('appfingerprint', 'app_fingerprint'):
|
||||
if name in columns:
|
||||
with op.batch_alter_table(TABLE) as batch:
|
||||
batch.drop_column(name)
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Retired: client certificate columns on assets.
|
||||
|
||||
Revision ID: endpoints0021clientcerts
|
||||
Revises: endpoints0020appfingerprints
|
||||
Create Date: 2026-09-14
|
||||
|
||||
This revision once added client certificate columns to the core assets table.
|
||||
Client certificate authentication was replaced by per-PC endpoint credentials
|
||||
(0023), so the revision is kept only so databases already stamped with it
|
||||
still resolve. It does nothing; 0023 drops the columns wherever an earlier
|
||||
build of this revision created them.
|
||||
"""
|
||||
|
||||
revision = 'endpoints0021clientcerts'
|
||||
down_revision = 'endpoints0020appfingerprints'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
plugin_name = 'endpoints'
|
||||
|
||||
|
||||
def upgrade():
|
||||
pass
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Retired: client certificate revocation columns on assets.
|
||||
|
||||
Revision ID: endpoints0022certrevocation
|
||||
Revises: endpoints0021clientcerts
|
||||
Create Date: 2026-09-14
|
||||
|
||||
Kept only so databases already stamped with this revision still resolve. It
|
||||
does nothing; 0023 drops the columns wherever an earlier build of this
|
||||
revision created them.
|
||||
"""
|
||||
|
||||
revision = 'endpoints0022certrevocation'
|
||||
down_revision = 'endpoints0021clientcerts'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
plugin_name = 'endpoints'
|
||||
|
||||
|
||||
def upgrade():
|
||||
pass
|
||||
|
||||
|
||||
def downgrade():
|
||||
pass
|
||||
@@ -0,0 +1,181 @@
|
||||
"""Per-PC endpoint credentials; remove the client certificate leftovers.
|
||||
|
||||
Revision ID: endpoints0023credentials
|
||||
Revises: endpoints0022certrevocation
|
||||
Create Date: 2026-09-15
|
||||
|
||||
Three things, each guarded against the live schema so the revision is safe on
|
||||
a database built by create_all, on one that ran the earlier client certificate
|
||||
builds of 0021/0022, and on a re-run:
|
||||
|
||||
1. Drop the client certificate columns (and their index) from the core assets
|
||||
table where an earlier build of 0021/0022 created them.
|
||||
2. Rename endpointapplications.app_fingerprint to appfingerprint where an
|
||||
earlier build of 0020 used the underscore spelling.
|
||||
3. Create endpointcredentials (with the reenrollblocked flag and its reason),
|
||||
endpointenrollrequests and endpointmigrationwaves.
|
||||
|
||||
MySQL 5.6 safe: no JSON columns, and the enroll request lookup index uses
|
||||
prefix lengths so the key stays under 767 bytes on utf8mb4.
|
||||
|
||||
Downgrade drops the three tables. The certificate columns are not restored:
|
||||
the code that read them is gone.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = 'endpoints0023credentials'
|
||||
down_revision = 'endpoints0022certrevocation'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
plugin_name = 'endpoints'
|
||||
|
||||
CREDENTIALS = 'endpointcredentials'
|
||||
REQUESTS = 'endpointenrollrequests'
|
||||
WAVES = 'endpointmigrationwaves'
|
||||
|
||||
_CERT_COLUMNS = ('client_cert_fingerprint', 'cert_issued_at', 'cert_expires_at',
|
||||
'cert_revoked_at', 'cert_revocation_reason')
|
||||
_APPS = 'endpointapplications'
|
||||
_FINGERPRINT_INDEX = 'idx_endpointapplications_fingerprint'
|
||||
|
||||
|
||||
def _inspector():
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return inspect(op.get_bind())
|
||||
|
||||
|
||||
def _has_table(name):
|
||||
return _inspector().has_table(name)
|
||||
|
||||
|
||||
def _columns(table):
|
||||
return {c['name'] for c in _inspector().get_columns(table)}
|
||||
|
||||
|
||||
def _indexes(table):
|
||||
return _inspector().get_indexes(table)
|
||||
|
||||
|
||||
def _drop_certificate_columns():
|
||||
if not _has_table('assets'):
|
||||
return
|
||||
present = _columns('assets') & set(_CERT_COLUMNS)
|
||||
if not present:
|
||||
return
|
||||
# Any index over a certificate column goes first: 0021 named its own,
|
||||
# create_all names by convention, and SQLite refuses to drop an indexed column.
|
||||
for index in _indexes('assets'):
|
||||
if set(index.get('column_names') or ()) & present:
|
||||
op.drop_index(index['name'], table_name='assets')
|
||||
for column in _CERT_COLUMNS:
|
||||
if column in present:
|
||||
op.drop_column('assets', column)
|
||||
|
||||
|
||||
def _rename_fingerprint():
|
||||
if not _has_table(_APPS):
|
||||
return
|
||||
columns = _columns(_APPS)
|
||||
if 'app_fingerprint' not in columns:
|
||||
return
|
||||
if 'appfingerprint' in columns:
|
||||
return
|
||||
for index in _indexes(_APPS):
|
||||
if 'app_fingerprint' in (index.get('column_names') or ()):
|
||||
op.drop_index(index['name'], table_name=_APPS)
|
||||
op.alter_column(_APPS, 'app_fingerprint', new_column_name='appfingerprint',
|
||||
existing_type=sa.String(64), existing_nullable=True)
|
||||
op.create_index(_FINGERPRINT_INDEX, _APPS, ['assetid', 'appfingerprint'],
|
||||
unique=True)
|
||||
|
||||
|
||||
def upgrade():
|
||||
_drop_certificate_columns()
|
||||
_rename_fingerprint()
|
||||
|
||||
if not _has_table(CREDENTIALS):
|
||||
op.create_table(
|
||||
CREDENTIALS,
|
||||
sa.Column('credentialid', sa.Integer(), nullable=False),
|
||||
sa.Column('assetid', sa.Integer(), nullable=False),
|
||||
sa.Column('credentialhash', sa.CHAR(length=64), nullable=False),
|
||||
sa.Column('credentialprefix', sa.String(length=12), nullable=False),
|
||||
sa.Column('enrolledvia', sa.String(length=16), nullable=False),
|
||||
sa.Column('issuedat', sa.DateTime(), nullable=False),
|
||||
sa.Column('lastusedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('revokedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('revokedby', sa.String(length=64), nullable=True),
|
||||
sa.Column('reenrollblocked', sa.Boolean(), nullable=False,
|
||||
server_default=sa.false()),
|
||||
sa.Column('reenrollblockedreason', sa.String(length=64),
|
||||
nullable=True),
|
||||
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
|
||||
ondelete='CASCADE'),
|
||||
sa.PrimaryKeyConstraint('credentialid'),
|
||||
sa.UniqueConstraint('assetid'),
|
||||
sa.UniqueConstraint('credentialhash'),
|
||||
)
|
||||
else:
|
||||
# A test database that ran an earlier build of this revision.
|
||||
columns = _columns(CREDENTIALS)
|
||||
if 'reenrollblocked' not in columns:
|
||||
op.add_column(CREDENTIALS, sa.Column(
|
||||
'reenrollblocked', sa.Boolean(), nullable=False,
|
||||
server_default=sa.false()))
|
||||
if 'reenrollblockedreason' not in columns:
|
||||
op.add_column(CREDENTIALS, sa.Column(
|
||||
'reenrollblockedreason', sa.String(length=64), nullable=True))
|
||||
|
||||
if not _has_table(REQUESTS):
|
||||
op.create_table(
|
||||
REQUESTS,
|
||||
sa.Column('requestid', sa.Integer(), nullable=False),
|
||||
sa.Column('hostname', sa.String(length=255), nullable=False),
|
||||
sa.Column('serialnumber', sa.String(length=128), nullable=False),
|
||||
sa.Column('deviceid', sa.String(length=64), nullable=True),
|
||||
sa.Column('enrollnonce', sa.String(length=128), nullable=False),
|
||||
sa.Column('agentversion', sa.String(length=32), nullable=True),
|
||||
sa.Column('status', sa.String(length=16), nullable=False),
|
||||
sa.Column('reason', sa.String(length=64), nullable=True),
|
||||
sa.Column('matchedassetid', sa.Integer(), nullable=True),
|
||||
sa.Column('sourceip', sa.String(length=45), nullable=True),
|
||||
sa.Column('requestedat', sa.DateTime(), nullable=False),
|
||||
sa.Column('lastseenat', sa.DateTime(), nullable=False),
|
||||
sa.Column('decidedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('decidedby', sa.String(length=64), nullable=True),
|
||||
sa.ForeignKeyConstraint(['matchedassetid'], ['assets.assetid'],
|
||||
ondelete='SET NULL'),
|
||||
sa.PrimaryKeyConstraint('requestid'),
|
||||
)
|
||||
existing = {tuple(ix.get('column_names') or ()) for ix in _indexes(REQUESTS)}
|
||||
if ('hostname', 'serialnumber') not in existing:
|
||||
op.create_index('idx_endpointenrollrequests_hostserial', REQUESTS,
|
||||
['hostname', 'serialnumber'],
|
||||
mysql_length={'hostname': 128, 'serialnumber': 63})
|
||||
|
||||
if not _has_table(WAVES):
|
||||
op.create_table(
|
||||
WAVES,
|
||||
sa.Column('waveid', sa.Integer(), nullable=False),
|
||||
sa.Column('createdat', sa.DateTime(), nullable=False),
|
||||
sa.Column('createdby', sa.String(length=64), nullable=True),
|
||||
sa.Column('assetids', sa.Text(), nullable=False),
|
||||
sa.Column('notes', sa.String(length=255), nullable=True),
|
||||
sa.PrimaryKeyConstraint('waveid'),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_table(WAVES):
|
||||
op.drop_table(WAVES)
|
||||
if _has_table(REQUESTS):
|
||||
for index in _indexes(REQUESTS):
|
||||
if index['name'] == 'idx_endpointenrollrequests_hostserial':
|
||||
op.drop_index(index['name'], table_name=REQUESTS)
|
||||
op.drop_table(REQUESTS)
|
||||
if _has_table(CREDENTIALS):
|
||||
op.drop_table(CREDENTIALS)
|
||||
@@ -0,0 +1,65 @@
|
||||
"""One-line bootstrap install tokens.
|
||||
|
||||
Revision ID: endpoints0024bootstraptokens
|
||||
Revises: endpoints0023credentials
|
||||
Create Date: 2026-09-15
|
||||
|
||||
Creates endpointbootstraptokens: the short-lived, admin-minted authorizations
|
||||
that let a tech fetch the bootstrap script and the keyless agent package with a
|
||||
single command. Only the sha256 of the secret is stored.
|
||||
|
||||
Guarded so the revision is safe on a database built by create_all and on a
|
||||
re-run. MySQL 5.6 safe: no JSON columns, and the only index is the unique key
|
||||
on a fixed-length CHAR(64), well under the 767-byte limit on utf8mb4.
|
||||
|
||||
Downgrade drops the table.
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = 'endpoints0024bootstraptokens'
|
||||
down_revision = 'endpoints0023credentials'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
plugin_name = 'endpoints'
|
||||
|
||||
TOKENS = 'endpointbootstraptokens'
|
||||
|
||||
|
||||
def _inspector():
|
||||
from sqlalchemy import inspect
|
||||
|
||||
return inspect(op.get_bind())
|
||||
|
||||
|
||||
def _has_table(name):
|
||||
return _inspector().has_table(name)
|
||||
|
||||
|
||||
def upgrade():
|
||||
if _has_table(TOKENS):
|
||||
return
|
||||
op.create_table(
|
||||
TOKENS,
|
||||
sa.Column('tokenid', sa.Integer(), nullable=False),
|
||||
sa.Column('tokenhash', sa.CHAR(length=64), nullable=False),
|
||||
sa.Column('tokenprefix', sa.String(length=12), nullable=False),
|
||||
sa.Column('createdat', sa.DateTime(), nullable=False),
|
||||
sa.Column('createdby', sa.String(length=64), nullable=True),
|
||||
sa.Column('expiresat', sa.DateTime(), nullable=False),
|
||||
sa.Column('maxuses', sa.Integer(), nullable=False),
|
||||
sa.Column('usecount', sa.Integer(), nullable=False,
|
||||
server_default='0'),
|
||||
sa.Column('revokedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('lastusedat', sa.DateTime(), nullable=True),
|
||||
sa.Column('lastusedip', sa.String(length=45), nullable=True),
|
||||
sa.PrimaryKeyConstraint('tokenid'),
|
||||
sa.UniqueConstraint('tokenhash'),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
if _has_table(TOKENS):
|
||||
op.drop_table(TOKENS)
|
||||
Reference in New Issue
Block a user