ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make main the lean product a site receives

Browse Source 
main was 16 days and 259 commits behind the line the product is actually built
on, at contract 0.30.0 against 0.35.0, with no per-PC credential code at all. It
was neither current nor curated - just stale. This brings it forward and, at the
same time, makes it mean something: main is what a receiving site installs, and
nothing else.

WHAT IT CARRIES. Twelve plugins - backups, computers, endpoints, guides,
knowledgebase, machines, network, notifications, printers, slides, tools,
warranty - plus applications, which is manifest-less core and always ships. The
only manifest dependency inside that set is endpoints on computers, and both are
in it, so the closure is self-contained.

WHAT IT DOES NOT. employees, geenforce, measuringtools, printedparts and usb
stay on the site line. So do the Windows installer tree, the VM rehearsal
helpers, the agent tooling config, one site's data loader, the internal audit
and findings records, and the planner spreadsheet. 1,135 tracked files against
1,438.

A REMOVAL THAT BREAKS THE APP IS NOT A REMOVAL, and two nearly shipped:

deploy/site-profile-universal.json still named all seventeen plugins. It is the
file docs/DEPLOY.md tells a site to provision from, so a fresh install would
have failed at `flask plugin apply-profile` with PluginError - the first thing a
receiving site does, broken by the branch meant to serve them. It now names the
twelve that exist, and its comment says the two must agree.

agent/ was never refreshed. Whoever populated this branch copied shopdb/,
plugins/, frontend/ and tests/ from the site line and left the agent tree at its
September state: 22 C# and PowerShell sources missing, including AgentCredentials,
EnrollmentService, AgentHttpFactory and the fleet-alert client, plus one file the
site line had deleted still present. The old tree is internally consistent, so it
compiles, and nothing in CI builds it - every gate was green over a server that
speaks enrollment and per-PC credentials to a bundled agent source that cannot.
Resynced whole.

DECISIONS THIS ENCODES, rather than defers:

- The plugin-authoring lab builds printedparts, whose manifest depends on
  employees, so it could not stay. The lab remains on the site line and main
  ships no build-along tutorial. That is a real loss for an adopting developer
  and is recorded here rather than discovered later.
- Recognition, Training and Recertification notifications are no longer seeded.
  Each splits one notification into a card per employee and renders a photo,
  which needs a plugin this branch does not bundle. Seeding them would offer an
  author a type that cannot resolve anybody. A site that installs employees gets
  them from that plugin's own seed.
- Windows/IIS is documented as the legacy path throughout, and the README leads
  with Docker, because that is what a receiving site gets.

The API description is regenerated for THIS branch - 347 paths and 483
operations, down from 415 and 573 - so the published contract describes what is
here rather than what the site line serves. docs/CONFIG.md lost the two
measuring-tool label keys the code no longer declares, which two lanes had
decided opposite ways.

Suite: 3124 passed, 6 skipped. Naming and style gates pass. The docs gates run
rather than skip, and every index line count is derived from the file it names.
This commit is contained in:
cproudlock
2026-09-19 11:24:40 -04:00
parent cb3975049c
commit 534f179ee6
675 changed files with 115759 additions and 60239 deletions
Download Patch File Download Diff File Expand all files Collapse all files

89
migrations/FIX_LOCATIONONLY_EQUIPMENT_TYPES.md
View File

@@ -1,89 +0,0 @@
# Fix LocationOnly Equipment Types Migration
## Issue
During the original migration from the VBScript ShopDB site, 110 machines were imported with `LocationOnly` type even though they have models assigned that specify the correct equipment type.
## Root Cause
The migration copied the `machinetypeid` directly from the `machines` table, but many machines had `machinetypeid=1` (LocationOnly) even when they had a model with a proper type.
## Affected Records
- **110 machines** in the `machines` table
- **110 equipment** records in the `equipment` table
## Example
| Machine | Current Type | Should Be | Model |
|---------|-------------|-----------|-------|
| 7502 | LocationOnly | Horizontal Machining Center | a81nx |
| 3108 | LocationOnly | Vertical Lathe | VTM-100 |
| 4001 | LocationOnly | 5-axis Mill | VP9000 |
## Fix SQL
### Step 1: Preview affected records
```sql
-- Preview what will be changed
SELECT
m.machineid,
m.machinenumber,
mt_current.machinetype as current_type,
mt_model.machinetype as correct_type,
mo.modelnumber
FROM machines m
JOIN models mo ON mo.modelnumberid = m.modelnumberid
JOIN machinetypes mt_current ON mt_current.machinetypeid = m.machinetypeid
JOIN machinetypes mt_model ON mt_model.machinetypeid = mo.machinetypeid
WHERE m.machinetypeid = 1 -- Currently LocationOnly
AND mo.machinetypeid != 1; -- Model has a real type
```
### Step 2: Fix the machines table
```sql
-- Update machines table to use the model's machine type
UPDATE machines m
JOIN models mo ON mo.modelnumberid = m.modelnumberid
SET m.machinetypeid = mo.machinetypeid
WHERE m.machinetypeid = 1 -- Currently LocationOnly
AND mo.machinetypeid != 1; -- Model has a real type
```
### Step 3: Fix the equipment table
```sql
-- Update equipment table to match
-- Note: equipmenttypeid values match machinetypeid values
UPDATE equipment e
JOIN assets a ON a.assetid = e.assetid
JOIN machines m ON m.machinenumber = SUBSTRING_INDEX(a.assetnumber, '-', 1)
SET e.equipmenttypeid = m.machinetypeid
WHERE e.equipmenttypeid = 1 -- Currently LocationOnly
AND m.machinetypeid != 1; -- Machine now has correct type
```
### Alternative Step 3 (if asset numbers don't match)
```sql
-- Update equipment based on their own model reference
UPDATE equipment e
JOIN models mo ON mo.modelnumberid = e.modelnumberid
SET e.equipmenttypeid = mo.machinetypeid
WHERE e.equipmenttypeid = 1 -- Currently LocationOnly
AND mo.machinetypeid != 1 -- Model has a real type
AND e.modelnumberid IS NOT NULL;
```
## Verification
```sql
-- Count remaining LocationOnly records that should have been fixed
SELECT COUNT(*) as remaining_fixable
FROM machines m
JOIN models mo ON mo.modelnumberid = m.modelnumberid
WHERE m.machinetypeid = 1
AND mo.machinetypeid != 1;
-- Should return 0 after fix
```
## Notes for Future Migrations
1. When importing equipment, always check if the model has a type and use that as the default
2. Only use "LocationOnly" for genuine location markers (rooms, offices) without models
3. Validate after import: any equipment with a model should NOT be LocationOnly
## Date Created
2026-01-27

92
migrations/MIGRATE_USB_DEVICES_FROM_EQUIPMENT.md
View File

@@ -1,92 +0,0 @@
# Migrate USB Devices from Equipment Table
## Issue
6 USB devices were incorrectly stored in the `equipment` table instead of the proper `usbdevices` table.
## Status
- [x] `pin` field added to `usbdevices` table for encrypted devices
- [x] `pin` field added to USBDevice model
- [x] USB Device equipment type deactivated (equipmenttypeid=44)
- [ ] Migrate 6 USB devices to usbdevices table
## Affected Records
| Asset Number | Name | Serial Number |
|--------------|------|---------------|
| 82841957-5891 | Green Kingston 64GB | 82841957 |
| 48854302-5892 | Blue Kingston 64GB | 48854302 |
| 75953637-5893 | Blue Kingston 64GB USB 3.0 | 75953637 |
| 41299370-5904 | Lenovo Portable DVD | 41299370 |
| 15492331-5905 | TEAC Portable Floppy Drive | 15492331 |
| 25777358-5906 | Netgear WiFi Adapter | 25777358 |
## Migration SQL
### Step 1: Insert into usbdevices
```sql
INSERT INTO usbdevices (
serialnumber,
label,
assetnumber,
usbdevicetypeid,
storagelocation,
notes,
createddate,
modifieddate,
isactive
)
SELECT
a.serialnumber,
a.name,
a.assetnumber,
NULL, -- Assign proper type later
NULL, -- Storage location TBD
a.notes,
a.createddate,
a.modifieddate,
a.isactive
FROM assets a
JOIN equipment e ON e.assetid = a.assetid
WHERE e.equipmenttypeid = 44; -- USB Device type
```
### Step 2: Delete from equipment table
```sql
DELETE e FROM equipment e
JOIN assets a ON a.assetid = e.assetid
WHERE e.equipmenttypeid = 44;
```
### Step 3: Delete from assets table
```sql
DELETE a FROM assets a
JOIN equipment e ON e.assetid = a.assetid
WHERE e.equipmenttypeid = 44;
-- Note: Run step 2 first since it references assets
```
### Alternative: Manual Migration
Since there are only 6 devices, you may prefer to:
1. Manually create them in the USB Devices section
2. Delete the equipment/asset records
## USB Device Types to Create
```sql
-- Check if types exist, create if needed
INSERT IGNORE INTO usbdevicetypes (typename, description, icon) VALUES
('Flash Drive', 'USB flash drive / thumb drive', 'usb'),
('External HDD', 'External hard disk drive', 'harddisk'),
('External SSD', 'External solid state drive', 'harddisk'),
('Card Reader', 'SD/CF card reader', 'sdcard'),
('Optical Drive', 'External CD/DVD/Blu-ray drive', 'disc'),
('WiFi Adapter', 'USB wireless network adapter', 'wifi'),
('Other', 'Other USB device', 'usb');
```
## Notes
- USB Device equipment type has been deactivated (isactive=0)
- New USB devices should be created in the USB Devices section
- USB devices do NOT need map positions (no mapleft/maptop)
## Date Created
2026-01-27

165
migrations/PRODUCTION_MIGRATION_GUIDE.md
View File

@@ -1,165 +0,0 @@
# Production Migration Guide
## Overview
This guide documents the process for migrating data from the legacy VBScript ShopDB site to the new Flask-based ShopDB.
## Database Architecture
### Legacy System (VBScript)
- Single `machines` table containing all equipment, PCs, printers
- `machinetypes` table for classification
- `models` table with `machinetypeid` reference
### New System (Flask)
- Core `assets` table (unified asset registry)
- Plugin-specific extension tables:
- `equipment` (equipmenttypeid -> equipmenttypes)
- `computers` (computertypeid -> computertypes)
- `printers` (printertypeid -> printertypes)
- `network_device` (networkdevicetypeid -> networkdevicetypes)
## Key Mappings
### Asset Type Mapping
| Legacy Category | New Asset Type | Extension Table |
|-----------------|---------------|-----------------|
| Equipment machines | Equipment | equipment |
| PC/Computer | Computer | computers |
| Printer | Printer | printers |
| Network (IDF, Switch, AP) | Network Device | network_device |
### Type ID Alignment
The `equipmenttypes` table IDs match `machinetypes` IDs for easy migration:
- equipmenttypeid = machinetypeid (where applicable)
## Migration Steps
### Step 1: Export from Legacy Database
```sql
-- Export machines with all related data
SELECT
m.*,
mt.machinetype,
mo.modelnumber,
mo.machinetypeid as model_typeid,
v.vendor,
bu.businessunit,
s.status
FROM machines m
LEFT JOIN machinetypes mt ON mt.machinetypeid = m.machinetypeid
LEFT JOIN models mo ON mo.modelnumberid = m.modelnumberid
LEFT JOIN vendors v ON v.vendorid = m.vendorid
LEFT JOIN businessunits bu ON bu.businessunitid = m.businessunitid
LEFT JOIN statuses s ON s.statusid = m.statusid;
```
### Step 2: Create Assets
For each machine, create an asset record:
```sql
INSERT INTO assets (assetnumber, name, assettypeid, statusid, locationid, businessunitid, mapleft, maptop)
SELECT
CONCAT(machinenumber, '-', machineid), -- Unique asset number
alias,
CASE
WHEN category = 'PC' THEN 2 -- Computer
WHEN category = 'Printer' THEN 4 -- Printer
ELSE 1 -- Equipment
END,
statusid,
locationid,
businessunitid,
mapleft,
maptop
FROM machines;
```
### Step 3: Create Extension Records
```sql
-- For Equipment
INSERT INTO equipment (assetid, equipmenttypeid, vendorid, modelnumberid)
SELECT
a.assetid,
COALESCE(mo.machinetypeid, m.machinetypeid), -- Use model's type if available!
m.vendorid,
m.modelnumberid
FROM machines m
JOIN assets a ON a.assetnumber = CONCAT(m.machinenumber, '-', m.machineid)
LEFT JOIN models mo ON mo.modelnumberid = m.modelnumberid
WHERE m.category = 'Equipment' OR m.category IS NULL;
```
### Step 4: Post-Migration Fixes
#### Fix LocationOnly Equipment Types
Equipment imported with LocationOnly type should inherit type from their model:
```sql
-- Fix equipment that has a model with a proper type
UPDATE equipment e
JOIN assets a ON a.assetid = e.assetid
JOIN machines m ON m.machinenumber = SUBSTRING_INDEX(a.assetnumber, '-', 1)
JOIN models mo ON mo.modelnumberid = m.modelnumberid
SET e.equipmenttypeid = mo.machinetypeid
WHERE e.equipmenttypeid = 1 -- LocationOnly
AND mo.machinetypeid != 1; -- Model has real type
```
## Validation Queries
### Check for Orphaned Assets
```sql
SELECT a.* FROM assets a
LEFT JOIN equipment e ON e.assetid = a.assetid
LEFT JOIN computers c ON c.assetid = a.assetid
LEFT JOIN printers p ON p.assetid = a.assetid
WHERE a.assettypeid = 1 -- Equipment type
AND e.assetid IS NULL;
```
### Check LocationOnly with Models
```sql
-- Should return 0 after migration fix
SELECT COUNT(*)
FROM equipment e
JOIN assets a ON a.assetid = e.assetid
JOIN machines m ON m.machinenumber = SUBSTRING_INDEX(a.assetnumber, '-', 1)
JOIN models mo ON mo.modelnumberid = m.modelnumberid
WHERE e.equipmenttypeid = 1
AND mo.machinetypeid != 1;
```
### Verify Type Distribution
```sql
SELECT et.equipmenttype, COUNT(*) as count
FROM equipment e
JOIN equipmenttypes et ON et.equipmenttypeid = e.equipmenttypeid
GROUP BY et.equipmenttype
ORDER BY count DESC;
```
## Common Issues
### Issue: Equipment marked as LocationOnly but has model
**Cause**: Migration copied machinetypeid from machines table instead of using model's type
**Fix**: See FIX_LOCATIONONLY_EQUIPMENT_TYPES.md
### Issue: Missing model relationship
**Cause**: Equipment table modelnumberid not populated during migration
**Fix**: Link through machines table using asset number pattern
### Issue: Duplicate asset numbers
**Cause**: Asset number generation didn't account for existing duplicates
**Fix**: Use unique suffix or check before insert
## Scripts Location
- `/migrations/FIX_LOCATIONONLY_EQUIPMENT_TYPES.md` - Fix for LocationOnly type issue
- `/scripts/import_from_mysql.py` - Original import script (may need updates)
- `/scripts/migration/` - Migration utilities
## Notes
- Always backup before running migration fixes
- Test on staging/dev before production
- Verify counts before and after each fix
- Keep legacy `machines` table for reference during transition
## Date Created
2026-01-27

151
migrations/versions/7d42_split_endpoints_deploy.py Normal file
View File

@@ -0,0 +1,151 @@
"""Split endpoints.deploy into three independent scope grants.
endpoints.deploy bundled three actions of very different risk behind one name:
scheduledtask_run -> starts a task already registered and allowlisted agent-side
update_app -> runs a pinned installer
run_script -> ARBITRARY code as SYSTEM unless the descriptor pins a user
Because they shared a grant, there was no way to let an operator push an
application update without also handing them arbitrary SYSTEM code execution on
every endpoint they could reach. The permission name described the lightest of
the three and silently carried the heaviest.
commands.ELEVATED_ACTION_PERMISSIONS now maps each action to its own permission
(endpoints.corptools, endpoints.updateapp, endpoints.runscript). The enqueue
route reads that table, so no route logic changed.
This migration preserves access EXACTLY. Anyone holding endpoints.deploy today
can already do all three, so every role holding it receives all three new
grants. Nothing widens: no role that lacked endpoints.deploy gains anything. The
security win is not in this migration - it is that from here the three can be
granted separately.
endpoints.deploy itself is then removed. Leaving it behind would be worse than
useless: nothing reads it any more, but the role grid would keep offering it and
operators would keep ticking it expecting an effect.
Revision ID: 7d42_split_endpoints_deploy
Revises: 7d41_drop_guest_tools_secrets
"""
from alembic import op
import sqlalchemy as sa
revision = '7d42_split_endpoints_deploy'
down_revision = '7d41_drop_guest_tools_secrets'
branch_labels = None
depends_on = None
OLD_PERMISSION = 'endpoints.deploy'
NEW_PERMISSIONS = [
('endpoints.corptools',
'Start approved scheduled tasks on endpoints (limited to tasks already '
'registered and allowlisted on the agent)',
'endpoints'),
('endpoints.updateapp',
'Push application installers to endpoints (runs a pinned installer as '
'SYSTEM)',
'endpoints'),
('endpoints.runscript',
'Run staged scripts on endpoints (ARBITRARY CODE, executes as SYSTEM '
'unless the script pins a user context; grant only to fully trusted '
'operators)',
'endpoints'),
]
def _tables_present(bind):
inspector = sa.inspect(bind)
tables = set(inspector.get_table_names())
return {'roles', 'permissions', 'rolepermissions'} <= tables
def _ensure_permission(bind, name, description, category):
"""Insert a permission if absent. Guarded in Python so the statement stays
portable - MySQL's FROM DUAL idiom does not run on SQLite, which the test
suite uses."""
exists = bind.execute(sa.text(
'SELECT 1 FROM permissions WHERE name = :name'), {'name': name}).first()
if exists:
return
bind.execute(sa.text(
'INSERT INTO permissions (name, description, category) '
'VALUES (:name, :description, :category)'),
{'name': name, 'description': description, 'category': category})
def upgrade():
bind = op.get_bind()
if not _tables_present(bind):
return
for name, description, category in NEW_PERMISSIONS:
_ensure_permission(bind, name, description, category)
# Preserve access: every role holding the old bundle gets all three halves.
for name, _description, _category in NEW_PERMISSIONS:
bind.execute(sa.text(
'INSERT INTO rolepermissions (roleid, permissionid) '
'SELECT rp.roleid, new.permissionid '
'FROM rolepermissions rp '
'JOIN permissions old ON old.permissionid = rp.permissionid '
'JOIN permissions new ON new.name = :newname '
'WHERE old.name = :oldname '
'AND NOT EXISTS (SELECT 1 FROM rolepermissions existing '
' WHERE existing.roleid = rp.roleid '
' AND existing.permissionid = new.permissionid)'),
{'newname': name, 'oldname': OLD_PERMISSION})
bind.execute(sa.text(
'DELETE FROM rolepermissions WHERE permissionid IN '
'(SELECT permissionid FROM permissions WHERE name = :oldname)'),
{'oldname': OLD_PERMISSION})
bind.execute(sa.text(
'DELETE FROM permissions WHERE name = :oldname'),
{'oldname': OLD_PERMISSION})
def downgrade():
"""Rebuild the bundle and grant it to anyone holding any of the three.
Deliberately generous in this direction: on the way back the only safe
reading of "held one of the three" is "held the bundle", because the bundle
is all-or-nothing. Downgrading therefore restores capability rather than
quietly removing it from an operator mid-shift.
"""
bind = op.get_bind()
if not _tables_present(bind):
return
_ensure_permission(
bind, OLD_PERMISSION,
'Run staged scripts, start approved scheduled tasks, and push '
'application installers to endpoints (scripts run as SYSTEM; grant '
'only to fully trusted operators)',
'endpoints')
newnames = [name for name, _d, _c in NEW_PERMISSIONS]
bind.execute(sa.text(
'INSERT INTO rolepermissions (roleid, permissionid) '
'SELECT DISTINCT rp.roleid, old.permissionid '
'FROM rolepermissions rp '
'JOIN permissions new ON new.permissionid = rp.permissionid '
'JOIN permissions old ON old.name = :oldname '
'WHERE new.name IN :newnames '
'AND NOT EXISTS (SELECT 1 FROM rolepermissions existing '
' WHERE existing.roleid = rp.roleid '
' AND existing.permissionid = old.permissionid)'
).bindparams(sa.bindparam('newnames', expanding=True)),
{'oldname': OLD_PERMISSION, 'newnames': newnames})
for name in newnames:
bind.execute(sa.text(
'DELETE FROM rolepermissions WHERE permissionid IN '
'(SELECT permissionid FROM permissions WHERE name = :name)'),
{'name': name})
bind.execute(sa.text(
'DELETE FROM permissions WHERE name = :name'), {'name': name})

59
migrations/versions/7d43_userpermissions.py Normal file
View File

@@ -0,0 +1,59 @@
"""Per-user direct permission grants.
Permissions were reachable only through roles. At 5-7 operators per instance
that forces a role to be minted for every one-off capability - "Bob, and only
Bob, may push scripts" - until the role list is a mixture of job titles and
single-capability flags and nobody can tell which is which.
A row here is an ORDINARY permission grant. It is identical in power to one held
through a role: it counts toward the API token scope ceiling, it is folded by
both User.haspermission() and User.getpermissions(), and it is revoked the same
way. There is deliberately no second class of permission.
Nothing is granted by this migration. The table is created empty; existing
authority is untouched.
Revision ID: 7d43_userpermissions
Revises: 7d42_split_endpoints_deploy
"""
from alembic import op
import sqlalchemy as sa
revision = '7d43_userpermissions'
down_revision = '7d42_split_endpoints_deploy'
branch_labels = None
depends_on = None
def upgrade():
bind = op.get_bind()
inspector = sa.inspect(bind)
if 'userpermissions' in set(inspector.get_table_names()):
return
op.create_table(
'userpermissions',
sa.Column('userid', sa.Integer(), nullable=False),
sa.Column('permissionid', sa.Integer(), nullable=False),
# Cascade on both sides: a direct grant is meaningless once either end
# is gone, and leaving orphans would let a recycled id silently inherit
# someone else's capability.
sa.ForeignKeyConstraint(['userid'], ['users.userid'],
ondelete='CASCADE'),
sa.ForeignKeyConstraint(['permissionid'],
['permissions.permissionid'],
ondelete='CASCADE'),
sa.PrimaryKeyConstraint('userid', 'permissionid'),
)
op.create_index('idx_userpermissions_userid', 'userpermissions', ['userid'])
def downgrade():
bind = op.get_bind()
inspector = sa.inspect(bind)
if 'userpermissions' not in set(inspector.get_table_names()):
return
op.drop_index('idx_userpermissions_userid', table_name='userpermissions')
op.drop_table('userpermissions')

42
migrations/versions/7d44_application_monitorable.py Normal file
View File

@@ -0,0 +1,42 @@
"""Add applications monitorable fields
Backs the "monitorable" checkbox on the Application form: an app that the
endpoint agent can watch as a process monitor. A monitor needs something to look
for (the Windows image name) and something to start when it is missing (the
launch path), so the API refuses to save ismonitorable without both.
The two text columns stay NULLABLE rather than carrying a constraint, because
every application row that existed before this migration has neither value and
must remain valid. The pairing rule is enforced in shopdb/core/api/applications.py
where it can return a useful message, not in the schema where it would only
produce an IntegrityError.
Revision ID: 7d44_app_monitorable
Revises: 7d43_userpermissions
Create Date: 2026-09-08
"""
from alembic import op
import sqlalchemy as sa
revision = '7d44_app_monitorable'
down_revision = '7d43_userpermissions'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('applications', sa.Column(
'ismonitorable', sa.Boolean(), nullable=True, server_default=sa.false(),
comment='Can be watched by the endpoint agent (needs launch path + process name)'))
op.add_column('applications', sa.Column(
'monitorlaunchpath', sa.String(length=512), nullable=True))
op.add_column('applications', sa.Column(
'monitorprocessname', sa.String(length=255), nullable=True))
def downgrade():
op.drop_column('applications', 'monitorprocessname')
op.drop_column('applications', 'monitorlaunchpath')
op.drop_column('applications', 'ismonitorable')

86
migrations/versions/7d45_purge_inactive_apps.py Normal file
View File

@@ -0,0 +1,86 @@
"""Purge soft-deleted applications that still reserve their names.
Delete used to set isactive=False and leave the row in place. The uniqueness
check on create looks at every row regardless of isactive, so each of those
rows kept its name reserved permanently: creating an application with the same
name failed with "already exists" against a row no screen would display, and
nothing in the UI could clear it.
Delete is a hard delete now, but that only fixes rows deleted from here on.
This clears the ones already stranded, in every environment rather than just
the one where the bug was noticed.
Dependent rows are cleared the same way the API does it, and for the same
reasons: appversions and computerinstalledapps are NOT NULL and meaningless
without their application, while knowledgebase.appid is nullable and the
article is somebody's writing, so it is unlinked rather than destroyed.
Uploaded installers belonging to these rows are deliberately NOT touched. A
migration that deletes files cannot be rolled back and cannot be dry-run, and a
stray file is harmless next to that risk. Deletes made through the API do clean
them up.
Revision ID: 7d45_purge_inactive_apps
Revises: 7d44_app_monitorable
"""
from alembic import op
import sqlalchemy as sa
revision = '7d45_purge_inactive_apps'
down_revision = '7d44_app_monitorable'
branch_labels = None
depends_on = None
def _table_exists(bind, name):
"""True when a table is present.
The plugin tables are only there when the plugin is installed, so each is
checked before being touched.
"""
return sa.inspect(bind).has_table(name)
def upgrade():
bind = op.get_bind()
if not _table_exists(bind, 'applications'):
return
doomed = [row[0] for row in bind.execute(sa.text(
'SELECT appid FROM applications WHERE isactive = false'
))]
if not doomed:
return
for appid in doomed:
params = {'appid': appid}
if _table_exists(bind, 'knowledgebase'):
bind.execute(sa.text(
'UPDATE knowledgebase SET appid = NULL WHERE appid = :appid'
), params)
if _table_exists(bind, 'computerinstalledapps'):
bind.execute(sa.text(
'DELETE FROM computerinstalledapps WHERE appid = :appid'
), params)
if _table_exists(bind, 'appversions'):
bind.execute(sa.text(
'DELETE FROM appversions WHERE appid = :appid'
), params)
bind.execute(sa.text(
'DELETE FROM applications WHERE appid = :appid'
), params)
def downgrade():
# The rows are gone and their contents were not captured, so there is
# nothing to restore. Failing loudly is better than pretending otherwise.
raise NotImplementedError(
'Purged applications cannot be restored; recover from a backup.')

136
migrations/versions/7d46_nest_app_packages.py Normal file
View File

@@ -0,0 +1,136 @@
"""Move uploaded installers into per-application directories.
Installers used to be stored flat as "application-<id>.<ext>", which threw away
the name the vendor shipped. That is what made a re-import look like new
software: the operator had no way to see that the file already in the catalogue
was the same one they were about to upload again, so duplicate applications and
versions accumulated. Installers now keep their filename, namespaced by
application id so a generic "setup.exe" from two vendors cannot collide.
That only helps files uploaded from here on. This moves the ones already stored,
so existing download links keep working instead of 404ing against a path the
route no longer serves.
The original filename cannot be recovered - the old scheme discarded it at
upload - so these land as "installer.<ext>". That is no worse than what they
were called before, and re-uploading the installer restores the real name.
Files are MOVED, not copied or deleted: a move is reversible, which is what
lets downgrade() put them back. This is deliberately unlike 7d45, which refused
to touch files because it could only have deleted them.
The database is the part that must not be left inconsistent, so installpath is
rewritten for every affected row whether or not its file was found on disk. A
row pointing at a missing file is a broken download; a row pointing at the old
scheme is a broken download that also cannot be repaired by re-uploading.
Revision ID: 7d46_nest_app_packages
Revises: 7d45_purge_inactive_apps
"""
import os
import re
from alembic import op
import sqlalchemy as sa
revision = '7d46_nest_app_packages'
down_revision = '7d45_purge_inactive_apps'
branch_labels = None
depends_on = None
OLD_PREFIX = '/api/applications/package/'
# application-<id>.<ext>, the only shape the old uploader produced.
OLD_NAME = re.compile(r'^application-(\d+)\.([A-Za-z0-9]+)$')
def _package_dir():
"""The instance-relative directory installers are stored in.
Resolved the same way the API resolves it. If the app context is not
available the migration still rewrites the database and simply leaves the
files alone, rather than failing the whole upgrade over a path lookup.
"""
try:
from flask import current_app
return os.path.join(current_app.instance_path, 'applicationpackages')
except Exception:
return None
def upgrade():
bind = op.get_bind()
rows = bind.execute(sa.text(
"SELECT appid, installpath FROM applications "
"WHERE installpath LIKE :prefix"
), {'prefix': OLD_PREFIX + '%'}).fetchall()
directory = _package_dir()
for appid, installpath in rows:
tail = installpath[len(OLD_PREFIX):]
match = OLD_NAME.match(tail)
if not match:
# Already nested (contains a slash) or otherwise not the old shape.
# Leaving it alone is correct: it is already being served.
continue
ext = match.group(2).lower()
stored = f'installer.{ext}'
if directory:
source = os.path.join(directory, tail)
target_dir = os.path.join(directory, str(appid))
if os.path.isfile(source):
os.makedirs(target_dir, exist_ok=True)
try:
os.replace(source, os.path.join(target_dir, stored))
except OSError:
# Best effort. The row is still repointed below so the
# schema is consistent; a missing file is a re-upload.
pass
bind.execute(sa.text(
"UPDATE applications SET installpath = :path WHERE appid = :appid"
), {'path': f'{OLD_PREFIX}{appid}/{stored}', 'appid': appid})
def downgrade():
bind = op.get_bind()
rows = bind.execute(sa.text(
"SELECT appid, installpath FROM applications "
"WHERE installpath LIKE :prefix"
), {'prefix': OLD_PREFIX + '%'}).fetchall()
directory = _package_dir()
for appid, installpath in rows:
tail = installpath[len(OLD_PREFIX):]
if '/' not in tail:
continue
nested_id, _, stored = tail.partition('/')
if nested_id != str(appid):
continue
ext = os.path.splitext(stored)[1].lstrip('.').lower()
if not ext:
continue
flat = f'application-{appid}.{ext}'
if directory:
source = os.path.join(directory, str(appid), stored)
if os.path.isfile(source):
try:
os.replace(source, os.path.join(directory, flat))
# Remove the now-empty directory so a re-upgrade does not
# find a stale one. Only if empty: a directory with other
# files in it is not ours to remove.
os.rmdir(os.path.join(directory, str(appid)))
except OSError:
pass
bind.execute(sa.text(
"UPDATE applications SET installpath = :path WHERE appid = :appid"
), {'path': OLD_PREFIX + flat, 'appid': appid})

97
migrations/versions/7d47_endpoints_operatorreports.py Normal file
View File

@@ -0,0 +1,97 @@
"""Split operator report viewing out of endpoints.view.
Operator reports carry a screenshot of whatever was on the operator's screen
when they filed the report. That is an uncontrolled human capture - it can
include part numbers, customer names, email, or anything else that happened to
be visible - whereas the rest of endpoints.view is uniform machine-collected
telemetry. Reading a PC's CPU load does not imply a right to read someone's
screen, so the reports and their images move to their own grant.
Backfill preserves access rather than revoking it: every user who can see
endpoints today gets the new grant, so nobody loses a panel they were already
using. Tightening from there is an administrative decision made per account in
the user editor, not something this migration should force. The grant is a
per-user direct permission (endpoints.operatorreports is in the scope
denylist), so the backfill writes userpermissions rows, not role rows.
Revision ID: 7d47_endpoints_operatorreports
Revises: 7d46_nest_app_packages
"""
from alembic import op
import sqlalchemy as sa
revision = '7d47_endpoints_operatorreports'
down_revision = '7d46_nest_app_packages'
branch_labels = None
depends_on = None
NEW_PERMISSION = (
'endpoints.operatorreports',
'View operator-submitted reports and their screenshots',
'endpoints',
)
SOURCE_PERMISSION = 'endpoints.view'
def _tables_present(bind):
inspector = sa.inspect(bind)
tables = set(inspector.get_table_names())
return {'users', 'roles', 'permissions', 'rolepermissions',
'userpermissions', 'userroles'} <= tables
def upgrade():
bind = op.get_bind()
if not _tables_present(bind):
return
name, description, category = NEW_PERMISSION
# Guarded in Python so the statement stays portable: MySQL's FROM DUAL
# idiom does not run on SQLite, which the test suite uses.
exists = bind.execute(sa.text(
'SELECT 1 FROM permissions WHERE name = :name'),
{'name': name}).first()
if not exists:
bind.execute(sa.text(
'INSERT INTO permissions (name, description, category) '
'VALUES (:name, :description, :category)'),
{'name': name, 'description': description, 'category': category})
# Preserve access for anyone who reaches operator reports today, which is
# anyone holding endpoints.view through a role. NOT EXISTS keeps the
# migration re-runnable and avoids duplicate rows for users in two
# qualifying roles.
bind.execute(sa.text(
'INSERT INTO userpermissions (userid, permissionid) '
'SELECT DISTINCT ur.userid, new.permissionid '
'FROM userroles ur '
'JOIN rolepermissions rp ON rp.roleid = ur.roleid '
'JOIN permissions old ON old.permissionid = rp.permissionid '
'JOIN permissions new ON new.name = :newname '
'WHERE old.name = :oldname '
'AND NOT EXISTS (SELECT 1 FROM userpermissions existing '
' WHERE existing.userid = ur.userid '
' AND existing.permissionid = new.permissionid)'),
{'newname': name, 'oldname': SOURCE_PERMISSION})
def downgrade():
bind = op.get_bind()
if not _tables_present(bind):
return
name = NEW_PERMISSION[0]
bind.execute(sa.text(
'DELETE FROM userpermissions WHERE permissionid IN '
'(SELECT permissionid FROM permissions WHERE name = :name)'),
{'name': name})
bind.execute(sa.text(
'DELETE FROM rolepermissions WHERE permissionid IN '
'(SELECT permissionid FROM permissions WHERE name = :name)'),
{'name': name})
bind.execute(sa.text(
'DELETE FROM permissions WHERE name = :name'), {'name': name})

41
migrations/versions/7d48_script_workingdirectory.py Normal file
View File

@@ -0,0 +1,41 @@
"""Add scripts workingdirectory
Backs the optional "run in this folder" field on a library script. The agent
otherwise runs a delivered script from its own staging directory, which is
correct for a self-contained script and wrong for one that shells out to a
corporate tool already installed on the endpoint (Sysinternals and similar,
delivered by Intune) and referenced by relative path.
NULLABLE with no default, so every script row that existed before this
migration stays valid and keeps today's behaviour exactly: empty means the
staging directory. The path is validated in shopdb/core/api/scripts.py where a
rejection can explain itself, not in the schema where it would only produce an
IntegrityError.
This column grants no capability. A script body is already arbitrary PowerShell
run as SYSTEM and can change its own directory; the boundary is the
endpoints.runscript grant and the digest-pinned library.
Revision ID: 7d48_script_workingdir
Revises: 7d47_endpoints_operatorreports
Create Date: 2026-09-09
"""
from alembic import op
import sqlalchemy as sa
revision = '7d48_script_workingdir'
down_revision = '7d47_endpoints_operatorreports'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('scripts', sa.Column(
'workingdirectory', sa.String(length=255), nullable=True,
comment='Folder to run the script in; empty means the agent staging dir'))
def downgrade():
op.drop_column('scripts', 'workingdirectory')

46
migrations/versions/7d49_script_errormode.py Normal file
View File

@@ -0,0 +1,46 @@
"""Add scripts errormode
Backs the per-script "how should a non-terminating error be treated" field.
PowerShell invoked with -File exits 0 whenever it reaches the end of the body,
even if every meaningful line inside wrote an error. The agent derives the
command verdict from that exit code, so a script could report SUCCEEDED while
doing none of its work - which is exactly how an account-provisioning script
came back green with the account only partly built.
NOT NULL with server_default 'default', so every script row that existed before
this migration keeps today's permissive behaviour exactly. Strict is opt-in per
script and never retroactive: turning it on globally would change the meaning of
bodies that were authored under the permissive rule and deliberately tolerate a
failing Remove-Item, probe with Get-LocalUser, or loop past a bad item. Those
would begin aborting partway through, which is worse than a false green because
it leaves the endpoint half-configured.
The value is validated in shopdb/core/api/scripts.py where a rejection can
explain itself, not in the schema where it would only produce an IntegrityError.
Revision ID: 7d49_script_errormode
Revises: 7d48_script_workingdir
Create Date: 2026-09-09
"""
from alembic import op
import sqlalchemy as sa
revision = '7d49_script_errormode'
down_revision = '7d48_script_workingdir'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('scripts', sa.Column(
'errormode', sa.String(length=16), nullable=False,
server_default='default',
comment="'default' keeps PowerShell's permissive behaviour; "
"'strict' stops on the first error so the exit code is honest"))
def downgrade():
op.drop_column('scripts', 'errormode')

48
migrations/versions/7d50_script_captureoutput.py Normal file
View File

@@ -0,0 +1,48 @@
"""Add scripts captureoutput
Backs the per-script "may a successful run send its output back" field.
The agent already captures a script's full output and always writes it to the
endpoint log. What this controls is narrower: whether that output also crosses
to the server and is persisted in the command's resultjson, where the Run
Scripts history can show it.
Failures have always sent output and still do - a failed run has to be
diagnosable without walking to the machine. Success is the case in question,
and it is opt-in because output carries whatever the body happened to print:
hostnames, share paths, an account name, a token an author echoed while
debugging. Accepting that exposure for every routine green run is a poor
trade, so the author decides per script, after reading the body.
BOOLEAN NOT NULL defaulting FALSE, so every script row written before this
migration keeps exactly today's behaviour: metadata on success, output only on
failure. Same rule as errormode - never retroactive, because a library-wide
flip would start persisting output from bodies nobody re-read with this
question in mind.
Revision ID: 7d50_script_captureoutput
Revises: 7d49_script_errormode
Create Date: 2026-09-09
"""
from alembic import op
import sqlalchemy as sa
revision = '7d50_script_captureoutput'
down_revision = '7d49_script_errormode'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('scripts', sa.Column(
'captureoutput', sa.Boolean(), nullable=False,
server_default=sa.false(),
comment='When true, a successful run also returns its output to the '
'server; failures always do. Off by default because script '
'output can contain whatever the body printed.'))
def downgrade():
op.drop_column('scripts', 'captureoutput')

92
migrations/versions/7d51_endpoint_logcollections.py Normal file
View File

@@ -0,0 +1,92 @@
"""Add endpointlogcollections
Backs the manual Log Collector tab: one row per plaintext log file an operator
collected from an endpoint by path.
Collected content gets its own table rather than living in
endpointcommands.resultjson for three reasons. It is bulky, and resultjson is
read by every command-history list that only wants a status. It needs its own
retention rule, which a command row cannot carry without deleting the audit
record of the command itself. And the download route has to address a single
collection by id and serve it as a file, which is awkward against a JSON column.
Retention is PER ASSET (20, enforced on insert in the result route), not global.
The history renders on a single machine's detail page, so a global cap would let
one busy endpoint quietly evict the history of another - the panel would be
unreliable exactly where it is read.
endpointcommandid is SET NULL rather than CASCADE: commands are reaped and
expired over time, and a collected log an operator is still reading must not
vanish because its originating command row was cleaned up. assetid stays CASCADE
because a log has no meaning once the machine is gone.
filesizebytes is BIGINT while contentbytes is INTEGER on purpose - the file on
disk may be gigabytes, while what we store is a bounded tail.
Revision ID: 7d51_endpoint_logcollections
Revises: 7d50_script_captureoutput
Create Date: 2026-09-09
"""
from alembic import op
import sqlalchemy as sa
revision = '7d51_endpoint_logcollections'
down_revision = '7d50_script_captureoutput'
branch_labels = None
depends_on = None
def upgrade():
op.create_table(
'endpointlogcollections',
sa.Column('endpointlogcollectionid', sa.Integer(), nullable=False),
sa.Column('assetid', sa.Integer(), nullable=False),
sa.Column('endpointcommandid', sa.Integer(), nullable=True),
sa.Column('sourcepath', sa.String(length=512), nullable=False,
comment='Path as the AGENT resolved it, which may differ '
'from what the operator typed once environment '
'variables are expanded endpoint-side.'),
sa.Column('content', sa.Text(length=16777215), nullable=True),
sa.Column('filesizebytes', sa.BigInteger(), nullable=True,
comment='Size of the file on disk. Differs from contentbytes '
'when the stored copy is a truncated tail.'),
sa.Column('contentbytes', sa.Integer(), nullable=True),
sa.Column('linecount', sa.Integer(), nullable=True),
sa.Column('istruncated', sa.Boolean(), nullable=False,
server_default=sa.false(),
comment='True when content is only the tail of a larger file.'),
sa.Column('filemodifiedat', sa.DateTime(), nullable=True),
sa.Column('collectedbyuserid', sa.Integer(), nullable=True),
sa.Column('collectedbyusername', sa.String(length=100), nullable=True,
comment='Username snapshot; the id alone becomes unreadable '
'if the user is later removed.'),
sa.Column('collectedat', sa.DateTime(), nullable=False),
sa.ForeignKeyConstraint(['assetid'], ['assets.assetid'],
ondelete='CASCADE'),
sa.ForeignKeyConstraint(['endpointcommandid'],
['endpointcommands.endpointcommandid'],
ondelete='SET NULL'),
sa.ForeignKeyConstraint(['collectedbyuserid'], ['users.userid'],
ondelete='SET NULL'),
sa.PrimaryKeyConstraint('endpointlogcollectionid'),
)
op.create_index('ix_endpointlogcollections_assetid',
'endpointlogcollections', ['assetid'])
op.create_index('ix_endpointlogcollections_collectedat',
'endpointlogcollections', ['collectedat'])
# Serves both hot paths: listing an asset's history newest-first, and
# finding that asset's oldest rows to prune on insert.
op.create_index('idx_endpointlogcollections_assetcollected',
'endpointlogcollections', ['assetid', 'collectedat'])
def downgrade():
op.drop_index('idx_endpointlogcollections_assetcollected',
table_name='endpointlogcollections')
op.drop_index('ix_endpointlogcollections_collectedat',
table_name='endpointlogcollections')
op.drop_index('ix_endpointlogcollections_assetid',
table_name='endpointlogcollections')
op.drop_table('endpointlogcollections')