A site running external mode keeps its devices and check-in/out history in a
separate cmmc_usb database. There was no way to bring that in, so the choice
was to keep the second database forever or lose the history. This adds
'flask usb import-external' - dry run by default, --commit to write - which
reads through the same connection settings the routes already use and never
writes to the source.
Two things it has to reconcile, and they are why it is a command rather than a
few INSERTs.
The schemas disagree about SHAPE. External is an event log, one row per action;
self-hosted is a paired row carrying both times. Pairing walks each device in
time order, and real logs after years of use are untidy. The rule is that
nothing is invented and nothing vanishes: a drive still out becomes an open
checkout and is NOT flagged (that is just a drive somebody has), a return with
no matching issue is skipped and named, and two issues in a row leave the first
open rather than inventing a return time nobody recorded.
Identity is the other. usbdevices.serialnumber is unique and NOT NULL and the
external devices table has no serial, so device_id becomes it. Blank or
duplicated ids stop that device rather than inventing a key. external
device_owner has no column here, so it is appended to the device notes rather
than dropped.
Also adds usbcheckouts.scannedviruses, the one field with nowhere to land.
waswiped already means what external calls sanitized - selfhosted.py was
mapping it - but scanned_viruses was HARDCODED None in the log output: the API
promised the field and had nothing behind it, so an imported history would have
rendered blank. It now reads and writes properly.
Nullable, never defaulted to false. An unrecorded scan is not a failed scan,
and for removable-media records those are very different claims.
Full suite 2131 passed; seven new tests pin the pairing rules.
CMMC removable-media tracking is an audit obligation, and answering "who had
this drive, and when" meant paging through the check-in/out list by hand.
GET /api/usb/reports/audit returns the events - a take and a return are separate
rows, because a count cannot answer the question an audit actually asks - with
summary figures riding along for the people who only want the shape of it, and
a CSV for the people who want it in a spreadsheet.
The window is named rather than a day count: today, last month, six months, a
year, all time. Nobody should have to work out that a year is 365.
Both directory modes answer in one voice. The external path reads the
cmmc_usb checkinoutlog; self-hosted expands one usbcheckouts row into its two
ends, so a returned drive is distinguishable from one still out, and the window
filter runs in SQL against either end of the loan - a drive taken before the
window and returned inside it is a check-in the window has to show. Rows come
back from the mode-specific half; the CSV, the summary and the response shape
are built once in the route.
Ordering is total in both modes. Timestamp alone ties for two events in the same
second, and the same query then returns them in a different order each run,
which reads as the log being unstable.
The people who actually take USB drives are standing at the cabinet, and most
of them have no account here. Check-out and check-in were gated behind a login
that was not doing much anyway: both modes already attribute the action to a
scanned BADGE rather than to the signed-in user, so the login was gating the
screen, not the record.
/usb-kiosk is a full-screen touch page modelled on the parts kiosk, off unless
a site sets usb_kiosk_enabled - every route 404s until then, rather than 403,
so a disabled kiosk does not advertise that there is a door here. It works
against the self-hosted tables or an existing cmmc_usb database.
THE LIST IS THE TRADE. These drives carry no barcode, so unlike the parts kiosk
there is nothing to present and the screen has to offer a list - which exposes
more than a scan-first design, where you had to already hold a drive to learn
anything. So the list is not anonymous: an SSO comes first and must resolve to
a real person before any drive is named, which is the same bar taking a drive
already had to clear. It is a POST so the badge stays out of proxy access logs.
The main screen shows the caller's own drives, what is free, and a COUNT of
everything else; holders of other people's drives appear only behind a second
deliberate tap, so nobody reads the whole cabinet off a wall panel in passing.
A badge that resolves to no employee is REFUSED, unlike the signed-in route
which creates the user row. A checkout nobody can trace to a person defeats the
tracking, and at a kiosk there is no operator standing there to vouch for it.
Returning somebody else's drive asks first and then records the badge that was
SCANNED, so the log says who actually handed it back. That is the honest record
and it needs no change to a schema this app does not own. Refusing a mismatch
outright is a setting but not the default: a drive left behind by somebody on
leave would otherwise be unreturnable at the kiosk, which just moves the problem
to a person with a login.
The authz sweep caught these writes, twice - once for the two mutations and
again for the two list endpoints. Both are exempted with the reasoning written
out beside the parts-kiosk entry rather than silenced. The list endpoints are
POSTs that change nothing.
Verified by driving the real page: SSO on the keypad, tap Take, database moves.
From the database review (verdict: sound-with-minor-issues). Applies the
actionable findings.
Redundant indexes: five non-unique secondary indexes duplicated a named idx_*
or a unique index on the same column - ix_communications_assetid,
ix_computers_hostname, ix_networkdevices_hostname, ix_printers_hostname (each
shadowing an idx_*), and idx_usb_serial (shadowing the serialnumber unique
index). Removed the redundant index source from the models (column index=True /
the extra db.Index) and added core migration 7d25 dropping the live duplicates.
The unique ix_*_assetid indexes are kept (they enforce assetid uniqueness).
Dead column: usbcheckouts.machineid was a NOT NULL soft-ref to the retired
machines table storing sentinel 0 (ADR-001). Dropped from the model + the
machineid=0 literal in selfhosted checkout; usb plugin migration 0002 drops it
live (downgrade restores it default 0).
Index: notifications.businessunitid (filtered by the shopfloor feed) was
unindexed; added index=True + notifications migration 0002.
CI: new migrations-mysql job proves the real multi-site deploy path - fresh
`flask db upgrade` + per-plugin install on utf8mb4 MySQL from empty, asserting
table count + charset and a clean second-run no-op. The pytest suite only
exercises SQLite create_all(), so a regression in the Alembic chain on MySQL
would otherwise ship undetected.
Verified: fresh core upgrade on a scratch utf8mb4 MySQL builds clean + no-op on
rerun (redundant indexes absent, unique assetid kept); plugin migrations applied
+ verified on the dev DB (machineid gone, bu index present). 953 backend tests
pass; naming + pyflakes green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Goal: an LLM or script can migrate an entire legacy database using only
the HTTP API - original history preserved, safely re-runnable.
- X-Import-Mode header (admin only): create/update endpoints across 15
timestamped entity types accept original createddate/modifieddate;
helper exposed via shopdb.api (contract 0.7.0 -> 0.8.0).
- Exact-match natural-key lookup filters on 13 list endpoints for the
lookup-then-upsert recipe.
- Selfhosted USB checkout/checkin accept backdated event times in
import mode.
- docs/IMPORT-API.md: operator manual grounded in the real legacy
schema - order of operations, full table-by-table mapping including
the machines fan-out, idempotent Python importer with dry-run, parity
checks, and decided dispositions for unmigrated tables (DNC config
stays live-fed via the collector; supportteams/appowners map to the
upcoming supportteams model).
635 tests pass; naming green; frontend untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make the app distributable to other GE Aerospace sites (one self-hosted
instance per site, ADR-004). GE values remain the shipped defaults; every
site-specific behavior is now a Setting an admin can change in the UI.
Settings-driven site config:
- Branding: site/QR/badge logos, favicon, primary color (upload endpoints
mirror the map-blueprint pattern; new Settings > Branding section).
- ServiceNow: search/incident/change URL templates ({ticket}), ticket
prefixes, enable toggle. Defaults point at the current
geaerospaceqa.service-now.com global search. Disabled = plain-text tickets.
- Employee-id regex (employeeid_pattern), printer hostname template,
QR label targets (qr_target_printer / qr_target_usb, blank = asset page,
else URL template with placeholders), usb_label_style (barcode|qr).
- West Jefferson floor-plan PNGs removed from the tree; generic placeholder
ships as the map default and sites upload their own blueprint.
Security closeout:
- dashboarddefaults writes now require admin.
- Collector: generic error messages (no str(exc) leak); API key accepted
via X-API-Key header only (BREAKING: querystring api_key removed).
- IP-based login rate limiting (AUTH_RATELIMIT_* knobs) atop account lockout.
- Setting.set() creation race fixed (IntegrityError retry).
Release engineering and docs:
- __version__ 0.5.0 (distinct from __contract_version__, ADR-007),
CHANGELOG.md, Gitea Actions CI config, frontend version aligned.
- One wizard-first install story across README/DEPLOY; new CONFIG.md,
UPGRADE.md, BACKUP-RESTORE.md; CLAUDE.md and ROADMAP de-staled.
- Dockerfile multi-stage build now bundles the frontend; compose binds
MySQL to 127.0.0.1; stale database/schema.sql and one-off SQL removed.
Debt and fixes:
- .query.get() -> db.session.get() sweep; datetime.utcnow() removed
(naive-UTC via timezone-aware now); users.py on authz decorators.
- Fixed 4 stale tests (slides feed shape, shopfloor splitperemployee,
plugin contract purity) and the USB label page field mapping (both usb
modes emit the cmmc shape: device_id/device_desc).
- Health endpoint reports the real version.
248 tests pass; naming/style check green; frontend builds; fresh-DB
flask db upgrade + seeds verified; QR targets verified by decoding
rendered codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Create the tables here" for USB now actually works. Add selfhosted.py backing
every USB endpoint with the app-owned usbdevices/usbcheckouts tables and
returning the same response shape as the external cmmc_usb path. Each route in
routes.py delegates to it when usb_directory_mode=selfhosted (the default).
- Device status derives from ischeckedout + isactive (retired); check-in/out
history is synthesized from usbcheckouts rows (check-out + check-in events);
holder names resolve from the self-hosted employee directory.
- External cmmc_usb mode unchanged, for sites already running that solution.
Verified E2E: create -> list -> checkout -> checkin -> history all correct
against the app tables.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
