84f9cfe13d14844b14a58ffdf2341f98d49bcf46
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
benjamin.baker
|
902339baf9 |
Persist and surface endpoint installed-software (SBOM)
The endpoint agent already reports an installed-software list (SBOM), but ingress dropped it: the collector schema had no entry for it and there was nowhere to store it, so operators could not see what is installed on a PC. Persist it as a per-asset snapshot in a new endpointapplications table, replaced on every report (current inventory is what operators ask for, not install history). Add applications to the collector schema and KNOWN_PAYLOAD_KEYS so ingress stops discarding it, a read route for the raw collection, and a collapsed-by-default Installed software panel to surface it. appid is nullable groundwork for a future Managed Applications link; no matcher is wired yet. |
||
cproudlock
|
60c35728e8 |
Import USB records off an external cmmc_usb database, so a site can retire it
A site running external mode keeps its devices and check-in/out history in a separate cmmc_usb database. There was no way to bring that in, so the choice was to keep the second database forever or lose the history. This adds 'flask usb import-external' - dry run by default, --commit to write - which reads through the same connection settings the routes already use and never writes to the source. Two things it has to reconcile, and they are why it is a command rather than a few INSERTs. The schemas disagree about SHAPE. External is an event log, one row per action; self-hosted is a paired row carrying both times. Pairing walks each device in time order, and real logs after years of use are untidy. The rule is that nothing is invented and nothing vanishes: a drive still out becomes an open checkout and is NOT flagged (that is just a drive somebody has), a return with no matching issue is skipped and named, and two issues in a row leave the first open rather than inventing a return time nobody recorded. Identity is the other. usbdevices.serialnumber is unique and NOT NULL and the external devices table has no serial, so device_id becomes it. Blank or duplicated ids stop that device rather than inventing a key. external device_owner has no column here, so it is appended to the device notes rather than dropped. Also adds usbcheckouts.scannedviruses, the one field with nowhere to land. waswiped already means what external calls sanitized - selfhosted.py was mapping it - but scanned_viruses was HARDCODED None in the log output: the API promised the field and had nothing behind it, so an imported history would have rendered blank. It now reads and writes properly. Nullable, never defaulted to false. An unrecorded scan is not a failed scan, and for removable-media records those are very different claims. Full suite 2131 passed; seven new tests pin the pairing rules. |
||
|
cproudlock
|
784964c907 |
Bring the wilm branch into main: endpoints plugin, agent, policies, CSV import
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
|
||
|
cproudlock
|
a9289ee670 |
printedparts 0003 was an unguarded ALTER, and no test could see it
Unlike its twenty siblings it ran a bare add_column plus create_index, so `flask plugin upgrade-all` failed with "duplicate column name: gagelabtag" on any database whose tables were built by create_all and stamped nowhere - and a chain stops at its first error, so 0004 never applied either. The index is guarded by COLUMN SET rather than by name: create_all names indexes by SQLAlchemy's convention and the migration by this repo's, so a name-keyed guard would have added a second index over the same column instead of skipping. The test that should have caught it was restricted to plugins whose whole chain is a single baseline, on the reasoning that a longer chain meets a second problem - an ALTER whose column is already present - that guarding a create_table does not fix. True, but the conclusion was wrong: the answer is to guard the ALTERs too, not to stop testing the chains that have them. It now covers every post-cutover chain, and was confirmed to fail against the unguarded migration before the guard went in. |
||
|
cproudlock
|
38d1e367fc |
Guides: write a procedure in ShopDB, not beside it
The knowledgebase plugin is a link registry - a description, a URL and some keywords pointing at a document held somewhere else. That leaves the documents themselves as HTML files on a share: no search, no history, no way to know whether the copy pinned to a noticeboard is current, and no way for a technician to tick off where they are. A guide is stored as an ordered list of typed blocks rather than a blob of HTML, because three of the four things this feature exists for depend on it. A tickable step needs a stable id to be remembered; dark mode needs semantic markup rather than pasted colours; and print needs to know where a step begins so it does not split one across a page. Free HTML gives none of that and lets pasted styling fight the theme. Inline text is a restricted Markdown subset - bold, italic, inline code, links, line breaks - escaped BEFORE formatting, so nothing an author pasted can reach the DOM as markup. No fonts, sizes or colours: emphasis is semantic and the theme decides how it looks. That is what makes a guide written next year match one written today, and dark mode work without an author thinking about it. DRAFTS AND VERSIONS. The guide row is the working copy; every save that changes something writes an immutable revision; and a separate pointer says which revision READERS get. Without that pointer, editing a live procedure changes it under a technician who is halfway through following it. Publishing promotes an existing revision rather than snapshotting, so a publish with unsaved edits cannot ship a version nobody reviewed. Restoring writes a new revision with old content, so the record of what people were told in between survives. Saves dedup on a content hash covering title, summary, tags and blocks, so an autosave that changed nothing writes no version - otherwise the history fills with identical entries and hides the three edits that mattered. Progress lives in the browser, not the database: shopfloor PCs share a login, so per-user ticks would show one technician's progress to the next person on that bay, which reads as "already done". Keyed by guide and version, because step 4 of version 3 is not step 4 of version 2. Five permissions, and the split between edit and publish is the point: a technician can write a procedure up while somebody else decides it is fit for the floor. create-without-edit works too - an author may edit their own guide until it is published, after which other people are following it. |
||
|
cproudlock
|
b96637e1a1 |
Let a baseline meet a table that is already there
Reported: `flask plugin upgrade-all` failed the tools plugin with "table securesendsecrets already exists". Every post-cutover baseline emitted a plain op.create_table. That is correct only while a table's existence and its plugin's Alembic stamp can never disagree, and they can. Anything that builds tables from the models - db.create_all(), which `flask db-utils create-all` calls - creates them and stamps nothing. A run that creates a table and then fails before its stamp commits leaves the same split, and re-running is the obvious thing to try. In both cases upgrade-all meets existing tables with an unstamped chain and the create is fatal. Nothing caught it because the migration test DROPS the post-cutover tables db.create_all() built before running upgrade-all. That arranges the one state where an unguarded create works, and never the state the report came from. The new test is the same sequence with the drop removed for the plugin under test. create_table_if_missing and create_index_if_missing are in alembic_template.py next to create_plugin_tables, which had the idempotency rule right and turns out to be called by nobody - its docstring claimed every 0001 used it. Fixed the docstring rather than rewiring the baselines onto it, because the reason they hand-write their DDL still stands: the helper's per-plugin MetaData cannot resolve a foreign key to a core table at CreateTable-compile time. INDEXES ARE MATCHED ON COLUMNS, NOT NAME, and that distinction matters. The two creators disagree about names - db.create_all() takes SQLAlchemy's default convention (ix_securesendsecrets_token) and a baseline spells it the way this repo's naming rules require (ixsecuresendsecretstoken). A name-keyed guard would have called the index missing and added a second one over the same column, which MySQL permits with a warning, so every affected site would have quietly grown duplicates. The test asserts no table ends up with two indexes over the same columns. Applied to all nine table-creating plugin migrations, not just the tools baseline that was reported: the four later migrations that create tables have the same exposure, and geenforce, measuringtools, printedparts and backups are the same defect waiting for a different site. WHAT THIS DOES NOT FIX. It makes a baseline idempotent, not a whole chain. A longer chain can still meet an ALTER whose column is already present - printedparts adding gagelabtag to a table create_all built at head is the case that showed up while testing. Guarding every op one at a time is the wrong answer to that; a database built from the models is already AT head and its chains want stamping rather than replaying, and there is no `flask plugin stamp` today. Left alone deliberately rather than half-built, and the new test is scoped to single-baseline plugins so it does not pretend otherwise. |
||
|
cproudlock
|
d4b49b91b4 |
Move secure send into Tech Tools, where somebody would look for it
|
||
|
cproudlock
|
98458672eb |
Hand somebody a password without leaving it in a chat window
The habit this replaces is pasting a credential into Teams, where it stays until someone deletes the conversation. Tabled on 2026-08-21 pending a decision about scope; the decision was text only, so this is the half that survives it. BACKEND ONLY. There is no page yet, so nothing here is reachable by a person. TEXT, NOT DOCUMENTS, and that shapes everything. Dropping files removed the export-control question entirely - a password is not technical data - along with blob storage, size and MIME handling, and the need to authenticate the recipient so a document fetch could be pinned to a name. TWO DOORS. Creating needs a login AND securesend.create: anonymous creation would be an untraceable way to move data out of a system that otherwise records who did what, and the creator is the only person these rows can ever name. Reading needs neither, because the recipient usually has no account here and the link is handed over on a call - the unguessable token IS the authorisation, which is why it is 128 bits of randomness and why the row is spent on first read. REDEEMING IS A POST, and that is the detail that would have broken this in practice. Teams, Outlook and Safe Links prefetch GET urls to build previews and scan for malware, so a GET that redeemed would let a preview window consume a secret before its recipient ever saw it. The GET route that remains reports only whether a link is still live, so the read page can say "opening this will destroy it" without spending the only view to find out. THE SIZE LIMIT IS THE WHOLE ENFORCEMENT. The key lives in the URL fragment, which browsers never transmit, so the server holds ciphertext it cannot read and cannot tell a password from a page of a drawing. MAX_CIPHERTEXT is therefore not a tuning knob: raising it quietly turns this into the document transfer it was scoped not to be. The refusal says what the limit is FOR, because somebody hitting it is usually trying to send a file and the answer is a different tool. Line counts, file signatures and refusing a pasted file belong in the browser, guide an honest person, and are bypassable by anyone willing to POST directly. NOT ENUMERABLE. Missing, expired and spent all answer identically - telling them apart would confirm to anyone holding a guessed token whether it ever named anything. A spent or expired row is deleted on the way past rather than waiting for a sweeper, because it has no reason to keep its ciphertext. Its own plugin rather than a Tech Tool: that plugin is browser-only by design, and this needs storage, expiry and a view counter. Not an asset extension either, so no assetid and no asset type - the scaffold assumes one and it was removed. FOUND BY ITS OWN TESTS: expiryhours=0 returned 201. `int(x or DEFAULT)` treats an explicit zero as absent, so "expire immediately" silently became "expire in a day". A caller who sends a number is now told when it is not allowed rather than quietly given a different one. BEFORE ANYONE RELIES ON THIS, test that your link rewriter preserves URL fragments: create a secret, send the link to yourself in Teams, click it from the message, and check the key is still after the #. If Safe Links strips it the secret is unrecoverable and the server cannot help, which is the point of the design. The fallback is passphrase mode, worth building only if that test says so. Three product gates caught what a new plugin has to declare, and each was right to: the route-parity gate wanted the endpoints described in api-inventory.json, the lean-build guard wanted the plugin listed in the universal profile the released installer builds from, and the authz sweep wanted the public redeem route named in EXEMPT_ENDPOINTS with a reason. That last one matters most - a mutating route that answers an unauthenticated caller is a decision somebody has to write down, not a default, and reading a secret is a mutation because it spends it. |
||
|
cproudlock
|
2d09fa3201 |
Collect what bays actually have, separately from what they are told to have
ShopDB knew what a bay SHOULD have and nothing about what it DOES. Adding the observed half makes a rollout a review instead of a typing exercise: the floor reports itself in, you look, and you adopt. The collection uses the mechanism that already exists rather than a new one. POST /api/collector/printers dispatches to the printers plugin's apply_collector_payload, the same ADR-006 hook the computers and backups plugins implement. New client script, new plugin-owned table, no new transport and no new credential. OBSERVED AND ASSIGNED STAY APART, and that is the point rather than a detail. A collector report can never write an assignment row: _reconcile_edges is the only function that writes usesprinter/defaultprinter, it has two call sites, and both are authenticated routes a human calls. If a drifted bay's own state were allowed to become what it is told to install, every configuration error would become permanent the next time that PC checked in. Seeding an assignment from observed state is explicit - POST /assignments/seed-from-observed - because a rollout adopts many machines at once. It routes through the same _reconcile_edges as the editor, so there is one write path with two doors, and a queue matching no known printer is REFUSED rather than guessed into an assignment. That last rule is the lesson from the measuring tools: adopting on a weak key produced 43 duplicate instruments. Two fixes on top of what the agents built. The replace deleted a host's previous rows by exact case-folded name while the read path treats a short name and its FQDN as one machine, so a PC that changed spelling appeared to hold every queue twice - which reads as drift that is not there. And the client sent 'reportedat' where the declared schema said 'observedat'. Also here: the legacy loader now imports machines.printerid, the classic system's record of each machine's default printer, which it silently dropped - the production import would have lost every one. And Set-ShopdbPrinters.ps1 finally registers the per-user logon task, staging Apply-ShopdbDefaultPrinter.ps1 to C:\ProgramData first because the share it lives on is mounted only during the enforcement cycle and the task runs at logon when it is gone. VALIDATED ON WINDOWS 11 (build 26200), not just on Linux pwsh, which parses these scripts happily and executes none of the spooler branches. The reporter: posts a correct payload with the X-API-Key header; resolves BaseUrl and CollectorKey from HKLM when given no arguments; suppresses the virtual queues by port; resolves port addresses; and reads the CONSOLE USER's default out of HKU rather than SYSTEM's own, which is a different and usually wrong answer. Two results matter more than the rest. With the spooler stopped, both the cmdlet and the CIM path fail and the script posts NOTHING - verified against a capture server that recorded zero requests, where an empty list would instead have erased that host's observed rows and read as a bay that lost its printers. A genuinely empty host still posts [], because that is a real and different fact. The logon task registers as the Users group at Limited, and falls back to the well-known SID S-1-5-32-545 when the group name will not resolve, as it will not on localised Windows. It was then run with the source directory RENAMED AWAY, to stand in for the share being unmounted, and it still moved the user's default - which is the whole reason the script is staged to C:\ProgramData rather than run from where it lives. The guarantees against damage were re-checked rather than assumed: an empty assignment changes nothing, an unreachable server changes nothing, -WhatIfOnly leaves no queue, no task, no staged file and no registry value behind, and a drifted queue is repointed IN PLACE with Set-Printer so whoever has it as their default keeps it. Not covered by any of this: the driver-staging path, which needs a real vendor package rather than the class drivers a VM ships with. |
||
|
cproudlock
|
8cedf674fb |
Resolve a driver by vendor, and converge a bay's printers from ShopDB
Two rows now cover 41 of 44 printers. printerdrivers could only bind a driver to ONE modelnumberid, so the HP and Xerox universal drivers - which between them cover almost the whole floor - would have needed 21 near-duplicate rows pointing at the same package. That is a table nobody keeps true, and it is why 42 of 44 printers resolved no driver at all. printerdrivers gains vendorid, and resolution runs most-specific-first: the printer's model, then its vendor, then the pre-vendorid convention of matching the vendor word in the driver's name so a site that populated the table before the column existed does not silently lose every driver on upgrade. A row that names a vendor is never matched by its text, because a mis-set vendor resolving to the wrong package is worse than resolving to none. Six rows now resolve 44 of 44 printers at the reference site, and the DesignJet correctly takes its own driver over the HP universal one. Set-ShopdbPrinters.ps1 is the client half: ask for-host, create the queues that are missing, record the desired default. It NEVER removes a queue - a bad minute from the API must not take printers away from a working bay - and it never fetches a driver, because downloading 48 MB while somebody waits to print is the wrong moment. The common scope stages those. Apply-ShopdbDefaultPrinter.ps1 applies the default in the USER's context, which is the only context that can: SYSTEM cannot set a per-user default for somebody else. It also turns off "Let Windows manage my default printer", without which Windows silently overwrites the choice the next time anyone prints elsewhere - a fix that undoes itself within a day. VALIDATED ON WINDOWS 11 AGAINST A LIVE SHOPDB, not only by tests. Printers were assigned to a MACHINE; a PC controlling it, holding no rows of its own, created both queues bound to the right universal drivers, recorded the default and set it, and a second run changed nothing. The first attempt failed with "Relationship types are not seeded - run: flask seed reference-data", which is the deployment trap the plan predicted, caught by an explicit error rather than silently resolving nothing. |
||
|
cproudlock
|
0dc0ac13c8 |
Assign printers to a machine, and let the PC that drives it inherit them
Printers belong to the bay, not to the box currently driving it. The assignment
goes on the MACHINE asset and reaches whichever PC controls it, so a reimaged or
swapped PC comes back with the right printers and nothing had to be saved off the
old one. The asset register is the backup.
New relationship type usesprinter ("this printer is installed here"), beside the
existing defaultprinter ("which of them is the default"), both seeded and both
given a propagation rail through controls. The rails are consumed at READ time
only: the create-time fan-out skips directional through-types, and controls is
directional, so assigning a printer to a machine does not copy rows onto its PC.
That is what keeps own-beats-inherited possible.
Resolution for a PC is its OWN rows if it has any, otherwise one hop out along
controls to the machines it drives. Whole set at a time, not merged: a PC with
its own assignment is overriding the bay deliberately, and the UI has to say so
or a tech "fixing" a bay by editing the PC will shadow the machine's record and
wonder why they keep disagreeing.
GET /api/printers/for-host/<hostname> is what the convergence client asks every
cycle. Resolved by hostname because the collector upserts PCs by hostname and an
office PC has no machine number. An unknown host, a site without the computers
plugin, and nothing assigned all return an empty set - that is the client's
designed no-op and it must stay indistinguishable from "assigned nothing".
PUT /api/printers/assignments/for-asset/<id> reconciles the whole set in one
call. The endpoint was specified, documented and asserted by three tests, and
never written - the verification pass caught that, with four failures. It
validates the default BEFORE any write, so a rejected request changes nothing;
soft-deletes rows that went away; and REACTIVATES soft-deleted rows rather than
inserting, because the unique constraint spans inactive rows and a blind insert
after an unassign raises IntegrityError on MySQL while passing on SQLite.
One default per asset, enforced here because the schema cannot: the constraint is
(source, target, type), which accepts two different defaults quite happily. Two
active defaults are still reachable through the generic relationships endpoint,
where the oldest silently wins - recorded in the proposal as the next thing to
close.
printerdrivers gains drivername: the exact string the INF declares, which
Add-PrinterDriver matches on and nothing else. Deriving it by parsing INFs on
hundreds of bays is fragile; a human confirming it once is not.
|
||
|
cproudlock
|
ab301df9ac |
network: generate a device's asset number instead of asking twice
Every network device on this fleet already follows one convention, applied by hand: AP-<name>, SW-<name>, SVR-<name>, IDF-<name>. 45 records, no exceptions. The create form demanded the asset number anyway, so the same value was typed twice and the convention held only as long as everyone remembered it. The prefix now lives on the device type, and a blank asset number is generated as <PREFIX>-<name>. Left explicit, an asset number always wins: a device carrying a real identifier of its own - a vendor tag, a controller name, a serial - keeps it. That is the platform rule, adopt where an identifier exists and derive only where none does. The prefix is NOT derived from the type name. "Access Point" and "Access Panel" both initialise to AP, and assetnumber is unique, so the second type would collide with the first on every device it created. It is nullable, so a type that wants no prefix generates the bare name rather than needing one invented. Names are sanitised before they reach a business key - the existing data already shows why, with IDF-Telco-Demarc-#1 carrying a '#' into an identifier. An existing prefix is never stacked: IDF-03 under type IDF stays IDF-03. |
||
|
cproudlock
|
84bf5d04ed |
geenforce: let a kiosk say what it is instead of guessing
A display knows whether it is a Dashboard, a Lobby screen or the 3D print room - the dispatcher reads C:\Enrollment\display-type.txt to choose which page to open. It never told shopdb, so the fleet table inferred it from the DashboardDefault fqdn mapping, which is empty unless somebody added a row per kiosk. The column was blank for every display. The client now reads that file and reports it, the report stores it, and the API prefers the reported value with the old mapping left as a fallback for hosts still on an older client. Reported by the device beats inferred from a lookup table, the same way enforcerversion already works. A PC with no display-type.txt reports nothing rather than something invented, and an empty string lands as NULL. Two guards had to learn about it. The DDL parity check read only the 0001 baseline, so a column added by a later revision looked like drift even though its migration existed; it now runs the whole chain, which is what 'do the models match what the migrations build' means. 0002 added a whole table rather than a column, which is why this is the first time it bit. |
||
|
cproudlock
|
f66763e6df |
tools: a Tech Tools section, starting with codes on label stock
A place for the small utilities a technician reaches for at a bench. The plugin owns no API and no tables: every tool runs entirely in the browser, so an air-gapped site gets them for free and a bad network cannot break them. Adding the next tool is a view, a route, and one entry in tools.js. First tool is a barcode/QR generator. Content is typed text, a URL, or a CSV (content,label,copies - quoted fields and an optional header both handled), so a batch of a few hundred is one paste. Label stock is adjustable in inches with five presets, and the code renders to an SVG data URI rather than a PNG: a bitmap gets downscaled to label size and smears the module edges a scanner reads, where SVG rasterizes at the printer's resolution with hard edges. It also carries the dot-grid rule that is easy to get wrong by eye. A thermal head cannot render a fraction of a dot, so a code sized off the grid gets uneven modules; pick a DPI and the page says what the current size lands on and what to use instead. The quiet zone is blank label rather than white baked into the code, so it can be tuned - and it applies to CODE128 too, which needs clear space at each end and was letting bars run into the caption. Tech Tools is the first bundled plugin that owns no schema, which two guards did not model: it belongs in the universal installer profile, and upgrade-all reports it 'no-migrations' where every plugin was assumed to report 'ok'. The migration test now asserts that status explicitly for schema-less plugins, so a table-owning plugin whose chain went missing still fails. |
||
|
cproudlock
|
e0e4cce8bd |
dashboard: fix what a real fleet showed, which tests could not
Three faults, visible only once the board ran against production data. BACKUPS SAID THE WHOLE FLEET HAD STOPPED. The lastseenat backfill was wrong. It seeded from collectedat, reasoning that the last change was the last provable moment - but an unchanged config writes no revision, so a machine whose settings last changed nine months ago got a nine-month-old lastseenat and was instantly reported as a dead backup. Every chain lit up at once, which is worse than no card: it says the site is broken when it is fine. The honest value is NULL. Before the column existed nothing recorded when a config was last confirmed, and inventing a date does not change that. Migration 0003 clears the backfill, and staleness now IGNORES a NULL chain rather than substituting timestamps that mean something else. A chain becomes measurable the first time its PC posts, which for NTLARS is within a day. TONER READ "None%". The supply dict has no 'percent' key - it is 'remaining'. Supply names are also shortened, because "Black Toner Level 4%" spends three words saying what the card already says. THE CARDS READ AS WALLS OF TEXT. Rows wrapped into paragraphs and a card with forty PCs pushed everything below it off the screen. Now: at most five rows with "and N more", one line per row that truncates rather than wraps, meta pushed right and dropped first since it matters least, and severity reduced to a small dot beside an uppercase label instead of a coloured card - six severity-painted cards read as a crisis, which is how a board stops being read. Worth recording that none of this could fail in a test. Every one needed real data on a real fleet. |
||
|
cproudlock
|
6c975a107c |
backups: record that a config was checked, not only that it changed
The stale-backup card could not be built as designed, and the reason is more important than the card. Dedup means an unchanged configuration writes no revision, so collectedat moves only on a CHANGE. A machine stable for six months has a six-month-old newest revision and is perfectly healthy. Keying a staleness card on revision age would have flagged most of the fleet - exactly the noise that makes a board worth ignoring. Underneath that: ShopDB could not distinguish those cases at all. On a no-op the server returned "unchanged" and wrote nothing, so "we checked yesterday and it matched" was discarded. That fact is the one thing a backup system must be able to prove, and the only record of it was a line in a log file on the PC. lastseenat records the check rather than the change. Touched on every matching post including the no-op; set on creation, since a new revision has by definition just been seen; backfilled from collectedat or createdat so existing rows start from the last moment the config can be PROVEN current, rather than from now - claiming a check that never happened would be worse than silence. The card keys on it, one row per CHAIN rather than per asset: a machine with two part markers can have one still reporting while the other stopped, and a per-asset view would report the machine as fine. It stays deliberately silent about assets never backed up, because whether one SHOULD be is a question only the manifest can answer, and guessing would list a hundred healthy machines. The rule lives in services/staleness.py rather than the route, so it is testable without an auth layer in the way - the same split retention.py uses. Threshold is backups_staledays, default 3, and 0 disables the card. |
||
|
cproudlock
|
442a631557 |
Let a site choose the order its board rows run in
Rows ran in display-style order and then alphabetically, so what led the screen was an accident of styling and the alphabet - a new type called Awareness landed above Recertification for no better reason than the letter A. Each type now carries a board position, lowest first, set on the Notification Types page. The migration seeds Recognition at 10 and Recertification at 20 and leaves everything else at 100, so an existing board keeps the order sites already expect. Steps of ten leave room to slot a row in without renumbering the rest. A row shared by several types sits wherever its earliest-ordered type puts it, so a category moves as a unit. |
||
|
cproudlock
|
fca775c737 |
backups plugin: per-asset config backups with revision history
Adds a kind-pluggable backups plugin. Configuration captured from a PC is filed against the MACHINE it controls, with a revision history and download back to the native format. NTLARS/DNC is the first kind. Settings live in the controlling PC's registry but describe the machine, so revisions attach to the machine's asset and carry no foreign key to the PC: history survives a PC being replaced or deleted, and sourcehostname records the handover. Storage splits by kind. Parseable kinds store a dialect-neutral JSON projection in ShopDB and re-render on download; opaque vendor formats (part marker and similar) keep their bytes on the SFLD share with ShopDB holding metadata and the UNC pointer. Two .reg dialects exist in the wild: NTLARS's own Save... export omits the WOW6432Node path segment, scripted exports include it. Parsing strips whichever root matched, so a stored revision commits to neither and download offers both (NTLARS Load... by default, WOW6432Node for direct reg import). Getting this backwards is silent, so the dedup hash deliberately excludes sourcedialect and both dialects of one config dedup to a single revision. Dedup is load-bearing: the collector runs every GE-Enforce cycle across the fleet, so a revision is inserted only when the content hash differs from that asset's latest for that kind. A freshly imaged PC opens NTLARS with a blank General tab. Recording that would make an empty config the newest revision exactly when someone needs the last good one, so a blank MachineNo is rejected rather than accepted as a change. Two of the 320 known-good backups on the share already have that shape. DNC Info card summarises the latest revision on the machine page: General (Cnc, NcIF, HostType), eFocas, Serial, NTSHR when populated (only 18 of 147 machines), and MARK when the machine is a marker. MARK is gated on Cnc=MARKER or the ShopDB machine type, not on the MARK key having content: MARK carries serial defaults on 145 of 147 machines and DncPatterns reads YES on 103 including ordinary lathes, so neither identifies a marker. The info card is owned by the kind (BackupKind.infopanel/buildinfo) and served by a generic endpoint, so the expected successor to DNC ships its own card by adding a class rather than changing the plugin or the panel wiring. Also: schedule and retention settings with a prune that never drops the newest or the oldest revision, and scripts/import_ntlars_backups.py to seed history from the existing per-machine .reg files (144 of 147 resolve to assets). Codec verified against all 320 real backups: round-trips clean through both dialects. Bay-side generation verified on Windows against reg.exe export. |
||
|
cproudlock
|
76c184fe91 |
Give every notification type its own row on the board
The shopfloor board grouped cards by display style alone, so every type set to grid landed inside the Recertification row and every carousel type inside Recognition's - under a heading naming somebody else's type. Setting Awareness to grid put awareness messages under "Recertification Required". Each type now gets a row of its own, titled by its own name, and rotation state is per row: two carousel rows advance on their own indexes instead of sharing one counter, and two grid rows page independently. For the other direction there is notificationtypes.boardcategory. Types sharing a category share one row under the category name, so Change, Awareness and Incident can sit together while Recognition and Recertification keep their own. Blank - the default - means a row of its own. The category is part of the grouping key along with the display style, since a category cannot merge a banner with a row of tiles. A card that names no employee now renders its message as the tile or card, rather than a placeholder face above a blank name, which is what a grid type like Awareness looked like before. The layout fingerprint that makes open kiosks reload now covers the category and the grace window, so a re-grouped board reaches screens that are already up. |
||
|
cproudlock
|
fef5e28952 |
Let an end time mean the card leaves the board
The shopfloor feed kept every ended notification up for a hardcoded 30 minutes, flagged resolved. A card with an 8:00 end time was still on the board at 8:29, which reads as an expiry that did not work - and in the carousel, grid and banner sections it read that way with no visual sign at all, since only the standard cards render the resolved state. The tail is now notificationtypes.gracewindowminutes, set per type on the Notification Types page and defaulting to 0, so an end time means what it says. A type whose cards are worth acknowledging after they clear - an incident, say - opts into a tail, and only that type's cards get one. The feed widens its query to the largest configured tail and then holds each row to its own type's window. That keeps one portable query rather than a per-type interval expression in SQL, and with every type at 0 it collapses to "still showing". Also fixes resolved serializing as null rather than false for a card with no end time, which the and-chain produced. |
||
|
cproudlock
|
f8c4246483 |
Fix model photo upload, and give network devices the model link the page assumed
Three faults around vendor-model photos, found while looking at why an uploaded image did not appear. Saving a model was blocked after uploading a photo. The Image URL field was type="url", and an upload sets it to an application path such as /api/models/image/model-120.png. Native url validation demands an absolute URL with a scheme, so the browser refused to submit the form with "Please enter a URL" for a value the page had just written itself. The field is now type="text", which is what it always needed to be: it holds either a full web address or a path on this server. documentationurl stays type="url". The upload button did not appear when adding a model, only when editing one. That was deliberate - the photo is stored as model-<id>.<ext>, so it cannot be sent before the record has an id - but it reads as a missing feature, and the hint explaining it was easy to miss. A photo chosen while creating is now held and uploaded as soon as the model is saved, and it is dropped if the dialog is cancelled, so it cannot land on the next model created in the same session. Network devices could never show a photo. NetworkDeviceDetail.vue binds its hero image to networkdevice.imageurl, but networkdevices carried only vendorid, with no link to a catalog model, so nothing could populate it - a feature that looked present and could not work. Machines, PCs and printers have carried modelnumberid since July. This adds the same column and relationship, the to_dict branch that exposes modelname and imageurl, the field on the API, and a Model selector on the form so the link can actually be set. The migration is guarded the same way employees0002photo is: on a fresh database the tables come from the SQLAlchemy models, which already declare the column, so an unconditional add fails with "duplicate column name". The foreign key is created only on databases that can add one by ALTER; routing it through batch_alter_table made Alembic's column sort raise "Circular dependency detected" on the fresh-database test. Deploying this needs `flask db upgrade` and `flask plugin upgrade-all` on the server, not just a file copy. |
||
|
cproudlock
|
9d65ef103d |
geenforce: display-readiness batch (server hardening, PS client wiring, display scope)
Get GE-Enforce closer to running on credential-less Intune/Entra display PCs that pull manifest + payloads over HTTPS instead of SMB. Server (plugins/geenforce/api/routes.py): - Rate-limit + 512MB served-size ceiling on GET /payload/<sha256> (reuses the login limiter's cache pattern, config-overridable via GEENFORCE_PAYLOAD_*). - New tests: payload hardening, manifestblobs model-vs-migration parity, and a report-contract test locking the lowercase per-entry report keys. PS client (plugins/geenforce/client/): - Fix New-ShopdbReport per-entry key casing to lowercase (name/action/selfhealed/ exitcode/message) to match what the server reads; the engine emits PascalCase. - Enforce TLS 1.2 in the network functions. - Fetch + merge the fleet-wide common scope alongside the pctype scope (pctype wins on conflict; -NoCommon opt-out). - Normalize whatever the engine returns into a well-formed summary. - Make the empty-cache fail-safe observable: event-log entry + report ping instead of a silent exit 0. Manifest (plugins/geenforce/seed_display_scope.py + docs/GE-ENFORCE-DISPLAY.md): - Seed a gea-shopfloor-display scope: 4 Edge kiosk drift-heal registry entries + 1 data-driven dispatcher (Dashboard/Lobby/3DPrintRoom via display-type.txt). Kiosk EXEs stay image-baked; the manifest heals policy/config drift only. - Documents the common SMB-payload audit (entries needing http/inline before a share-less display can inherit common). Migration registry (shopdb/plugins/alembic_template.py + test): - Register the pre-existing manifestblobs and the new printersupplyalerts tables in PLUGIN_TABLE_OWNERS; update EXPECTED_HEAD_REVISION for geenforce (0002blobs), printers (0002supplyalerts), and printedparts (0004txnrev) which had drifted. |
||
|
cproudlock
|
3a3dff285e |
printedparts stage 17: gage-lab asset tag + print-files redesign
The gage lab assigns real WJRP asset numbers, so identity splits: the internal itemcode stays auto-minted and a new optional unique gagelabtag (migration 0003) carries the lab's number - settable on create/edit, searchable, and resolved by the kiosk for scans and bare keypad digits against the numeric tail of either identifier (unique-match only). The print-files table becomes stacked revision cards - filename with rev/current badges, one meta line, delete pinned right - ending the horizontal scroll in that column. |
||
|
cproudlock
|
aa4bfcd41c |
printedparts stage 15: print-file revision history + role-based alerts
printeditemfiles lands as the plugin's first incremental migration (0002 on the plugin chain - the ADR-008 payoff). Revisions are append-only per item: upload assigns the next number, records the uploader from the JWT, enforces an extension allowlist and a 100 MB cap; download serves the original filename; a permission-gated delete covers wrong-file mistakes. The detail page gains the revision table with a current badge. Unique storedfilename is sized 191 so the index fits MySQL's 767-byte prefix - the per-plugin chain does not apply the core env's ROW_FORMAT hook. Alert recipients gain roles: Role joins the 0.13.0 surface, a role picker on the settings page, and every active member of the selected roles is folded into the deduped recipient list. |
||
|
cproudlock
|
f5cfac33b4 |
printedparts stage 2: models, real 0001 baseline, tables live
PrintedItem (catalog: code, name, image, cached quantityonhand, per-item threshold, bin) and PrintedItemTransaction (the ledger: signed quantity change attributed to a badge-resolved employee). Both registered in PLUGIN_TABLE_OWNERS; 0001 is a post-cutover real baseline. The migration-guard test learns the new expected head. Routes are a placeholder ping until the next stage - the scaffold's list route imported the deleted scaffold model, which surfaces as an empty 'Migration error' because the alembic env imports the models package. |
||
|
cproudlock
|
1c6c7ba14b |
DB review fixes: drop redundant indexes + dead column, add CI MySQL-upgrade job
From the database review (verdict: sound-with-minor-issues). Applies the actionable findings. Redundant indexes: five non-unique secondary indexes duplicated a named idx_* or a unique index on the same column - ix_communications_assetid, ix_computers_hostname, ix_networkdevices_hostname, ix_printers_hostname (each shadowing an idx_*), and idx_usb_serial (shadowing the serialnumber unique index). Removed the redundant index source from the models (column index=True / the extra db.Index) and added core migration 7d25 dropping the live duplicates. The unique ix_*_assetid indexes are kept (they enforce assetid uniqueness). Dead column: usbcheckouts.machineid was a NOT NULL soft-ref to the retired machines table storing sentinel 0 (ADR-001). Dropped from the model + the machineid=0 literal in selfhosted checkout; usb plugin migration 0002 drops it live (downgrade restores it default 0). Index: notifications.businessunitid (filtered by the shopfloor feed) was unindexed; added index=True + notifications migration 0002. CI: new migrations-mysql job proves the real multi-site deploy path - fresh `flask db upgrade` + per-plugin install on utf8mb4 MySQL from empty, asserting table count + charset and a clean second-run no-op. The pytest suite only exercises SQLite create_all(), so a regression in the Alembic chain on MySQL would otherwise ship undetected. Verified: fresh core upgrade on a scratch utf8mb4 MySQL builds clean + no-op on rerun (redundant indexes absent, unique assetid kept); plugin migrations applied + verified on the dev DB (machineid gone, bu index present). 953 backend tests pass; naming + pyflakes green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
d85b33bd68 |
Build GE-Enforce manifest-store plugin (P0/P1): model, importer, parity gate
First execution phases of docs/proposals/ge-enforce-plugin.md. The GE-Enforce manifest becomes shopdb data. P0 scaffold: new geenforce plugin (api_prefix /api/geenforce, default_enabled false, core_version >=0.7.0). Registered in PLUGIN_TABLE_OWNERS (ADR-008); its 0001 baseline really creates the tables. P1a model: one wide manifestentries table + entrytype discriminator (not STI, not JSON blob), manifestscopes (UNIQUE scopename+phase), the three multi-value filter child tables, inusechecks + processes, immutable manifestpublishedversions (frozen rendered JSON), manifestpayloads (inline, capped), pctypealiases (mirror of the engine lib's alias graph). regvalue stored as its raw JSON literal so DWord typing survives. P1c importer + exporter: parse common + gea-shopfloor-* + preinstall.json into draft rows and rebuild the JSON verbatim from rows in sortorder. P1d parity harness (GATE A): filters.py mirrors the engine's four filter functions + alias graph; parity.py proves import+export is behaviorally lossless (field-identical + same-entries-fire across 18 machine-profile fixtures) WITHOUT byte-diffing. Verified PASS against all 11 real reference manifests (64 entries) and a synthetic site-neutral fixture covering every type/filter (the CI gate). First slice (gea-shopfloor-cmm shape): service layer (import/publish/rollback/ export-to-share), CLI (parity, import-share, publish, export-share), and the client endpoint GET /api/geenforce/manifest serving the current published snapshot (never the draft) with ETag/304. Split permissions geenforce.manage/publish/fetch. Tests prove import->publish->serve, draft edits never change served bytes, publish+rollback, and auth (401 unauth/wrong-scope). Contract 0.11.0: added service_token_authorized(scope) to shopdb.api so plugin service endpoints authorize a scoped managed token without importing core token internals. Documented in PLUGIN-HOOKS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
1d21bf0206 |
Add photo management for models and employees; fix stale detail navigation
Model photos: upload/replace/delete on /api/models/<id>/image (admin), stored under instance/modelimages/ with a public serve route; thumbnail plus Upload/Replace/Remove controls in the Models settings modal; the URL field remains as a manual alternative. Employee photos, mode-aware: self-hosted directory employees get upload/replace/delete (photo-<sso> under instance/employeephotos/, employees plugin migration 0002); external directory mode passes the HR-supplied picture URL through read-only (writes 409). One resolver feeds both consumers - the shopfloor recognition/recert kiosk cards and the employee detail hero - in either mode. Navigation fix: router-view is keyed on route path, so following a relationship link between two assets of the same type (machine -> dualpath machine) reloads the page instead of showing stale content; query-only URL changes still avoid a remount. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
cproudlock
|
48d3160bc5 |
Rename the equipment domain to machines; retype the models catalog (ADR-011)
The equipment plugin is now the machines plugin, ending the UI-vs-code vocabulary split while the contract is pre-1.0 and nothing external depends on the old names. - plugins/equipment -> plugins/machines: manifest, class, /api/machines, machines.* permissions, registry key (with an auto-migrating load shim for existing installs). - Tables: equipment -> machines (equipmentid -> machineid) and equipmenttypes -> machinetypes, renamed in the plugin's own migration chain (machines0002rename), idempotent for both upgrading and fresh installs. - The legacy core machinetypes lookup actually types the vendor MODELS catalog, so it is renamed losslessly to modeltypes (models.modeltypeid, /api/modeltypes, Model Types settings page) rather than collapsed, freeing the machinetypes name. Core migration 7d17_machines_rename also flips data in place: assettypes row equipment -> machine, auditlog entitytype, identifier_/search_ settings keys, permission rows, and renames alembic_version_equipment. - Frontend: machinesApi/modeltypesApi, item.machine response shape, assettype value compares 'equipment' -> 'machine' (map, search, custom fields, relationships), routes machines.js with plugin gating retagged, /print/machine-badge, Machine Types (subtypes) and Model Types (catalog) settings pages, machines-by-type report id. - Docs swept; ADRs left as history per the authoring rule. Upgrade: flask db upgrade then flask plugin upgrade-all. Verified: dev DB flipped live (262 machines, 35 modeltypes, 95 models retyped, zero equipment tables remain); fresh scratch-MySQL install produces the new names; 341 tests green; naming/style green; frontend builds; live E2E on machines list/detail, PC relationships, map, reports, and both settings pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
cproudlock
|
22e623c1f6 |
Plugin framework maturation, reports overhaul, theming, and USB frontend repair
Framework: - Per-plugin Alembic migration chains (ADR-008): every bundled plugin carries its own chain with a stamp-only anchor at the ownership cutover; new plugin schema lands in plugins/<name>/migrations/, never the core chain. Deploys add flask plugin upgrade-all. Fixed a latent bug in the shared alembic template (engine URL resolution) and taught the metadata filter to include FK-referenced core tables. - Frontend plugin route gating (ADR-009): plugin routes carry meta.plugin; a disabled plugin's pages redirect to the dashboard via a cached, fail-open check against the new public GET /api/plugins/enabled. - get_reports() plugin hook (contract 0.5.0 -> 0.6.0): plugins contribute report cards; warranty and toner cards moved off the hardcoded list. Reports: - Hub grouped by category with search; inline reports render at the top, are URL-backed (?report=id, back-button and deep links work), expose their server-side filter params as controls, and export CSV. Warranty and Toner pages gained CSV export. - Deleted the dead legacy Warranty Status report (always-zero buckets from a retired column). Theming and fonts: - Inter (variable) bundled locally via @fontsource, replacing the Google Fonts Roboto import - air-gapped installs now render correctly; tables use tabular numerals. - Optional brand_primary_dark_color, brand_accent_color, brand_sidebar_color settings applied to CSS vars at bootstrap. USB frontend repair (views were reading a dead legacy shape): - List/detail/form and the employee profile USB panels remapped to the real API shape (device_id/device_desc/checkinoutlog); employee panels now use /usb/checkouts endpoints; external-mode /usb/checkouts/active honors the badge filter; dead client methods pruned. Also: warranties list page no longer requires login (matches app convention); collector doc rewritten with a GE-Enforce integration guide and paste-ready PowerShell reporter; ADR index and CHANGELOG updated. Verified: 323 tests pass, naming/style green, frontend builds, plugin migration dry-run green on scratch MySQL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
cproudlock
|
4f1638bda0 |
Consolidate bundled plugin schema into the core migration chain
The core chain already owns and reproduces the full bundled schema (deploys run `flask db upgrade` only). The per-plugin Alembic baselines duplicated those tables, so `flask plugin upgrade-all` would conflict - a footgun. Remove the 6 bundled plugin migration dirs; the per-plugin Alembic helpers (alembic_template, PluginMigrationRunner) remain for external/filesystem plugins. upgrade-all now cleanly no-ops for bundled plugins. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
cproudlock
|
42ae985d72 |
Phase 7B: per-plugin Alembic chains for bundled plugins
Each of the six bundled plugins (computers, equipment, network,
notifications, printers, usb) now has its own Alembic chain with a
baseline migration. Sister sites adopting one of these plugins can
manage its schema via `flask plugin migrate <name>` instead of relying
on db.create_all to bootstrap everything.
Existing single-site deploys that bootstrap via db.create_all continue
to work unchanged. The chains coexist; the bootstrap path stays the
operator's choice.
Framework
- shopdb/plugins/alembic_template.py: shared env.py logic + helpers.
PLUGIN_TABLE_OWNERS pins which tables belong to which plugin (explicit
registry, not import-side-effect). _get_plugin_metadata filters
db.metadata to only the named plugin's tables. create_plugin_tables /
drop_plugin_tables emit DDL via SQLAlchemy CreateTable so the table
definitions stay sourced from the models, not duplicated.
- shopdb/plugins/__init__.py: PluginManager.upgrade_all_plugins() runs
pending migrations across every discovered plugin and returns a status
dict. Idempotent (Alembic skips applied revisions).
CLI
- `flask plugin upgrade-all` runs pending migrations for every plugin.
Used on a fresh deploy after the core schema is in place.
Per-plugin scaffolding
- plugins/{computers,equipment,network,notifications,printers,usb}/
migrations/{alembic.ini, env.py, script.py.mako, versions/0001_baseline.py}
- Each env.py is a 5-line shim that sets PLUGIN_NAME and delegates to
the shared template. Each 0001_baseline calls create_plugin_tables(name)
/ drop_plugin_tables(name); no duplication of column definitions.
Tests
- tests/test_plugin_migrations.py (18 cases): every bundled plugin has
an entry in PLUGIN_TABLE_OWNERS, has the on-disk Alembic scaffolding,
and the filtered MetaData contains every owned table (catches drift
between the template's table list and what the models declare).
- 129 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|