Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
Every job on GitHub is warning:
Node.js 20 is deprecated. The following actions target Node.js 20 but are
being forced to run on Node.js 24: actions/checkout@v4, actions/setup-node@v4
Working today because the runners shim it, and a hard failure the day that shim
goes. checkout@v5, setup-node@v5 and setup-python@v6 are the releases built for
Node 24, so the warning goes away rather than being suppressed.
Nine references across five jobs. The comment above them says why, so nobody
pins them back to the v4 generation on the assumption it is the safer number.
The org's actions policy allows GitHub-owned actions with no SHA pinning
required, so the tag change needs no allow-list update.
Two faults with one cause: docs/ is excluded from publication because the prose
becomes the wiki, and two things under docs/ are not prose.
shopdb/core/api/docs.py reads docs/openapi.json AT RUNTIME to serve /api/docs.
Excluding it meant every site installed from the published repo shipped the
ReDoc page, loaded it, and got {"error":"openapi.json not generated"}. The spec
now publishes - it is a product file that happens to live under docs/. The prose
still goes to the wiki and nowhere else, and docs/api-inventory.json stays out,
because nothing reads it at runtime.
The second fault is the reason this was worth chasing. Three gate modules skip
themselves when the file they read is absent, which is correct in a published
checkout and silently wrong on GitHub, which runs against exactly that checkout.
The modules skipped, CI reported green, and an outside branch added nine
undocumented routes and passed. A gate that stops running is worse than no gate,
because nobody looks at it again.
scripts/check-gates-ran.sh asserts each gate ran, with two lists: the ones that
must run everywhere, and the ones needing files publication deliberately leaves
behind - required here, allowed to be absent there. A `gates` job runs it in CI.
Proven in both directions: green in each checkout, and red when the file a gate
reads is removed.
Also purge instance/plugins.json from the publication tree. It was never
published - it is untracked there - but an old copy had sat in that tree for
months, protected from --delete by its own exclude, and it made the published
repo fail its own suite locally: 13 plugins enabled where the code expects 16.
A publication tree that cannot pass its own tests cannot be verified before
shipping. That is the exact trap the comment at the top of the script warns
about, which had claimed a second victim.
Also: concurrency on the workflow, so a second push supersedes the first rather
than queueing behind it holding a self-hosted runner.
A link on its own invites being pasted onto text that is already in the chat
box, and every chat client turns the selection into the link's caption. The
secret then travels as the visible message with the secure link wrapped around
it, protecting nothing.
This is not hypothetical. Six LAPS passwords went out that way in production -
machine names and passwords as the caption, the correct /s/<token>#<key> URL
underneath, doing nothing at all.
Copy message puts a whole sentence on the clipboard: what the link is, how many
opens it has, when it expires, and then the URL. A sentence has nowhere to
attach itself - pasted into an empty box it is a message, pasted over a
selection it replaces it rather than decorating it. Plain text on purpose; a
rich-text flavour is exactly what would let a secret ride along as markup.
The success card now says to delete the password from the chat box before
pasting. The existing hint explains that the key cannot be recovered, which is
true and does not stop anybody pasting the password itself.
Nothing about the link was wrong and nothing about it changed: the URL carries
the token, the key rides in the fragment the browser never sends, the plaintext
is dropped the moment it is sealed, and redeeming stays a POST so a preview
cannot spend it.
Sending somebody a label meant sending a screenshot and a list of settings to
retype, and they would get one of them wrong. Copy share link hands them a URL
that reopens the generator with the content, the label, the copy count and
every layout value that was actually changed.
The whole payload rides in the fragment. A fragment is never transmitted, so it
does not reach Flask, it is not in the IIS log, and the proxy between them never
sees it - which matters because the content is usually an asset number or a
serial, and those have no business scattered through logs. It is NOT secrecy:
the link sits in a chat message and in browser history, and anyone holding it
can read the content. Secure Send exists for the case where that matters, and
the note under the box says so.
Only values that differ from a first-time visitor's defaults are carried, so
the link stays short enough to paste into a chat box; a shared link then wins
over the recipient's own saved settings, because somebody following one is
asking to see THAT code rather than their own last one. An unknown key is
ignored rather than refused, so a link made by a newer version still opens on
an older one. The picture is never carried - a data URL runs to megabytes.
Values are coerced back to the type the field holds. Everything arrives from a
URL as a string, and a nudge of 0 plus '2' is '02'.
Proven end to end: a link made in one browser, opened in a clean context with
no saved settings, rendering the same code.
It struck the heading and left everything underneath looking untouched, which
is the opposite of what somebody ticking "this section is done" means. The
strike now covers every block under that heading, down to the next heading of
the same or higher rank - so a top-level section takes its sub-sections with
it.
That needed the rank, and the HTML importer was throwing it away: h2, h3 and h4
all arrived as a flat 'heading', so a parent heading had no children to cross
out. It was the visible half of a real loss - a converted article kept its
words and lost its shape. Headings now carry their level and render at it, and
a guide written before this reads as flat, exactly as it did.
Code blocks, tables and images inside a finished section fade but are not
struck. A line through a command makes it unreadable, and re-reading the
command is usually the reason to look back at a step already done.
The five imported guides were re-imported so they carry their structure.
Every tool on that page is usable without an account, on purpose - the routes
carry no requiresAuth and say so - and Secure Send is the one write this
product offers a guest. The sidebar entry still never appeared for them: it
named no permission, so the filter that decides what a signed-out visitor may
see had nothing to go on and dropped it.
The plugin had no read permission to name, only tools.secrets, so it gains
tools.view: reaching the page at all. The tools themselves run in the browser
and touch no data, which is why this is about whether the entry appears rather
than about what anybody can read.
The item names either right. Somebody granted only Secure Send needs the entry
that leads to it, and a manager who holds neither still does not see it.
The sidebar hides what the caller cannot reach - that is what makes it useful
to a guest, and what stops a member with a sparse role being offered links that
can only answer 403. It reads the permission off the nav item to decide.
Only four items ever declared one. Machines, PCs, Printers, Network, USB,
Knowledge Base, Notifications, Calendar, Dashboard, Map, Applications and
Reports declared nothing, so the filter had nothing to go on: granting the
guest role machines.view opened the API and left the sidebar empty, which reads
as the grant not working.
Every item now names its read permission - in the plugins, in the core
navigation endpoint, and in the frontend's fallback list, which is what renders
when the endpoint is unreachable and would otherwise disagree with it.
A sweep of the URL map found 165 GET routes that answered anybody, with no
token at all: the whole asset inventory, every machine, PC, printer and network
device, the USB checkout history and who is holding which drive, the staff
directory with its photographs, every core report. Guest access had been argued
about one feature-flag at a time while the front door stood open.
Each of those now carries the read permission it belongs to. That also makes
the guest role mean something: a site can open exactly the reads it chooses,
and nothing follows by accident.
Thirty-three endpoints stay public, and each is named with its reason rather
than left to be inferred: the app shell and its bootstrap, the settings
allowlist (the handler already filters an anonymous caller down to branding and
a few keys), images fetched by <img src> which carries no Authorization header,
what a display kiosk renders before anybody signs in, the parts and USB kiosks,
the client scripts that run as SYSTEM before a user session exists, and a
secure-send link whose unguessable token IS the authorisation.
tests/test_core/test_public_read_surface.py is the standing gate, the companion
to the mutating-route sweep that has guarded writes for months. A new public
read fails CI until somebody adds it to that list in front of a reviewer. Four
further reads need a login but no permission, because the handler decides from
who you are - your own user record, your own API tokens, the role catalogue the
grid renders - and the same test proves each still refuses an anonymous caller.
Verified not vacuous by ungating one route and watching the sweep name it.
The employees plugin gained employees.view; the directory is names, SSOs,
photographs and reporting lines, and it had no permission of any kind.
Around ninety tests asserted payload shapes against endpoints that answered
anybody. They are about what the answer says, not about who may ask, so they
are signed in now rather than the gate being loosened. The ones that assert an
anonymous caller is refused were left exactly as they were.
Only checklist items could be ticked, which answers "what have I done" but not
the question a long procedure actually raises: where was I. Somebody looks away
to do the thing the guide asked for and looks back at a wall of text.
Every step and every heading now carries a checkbox while reading. Ticking one
strikes it through and fades it rather than hiding it, because somebody
scanning back up the page has to still be able to read what they did. The
number is struck through with it, but the step's own "Step 3" label is not -
that is the address of the thing, not part of what was done.
Headings count, not just steps. A guide converted from an HTML article often
has fifty headings and no steps at all, and those are exactly the guides long
enough to lose your place in. That also means the card's "N steps to tick"
stops reading as a broken feature on a guide that is entirely tickable.
Reading only, as the checklists already were: in the editor a tick would be an
edit, and an author clicking around their own draft would quietly change what
the next reader sees as done. Progress stays per browser and per version -
shopfloor PCs share a login, so one technician's ticks must not greet the next
person on that bay, and step 4 of version 3 is not step 4 of version 2.
Whether somebody with no account may see something was being answered one
feature at a time: a securesend_allow_guests setting, an enable flag per kiosk,
and very nearly a third for guides - each with its own settings page, its own
vocabulary and its own hand-written branch inside a route. Every one of them
answered a permissions question, and this app already administers permissions
per role.
So a caller with no token is not nobody: they hold the seeded `guest` role.
require_permission consults it when there is no identity, which is the entire
mechanism. The role is seeded holding NOTHING, so an upgrade opens no door; a
site opens one by ticking a box in the same grid it uses for every other role,
and closes it the same way. Revocation is read on every request rather than
cached, because taking guest access away is what somebody does the moment they
realise what they opened, and it has to bite immediately.
The redundant @jwt_required() above require_permission is gone from 184 routes.
It was belt-and-braces before; now it is actively wrong, because it refuses a
request with no token before the permission check can have an opinion, so a
grant would appear to do nothing.
What a guest may hold is bounded. Any .view or .read permission is grantable.
A WRITE has to be blessed by whoever owns it, through the new
get_guest_grantable_permissions hook (contract 0.21.0), and Secure Send is the
only case that exists: sending a password to a vendor is the one write this
product means to offer somebody who never signs in. Everything else is refused
by the API and greyed out in the grid rather than warned about - a checkbox
that quietly makes deleting an asset anonymous is not a warning anybody reads
twice. A guest write records no creator: the trail says when and from where,
never who, and that is the reason for the bar.
GE-Enforce and Slides gained read permissions so they can be opened to guests
without handing out the manage right. Their read routes accept either through
require_any_permission, so nothing an existing manager could reach has moved.
The navigation endpoint now returns only what the caller can actually reach.
Without that a signed-out visitor gets a sidebar of the whole app where every
link bounces off the login page, which reads as a broken site rather than as a
closed door. A nav item may name several permissions; holding any one is
enough, which is what GE-Enforce and Slides need now that a viewer and a
manager hold different strings.
Two things this found on the way: current_user is a LocalProxy, so
`current_user is None` is False for a request with nobody behind it - harmless
under a required JWT, an AttributeError the moment verification became
optional. And an image pasted into a guide before its first save was never
attached to it, so it was visible to editors and invisible to every reader;
saving now attaches it, and `flask guides fix-images` repairs what is stored.
The guides guest setting written earlier today is gone. Granting guides.view to
the guest role replaces it, which is the point.
Guides are written on one site and needed on all of them, and the sites that
matter are air-gapped from wherever the writing happens. Until now a procedure
was stranded in the database it was typed into: the HTML importer could rebuild
one from its original article, but anything authored in the editor had nowhere
to go.
flask guides export writes one .guide.json per guide, with every image the
blocks reference carried inside it as base64. One file on purpose - a bundle
that needs its images copied alongside it is a bundle that will eventually
arrive without them. flask guides import reads them back, a directory at a
time, and refuses anything that is not a bundle before it writes a row rather
than half-importing a stray .json sitting in the same folder.
What travels is the content: title, summary, tags, blocks, images. What does
not is ids, revision history, authorship and publication state, because those
belong to whichever database issued them - an imported guide starts as a draft
at revision 1, owned by whoever ran the import. The SLUG does travel: it is the
address somebody pastes into a work order, and the same guide should answer to
the same name on every site.
--replace updates the guide already carrying that slug instead of creating a
second one, which is how a corrected guide is re-shipped. It drops the stored
images first: the incoming bundle is the whole truth for that guide, and a
guide re-shipped monthly would otherwise accumulate orphans nothing points at.
Bundles refuse a version they are too old to read rather than importing the
parts they recognise. A newer site's guide is not a guide this one can be
trusted to render.
Proven by round-tripping a five-guide export back into the same site, including
one carrying two PNG screenshots, and reading it in a browser.
A site adopting this app arrives holding procedures already written: styled
HTML pages that get mailed around, opened off a share, and edited by whoever
still has the file. Those pages are exactly what a guide is for, and retyping
one into the editor is how a good procedure stays on a share forever.
flask guides import-html reads the page and produces blocks. It recognises the
vocabulary those pages actually use - a step box with its own number, an alert
in one of three colours, a code block with a copy button, a checkbox list, a
table with a header row - and maps each to the block that already renders it,
lifting embedded images into guideimages so a document stops carrying its
screenshots as base64. Page furniture goes: the header, the footer, and the
table of contents a guide builds for itself from its own headings.
The conversion is lossy on purpose. Colour, fonts, width and nested layout are
dropped, because storing blocks rather than markup is what makes dark mode,
printing and tickable steps work at all. So an import is always a DRAFT and the
default is a dry run that reports what it would produce and writes nothing.
Blocks go through the same validator the API uses: an import must not be able
to store something the editor could not have produced.
The parser is tolerant, because real articles are not well-formed. An unclosed
<p> is normal HTML and the browser closes it at the next block element; without
the same rule one unclosed tag swallows the rest of the article into a single
paragraph, which is what it did before the test caught it.
Proven end to end against four real articles, including one carrying two PNG
screenshots, read back in a browser.
POST /api/reports/email has always taken any {columns, rows} from any caller,
but the Email report button existed on the toner and warranty pages only - not
because those reports are special, but because they happen to be their own
routed pages and the reports page had no button at all. It is now beside Export
CSV for every report the page can build a table from, which is all of them:
plugin reports through their declared columns, core reports through the flat
list they return, and the asset inventory summary by flattening its three
tables into one with the grouping named in a column. Presentation-only fields -
a status colour, a link id - are left out; a column of hex codes is worse than
no column. The shapes live in a module with tests rather than in the view.
The button is now hidden without reports.export on all three pages. Offering
somebody a button that can only answer 403 is worse than not offering it.
Separately, on the USB kiosk a long drive name pushed Take or Return out
through the side of its card, leaving a drive nobody could press. The name is a
flex item and so defaulted to min-width:auto, keeping its full intrinsic width;
it now yields, wraps mid-word for the serial-style names that carry no spaces,
and clamps its description to two lines so one long name cannot make every card
in the row tall. The locker line is never clipped - it is how somebody finds
the drive.
A cabinet of thirty drives was one long column to read through. The free list
now filters from a search box and lays out in as many columns as the panel is
wide - four on a wall display, two on a tablet, one on a phone - while the
steps that hold a single control stay narrow, because stretching one input
across a 24 inch screen reads as a broken page.
A drive is found by its id, by what is written on it, or by where it lives. The
words can arrive in any order, and "locker" is part of what is searched, so
"locker engineer" finds the engineering locker - that is how people say it out
loud. Filtering is client-side: every free drive is already on the screen, and a
round trip per keystroke on a wall panel would be worse than no filter at all.
The matcher lives in its own module with tests rather than inside the view.
The box takes focus as the picker opens so somebody can simply start typing,
and it is cleared when the kiosk resets - the next person at the cabinet must
not inherit a filter and conclude there are no drives left.
Also: both this view and the guides blocks coloured muted text with a
`--textmuted` variable this app has never defined. The browser dropped the
declaration, so every hint, subtitle and locker line was rendering at full
strength. The variable is `--text-light`.
The reports page ran a hardcoded switch over five core report ids and fell
through to `default: console.error('Unknown report')`, and its results panel was
a matching chain of per-id templates ending in a `JSON.stringify`. So a plugin
could register a report, watch it appear in the catalogue, and get a wall of
JSON when it was opened - which is where the three printedparts reports have
been since they were added.
Any report that declares an endpoint is now fetched from it, and any report
returning {columns, rows} renders as a table. That shape is already the
convention on the server side, so a plugin needs no core code at all to have a
working report. The per-id templates stay for the core reports whose payloads
are not tabular.
Export CSV goes through the api client rather than window.open. A new window
carries no Authorization header, so every report behind a real jwt_required -
which is every plugin one - answered the export button with a 401. The core
reports are jwt-optional, which is the only reason the old way ever appeared to
work. It also followed /api/reports/<id>, which does not exist for a plugin
report, instead of the endpoint the report itself came from.
A report that fails now says so instead of leaving an empty panel, and the page
offers the time window the endpoints already accepted - printedparts-consumption
was permanently stuck on its 30-day default because nothing could send one.
Verified in a browser against both plugin report families and the core reports.
CMMC removable-media tracking is an audit obligation, and answering "who had
this drive, and when" meant paging through the check-in/out list by hand.
GET /api/usb/reports/audit returns the events - a take and a return are separate
rows, because a count cannot answer the question an audit actually asks - with
summary figures riding along for the people who only want the shape of it, and
a CSV for the people who want it in a spreadsheet.
The window is named rather than a day count: today, last month, six months, a
year, all time. Nobody should have to work out that a year is 365.
Both directory modes answer in one voice. The external path reads the
cmmc_usb checkinoutlog; self-hosted expands one usbcheckouts row into its two
ends, so a returned drive is distinguishable from one still out, and the window
filter runs in SQL against either end of the loan - a drive taken before the
window and returned inside it is a check-in the window has to show. Rows come
back from the mode-specific half; the CSV, the summary and the response shape
are built once in the route.
Ordering is total in both modes. Timestamp alone ties for two events in the same
second, and the same query then returns them in a different order each run,
which reads as the log being unstable.
The kiosk was a single column of full-width rows top-aligned on the page, so a
cabinet of a dozen drives meant scrolling a wall panel, and the page itself
scrolled - which takes the "Your drives" heading off screen and leaves somebody
looking at a list with no idea whose drives they are. The lists now sit in a
grid that fills the width and scroll INSIDE themselves against a bounded
height, with the panel centred as one block the way the parts kiosk beside it
already is. Two kiosks on the same floor that sit differently on their screens
read as two different products.
Drive ids sorted as text, which puts USB-10 ahead of USB-3. Compared with a
numeric collator so the list runs in the order the labels on the drives read.
fetchUser() existed and was called from nowhere. The signed-in user - including
the permission list every v-if in the app reads - was written once at login and
cached in localStorage, and nothing ever refreshed it. So an admin granted
somebody guides.create, the person was told they had it, and the New guide
button stayed hidden until they happened to log out and back in. Nothing on
screen explained why, and the natural conclusion is that the permission system
is broken.
Called on app boot, so a permission change lands on the next page load.
Reproduced before and after: button hidden, permission granted mid-session,
plain reload, button there.
Only a 401 logs out now. The old handler logged out on ANY failure, so an API
restart or a network blip would have thrown people out of sessions they still
held - a cached user is stale, not invalid.
The kiosk page tells an administrator to enable it in Settings under USB. That
place did not exist - the setting was reachable only through the API or a shell,
which is the same trap the backups plugin shipped with. A settings page and a
catalogue card, so the instruction leads somewhere.
Machines, PCs, printers and measuring tools all carry Print Label / Edit /
Retire / Back to List in a page header. A network device had no header at all:
Edit was tucked inside the hero title row, and Print Label and Delete sat in a
bar at the very bottom of the page, past every card. Managing one meant looking
somewhere different from every other asset type.
They move to the standard header. The bottom bar is removed rather than left as
a second copy, and the two delete handlers collapse into one - the route is a
soft delete, so it is called Retire here as it is everywhere else, and the
confirm says what is kept.
Four things people kept tripping over.
The move controls were a literal ^ and v character. They are arrows now.
Sections can be dragged. The handle is a grip in the block bar, not the block
itself: making the whole container draggable swallows text selection inside its
own textarea, so you could no longer select a word to bold it.
The type palette lived once, at the bottom of the entire guide. Adding a step in
the middle meant adding it at the end and walking it up one press at a time.
There is now an insert point in every gap - and at the end, so there is one way
to add a section rather than two.
Linking to another guide or a KB article meant knowing its slug or id by heart.
A picker searches both by title and writes the markdown link at the cursor. It
offers published guides only - never send a reader to a draft - and it stores an
in-app path rather than an absolute URL, so the link survives the site being
reached by another hostname or mounted under a subpath. The knowledge base is a
separate plugin that a site may not have installed; its absence costs nothing
but its results.
The section headings were derived from position ranges: 10-29 became Assets,
30-49 became Information. A number is for ORDER; it was never a statement about
meaning, and using it as one meant a plugin landed in whichever bucket its
number happened to fall in.
That is how USB Devices - an asset - came to sit under Information beside
GE-Enforce and Tech Tools, and 3D Parts with it. Nobody chose that. It also
meant two plugins picking the same number silently reordered each other:
Warranties and Guides both held 8, the tie broke on plugin load order, and
Guides rendered under Warranties rather than under Slides where it was asked to
go. Warranties moves to 9.
Each nav item now names its section and the renderer groups by what was
declared. An item with no section stays in the unlabelled group at the top,
which is the safe default: it appears, it is simply not filed.
Assets gains USB Devices and 3D Parts. GE-Enforce and Tech Tools move to a new
Fleet section - they are neither assets nor reference material, and calling them
Information was only ever an artefact of their position numbers.
Also: sorting used `position || 99`, and 0 is falsy in JavaScript, so Dashboard -
the only item that uses 0 - sorted to the end of its own group.
The kiosk route lives outside AppLayout, so nothing linked to it - a full-screen
page with no way in from the app is a page nobody finds. It joins the Displays
section beside Parts Kiosk, TV Slideshow and the Shopfloor Dashboard, gated on
the route existing so a build without the USB plugin does not advertise it.
The people who actually take USB drives are standing at the cabinet, and most
of them have no account here. Check-out and check-in were gated behind a login
that was not doing much anyway: both modes already attribute the action to a
scanned BADGE rather than to the signed-in user, so the login was gating the
screen, not the record.
/usb-kiosk is a full-screen touch page modelled on the parts kiosk, off unless
a site sets usb_kiosk_enabled - every route 404s until then, rather than 403,
so a disabled kiosk does not advertise that there is a door here. It works
against the self-hosted tables or an existing cmmc_usb database.
THE LIST IS THE TRADE. These drives carry no barcode, so unlike the parts kiosk
there is nothing to present and the screen has to offer a list - which exposes
more than a scan-first design, where you had to already hold a drive to learn
anything. So the list is not anonymous: an SSO comes first and must resolve to
a real person before any drive is named, which is the same bar taking a drive
already had to clear. It is a POST so the badge stays out of proxy access logs.
The main screen shows the caller's own drives, what is free, and a COUNT of
everything else; holders of other people's drives appear only behind a second
deliberate tap, so nobody reads the whole cabinet off a wall panel in passing.
A badge that resolves to no employee is REFUSED, unlike the signed-in route
which creates the user row. A checkout nobody can trace to a person defeats the
tracking, and at a kiosk there is no operator standing there to vouch for it.
Returning somebody else's drive asks first and then records the badge that was
SCANNED, so the log says who actually handed it back. That is the honest record
and it needs no change to a schema this app does not own. Refusing a mismatch
outright is a setting but not the default: a drive left behind by somebody on
leave would otherwise be unreturnable at the kiosk, which just moves the problem
to a person with a login.
The authz sweep caught these writes, twice - once for the two mutations and
again for the two list endpoints. Both are exempted with the reasoning written
out beside the parts-kiosk entry rather than silenced. The list endpoints are
POSTs that change nothing.
Verified by driving the real page: SSO on the keypad, tap Take, database moves.
.env is committed git-crypt encrypted. Without the key the checked out
file is the raw encrypted blob, and python-dotenv fails reading it as
UTF-8 with "invalid start byte". The migrations job already provides
DATABASE_URL, SECRET_KEY and JWT_SECRET_KEY through env:, so removing
the file is enough. Previously masked by the job dying earlier on a
missing mysql client.
A service container is reachable on 127.0.0.1 only when the job runs
directly on the runner host, which is how GitHub-hosted runners work and
how the old host-executor runner worked. Under a container-mode runner
the job is itself a container on the services network, published ports
land on the runner host rather than the job's own localhost, and the
service answers to its name.
Fixes: pymysql.err.OperationalError (2003, "Can't connect to MySQL
server on '127.0.0.1' ([Errno 111] Connection refused)").
Three failures, all from the workflow inheriting whatever the runner
happened to have rather than declaring what it needs.
- backend ran pytest without installing it. pytest is in
requirements-dev.txt, which nothing installed, so the step died with
"No module named pytest".
- Both Python jobs used the runner's system interpreter. The old host
executor had a matching Python by accident; a container image has
3.10, where alembic additionally requires tomli. requirements.txt is
hash-pinned and compiled on 3.14, so tomli has no hash and pip refuses
the whole install under --require-hashes. 3.14 matches the Dockerfile,
the README and the .github workflow.
- The utf8mb4 step shelled out to the mysql client, which is not in
every runner image. pymysql is already a runtime dependency.
The comment explaining the concatenation fix spelled the example hostname out
in full, which tripped the publication scrub - the same trap the surrounding
comment warns about, since the scrub greps this file for the very strings the
patterns are assembled from. Reworded with a neutral example.
Re-verified after the reword by planting a violation and watching the gate fail.
Guides was shipped without an entry in PLUGINS.md or START-HERE, so a site
adopting the product had no way to find out it exists.
BACKUP-KINDS.md described a host's badge as coming from the most recently
confirmed kind. It has not worked that way since the fleet report was reworked:
the freshest revision per (host, kind) is taken, and then the STALEST of those
kinds decides, because reporting the freshest would let a healthy NTLARS backup
mask a part-marker capture that stopped a month ago. Stalest among kinds that
can be JUDGED - a kind never confirmed is unmeasurable, not stale, and treating
it as the worst value made a host with a working backup show no badge at all.
The backups row also still advertised UDC and CMM kinds that do not exist.
api-inventory, openapi.json and PROJECT-MAP regenerated.
Unlike its twenty siblings it ran a bare add_column plus create_index, so
`flask plugin upgrade-all` failed with "duplicate column name: gagelabtag" on
any database whose tables were built by create_all and stamped nowhere - and a
chain stops at its first error, so 0004 never applied either.
The index is guarded by COLUMN SET rather than by name: create_all names indexes
by SQLAlchemy's convention and the migration by this repo's, so a name-keyed
guard would have added a second index over the same column instead of skipping.
The test that should have caught it was restricted to plugins whose whole chain
is a single baseline, on the reasoning that a longer chain meets a second
problem - an ALTER whose column is already present - that guarding a create_table
does not fix. True, but the conclusion was wrong: the answer is to guard the
ALTERs too, not to stop testing the chains that have them. It now covers every
post-cutover chain, and was confirmed to fail against the unguarded migration
before the guard went in.
The NTLARS/DNC registry carries an FTP password in cleartext and it was stored
verbatim. That put it in the backuprevisions table, on the revision panel which
renders values, and in the download route which hands back a ready-to-import
.reg - so anyone holding backups.view could read it off a web page.
Redaction happens at PARSE, so the secret never reaches the database at all.
Redacting on display would have left it at rest and one query away.
What is deliberately NOT done is dropping the value silently. This codec already
refuses to discard an unparseable value, precisely because a backup that looks
complete and restores a machine with a setting missing is the worst outcome. A
redacted value keeps its name, is listed in the projection, and renders as a
COMMENT naming what must be set by hand. Emitting a placeholder as the password
would produce a machine that looks restored and cannot connect, and the
technician would go debugging the network.
scripts/redact_backup_secrets.py scrubs revisions recorded before this, dry run
by default. It leaves contenthash alone: the hash identifies the configuration a
bay reported, so rewriting it would make every bay's next post look like a
change and insert a duplicate revision across the fleet.
Separately, sourcehostname is now normalized at ingest. samesource() maps '' to
IS NULL for the query while the insert stored the raw '', so rows written by a
client posting an empty hostname could never match their own next post: every
post inserted, the table grew without bound, and the history filled with
identical entries nobody could read. Whitespace did the same thing more quietly.
Every domain catches broadly and returns no rows so one broken table cannot take
the whole search box down. That part is right. On its own, though, it makes a
DOWN domain indistinguishable from an EMPTY one: the technician sees no results,
is offered a spelling correction, and concludes the record does not exist. It
does - the search just could not look.
Failed domains are now named in the response and stated above the results.
The suggestion is deliberately still offered on a degraded run. Suppressing it
was tried first and is worse: a chronically unreachable domain, an external HR
directory say, would silently disable spelling help for every search for ever
with nothing saying why. Naming the failure is the honest fix; withholding a
second useful signal is not.
Guides search also moves to the published revision. It correctly limited itself
to published GUIDES, but matched and displayed title/summary/tags/blocksjson
from the Guide row - which is the WORKING copy. A published guide with
unpublished edits matched on wording that only exists in the draft, and the
reader opened it to find the word absent.
The audit log is append-only and nothing ever removed a row, so it grows for as
long as a site runs. This site reached tens of thousands of pages, which is past
the point of being useful: nobody pages through that to find anything.
Retention is OPT-IN and the default keeps everything. An upgrade that silently
started deleting a site's audit history would be indefensible, so
audit_retentiondays defaults to 0 and nothing is removed until somebody sets a
number. The Site & Facility card shows the log's size and exactly how many
entries a purge would delete BEFORE anything is pressed.
Deleting happens in committed batches. That is not an optimisation: one DELETE
over a million rows builds a single enormous transaction and holds locks across
the table while it runs, which on this server means blocking the collectors
posting inventory. The API caps its batch count so the request cannot outlive a
proxy timeout, and a run that stops early reports how much is left rather than
claiming it finished - a purge reporting success over 900k remaining rows would
be the same silent half-failure this codebase keeps getting bitten by.
`flask audit stats` and `flask audit purge` for the server, the latter reporting
by default and deleting only with --apply, with no batch cap because it is not
behind a proxy and a purge that stops halfway leaves the operator guessing.
audit.purge is a separate permission from audit.view. Reading the log and
destroying it are different acts, and plenty of people need only the first.
Chunking selects a page of ids and deletes those rather than using
DELETE ... LIMIT, which is MySQL-only while the suite runs on SQLite.
The patterns are assembled from fragments so the published script does not
contain the strings it greps for. They used the apostrophe-escape form
('a'\''b'), which EMBEDS a literal apostrophe, instead of the concatenation form
('a''b'), which joins them. Every pattern therefore carried a stray quote in the
middle and could never match a line of real code, so the check passed everything
from the day it was written.
Verified the only way that means anything: plant a violation, watch it fail,
revert. Fixed, it immediately found 25 real hits.
Among them, in seven places, the default ServiceNow URLs - one organisation's QA
tenant, shipped with the integration ENABLED. Any site that had not configured
ServiceNow still rendered ticket links, pointing its technicians at a tenant
they cannot reach; an adopting site would have been pointed into somebody else's
instance entirely. There is no sensible cross-site default for "which ServiceNow
is yours", so the URLs ship blank and the integration ships off, matching the
Zabbix block that got this right.
A named SaaS tenant is as site-specific as a hostname, so it joins the patterns.
Colocated .spec.js files are excluded the same way tests/ already is: sample data
naming a facility is not a shipped literal.
Images were a broken icon for every user. The stored URL was bound straight
to <img src>, and a plain image request carries no Authorization header while
this app keeps its JWT in one, so every fetch was a 401. They are now fetched
through the API client, which also fixes the stored '/api/...' path under a
subpath mount where the API is not at /api at all. A failed image says so in
words rather than rendering as a broken icon, and the editor's own thumbnails
go through the same path - an author needs to see the screenshot they pasted.
Revision and image routes gated on their parent guide. Both were gated on
guides.view alone, and revision ids are sequential, so being able to read one
guide was enough to walk the ids and read every unpublished draft on the site -
the opposite of what docs/GUIDES.md promises. An image in a PUBLISHED guide
still serves to any reader; one that only appears in a draft does not.
The guide list, the tag chips and search now match the PUBLISHED text rather
than the working copy. Matching the working copy handed readers a hit on
wording that is not in the page when they open it.
The slug no longer follows the title. It was regenerated on every save that
carried a title and the editor autosaves every three seconds, so fixing a typo
in a heading silently moved the URL and broke exactly the pasted links the
stored slug exists to protect.
publish() now aborts when the pre-publish save failed. It ran regardless, so
the author saw "Published version N" while readers got the previous revision.
Link rendering: text is escaped before formatting, so a captured URL arrives
carrying entities and was escaped a second time - every multi-parameter
ServiceNow and SharePoint link opened with &amp; in it. Emphasis could also
reach inside an href, and '//host' passed as an in-app link when the browser
reads it as off-site.
Tests are now HTTP-level. The draft-invisibility rule lived entirely in routes
that no test executed, which is how three of them shipped serving drafts.
Five of the seven asset types had no way to take a record out of ShopDB from
the interface. Network devices and 3D printed parts had a button; machines,
PCs, printers and measuring tools did not, so a scrapped tool stayed in every
list and report until somebody went to the API.
The endpoints were already there and already correct. Every one of these
DELETEs sets asset.isactive = False and nothing more, so the record leaves the
lists while its calibration history, backups and relationships stay exactly
where they were.
Called "Retire" rather than "Delete" for that reason: the two existing buttons
say Delete, which reads as destruction of a record that is in fact kept and can
be restored by setting isactive back. The confirm says so too, because the
question somebody needs answered before clicking is whether the history goes
with it.
The button is guarded on the record having loaded, on the two pages whose
action row renders before the fetch resolves - measuring tools showed Retire
beside a hidden Edit while the tool was still loading, which would have called
delete with an undefined id.
printersApi had no delete method at all; added.
USB is deliberately left out. It retires through a status column in the
external CMMC database rather than isactive, has its own /retire endpoint, and
needs its own decision about what retiring means there.
Every detail page read "West Jefferson ShopDB - Machine". Two tabs on two
different machines were indistinguishable, a bookmark said nothing about what
it pointed at, and a link shared into Teams previewed as the section rather
than the thing.
The router cannot fix this alone: afterEach runs before the view has fetched
anything, so it only ever knows the KIND of page. So a detail view names its
record once the record arrives, and both writers go through
composables/pageTitle.js.
THE ORDERING IS THE WHOLE PROBLEM. The router awaits the site settings to build
the "<facility> ShopDB" prefix, so on a cold load it finishes AFTER a view that
fetched quickly and would overwrite the good title with the generic one. The
route's fullPath is claimed before that await, and the generic write stands
down when a record title already belongs to the route being landed on. The
reverse case is handled too: a view that names its record BEFORE the facility
name arrives composed against the placeholder prefix, so the late resolve
rewrites it rather than leaving "ShopDB - CSF16" on a site called West
Jefferson. Both are pinned by tests, because both fail only under timing.
Covers machines, PCs, printers, network devices, subnets, measuring tools, USB
devices, printed parts, applications, employees, knowledge base articles and
guides. Printers reuse the page's own displayTitle, which already discards the
literal name "NONE" a discovered queue can carry, so the tab and the heading
agree.
A record with no usable name, or one that failed to load, keeps the section
name: "Machine" beats "Machine 4192" for somebody scanning a row of tabs.
Long names are cut at a word boundary - knowledge base articles are named with
a whole sentence of description, which a browser otherwise truncates mid-word.
Reports gained meta.title, which the router already supported and nothing used.
That also fixes "Pc Relationships", which the name-derived spelling produced.
Two faults the guides plugin shipped with, both caught by gates that were
already there.
deploy/site-profile-universal.json never listed `guides`. That file is the
profile the released Windows installer is built from, so the wizard would not
have offered Guides at all - and worse, `flask plugin prune-schema` drops the
tables of any plugin absent from the profile at provisioning (ADR-014). A site
installing the next release would have had its guides table dropped.
plugins/guides/models/guide.py imported shopdb.core.models.base directly.
Plugins reach core only through shopdb.api or shopdb.plugins.base (ADR-001),
and shopdb.api already re-exports BaseModel, so this is the same symbol by the
supported path.
The suite was red on main because of these two; it passes again.
Global search has per-domain toggles (search_<type>_enabled), enforced
server-side and edited from Settings. Guides answered the search box but had no
toggle, so it was the one domain nobody could turn off - and the omission would
only have surfaced when somebody went looking for the switch and could not find
it.
Missing key still means enabled, so nothing changes until somebody decides
otherwise.
The repo already enforces that every served route appears in
docs/api-inventory.json, and the guides endpoints did not - so the parity test
was failing. They are now listed with auth, params and a working example each,
beside knowledgebase because the two get read for the same reason.
docs/GUIDES.md covers what the inventory cannot: the block schema and why it is
blocks rather than HTML, the id rule that makes a tick rememberable, what the
server refuses and why validation happens at the door, the working-copy /
revision / published-pointer split that stops a live procedure changing under
somebody following it, and a worked example of creating and publishing a guide
from a script.
It also states plainly what does not exist yet - no importer, orphaned images
never pruned - and what converting existing documents will actually be like:
structured HTML and Markdown map onto block types nearly one for one, while a
plain-text runbook lands as paragraphs and needs marking up by hand afterwards.
Generated ids have to stay stable across re-imports, or every re-import resets
readers' progress.
Guides moves to position 8 in the sidebar, beside Slides, rather than down among
the reference lists in Information. A guide is something somebody goes and does
something with, not a catalogue they look things up in.
A guide was invisible to global search. Core search has a hardcoded block per
domain rather than a hook, so a new plugin stays unreachable until someone adds
one - which is the same shape as the gap this test file already documents for
USB devices and printed parts: records with a list page, labels in the world,
and no way to find them from the search box.
Guides are searched by title, summary, tags AND body. The body matters more here
than elsewhere: people look for a procedure by something written halfway down it,
not by a title they never read. PUBLISHED ONLY - a draft is deliberately
invisible to readers, and search is exactly where that rule leaks if nobody
writes it down. The test proves it with a draft sharing a tag and a body word
with a published guide, so a missing filter would show up as a ride-along.
PROOFING catches what a spellchecker cannot, because none of these are spelling:
a smart quote in a command so it will not run, a checklist item with no text so
a step is silently missing, a heading left with nothing under it after a
reorganise, an image with no alt text, a TODO published by accident. Warnings,
never blocks - somebody publishing at 14:40 with one empty bullet should be
told, not stopped, and a procedure that exists imperfectly beats one nobody
wrote.
The smart-character issue reports once per block with a one-click fix, rather
than once per offending character: an opening and a closing quote are one
problem, and listing both makes the panel look like it is nagging.
The character table itself is written as escapes. The style gate forbids
non-ASCII in source, and a file whose job is detecting smart quotes cannot
contain any - which the gate caught, correctly.
Seven changes, all aimed at the same thing: think about the procedure, not
about blocks.
Live preview beside the editor, rendered with the SAME component readers get,
so what an author sees is what will be published - a separate preview
implementation would drift within a month.
Formatting buttons wrap the selection, so nobody needs to know the markup is
Markdown. New blocks and pasted images land AFTER the block being worked on
rather than at the bottom of the document, which removes the scroll-back that
made adding a screenshot mid-guide annoying.
Enter in a list makes the next item and focuses it, the way every other list in
every other tool behaves; backspace on an already-empty item removes it and
steps back up. Typing a ten-item checklist no longer needs the mouse.
Templates, because the blank page is the real barrier to somebody writing a
procedure down at all. The shapes are lifted from the KB articles the team
already writes, so guides written here read like the ones written before them.
Autosave on a three-second debounce with an honest status line, safe precisely
because the server dedups on content - an autosave that changed nothing writes
no version. Leaving with unsaved work now asks first.
CLIPBOARD: the code block no longer writes to the clipboard. A page that
programmatically copies a shell command is the exact shape of a ClickFix attack,
where malware plants a command in your clipboard and talks you into pasting it
into Run - and uBlock Origin blocked ours, correctly, because it was writing
PowerShell on click. The button now SELECTS the code and says to press Ctrl+C:
same convenience, decision back with the person, and nothing for security
tooling to object to.
AUTOCORRECT is on for prose and off for code and table cells. A code block that
autocorrects produces a guide telling somebody to run a command that does not
work - smart quotes replacing a straight one, an autocapitalised cmdlet - and it
still LOOKS right to the author. The person at the bay gets the syntax error.
Printing a dark-mode page lays a full page of toner down for the background and
knocks the text out of it in white. On a shopfloor printer that is a cartridge
for a handful of guides, and the result is harder to read than the plain
version. Somebody would have found this the expensive way.
The theme tokens are redefined for print only, in an unscoped block because
overriding them has to reach :root. Every guide element already renders through
those variables rather than literal colours, so this is a dozen lines instead of
a rewrite - the same discipline that made dark mode work without dark-mode code.
Backgrounds are dropped rather than printed grey: a callout is identified by its
rule and its heading, which cost nothing. Checkboxes print EMPTY even where they
are ticked on screen, because a printed guide is a worksheet somebody carries to
the bay, and one person's progress is not part of the procedure. The footer
gains the print date beside the version, so a stale copy pinned to a noticeboard
can be spotted.
Also corrects the model and migration found by running this against a real
database rather than SQLite: BaseModel contributes createddate, modifieddate and
isactive to every table and the migration never created them, so the first read
failed on unknown columns; the model had also declared duplicate timestamps on
top, now removed with the API field names kept stable. And the slug is 191
characters, not 300 - it is UNIQUE-indexed, MySQL caps an index key at 767
bytes, and utf8mb4 is 4 bytes per character.
The knowledgebase plugin is a link registry - a description, a URL and some
keywords pointing at a document held somewhere else. That leaves the documents
themselves as HTML files on a share: no search, no history, no way to know
whether the copy pinned to a noticeboard is current, and no way for a technician
to tick off where they are.
A guide is stored as an ordered list of typed blocks rather than a blob of
HTML, because three of the four things this feature exists for depend on it. A
tickable step needs a stable id to be remembered; dark mode needs semantic
markup rather than pasted colours; and print needs to know where a step begins
so it does not split one across a page. Free HTML gives none of that and lets
pasted styling fight the theme.
Inline text is a restricted Markdown subset - bold, italic, inline code, links,
line breaks - escaped BEFORE formatting, so nothing an author pasted can reach
the DOM as markup. No fonts, sizes or colours: emphasis is semantic and the
theme decides how it looks. That is what makes a guide written next year match
one written today, and dark mode work without an author thinking about it.
DRAFTS AND VERSIONS. The guide row is the working copy; every save that changes
something writes an immutable revision; and a separate pointer says which
revision READERS get. Without that pointer, editing a live procedure changes it
under a technician who is halfway through following it. Publishing promotes an
existing revision rather than snapshotting, so a publish with unsaved edits
cannot ship a version nobody reviewed. Restoring writes a new revision with old
content, so the record of what people were told in between survives.
Saves dedup on a content hash covering title, summary, tags and blocks, so an
autosave that changed nothing writes no version - otherwise the history fills
with identical entries and hides the three edits that mattered.
Progress lives in the browser, not the database: shopfloor PCs share a login, so
per-user ticks would show one technician's progress to the next person on that
bay, which reads as "already done". Keyed by guide and version, because step 4
of version 3 is not step 4 of version 2.
Five permissions, and the split between edit and publish is the point: a
technician can write a procedure up while somebody else decides it is fit for
the floor. create-without-edit works too - an author may edit their own guide
until it is published, after which other people are following it.
CI caught this: judging a host by its stalest kind treated a never-confirmed
revision (lastseenat NULL) as the worst case, so it won. The verdict for a NULL
confirmation is tri-state None, which renders as NO BADGE - so a host with a
healthy ntlars backup and one backfilled row showed nothing at all. Hiding a
working backup is the one thing that badge must never do.
A NULL confirmation is not stale, it is unmeasurable. The stalest kind is now
chosen among kinds that can actually be judged, which is the rule staleness.py
already applies: a chain becomes measurable the first time its PC posts and says
nothing until then. If every kind is unmeasurable the host still reports which
kind it has, with a None verdict and no badge.
The test that caught it pinned the old newest-wins behaviour, which part markers
made wrong; it now pins stalest-measurable-wins and keeps its NULL clause, which
was right all along and is what found the bug.
Also worth recording how it was missed: the change was verified against
test_geenforce_reporting.py and test_backups.py, and this assertion lives in
test_geenforce_publish_gate.py, which was never run.
