.env is committed git-crypt encrypted. Without the key the checked out
file is the raw encrypted blob, and python-dotenv fails reading it as
UTF-8 with "invalid start byte". The migrations job already provides
DATABASE_URL, SECRET_KEY and JWT_SECRET_KEY through env:, so removing
the file is enough. Previously masked by the job dying earlier on a
missing mysql client.
A service container is reachable on 127.0.0.1 only when the job runs
directly on the runner host, which is how GitHub-hosted runners work and
how the old host-executor runner worked. Under a container-mode runner
the job is itself a container on the services network, published ports
land on the runner host rather than the job's own localhost, and the
service answers to its name.
Fixes: pymysql.err.OperationalError (2003, "Can't connect to MySQL
server on '127.0.0.1' ([Errno 111] Connection refused)").
Three failures, all from the workflow inheriting whatever the runner
happened to have rather than declaring what it needs.
- backend ran pytest without installing it. pytest is in
requirements-dev.txt, which nothing installed, so the step died with
"No module named pytest".
- Both Python jobs used the runner's system interpreter. The old host
executor had a matching Python by accident; a container image has
3.10, where alembic additionally requires tomli. requirements.txt is
hash-pinned and compiled on 3.14, so tomli has no hash and pip refuses
the whole install under --require-hashes. 3.14 matches the Dockerfile,
the README and the .github workflow.
- The utf8mb4 step shelled out to the mysql client, which is not in
every runner image. pymysql is already a runtime dependency.
The comment explaining the concatenation fix spelled the example hostname out
in full, which tripped the publication scrub - the same trap the surrounding
comment warns about, since the scrub greps this file for the very strings the
patterns are assembled from. Reworded with a neutral example.
Re-verified after the reword by planting a violation and watching the gate fail.
Guides was shipped without an entry in PLUGINS.md or START-HERE, so a site
adopting the product had no way to find out it exists.
BACKUP-KINDS.md described a host's badge as coming from the most recently
confirmed kind. It has not worked that way since the fleet report was reworked:
the freshest revision per (host, kind) is taken, and then the STALEST of those
kinds decides, because reporting the freshest would let a healthy NTLARS backup
mask a part-marker capture that stopped a month ago. Stalest among kinds that
can be JUDGED - a kind never confirmed is unmeasurable, not stale, and treating
it as the worst value made a host with a working backup show no badge at all.
The backups row also still advertised UDC and CMM kinds that do not exist.
api-inventory, openapi.json and PROJECT-MAP regenerated.
Unlike its twenty siblings it ran a bare add_column plus create_index, so
`flask plugin upgrade-all` failed with "duplicate column name: gagelabtag" on
any database whose tables were built by create_all and stamped nowhere - and a
chain stops at its first error, so 0004 never applied either.
The index is guarded by COLUMN SET rather than by name: create_all names indexes
by SQLAlchemy's convention and the migration by this repo's, so a name-keyed
guard would have added a second index over the same column instead of skipping.
The test that should have caught it was restricted to plugins whose whole chain
is a single baseline, on the reasoning that a longer chain meets a second
problem - an ALTER whose column is already present - that guarding a create_table
does not fix. True, but the conclusion was wrong: the answer is to guard the
ALTERs too, not to stop testing the chains that have them. It now covers every
post-cutover chain, and was confirmed to fail against the unguarded migration
before the guard went in.
The NTLARS/DNC registry carries an FTP password in cleartext and it was stored
verbatim. That put it in the backuprevisions table, on the revision panel which
renders values, and in the download route which hands back a ready-to-import
.reg - so anyone holding backups.view could read it off a web page.
Redaction happens at PARSE, so the secret never reaches the database at all.
Redacting on display would have left it at rest and one query away.
What is deliberately NOT done is dropping the value silently. This codec already
refuses to discard an unparseable value, precisely because a backup that looks
complete and restores a machine with a setting missing is the worst outcome. A
redacted value keeps its name, is listed in the projection, and renders as a
COMMENT naming what must be set by hand. Emitting a placeholder as the password
would produce a machine that looks restored and cannot connect, and the
technician would go debugging the network.
scripts/redact_backup_secrets.py scrubs revisions recorded before this, dry run
by default. It leaves contenthash alone: the hash identifies the configuration a
bay reported, so rewriting it would make every bay's next post look like a
change and insert a duplicate revision across the fleet.
Separately, sourcehostname is now normalized at ingest. samesource() maps '' to
IS NULL for the query while the insert stored the raw '', so rows written by a
client posting an empty hostname could never match their own next post: every
post inserted, the table grew without bound, and the history filled with
identical entries nobody could read. Whitespace did the same thing more quietly.
Every domain catches broadly and returns no rows so one broken table cannot take
the whole search box down. That part is right. On its own, though, it makes a
DOWN domain indistinguishable from an EMPTY one: the technician sees no results,
is offered a spelling correction, and concludes the record does not exist. It
does - the search just could not look.
Failed domains are now named in the response and stated above the results.
The suggestion is deliberately still offered on a degraded run. Suppressing it
was tried first and is worse: a chronically unreachable domain, an external HR
directory say, would silently disable spelling help for every search for ever
with nothing saying why. Naming the failure is the honest fix; withholding a
second useful signal is not.
Guides search also moves to the published revision. It correctly limited itself
to published GUIDES, but matched and displayed title/summary/tags/blocksjson
from the Guide row - which is the WORKING copy. A published guide with
unpublished edits matched on wording that only exists in the draft, and the
reader opened it to find the word absent.
The audit log is append-only and nothing ever removed a row, so it grows for as
long as a site runs. This site reached tens of thousands of pages, which is past
the point of being useful: nobody pages through that to find anything.
Retention is OPT-IN and the default keeps everything. An upgrade that silently
started deleting a site's audit history would be indefensible, so
audit_retentiondays defaults to 0 and nothing is removed until somebody sets a
number. The Site & Facility card shows the log's size and exactly how many
entries a purge would delete BEFORE anything is pressed.
Deleting happens in committed batches. That is not an optimisation: one DELETE
over a million rows builds a single enormous transaction and holds locks across
the table while it runs, which on this server means blocking the collectors
posting inventory. The API caps its batch count so the request cannot outlive a
proxy timeout, and a run that stops early reports how much is left rather than
claiming it finished - a purge reporting success over 900k remaining rows would
be the same silent half-failure this codebase keeps getting bitten by.
`flask audit stats` and `flask audit purge` for the server, the latter reporting
by default and deleting only with --apply, with no batch cap because it is not
behind a proxy and a purge that stops halfway leaves the operator guessing.
audit.purge is a separate permission from audit.view. Reading the log and
destroying it are different acts, and plenty of people need only the first.
Chunking selects a page of ids and deletes those rather than using
DELETE ... LIMIT, which is MySQL-only while the suite runs on SQLite.
The patterns are assembled from fragments so the published script does not
contain the strings it greps for. They used the apostrophe-escape form
('a'\''b'), which EMBEDS a literal apostrophe, instead of the concatenation form
('a''b'), which joins them. Every pattern therefore carried a stray quote in the
middle and could never match a line of real code, so the check passed everything
from the day it was written.
Verified the only way that means anything: plant a violation, watch it fail,
revert. Fixed, it immediately found 25 real hits.
Among them, in seven places, the default ServiceNow URLs - one organisation's QA
tenant, shipped with the integration ENABLED. Any site that had not configured
ServiceNow still rendered ticket links, pointing its technicians at a tenant
they cannot reach; an adopting site would have been pointed into somebody else's
instance entirely. There is no sensible cross-site default for "which ServiceNow
is yours", so the URLs ship blank and the integration ships off, matching the
Zabbix block that got this right.
A named SaaS tenant is as site-specific as a hostname, so it joins the patterns.
Colocated .spec.js files are excluded the same way tests/ already is: sample data
naming a facility is not a shipped literal.
Images were a broken icon for every user. The stored URL was bound straight
to <img src>, and a plain image request carries no Authorization header while
this app keeps its JWT in one, so every fetch was a 401. They are now fetched
through the API client, which also fixes the stored '/api/...' path under a
subpath mount where the API is not at /api at all. A failed image says so in
words rather than rendering as a broken icon, and the editor's own thumbnails
go through the same path - an author needs to see the screenshot they pasted.
Revision and image routes gated on their parent guide. Both were gated on
guides.view alone, and revision ids are sequential, so being able to read one
guide was enough to walk the ids and read every unpublished draft on the site -
the opposite of what docs/GUIDES.md promises. An image in a PUBLISHED guide
still serves to any reader; one that only appears in a draft does not.
The guide list, the tag chips and search now match the PUBLISHED text rather
than the working copy. Matching the working copy handed readers a hit on
wording that is not in the page when they open it.
The slug no longer follows the title. It was regenerated on every save that
carried a title and the editor autosaves every three seconds, so fixing a typo
in a heading silently moved the URL and broke exactly the pasted links the
stored slug exists to protect.
publish() now aborts when the pre-publish save failed. It ran regardless, so
the author saw "Published version N" while readers got the previous revision.
Link rendering: text is escaped before formatting, so a captured URL arrives
carrying entities and was escaped a second time - every multi-parameter
ServiceNow and SharePoint link opened with &amp; in it. Emphasis could also
reach inside an href, and '//host' passed as an in-app link when the browser
reads it as off-site.
Tests are now HTTP-level. The draft-invisibility rule lived entirely in routes
that no test executed, which is how three of them shipped serving drafts.
Five of the seven asset types had no way to take a record out of ShopDB from
the interface. Network devices and 3D printed parts had a button; machines,
PCs, printers and measuring tools did not, so a scrapped tool stayed in every
list and report until somebody went to the API.
The endpoints were already there and already correct. Every one of these
DELETEs sets asset.isactive = False and nothing more, so the record leaves the
lists while its calibration history, backups and relationships stay exactly
where they were.
Called "Retire" rather than "Delete" for that reason: the two existing buttons
say Delete, which reads as destruction of a record that is in fact kept and can
be restored by setting isactive back. The confirm says so too, because the
question somebody needs answered before clicking is whether the history goes
with it.
The button is guarded on the record having loaded, on the two pages whose
action row renders before the fetch resolves - measuring tools showed Retire
beside a hidden Edit while the tool was still loading, which would have called
delete with an undefined id.
printersApi had no delete method at all; added.
USB is deliberately left out. It retires through a status column in the
external CMMC database rather than isactive, has its own /retire endpoint, and
needs its own decision about what retiring means there.
Every detail page read "West Jefferson ShopDB - Machine". Two tabs on two
different machines were indistinguishable, a bookmark said nothing about what
it pointed at, and a link shared into Teams previewed as the section rather
than the thing.
The router cannot fix this alone: afterEach runs before the view has fetched
anything, so it only ever knows the KIND of page. So a detail view names its
record once the record arrives, and both writers go through
composables/pageTitle.js.
THE ORDERING IS THE WHOLE PROBLEM. The router awaits the site settings to build
the "<facility> ShopDB" prefix, so on a cold load it finishes AFTER a view that
fetched quickly and would overwrite the good title with the generic one. The
route's fullPath is claimed before that await, and the generic write stands
down when a record title already belongs to the route being landed on. The
reverse case is handled too: a view that names its record BEFORE the facility
name arrives composed against the placeholder prefix, so the late resolve
rewrites it rather than leaving "ShopDB - CSF16" on a site called West
Jefferson. Both are pinned by tests, because both fail only under timing.
Covers machines, PCs, printers, network devices, subnets, measuring tools, USB
devices, printed parts, applications, employees, knowledge base articles and
guides. Printers reuse the page's own displayTitle, which already discards the
literal name "NONE" a discovered queue can carry, so the tab and the heading
agree.
A record with no usable name, or one that failed to load, keeps the section
name: "Machine" beats "Machine 4192" for somebody scanning a row of tabs.
Long names are cut at a word boundary - knowledge base articles are named with
a whole sentence of description, which a browser otherwise truncates mid-word.
Reports gained meta.title, which the router already supported and nothing used.
That also fixes "Pc Relationships", which the name-derived spelling produced.
Two faults the guides plugin shipped with, both caught by gates that were
already there.
deploy/site-profile-universal.json never listed `guides`. That file is the
profile the released Windows installer is built from, so the wizard would not
have offered Guides at all - and worse, `flask plugin prune-schema` drops the
tables of any plugin absent from the profile at provisioning (ADR-014). A site
installing the next release would have had its guides table dropped.
plugins/guides/models/guide.py imported shopdb.core.models.base directly.
Plugins reach core only through shopdb.api or shopdb.plugins.base (ADR-001),
and shopdb.api already re-exports BaseModel, so this is the same symbol by the
supported path.
The suite was red on main because of these two; it passes again.
Global search has per-domain toggles (search_<type>_enabled), enforced
server-side and edited from Settings. Guides answered the search box but had no
toggle, so it was the one domain nobody could turn off - and the omission would
only have surfaced when somebody went looking for the switch and could not find
it.
Missing key still means enabled, so nothing changes until somebody decides
otherwise.
The repo already enforces that every served route appears in
docs/api-inventory.json, and the guides endpoints did not - so the parity test
was failing. They are now listed with auth, params and a working example each,
beside knowledgebase because the two get read for the same reason.
docs/GUIDES.md covers what the inventory cannot: the block schema and why it is
blocks rather than HTML, the id rule that makes a tick rememberable, what the
server refuses and why validation happens at the door, the working-copy /
revision / published-pointer split that stops a live procedure changing under
somebody following it, and a worked example of creating and publishing a guide
from a script.
It also states plainly what does not exist yet - no importer, orphaned images
never pruned - and what converting existing documents will actually be like:
structured HTML and Markdown map onto block types nearly one for one, while a
plain-text runbook lands as paragraphs and needs marking up by hand afterwards.
Generated ids have to stay stable across re-imports, or every re-import resets
readers' progress.
Guides moves to position 8 in the sidebar, beside Slides, rather than down among
the reference lists in Information. A guide is something somebody goes and does
something with, not a catalogue they look things up in.
A guide was invisible to global search. Core search has a hardcoded block per
domain rather than a hook, so a new plugin stays unreachable until someone adds
one - which is the same shape as the gap this test file already documents for
USB devices and printed parts: records with a list page, labels in the world,
and no way to find them from the search box.
Guides are searched by title, summary, tags AND body. The body matters more here
than elsewhere: people look for a procedure by something written halfway down it,
not by a title they never read. PUBLISHED ONLY - a draft is deliberately
invisible to readers, and search is exactly where that rule leaks if nobody
writes it down. The test proves it with a draft sharing a tag and a body word
with a published guide, so a missing filter would show up as a ride-along.
PROOFING catches what a spellchecker cannot, because none of these are spelling:
a smart quote in a command so it will not run, a checklist item with no text so
a step is silently missing, a heading left with nothing under it after a
reorganise, an image with no alt text, a TODO published by accident. Warnings,
never blocks - somebody publishing at 14:40 with one empty bullet should be
told, not stopped, and a procedure that exists imperfectly beats one nobody
wrote.
The smart-character issue reports once per block with a one-click fix, rather
than once per offending character: an opening and a closing quote are one
problem, and listing both makes the panel look like it is nagging.
The character table itself is written as escapes. The style gate forbids
non-ASCII in source, and a file whose job is detecting smart quotes cannot
contain any - which the gate caught, correctly.
Seven changes, all aimed at the same thing: think about the procedure, not
about blocks.
Live preview beside the editor, rendered with the SAME component readers get,
so what an author sees is what will be published - a separate preview
implementation would drift within a month.
Formatting buttons wrap the selection, so nobody needs to know the markup is
Markdown. New blocks and pasted images land AFTER the block being worked on
rather than at the bottom of the document, which removes the scroll-back that
made adding a screenshot mid-guide annoying.
Enter in a list makes the next item and focuses it, the way every other list in
every other tool behaves; backspace on an already-empty item removes it and
steps back up. Typing a ten-item checklist no longer needs the mouse.
Templates, because the blank page is the real barrier to somebody writing a
procedure down at all. The shapes are lifted from the KB articles the team
already writes, so guides written here read like the ones written before them.
Autosave on a three-second debounce with an honest status line, safe precisely
because the server dedups on content - an autosave that changed nothing writes
no version. Leaving with unsaved work now asks first.
CLIPBOARD: the code block no longer writes to the clipboard. A page that
programmatically copies a shell command is the exact shape of a ClickFix attack,
where malware plants a command in your clipboard and talks you into pasting it
into Run - and uBlock Origin blocked ours, correctly, because it was writing
PowerShell on click. The button now SELECTS the code and says to press Ctrl+C:
same convenience, decision back with the person, and nothing for security
tooling to object to.
AUTOCORRECT is on for prose and off for code and table cells. A code block that
autocorrects produces a guide telling somebody to run a command that does not
work - smart quotes replacing a straight one, an autocapitalised cmdlet - and it
still LOOKS right to the author. The person at the bay gets the syntax error.
Printing a dark-mode page lays a full page of toner down for the background and
knocks the text out of it in white. On a shopfloor printer that is a cartridge
for a handful of guides, and the result is harder to read than the plain
version. Somebody would have found this the expensive way.
The theme tokens are redefined for print only, in an unscoped block because
overriding them has to reach :root. Every guide element already renders through
those variables rather than literal colours, so this is a dozen lines instead of
a rewrite - the same discipline that made dark mode work without dark-mode code.
Backgrounds are dropped rather than printed grey: a callout is identified by its
rule and its heading, which cost nothing. Checkboxes print EMPTY even where they
are ticked on screen, because a printed guide is a worksheet somebody carries to
the bay, and one person's progress is not part of the procedure. The footer
gains the print date beside the version, so a stale copy pinned to a noticeboard
can be spotted.
Also corrects the model and migration found by running this against a real
database rather than SQLite: BaseModel contributes createddate, modifieddate and
isactive to every table and the migration never created them, so the first read
failed on unknown columns; the model had also declared duplicate timestamps on
top, now removed with the API field names kept stable. And the slug is 191
characters, not 300 - it is UNIQUE-indexed, MySQL caps an index key at 767
bytes, and utf8mb4 is 4 bytes per character.
The knowledgebase plugin is a link registry - a description, a URL and some
keywords pointing at a document held somewhere else. That leaves the documents
themselves as HTML files on a share: no search, no history, no way to know
whether the copy pinned to a noticeboard is current, and no way for a technician
to tick off where they are.
A guide is stored as an ordered list of typed blocks rather than a blob of
HTML, because three of the four things this feature exists for depend on it. A
tickable step needs a stable id to be remembered; dark mode needs semantic
markup rather than pasted colours; and print needs to know where a step begins
so it does not split one across a page. Free HTML gives none of that and lets
pasted styling fight the theme.
Inline text is a restricted Markdown subset - bold, italic, inline code, links,
line breaks - escaped BEFORE formatting, so nothing an author pasted can reach
the DOM as markup. No fonts, sizes or colours: emphasis is semantic and the
theme decides how it looks. That is what makes a guide written next year match
one written today, and dark mode work without an author thinking about it.
DRAFTS AND VERSIONS. The guide row is the working copy; every save that changes
something writes an immutable revision; and a separate pointer says which
revision READERS get. Without that pointer, editing a live procedure changes it
under a technician who is halfway through following it. Publishing promotes an
existing revision rather than snapshotting, so a publish with unsaved edits
cannot ship a version nobody reviewed. Restoring writes a new revision with old
content, so the record of what people were told in between survives.
Saves dedup on a content hash covering title, summary, tags and blocks, so an
autosave that changed nothing writes no version - otherwise the history fills
with identical entries and hides the three edits that mattered.
Progress lives in the browser, not the database: shopfloor PCs share a login, so
per-user ticks would show one technician's progress to the next person on that
bay, which reads as "already done". Keyed by guide and version, because step 4
of version 3 is not step 4 of version 2.
Five permissions, and the split between edit and publish is the point: a
technician can write a procedure up while somebody else decides it is fit for
the floor. create-without-edit works too - an author may edit their own guide
until it is published, after which other people are following it.
CI caught this: judging a host by its stalest kind treated a never-confirmed
revision (lastseenat NULL) as the worst case, so it won. The verdict for a NULL
confirmation is tri-state None, which renders as NO BADGE - so a host with a
healthy ntlars backup and one backfilled row showed nothing at all. Hiding a
working backup is the one thing that badge must never do.
A NULL confirmation is not stale, it is unmeasurable. The stalest kind is now
chosen among kinds that can actually be judged, which is the rule staleness.py
already applies: a chain becomes measurable the first time its PC posts and says
nothing until then. If every kind is unmeasurable the host still reports which
kind it has, with a None verdict and no badge.
The test that caught it pinned the old newest-wins behaviour, which part markers
made wrong; it now pins stalest-measurable-wins and keeps its NULL clause, which
was right all along and is what found the bug.
Also worth recording how it was missed: the change was verified against
test_geenforce_reporting.py and test_backups.py, and this assertion lives in
test_geenforce_publish_gate.py, which was never run.
The publication scrub gate rejects internal references, and a real site
hostname in a test is still one in the published repo. The rewrite does not
care which server it is.
Snapshots written before the client was fixed stored "W:\backup\partmarker\..."
as their location. That mapping exists only inside the SYSTEM logon session,
only during an enforcement cycle, and only on the bay that wrote it, so the path
recorded against a marker resolved to nothing for the person most likely to open
it.
The script rewrites backuprevisions.sharepath and nothing else - no revision is
created, deleted, re-filed or re-hashed, and no file on the share moves, since
the bytes are already where the UNC will point. Dry run by default.
It matches on the drive letter alone and leaves everything else untouched,
including paths that are already UNC and paths on another drive. This repairs a
specific known mistake rather than normalising whatever it can reach, which is
the version of this script that would eventually rewrite something it should
not have.
rewrite() is kept pure so the decision of WHICH paths to touch is testable
without an app context; the tests cover the already-UNC, other-drive, empty,
case-difference and trailing-separator cases.
The plugin shipped five settings and no page for any of them. The core settings
catalog is a hardcoded list a plugin cannot extend, so backups_shareroot and the
collection interval were reachable only through the API or a flask shell on the
server - which is not a configurable setting in any useful sense, and it is how
today went: the setting simply was not there to set.
The page is contributed the ADR-010 way rather than by editing core - a card
from get_settings_cards plus a route in the plugin's own routes.js, both of
which disappear when the plugin is disabled. Filed under Machines, next to
Machine Types, because that is where the assets these back up live.
It also states the two things the settings do not say for themselves: that
health is judged on when a backup was last CONFIRMED rather than when it last
changed, and that retention prunes database revisions only - files on the share
stay, so enabling it on a share-backed kind leaves orphaned snapshot
directories.
Snapshot paths are now shown as full UNC text in both places: a Snapshot row on
the Marker Contents card, and the existing path on the history page, which
gains a copy button. Deliberately NOT a link - browsers refuse to navigate from
an https page to file://, so a clickable path looks actionable and silently
does nothing, which is worse than no link. Copy, then paste into Explorer.
The path comes off the revision, never composed from the configured share root:
a snapshot written before the layout changed still lives where it was put.
Until part markers, a PC reported exactly one backup kind and "newest revision
wins" was the same thing as the truth. A marker bay reports two: ntlars for the
machine's DNC settings, partmarker for the controller's patterns.
With newest-wins, the daily ntlars post keeps the badge green while the marker
backup has been failing for a month. The failure shows only on that marker's
asset page, which is not where anyone looks to find out something stopped - and
"is this still being checked" is the one question this badge exists to answer.
So a host is green only when everything it backs up is current, and the badge
names the kind that is not. Freshest revision per kind first, because
lastseenat only ever moves on the newest revision of a chain; judging every row
would report a healthy, frequently changing config as permanently stale.
Known consequence, and the right direction to err: a device retired from a PC
leaves its last revision behind, so that host stays stale until the revision is
pruned. It complains about a backup that genuinely is not running rather than
staying quiet about one.
The dashboard card needed no change - it already lists every stale chain
independently, so a stale partmarker chain was never hidden there.
A 'share' kind keeps its bytes on the file share and gives ShopDB only a hash
and a UNC path. That is right for an opaque vendor file, but it means the app
cannot answer the question a tech actually arrives with - what does this part
marker hold, and which pattern disappeared last month. Reading the share from
the app server is not an option: the download routes refuse to mount it on
purpose, because a permissions slip there becomes an unexplained empty panel.
So the collector now accepts an optional manifest alongside the hash: the
index, not the content. Names, sizes, per-file hashes, and the marking system
each pattern was authored for. Advisory throughout - a client that sends none
behaves exactly as before, and dedup still keys on contenthash.
The marker asset gains a Marker Contents card built from it: counts by type,
the controller family, and a diff against the previous snapshot in the same
source chain. Removed files are named rather than counted, because "1 removed"
only tells someone to go and look. Files edited in place are named too - the
file set can be identical while the content hash moved, and reporting no
change there would contradict the reason the revision exists.
The manifest is deliberately NOT covered by contenthash, which attests the
bytes on the share. A client that starts describing itself therefore posts an
unchanged hash, so a no-op backfills a manifest the row does not yet have -
otherwise the card would appear only on markers that had recently changed, and
stay hidden on every healthy one. The same no-op refreshes sharepath, so a
snapshot that moved on the share stops being pointed at where it used to be.
Manifests are validated at ingest rather than at render: a malformed one would
otherwise become the latest revision and fail every view of the asset page
until the content next changed, outliving the client bug that caused it.
sharedir() for the marker kind is overridden to the layout the site actually
uses, so what ShopDB displays and validates matches what the collector writes.
Canonical names chosen by the site: Dell Inc., HP, Zebra. The rule is the same
in each case - keep the spelling that holds the entries - it just lands on a
different form per company. Dell Inc. carries 310 references and 33 models
against 1 apiece for Dell and DellInc.; HP carries 51 and 24 while HP Inc. has
none; Zebra carries 7 and 2 while Zebra Technologies has none.
So HP Inc. and Zebra Technologies are gone from the shipped catalog. Both were
empty rows that only ever offered a second way to spell a vendor already
present, and neither exists in this site's database at all - they would have
arrived as duplicates at a NEW site and nowhere else.
Also removed UnscopedLiveVendor, which is a test artifact. It reached the
catalog because export_catalog.py reads the live development database, so a row
left behind by a lab run was published as reference data and has shipped to
every site seeded since.
Vendors 51 to 48. Verified nothing referenced any of the three before removing
them, and the catalog still has no duplicates by any spelling.
The seed catalog is what a new site inherits, so its duplicates propagate.
Removed the four space-stripped twins it carried: vendor 'DellInc.', model
'OptiPlex7080', and operating systems 'MicrosoftWindows10Enterprise' and
'Windows10', each of which had a correctly spaced original already present.
'OptiPlex5000' was NOT a duplicate - no spaced twin existed - so it was
renamed to 'OptiPlex 5000' rather than dropped, matching every sibling in the
list. Its vendor moved to 'Dell Inc.', which this site treats as the one
canonical Dell.
Vendors 53 to 51, models 128 to 127, operating systems 14 to 12. Verified
afterwards: no exact duplicate rows, no near-duplicates, no orphaned
model-to-vendor or supply-to-model reference.
The legacy data carried space-stripped twins: 'DellInc.' beside 'Dell Inc.',
'Windows10' beside 'Windows 10', 'OptiPlex7080' beside 'OptiPlex 7080'. They
rode into the shipped seed catalog, so every site that ran `seed catalog`
inherited them, and they stay invisible until somebody opens a vendor picker
and finds the same company listed twice.
Not a code fault. The collector strips only leading and trailing whitespace,
matches with ilike, and treats the OS list as lookup-only, so it can neither
have created these nor be creating more. They arrived with the classic-ASP
import, which holds the same three pairs today.
`flask catalog dedupe` merges only whitespace-and-punctuation twins, because
those are one name typed two ways. 'Dell' and 'Dell Inc.' normalise apart and
are left alone: that is a naming choice and only a person can make it. The
surviving row is the most referenced, tie-broken toward the spaced spelling,
which is the one somebody pressed the space bar for.
`flask catalog merge` makes that judgment call explicitly, and refuses to
invent a target that does not exist, so a typo cannot move rows into a new
vendor. `flask catalog rename` handles the case neither can reach: a stripped
name with NO twin, where there is nothing to merge into.
All three report and change nothing unless --apply is passed, and dedupe is
idempotent so it is safe in a deploy script.
References are discovered from the LIVE schema rather than listed in code.
There are 15 foreign keys into these three tables across 7 tables, and
machines.controllervendorid and printerdrivers.modelnumberid are not ones
anybody would enumerate by hand. Every reference is repointed before the row
is deleted, so nothing is orphaned, and a plugin adding a table is covered
without editing this command.
Running `python scripts/export_catalog.py` on a deployed site died with
1045 "Access denied for user 'root'@'localhost'", naming a user that site
does not have.
Nothing loads .env for a plain script run. wsgi.py calls load_dotenv() for the
served app and the flask CLI loads .env for itself, so both reach the real
database; a bare `python scripts/...` gets neither. DATABASE_URL then stays
unset, FLASK_ENV defaults to development, and config.py falls back to a
hardcoded root@localhost URL.
The read-only failure above is the harmless version. Six of these seven
scripts WRITE - backfill, reclassify, retype, collapse, refile, import - and
on any box where those fallback root credentials happen to resolve, they would
have silently rewritten the wrong database with no error at all. That is also
why this survived: the dev .env points at root, so the fallback matched what
the developer expected and nothing ever looked wrong here.
All seven now load .env from the repo root, resolved from __file__ rather than
the working directory, since these are run by hand during a deployment and
rarely from the repo root.
tests/test_maintenance_scripts.py pins it for any future script: every script
that calls create_app must call load_dotenv, before create_app rather than
after it, and with an explicit path rather than a cwd-relative search.
PLUGINS.md listed 14 of the 15 bundled plugins. `backups` had no row: the
per-asset configuration backups the collector takes off shopfloor machines,
kind-pluggable, with view and download as separate permissions because a .reg
is a restorable artifact. Added, with the pointer that separates it from
BACKUP-RESTORE.md, which is the database and a different subject.
PLUGIN-LAB-PRINTEDPARTS said the finished implementation lives on branch
feat/printedparts-plugin. That branch was deleted in the v0.8.0 cleanup; only
the 17 lab-stage tags survived, and they are enough. Says so now, and warns
that the tags are a record of contract 0.11-0.13 rather than today's
conventions. Its manifest example carried core_version >=0.12.0 against a
shipped >=0.16.0.
The printer-assignment proposal still said the client script is not built. It
shipped and released, across v0.11.3 and v0.12.0, and the collector half that
contradicted the assigned-vs-observed split was retired in adc7ef4.
PROJECT-MAP regenerated for the test count.
`flask plugin new` wrote views to frontend/src/views/<name>/ and a route file
to frontend/src/router/routes/<name>.js. Every one of the 16 bundled plugins
keeps its frontend in plugins/<name>/frontend/ instead, staged into
.plugins-staged by stage-frontend.mjs and aggregated into routes.gen.js.
It was not visibly broken, which is why it lasted: the router still globs
./routes/*.js during the Phase 4 dual-location transition, so a scaffolded
plugin's pages loaded. What a new plugin author silently lost was ADR-013 lean
builds. Pruning works off the staged tree, so a frontend in the core tree
cannot be excluded from a site build, and deleting the plugin strands its
views behind.
The templates for the correct layout already existed; only the output paths
were wrong. Also fixed the imports they carry: core imports now use the @
alias (@/api, @/components/...), which resolves both before and after staging,
where the previous ../../api was correct only in the old location. Sibling
views stay relative and travel with the route file.
frontend_dir is gone from scaffold_plugin. There is no second output tree to
point at, and a plugin developed in its own repo now gets the same frontend as
a bundled one rather than being skipped.
Proven end to end: scaffolded a throwaway plugin, confirmed it stages, builds
real chunks, and that SITE_PLUGINS=network prunes it, which is what the old
layout could not do.
A NOTICE result linked to /notifications, so clicking the row search had
already identified dropped the reader on the list page to find it again by
hand. The id was in the payload the whole time; only the URL ignored it.
`/notifications/<id>` is the existing notification-detail route.
The subnet searcher had the same defect two functions away: an IP search
resolves which subnet the address falls inside, then linked to /network, the
hub. It now links to /networks/<id>, the subnet detail route.
Every other search result type already carried its id. These were the only
two that did not.
The printer feature rests on one rule: a collector report is OBSERVED state and
never ASSIGNED state. If a drifted bay's own state can become what it is told to
install, enforcement stops meaning anything, because a bay that installed the
wrong printer makes itself right simply by reporting it.
_sync_printer_links broke that rule. It ran on every collector PC push and, when
the payload carried defaultprinter or printers, wrote defaultprinter and
connectedto rows from the report - defaultprinter being one of the two types
_reconcile_edges is meant to own alone, from authenticated routes a human calls.
It was recorded as dormant. It was not. The collector schema ADVERTISED both
keys and the example client in COLLECTOR-INTEGRATION.md set
$payload['defaultprinter'], so the documentation was recruiting callers for it -
while the same document, under "Observed is not assigned", promised that nothing
on this path writes a usesprinter or defaultprinter row. The document
contradicted itself and the code followed the wrong half. Nothing in the fleet
armed it, because the first-party reporter never sent those fields, but any
third-party client written from the docs would have.
Retired the function, its _sync_one helper and the now-unused
PRINTER_LINK_ORIGIN label, and removed both keys from the schema.
IGNORED, NOT REJECTED. Unknown payload keys are not validated, so a client
written against the old docs keeps pushing successfully and only its printer
keys stop doing anything. The response warns, naming the keys and pointing at
POST /api/collector/printers for reporting queues and seed-from-observed for
adopting them. Dropping them in silence would leave an integrator believing
assignments were still being written, which is a worse failure than the one
being fixed.
The docs were the larger half of this. The schema table rows are gone, the
"PC -> printer relationship sync" section is replaced by a note saying what
happens now to anyone still sending the keys, and the example client no longer
enumerates Win32_Printer at all - it points at Report-PrintersToShopDB.ps1,
which posts to the observed endpoint and posts NOTHING on a failed enumeration,
since an empty list would erase the host's rows.
test_collector_printers.py pinned the old behaviour, so it now pins the
opposite: the keys create no relationships, the push still succeeds and says
why, a payload without them stays quiet, a human-made default survives a report
naming a different printer, and the schema no longer offers the keys.
Reported: the warranty page cannot be searched by machine number.
It was there to be found. _related_machine puts the machine's number in the
payload, and it does so for a stated reason - a shopfloor PC is bought,
warranted and replaced as a PC, but it is FOUND by the machine it drives, and
nobody walks the floor looking for an asset number. The table renders it as a
chip. The filter looked at vendor, service level, service tag and asset number,
and never at the machine. So the one identifier the feature was designed around
was the one thing the search box could not match.
This is how a field list goes wrong: a column was added to the table and not to
the filter, and nothing connects the two. Moved the field list into
warrantySearch.js so a test can see it, following controllingPc.js and
labelPages.js. Machine name is searched as well - somebody who knows the bay by
"the Makino" should not have to look up its number to find its warranty.
The specs also pin the case that made the old inline version fragile: a covered
asset whose machine is null, which is every printer and any PC nobody has linked
yet, must not throw. And a missing field must not match: String(undefined) is
'undefined', so a naive coercion makes a search for "und" hit every row with an
empty vendor.
Not changed, worth knowing: the covered asset's NAME is still not searched, only
its number. That looks like the same omission, but it is a wider change than the
report - the placeholder has always said "asset" without saying which half - so
it is left for a deliberate decision rather than folded in here.
Reported: `flask plugin upgrade-all` failed the tools plugin with "table
securesendsecrets already exists".
Every post-cutover baseline emitted a plain op.create_table. That is correct
only while a table's existence and its plugin's Alembic stamp can never
disagree, and they can. Anything that builds tables from the models -
db.create_all(), which `flask db-utils create-all` calls - creates them and
stamps nothing. A run that creates a table and then fails before its stamp
commits leaves the same split, and re-running is the obvious thing to try. In
both cases upgrade-all meets existing tables with an unstamped chain and the
create is fatal.
Nothing caught it because the migration test DROPS the post-cutover tables
db.create_all() built before running upgrade-all. That arranges the one state
where an unguarded create works, and never the state the report came from. The
new test is the same sequence with the drop removed for the plugin under test.
create_table_if_missing and create_index_if_missing are in alembic_template.py
next to create_plugin_tables, which had the idempotency rule right and turns out
to be called by nobody - its docstring claimed every 0001 used it. Fixed the
docstring rather than rewiring the baselines onto it, because the reason they
hand-write their DDL still stands: the helper's per-plugin MetaData cannot
resolve a foreign key to a core table at CreateTable-compile time.
INDEXES ARE MATCHED ON COLUMNS, NOT NAME, and that distinction matters. The two
creators disagree about names - db.create_all() takes SQLAlchemy's default
convention (ix_securesendsecrets_token) and a baseline spells it the way this
repo's naming rules require (ixsecuresendsecretstoken). A name-keyed guard would
have called the index missing and added a second one over the same column, which
MySQL permits with a warning, so every affected site would have quietly grown
duplicates. The test asserts no table ends up with two indexes over the same
columns.
Applied to all nine table-creating plugin migrations, not just the tools
baseline that was reported: the four later migrations that create tables have
the same exposure, and geenforce, measuringtools, printedparts and backups are
the same defect waiting for a different site.
WHAT THIS DOES NOT FIX. It makes a baseline idempotent, not a whole chain. A
longer chain can still meet an ALTER whose column is already present -
printedparts adding gagelabtag to a table create_all built at head is the case
that showed up while testing. Guarding every op one at a time is the wrong
answer to that; a database built from the models is already AT head and its
chains want stamping rather than replaying, and there is no `flask plugin stamp`
today. Left alone deliberately rather than half-built, and the new test is
scoped to single-baseline plugins so it does not pretend otherwise.
Reported: moving machine 3016 on the map and saving answered "this
relationship already exists" and refused the save.
The map had nothing to do with it. Every save of the machine form re-POSTed the
controlling-PC link, and the server was right to refuse it - the row was already
active, and assetrelationships is unique on (source, target, type). Any edit to
a machine that has a controlling PC hit this; a position change is just the one
somebody happened to be making.
The form asked the wrong question. It used "do I know the existing link's id" as
a stand-in for "is there an existing link", and read that id from
rel.assetrelationshipid - a key the API has never emitted. to_dict serialises
column names and the column is relationshipid, so the read was undefined on
every load while controllingPcId was populated from the same relationship. The
two answers disagreed, and the disagreement was invisible: undefined is not an
error, it is just a falsy id.
That single conflation caused both failures. Saving anything re-created the
link, and CLEARING the controlling PC did nothing at all, because "no selection
and no known id" reads as "nothing to do" - so removing a machine's controlling
PC silently kept it. Only the first of those was reported, and the second is the
one somebody would have trusted.
Fixed by making the LOADED PC the authority on whether a link exists, with the
relationship id used only to delete a row. If that id ever goes missing again
the worst case is a link that cannot be removed, never a duplicate create.
The decision moved to controllingPc.js as a pure function so it could be pinned
by tests, following labelPages.js. Unchanged is now explicitly no-op rather than
delete-then-create: re-linking a machine to the same PC opens a window where it
has no controlling PC at all, and resolve_asset_position walks that edge, so a
failure between the two calls would drop the machine's inherited position along
with its link. The specs also cover a select handing back '42' against an API
that gave 42, which would make an untouched link look like a swap and reach the
same duplicate POST by another route.
Not the printer assignment, which was the first guess: PrinterAssignmentPicker
makes no relationship calls, and its save is wrapped in a catch that only logs,
so it cannot surface an error from this form at all.
Secure Send shipped on 21 August and nothing outside the code caught up. The
roadmap still listed it under Deferred as "tabled", and gave as the reason that
the tools plugin is browser-only by design with get_blueprint and get_models
both empty - which stopped being true in the same day's commits that gave it a
blueprint, a table, a migration chain, a permission and a setting. A reader
following the roadmap would have concluded the feature does not exist.
The generated API docs were fine: api-inventory.json and openapi.json both
carry /api/tools/secrets, so the contract gates did their job. What was missing
was every word written for a person, and that is the half where the security
properties live.
docs/SECURE-SEND.md is the new page. The parts worth having in prose rather
than inferring from the code: that a lost secret is lost to everybody including
an administrator, which is the same property as "a compromised database yields
nothing" seen from the other side; that the 4096-character limit is the entire
server-side enforcement, because a server that cannot read what it holds cannot
have an opinion about content, so raising it turns a password tool into an
unaudited file transfer; that redeeming is a POST because preview prefetch in
chat and mail clients would otherwise spend a one-view secret before the
recipient ever sees it; and that somebody should send a link to themselves
through their own mail or chat client and confirm the key survives after the #
before a real credential goes through it, because a product that rewrites links
can strip the fragment and nothing on this side can help.
securesend_allow_guests was in the database, on a settings page, and in no
document. It is now in CONFIG.md with what it actually does - an unauthenticated
write endpoint, rate limited per address, failing closed - rather than only its
one-line description.
Also corrected along the way: site_base_url was documented as scheme + host
while the code deliberately tolerates a subpath in it, and the bundled plugin
table in PLUGINS.md had no row for tools at all.
Still missing after this: PLUGINS.md has no row for backups either.
Reported: no Tools section in Settings. The setting existed in the database and
had no page, which from a person's side is indistinguishable from the feature
not existing.
get_settings_defaults SEEDS a setting. get_settings_cards (ADR-010) is what puts
it in the rail. Declaring only the first is a silent half-feature: nothing fails,
no test notices, and the switch is unreachable.
The page states the cost where the DECISION is made rather than in a document
nobody opens - turn it on and it says a guest-created link records no creator,
so the audit trail can say when and from where but never who.
Admin-only, like every other settings page, because this one decides whether an
unauthenticated caller may write to the site.
Also caught while wiring it: the card asked for a 'shield' icon, which is not in
the settings iconMap, so it would have rendered the fallback puzzle piece - the
glyph that reads as "nobody chose this". Uses wrench, matching Tech Tools.
The row still comes from `flask plugin upgrade-all`; the card appears either
way, and without the row the page loads empty.
/tools, /tools/codes and /tools/secure-send were all behind requiresAuth. None
of them needed to be, and one of them actively contradicted a setting added an
hour earlier: securesend_allow_guests lets somebody with no account create a
link, and the page they would create it on was behind a login wall.
The label generator runs entirely in the browser - no API, no data, nothing to
protect - so gating it only ever cost a guest the use of it.
THE SERVER STILL DECIDES who may create a secret. A signed-in user needs
tools.secrets; a guest needs the site to have opened guest creation. Making the
page reachable is not the same as making the action allowed, and the two were
conflated.
The page now reads the setting on load, so a guest at a site that has NOT opened
guest creation is told to sign in instead of being handed a form that was always
going to be refused - with a line saying that receiving a link never needs an
account, because somebody who lands there confused usually has a link they
should be opening instead.
A signed-in user always sees the form, even without the permission: the server's
"you do not have permission" is a different answer from "sign in", and
collapsing them would send somebody to a login page they are already past.
