The docs lived on one lineage and the fleet code on another. That is why six
pages described a shared fleet key as how an agent authenticates, why
COLLECTOR-AUTH had to warn that the per-PC credential code was not in the
repository it sat in, and why ADR-019 recorded a decision whose implementation
was somewhere else. Documentation that cannot see the code it documents goes
stale by construction, and no amount of editing fixes that.
So the corpus moves here: 84 files, 56 pages, 20 ADRs, the proposals, INDEX.md,
llms.txt and both gates. Filenames are unchanged, because a wiki page name is
derived from the filename and a rename breaks every link anybody saved.
THE POINT OF THE MOVE IS THE CORRECTION IT ENABLES. Every caveat that said "not
in this tree" was false the moment the files landed beside the code, so each was
rewritten from what is actually here:
- COLLECTOR-AUTH gained the real enroll request body, the two bounds on the
unauthenticated route (per-IP rate limit, pending cap with stale rows dropped
before a real PC is refused), and the rejected pending reason that binds to
the endpoint identity rather than the nonce.
- ADR-019 now describes an implementation that is present, including its tests,
and says plainly that what remains is the FLEET migration, not the code.
- proposals/device-identity claimed phase 1 was unbuilt. plugins/computers
declares deviceid on the collector schema and prefers it over the older
fallbacks. Corrected in the page, its summary and the index.
WINDOWS/IIS IS NOW DOCUMENTED AS THE LEGACY PATH, per the product decision, and
carefully: it is what production runs today. Every Windows page opens by saying
it is supported, not for a new site, and where a new site should go instead. No
procedure was cut - an operator on an IIS box at 2am needs these pages to work
exactly as well tomorrow. MIGRATE-TOPOLOGY stops being a footnote and becomes
the off-ramp, which is why it nearly doubled: what was there did not match how
the Docker stack really works here.
The Docker material that already lived on this line was kept and given
ownership rather than buried under the ported Windows-centric prose: DEPLOY now
describes the four services, the migrate service that runs the upgrade chain
before api starts, TLS terminated by nginx, and the plugin count corrected from
thirteen to seventeen - a number both lineages had wrong.
ROADMAP is rewritten from evidence rather than memory: what has shipped, read
out of the changelog, the ADRs and the manifests, and ahead of it exactly two
items - finalize a distributed release and polish the setup wizard.
PROJECT-MAP is generated and was stale; the GENERATOR was what was wrong.
--check passes now.
API-REFERENCE is retired to a pointer. /api/docs serves 573 operations with
derived auth, query parameters and harvested examples, and a hand-written page
competing with that is a liability. The file stays so its wiki page and its
inbound links survive.
AND THE MOVE TURNED ON A SCRUB THAT HAD NEVER RUN HERE. test_docs_publishable
keys on docs/*.md, which this line did not have, so it skipped - its own comment
predicted what would surface. Porting the corpus activated it and it found 48
lines that must not reach a published wiki: branch names abbreviating the site,
assistant-tooling paths, the site name in two deployment pages, and an employee
ID in a Windows path that the scrub does not even look for.
The spec described 573 operations with a schema each and almost nothing a
reader could copy: 12 operations carried an example, and no operation described
a query parameter at all, because a Flask handler's query string is invisible to
url_map. A route whose real interface is ?code= documented nothing.
Both are now derived from evidence rather than written by hand.
QUERY PARAMETERS come from each view's AST - request.args.get, the subscript
form, getlist, and this repo's own helpers - and a read that does not resolve to
a literal name is not emitted. 0 to 87 operations, 350 parameter objects. The
enumerations some handlers validate (sort=assetnumber|hostname) are deliberately
NOT published: deriving the accepted set reliably needs more than a literal
read, and a wrong enum is exactly the invented contract this work avoids.
EXAMPLES are harvested from the test suite, which already sends real bodies and
asserts real responses, so they are transcripts rather than inventions. 12 to
about 324. The harvest is re-runnable and optional; generation works without it.
THE HARVEST PUBLISHED TWO REAL CREDENTIALS, and that is the important part of
this commit. A bootstrap token and a kiosk setup code reached docs/openapi.json
inside URL values - a file served at /api/docs on production and copied to a
wiki. Neither scrubber layer could see them: the keys are 'scripturl' and 'url',
which name no secret, and the opaque-value rule excludes ':' and '/' on purpose
so that example URLs survive. Nothing was committed.
Four changes came out of it:
- a URL-credential rule that scrubs the parameter VALUE and keeps the URL, so
?t= is redacted and ?page=2&perpage=50 still reads as an example
- redaction runs on the way OUT as well as on the way in. Harvested examples
live in the inventory and a refresh preserves them, so a scrubber fixed after
a harvest protects only future harvests while the stored copy keeps
publishing. The published file can no longer carry a secret the current rules
would catch, whatever the inventory holds
- the leak gate now holds its own patterns instead of importing the generator's.
Checked with the scrubber's own rules it could only ever assert that redact()
ran, and it was green while both tokens sat in the file. It now knows the
product's token prefixes, a URL-credential shape, and a crude entropy
backstop. Replanting the exact credential that shipped fails it
- a DELETE only declares a body when the HANDLER reads one, derived from its
AST. A transcript proves the client sent something: the suite posts {} to
routes that ignore it, and that published "this route reads a JSON body" on
DELETE /api/endpoints/agentkey, whose handler never calls get_json.
249 operations still carry no example, because no test exercises them with a
JSON body. The honest fix is a test, not a hand-written example, and the next
harvest picks each one up for free.
The lean-build job proved the delete-a-plugin guarantee with two plugins and
four hand-picked marker components. It now builds a real twelve-plugin set, and
the assertions follow a rule instead of a hunch: one component from EVERY
omitted plugin must be absent, one from EVERY chosen plugin must be present, and
the backend tree is checked both ways. An omitted set that grows without its
check growing is how a leak ships.
CHANGELOG had not been touched across the whole range, so the largest
security-behaviour change in the product - per-PC endpoint credentials, the
shared-key refusal for enrolled PCs, server pinning, agent 1.20.1 and tray 1.7.1
- had no release record. An [Unreleased] section now carries it, written from
the commits rather than from memory.
CLAUDE.md and the agent quick-reference stated that ADR-016 bans storing
credentials in the registry. It does not, and this fleet depends on HKLM for its
collector key and server pins: the next person following that text removes them
and silently breaks reporting and pinning on every un-enrolled PC. Corrected,
along with two other statements that were false on this branch - the database is
MySQL, not PostgreSQL, and column names are lowercase concatenated, not
snake_case, which is the opposite of what the gate enforces.
The wiki drafts are updated against the code as it stands, each old finding
given a verdict with a file and line rather than left as an assertion from a
week ago, and a new page records the ADR conformance result - including that the
ADR set itself is not on this branch, so nothing here enforces it except the
naming and style script.
They are also scrubbed of the things that must not reach a published wiki: the
site name, the internal git server, developer home paths and the assistant
tooling. They now pass the repository's own publishability patterns, which is
what the scrub is for.
docs/openapi.json is served live at /api/docs on the production box and
documented none of the routes added since 2026-09-11. It is generated from
docs/api-inventory.json, which is not tracked on this branch, so the generator
had no input - and tests/test_api_inventory_parity.py skipped itself for the
same reason. No route here had ever been checked against the published contract.
The inventory is now generated from this branch's own url_map rather than
carried over from elsewhere, with each route's auth derived from the decorators
and gates on its view rather than guessed. A route whose gate cannot be
classified is marked for review, not quietly called public. gen_openapi.py can
refresh it from the live app, so this does not rot again, and human-authored
prose survives a refresh.
The parity test now RUNS instead of skipping.
The thing that mattered more than the missing pages: nine operations were
published with "security: []" - the machine-readable claim that no credentials
are needed - while their handlers require one. Two of them serve the bootstrap
installer and the agent package. A generated client, a firewall plan or anything
else reading the spec would have been told those are anonymous. Required auth is
now expressed as required, and a token-or-network gate is expressed as optional
auth with the network half named in the description, which is what it actually
is.
A test re-derives auth from the live app and fails when the committed inventory
disagrees, because the parity check only ever compared route lists: a wrong auth
string passed everything.
test_openapi_spec.py asserted that optional auth was the majority of the
surface, a premise the guest-role and permission model has made false. It now
holds the property that matters - an optional-auth route is expressed as
optional - rather than a count that was true once.
gen_openapi.py also loads .env before building the app, like every other script
that builds one, or a plain python run falls back to the hardcoded localhost URL.
Finding F12 of the architectural review, checked against the code and found
worse than reported: the dynamic component import cannot be rewritten by the
bundler, so the built file ships the unresolved path and a plugin tab answers
404 in every production build, staged or not. It works only under the dev
server. Nothing shipping is affected, because the one plugin that declares a
tab also registers the same page as an ordinary route, and that is the one
sites reach.
- The ADR draft now carries a "Constraints as implemented" section: only the
model catalog hub registers plugin tabs, registration happens when that hub
mounts so a tab is not deep-linkable and does not survive a refresh, and the
build finding above with the bundler's own warning. The two ways out are
stated as the open decision rather than taken.
- The hook's docstring reads as instructions to a plugin author, and its
example no longer names a path that does not exist at build time.
- A route declared for a hub that cannot render it is dropped with an error
naming the plugin, the hub and the hubs that do work, and the same is logged
at startup. Enabling the plugin still succeeds: a tab nobody can reach is no
reason to take a working backend offline.
- Tests hold the two halves together: no bundled plugin may declare a hub the
frontend does not implement, and the list in the server must equal the hubs
that really register routes, so the two cannot drift apart again.
Findings F2, F4, F5, F6, F7, F8, F9, F10, F11 and F13 from
wj-architectural-review.md. F1/F1a (the non-sargable LOWER pattern) and F3
(the warranty sync holding a pooled connection) are being handled separately
and are untouched here.
- A collector check-in that raced another for the same PC lost everything:
the unique index on the application fingerprint raised, nothing caught it,
and the whole check-in rolled back, discarding state, events and hardware,
answering the agent with form copy meant for a person. The insert now runs
in a savepoint, reconciles what the other transaction already wrote, one row
per savepoint so a second collision skips a row rather than the batch, and
the check-in commits with a warning. This is the shape the delete side of
the same function already had.
- Enrollment is deliberately unauthenticated, so it now carries a per-address
rate limit and a cap on pending requests, and the cap is a setting an admin
can see and change. A flood can no longer bury the approval queue or grow
the table without bound.
- The fleet alert poll no longer formats debug strings on every request.
- BusinessUnit is on the contract surface (0.34.0), so the endpoints plugin
stops reaching into a core table with raw SQL.
- A test walks every asset type descriptor and proves the endpoint it names
really refuses a caller without the declared permission.
- The plugin settings routes now need a permission; they were telling anyone
which modules a site runs and where its components live.
- The connection pool is sized explicitly with a short timeout, and the
arithmetic against the server's connection ceiling is written next to the
worker count instead of being left to defaults nobody chose.
- Fleet alerts carry both the correct lowercase keys and the old camelCase
pair. Trays older than 1.7.0 match keys case-sensitively and would have gone
quiet on every PC not yet upgraded; the old pair goes when they are.
- The dev server proxy points at localhost again, with environment variables
for the remote-box workflow, and one developer's address is out of the
server certificate page.
Document new endpoint that exposes plugin-contributed settings routes
for dynamic frontend UI extension.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
wilm has been red for eight consecutive runs across backend, gates and
frontend. Three failures, two unrelated causes, neither introduced by the
endpoint agent work.
docs/ prose (backend + gates):
The docs gates detect a published checkout by the ABSENCE of docs/*.md,
because docs/ becomes the wiki on that side. DEV-BUILD-TEST.md and
PLUGIN-HOOKS.md were committed in unrelated changes, which made CI look
like an internal checkout carrying almost no prose. test_there_are_docs_
to_check then failed on `assert 3 > 20`, test_the_map_itself_is_current
failed on a generated PROJECT-MAP.md that is correctly untracked, and
check-gates-ran.sh failed on the resulting skip - a skipped gate reports
green and checks nothing, so it refuses to accept one.
The existing .gitignore comment predicted this exactly and was defeated by
naming files individually. Untracked both (kept on disk) and replaced the
per-file list with docs/*.md so the next authored doc cannot reintroduce
it. Scoped to the top level: docs/notes/ prose and docs/openapi.json stay
tracked, and the gates only glob docs/*.md.
Frontend specs:
getEntraDeviceUrls() gained lapsUrl and loaded, but two siteSettings specs
still asserted the old two-key shape with toEqual, which is exact-match.
The implementation is correct - loaded is what lets a caller distinguish a
failed fetch from an unconfigured site, since both read as empty strings -
so the specs were brought up to the current shape rather than the return
value narrowed.
pinnedPanels.spec.js sliced 420 characters from getEntraDeviceUrls() and
asserted the finally block appeared inside it. Adding the retry logic
pushed finally past that offset, so a correct implementation reported as
"the finally is missing". Rebounded the slice to the function's own extent,
matching the structural approach already used elsewhere in this file for
the same reason. Verified by mutation: renaming the flag in the source
fails the test, which also caught that the spec reads
plugins/computers/frontend/views/PCDetail.vue and not the .plugins-staged
build copy.
A table panel with no rows rendered its column headers over an empty body. On a live PC Details page that made Network Interfaces and Listening Ports look like cards that had failed to load, when the real answer was that the agent had not reported any yet. List panels already honoured panel.empty; table panels did not.
Also records the simulation-suite requirement uncovered when rolling containers: the pytest suite builds its schema from the models, so an unapplied migration is invisible to it while every query against the affected table 500s in the running image.
Mapped drives existed only as a comma-joined line in the user context card, which is fine for three drives and unreadable for a dozen. They now get their own list card in Monitors: one row per letter, remote path as meta, health as a badge. Reuses the statecolor/levels vocabulary the tracked-services cards already declare rather than new CSS. Only unhealthy drives are coloured. Green on every working drive makes the broken one just another badge in a row of badges, and the card exists to make the exception findable at a glance. Unhealthy rows sort first for the same reason - a user with a dozen mappings should not hunt for the broken one - and the sort is on health alone, so drive-letter order survives within each group and the list does not reshuffle between refreshes. The merge logic is untouched. _mapped_drive_rows now returns structured rows and the user-context summary line is built from them, so the card and the line cannot drift apart about a drive's status. The two silent-failure paths are mutation-tested: forcing every badge green failed 7 tests, and the first sort test passed a mutation that added a secondary key because every broken drive in it said the same word - it now uses statuses that would reorder under a text sort. Also corrects a stale warning claiming the merge is untested; it has been covered since Phase 1.
Volume capacity and physical drive health were two adjacent cards, and the operator reading a full disk is usually the operator reading a dying disk in the same minute. They are now one Storage card with two sections. Visual grouping only - the two tables share no key, storage being per volume and health per physical device, and one drive can back several volumes or none the operator sees. Nothing is reconciled across the sections; this is not the mapped-drive merge where two sources described the same drive and the worse status won. The tabs render type was the only sections-in-one-card mechanism and emitted label/value fields with no table path, so it now takes table sections too. That is the renderer every plugin uses, so both behaviour changes in it are mutation-tested. Two things only surfaced on contact: the empty-section filter had to grow a table branch, because a table section declares columns up front and dropping it when it has no rows takes its tab with it and reads as no disks rather than nothing reported; and full-width span was keyed off the declared render type, but a tabs panel holding a six-column table needs the same span and that is only knowable from fetched data. The diskhealth route stays. It no longer backs a panel, but it is a stable read for callers outside the renderer and removing it for a UI-only change would break them.
Records why rebootreasons is not redundant with pendingreboot, why bootinfo carries two facts that must not be collapsed, and why defender is stored unsurfaced.
Also corrects the notes: pending file-rename was already being collected and OR-ed into pendingreboot, so the gap was that the stronger signal was flattened into the weaker one rather than missing. Records that the columns landed as 0015 because 0014 was taken by disk health earlier in this phase.
Records the agreed plan to combine the two cards into one with two collapsible sections, as VISUAL GROUPING ONLY. The two tables cannot share rows: Storage is per volume, Disk health is per physical device, and one drive can back several volumes or none the operator sees. Stating that explicitly so this is never mistaken for the Phase 1 mapped-drive merge, where two sources described the same drive and the worse status won.
Deferred to its own phase because the tabs render type emits label/value fields only and has no table path, so this means extending the renderer every plugin uses. Bundling that with the Phase 2 telemetry work would turn a UI regression and a telemetry regression into one commit to bisect.
Also scopes the not nested cards density rule to the header block it was written for, having checked it does not block this.
Records why there are two sources and why they are not reconciled, so the next person does not simplify the ATA SMART path back in and silently blind the fleet on NVMe hardware.
Also notes that diskhealth is per physical device and is not the same list as disks, and that a null predictfailure is unknown rather than healthy.
w32time is Stopped on provisioned endpoints, confirmed on more than one box, so w32tm returns 0x80070426 fleet-wide and there is no offset to read. It appears to be stopped by policy after provisioning. Collecting it would return null everywhere and give us an alerting signal that can never fire, which is worse than not having one: a permanently-null health check reads as no problem found.
Implemented first, then removed - collector, both parsers, the process helper, and the contract field. Kept one note in the notes file: parse the offset by label, not by first value ending in s, because w32tm prints Root Delay and Root Dispersion in the same format beforehand and a first-match parser reports network latency as clock skew. That bug was written and caught here.
Windows Update history is pinned rather than cut. The COM update searcher is slow, but the blocking concern is that some endpoints host COM server applications where driving it risks interfering with collections or the hosted application. Needs its own investigation. The Phase 4 card that depended on it is marked blocked so the two do not drift apart.
Windows exposes mapped drives twice and the two views disagree. Captured on one
box at one moment, same two mappings: Get-SmbMapping reported OK for both while
net use reported Disconnected and Reconnecting. Collecting only Get-SmbMapping -
the nicer, more structured one - would have reported a healthy Q: on a machine
whose Q: was down, which is the exact failure this telemetry exists to catch.
So collect both and keep them apart. The probe emits netuse and smbmapping as
separate keys and does not reconcile them; the disagreement is the signal and
normalising it in the agent would discard it before anyone could see it.
The server merges by drive letter and shows the WORSE of the two. A drive that
either source calls broken is broken for the operator, and a drive neither
source gave a status for renders as Unknown rather than blank - an unannotated
row reads as healthy, so a failed check would present as a working drive.
Drives seen by only one source are kept, not dropped: Get-SmbMapping is absent
on some builds and session types, and requiring agreement would silently hide
every mapped drive on those machines.
No active reachability testing. Windows already knows the status, so touching
each share would be wasted work and would reintroduce the hang risk the bounded
probe exists to avoid.
The probe stays unelevated in the operator's session. UAC token separation
gives an elevated context a different mapped-drive table, often empty, so
elevating this later would quietly empty the card.
Adds a commented-out remap_drive seam in the command catalog. Not implemented,
but placed so the next person finds the user-session and elevation constraints
before writing it rather than after.
The merge logic has no test coverage yet; the docstring names the three
behaviours that must not be broken.
Machine data arrives as a Maximo asset export (validated against the WINC
site) that must update existing machines in place, not create canonical
per-table rows, so it does not fit the ShopDB-native/Azure/Thor CSV adapter
path. Bulk edits to live machines also need an escape hatch, so the whole
commit is journaled and can be undone as one batch.
Add a dedicated dry-run analyzer, approval packet, resolver/suggest helpers,
and a reversible writer/undo backed by a new importbatches/importjournalentries
schema (with migration). Expose preview/commit/batches/undo settings routes and
a matching frontend source card and panel, and surface the batch/journal
contract through shopdb.api for plugins.
Bring main's endpoint hardening, operator-report validation, launch-path
policy, and theme-variable CSS into wilm. Channel model reconciled to this
repo's shipping (stable=release tag, edge=main trunk) while keeping wilm's
remote-BuildKit publish step and GHCR pull-based launcher.
Pilots behind the inspecting proxy cannot reliably build the frontend from source; npm's many small fetches get reset mid-install. Build the image once in CI on the clean runners and have pilots pull the finished channel image. One resumable image pull survives the proxy where hundreds of npm requests do not, and a pilot now needs only Docker.
The iex (irm ...) one-liner could never work against a private repo: raw.githubusercontent.com returns 404 without a token, and a git operation's Credential Manager auth does not apply to a plain irm. The prior token-passing workaround (get.ps1, 717387f) asked pilots to paste a PAT, which they will not do and may not have.
Start-ShopDBDev.ps1 is a downloaded FILE, not a web pipe, so only its git clone/fetch need auth - and those go through Git Credential Manager using the login pilots already have from cloning this repo or signing in via Visual Studio / VS Code. No token, no paste. It takes -Channel stable|edge and hands off to dev-bootstrap.ps1. DEV-BUILD-TEST.md is rewritten for the save-and-run flow with a GCM-seeding fallback.
Follows the config-driven-badge decision through to packaging: a single channel-parameterized builder emits two distinct installers (stable from main, edge from dev-be) rather than two drifting scripts. Each package ships the image bundle, the airgap+channel compose templates, a pre-filled .env, and a one-command site installer.
Secrets ship as CHANGE_ME_* sentinels and Install-ShopDB.ps1 refuses to start until every one is rotated, so a distributable can never come up on shipped credentials. docker-compose.airgap.yml now threads SHOPDB_ENV_LABEL through the shared app-env anchor, otherwise a distributed edge stack would run the badge-capable image but never light the badge.
get.ps1 is the web launcher that makes the iex one-liners work: dev-bootstrap.ps1 operates on a local clone, so get.ps1 clones/updates the channel branch and hands off. docs/DEV-BUILD-TEST.md is the short reviewer handout with the two one-liners for Cameron to try.
Operators had no way to flag a problem PC from the tray. This adds a report path: the tray captures a reason, free-text note, and optional screenshot and the agent uploads it on the collector token to a new operatorreports table linked to the asset. Ships the endpoints 0009 migration so the table exists before the code queries it; the running DB had drifted behind the code and returned HTTP 500 on upload.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
The branch stored scopejson, desiredstate and params as sa.JSON, the only JSON
columns in the product. MySQL 5.6 has no JSON type - it arrived in 5.7 - and the
migration failed there with a bare syntax error near the word JSON.
They are TEXT now, through a JSONText type decorator, so the seventeen call sites
still assign and read dicts. The convention elsewhere is already TEXT:
guides.blocksjson, geenforce.manifestjson.
Spec regenerated: 331 paths, 477 operations across 22 surfaces.
