The claim poll holds a connection open server-side. Under gthread that
pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a
worker with 8 of 9 threads parked in timed sleep and UI requests hashed
to that worker waiting 12-24s. Hold was cut to 5s as an interim fix.
gevent removes the constraint rather than working around it: a held
connection is a greenlet and the per-worker ceiling is worker-connections
(2000), not 8 threads, so an idle hold no longer costs a UI request
anything. Measured at 700 simulated agents plus UI, integration and kiosk
traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding
resource, not memory or DB connections.
With that gone the 5s hold is a liability. Hold length sets the fleet's
request rate, not its concurrency, because the agent re-polls immediately
on a 204 - so hold=5 is 5x the requests of hold=25, against a burst
bottleneck that measured as TLS handshake CPU. 25 also restores the
margin against the agent's MinIdlePollDuration=5s floor used to tell a
normal hold-window close apart from a transient failure; at 5 it was
~20ms.
Timeout raised to 60 so gunicorn cannot kill a healthy long-poll
mid-hold, and the access log now carries request duration - without it a
slow request is indistinguishable from a fast one and latency has to be
reconstructed from outside the container. Healthcheck moved off
POST /api/auth/login, which logged a 400 every 30s and exercised the
credential path from a probe.
Corporate Tools tab on PC Details, gated by endpoints.deploy and liveness.
Starts already-registered scheduled tasks; the agent's ScheduledTaskPolicy
allowlist remains the authority on what may actually run.
Dell Command | Update split into two actions (Scan, Apply Updates) because
they have very different consequences. Apply Updates carries a persistent
breakglass warning in the card naming Tanium as the primary update utility,
rendered between the row title and description so it is visible at all times
rather than only in the confirm dialog. Destructive actions keep the typed
RUN confirmation.
Fix scheduledtask_run being stripped by Configuration Policy: the
intersection in ALLOWED_ACTIONS_FOR dropped it on any endpoint whose
effective policy was non-empty, so every Corporate Tools click failed with
"not permitted by policy" regardless of the caller's role. Corporate tools
are role-gated only, via POLICY_EXEMPT_ACTIONS; policy keeps authority over
monitors and service/process primitives.
Fix blank PC Details page: watch(detailtabs) evaluates its source during
setup and read canDeploy before its const declaration, throwing a TDZ
ReferenceError that aborted setup with no API request made.
Mapped-drive refresh moves to the tray. SYSTEM cannot probe user-scoped
drives, so the service requests a re-probe over a named event and waits for
the spool collectedat stamp to advance before collecting. The probe task is
registered as a ShopFloor user principal; the Users-group principal remains
only as a degraded fallback since a group member cannot start it.
Placed between Health and Logged-in User on /endpoints. The main agent
version, not the tray: both ship side by side in the payload as
agentversion and trayversion, so the wrong one is an easy pick.
Frontend only. EndpointState.to_dict() has carried agentversion all
along, so the fleet list API was already sending the field and nothing
rendered it.
Also bumped the empty-state colspan from 9/8 to 10/9. It is hardcoded
rather than derived from the header count, so it does not track a new
column on its own and silently under-spans the "No endpoints found" row
if missed.
Mapped drives reported Unavailable on endpoints where the operator could
see them fine. The data path was intact; the collection was running under
the wrong token. Mapped drives are per logon token, so an elevated probe
enumerates a different drive table than the interactive user.
Collection:
- Pin the user-context spool path so the probe and service agree on one
location, and persist/overwrite it instead of deleting on read.
- Refuse to collect while elevated (exit code 2) rather than emitting a
confident wrong answer, and stamp elevated/username on the spool.
- Reject spools stamped elevated or from an unexpected account on read,
so an admin run cannot masquerade as valid.
- Log netuse/smbmapping/userlistener counts so each run leaves evidence
of what it actually saw.
Install and launch context:
- Move collection to a 'ShopDB user-context probe' scheduled task under
S-1-5-32-545 at RunLevel Limited, the only mechanism that reliably runs
in the operator's own session. Verify registration and record it in the
install ledger.
- Stop starting the tray as a child of the elevated installer; explorer.exe
de-elevates without changing ACCOUNT, so an admin install produced an
admin tray. The Run key only fires at logon, which left the tray dead on
a machine already signed in, so add a transient 'ShopDB tray first launch'
task that starts it once in the existing session and unregisters itself.
- Hide the console window during --collect-usercontext. The agent is a
console exe; the tray used to mask this with CreateNoWindow, but the
scheduled task invokes it directly and flashed a window every cycle.
Installer edits are in _install_script() in plugins/endpoints/api/siteagent.py,
which generates the install.ps1 that actually ships. The standalone
agent/installer/install-shopdbagent.ps1 was never shipped once the package
moved to the UI builder and is deleted here to stop the divergence that
caused an earlier fix to land in a file no endpoint ever ran.
Versions: agent 1.10.1, tray 1.3.0. These share the pinned spool path and
must be deployed together.
Pre-existing test failures unrelated to this work (8, identical on a clean
tree): docs publishable/versions, plugin contract symlink scan, import
guard tamper check, security headers upload sandbox, and three endpoint
timestamp tests.
Details tab card layout
-----------------------
The Details tab sat on .content-grid, a two-column CSS grid. Grid packs
items into implicit rows, so every row is as tall as its tallest card and
align-items:start leaves the shorter one with dead space beneath it:
Warranty is short, Endpoint Status is tall, and Policy had to wait for the
taller of the two rather than sliding up. That read as a staggered layout.
Switched that one tab to multicol via a .details-flow modifier, which has no
rows at all - each card starts where the previous one ended, so the holes
close while cards still size to their content.
Multicol is deliberately NOT applied to .content-grid globally. The note by
that rule in style.css records why it was abandoned there: a column-span:all
child (.pap-span, used by table panels) overlaps the shorter column when the
two are unequal. That cannot happen on this tab - every panel in the details
group is a keyvalue or list render, and spansFullWidth() only returns true
for table (or a tabs panel holding one), so nothing in the subtree spans.
Monitors and Service/Process DO carry table panels and stay on grid.
The modifier is scoped to PCDetail, so the nine other detail views on
.content-grid are untouched.
Chosen over JS masonry on purpose. Masonry packs perfectly but has to
measure every card and re-measure on resize, on panel collapse and on every
async panel load - and these panels fetch after paint, so it would mean
continuous forced reflow on a page that already polls. This is pure CSS:
no measurement, no observer, no runtime cost. The tradeoff accepted is that
column bottoms can end at slightly different heights.
Cards carry break-inside: avoid so none fragments across the column break,
and the 25px spacing returns as a margin because multicol has no gap between
stacked items the way grid does.
IP Configuration card
---------------------
Removed: the data has no purpose on the page anymore. Taken out at the root
rather than hidden - the panel declaration, the /asset/<id>/ipconfig route
and the _IPCONFIG_COLUMNS constant all go, since a route with no consumer is
dead surface that still answers requests.
test_the_two_cards_do_not_duplicate_columns_beyond_the_address existed only
to police the history/config column split; with one card gone it guards
nothing, so it and its import go with it. The panel-count assertion in
test_endpoints_asset_panels_are_computer_scoped moves 15 -> 14, which is the
test doing its job and catching the removal.
IP History is unchanged and still carries the address list.
Not addressed here: three tzset() tests in test_endpoints_plugin.py fail on
Windows because time.tzset() is Unix-only. Verified pre-existing against
unmodified code; they pass in the Linux container.
Mapped drives stopped reporting because the user-context path failed silently
in three separate places, and none of them left a trace to read back.
- Probe: launch via -EncodedCommand instead of staging a temp .ps1 and running
powershell.exe -File. A missing or quarantined script file fails BEFORE any
statement runs, so the script's own try/catch never sees it and the only
symptom is exit -196608.
- Spool: pin to the ShopFloor profile. Resolving it through WTSQueryUserToken
is privileged, endpoint protection blocks it, and on failure the tray
(writer) and service (reader) silently disagree about the path.
- Spool: consume after a successful parse, so a stopped tray cannot keep
re-reporting stale mapped drives as current. Unparseable files are left in
place for the next cycle.
- Logging: user-session components cannot write to Program Files or to the
SYSTEM-owned ProgramData log directory, and the file sink degrades to a
silent no-op when it cannot create its directory. Tray and probe now log to
C:\Users\ShopFloor\ShopDB\Logs, installs transcript to ProgramData, and the
sink records a SinkProblem instead of swallowing UnauthorizedAccessException.
- Installer: provision the operator spool/log directories with ShopFloor:Modify
before the tray first runs, and stop the transcript via trap so a throw
cannot orphan exactly the failed install worth reading.
- Upload cap 64 -> 128 MB: the self-contained tray publishes at ~69 MB and was
rejected on SIZE, but the size check returns before the PE is parsed, so the
operator only ever saw a version-detection failure.
AGENT-CONTRACT.md gains a normative section defining the three security
contexts, their log destinations, the fixed spool path and consume-on-read.
Agent 1.8.0 -> 1.8.1, tray 1.1.0 -> 1.1.1 so the deployed build is
identifiable from a log line.
asset_panels() rewrites any group missing from ASSET_PANEL_GROUPS to 'details' before the client sees it, and Phase 3 never added 'serviceprocess' there. Both panels were relabelled on the wire, so the tab could not appear no matter how consistent the frontend was. That is why the tab was still missing after the schema fix - a second, unrelated cause wearing the same symptom.
Operator Reports moves out of System Monitors into a tab of its own: that tab is machine-collected health, this is a person reporting something, and the thumbnail grid dwarfed the fact cards it sat between. Full width, since it is alone in the tab.
panelGroups.spec.js claimed to cover four files and checked two; it now reads the server tuple as well. The plugin test's hand-copied set was a fifth copy and now derives from the server tuple instead.
Mapped drives existed only as a comma-joined line in the user context card, which is fine for three drives and unreadable for a dozen. They now get their own list card in Monitors: one row per letter, remote path as meta, health as a badge. Reuses the statecolor/levels vocabulary the tracked-services cards already declare rather than new CSS. Only unhealthy drives are coloured. Green on every working drive makes the broken one just another badge in a row of badges, and the card exists to make the exception findable at a glance. Unhealthy rows sort first for the same reason - a user with a dozen mappings should not hunt for the broken one - and the sort is on health alone, so drive-letter order survives within each group and the list does not reshuffle between refreshes. The merge logic is untouched. _mapped_drive_rows now returns structured rows and the user-context summary line is built from them, so the card and the line cannot drift apart about a drive's status. The two silent-failure paths are mutation-tested: forcing every badge green failed 7 tests, and the first sort test passed a mutation that added a secondary key because every broken drive in it said the same word - it now uses statuses that would reorder under a text sort. Also corrects a stale warning claiming the merge is untested; it has been covered since Phase 1.
Volume capacity and physical drive health were two adjacent cards, and the operator reading a full disk is usually the operator reading a dying disk in the same minute. They are now one Storage card with two sections. Visual grouping only - the two tables share no key, storage being per volume and health per physical device, and one drive can back several volumes or none the operator sees. Nothing is reconciled across the sections; this is not the mapped-drive merge where two sources described the same drive and the worse status won. The tabs render type was the only sections-in-one-card mechanism and emitted label/value fields with no table path, so it now takes table sections too. That is the renderer every plugin uses, so both behaviour changes in it are mutation-tested. Two things only surfaced on contact: the empty-section filter had to grow a table branch, because a table section declares columns up front and dropping it when it has no rows takes its tab with it and reads as no disks rather than nothing reported; and full-width span was keyed off the declared render type, but a tabs panel holding a six-column table needs the same span and that is only knowable from fetched data. The diskhealth route stays. It no longer backs a panel, but it is a stable read for callers outside the renderer and removing it for a UI-only change would break them.
Tracked Services and Tracked Processes were sharing a tab with system health cards, so what is running on the box and how the box itself is doing read as one list. They now sit in a Service/Process Monitors tab and the old tab is System Monitors. The group vocabulary is duplicated across the renderer, the tab map and the panel declarations with nothing checking they agree, and a group named in one but not the others is dropped in silence - that has eaten cards twice. Tests now pin the vocabulary on both sides. Boot type and NIC link state were collected in Phase 2 but shown nowhere. Boot type only renders when fast startup applies, because uptime measured from a resume understates how long the kernel has been up and that is the case worth naming. Link state leaves a disconnected NIC unknown rather than half duplex: live hardware returns empty speed and duplex when unplugged, and half duplex is the cable fault signature this column exists to catch.
Records why rebootreasons is not redundant with pendingreboot, why bootinfo carries two facts that must not be collapsed, and why defender is stored unsurfaced.
Also corrects the notes: pending file-rename was already being collected and OR-ed into pendingreboot, so the gap was that the stronger signal was flattened into the weaker one rather than missing. Records that the columns landed as 0015 because 0014 was taken by disk health earlier in this phase.
Adds rebootreasonsjson, bootinfojson and defenderjson with their migration in the same change, so the model never references a column the schema has not applied.
rebootreasons is not redundant with the existing pendingreboot column. That column is a single bool OR-ed from three registry signals and cannot say which fired, and the difference matters: a staged file rename means the box is mid-servicing, while a lone servicing key can sit set for weeks on a machine nobody rebooted.
Null means never reported and is deliberately distinct from an empty list, which means checked and nothing pending. Backfilling [] would assert every pre-existing endpoint was confirmed clean.
Records why there are two sources and why they are not reconciled, so the next person does not simplify the ATA SMART path back in and silently blind the fleet on NVMe hardware.
Also notes that diskhealth is per physical device and is not the same list as disks, and that a null predictfailure is unknown rather than healthy.
Its own card rather than folded into Storage: Storage is per volume and this is per physical device, and one drive can back several volumes or none the operator sees, so the two do not share a key.
Rows sort failure-first, then unknown, then OK. A failing drive must not sit below the fold on a multi-disk box, and a drive that stopped answering is a weaker claim of health than one that said OK. An unrecognised verdict sorts as unknown rather than last, so a future verdict string cannot bury itself under the healthy drives.
Both agent sources are shown rather than merged, matching the mapped-drive card: when two sources disagree about one drive, that disagreement is the thing worth looking at.
Adds diskhealthjson with its migration in the same change, so the model never references a column the schema has not applied.
Kept separate from disksjson rather than merged into it. That column is per volume and answers how full a drive is; this one is per physical device and answers whether the hardware is dying. They come from different WMI sources with different failure modes, so merging them would make a capacity collection failure indistinguishable from a health collection failure.
Nullable with no backfill: a drive whose health has never been reported is not a healthy drive, and defaulting to an empty list would read as no disks have problems on every endpoint predating the column.
Two statuses sharing a rank makes min() break the tie by argument order, so the same drive reads healthy or broken depending on which source enumerated first. Deliberate synonyms are safe because either answer is the same answer; a new status landing on an existing rank is not, and the rank table is where someone adding one will be looking.
Names the test that catches it, since the single-direction tests stay green through this and would otherwise suggest the change was safe.
Windows exposes mapped drives twice and the two views disagree. Captured on one
box at one moment, same two mappings: Get-SmbMapping reported OK for both while
net use reported Disconnected and Reconnecting. Collecting only Get-SmbMapping -
the nicer, more structured one - would have reported a healthy Q: on a machine
whose Q: was down, which is the exact failure this telemetry exists to catch.
So collect both and keep them apart. The probe emits netuse and smbmapping as
separate keys and does not reconcile them; the disagreement is the signal and
normalising it in the agent would discard it before anyone could see it.
The server merges by drive letter and shows the WORSE of the two. A drive that
either source calls broken is broken for the operator, and a drive neither
source gave a status for renders as Unknown rather than blank - an unannotated
row reads as healthy, so a failed check would present as a working drive.
Drives seen by only one source are kept, not dropped: Get-SmbMapping is absent
on some builds and session types, and requiring agreement would silently hide
every mapped drive on those machines.
No active reachability testing. Windows already knows the status, so touching
each share would be wasted work and would reintroduce the hang risk the bounded
probe exists to avoid.
The probe stays unelevated in the operator's session. UAC token separation
gives an elevated context a different mapped-drive table, often empty, so
elevating this later would quietly empty the card.
Adds a commented-out remap_drive seam in the command catalog. Not implemented,
but placed so the next person finds the user-session and elevation constraints
before writing it rather than after.
The merge logic has no test coverage yet; the docstring names the three
behaviours that must not be broken.
listAll goes through fetchAllPages, which pages past the server's 100-row cap
and returns the rows themselves. Three asset forms read .data.data off that
array. An array has no .data, so the second access threw and the surrounding
Promise.all rejected, taking the whole load into the catch.
On the PC edit page the throw landed before the record fetch, so no request was
made for the computer at all. The form rendered with its reference data and no
record, and the banner said only "Failed to load data" - which reads like a
backend or schema fault and was twice diagnosed as one. The server logs settled
it: every reference endpoint answered 200 and the record was never requested.
MachineForm already carried a comment describing this exact trap, added when
pcsRes hit it. Two adjacent lines had the same defect and were missed, so the
comment now sits above all of them.
Report the failure honestly while here. apiError only reads err.response, so a
client-side TypeError produced nothing but the fallback string no matter how
many times the page was reloaded. Fall back to err.message when there is no
response, which is what identified this.
The hardware, OS and storage inventory had columns and data but nowhere to
be read: the only per-PC card was the status keyvalue, so a machine's
processor, firmware, disks and listening ports were reachable only by
querying the database.
Adds four Monitors cards - Hardware, Operating system, Storage, Listening
ports - split by the question each answers rather than by which JSON blob
they came from. OS and BIOS share one card deliberately; apart they are
three short rows each. A card yields no fields when its columns are null,
so an endpoint on an older agent hides them rather than showing a grid of
blanks.
User context is declared last on the page. The probe feeding it is not yet
producing output, so it reads empty for now, and being last means an empty
card sits below the populated ones instead of pushing them down.
The Activity feed showed a title and an event id, but the title is the
catalog name for that id - the same text on every occurrence - so two
different faults read identically. Rows now carry the provider name, and
the stored message body expands in place when the severity badge is
clicked. Commands expand to their result message, which is the reason a
failed one failed.
The badge doubles as the expander rather than gaining a control beside it,
and it keeps its severity label when open: swapping that for "Hide" would
cost the at-a-glance severity scan down the feed. Being a button rather
than a span, it is also now keyboard reachable.
map.body is additive in the shared renderer - panels that do not declare
it render exactly as before.
Curated events were re-read from the newest 25 entries per channel on every
collect, and the server appended whatever arrived. One real event was
therefore stored again on every cadence tick for as long as it stayed in
that window, so event counts measured how long a box had been up rather
than what had happened to it.
The agent now keeps a per-channel watermark of the last Windows event
record id it reported, persisted under ProgramData so a service restart
does not replay history, and sends eventrecordid with each event. The
server dedupes on (assetid, eventchannel, eventrecordid) behind a unique
index, so a replay from an older agent cannot double-store either.
Six inventory keys the agent had been sending all along - bios, disks,
domain, memory, osinfo, processor - were parsed, logged as unpersisted,
and dropped. They now have columns and are written on check-in, along
with user context, pending reboot, and the cpupercent/memorypercent/
idleseconds fields that were declared in the contract but always null.
Session-scoped data cannot be read from the agent's LocalSystem session 0
at all: mapped drives and user-owned listeners do not exist there. A probe
runs in the interactive user's session and hands results back through a
spool file. It is not yet producing output on a live endpoint; the column
and panel are in place and read empty until that is fixed.
Also bounds the user-context script wait. It was WaitForSingleObject with
INFINITE, so a hung script pinned the command worker permanently.
Uploading an agent build whose product version carried semver build metadata
(1.5.0+abcdef) was rejected, because the typed version string was validated as
a dotted-numeric value before the file was ever read. That string is only a
hint from the operator, and the build metadata it carries is not a legal
directory name for the version store.
The authority is the dotted-numeric FILE version stamped in the binary PE
resource, so validation now happens after the upload is parsed: when the typed
value is missing or not dotted-numeric, fall back to the detected version, and
reject only when neither yields a usable one.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
The board was one card per source in a fixed grid: every source got the same
small box whatever it had to say (warranties showed 5 of 50), and a box per
source flattened priority, while the largest type on the page belonged to counts
that need nobody.
Now a severity rollup, one worst-first queue of everything needing a person, and
a row of context beneath. The queue is full width, capped per source so a noisy
check cannot bury a rare one, and names what did not fit with a link to it. Chips
filter the queue by check as well as by severity; both narrow what is already
loaded, so the counts and rows cannot disagree.
Dropped: an 'All assets' tile shown beside its own four addends, a printer strip
carrying the same count a third time, three cards whose job was to display 0, and
a Recent Devices table listing five machines all 'In Use'.
Contract 0.26.0 adds optional 'shortname' on dashboard cards - a title is written
as a heading and ran wider than the hostname it qualified as a row badge.
Also: the supplies card reported 'Low Supplies 0' when Zabbix was unreachable,
which reads as reassurance rather than the truth that nothing could see the
cartridges. It now distinguishes unreachable (says so) from unconfigured (stays
quiet - the site declined the feature) from a real zero.
Full suite 2182 passed, frontend 314 passed.
The machine page opened with a dual-bay banner directly above a hero band saying
'Controlled by'. Both answer 'what am I looking at', so the sibling is now a chip
in the band. The band takes a generic extraitems prop rather than learning what
dualpath is - the pairing is resolved by the machines plugin, not from the
relationships endpoint.
MeasuringToolDetail filtered for a machine counterpart, but the import creates a
controls edge PC -> tool, so a gage's counterpart is a computer. It rendered
nothing on every metrology station that has a PC.
Verified on screen. Full suite 2178 passed, frontend 280 passed.
The machine page now shows the PC that controls it. The endpoint was correct all
along and returned both directions; AssetTriageBand discarded every peer whose
type was not 'machine' and was mounted only on the PC page. The counterpart type,
label and relationship types are now props.
Fixed while adopting it more widely: every band chip linked to /assets/<id>, and
there is no such route. The resolver that AssetRelationships.vue kept privately
is now frontend/src/utils/assetRoute.js, used by both - and its type lookup is
case-insensitive, because seed data capitalises the type names while this site's
database does not.
The legacy import is now safe to resume. The 409-resolve asked for
per_page=10000 against a server that clamps to 100, so it silently searched the
first 100 rows and dropped links past that on every re-run. The crosswalk is now
bound to the database it describes, so one carried across a rebuild is refused
rather than attaching rows to the wrong assets. Source database names are
resolved from the environment, so the loader can be aimed without editing it.
Full suite 2178 passed, frontend 280 passed.
