Both agent loops made their first request the instant the service
started, and the service is installed start=auto - so a patch reboot put
the whole fleet on the wire together, twice per machine (a telemetry
report and a command poll). The existing jitterseconds does not help: it
spreads the gap between runs, not first contact.
Measured on a 700-endpoint burst: simultaneous arrival drove TLS
handshakes to 1482ms worst case and pinned CPU at 305% while the
database sat idle at 3 active queries. Spreading the same arrivals over
30s cut worst-case handshake to 63.7ms with no server-side change. The
server was never the constraint; simultaneity was. Raising the
SQLAlchemy pool was tried and rejected - it moved the queue from
SQLAlchemy into MySQL without adding throughput.
StartupJitter is shared by both workers so the clamp and the
randomisation cannot drift apart between them. Randomised in
milliseconds rather than whole seconds because a 30s window at second
granularity re-clumps a large fleet into 30 buckets of simultaneous
arrivals - the same pile-up in miniature. Clamped to 300s and warned
about when out of range: the command channel is how an operator would
correct a bad value, so an unbounded delay would remove the means of
recovery.
The cost is up to startupjitterseconds of extra latency before a
rebooted endpoint picks up a queued command, on top of a reboot the
machine was already unreachable for. The once-only console switch
bypasses the delay entirely, so smoke tests stay instant.
Agent 1.12.0. Minor because the appsettings schema gains a key; packages
without it fall back to the AgentConfig default. The shipped installer
template in siteagent.py carries the new key, so the containers need
rolling before newly built packages pick it up.
The claim poll holds a connection open server-side. Under gthread that
pinned one of a worker's 8 OS threads, so 14 agents at hold=25 left a
worker with 8 of 9 threads parked in timed sleep and UI requests hashed
to that worker waiting 12-24s. Hold was cut to 5s as an interim fix.
gevent removes the constraint rather than working around it: a held
connection is a greenlet and the per-worker ceiling is worker-connections
(2000), not 8 threads, so an idle hold no longer costs a UI request
anything. Measured at 700 simulated agents plus UI, integration and kiosk
traffic: zero errors, UI median 7.5ms, peak RSS 479MB. CPU is the binding
resource, not memory or DB connections.
With that gone the 5s hold is a liability. Hold length sets the fleet's
request rate, not its concurrency, because the agent re-polls immediately
on a 204 - so hold=5 is 5x the requests of hold=25, against a burst
bottleneck that measured as TLS handshake CPU. 25 also restores the
margin against the agent's MinIdlePollDuration=5s floor used to tell a
normal hold-window close apart from a transient failure; at 5 it was
~20ms.
Timeout raised to 60 so gunicorn cannot kill a healthy long-poll
mid-hold, and the access log now carries request duration - without it a
slow request is indistinguishable from a fast one and latency has to be
reconstructed from outside the container. Healthcheck moved off
POST /api/auth/login, which logged a 400 every 30s and exercised the
credential path from a probe.
Corporate Tools tab on PC Details, gated by endpoints.deploy and liveness.
Starts already-registered scheduled tasks; the agent's ScheduledTaskPolicy
allowlist remains the authority on what may actually run.
Dell Command | Update split into two actions (Scan, Apply Updates) because
they have very different consequences. Apply Updates carries a persistent
breakglass warning in the card naming Tanium as the primary update utility,
rendered between the row title and description so it is visible at all times
rather than only in the confirm dialog. Destructive actions keep the typed
RUN confirmation.
Fix scheduledtask_run being stripped by Configuration Policy: the
intersection in ALLOWED_ACTIONS_FOR dropped it on any endpoint whose
effective policy was non-empty, so every Corporate Tools click failed with
"not permitted by policy" regardless of the caller's role. Corporate tools
are role-gated only, via POLICY_EXEMPT_ACTIONS; policy keeps authority over
monitors and service/process primitives.
Fix blank PC Details page: watch(detailtabs) evaluates its source during
setup and read canDeploy before its const declaration, throwing a TDZ
ReferenceError that aborted setup with no API request made.
Mapped-drive refresh moves to the tray. SYSTEM cannot probe user-scoped
drives, so the service requests a re-probe over a named event and waits for
the spool collectedat stamp to advance before collecting. The probe task is
registered as a ShopFloor user principal; the Users-group principal remains
only as a degraded fallback since a group member cannot start it.
Mapped drives reported Unavailable on endpoints where the operator could
see them fine. The data path was intact; the collection was running under
the wrong token. Mapped drives are per logon token, so an elevated probe
enumerates a different drive table than the interactive user.
Collection:
- Pin the user-context spool path so the probe and service agree on one
location, and persist/overwrite it instead of deleting on read.
- Refuse to collect while elevated (exit code 2) rather than emitting a
confident wrong answer, and stamp elevated/username on the spool.
- Reject spools stamped elevated or from an unexpected account on read,
so an admin run cannot masquerade as valid.
- Log netuse/smbmapping/userlistener counts so each run leaves evidence
of what it actually saw.
Install and launch context:
- Move collection to a 'ShopDB user-context probe' scheduled task under
S-1-5-32-545 at RunLevel Limited, the only mechanism that reliably runs
in the operator's own session. Verify registration and record it in the
install ledger.
- Stop starting the tray as a child of the elevated installer; explorer.exe
de-elevates without changing ACCOUNT, so an admin install produced an
admin tray. The Run key only fires at logon, which left the tray dead on
a machine already signed in, so add a transient 'ShopDB tray first launch'
task that starts it once in the existing session and unregisters itself.
- Hide the console window during --collect-usercontext. The agent is a
console exe; the tray used to mask this with CreateNoWindow, but the
scheduled task invokes it directly and flashed a window every cycle.
Installer edits are in _install_script() in plugins/endpoints/api/siteagent.py,
which generates the install.ps1 that actually ships. The standalone
agent/installer/install-shopdbagent.ps1 was never shipped once the package
moved to the UI builder and is deleted here to stop the divergence that
caused an earlier fix to land in a file no endpoint ever ran.
Versions: agent 1.10.1, tray 1.3.0. These share the pinned spool path and
must be deployed together.
Pre-existing test failures unrelated to this work (8, identical on a clean
tree): docs publishable/versions, plugin contract symlink scan, import
guard tamper check, security headers upload sandbox, and three endpoint
timestamp tests.
Details tab card layout
-----------------------
The Details tab sat on .content-grid, a two-column CSS grid. Grid packs
items into implicit rows, so every row is as tall as its tallest card and
align-items:start leaves the shorter one with dead space beneath it:
Warranty is short, Endpoint Status is tall, and Policy had to wait for the
taller of the two rather than sliding up. That read as a staggered layout.
Switched that one tab to multicol via a .details-flow modifier, which has no
rows at all - each card starts where the previous one ended, so the holes
close while cards still size to their content.
Multicol is deliberately NOT applied to .content-grid globally. The note by
that rule in style.css records why it was abandoned there: a column-span:all
child (.pap-span, used by table panels) overlaps the shorter column when the
two are unequal. That cannot happen on this tab - every panel in the details
group is a keyvalue or list render, and spansFullWidth() only returns true
for table (or a tabs panel holding one), so nothing in the subtree spans.
Monitors and Service/Process DO carry table panels and stay on grid.
The modifier is scoped to PCDetail, so the nine other detail views on
.content-grid are untouched.
Chosen over JS masonry on purpose. Masonry packs perfectly but has to
measure every card and re-measure on resize, on panel collapse and on every
async panel load - and these panels fetch after paint, so it would mean
continuous forced reflow on a page that already polls. This is pure CSS:
no measurement, no observer, no runtime cost. The tradeoff accepted is that
column bottoms can end at slightly different heights.
Cards carry break-inside: avoid so none fragments across the column break,
and the 25px spacing returns as a margin because multicol has no gap between
stacked items the way grid does.
IP Configuration card
---------------------
Removed: the data has no purpose on the page anymore. Taken out at the root
rather than hidden - the panel declaration, the /asset/<id>/ipconfig route
and the _IPCONFIG_COLUMNS constant all go, since a route with no consumer is
dead surface that still answers requests.
test_the_two_cards_do_not_duplicate_columns_beyond_the_address existed only
to police the history/config column split; with one card gone it guards
nothing, so it and its import go with it. The panel-count assertion in
test_endpoints_asset_panels_are_computer_scoped moves 15 -> 14, which is the
test doing its job and catching the removal.
IP History is unchanged and still carries the address list.
Not addressed here: three tzset() tests in test_endpoints_plugin.py fail on
Windows because time.tzset() is Unix-only. Verified pre-existing against
unmodified code; they pass in the Linux container.
Mapped drives stopped reporting because the user-context path failed silently
in three separate places, and none of them left a trace to read back.
- Probe: launch via -EncodedCommand instead of staging a temp .ps1 and running
powershell.exe -File. A missing or quarantined script file fails BEFORE any
statement runs, so the script's own try/catch never sees it and the only
symptom is exit -196608.
- Spool: pin to the ShopFloor profile. Resolving it through WTSQueryUserToken
is privileged, endpoint protection blocks it, and on failure the tray
(writer) and service (reader) silently disagree about the path.
- Spool: consume after a successful parse, so a stopped tray cannot keep
re-reporting stale mapped drives as current. Unparseable files are left in
place for the next cycle.
- Logging: user-session components cannot write to Program Files or to the
SYSTEM-owned ProgramData log directory, and the file sink degrades to a
silent no-op when it cannot create its directory. Tray and probe now log to
C:\Users\ShopFloor\ShopDB\Logs, installs transcript to ProgramData, and the
sink records a SinkProblem instead of swallowing UnauthorizedAccessException.
- Installer: provision the operator spool/log directories with ShopFloor:Modify
before the tray first runs, and stop the transcript via trap so a throw
cannot orphan exactly the failed install worth reading.
- Upload cap 64 -> 128 MB: the self-contained tray publishes at ~69 MB and was
rejected on SIZE, but the size check returns before the PE is parsed, so the
operator only ever saw a version-detection failure.
AGENT-CONTRACT.md gains a normative section defining the three security
contexts, their log destinations, the fixed spool path and consume-on-read.
Agent 1.8.0 -> 1.8.1, tray 1.1.0 -> 1.1.1 so the deployed build is
identifiable from a log line.
Mapped drives existed only as a comma-joined line in the user context card, which is fine for three drives and unreadable for a dozen. They now get their own list card in Monitors: one row per letter, remote path as meta, health as a badge. Reuses the statecolor/levels vocabulary the tracked-services cards already declare rather than new CSS. Only unhealthy drives are coloured. Green on every working drive makes the broken one just another badge in a row of badges, and the card exists to make the exception findable at a glance. Unhealthy rows sort first for the same reason - a user with a dozen mappings should not hunt for the broken one - and the sort is on health alone, so drive-letter order survives within each group and the list does not reshuffle between refreshes. The merge logic is untouched. _mapped_drive_rows now returns structured rows and the user-context summary line is built from them, so the card and the line cannot drift apart about a drive's status. The two silent-failure paths are mutation-tested: forcing every badge green failed 7 tests, and the first sort test passed a mutation that added a secondary key because every broken drive in it said the same word - it now uses statuses that would reorder under a text sort. Also corrects a stale warning claiming the merge is untested; it has been covered since Phase 1.
Volume capacity and physical drive health were two adjacent cards, and the operator reading a full disk is usually the operator reading a dying disk in the same minute. They are now one Storage card with two sections. Visual grouping only - the two tables share no key, storage being per volume and health per physical device, and one drive can back several volumes or none the operator sees. Nothing is reconciled across the sections; this is not the mapped-drive merge where two sources described the same drive and the worse status won. The tabs render type was the only sections-in-one-card mechanism and emitted label/value fields with no table path, so it now takes table sections too. That is the renderer every plugin uses, so both behaviour changes in it are mutation-tested. Two things only surfaced on contact: the empty-section filter had to grow a table branch, because a table section declares columns up front and dropping it when it has no rows takes its tab with it and reads as no disks rather than nothing reported; and full-width span was keyed off the declared render type, but a tabs panel holding a six-column table needs the same span and that is only knowable from fetched data. The diskhealth route stays. It no longer backs a panel, but it is a stable read for callers outside the renderer and removing it for a UI-only change would break them.
Tracked Services and Tracked Processes were sharing a tab with system health cards, so what is running on the box and how the box itself is doing read as one list. They now sit in a Service/Process Monitors tab and the old tab is System Monitors. The group vocabulary is duplicated across the renderer, the tab map and the panel declarations with nothing checking they agree, and a group named in one but not the others is dropped in silence - that has eaten cards twice. Tests now pin the vocabulary on both sides. Boot type and NIC link state were collected in Phase 2 but shown nowhere. Boot type only renders when fast startup applies, because uptime measured from a resume understates how long the kernel has been up and that is the case worth naming. Link state leaves a disconnected NIC unknown rather than half duplex: live hardware returns empty speed and duplex when unplugged, and half duplex is the cable fault signature this column exists to catch.
Its own card rather than folded into Storage: Storage is per volume and this is per physical device, and one drive can back several volumes or none the operator sees, so the two do not share a key.
Rows sort failure-first, then unknown, then OK. A failing drive must not sit below the fold on a multi-disk box, and a drive that stopped answering is a weaker claim of health than one that said OK. An unrecognised verdict sorts as unknown rather than last, so a future verdict string cannot bury itself under the healthy drives.
Both agent sources are shown rather than merged, matching the mapped-drive card: when two sources disagree about one drive, that disagreement is the thing worth looking at.
Two statuses sharing a rank makes min() break the tie by argument order, so the same drive reads healthy or broken depending on which source enumerated first. Deliberate synonyms are safe because either answer is the same answer; a new status landing on an existing rank is not, and the rank table is where someone adding one will be looking.
Names the test that catches it, since the single-direction tests stay green through this and would otherwise suggest the change was safe.
Windows exposes mapped drives twice and the two views disagree. Captured on one
box at one moment, same two mappings: Get-SmbMapping reported OK for both while
net use reported Disconnected and Reconnecting. Collecting only Get-SmbMapping -
the nicer, more structured one - would have reported a healthy Q: on a machine
whose Q: was down, which is the exact failure this telemetry exists to catch.
So collect both and keep them apart. The probe emits netuse and smbmapping as
separate keys and does not reconcile them; the disagreement is the signal and
normalising it in the agent would discard it before anyone could see it.
The server merges by drive letter and shows the WORSE of the two. A drive that
either source calls broken is broken for the operator, and a drive neither
source gave a status for renders as Unknown rather than blank - an unannotated
row reads as healthy, so a failed check would present as a working drive.
Drives seen by only one source are kept, not dropped: Get-SmbMapping is absent
on some builds and session types, and requiring agreement would silently hide
every mapped drive on those machines.
No active reachability testing. Windows already knows the status, so touching
each share would be wasted work and would reintroduce the hang risk the bounded
probe exists to avoid.
The probe stays unelevated in the operator's session. UAC token separation
gives an elevated context a different mapped-drive table, often empty, so
elevating this later would quietly empty the card.
Adds a commented-out remap_drive seam in the command catalog. Not implemented,
but placed so the next person finds the user-session and elevation constraints
before writing it rather than after.
The merge logic has no test coverage yet; the docstring names the three
behaviours that must not be broken.
The hardware, OS and storage inventory had columns and data but nowhere to
be read: the only per-PC card was the status keyvalue, so a machine's
processor, firmware, disks and listening ports were reachable only by
querying the database.
Adds four Monitors cards - Hardware, Operating system, Storage, Listening
ports - split by the question each answers rather than by which JSON blob
they came from. OS and BIOS share one card deliberately; apart they are
three short rows each. A card yields no fields when its columns are null,
so an endpoint on an older agent hides them rather than showing a grid of
blanks.
User context is declared last on the page. The probe feeding it is not yet
producing output, so it reads empty for now, and being last means an empty
card sits below the populated ones instead of pushing them down.
The Activity feed showed a title and an event id, but the title is the
catalog name for that id - the same text on every occurrence - so two
different faults read identically. Rows now carry the provider name, and
the stored message body expands in place when the severity badge is
clicked. Commands expand to their result message, which is the reason a
failed one failed.
The badge doubles as the expander rather than gaining a control beside it,
and it keeps its severity label when open: swapping that for "Hide" would
cost the at-a-glance severity scan down the feed. Being a button rather
than a span, it is also now keyboard reachable.
map.body is additive in the shared renderer - panels that do not declare
it render exactly as before.
Uploading an agent build whose product version carried semver build metadata
(1.5.0+abcdef) was rejected, because the typed version string was validated as
a dotted-numeric value before the file was ever read. That string is only a
hint from the operator, and the build metadata it carries is not a legal
directory name for the version store.
The authority is the dotted-numeric FILE version stamped in the binary PE
resource, so validation now happens after the upload is parsed: when the typed
value is missing or not dotted-numeric, fall back to the detected version, and
reject only when neither yields a usable one.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
A command reaching 'succeeded' only proves the agent ran the action, not that it posted a new telemetry snapshot. Reloading panels on command success alone re-rendered stale service/process rows, so a stopped process could still show running right after Refresh Status.
Add a lightweight telemetrymeta probe exposing lastreportedat and gate both the header Refresh Status action and inline row Start/Stop on that stamp advancing past a pre-action baseline. Polling is bounded so an offline or idle agent never hangs the button, and messaging distinguishes a confirmed refresh from a queued one.
Bring the .NET endpoint agent source (service, tray, installer) into the repo so Cameron can merge and test, with an agent/.gitignore keeping bin/obj/publish out. Includes the DncMain launch-path fix, tray single-file host env scrub, and best-effort file logger.
Default fuzzy reconcile threshold to 0 and refuse merges when digit runs differ so numbered labels like Bay 311 never bind to Bay 310. Keep edit forms from clearing saved model fields when category filtering hides untyped rows. Includes in-progress computers policy and endpoints command feature work.
Server-side RMM telemetry plugin: current-snapshot endpointstates plus append-only curated endpointevents, resolved to existing Computer assets by hostname. Adds ADR-010 asset panels (status, services, recent events), dashboard quiet and stability cards, and a minimal Fleet list view that links into existing PC detail pages. Registers the activity sidebar icon and a dev-only sample seeder. Ingest arrives on the generic collector route; the future C# agent design record lives in docs/DESIGN-DECISIONS-agent.md.
