ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge wilm: gevent migration, endpoint telemetry phase 2, backup/restore

Browse Source 
Brings in 98 commits from wilm (origin/wilm @ 811e6ca61a):

Major features:
- Move API to gevent workers for better concurrency
- Endpoint agent startup jitter (prevents thundering herd)
- Complete backup/restore with restic
- Endpoint telemetry phase 2 (disk health, mapped drives, user context)
- PC Detail UI rework with new cards and tabs
- Dell Command actions and Corporate Tools tab
- Agent version column in Fleet view

Conflict resolution:
- Backup/restore: Accepted wilm implementation (10 files)
- Agent startup jitter: Accepted wilm v1.12.0
- PC Types list fix: Accepted wilm version
- Endpoint plugin domain clamping: Kept ours (defensive)
- Test files: Accepted wilm versions for consistency

Testing required:
- Verify gevent handles concurrent requests
- Test backup/restore functionality
- Verify agent startup jitter timing
- Test endpoint telemetry collection
- Validate PC Detail UI changes

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Proudlock
2026-09-06 21:23:42 -04:00
parent 95abe72729 811e6ca61a
commit a6af09858c
32 changed files with 986 additions and 710 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
.gitignore vendored
View File

@@ -91,20 +91,24 @@ _transfer/
# Local developer scratch notes and chat handoffs - never committed
devnotes/
# docs/ is generated/published to the wiki, not tracked in the product repo.
# A single tracked file here makes docs/ exist in the CI checkout, which
# defeats the intentional skip in the docs publish/version tests. Keep this
# local-only guide out of version control so those tests skip as designed.
docs/CSV-IMPORT-GUIDE.md
# docs/ prose is published to the wiki, not tracked in the product repo. The
# docs gates detect a published checkout by the ABSENCE of docs/*.md, so a
# single tracked prose file here makes CI look like an internal checkout while
# carrying almost no prose: test_there_are_docs_to_check then fails asserting
# >20 files, test_the_map_itself_is_current fails on a generated PROJECT-MAP
# that is correctly untracked, and check-gates-ran.sh fails on the resulting
# skip. That happened - DEV-BUILD-TEST.md and PLUGIN-HOOKS.md were committed in
# unrelated changes and broke the branch for days. Ignore the whole pattern
# rather than naming files one at a time, so the next authored doc cannot
# reintroduce it.
#
# Scoped to the top level on purpose: docs/notes/ prose stays tracked, and the
# gates only glob docs/*.md. docs/openapi.json is tracked (the app serves it)
# and is unaffected by a *.md rule.
docs/*.md
# Local-only authored docs (executive summaries, workflow/security writeups,
# planner exports, draft decks and diagrams). Same reasoning as above: keep
# personal docs out of version control. docs/openapi.json stays tracked and
# is intentionally NOT ignored here.
docs/EXECUTIVE-SUMMARY.md
docs/DESIGN-DECISIONS-agent.md
docs/OPERATOR-WORKFLOW.md
docs/SECURITY-RISK-ASSESSMENT.md
# Local-only authored docs (planner exports, draft decks and diagrams). Same
# reasoning as above: keep personal docs out of version control.
docs/ShopDB-Planner-Tasks.csv
docs/DRAFT--ShopDB-CSV-Import-Guide.pptx
docs/diagrams/

27
Dockerfile
View File

@@ -109,8 +109,13 @@ RUN apt-get update \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt ./
# gunicorn and gevent are container-only: requirements.txt is a universal,
# hash-checked lockfile that also serves the Windows/IIS install, where waitress
# is the server and neither of these is used. gevent is the worker class (see
# the CMD below) because the endpoint long-poll holds a connection per agent,
# which a thread pool cannot absorb at fleet scale.
RUN pip install --no-cache-dir -r requirements.txt \
&& pip install --no-cache-dir gunicorn
&& pip install --no-cache-dir gunicorn gevent
COPY shopdb/ ./shopdb/
COPY plugins/ ./plugins/
@@ -134,21 +139,27 @@ USER shopdb
EXPOSE 5001
# Probe the public health route, not the login endpoint. Probing POST
# /api/auth/login logged a 400 every 30s and ran the credential path from a
# health check. The route answers 200 even when the database is degraded, so the
# body's status field is what decides health. Compose files that terminate TLS
# override this with an https probe; plain http is correct for the pull and
# airgap stacks, which inherit this unchanged.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl --silent http://localhost:5001/api/auth/login -X POST \
-H "Content-Type: application/json" -d '{}' \
| grep -q "VALIDATION_ERROR" || exit 1
CMD curl --silent http://localhost:5001/api/dashboard/health \
| grep -q '"status":"ok"' || exit 1
# Threaded workers, matching docker-compose.yml - see the reasoning there. A
# Async workers, matching docker-compose.yml - see the reasoning there. A
# compose file overrides this CMD, but an image run without one (a plain
# `docker run`, a site's own orchestrator) would otherwise still get the
# blocking sync default and the same starvation.
CMD ["gunicorn", \
"--bind", "0.0.0.0:5001", \
"--worker-class", "gthread", \
"--worker-class", "gevent", \
"--workers", "4", \
"--threads", "8", \
"--timeout", "30", \
"--worker-connections", "2000", \
"--timeout", "60", \
"--access-logfile", "-", \
"--access-logformat", "%(h)s %(l)s %(u)s %(t)s \"%(r)s\" %(s)s %(b)s %(D)sus \"%(f)s\" \"%(a)s\"", \
"--error-logfile", "-", \
"wsgi:app"]

17
agent/ShopDbEndpointAgent.Core/AgentConfig.cs
View File

@@ -17,6 +17,23 @@ public sealed class AgentConfig
// (thundering herd). Actual delay = cadence +/- up to jitterseconds.
[JsonPropertyName("jitterseconds")] public int JitterSeconds { get; set; } = 60;
// One-time random delay before FIRST contact, applied by both the telemetry
// and command loops at startup. JitterSeconds only spreads the gap BETWEEN
// runs, so without this every agent in the fleet still makes its first
// request the instant its service starts - and the service is installed
// start=auto, so a patch reboot starts them all together.
//
// Measured on a 700-endpoint burst (devnotes/gevent-sizing-simulation.md):
// simultaneous arrival drove TLS handshakes to 1482ms worst case and pinned
// CPU at 305% while the database sat idle at 3 active queries. Spreading the
// same arrivals over 30s cut worst-case handshake to 63.7ms. The server is
// not the constraint; simultaneity is.
//
// Costs up to this many seconds before a rebooted endpoint picks up a queued
// command. That window is on top of a reboot the machine was already
// unreachable for. 0 disables the delay.
[JsonPropertyName("startupjitterseconds")] public int StartupJitterSeconds { get; set; } = 30;
[JsonPropertyName("adoptonfirstrun")] public bool AdoptOnFirstRun { get; set; } = true;
// Directory holding site-provided PowerShell "agent utilities" the runtime

17
agent/ShopDbEndpointAgent.Core/CommandWorker.cs
View File

@@ -41,15 +41,18 @@ public sealed class CommandWorker : BackgroundService
private readonly CommandExecutor executor;
private readonly ILogger<CommandWorker> logger;
private readonly CollectSignal? collectSignal;
private readonly AgentConfig? config;
private readonly string hostname;
public CommandWorker(CommandClient client, CommandExecutor executor,
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null)
ILogger<CommandWorker> logger, CollectSignal? collectSignal = null,
AgentConfig? config = null)
{
this.client = client;
this.executor = executor;
this.logger = logger;
this.collectSignal = collectSignal;
this.config = config;
hostname = Environment.MachineName;
}
@@ -57,6 +60,18 @@ public sealed class CommandWorker : BackgroundService
{
logger.LogInformation("Command channel started for host {Host}", hostname);
// Spread first contact across the fleet. Every agent otherwise polls the
// instant its service starts, and the service is start=auto, so a patch
// reboot puts the whole fleet on the wire together. See StartupJitter.
var startupDelay = StartupJitter.Next(config?.StartupJitterSeconds ?? 0, logger);
if (startupDelay > TimeSpan.Zero)
{
logger.LogInformation(
"Delaying first command poll by {Delay}ms (startup jitter)",
(int)startupDelay.TotalMilliseconds);
await SafeDelay(startupDelay, stoppingToken);
}
while (!stoppingToken.IsCancellationRequested)
{
var pollStarted = System.Diagnostics.Stopwatch.StartNew();

51
agent/ShopDbEndpointAgent.Core/StartupJitter.cs Normal file
View File

@@ -0,0 +1,51 @@
using Microsoft.Extensions.Logging;
namespace ShopDbEndpointAgent.Core;
// One-time startup delay, shared by TelemetryWorker and CommandWorker.
//
// Both loops otherwise make their first request the instant the service starts,
// and the service is installed start=auto - so a patch-day reboot has an entire
// fleet arriving together, twice per machine (a telemetry report AND a command
// poll). AgentConfig.JitterSeconds does not help: it spreads the gap BETWEEN
// runs, not the first contact.
//
// Lives here rather than in either worker so the two cannot drift apart, and so
// the clamp is enforced in one place.
internal static class StartupJitter
{
// Ceiling on the configured value. A fat-fingered appsettings.json must not
// be able to park an endpoint for hours before it first checks in: the
// command channel is how an operator would correct it, so an unbounded
// delay would take away the means of recovery.
private const int MaxStartupJitterSeconds = 300;
public static TimeSpan Next(int configuredSeconds, ILogger logger)
{
// Warn rather than clamp silently: an operator who deliberately set a
// large value and got a smaller one has config and behaviour disagreeing
// with no way to tell why. A negative value is a mistake too, so it is
// worth the same line.
if (configuredSeconds > MaxStartupJitterSeconds)
{
logger.LogWarning(
"startupjitterseconds {Configured} exceeds the {Max}s ceiling; using {Max}s",
configuredSeconds, MaxStartupJitterSeconds, MaxStartupJitterSeconds);
}
else if (configuredSeconds < 0)
{
logger.LogWarning(
"startupjitterseconds {Configured} is negative; startup jitter disabled",
configuredSeconds);
}
int spread = Math.Clamp(configuredSeconds, 0, MaxStartupJitterSeconds);
if (spread == 0)
return TimeSpan.Zero;
// Milliseconds, not whole seconds: 700 endpoints over a 30s window would
// otherwise land in 30 clumps of ~23 simultaneous arrivals, which is the
// same pile-up in miniature.
return TimeSpan.FromMilliseconds(Random.Shared.Next(0, spread * 1000));
}
}

19
agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs
View File

@@ -139,6 +139,25 @@ public sealed class TelemetryWorker : BackgroundService
"Endpoint agent {Version} -> {BaseUrl}, cadence {Cadence}s +/-{Jitter}s",
agentVersion, config.BaseUrl, config.CadenceSeconds, config.JitterSeconds);
// Spread first contact across the fleet. JitterSeconds only spreads the
// gap BETWEEN runs, so without this the whole fleet reports the instant
// its service starts. See StartupJitter.
var startupDelay = StartupJitter.Next(config.StartupJitterSeconds, logger);
if (startupDelay > TimeSpan.Zero)
{
logger.LogInformation(
"Delaying first telemetry report by {Delay}ms (startup jitter)",
(int)startupDelay.TotalMilliseconds);
try
{
await Task.Delay(startupDelay, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
}
bool firstRun = true;
while (!stoppingToken.IsCancellationRequested)
{

3
agent/ShopDbEndpointAgent/Program.cs
View File

@@ -346,7 +346,8 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config
sp.GetRequiredService<CommandClient>(),
sp.GetRequiredService<CommandExecutor>(),
sp.GetRequiredService<ILogger<CommandWorker>>(),
sp.GetRequiredService<CollectSignal>()));
sp.GetRequiredService<CollectSignal>(),
config));
}
static void ConfigureLogging(ILoggingBuilder logging, string logDirectory)

23
agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj
View File

@@ -131,6 +131,27 @@
(AgentConfig.LaunchPaths, shipped defaults overridable per site).
1.1.0: generic, data-driven service_*/process_* actions carry their
target over the wire, so new policies/apps need no agent rebuild. -->
<!-- 1.12.0: startup jitter before first contact. Both loops previously made
their first request the instant the service started, and the service is
installed start=auto, so a patch reboot put the whole fleet on the wire
together - twice per machine (a telemetry report AND a command poll).
jitterseconds did not help: it spreads the gap BETWEEN runs, not first
contact. A 700-endpoint burst drove TLS handshakes to 1482ms worst case
and pinned CPU at 305% while the database sat idle at 3 active queries;
spreading the same arrivals over 30s cut worst-case handshake to 63.7ms
with no server-side change. New startupjitterseconds (default 30, 0
disables) is applied by StartupJitter.Next in both TelemetryWorker and
CommandWorker. Randomized in milliseconds rather than whole seconds
because a 30s window at second granularity re-clumps a large fleet into
30 buckets of simultaneous arrivals. Clamped to 300s and warned about
when out of range: the command channel is how an operator would correct
a bad value, so an unbounded delay would remove the means of recovery.
Costs up to startupjitterseconds of extra latency before a rebooted
endpoint picks up a queued command, on top of a reboot the machine was
already unreachable for. The once-only console switch bypasses the delay
entirely (it calls RunOnceAsync directly), so smoke tests stay instant.
appsettings schema gains a key; older packages without it get the
AgentConfig default. No tray change - tray stays 1.4.0. -->
<!-- 1.11.0: on-demand user-context refresh. "Refresh Status" (collect_now)
and the tray's "Send Status" re-sent whatever was already in the spool,
so a drive the operator had just unmapped kept reporting OK until the
@@ -150,7 +171,7 @@
member of that group, so Start-ScheduledTask returned access denied and
the task only ever ran on its own trigger. Requires tray 1.4.0 - the tray
is the side that listens, so an older tray simply never answers. -->
<Version>1.11.0</Version>
<Version>1.12.0</Version>
<InvariantGlobalization>true</InvariantGlobalization>
<RuntimeIdentifier>win-x64</RuntimeIdentifier>
<SelfContained>true</SelfContained>

1
agent/ShopDbEndpointAgent/appsettings.json
View File

@@ -3,6 +3,7 @@
"apikey": "REPLACE_WITH_COLLECTOR_KEY",
"cadenceseconds": 900,
"jitterseconds": 60,
"startupjitterseconds": 30,
"adoptonfirstrun": true,
"utilitiespath": "",
"logpath": "",

48
docker-compose.yml
View File

@@ -79,28 +79,39 @@ services:
- gunicorn
- --bind
- 0.0.0.0:5001
# Threaded workers, not the blocking sync default. An agent that fails
# the TLS handshake retries in a loop, and with 4 sync workers a handful
# of those pinned every worker: real UI requests queued 20+ seconds while
# the CPU sat idle. A stuck connection now holds one THREAD, and 32
# (4 x 8) have to be exhausted before a person waits.
# Async workers, not threads. The endpoint agent long-polls
# /api/endpoints/commands/next and re-polls immediately on a 204, so every
# agent holds a connection essentially continuously and fleet size IS the
# concurrency. Under gthread that meant 4 x 8 = 32 slots total, and a
# single worker's 8 threads could fill with sleeping agents while the CPU
# sat idle - measured: 8 of 9 threads in timed sleep, UI p95 24s.
#
# Found in production at the other site (their commit db75ca7). Their
# server, our agent - so the same client behaviour will reach any site
# running the endpoint agent, and the fix belongs in the product rather
# than on one branch.
# A held connection under gevent is a greenlet, not an OS thread, so idle
# is nearly free. Measured on this stack (devnotes/gevent-sizing-simulation.md):
# 700 simulated agents plus UI, integration and kiosk traffic served with
# zero errors at UI median 7.5ms, peak RSS 479MB. CPU, not memory or DB
# connections, is the binding resource.
- --worker-class
- gthread
- gevent
- --workers
- "4"
- --threads
- "8"
# 30, not 60: a hung connection frees its thread in half the time, which
# is what decides whether a retry storm is absorbed or queues.
# Per-worker greenlet ceiling. 4 x 2000 is far above any realistic fleet;
# it exists so a connection spike queues rather than being refused.
- --worker-connections
- "2000"
# 60, not 30: the timeout must exceed _CLAIM_HOLD_SECONDS in
# plugins/endpoints/api/routes.py or gunicorn kills a healthy long-poll
# mid-hold. Cheap here because a stuck connection costs a greenlet.
- --timeout
- "30"
- "60"
- --access-logfile
- "-"
# %(D)s is request duration in microseconds. Without it a slow request is
# indistinguishable from a fast one in the log, and latency has to be
# reconstructed from outside the container. Keep in sync with the
# Dockerfile CMD.
- --access-logformat
- '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"'
- --error-logfile
- "-"
- --certfile
@@ -112,8 +123,11 @@ services:
# Match the TLS listener: probe over https and accept the self-signed cert
# (-k). The Dockerfile default probes http and would stay red once gunicorn
# only speaks TLS.
test: ["CMD", "curl", "--silent", "--insecure", "https://localhost:5001/api/auth/login",
"-X", "POST", "-H", "Content-Type: application/json", "-d", "{}"]
# /api/dashboard/health is the public probe route and answers 200 even when
# the database is down, so the body's status field is what decides health.
# Probing POST /api/auth/login instead logged a 400 every 30s and exercised
# the credential path from a health check.
test: ["CMD-SHELL", "curl --silent --insecure https://localhost:5001/api/dashboard/health | grep -q '\"status\":\"ok\"'"]
interval: 30s
timeout: 5s
start_period: 10s

0
docs/ShopDB-Planner-Tasks-v2.csv Normal file
View File

631
docs/notes/endpoint-telemetry-and-ui-phases.md Normal file
View File

@@ -0,0 +1,631 @@
# Endpoint telemetry and PC Details UI - phased working notes
Scratch/reference notes for a multi-phase effort. Not a contract document. When
a phase lands, its details belong in AGENT-CONTRACT.md, the csproj changelog, or
the panel declarations - not here. Delete this file when all phases are done.
Branch: wilm. Current versions at time of writing: agent 1.8.0, tray 1.1.0.
---
## Ground truth established before scoping
Verified in code, not assumed. These constrain everything below.
**Panel groups are a hardcoded frontend list.**
`frontend/src/components/PluginAssetPanels.vue:423`:
const RENDERED_GROUPS = ['details', 'monitors', 'software', 'activity']
Any panel whose `group` is not in that list silently falls back to `details`.
Nothing is logged. The comment directly above it records that this has already
caused two disappearing-card incidents (`actions` -> `monitors`, and the
`metrics` -> `monitors` rename). Tab display labels are separately hardcoded in
`plugins/computers/frontend/views/PCDetail.vue:441-449`.
Consequence: introducing a new tab is a THREE-file change that must land in one
commit - panel `group` in `plugins/endpoints/plugin.py`, `RENDERED_GROUPS`, and
the PCDetail label map. Split it and cards vanish with no error.
**There is no pinned/locked card concept today.**
Layout is a CSS grid ordered purely by the integer `position` on each panel.
There is no "top row", no span control except one special case: the Network
interfaces table sets full-width span because at half width it wrapped every
cell to three lines (`PluginAssetPanels.vue:693`).
Consequence: "lock this card top-left/top-right" does not exist yet and is
itself a feature, not a config value. See Phase 5.
**Two of the requested cards are not plugin panels at all.**
Both the Entra/Intune card (`PCDetail.vue:202-243`) and the Network/IP card
(`PCDetail.vue:249-265`) are hand-written Vue inside PCDetail's details grid,
not declarations in `plugin.py`. Moving them is a component refactor, not a
`position`/`group` edit.
**There is no LAPS retrieval feature in this codebase.**
Searched: no LAPS button, endpoint, or credential path exists. The only hit is
a historical incident note in `plugins/tools/frontend/views/SecureSendCreate.vue:254`
about LAPS passwords once being sent over a chat link. The request assumes a
click-to-retrieve LAPS button already exists on the Entra card; it does not.
This needs its own scoping conversation - it is a credential-disclosure path
with real audit/permission implications and must not be smuggled in as part of
a layout change. See Open Questions.
**Existing panel positions in `monitors`** (so new cards do not collide):
10 status, 12 tracked services, 13 tracked processes, 14 hardware, 15 osinfo,
16 interfaces, 17 storage, 18 operator reports, 19 listeners, 40 user context.
`activity` uses 20. `software` uses 30.
---
## Phase 1 - Mapped drive reachability (agent + tray)
STATUS: code complete, NOT fully validated. See "Where we left off" at the end
of this file before resuming.
Highest value, and the reason the user-context work exists. Extends the probe
already added in agent 1.8.0 / tray 1.1.0.
**No active reachability testing.** An earlier draft proposed touching each
share to see if it responded, with a per-mapping timeout. Rejected: Windows
already reports the status, so probing the share is wasted work and reintroduces
the hang risk the bounded probe exists to avoid. Read the status, do not test it.
**The two status sources disagree, and that disagreement is the signal.**
Captured live on the same box at the same moment, same two mappings:
Get-SmbMapping net use
OK Q: ...specdata Disconnected Q: ...specdata
OK S: ...shared Reconnecting S: ...shared
`Get-SmbMapping` reported OK for both while `net use` reported Disconnected and
Reconnecting. So neither source alone is trustworthy: collecting only
Get-SmbMapping (the nicer, more structured one) would have reported a healthy
Q: on a box whose Q: was actually down - exactly the failure this card exists
to catch.
Collect BOTH and keep them as separate fields per mapping. Do not silently
reconcile them in the agent; the server/UI decides presentation, and preserving
the disagreement is diagnostically useful in its own right. The card should
surface the WORSE of the two rather than the prettier one.
Status vocabulary must include `Reconnecting` - it appears in the real capture
above and an earlier draft's `Ok`/`Disconnected`/`Unknown` set would have
forced it into the wrong bucket. Keep `Unknown` for probe failure so a failed
check never renders as green Ok.
**Elevation matters and confirms the tray approach.** The capture was taken
from an elevated shell (`start powershell -verb runas`). Because of UAC token
separation, an elevated context sees a DIFFERENT mapped-drive table than the
normal user - often empty. The tray runs unelevated as the operator, which is
the correct context to enumerate from. Do not "improve" this later by elevating
the probe; that would silently empty the card.
Mechanism: the probe already writes JSON to the spool file, so this rides the
existing handoff - run both commands in-session, write the combined result,
service reads it. No new transport.
Server: persist as part of the existing `usercontext` payload key (no new
column, no migration needed - it rides `usercontextjson`).
Note in code, commented out, per request: a future one-click remediation
command will re-establish a Disconnected mapping. Do not implement now, but
leave the command-catalog seam obvious. It will be an allowlisted action in
`plugins/endpoints/commands.py` and must run in the USER session (a SYSTEM
remap creates a mapping the operator cannot see, and per the elevation note
above an elevated remap has the same problem).
## Phase 2 - Cheap system-side telemetry (agent service)
All service-side, no session needed. Group these because they share the
inventory-snapshot plumbing and one migration.
- **Time sync offset and source** - CUT. Do not collect, do not surface.
w32time is Stopped on provisioned endpoints, confirmed on more than one box
(`get-service w32time` -> Stopped). The service appears to be stopped by
server-side policy after provisioning, so `w32tm /query /status` returns
0x80070426 "service has not been started" fleet-wide and there is no offset to
read. Collecting it would return null everywhere and produce an alerting
signal that never fires - worse than nothing, because a permanently-null
health check reads as "no problem found".
Implemented and then removed in this phase. If time skew becomes a real
suspect later, the prerequisite is getting w32time RUNNING on the fleet; the
collection is the easy part. One implementation note worth keeping: parse the
offset by LABEL, not by "first value ending in s". w32tm prints Root Delay and
Root Dispersion in the same seconds format BEFORE Phase Offset, so a
first-match parser silently reports network latency as clock skew. That bug
was written and caught here; do not rewrite it.
- **Disk errors / SMART predictive failure** - DONE. Landed as its own "Disk
health" card rather than folded into the Storage card as originally planned:
Storage is per VOLUME (C:, D:) and this is per PHYSICAL device, and one drive
can back several volumes or none the operator sees, so the two do not share a
key and could not be one table.
The important finding: the classic ATA SMART class
(`MSStorageDriver_FailurePredictStatus`) returns ZERO instances on an NVMe
machine. Verified on a two-NVMe box where both drives were present and
healthy. An agent collecting only that class would have reported an empty
disk health list on every modern endpoint and read as "no problems" - the
exact silent-healthy failure mode this telemetry exists to prevent. So the
agent collects BOTH `MSFT_PhysicalDisk` (the only NVMe-aware source) and
`Win32_DiskDrive.Status` (ATA/SATA, and hosts where the Storage namespace is
unavailable), and the server does not reconcile them - same reasoning as the
Phase 1 mapped-drive merge, where two disagreeing sources was the signal.
Note `MSFT_PhysicalDisk` returns raw enum INTEGERS through
ManagementObjectSearcher, not the friendly strings `Get-PhysicalDisk` prints
(Health=0 Op=2 Media=4 Bus=17 for a healthy NVMe); the mapping lives in the
collector so the wire stays self-describing.
- **NIC link speed and duplex** - DONE. Paired onto the existing interfaces
entries by MAC from `MSFT_NetAdapter` (`Win32_NetworkAdapterConfiguration`
carries no link state, so it is a second query). Both fields are null when
unreadable: verified on the live box that a DISCONNECTED adapter returns
empty Speed and FullDuplex, so defaulting would have reported every unplugged
NIC as half-duplex. Not surfaced.
- **Boot type** - DONE. Two facts kept separate: `hiberbootenabled` from the
registry (fast startup configured) and `lastboottype` from Kernel-Boot event
27 (what the last boot did). Neither substitutes for the other - the registry
cannot say what the last boot did, and the event can age out of the log.
Verified live: BootType=0 with HiberbootEnabled=0. Not surfaced; placement
still undecided.
- **Windows Update history, last 5** - PINNED, deferred out of Phase 2. The COM
update searcher is not cheap: it routinely takes seconds and sometimes tens of
seconds, which would make it the most expensive thing in an otherwise
WMI-and-registry `Collect()` running on the heartbeat path.
The blocking concern is broader than cost. Some endpoints host COM server
applications, and driving the update searcher through COM on those boxes risks
interfering with collections - or with the hosted application itself. Needs
its own investigation before any implementation, and if it lands it wants its
own cadence (hourly/daily), never the every-cycle path.
- **Defender/AV health** - DONE, collected and persisted, NOT surfaced, exactly
as specified. Note the ages are in days and a real 0 means updated today, so
the collector sends null rather than 0 when Defender did not report. A box
with third-party AV or a stripped image has no Defender WMI provider at all,
which is normal here and not a fault.
Explicitly dropped after discussion, do not collect:
certificates (corporate-managed), BitLocker (network join already requires it),
display topology (single monitor on the floor, trained users), session
type/RDP (RDP-in not permitted on the shop floor), logon server (local users).
**Pending file-rename operations** - DONE, but NOT as a new collection. It was
already being read: `GetPendingReboot()` OR-s it together with the CBS and
Windows Update keys into one bool. The gap was that the bool cannot say WHICH
signal fired, so the "stronger signal" was already being flattened into the
weaker one. Shipped as `rebootreasons`, a list naming the signals that fired,
leaving `pendingreboot` untouched for compatibility. Empty list = checked,
nothing pending; null = the check failed.
Schema: these add columns to `endpointstates`. This landed as `0015`, not the
`0014` reserved here - `0014` was taken by the disk health column earlier in
Phase 2.
## Phase 3 - Service/Process Monitors tab split
Tab churn is exactly what has silently eaten cards twice before, so this phase
is deliberately isolated from any collection work.
- Rename the existing `Monitors` tab to `System Monitors`.
- Add a new `Service/Process Monitors` tab.
- Move `Tracked Services` (position 12) and `Tracked Processes` (13) into it.
Must land atomically across all three places:
1. `group` on those panels in `plugins/endpoints/plugin.py`
2. `RENDERED_GROUPS` in `PluginAssetPanels.vue`
3. the tab label map in `PCDetail.vue`
Also update the hardcoded panel-count assertion in
`tests/test_plugins/test_endpoints_plugin.py`
(`test_endpoints_asset_panels_are_computer_scoped`) - it has already needed
bumping once (7 -> 12) and will fail again here.
Printers: a location-based printer auto-detect collection is in progress
elsewhere. Reserve space and a position band for a printer card in the monitors
family so this work does not have to be redone when it lands. Do not build the
card now.
## Phase 4 - Card content additions
Depends on Phase 2 data existing.
- **Mapped drives card** - DONE. `list` render at position 21 in Monitors, one
row per drive letter with the remote path as meta and a status badge. Reuses
the existing status/badge vocabulary (`statecolor` + the ok/warning/critical
levels the tracked-services cards already declare) rather than new CSS.
Only UNHEALTHY drives are coloured. Green on every working drive makes the
broken one just another badge in a row of badges; the card exists to make the
exception findable. For the same reason unhealthy rows sort first - a user
with a dozen mappings should not hunt for the broken one - and the sort is on
health ALONE so drive-letter order survives within each group and the list
does not reshuffle between refreshes.
The merge itself was not touched. `_mapped_drive_rows` now returns structured
rows and `_mapped_drives` (the user-context summary line) is built on top of
it, so the card and the summary cannot drift apart about a drive's status.
Room reserved for the one-click reconnect: the `list` render already supports
a row action, so adding it is a map/commandendpoint change, not a new card.
- **Storage card** - superseded. Disk health shipped in Phase 2 as its own
"Disk health" card (position 18, directly under Storage) instead of being
folded in: Storage is per volume, disk health is per physical device, so they
do not share a key. Rows sort failure-first, then unknown, then OK - a failing
drive must not sit below the fold on a multi-disk box, and a drive that
stopped answering is a weaker claim of health than one that said OK.
Later merged into a single two-section card; see the entry below.
- **Windows Update card** - BLOCKED, not scheduled. Depends on Phase 2 data
that is pinned pending a COM-interference investigation. Do not build the card
until the collection question is settled.
- Boot type placement - DONE in Phase 3. Rendered in the OS card only when fast
startup applies, because that is the case where uptime understates how long
the kernel has been up. A permanent "Cold" row would be noise.
- **Storage + Disk health into one card with two collapsible sections** -
DONE. VISUAL GROUPING ONLY. These two tables are not merged into shared rows:
Storage is per volume, Disk health is per physical device, and one drive can
back several volumes or none the operator sees. This is emphatically NOT the
Phase 1 mapped-drive treatment, where two sources described the same drive and
the worse status won. Nothing here is reconciled.
Built as predicted: `tabs` was extended to support table sections (columns/
rows/empty per section) in the shared renderer, and the two routes collapsed
into one `/storage` returning `sections`. `/diskhealth` is kept as a route -
it is a stable read for non-panel callers, and removing it for a UI-only
change would break them.
Two things only found by building it:
- The empty-section filter had to grow a table branch. A table section
declares its columns up front and so must survive having no rows; dropping
it takes its TAB with it, and a missing tab reads as "this machine has no
disks" rather than "nothing was reported".
- Full-width span was keyed off `render === 'table'`. A `tabs` panel holding a
six-column table needs the same span, and whether it does is only knowable
from FETCHED DATA, not the declaration - hence `spansFullWidth()`.
Both are mutation-tested in `tabSectionsTable.spec.js`.
Checked and cleared: the "inline label/value pairs, not nested cards" density
rule in Phase 5 does NOT block this. That rule is scoped to the PC Details
HEADER block, where the constraint is adding zero height so the tab strip is
not pushed toward the fold. It says nothing about panel bodies in the
Monitors tab.
## Phase 5 - Header right-hand block (DEFERRED - revisit before building)
Status: on hold at user request. The original ask assumed pinned cards were a
config value; they are not, and building a pinning system was not what the user
wanted to sign up for. Not scheduled. Nothing below is committed.
The cascade worry that motivated pinning is explicitly parked - do not build
anything for it unless reflow problems actually surface.
**Revised direction: use the empty right-hand half of the existing header.**
The PC Details header is a full-width card whose right side is currently dead
space - the hostname, badges, PC type, OS and location all sit left, and the
Status / Remote access / Refresh / Restart row runs along the bottom. The block
to the right of the identity lines is empty at every width.
This is a better target than either pinned grid cells or extending the header
downward:
- It consumes space that already exists, so it costs ZERO additional vertical
height and cannot push the tab strip toward the fold. That was the hard
constraint, and this approach satisfies it structurally rather than by
budgeting.
- It sidesteps the Network interfaces width problem entirely (that table is
full-width by necessity and would fight a half-width pinned slot).
- Header content lives outside the panel grid, so it cannot be reflowed by
cards appearing or disappearing below it.
Candidates for the block, unresolved:
- Entra/Intune deep links (currently a Details-tab card). Strongest candidate:
they are navigation, not data, and the user has called out their outbound
usability value.
- IP / network summary (currently the hand-written Network card).
Density rules if this proceeds: inline label/value pairs, not nested cards. Do
not let the right block grow taller than the left identity block, or it starts
adding height after all and the zero-cost property is lost.
Scope note: that "not nested cards" rule is about THIS header block only, where
the whole value is adding zero height. It is not a general ban on sections
inside a panel body - the Phase 4 Storage + Disk health grouping was checked
against it and is not blocked.
Still true and still unbuilt if this proceeds:
- `Collectors` is not a current tab or group. If IP History moves to a new tab,
that is the same three-file atomic change as Phase 3.
- Entra empty state, requested: show `Please provide DeviceID` /
`Please provide ObjectID` when the backend has no value, rather than an
empty or broken card.
LAPS is NOT a feature request - clarified by the user as an illustration of why
these outbound portal links matter to usability. No LAPS work is scoped.
## Phase 6 - Intune/Entra link bug (independent, can be done anytime)
Reported: the INTUNE and ENTRA buttons sometimes do not appear until the page
is refreshed. Diagnosed, root cause found, NOT yet fixed.
**Cause: a cold-cache race, not a render problem.**
`frontend/src/utils/siteSettings.js:9` caches only the RESOLVED settings object:
let settingsCache = null
async function loadSettings() {
if (settingsCache) return settingsCache
...
The in-flight promise is not cached, so every caller during the first load
fires its own `settingsApi.list()`.
In `plugins/computers/frontend/views/PCDetail.vue:513`, `getEntraDeviceUrls()`
is deliberately fire-and-forget (`.then(...)`, not awaited) so a settings
failure cannot block the page. Meanwhile `computersApi.get()` on line 519 IS
awaited. On a cold cache the settings response can therefore land AFTER the
computer data has rendered. Until it does, `entraUrlTemplate` /
`intuneUrlTemplate` are still `''`, so `fillTemplate()` returns null
(line 426: `if (!template || !value) return null`), and the `v-if` on the
anchors hides both links. A later refresh hits the warm cache and they appear.
That also explains the intermittency: it depends on which response wins, so it
varies by network timing and by whether another view already warmed the cache.
**Fix: cache the in-flight promise, not just the result.** Assign the promise
to `settingsCache` immediately so concurrent callers await the same request.
This is a one-line-shaped change in `siteSettings.js` and fixes every consumer
of site settings, not just this card - the same race exists for any view that
reads settings during mount.
Watch for: the existing `catch` sets `settingsCache = {}` on failure, which
permanently caches the empty result for the page session. Confirm whether a
failed settings load should stay cached (currently it does) or be retryable.
Do not change that behavior silently while fixing the race.
Verify after fixing that the links appear on a hard reload with a cold cache
and throttled network, which is the condition that reproduces it.
DONE. Fixed in `siteSettings.js`: the in-flight promise is now cached alongside
the result, and a failed load is cached for 30s rather than the page session.
Covered by `frontend/src/utils/siteSettings.spec.js` (6 tests). Open question 5
resolved in favour of the short TTL.
Measured side benefit: the old code assigned `settingsCache` only AFTER the
response resolved, so each sequential `await` re-entered `loadSettings()` before
the cache existed. `applyBranding()` awaits five settings in a row at bootstrap,
`getServicenowUrls()` four, `getEntraDeviceUrls()` two - all of which were
separate GETs on a cold load. Now one.
## Phase 7 - PCForm "Failed to load data" (NOT the same bug)
Separate from Phase 6 and NOT caused by it - `PCForm.vue` does not import
`siteSettings.js`. Surfaced on the F3KKSGH4 edit page.
Partially addressed, root cause still UNKNOWN.
Evidence: the form renders with reference data present (printer list populated,
dropdowns built) but record fields empty. That means the eight-way
`Promise.all` of reference endpoints SUCCEEDED and the failure is in the
edit-only block, `PCForm.vue` lines 589-635. The only unprotected call there is
`computersApi.get(route.params.id)`; `ensureModelLoaded()` already swallows its
own errors, so it is not the source.
Two candidates remain, with different fixes and different layers:
- the `computersApi.get` request fails (backend/route problem), or
- a TypeError while mapping the response into `form.value` (frontend shape
mismatch, plausibly from recent schema work).
Both land in the same catch and both printed the same generic string, which is
why this could not be diagnosed from the UI.
Done so far: the catch now uses `apiError(err, ...)` so the real server message
surfaces instead of a generic one, and the stale comment claiming a call
"handles its own failure" was removed - no such call remains in that list.
NEXT: reload the edit page and read the now-specific error. An axios status
code points at the backend; a TypeError names the property and points at the
mapping block. Do not guess between them.
Also worth doing regardless: a single failed reference list currently blanks
the whole form via `Promise.all` rejection. Degrading one dropdown instead
would be the behaviour the removed comment falsely claimed.
---
## Simulation suite - required check (from a real dev-environment failure)
The suite must assert that CODE AND SCHEMA ARE NEVER SERVED OUT OF STEP, and
must do it by bringing the image UP, not by building a schema from the models.
Observed 2026-09-05 in dev, after rolling containers for the Phase 3/4 UI work:
every query against `endpointstates` returned 500 with
(1054, "Unknown column 'endpointstates.diskhealthjson' in 'field list'")
The Phase 2 migrations (endpoints 0014, 0015) were written and committed WITH
their models, correctly. They had simply never been applied to that database.
`flask plugin upgrade-all` fixed it in one command.
Why nothing caught it:
- The pytest suite builds its schema from the models, so model and schema are
by construction in step. This class of fault is INVISIBLE to it. A green test
run says nothing about whether a deployed database has the columns.
- Nothing in the startup path checks. `create_app()` and `wsgi.py` do no
migration-state validation, so the container reports HEALTHY while every
query against the affected table 500s. The health check passes because it
does not touch the table.
- `flask db preflight` exists but is manual and pre-migration; it is not a
guard on serving traffic.
The failure is also SILENT IN THE UI in a way that misdirects: a panel whose
endpoint 500s is dropped from `populatedGroups`, so its whole TAB disappears
from PC Details. The visible symptom was a missing "Service/Process Monitors"
tab, which reads as a UI regression in the tab-vocabulary code and sends you
to entirely the wrong file. Nothing on the page mentions the database.
So the suite needs, at minimum:
1. A start-up case that runs the real image against a database deliberately
left one migration behind, and asserts the mismatch is reported LOUDLY -
not a healthy container serving 500s.
2. Coverage that the documented order (`flask db upgrade`, then
`flask plugin upgrade-all`, THEN serve) is what the container actually does.
3. A UI-level assertion that a 500ing panel endpoint does not silently remove a
tab with no explanation on screen.
Worth considering separately from the suite: a startup guard that refuses to
serve, or at least logs at ERROR, when a model references a column the database
does not have. Cheap here, expensive on a fleet rollout.
## Open questions - answer before the relevant phase
1. ~~LAPS~~ - RESOLVED: not a feature request. The user was illustrating why the
outbound Intune/Entra links matter to usability. No LAPS work scoped. The
security concerns noted earlier stand if it is ever revisited.
2. ~~Pending file-rename operations~~ - RESOLVED: collect. Folded into Phase 2.
3. ~~Phase 5 pinning / cascade symptom~~ - RESOLVED: not worrying about the
cascade unless it surfaces. Pinned cards dropped in favour of the header's
empty right-hand block; Phase 5 deferred.
4. **`Collectors` tab** - only relevant if Phase 5 proceeds. New tab for one
moved card, or does IP History belong in an existing tab?
5. **Failed settings load caching** - Phase 6. Should a failed
`settingsApi.list()` stay cached for the page session (current behavior) or
be retryable? Decide while fixing the race, do not change it silently.
## Sequencing
1 -> 2 -> 3 -> 4. Phase 6 is independent and can be slotted in anytime; it is
small, self-contained, and fixes a user-visible bug, so it is a good candidate
to do early. Phase 5 is deferred and unscheduled.
Phases 1 and 2 are additive and independently deployable. Phase 3 is a
contained tab move. Phase 4 needs 2.
Agent/tray version bumps: Phase 1 touches the probe (tray behavior) and Phase 2
touches the service collector. Both need csproj changelog entries. Note the
existing coupling hazard - `Program.cs` has no unknown-arg guard, so an older
agent receiving a newer tray's verb starts a full second host rather than
no-oping. Fix that guard before any deployment where tray and agent could skew.
## Phase 8 - fetchAllPages returns a bare array (LAST, and it affects the above)
Deliberately last: it is a cross-cutting API-surface change, and every earlier
phase should land against the current shape rather than a moving one.
`frontend/src/api/index.js:131` - `fetchAllPages()` returns the ROWS:
return items
Every other method on every API object returns the axios RESPONSE. So callers
must remember, per call site, which of two shapes they are holding:
const rows = await vendorsApi.listAll() // array
const response = await vendorsApi.list() // { data: { data: [...] } }
Reading `.data.data` off the array yields `Cannot read properties of undefined
(reading 'data')`. Inside a `Promise.all` that rejects the whole batch, so one
wrong unwrap blanks an entire form - and if it throws before the record fetch,
NO request is made at all, which makes it look like a backend or schema fault.
That is exactly how the PC edit page presented, and why it was misdiagnosed
twice before the log showed `/api/computers/729` was never requested.
Known history - this class of bug has now hit at least four times:
- `MachineForm.vue` `pcsRes` (fixed previously, comment left in place)
- `PCForm.vue` `vendorRes` (fixed now)
- `MachineForm.vue` `vendorRes` and `buRes` (fixed now - the warning comment was
already sitting three lines away and the adjacent defects were still missed)
- `MeasuringToolForm.vue` `buResponse` (fixed now)
`ConfigurationPoliciesList.vue:187` defends with
`Array.isArray(value) ? value : ...`, which is evidence the shape is not
trustworthy at the call site.
Why this matters to the phases above: Phase 1/2 UI work adds call sites that
consume paged reference data, and Phase 5 (if revived) moves cards between
components. Any new `listAll()` caller is another chance to hit this. Landing
the phases first and this last means one migration pass, not a moving target.
Options, in order of preference:
1. Make `fetchAllPages` return `{ data: { data: items } }` so every API method
has ONE shape. Mechanical but touches every existing `listAll()` caller in
the same commit, or they break in the opposite direction. Highest value,
highest blast radius - needs the full grep and a single atomic change.
2. Rename to something the shape is obvious from (`fetchAllRows`,
`listAllRows`). Cheap, no behavior change, keeps the trap but makes it
visible at the call site. Good interim step if 1 is too large right now.
3. Leave it and rely on review. Rejected - four occurrences say review does not
catch it, and one of them happened directly beneath a comment warning about
it.
Whichever is chosen, do the full-codebase grep for `listAll(` first; there are
callers in core views, print views, and at least seven plugin frontends.
---
# Where we left off
Written mid-session when the test tooling stopped working. Read this first.
## Phase 1 is code complete but NOT fully validated
What landed, all uncommitted in the working tree:
**`agent/ShopDbEndpointAgent.Core/UserContextProbe.cs`** - added `Get-SmbMapping`
alongside the existing `Win32_NetworkConnection` capture, emitted as a separate
`smbmapping` key with its own `smbmappingerror`. The two sources are NOT
reconciled in the probe, deliberately - see the Phase 1 notes above. Still
unelevated, still in the operator session.
**`plugins/endpoints/api/routes.py`** - added `_drive_letter`,
`_DRIVE_STATUS_RANK`, `_worse_drive_status`, `_mapped_drives`, and pointed
`panel_usercontext()` at `_mapped_drives()`. Merges the two views by drive
letter and shows the WORSE status. Carries a long WARNING comment naming the
three load-bearing behaviours; that comment is doing real work, leave it.
**`plugins/endpoints/commands.py`** - commented-out `remap_drive` seam with the
user-session/elevation reasoning. Comment only, no behavior.
## Validation state - the important part
Last KNOWN-GOOD full run: `8 failed, 2489 passed, 11 skipped`. All 8 failures
are pre-existing and unrelated:
- 3x `test_endpoints_plugin.py` timestamp tests - these fail ONLY on Windows.
`_in_zone()` calls `time.tzset()`, which does not exist on Windows (Unix-only
in CPython), so it raises AttributeError before the assertions ever run.
`_parse_dt` itself is correct. These pass on CI. Consider a
`skipif(not hasattr(time, 'tzset'))` so the suite is honest on Windows.
- 5x docs / security-headers / plugin-guard, all pre-existing.
TWO CHANGES LANDED AFTER THAT RUN AND ARE UNVERIFIED:
1. `_mapped_drives()` now renders `(Unknown)` for a drive neither source gave a
status for. Previously it rendered blank, which reads as healthy - the exact
silent-green failure the warning comment warns about. Small but real,
behavioral, and untested.
2. The `commands.py` seam. Comment only, zero risk.
`get_errors` on both changed files is clean, but that only proves they parse.
## Tooling failure that stopped validation
Every `run_command_in_terminal` call began failing with "A task was canceled",
including a two-file pytest run that should finish in under a second, and a
background launch that never wrote its `Tee-Object` output file. Two full-suite
runs had succeeded earlier in the same session, so this is not slowness - the
execution path went down partway through.
Test Explorer discovers all 2508 tests but returns zero results for every
filter tried (`TypeName`, `MethodName`, `FullyQualifiedName`). Likely needs a VS
restart.
## Next actions, in order
1. Restart VS, then run `python -m pytest tests/test_plugins/test_endpoints_plugin.py -q`.
Expect 3 failed (the tzset ones) / 23 passed. Anything else is ours.
2. Consider adding tests for `_mapped_drives()` - currently zero coverage on
logic that decides whether a broken drive renders as broken. The cases that
matter: sources disagree (worse wins), drive in only one source (kept), no
status at all (Unknown, not blank), `Get-SmbMapping` absent entirely
(degrades to netuse alone).
3. Only then start Phase 2. Do not begin new collection work on an unverified
base - the untested `(Unknown)` delta would get buried under it.
## Deferred, agreed explicitly
Agent and tray republish happens at the END of all phases, not per-phase. The
server change is safe to deploy ahead of the agent: until the new probe ships,
`smbmapping` is simply absent from the payload and `_mapped_drives()` falls
through to `netuse` alone, rendering exactly as it does today. No ordering
constraint between them.

5
frontend/src/components/panelGroups.spec.js
View File

@@ -13,10 +13,7 @@ import { fileURLToPath } from 'node:url'
// constant in the .vue file with no export to assert against.
function read(relative) {
// .href, not the URL object: the jsdom test environment supplies its own
// URL global, and node's fileURLToPath rejects a foreign instance with
// "must be ... an instance of URL. Received an instance of URL".
return readFileSync(fileURLToPath(new URL(relative, import.meta.url).href), 'utf8')
return readFileSync(fileURLToPath(new URL(relative, import.meta.url)), 'utf8')
}
function renderedGroups() {

5
frontend/src/components/pinnedPanels.spec.js
View File

@@ -13,10 +13,7 @@ import { fileURLToPath } from 'node:url'
// than a layout bug.
function read(relative) {
// .href, not the URL object: the jsdom test environment supplies its own
// URL global, and node's fileURLToPath rejects a foreign instance with
// "must be ... an instance of URL. Received an instance of URL".
return readFileSync(fileURLToPath(new URL(relative, import.meta.url).href), 'utf8')
return readFileSync(fileURLToPath(new URL(relative, import.meta.url)), 'utf8')
}
const panelsSource = () => read('./PluginAssetPanels.vue')

29
frontend/src/views/settings/ShopdbBackupsList.vue
View File

@@ -4,18 +4,9 @@
<h2>Backups</h2>
</div>
<div v-if="!isConfigured" class="not-configured">
<strong>Site Backups is not configured on this deployment.</strong>
<span>
The feature needs the backup overlay (a separate container writing sets
onto a shared volume) and a readable backups root. A Windows/IIS install
does not run it, so nothing here would have a store behind it.
</span>
</div>
<!-- Confidence banner: the single most reassuring thing an admin wants to
see - is there a recent, verified backup? Everything else is secondary. -->
<div v-if="isConfigured" class="confidence-banner" :class="bannerClass">
<div class="confidence-banner" :class="bannerClass">
<span class="banner-icon">
<component :is="bannerIcon" :size="22" />
</span>
@@ -311,9 +302,6 @@ const sets = ref([])
const loading = ref(true)
const isBusy = ref(false)
const lastGoodAt = ref(null)
// Assume configured until the list says otherwise, so a slow first load
// does not flash the not-configured notice on a working install.
const isConfigured = ref(true)
const restoreTarget = ref(null)
// Live restore progress. restoreStatus mirrors the api /status.restore payload;
@@ -417,10 +405,6 @@ async function load() {
const response = await api.get('/shopdbbackups')
sets.value = response.data.data || []
lastGoodAt.value = response.data.meta?.lastgoodat || null
// A deployment without the backup overlay has no store behind any of
// this. Say so instead of showing 'No verified backup yet', which reads
// as a backup that failed rather than a feature that is not installed.
isConfigured.value = response.data.meta?.configured !== false
} catch (error) {
toast.error('Could not load backups.')
} finally {
@@ -663,17 +647,6 @@ onUnmounted(() => {
.banner-none { border-color: #c53030; }
.banner-none .banner-icon { color: #e53e3e; }
.not-configured {
display: flex;
flex-direction: column;
gap: 0.35rem;
padding: 1rem 1.25rem;
margin-bottom: 1rem;
border: 1px solid var(--border-color, #d8dbe0);
border-radius: 8px;
background: var(--surface-muted, #f6f7f9);
}
.explain {
margin-bottom: 1.25rem;
border: 1px solid var(--border);

7
plugins/computers/frontend/views/PCTypesList.vue
View File

@@ -107,10 +107,9 @@ onMounted(() => loadData())
async function loadData() {
loading.value = true
try {
// listAll resolves to a plain array (fetchAllPages), not an axios
// response. Reading .data.data threw and left the page empty.
const response = await computersApi.types.listAll({ active: false })
pcTypes.value = response || []
// listAll() pages past the server's 100-row cap and returns the ROWS, not
// an axios response - there is no .data to unwrap.
pcTypes.value = await computersApi.types.listAll({ active: false })
} catch (err) {
console.error('Error loading PC types:', err)
} finally {

19
plugins/endpoints/api/routes.py
View File

@@ -1828,6 +1828,25 @@ def revoke_agentkey():
# How long the agent claim poll is held open server-side before returning empty.
# Kept below the agent HttpClient timeout and typical proxy idle timeouts so a
# quiet fleet re-polls cleanly instead of tripping a read timeout.
#
# 25s is the value this endpoint is designed around; it was temporarily dropped
# to 5s under the blocking gthread worker pool, where a held connection pinned
# one of a worker's 8 OS threads. Measured then: 14 agents at hold=25 left a
# single worker with 8 of 9 threads parked in timed sleep and UI requests hashed
# to that worker waited 12-24s. Under gevent a held connection is a greenlet and
# the per-worker ceiling is --worker-connections (2000), not 8 threads, so an
# idle hold no longer costs anything a UI request needs.
#
# Restored to 25 because hold length sets the fleet's REQUEST RATE, not its
# concurrency: the agent re-polls immediately on a 204, so hold=5 is 5x the
# requests of hold=25, and TLS handshake CPU was the measured burst bottleneck
# (devnotes/gevent-sizing-simulation.md). 25 also restores the margin against
# the agent's MinIdlePollDuration=5s floor, which CommandWorker.IsIdlePollClose
# uses to tell a normal hold-window close apart from a transient failure; at
# hold=5 that margin was ~20ms.
#
# Constraint: gunicorn --timeout must stay above this or a healthy long-poll is
# killed mid-hold. docker-compose.yml sets 60.
_CLAIM_HOLD_SECONDS = 25
_CLAIM_POLL_INTERVAL = 1.0

1
plugins/endpoints/api/siteagent.py
View File

@@ -118,6 +118,7 @@ _BASE_APPSETTINGS = {
'apikey': 'REPLACE_WITH_COLLECTOR_KEY',
'cadenceseconds': 900,
'jitterseconds': 60,
'startupjitterseconds': 30,
'adoptonfirstrun': True,
'utilitiespath': '',
'logpath': '',

19
plugins/printers/api/asset_routes.py
View File

@@ -1123,12 +1123,18 @@ def set_asset_printer_assignment(asset_id: int):
', '.join(str(assetid) for assetid in sorted(notprinters))))
typeids = _relationship_typeids(_USES_PRINTER, _DEFAULT_PRINTER)
if not typeids[_USES_PRINTER] or not typeids[_DEFAULT_PRINTER]:
missingtypes = [name for name in (_USES_PRINTER, _DEFAULT_PRINTER)
if not typeids[name]]
if missingtypes:
# Seed data, not a migration. An un-seeded database cannot hold an
# assignment, and saying so beats writing rows nothing can read.
# assignment, and saying so beats writing rows nothing can read. Name
# the types that are actually missing: a partially seeded table is the
# common case, and "types are not seeded" sends someone looking for an
# empty table they will not find.
return error_response(
ErrorCodes.INTERNAL_ERROR,
'Relationship types are not seeded - run: flask seed reference-data',
'Relationship type(s) not seeded: {0} - run: flask seed '
'reference-data'.format(', '.join(missingtypes)),
http_code=500)
_reconcile_edges(asset_id, typeids[_USES_PRINTER][0],
@@ -1784,10 +1790,13 @@ def seed_assignment_from_observed(asset_id: int):
details={'hostname': hostname, 'skipped': candidate['skipped']})
typeids = _relationship_typeids(_USES_PRINTER, _DEFAULT_PRINTER, _CONTROLS)
if not typeids[_USES_PRINTER] or not typeids[_DEFAULT_PRINTER]:
missingtypes = [name for name in (_USES_PRINTER, _DEFAULT_PRINTER)
if not typeids[name]]
if missingtypes:
return error_response(
ErrorCodes.INTERNAL_ERROR,
'Relationship types are not seeded - run: flask seed reference-data',
'Relationship type(s) not seeded: {0} - run: flask seed '
'reference-data'.format(', '.join(missingtypes)),
http_code=500)
warnings = []

19
scripts/backup/Dockerfile.backup
View File

@@ -14,23 +14,8 @@ FROM mysql:8.4
# bzip2/python3/alembic or a base image that carries them) so the release image
# needs no outbound calls and no embedded cert. The cert is gitignored (*.pem)
# and must be staged at scripts/backup/zscaler-root-ca.pem before building.
#
# Copied via the directory rather than by name, because the cert is OPTIONAL: it
# is gitignored, so a plain `COPY scripts/backup/zscaler-root-ca.pem` fails the
# build outright and nobody outside a TLS-intercepting network - another site,
# or our own CI - could build this image at all. A trailing glob does not fix
# that either: BuildKit fails a COPY whose pattern matches nothing, which is
# precisely the case being handled. Copying a directory that always exists and
# testing for the file inside it is the form that works both ways.
COPY scripts/backup/ /tmp/buildctx/
RUN if [ -f /tmp/buildctx/zscaler-root-ca.pem ]; then \
echo "staging corporate root CA"; \
cp /tmp/buildctx/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/; \
update-ca-trust; \
else \
echo "no corporate root CA staged - using the public trust store"; \
fi \
&& rm -rf /tmp/buildctx
COPY scripts/backup/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/zscaler-root-ca.pem
RUN update-ca-trust
# restic from the distro is often stale; pull the pinned release binary so the
# snapshot format and retention flags are predictable across sites.

0
scripts/backup/_probe_manifest.py Normal file
View File

53
scripts/backup/backup.sh
View File

@@ -47,12 +47,6 @@ mkdir -p "${SETDIR}"
status_write backup database "${SETNAME}" "Dumping database"
# 1. Database: whole-DB logical dump, consistent and non-blocking on InnoDB.
# Dumped WITHOUT --databases on purpose. That flag bakes CREATE DATABASE and USE
# <name> into the file, which pins the dump to the schema name it was taken
# from: restoring onto a site whose MYSQL_DATABASE differs loaded the rows into
# the dump's schema, left the live one untouched, and reported success.
# --no-tablespaces because dumping tablespace metadata needs the PROCESS
# privilege, which the app user does not have on MySQL 8.
log "dumping database ${MYSQL_DATABASE} from ${MYSQL_HOST}"
mysqldump \
--host="${MYSQL_HOST}" \
@@ -61,8 +55,7 @@ mysqldump \
--single-transaction \
--routines \
--triggers \
--no-tablespaces \
"${MYSQL_DATABASE}" \
--databases "${MYSQL_DATABASE}" \
| gzip -c > "${SETDIR}/db.sql.gz"
log "database dump written ($(du -h "${SETDIR}/db.sql.gz" | cut -f1))"
@@ -80,48 +73,7 @@ DATAREV="$(mysql \
--skip-column-names --batch \
-e "SELECT version_num FROM alembic_version LIMIT 1;" \
"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
if [ -z "${DATAREV}" ]; then
# Recorded loudly rather than shrugged off: a set with no data revision
# cannot be gated on restore, and the gate treats an empty stamp as "no
# evidence" and refuses it. Better a visible warning now than a set that
# looks restorable and is not.
log "WARN could not read alembic_version - this set will NOT be restorable"
history_append backup warning "${SETNAME}" \
"Could not read alembic_version; set has no data revision and cannot be restored"
fi
# The same question for every PLUGIN chain. Each plugin owns an
# alembic_version_<plugin> table, and those move independently of the core one:
# code rolled back to an older plugin revision can sit at an identical core
# head while the dump carries plugin tables it has never heard of. Stamping only
# the core revision let that combination pass the restore gate, load, and then
# fail during the plugin upgrade with the data already replaced. Read straight
# from the dumped DB, same as the core revision above.
log "reading applied plugin revisions (alembic_version_*)"
DATAPLUGINS="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT table_name FROM information_schema.tables
WHERE table_schema = '${MYSQL_DATABASE}'
AND table_name LIKE 'alembic\\_version\\_%';" 2>/dev/null \
| tr -d '\r' \
| while read -r table; do
[ -n "${table}" ] || continue
plugin="${table#alembic_version_}"
rev="$(mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--skip-column-names --batch \
-e "SELECT version_num FROM \`${table}\` LIMIT 1;" \
"${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')"
[ -n "${rev}" ] && printf '%s\t%s\n' "${plugin}" "${rev}"
done \
| python3 -c 'import json,sys; print(json.dumps(dict(l.rstrip("\n").split("\t",1) for l in sys.stdin if "\t" in l)))')"
[ -n "${DATAPLUGINS}" ] || DATAPLUGINS='{}'
log "plugin revisions: ${DATAPLUGINS}"
[ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty"
log "writing manifest"
python3 /opt/backup/manifest.py \
@@ -130,7 +82,6 @@ python3 /opt/backup/manifest.py \
--site "${SITE}" \
--imagetag "${SHOPDB_IMAGE_TAG:-source}" \
--datarev "${DATAREV}" \
--dataplugins "${DATAPLUGINS}" \
--migrations /app/migrations \
--plugins /app/plugins

0
scripts/backup/backup.sh.tmp Normal file
View File

39
scripts/backup/manifest.py
View File

@@ -10,19 +10,12 @@ restore." It records, at capture time:
each plugin chain's heads - the versions a restore target must
be at least, or restored rows reference columns the live schema
has not applied yet.
- datarev : the CORE Alembic revision actually applied to the DATABASE that
was dumped (its alembic_version), which is the number a restore
must gate on. schemarev stamps the code on disk; datarev stamps
the data in the dump. They diverge when an image is ahead of a DB
- datarev : the Alembic revision actually applied to the DATABASE that was
dumped (its alembic_version), which is the number a restore must
gate on. schemarev stamps the code on disk; datarev stamps the
data in the dump. They diverge when an image is ahead of a DB
that has not run 'flask db upgrade' yet, and that divergence is
exactly what a code-to-code check cannot see.
- dataplugins: the same thing for every PLUGIN chain - the applied revision of
each alembic_version_<plugin> table. The core chain alone cannot
answer compatibility: plugin schemas move independently, so code
rolled back to an older plugin revision can sit at an identical
core head while the dump carries plugin tables it has never heard
of. Gating on core only let that pass, load, and then fail during
the plugin upgrade with the data already replaced.
Two modes:
--out ... write a fresh manifest for a new set
@@ -74,25 +67,6 @@ def _heads_for(chain_dir):
return sorted(rev for rev in revisions if rev not in down_refs)
def _parse_dataplugins(raw):
"""The applied per-plugin revisions, as a {plugin: revision} map.
Bad or absent JSON becomes an empty map rather than an error: a manifest
that cannot say what the plugin chains were at is exactly the legacy case
the restore gate already knows how to treat conservatively, and failing the
backup here would lose the set over a stamp.
"""
if not raw:
return {}
try:
value = json.loads(raw)
except ValueError:
return {}
if not isinstance(value, dict):
return {}
return {str(k): str(v) for k, v in value.items() if k and v}
def _collect_schema(migrations_dir, plugins_dir):
"""Core heads plus each plugin's chain heads, keyed for compatibility checks."""
schema = {'core': _heads_for(migrations_dir), 'plugins': {}}
@@ -112,7 +86,6 @@ def write_manifest(args):
'imagetag': args.imagetag,
'schemarev': _collect_schema(args.migrations, args.plugins),
'datarev': args.datarev or '',
'dataplugins': _parse_dataplugins(args.dataplugins),
'ischeckpassed': False,
}
with open(args.out, 'w', encoding='utf-8') as handle:
@@ -130,8 +103,6 @@ def amend_manifest(args):
manifest['ischeckpassed'] = args.ischeckpassed.lower() == 'true'
if args.datarev is not None:
manifest['datarev'] = args.datarev
if args.dataplugins is not None:
manifest['dataplugins'] = _parse_dataplugins(args.dataplugins)
with open(args.amend, 'w', encoding='utf-8') as handle:
json.dump(manifest, handle, indent=2, sort_keys=True)
return 0
@@ -147,8 +118,6 @@ def main(argv=None):
parser.add_argument('--migrations', default='/app/migrations')
parser.add_argument('--plugins', default='/app/plugins')
parser.add_argument('--datarev')
parser.add_argument('--dataplugins',
help='JSON object of {plugin: applied revision}')
parser.add_argument('--ischeckpassed')
args = parser.parse_args(argv)

171
scripts/backup/restore.sh
View File

@@ -21,22 +21,8 @@ set -euo pipefail
BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}"
SETNAME="${1:-}"
# Overridable so the gate can be exercised outside a container against throwaway
# migration trees. In the image these are exactly the mounted paths, so the
# shipped behaviour is unchanged; test-restore-datarev.sh drives THIS script
# rather than a copy of its logic, which is the only way the test can notice
# this script drifting.
BACKUP_LIB_DIR="${BACKUP_LIB_DIR:-/opt/backup}"
APP_MIGRATIONS_DIR="${APP_MIGRATIONS_DIR:-/app/migrations}"
APP_PLUGINS_DIR="${APP_PLUGINS_DIR:-/app/plugins}"
APP_INSTANCE_DIR="${APP_INSTANCE_DIR:-/app/instance}"
# When set, stop after the compatibility gate and report the verdict as the exit
# code, touching nothing. This is what makes the gate testable.
RESTORE_CHECK_ONLY="${RESTORE_CHECK_ONLY:-}"
# shellcheck source=scripts/backup/status.sh
. "${BACKUP_LIB_DIR}/status.sh"
. /opt/backup/status.sh
log() { echo "[restore] $*"; }
@@ -75,7 +61,7 @@ DBDUMP="${SETDIR}/db.sql.gz"
if [ ! -f "${DBDUMP}" ]; then
log "local dump not found; restoring set from restic"
command -v restic >/dev/null 2>&1 || die "restic not available and no local dump"
restic restore latest --tag "${SETNAME}" --target / --include "${SETDIR}" \
restic restore latest --target / --include "${SETDIR}" \
|| die "restic restore of ${SETNAME} failed"
fi
[ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}"
@@ -105,173 +91,30 @@ if [ -z "${SET_DATAREV}" ]; then
die "cannot determine the set's data revision - refusing an unverified restore"
fi
# Does the running code know this revision at all? If a migration file with that
# revision id exists in the core chain, the code is at-or-past it (the head is
# the tip of the same chain), so a post-load upgrade can carry the data forward.
# If it is absent, the dump is NEWER than the code - refuse.
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" "${APP_MIGRATIONS_DIR}/versions" 2>/dev/null; then
if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then
log "set datarev : ${SET_DATAREV}"
die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring"
fi
log "data revision ${SET_DATAREV} is known to this code - restore may proceed"
# The same gate for every PLUGIN chain, because the core chain alone cannot
# answer this. Plugin schemas move independently of core: roll the code back to
# an older plugin revision and the core head can be identical while the dump
# carries plugin tables the code has never heard of. Gating on core only let
# that pass, load the dump, and then fail in 'flask plugin upgrade-all' with
# "Can't locate revision" and the data already replaced - a failed restore that
# has destroyed what it was restoring over.
#
# A manifest with no dataplugins is a legacy set taken before this was stamped.
# Its plugin revisions are unknowable, so it is allowed through on the core gate
# alone (the behaviour before this check existed) rather than being made
# unrestorable retroactively.
SET_DATAPLUGINS="$(python3 -c \
'import json,sys
try:
data = json.load(open(sys.argv[1])).get("dataplugins") or {}
except Exception:
data = {}
for name, rev in sorted(data.items()):
print(name + " " + rev)' \
"${SETDIR}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${SET_DATAPLUGINS}" ]; then
log "manifest stamps no plugin revisions (legacy set) - core gate only"
else
while read -r plugin rev; do
[ -n "${plugin}" ] || continue
chain="${APP_PLUGINS_DIR}/${plugin}/migrations/versions"
if [ ! -d "${chain}" ]; then
die "the backup carries data for plugin '${plugin}', which this build does not have - deploy a build that includes it before restoring"
fi
if ! grep -rqE "^revision = ['\"]${rev}['\"]" "${chain}" 2>/dev/null; then
log "plugin ${plugin} data revision: ${rev}"
die "plugin '${plugin}' data is at a revision this code does not have - deploy code at or past '${rev}' before restoring"
fi
log "plugin ${plugin} at ${rev} - known to this code"
done <<PLUGINREVS
${SET_DATAPLUGINS}
PLUGINREVS
fi
# Last gate: which schema does this dump target? A dump taken with --databases
# carries CREATE DATABASE and USE <name>, pinning it to the schema it came from.
# Loaded into a site whose MYSQL_DATABASE differs, the rows land in the dump's
# schema, the live database is untouched, and the restore reports success. New
# sets are dumped without it and load into whatever schema is named here; older
# sets are still bound, so refuse rather than restore beside the live data.
# Checked HERE, with the other gates, so it costs nothing and stops before the
# safety copy rather than after it.
DUMP_SCHEMA="$(gunzip -c "${DBDUMP}" \
| grep -m1 -oE '^USE `[^`]+`' \
| sed -E 's/^USE `(.*)`/\1/' || true)"
if [ -n "${DUMP_SCHEMA}" ]; then
if [ "${DUMP_SCHEMA}" != "${MYSQL_DATABASE}" ]; then
die "this set is bound to schema '${DUMP_SCHEMA}' but this site's database is '${MYSQL_DATABASE}' - restoring it would load the data beside the live database and report success"
fi
log "legacy schema-bound dump for '${DUMP_SCHEMA}' - matches this site"
fi
if [ -n "${RESTORE_CHECK_ONLY}" ]; then
log "check-only: every gate passed, stopping before anything destructive"
RESTORE_LOADED=false
trap - EXIT
exit 0
fi
# Safety set FIRST. Everything below this point is destructive and none of it
# is transactional: the dump carries DROP TABLE per table, and the instance tree
# is replaced wholesale. Without a copy taken here, a restore of the wrong set -
# or a failure part-way through the load - leaves nothing to go back to. This is
# a local copy on the same volume, so it is cheap and it is NOT a substitute for
# a real backup; it exists to make THIS operation reversible.
SAFETY_STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
SAFETY_DIR="${BACKUPS_ROOT}/pre-restore-${SAFETY_STAMP}"
status_write restore safety "${SETNAME}" "Copying the current state aside"
log "taking a pre-restore safety copy into ${SAFETY_DIR}"
mkdir -p "${SAFETY_DIR}" || die "could not create ${SAFETY_DIR}"
mysqldump \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--single-transaction \
--routines \
--triggers \
--no-tablespaces \
"${MYSQL_DATABASE}" \
| gzip -c > "${SAFETY_DIR}/db.sql.gz" \
|| die "pre-restore dump failed - refusing to touch the live database"
if [ -d /app/instance ]; then
tar -czf "${SAFETY_DIR}/instance.tar.gz" -C /app instance \
|| die "pre-restore instance copy failed - refusing to touch the live files"
fi
# Named so an operator finding this directory months later knows what it is and
# how to use it, without reading this script.
cat > "${SAFETY_DIR}/README.txt" <<SAFETYNOTE
Automatic pre-restore copy of the state this site was in immediately before
restoring set '${SETNAME}' at ${SAFETY_STAMP}.
db.sql.gz mysqldump of ${MYSQL_DATABASE} as it was
instance.tar.gz /app/instance as it was (absent if there was no such tree)
To go back to this state, load the dump the same way a restore does:
gunzip -c db.sql.gz | mysql --host=... --user=... --password=... \
--database=${MYSQL_DATABASE}
tar -xzf instance.tar.gz -C /app
This is a local copy on the backups volume, taken for reversibility of one
restore. It is not a backup set: it has no manifest and no restic snapshot, and
nothing prunes it.
SAFETYNOTE
log "safety copy written ($(du -sh "${SAFETY_DIR}" | cut -f1))"
# Files first, then DB, so the two land together.
if command -v restic >/dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then
status_write restore files "${SETNAME}" "Restoring instance files"
# Scoped to THIS set's snapshot. A bare 'latest' restores whatever was
# snapshotted most recently, so restoring anything other than the newest set
# paired that set's DATABASE with the NEWEST snapshot's uploads and
# plugins.json - a mismatch nothing downstream could detect.
log "restoring instance files from the ${SETNAME} snapshot"
STAGE="$(mktemp -d)"
restic restore latest --tag "${SETNAME}" --target "${STAGE}" \
--include /app/instance \
|| die "restic restore of instance files for ${SETNAME} failed"
# Replace the CONTENTS of the instance tree rather than restoring over it,
# so files created after the snapshot are gone afterwards - a restore has to
# put the site back to the snapshot, not merge into it. Done as an explicit
# staged swap rather than 'restic restore --delete --target /', because that
# aims a delete pass at the container root and the blast radius of getting
# its filter wrong is the whole filesystem. /app/instance is a mount, so the
# directory itself has to survive; only what is inside it is replaced.
if [ -d "${STAGE}/app/instance" ]; then
find /app/instance -mindepth 1 -delete \
|| die "could not clear the instance tree before restoring it"
cp -a "${STAGE}/app/instance/." /app/instance/ \
|| die "could not lay down the restored instance tree"
else
log "WARN snapshot for ${SETNAME} carried no /app/instance - leaving files alone"
fi
rm -rf "${STAGE}"
log "restoring instance files from restic"
restic restore latest --target / --include /app/instance \
|| die "restic restore of instance files failed"
fi
status_write restore database "${SETNAME}" "Loading database dump"
log "loading database dump into ${MYSQL_DATABASE}"
log "loading database dump"
gunzip -c "${DBDUMP}" | mysql \
--host="${MYSQL_HOST}" \
--user="${MYSQL_USER}" \
--password="${MYSQL_PASSWORD}" \
--database="${MYSQL_DATABASE}" \
|| die "database load failed"
# Hand off to the api container for the schema upgrade. The dump may sit at an

16
scripts/backup/simulate-restore-results.txt Normal file
View File

@@ -0,0 +1,16 @@
ShopDB restore simulation results (FOR DEVELOPER USE ONLY)
run (UTC) : 2026-09-04T09:00:19Z
marker : simrestore-20260904T085538Z
backup set : wilm-2026-09-04T084716Z
tables checked : 110
cascade edge : computers->assets delete_rule=CASCADE (not exercised - live asset delete not permitted)
Expected, not-a-failure drift (excluded from strict equality):
- apitokens: app rewrites token last-used; restore is correct at backup time.
- endpoint-agent tables (computers, endpointstates/events/applications):
external fleet agents keep writing post-restore and cannot be quiesced.
- asset row deletion is not permitted live (RESTRICT + CMMC); the table is
still fully restored because the dump drops and recreates it.
RESULT: PASS - all 110 tables restored to backup-time state
(6 volatile table(s) excluded from strict equality)

157
scripts/backup/test-restore-datarev.sh
View File

@@ -79,62 +79,43 @@ build_dump() {
} | gzip -c > "${out}"
}
# Build a throwaway plugin chain, shaped like plugins/<name>/migrations/versions.
build_plugin_chain() {
root="$1"; name="$2"; rev="$3"
versions="${root}/${name}/migrations/versions"
mkdir -p "${versions}"
printf "revision = '%s'\ndown_revision = None\n" \
"${rev}" > "${versions}/${rev}.py"
# Mirror of the restore.sh predicate: does the code chain know this revision?
# Kept identical to restore.sh so this test tracks the shipped decision.
code_knows_revision() {
versions_dir="$1"; datarev="$2"
grep -rqE "^revision = ['\"]${datarev}['\"]" "${versions_dir}" 2>/dev/null
}
# Drive the REAL restore.sh gate. This is the point of the rewrite: the test
# used to carry its own copy of the predicate, so it stayed green no matter how
# far restore.sh drifted from it. RESTORE_CHECK_ONLY stops the script right
# after the gate, so the verdict here IS the shipped verdict.
run_gate() {
backups_root="$1"; setname="$2"; chain="$3"; plugins="$4"
RESTORE_CHECK_ONLY=1 \
SHOPDB_BACKUPS_ROOT="${backups_root}" \
BACKUP_LIB_DIR="${SCRIPT_DIR}" \
APP_MIGRATIONS_DIR="${chain}" \
APP_PLUGINS_DIR="${plugins}" \
bash "${RESTORE_SH}" "${setname}" >"${WORKDIR}/gate.log" 2>&1
# Mirror of the restore.sh datarev extraction: manifest field first, else parse
# the dump.
set_datarev() {
setdir="$1"; dbdump="$2"
value="$(${PY} -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
"${setdir}/manifest.json" 2>/dev/null || echo "")"
if [ -z "${value}" ]; then
value="$(gunzip -c "${dbdump}" \
| grep -A2 -i 'INSERT INTO `alembic_version`' \
| grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)"
fi
printf '%s' "${value}"
}
CHAIN="${WORKDIR}/migrations"
build_chain "${CHAIN}"
PLUGINS="${WORKDIR}/plugins"
build_plugin_chain "${PLUGINS}" 'endpoints' 'endpoints0002known'
ROOT="${WORKDIR}/backups"
mkdir -p "${ROOT}"
VERSIONS="${CHAIN}/versions"
# Lay out one set the way a real backup does, under the backups root.
make_set() {
setname="$1"; datarev="$2"; dumprev="$3"; dataplugins="$4"
setdir="${ROOT}/${setname}"
mkdir -p "${setdir}"
if [ "${datarev}" = "__legacy__" ]; then
printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \
> "${setdir}/manifest.json"
else
${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \
--migrations "${CHAIN}" --plugins "${PLUGINS}" \
--datarev "${datarev}" --dataplugins "${dataplugins}" \
--takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
fi
build_dump "${setdir}/db.sql.gz" "${dumprev}"
printf '%s' "${setdir}"
}
# --- 1. manifest.py records datarev separately from the code head ----------
make_set 'set-behind' '0002_mid' '0002_mid' '{}' >/dev/null
# --- 1. manifest.py records datarev distinct from the code head -------------
SET1="${WORKDIR}/set1"; mkdir -p "${SET1}"
${PY} "${MANIFEST_PY}" --out "${SET1}/manifest.json" \
--migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \
--datarev '0002_mid' --takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
recorded="$(${PY} -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \
"${ROOT}/set-behind/manifest.json" 2>/dev/null)"
"${SET1}/manifest.json" 2>/dev/null)"
head_stamp="$(${PY} -c \
'import json,sys;print(json.load(open(sys.argv[1])).get("schemarev",{}).get("core",[]))' \
"${ROOT}/set-behind/manifest.json" 2>/dev/null)"
"${SET1}/manifest.json" 2>/dev/null)"
if [ "${recorded}" = '0002_mid' ]; then
pass "manifest.py records datarev (${recorded}) separate from code head ${head_stamp}"
else
@@ -142,26 +123,38 @@ else
fi
# --- 2. datarev the code KNOWS (behind head) is allowed --------------------
if run_gate "${ROOT}" 'set-behind' "${CHAIN}" "${PLUGINS}"; then
pass "data behind code (0002_mid) is known and allowed to restore"
DBDUMP1="${SET1}/db.sql.gz"; build_dump "${DBDUMP1}" '0002_mid'
rev="$(set_datarev "${SET1}" "${DBDUMP1}")"
if [ "${rev}" = '0002_mid' ] && code_knows_revision "${VERSIONS}" "${rev}"; then
pass "data behind code (${rev}) is known and allowed to restore"
else
fail "data behind code was refused: $(tail -1 "${WORKDIR}/gate.log")"
fail "data behind code (${rev}) was not accepted"
fi
# --- 3. datarev the code does NOT know (newer than build) is refused -------
make_set 'set-ahead' '0004_future' '0004_future' '{}' >/dev/null
if run_gate "${ROOT}" 'set-ahead' "${CHAIN}" "${PLUGINS}"; then
fail "data ahead of code (0004_future) was NOT refused"
SET2="${WORKDIR}/set2"; mkdir -p "${SET2}"
${PY} "${MANIFEST_PY}" --out "${SET2}/manifest.json" \
--migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \
--datarev '0004_future' --takenat '2024-02-01T00:00:00Z' >/dev/null 2>&1
DBDUMP2="${SET2}/db.sql.gz"; build_dump "${DBDUMP2}" '0004_future'
rev="$(set_datarev "${SET2}" "${DBDUMP2}")"
if [ "${rev}" = '0004_future' ] && ! code_knows_revision "${VERSIONS}" "${rev}"; then
pass "data ahead of code (${rev}) is unknown and correctly refused"
else
pass "data ahead of code (0004_future) is unknown and correctly refused"
fail "data ahead of code (${rev}) was not refused"
fi
# --- 4. legacy manifest (no datarev) falls back to the dump ----------------
make_set 'set-legacy' '__legacy__' '0001_baseline' '{}' >/dev/null
if run_gate "${ROOT}" 'set-legacy' "${CHAIN}" "${PLUGINS}"; then
pass "legacy set with no datarev falls back to the dump revision"
SET3="${WORKDIR}/set3"; mkdir -p "${SET3}"
# A pre-datarev manifest: schemarev present, datarev absent/empty.
printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \
> "${SET3}/manifest.json"
DBDUMP3="${SET3}/db.sql.gz"; build_dump "${DBDUMP3}" '0001_baseline'
rev="$(set_datarev "${SET3}" "${DBDUMP3}")"
if [ "${rev}" = '0001_baseline' ] && code_knows_revision "${VERSIONS}" "${rev}"; then
pass "legacy set with no datarev falls back to dump revision (${rev})"
else
fail "legacy set fallback failed: $(tail -1 "${WORKDIR}/gate.log")"
fail "legacy set fallback failed (got '${rev}')"
fi
# --- 5. restore.sh hands off to the api instead of self-completing ---------
@@ -173,58 +166,6 @@ else
fail "restore.sh does not hand off the upgrade to the api"
fi
# --- 6. a PLUGIN chain the code does not know is refused -------------------
# The core head is identical in both of these; only the plugin revision differs.
# That is the whole failure: a core-only gate cannot tell them apart.
make_set 'set-plugin-known' '0002_mid' '0002_mid' '{"endpoints":"endpoints0002known"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-known' "${CHAIN}" "${PLUGINS}"; then
pass "a set whose plugin revision the code knows is allowed"
else
fail "known plugin revision was refused: $(tail -1 "${WORKDIR}/gate.log")"
fi
make_set 'set-plugin-future' '0002_mid' '0002_mid' '{"endpoints":"endpoints0099future"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-future' "${CHAIN}" "${PLUGINS}"; then
fail "plugin data ahead of code was NOT refused (core-only gate regression)"
else
pass "plugin data ahead of code is refused even though the core head matches"
fi
# --- 7. a plugin the build does not have at all is refused -----------------
make_set 'set-plugin-missing' '0002_mid' '0002_mid' '{"nosuchplugin":"whatever0001"}' >/dev/null
if run_gate "${ROOT}" 'set-plugin-missing' "${CHAIN}" "${PLUGINS}"; then
fail "a set carrying data for an absent plugin was NOT refused"
else
pass "a set carrying data for a plugin this build lacks is refused"
fi
# --- 8. a dump bound to another schema is refused --------------------------
# --databases bakes 'USE <name>' in, which silently loaded the rows beside the
# live database and reported success.
setdir="${ROOT}/set-otherschema"
mkdir -p "${setdir}"
${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \
--migrations "${CHAIN}" --plugins "${PLUGINS}" \
--datarev '0002_mid' --dataplugins '{}' \
--takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1
{
echo 'CREATE DATABASE `someothersite`;'
echo 'USE `someothersite`;'
echo 'INSERT INTO `alembic_version` VALUES'
echo "('0002_mid');"
} | gzip -c > "${setdir}/db.sql.gz"
# Now a gate, so check-only reaches it without needing a database.
out="$(RESTORE_CHECK_ONLY=1 SHOPDB_BACKUPS_ROOT="${ROOT}" \
BACKUP_LIB_DIR="${SCRIPT_DIR}" \
APP_MIGRATIONS_DIR="${CHAIN}" APP_PLUGINS_DIR="${PLUGINS}" \
MYSQL_DATABASE='shopdb_flask' \
bash "${RESTORE_SH}" 'set-otherschema' 2>&1)" || true
if printf '%s' "${out}" | grep -q "bound to schema 'someothersite'"; then
pass "a dump bound to another site's schema is refused"
else
fail "a schema-bound dump was not caught (got: $(printf '%s' "${out}" | tail -1))"
fi
echo "=================================================="
echo "[test] datarev gate: ${passes} passed, ${fails} failed"
echo "=================================================="

0
scripts/backup/watch.sh.tmp Normal file
View File

0
scripts/site_imports/map_from_grid.py Normal file
View File

287
shopdb/core/api/shopdbbackups.py
View File

@@ -17,8 +17,6 @@ load rows that reference a column the live schema has not applied yet.
import json
import os
import threading
import time
import subprocess
from datetime import datetime, timezone
@@ -87,30 +85,6 @@ def _backups_root():
return os.environ.get(BACKUPS_ROOT_ENV, DEFAULT_BACKUPS_ROOT)
def _backups_configured():
"""Whether this deployment has a backup store behind the feature.
The overlay is a compose profile: a third container writes the sets onto a
shared volume, and the api container only ever reads them and drops a
sentinel. A Windows/IIS install has no such container, and the default root
resolves there to a drive-letter path nothing creates - so every button
either 500s on PermissionError or reports 'queued' for a run that will
never happen. Answer the caller honestly instead.
"""
return os.path.isdir(_backups_root())
def _unavailable_response():
# 404 + NOT_FOUND is the established shape for "this feature is not
# enabled here" (the configuration-policy routes answer the same way), and
# the UI already tells a 404 apart from a real failure.
return error_response(
ErrorCodes.NOT_FOUND,
'Site Backups is not configured on this deployment. It needs the '
'backup overlay and a readable backups root (%s).' % BACKUPS_ROOT_ENV,
http_code=404)
def _read_json(path):
"""Load a JSON file, or None when it is absent or unreadable."""
try:
@@ -184,106 +158,6 @@ def _append_history(kind, outcome, setname, message):
pass
# The post-restore upgrade is claimed by renaming the sentinel. rename(2) is
# atomic, so exactly one claimant wins even with several gunicorn workers and a
# browser poll all racing - which matters because the loser would otherwise run
# Alembic concurrently against the same database.
UPGRADE_SENTINEL = '.upgrade-requested'
UPGRADE_CLAIM = '.upgrade-inprogress'
UPGRADE_FAILED = '.upgrade-failed'
def _claim_upgrade():
"""Take exclusive ownership of a pending upgrade, or return None.
Returns the set name on success. Losing the race is not an error: it means
another worker is already finalizing the same restore.
"""
root = _backups_root()
sentinel = os.path.join(root, UPGRADE_SENTINEL)
claim = os.path.join(root, UPGRADE_CLAIM)
try:
os.rename(sentinel, claim)
except OSError:
return None
try:
with open(claim, 'r', encoding='utf-8') as handle:
return handle.read().strip() or 'unknown'
except OSError:
return 'unknown'
def _finalize_claimed(setname):
"""Run the upgrade for a claimed restore and record how it ended.
A failure parks the claim as .upgrade-failed rather than putting it back:
re-arming it here would retry a migration that just failed, every poll,
forever. The operator re-arms it explicitly through the finalize endpoint.
"""
root = _backups_root()
claim = os.path.join(root, UPGRADE_CLAIM)
_write_status('restore', 'upgrading', setname,
'Upgrading restored schema to code')
try:
_run_schema_upgrade()
except Exception as error: # noqa: BLE001 - report any upgrade failure
message = 'Schema upgrade failed: ' + str(error)
try:
os.replace(claim, os.path.join(root, UPGRADE_FAILED))
except OSError:
pass
_write_status('restore', 'failed', setname, message)
_append_history('restore', 'failed', setname, message)
current_app.logger.error('Post-restore schema upgrade failed: %s', error)
return False
try:
os.remove(claim)
except OSError:
pass
_write_status('restore', 'done', setname, 'Restore complete')
_append_history('restore', 'success', setname, 'Restore complete')
current_app.logger.info('Restore finalized: %s', setname)
return True
def start_restore_finalizer(app):
"""Watch for a restore awaiting its schema upgrade, and run it.
WHY THIS EXISTS: the restore itself runs in the Flask-less backup sidecar,
which loads the dump and drops a sentinel because it cannot run
'flask db upgrade'. Until this watcher, the ONLY thing that acted on that
sentinel was the browser poll loop of the modal that queued the restore -
so closing the tab, or letting a token expire, during the minutes a load
takes left the data at the old revision under new code, with every query on
a newer column failing and nothing retrying.
Runs in the api container because that is where the app context and the
engine live, and out of band rather than in a request because Alembic can
outrun the gunicorn worker timeout and get killed mid-chain.
"""
interval = max(1, int(os.environ.get('SHOPDB_FINALIZE_POLL_SECONDS', '10')))
def _loop():
while True:
time.sleep(interval)
try:
with app.app_context():
if not _backups_configured():
continue
setname = _claim_upgrade()
if setname:
_finalize_claimed(setname)
except Exception: # noqa: BLE001 - a watcher must not die
try:
app.logger.exception('Restore finalizer pass failed')
except Exception: # noqa: BLE001
pass
thread = threading.Thread(target=_loop, name='restore-finalizer', daemon=True)
thread.start()
return thread
def _run_schema_upgrade():
"""Bring the just-restored database up to the running code, in-process.
@@ -383,63 +257,6 @@ def _known_core_revisions():
return set()
def _known_plugin_revisions():
"""Every revision each plugin chain knows, as {plugin: {revisions}}.
Read by text scan rather than through Alembic. Each plugin chain has its own
env and version table, so loading them properly would mean building 17
ScriptDirectories on a request that only needs to answer "is this id in
there" - and manifest.py already reads the same trees the same way, so the
two agree by construction.
"""
root = os.path.dirname(current_app.root_path)
plugins_dir = os.path.join(root, 'plugins')
known = {}
if not os.path.isdir(plugins_dir):
return known
for name in sorted(os.listdir(plugins_dir)):
versions = os.path.join(plugins_dir, name, 'migrations', 'versions')
if not os.path.isdir(versions):
continue
revisions = set()
for filename in os.listdir(versions):
if not filename.endswith('.py'):
continue
try:
with open(os.path.join(versions, filename), 'r',
encoding='utf-8') as handle:
for line in handle:
stripped = line.strip()
if stripped.startswith('revision ='):
revisions.add(
stripped.split('=', 1)[1].strip().strip('"\''))
break
except OSError:
continue
if revisions:
known[name] = revisions
return known
def _plugin_data_is_known(manifest):
"""Whether this build knows every plugin revision the set's data sits at.
A set with no `dataplugins` predates the stamp; its plugin revisions are
unknowable, so it keeps the old core-only verdict rather than being marked
unrestorable retroactively.
"""
stamped = manifest.get('dataplugins') or {}
if not isinstance(stamped, dict) or not stamped:
return True
known = _known_plugin_revisions()
for name, revision in stamped.items():
if not revision:
continue
if revision not in known.get(name, ()):
return False
return True
def _is_compatible(manifest, current):
"""Decide whether a set restores onto THIS build.
@@ -452,34 +269,17 @@ def _is_compatible(manifest, current):
revision, the dump is from a NEWER schema than this build and would 500 on
restored rows, so we block until the code is upgraded.
The same question is asked of every PLUGIN chain, because plugin schemas
move independently: code rolled back to an older plugin revision can sit at
an identical core head while the dump carries plugin tables it cannot
migrate.
Legacy sets predate `datarev`; fall back to the old head-subset check on
`schemarev` so they stay listable and restorable as before.
"""
if not manifest:
return False
# A manifest that CARRIES the key but left it empty is not a legacy set: it
# is a set whose data revision could not be read when it was taken. Falling
# through to the code-to-code check would badge it restorable on no
# evidence, using exactly the comparison the datarev work replaced. Only a
# manifest with no datarev key at all predates the stamp.
if 'datarev' in manifest and not manifest.get('datarev'):
return False
datarev = manifest.get('datarev')
if datarev:
known = _known_core_revisions()
if not known:
return False
# Core AND every plugin chain. Core alone said yes to a set whose plugin
# data the code had never heard of, which loads and then dies during the
# plugin upgrade with the old data already gone.
return datarev in known and _plugin_data_is_known(manifest)
return datarev in known
setrevs = set(manifest.get('schemarev', {}).get('core', []))
currevs = set(current.get('core', []))
if not setrevs or not currevs:
@@ -539,7 +339,6 @@ def list_backups():
'retention': RETENTION,
'lastgoodat': (laststgood or {}).get('takenat'),
'count': len(sets),
'configured': _backups_configured(),
}
return success_response(data=sets, meta=meta)
@@ -574,9 +373,6 @@ def trigger_backup():
up. Returning isrunning=false with a clear message keeps the UI honest
rather than pretending the run started inside this process.
"""
if not _backups_configured():
return _unavailable_response()
root = _backups_root()
try:
os.makedirs(root, exist_ok=True)
@@ -605,9 +401,6 @@ def restore_backup(snapshotid):
The actual restore is performed by scripts/backup/restore.sh in the backup
container; here we validate and record intent.
"""
if not _backups_configured():
return _unavailable_response()
root = _backups_root()
setdir = os.path.join(root, os.path.basename(snapshotid))
if not os.path.isdir(setdir):
@@ -645,53 +438,52 @@ def restore_backup(snapshotid):
@shopdbbackups_bp.route('/finalize-restore', methods=['POST'])
@require_role('admin')
def finalize_restore():
"""Make sure a restore awaiting its schema upgrade is queued to be finalized.
"""Run the post-restore schema upgrade in the app context.
This used to RUN the upgrade inline, which made two things true that should
not have been: a restore only completed while a browser tab stayed open on
it, and the upgrade ran inside a request, where Alembic can outrun the
gunicorn worker timeout and be killed part-way through the chain, leaving a
half-applied revision the next attempt trips over.
The restore itself runs in the Flask-less backup sidecar, which loads the
dump and then drops an .upgrade-requested sentinel because it cannot run
'flask db upgrade' itself. This endpoint is the api-side half: it brings the
just-restored DATA up to the running CODE with the exact idempotent commands
the deploy path uses, then stamps the final 'done'. Without it a restored
dump that sits at an older revision leaves every newer-migration table
throwing 1054 Unknown column.
The work now belongs to the finalizer watcher in this container
(start_restore_finalizer). This endpoint only ensures something is pending
for it to claim, so it is safe to call repeatedly, and it is how an operator
retries an upgrade that failed.
Idempotent and safe to call when nothing is pending: if there is no
.upgrade-requested sentinel it reports there was nothing to finalize.
"""
if not _backups_configured():
return _unavailable_response()
root = _backups_root()
sentinel = os.path.join(root, UPGRADE_SENTINEL)
claim = os.path.join(root, UPGRADE_CLAIM)
failed = os.path.join(root, UPGRADE_FAILED)
if os.path.exists(claim):
sentinel = os.path.join(root, '.upgrade-requested')
try:
with open(sentinel, 'r', encoding='utf-8') as handle:
setname = handle.read().strip()
except OSError:
return success_response(
data={'queued': True, 'inprogress': True},
message='The schema upgrade is already running.')
data={'finalized': False},
message='No restore is awaiting schema upgrade.')
if os.path.exists(sentinel):
return success_response(
data={'queued': True, 'inprogress': False},
message='The schema upgrade is queued and will run shortly.')
# A previous attempt failed and parked itself. Re-arming is the retry.
if os.path.exists(failed):
try:
os.replace(failed, sentinel)
except OSError as error:
return error_response(ErrorCodes.INTERNAL_ERROR,
'Could not re-queue the schema upgrade: ' + str(error),
http_code=500)
AuditLog.log('restore-finalize-retried', 'Backup')
return success_response(
data={'queued': True, 'inprogress': False, 'retried': True},
message='The failed schema upgrade has been queued to run again.')
_write_status('restore', 'upgrading', setname,
'Upgrading restored schema to code')
try:
_run_schema_upgrade()
except Exception as error: # noqa: BLE001 - report any upgrade failure
_write_status('restore', 'failed', setname,
'Schema upgrade failed: ' + str(error))
_append_history('restore', 'failed', setname,
'Schema upgrade failed: ' + str(error))
return error_response(ErrorCodes.INTERNAL_ERROR,
'Post-restore schema upgrade failed: ' + str(error),
http_code=500)
try:
os.remove(sentinel)
except OSError:
pass
_write_status('restore', 'done', setname, 'Restore complete')
_append_history('restore', 'success', setname, 'Restore complete')
AuditLog.log('restore-finalized', 'Backup', entityname=setname)
return success_response(
data={'queued': False},
message='No restore is awaiting schema upgrade.')
data={'finalized': True, 'snapshotid': setname},
message='Restore finalized: schema upgraded to the running code.')
@shopdbbackups_bp.route('/status', methods=['GET'])
@@ -704,7 +496,6 @@ def get_status():
value is None when that kind of run has never been requested.
"""
return success_response(data={
'configured': _backups_configured(),
'backup': _read_status('backup'),
'restore': _read_status('restore'),
})

2
tests/test_docs_publishable.py
View File

@@ -104,7 +104,7 @@ def test_there_are_docs_to_check():
docs/*.md prose is wiki-published and MUST stay untracked. The gates decide
'is this a published checkout?' by the ABSENCE of docs/*.md, so one tracked
prose file makes CI look internal while carrying no prose, and takes down
backend AND gates together. the DEV-BUILD-TEST wiki page and the PLUGIN-HOOKS wiki page
backend AND gates together. docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md
did exactly this and held the branch red for eight runs; both were untracked
(kept on disk) and .gitignore now ignores docs/*.md wholesale.

0
tests/test_plugins/test_endpoints_siteagent_build.py Normal file
View File