diff --git a/.gitignore b/.gitignore index 31591d72..20e29de4 100644 --- a/.gitignore +++ b/.gitignore @@ -91,20 +91,24 @@ _transfer/ # Local developer scratch notes and chat handoffs - never committed devnotes/ -# docs/ is generated/published to the wiki, not tracked in the product repo. -# A single tracked file here makes docs/ exist in the CI checkout, which -# defeats the intentional skip in the docs publish/version tests. Keep this -# local-only guide out of version control so those tests skip as designed. -docs/CSV-IMPORT-GUIDE.md +# docs/ prose is published to the wiki, not tracked in the product repo. The +# docs gates detect a published checkout by the ABSENCE of docs/*.md, so a +# single tracked prose file here makes CI look like an internal checkout while +# carrying almost no prose: test_there_are_docs_to_check then fails asserting +# >20 files, test_the_map_itself_is_current fails on a generated PROJECT-MAP +# that is correctly untracked, and check-gates-ran.sh fails on the resulting +# skip. That happened - DEV-BUILD-TEST.md and PLUGIN-HOOKS.md were committed in +# unrelated changes and broke the branch for days. Ignore the whole pattern +# rather than naming files one at a time, so the next authored doc cannot +# reintroduce it. +# +# Scoped to the top level on purpose: docs/notes/ prose stays tracked, and the +# gates only glob docs/*.md. docs/openapi.json is tracked (the app serves it) +# and is unaffected by a *.md rule. +docs/*.md -# Local-only authored docs (executive summaries, workflow/security writeups, -# planner exports, draft decks and diagrams). Same reasoning as above: keep -# personal docs out of version control. docs/openapi.json stays tracked and -# is intentionally NOT ignored here. -docs/EXECUTIVE-SUMMARY.md -docs/DESIGN-DECISIONS-agent.md -docs/OPERATOR-WORKFLOW.md -docs/SECURITY-RISK-ASSESSMENT.md +# Local-only authored docs (planner exports, draft decks and diagrams). Same +# reasoning as above: keep personal docs out of version control. docs/ShopDB-Planner-Tasks.csv docs/DRAFT--ShopDB-CSV-Import-Guide.pptx docs/diagrams/ diff --git a/Dockerfile b/Dockerfile index 7fd22a02..7d7c4242 100644 --- a/Dockerfile +++ b/Dockerfile @@ -109,8 +109,13 @@ RUN apt-get update \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt ./ +# gunicorn and gevent are container-only: requirements.txt is a universal, +# hash-checked lockfile that also serves the Windows/IIS install, where waitress +# is the server and neither of these is used. gevent is the worker class (see +# the CMD below) because the endpoint long-poll holds a connection per agent, +# which a thread pool cannot absorb at fleet scale. RUN pip install --no-cache-dir -r requirements.txt \ - && pip install --no-cache-dir gunicorn + && pip install --no-cache-dir gunicorn gevent COPY shopdb/ ./shopdb/ COPY plugins/ ./plugins/ @@ -134,21 +139,27 @@ USER shopdb EXPOSE 5001 +# Probe the public health route, not the login endpoint. Probing POST +# /api/auth/login logged a 400 every 30s and ran the credential path from a +# health check. The route answers 200 even when the database is degraded, so the +# body's status field is what decides health. Compose files that terminate TLS +# override this with an https probe; plain http is correct for the pull and +# airgap stacks, which inherit this unchanged. HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ - CMD curl --silent http://localhost:5001/api/auth/login -X POST \ - -H "Content-Type: application/json" -d '{}' \ - | grep -q "VALIDATION_ERROR" || exit 1 + CMD curl --silent http://localhost:5001/api/dashboard/health \ + | grep -q '"status":"ok"' || exit 1 -# Threaded workers, matching docker-compose.yml - see the reasoning there. A +# Async workers, matching docker-compose.yml - see the reasoning there. A # compose file overrides this CMD, but an image run without one (a plain # `docker run`, a site's own orchestrator) would otherwise still get the # blocking sync default and the same starvation. CMD ["gunicorn", \ "--bind", "0.0.0.0:5001", \ - "--worker-class", "gthread", \ + "--worker-class", "gevent", \ "--workers", "4", \ - "--threads", "8", \ - "--timeout", "30", \ + "--worker-connections", "2000", \ + "--timeout", "60", \ "--access-logfile", "-", \ + "--access-logformat", "%(h)s %(l)s %(u)s %(t)s \"%(r)s\" %(s)s %(b)s %(D)sus \"%(f)s\" \"%(a)s\"", \ "--error-logfile", "-", \ "wsgi:app"] diff --git a/agent/ShopDbEndpointAgent.Core/AgentConfig.cs b/agent/ShopDbEndpointAgent.Core/AgentConfig.cs index 92320022..e8f11fea 100644 --- a/agent/ShopDbEndpointAgent.Core/AgentConfig.cs +++ b/agent/ShopDbEndpointAgent.Core/AgentConfig.cs @@ -17,6 +17,23 @@ public sealed class AgentConfig // (thundering herd). Actual delay = cadence +/- up to jitterseconds. [JsonPropertyName("jitterseconds")] public int JitterSeconds { get; set; } = 60; + // One-time random delay before FIRST contact, applied by both the telemetry + // and command loops at startup. JitterSeconds only spreads the gap BETWEEN + // runs, so without this every agent in the fleet still makes its first + // request the instant its service starts - and the service is installed + // start=auto, so a patch reboot starts them all together. + // + // Measured on a 700-endpoint burst (devnotes/gevent-sizing-simulation.md): + // simultaneous arrival drove TLS handshakes to 1482ms worst case and pinned + // CPU at 305% while the database sat idle at 3 active queries. Spreading the + // same arrivals over 30s cut worst-case handshake to 63.7ms. The server is + // not the constraint; simultaneity is. + // + // Costs up to this many seconds before a rebooted endpoint picks up a queued + // command. That window is on top of a reboot the machine was already + // unreachable for. 0 disables the delay. + [JsonPropertyName("startupjitterseconds")] public int StartupJitterSeconds { get; set; } = 30; + [JsonPropertyName("adoptonfirstrun")] public bool AdoptOnFirstRun { get; set; } = true; // Directory holding site-provided PowerShell "agent utilities" the runtime diff --git a/agent/ShopDbEndpointAgent.Core/CommandWorker.cs b/agent/ShopDbEndpointAgent.Core/CommandWorker.cs index 2d71232d..5327f8e8 100644 --- a/agent/ShopDbEndpointAgent.Core/CommandWorker.cs +++ b/agent/ShopDbEndpointAgent.Core/CommandWorker.cs @@ -41,15 +41,18 @@ public sealed class CommandWorker : BackgroundService private readonly CommandExecutor executor; private readonly ILogger logger; private readonly CollectSignal? collectSignal; + private readonly AgentConfig? config; private readonly string hostname; public CommandWorker(CommandClient client, CommandExecutor executor, - ILogger logger, CollectSignal? collectSignal = null) + ILogger logger, CollectSignal? collectSignal = null, + AgentConfig? config = null) { this.client = client; this.executor = executor; this.logger = logger; this.collectSignal = collectSignal; + this.config = config; hostname = Environment.MachineName; } @@ -57,6 +60,18 @@ public sealed class CommandWorker : BackgroundService { logger.LogInformation("Command channel started for host {Host}", hostname); + // Spread first contact across the fleet. Every agent otherwise polls the + // instant its service starts, and the service is start=auto, so a patch + // reboot puts the whole fleet on the wire together. See StartupJitter. + var startupDelay = StartupJitter.Next(config?.StartupJitterSeconds ?? 0, logger); + if (startupDelay > TimeSpan.Zero) + { + logger.LogInformation( + "Delaying first command poll by {Delay}ms (startup jitter)", + (int)startupDelay.TotalMilliseconds); + await SafeDelay(startupDelay, stoppingToken); + } + while (!stoppingToken.IsCancellationRequested) { var pollStarted = System.Diagnostics.Stopwatch.StartNew(); diff --git a/agent/ShopDbEndpointAgent.Core/StartupJitter.cs b/agent/ShopDbEndpointAgent.Core/StartupJitter.cs new file mode 100644 index 00000000..14e4157a --- /dev/null +++ b/agent/ShopDbEndpointAgent.Core/StartupJitter.cs @@ -0,0 +1,51 @@ +using Microsoft.Extensions.Logging; + +namespace ShopDbEndpointAgent.Core; + +// One-time startup delay, shared by TelemetryWorker and CommandWorker. +// +// Both loops otherwise make their first request the instant the service starts, +// and the service is installed start=auto - so a patch-day reboot has an entire +// fleet arriving together, twice per machine (a telemetry report AND a command +// poll). AgentConfig.JitterSeconds does not help: it spreads the gap BETWEEN +// runs, not the first contact. +// +// Lives here rather than in either worker so the two cannot drift apart, and so +// the clamp is enforced in one place. +internal static class StartupJitter +{ + // Ceiling on the configured value. A fat-fingered appsettings.json must not + // be able to park an endpoint for hours before it first checks in: the + // command channel is how an operator would correct it, so an unbounded + // delay would take away the means of recovery. + private const int MaxStartupJitterSeconds = 300; + + public static TimeSpan Next(int configuredSeconds, ILogger logger) + { + // Warn rather than clamp silently: an operator who deliberately set a + // large value and got a smaller one has config and behaviour disagreeing + // with no way to tell why. A negative value is a mistake too, so it is + // worth the same line. + if (configuredSeconds > MaxStartupJitterSeconds) + { + logger.LogWarning( + "startupjitterseconds {Configured} exceeds the {Max}s ceiling; using {Max}s", + configuredSeconds, MaxStartupJitterSeconds, MaxStartupJitterSeconds); + } + else if (configuredSeconds < 0) + { + logger.LogWarning( + "startupjitterseconds {Configured} is negative; startup jitter disabled", + configuredSeconds); + } + + int spread = Math.Clamp(configuredSeconds, 0, MaxStartupJitterSeconds); + if (spread == 0) + return TimeSpan.Zero; + + // Milliseconds, not whole seconds: 700 endpoints over a 30s window would + // otherwise land in 30 clumps of ~23 simultaneous arrivals, which is the + // same pile-up in miniature. + return TimeSpan.FromMilliseconds(Random.Shared.Next(0, spread * 1000)); + } +} diff --git a/agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs b/agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs index 97b94bb4..bf689192 100644 --- a/agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs +++ b/agent/ShopDbEndpointAgent.Core/TelemetryWorker.cs @@ -139,6 +139,25 @@ public sealed class TelemetryWorker : BackgroundService "Endpoint agent {Version} -> {BaseUrl}, cadence {Cadence}s +/-{Jitter}s", agentVersion, config.BaseUrl, config.CadenceSeconds, config.JitterSeconds); + // Spread first contact across the fleet. JitterSeconds only spreads the + // gap BETWEEN runs, so without this the whole fleet reports the instant + // its service starts. See StartupJitter. + var startupDelay = StartupJitter.Next(config.StartupJitterSeconds, logger); + if (startupDelay > TimeSpan.Zero) + { + logger.LogInformation( + "Delaying first telemetry report by {Delay}ms (startup jitter)", + (int)startupDelay.TotalMilliseconds); + try + { + await Task.Delay(startupDelay, stoppingToken); + } + catch (OperationCanceledException) + { + return; + } + } + bool firstRun = true; while (!stoppingToken.IsCancellationRequested) { diff --git a/agent/ShopDbEndpointAgent/Program.cs b/agent/ShopDbEndpointAgent/Program.cs index ffa770bd..1e041166 100644 --- a/agent/ShopDbEndpointAgent/Program.cs +++ b/agent/ShopDbEndpointAgent/Program.cs @@ -346,7 +346,8 @@ static void RegisterCoreServices(IServiceCollection services, AgentConfig config sp.GetRequiredService(), sp.GetRequiredService(), sp.GetRequiredService>(), - sp.GetRequiredService())); + sp.GetRequiredService(), + config)); } static void ConfigureLogging(ILoggingBuilder logging, string logDirectory) diff --git a/agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj b/agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj index ec456c3b..e3ca12f1 100644 --- a/agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj +++ b/agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj @@ -131,6 +131,27 @@ (AgentConfig.LaunchPaths, shipped defaults overridable per site). 1.1.0: generic, data-driven service_*/process_* actions carry their target over the wire, so new policies/apps need no agent rebuild. --> + - 1.11.0 + 1.12.0 true win-x64 true diff --git a/agent/ShopDbEndpointAgent/appsettings.json b/agent/ShopDbEndpointAgent/appsettings.json index 6db6ca67..78479ed8 100644 --- a/agent/ShopDbEndpointAgent/appsettings.json +++ b/agent/ShopDbEndpointAgent/appsettings.json @@ -3,6 +3,7 @@ "apikey": "REPLACE_WITH_COLLECTOR_KEY", "cadenceseconds": 900, "jitterseconds": 60, + "startupjitterseconds": 30, "adoptonfirstrun": true, "utilitiespath": "", "logpath": "", diff --git a/docker-compose.yml b/docker-compose.yml index 8af86016..c80a3588 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -79,28 +79,39 @@ services: - gunicorn - --bind - 0.0.0.0:5001 - # Threaded workers, not the blocking sync default. An agent that fails - # the TLS handshake retries in a loop, and with 4 sync workers a handful - # of those pinned every worker: real UI requests queued 20+ seconds while - # the CPU sat idle. A stuck connection now holds one THREAD, and 32 - # (4 x 8) have to be exhausted before a person waits. + # Async workers, not threads. The endpoint agent long-polls + # /api/endpoints/commands/next and re-polls immediately on a 204, so every + # agent holds a connection essentially continuously and fleet size IS the + # concurrency. Under gthread that meant 4 x 8 = 32 slots total, and a + # single worker's 8 threads could fill with sleeping agents while the CPU + # sat idle - measured: 8 of 9 threads in timed sleep, UI p95 24s. # - # Found in production at the other site (their commit db75ca7). Their - # server, our agent - so the same client behaviour will reach any site - # running the endpoint agent, and the fix belongs in the product rather - # than on one branch. + # A held connection under gevent is a greenlet, not an OS thread, so idle + # is nearly free. Measured on this stack (devnotes/gevent-sizing-simulation.md): + # 700 simulated agents plus UI, integration and kiosk traffic served with + # zero errors at UI median 7.5ms, peak RSS 479MB. CPU, not memory or DB + # connections, is the binding resource. - --worker-class - - gthread + - gevent - --workers - "4" - - --threads - - "8" - # 30, not 60: a hung connection frees its thread in half the time, which - # is what decides whether a retry storm is absorbed or queues. + # Per-worker greenlet ceiling. 4 x 2000 is far above any realistic fleet; + # it exists so a connection spike queues rather than being refused. + - --worker-connections + - "2000" + # 60, not 30: the timeout must exceed _CLAIM_HOLD_SECONDS in + # plugins/endpoints/api/routes.py or gunicorn kills a healthy long-poll + # mid-hold. Cheap here because a stuck connection costs a greenlet. - --timeout - - "30" + - "60" - --access-logfile - "-" + # %(D)s is request duration in microseconds. Without it a slow request is + # indistinguishable from a fast one in the log, and latency has to be + # reconstructed from outside the container. Keep in sync with the + # Dockerfile CMD. + - --access-logformat + - '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s %(D)sus "%(f)s" "%(a)s"' - --error-logfile - "-" - --certfile @@ -112,8 +123,11 @@ services: # Match the TLS listener: probe over https and accept the self-signed cert # (-k). The Dockerfile default probes http and would stay red once gunicorn # only speaks TLS. - test: ["CMD", "curl", "--silent", "--insecure", "https://localhost:5001/api/auth/login", - "-X", "POST", "-H", "Content-Type: application/json", "-d", "{}"] + # /api/dashboard/health is the public probe route and answers 200 even when + # the database is down, so the body's status field is what decides health. + # Probing POST /api/auth/login instead logged a 400 every 30s and exercised + # the credential path from a health check. + test: ["CMD-SHELL", "curl --silent --insecure https://localhost:5001/api/dashboard/health | grep -q '\"status\":\"ok\"'"] interval: 30s timeout: 5s start_period: 10s diff --git a/docs/ShopDB-Planner-Tasks-v2.csv b/docs/ShopDB-Planner-Tasks-v2.csv new file mode 100644 index 00000000..e69de29b diff --git a/docs/notes/endpoint-telemetry-and-ui-phases.md b/docs/notes/endpoint-telemetry-and-ui-phases.md new file mode 100644 index 00000000..4423331a --- /dev/null +++ b/docs/notes/endpoint-telemetry-and-ui-phases.md @@ -0,0 +1,631 @@ +# Endpoint telemetry and PC Details UI - phased working notes + +Scratch/reference notes for a multi-phase effort. Not a contract document. When +a phase lands, its details belong in AGENT-CONTRACT.md, the csproj changelog, or +the panel declarations - not here. Delete this file when all phases are done. + +Branch: wilm. Current versions at time of writing: agent 1.8.0, tray 1.1.0. + +--- + +## Ground truth established before scoping + +Verified in code, not assumed. These constrain everything below. + +**Panel groups are a hardcoded frontend list.** +`frontend/src/components/PluginAssetPanels.vue:423`: + + const RENDERED_GROUPS = ['details', 'monitors', 'software', 'activity'] + +Any panel whose `group` is not in that list silently falls back to `details`. +Nothing is logged. The comment directly above it records that this has already +caused two disappearing-card incidents (`actions` -> `monitors`, and the +`metrics` -> `monitors` rename). Tab display labels are separately hardcoded in +`plugins/computers/frontend/views/PCDetail.vue:441-449`. + +Consequence: introducing a new tab is a THREE-file change that must land in one +commit - panel `group` in `plugins/endpoints/plugin.py`, `RENDERED_GROUPS`, and +the PCDetail label map. Split it and cards vanish with no error. + +**There is no pinned/locked card concept today.** +Layout is a CSS grid ordered purely by the integer `position` on each panel. +There is no "top row", no span control except one special case: the Network +interfaces table sets full-width span because at half width it wrapped every +cell to three lines (`PluginAssetPanels.vue:693`). + +Consequence: "lock this card top-left/top-right" does not exist yet and is +itself a feature, not a config value. See Phase 5. + +**Two of the requested cards are not plugin panels at all.** +Both the Entra/Intune card (`PCDetail.vue:202-243`) and the Network/IP card +(`PCDetail.vue:249-265`) are hand-written Vue inside PCDetail's details grid, +not declarations in `plugin.py`. Moving them is a component refactor, not a +`position`/`group` edit. + +**There is no LAPS retrieval feature in this codebase.** +Searched: no LAPS button, endpoint, or credential path exists. The only hit is +a historical incident note in `plugins/tools/frontend/views/SecureSendCreate.vue:254` +about LAPS passwords once being sent over a chat link. The request assumes a +click-to-retrieve LAPS button already exists on the Entra card; it does not. +This needs its own scoping conversation - it is a credential-disclosure path +with real audit/permission implications and must not be smuggled in as part of +a layout change. See Open Questions. + +**Existing panel positions in `monitors`** (so new cards do not collide): +10 status, 12 tracked services, 13 tracked processes, 14 hardware, 15 osinfo, +16 interfaces, 17 storage, 18 operator reports, 19 listeners, 40 user context. +`activity` uses 20. `software` uses 30. + +--- + +## Phase 1 - Mapped drive reachability (agent + tray) + +STATUS: code complete, NOT fully validated. See "Where we left off" at the end +of this file before resuming. + +Highest value, and the reason the user-context work exists. Extends the probe +already added in agent 1.8.0 / tray 1.1.0. + +**No active reachability testing.** An earlier draft proposed touching each +share to see if it responded, with a per-mapping timeout. Rejected: Windows +already reports the status, so probing the share is wasted work and reintroduces +the hang risk the bounded probe exists to avoid. Read the status, do not test it. + +**The two status sources disagree, and that disagreement is the signal.** +Captured live on the same box at the same moment, same two mappings: + + Get-SmbMapping net use + OK Q: ...specdata Disconnected Q: ...specdata + OK S: ...shared Reconnecting S: ...shared + +`Get-SmbMapping` reported OK for both while `net use` reported Disconnected and +Reconnecting. So neither source alone is trustworthy: collecting only +Get-SmbMapping (the nicer, more structured one) would have reported a healthy +Q: on a box whose Q: was actually down - exactly the failure this card exists +to catch. + +Collect BOTH and keep them as separate fields per mapping. Do not silently +reconcile them in the agent; the server/UI decides presentation, and preserving +the disagreement is diagnostically useful in its own right. The card should +surface the WORSE of the two rather than the prettier one. + +Status vocabulary must include `Reconnecting` - it appears in the real capture +above and an earlier draft's `Ok`/`Disconnected`/`Unknown` set would have +forced it into the wrong bucket. Keep `Unknown` for probe failure so a failed +check never renders as green Ok. + +**Elevation matters and confirms the tray approach.** The capture was taken +from an elevated shell (`start powershell -verb runas`). Because of UAC token +separation, an elevated context sees a DIFFERENT mapped-drive table than the +normal user - often empty. The tray runs unelevated as the operator, which is +the correct context to enumerate from. Do not "improve" this later by elevating +the probe; that would silently empty the card. + +Mechanism: the probe already writes JSON to the spool file, so this rides the +existing handoff - run both commands in-session, write the combined result, +service reads it. No new transport. + +Server: persist as part of the existing `usercontext` payload key (no new +column, no migration needed - it rides `usercontextjson`). + +Note in code, commented out, per request: a future one-click remediation +command will re-establish a Disconnected mapping. Do not implement now, but +leave the command-catalog seam obvious. It will be an allowlisted action in +`plugins/endpoints/commands.py` and must run in the USER session (a SYSTEM +remap creates a mapping the operator cannot see, and per the elevation note +above an elevated remap has the same problem). + +## Phase 2 - Cheap system-side telemetry (agent service) + +All service-side, no session needed. Group these because they share the +inventory-snapshot plumbing and one migration. + +- **Time sync offset and source** - CUT. Do not collect, do not surface. + w32time is Stopped on provisioned endpoints, confirmed on more than one box + (`get-service w32time` -> Stopped). The service appears to be stopped by + server-side policy after provisioning, so `w32tm /query /status` returns + 0x80070426 "service has not been started" fleet-wide and there is no offset to + read. Collecting it would return null everywhere and produce an alerting + signal that never fires - worse than nothing, because a permanently-null + health check reads as "no problem found". + + Implemented and then removed in this phase. If time skew becomes a real + suspect later, the prerequisite is getting w32time RUNNING on the fleet; the + collection is the easy part. One implementation note worth keeping: parse the + offset by LABEL, not by "first value ending in s". w32tm prints Root Delay and + Root Dispersion in the same seconds format BEFORE Phase Offset, so a + first-match parser silently reports network latency as clock skew. That bug + was written and caught here; do not rewrite it. +- **Disk errors / SMART predictive failure** - DONE. Landed as its own "Disk + health" card rather than folded into the Storage card as originally planned: + Storage is per VOLUME (C:, D:) and this is per PHYSICAL device, and one drive + can back several volumes or none the operator sees, so the two do not share a + key and could not be one table. + The important finding: the classic ATA SMART class + (`MSStorageDriver_FailurePredictStatus`) returns ZERO instances on an NVMe + machine. Verified on a two-NVMe box where both drives were present and + healthy. An agent collecting only that class would have reported an empty + disk health list on every modern endpoint and read as "no problems" - the + exact silent-healthy failure mode this telemetry exists to prevent. So the + agent collects BOTH `MSFT_PhysicalDisk` (the only NVMe-aware source) and + `Win32_DiskDrive.Status` (ATA/SATA, and hosts where the Storage namespace is + unavailable), and the server does not reconcile them - same reasoning as the + Phase 1 mapped-drive merge, where two disagreeing sources was the signal. + Note `MSFT_PhysicalDisk` returns raw enum INTEGERS through + ManagementObjectSearcher, not the friendly strings `Get-PhysicalDisk` prints + (Health=0 Op=2 Media=4 Bus=17 for a healthy NVMe); the mapping lives in the + collector so the wire stays self-describing. +- **NIC link speed and duplex** - DONE. Paired onto the existing interfaces + entries by MAC from `MSFT_NetAdapter` (`Win32_NetworkAdapterConfiguration` + carries no link state, so it is a second query). Both fields are null when + unreadable: verified on the live box that a DISCONNECTED adapter returns + empty Speed and FullDuplex, so defaulting would have reported every unplugged + NIC as half-duplex. Not surfaced. +- **Boot type** - DONE. Two facts kept separate: `hiberbootenabled` from the + registry (fast startup configured) and `lastboottype` from Kernel-Boot event + 27 (what the last boot did). Neither substitutes for the other - the registry + cannot say what the last boot did, and the event can age out of the log. + Verified live: BootType=0 with HiberbootEnabled=0. Not surfaced; placement + still undecided. +- **Windows Update history, last 5** - PINNED, deferred out of Phase 2. The COM + update searcher is not cheap: it routinely takes seconds and sometimes tens of + seconds, which would make it the most expensive thing in an otherwise + WMI-and-registry `Collect()` running on the heartbeat path. + + The blocking concern is broader than cost. Some endpoints host COM server + applications, and driving the update searcher through COM on those boxes risks + interfering with collections - or with the hosted application itself. Needs + its own investigation before any implementation, and if it lands it wants its + own cadence (hourly/daily), never the every-cycle path. +- **Defender/AV health** - DONE, collected and persisted, NOT surfaced, exactly + as specified. Note the ages are in days and a real 0 means updated today, so + the collector sends null rather than 0 when Defender did not report. A box + with third-party AV or a stripped image has no Defender WMI provider at all, + which is normal here and not a fault. + +Explicitly dropped after discussion, do not collect: +certificates (corporate-managed), BitLocker (network join already requires it), +display topology (single monitor on the floor, trained users), session +type/RDP (RDP-in not permitted on the shop floor), logon server (local users). + +**Pending file-rename operations** - DONE, but NOT as a new collection. It was +already being read: `GetPendingReboot()` OR-s it together with the CBS and +Windows Update keys into one bool. The gap was that the bool cannot say WHICH +signal fired, so the "stronger signal" was already being flattened into the +weaker one. Shipped as `rebootreasons`, a list naming the signals that fired, +leaving `pendingreboot` untouched for compatibility. Empty list = checked, +nothing pending; null = the check failed. + +Schema: these add columns to `endpointstates`. This landed as `0015`, not the +`0014` reserved here - `0014` was taken by the disk health column earlier in +Phase 2. + +## Phase 3 - Service/Process Monitors tab split + +Tab churn is exactly what has silently eaten cards twice before, so this phase +is deliberately isolated from any collection work. + +- Rename the existing `Monitors` tab to `System Monitors`. +- Add a new `Service/Process Monitors` tab. +- Move `Tracked Services` (position 12) and `Tracked Processes` (13) into it. + +Must land atomically across all three places: +1. `group` on those panels in `plugins/endpoints/plugin.py` +2. `RENDERED_GROUPS` in `PluginAssetPanels.vue` +3. the tab label map in `PCDetail.vue` + +Also update the hardcoded panel-count assertion in +`tests/test_plugins/test_endpoints_plugin.py` +(`test_endpoints_asset_panels_are_computer_scoped`) - it has already needed +bumping once (7 -> 12) and will fail again here. + +Printers: a location-based printer auto-detect collection is in progress +elsewhere. Reserve space and a position band for a printer card in the monitors +family so this work does not have to be redone when it lands. Do not build the +card now. + +## Phase 4 - Card content additions + +Depends on Phase 2 data existing. + +- **Mapped drives card** - DONE. `list` render at position 21 in Monitors, one + row per drive letter with the remote path as meta and a status badge. Reuses + the existing status/badge vocabulary (`statecolor` + the ok/warning/critical + levels the tracked-services cards already declare) rather than new CSS. + Only UNHEALTHY drives are coloured. Green on every working drive makes the + broken one just another badge in a row of badges; the card exists to make the + exception findable. For the same reason unhealthy rows sort first - a user + with a dozen mappings should not hunt for the broken one - and the sort is on + health ALONE so drive-letter order survives within each group and the list + does not reshuffle between refreshes. + The merge itself was not touched. `_mapped_drive_rows` now returns structured + rows and `_mapped_drives` (the user-context summary line) is built on top of + it, so the card and the summary cannot drift apart about a drive's status. + Room reserved for the one-click reconnect: the `list` render already supports + a row action, so adding it is a map/commandendpoint change, not a new card. +- **Storage card** - superseded. Disk health shipped in Phase 2 as its own + "Disk health" card (position 18, directly under Storage) instead of being + folded in: Storage is per volume, disk health is per physical device, so they + do not share a key. Rows sort failure-first, then unknown, then OK - a failing + drive must not sit below the fold on a multi-disk box, and a drive that + stopped answering is a weaker claim of health than one that said OK. + Later merged into a single two-section card; see the entry below. +- **Windows Update card** - BLOCKED, not scheduled. Depends on Phase 2 data + that is pinned pending a COM-interference investigation. Do not build the card + until the collection question is settled. +- Boot type placement - DONE in Phase 3. Rendered in the OS card only when fast + startup applies, because that is the case where uptime understates how long + the kernel has been up. A permanent "Cold" row would be noise. +- **Storage + Disk health into one card with two collapsible sections** - + DONE. VISUAL GROUPING ONLY. These two tables are not merged into shared rows: + Storage is per volume, Disk health is per physical device, and one drive can + back several volumes or none the operator sees. This is emphatically NOT the + Phase 1 mapped-drive treatment, where two sources described the same drive and + the worse status won. Nothing here is reconciled. + Built as predicted: `tabs` was extended to support table sections (columns/ + rows/empty per section) in the shared renderer, and the two routes collapsed + into one `/storage` returning `sections`. `/diskhealth` is kept as a route - + it is a stable read for non-panel callers, and removing it for a UI-only + change would break them. + Two things only found by building it: + - The empty-section filter had to grow a table branch. A table section + declares its columns up front and so must survive having no rows; dropping + it takes its TAB with it, and a missing tab reads as "this machine has no + disks" rather than "nothing was reported". + - Full-width span was keyed off `render === 'table'`. A `tabs` panel holding a + six-column table needs the same span, and whether it does is only knowable + from FETCHED DATA, not the declaration - hence `spansFullWidth()`. + Both are mutation-tested in `tabSectionsTable.spec.js`. + Checked and cleared: the "inline label/value pairs, not nested cards" density + rule in Phase 5 does NOT block this. That rule is scoped to the PC Details + HEADER block, where the constraint is adding zero height so the tab strip is + not pushed toward the fold. It says nothing about panel bodies in the + Monitors tab. + +## Phase 5 - Header right-hand block (DEFERRED - revisit before building) + +Status: on hold at user request. The original ask assumed pinned cards were a +config value; they are not, and building a pinning system was not what the user +wanted to sign up for. Not scheduled. Nothing below is committed. + +The cascade worry that motivated pinning is explicitly parked - do not build +anything for it unless reflow problems actually surface. + +**Revised direction: use the empty right-hand half of the existing header.** +The PC Details header is a full-width card whose right side is currently dead +space - the hostname, badges, PC type, OS and location all sit left, and the +Status / Remote access / Refresh / Restart row runs along the bottom. The block +to the right of the identity lines is empty at every width. + +This is a better target than either pinned grid cells or extending the header +downward: +- It consumes space that already exists, so it costs ZERO additional vertical + height and cannot push the tab strip toward the fold. That was the hard + constraint, and this approach satisfies it structurally rather than by + budgeting. +- It sidesteps the Network interfaces width problem entirely (that table is + full-width by necessity and would fight a half-width pinned slot). +- Header content lives outside the panel grid, so it cannot be reflowed by + cards appearing or disappearing below it. + +Candidates for the block, unresolved: +- Entra/Intune deep links (currently a Details-tab card). Strongest candidate: + they are navigation, not data, and the user has called out their outbound + usability value. +- IP / network summary (currently the hand-written Network card). + +Density rules if this proceeds: inline label/value pairs, not nested cards. Do +not let the right block grow taller than the left identity block, or it starts +adding height after all and the zero-cost property is lost. + +Scope note: that "not nested cards" rule is about THIS header block only, where +the whole value is adding zero height. It is not a general ban on sections +inside a panel body - the Phase 4 Storage + Disk health grouping was checked +against it and is not blocked. + +Still true and still unbuilt if this proceeds: +- `Collectors` is not a current tab or group. If IP History moves to a new tab, + that is the same three-file atomic change as Phase 3. +- Entra empty state, requested: show `Please provide DeviceID` / + `Please provide ObjectID` when the backend has no value, rather than an + empty or broken card. + +LAPS is NOT a feature request - clarified by the user as an illustration of why +these outbound portal links matter to usability. No LAPS work is scoped. + +## Phase 6 - Intune/Entra link bug (independent, can be done anytime) + +Reported: the INTUNE and ENTRA buttons sometimes do not appear until the page +is refreshed. Diagnosed, root cause found, NOT yet fixed. + +**Cause: a cold-cache race, not a render problem.** + +`frontend/src/utils/siteSettings.js:9` caches only the RESOLVED settings object: + + let settingsCache = null + async function loadSettings() { + if (settingsCache) return settingsCache + ... + +The in-flight promise is not cached, so every caller during the first load +fires its own `settingsApi.list()`. + +In `plugins/computers/frontend/views/PCDetail.vue:513`, `getEntraDeviceUrls()` +is deliberately fire-and-forget (`.then(...)`, not awaited) so a settings +failure cannot block the page. Meanwhile `computersApi.get()` on line 519 IS +awaited. On a cold cache the settings response can therefore land AFTER the +computer data has rendered. Until it does, `entraUrlTemplate` / +`intuneUrlTemplate` are still `''`, so `fillTemplate()` returns null +(line 426: `if (!template || !value) return null`), and the `v-if` on the +anchors hides both links. A later refresh hits the warm cache and they appear. + +That also explains the intermittency: it depends on which response wins, so it +varies by network timing and by whether another view already warmed the cache. + +**Fix: cache the in-flight promise, not just the result.** Assign the promise +to `settingsCache` immediately so concurrent callers await the same request. +This is a one-line-shaped change in `siteSettings.js` and fixes every consumer +of site settings, not just this card - the same race exists for any view that +reads settings during mount. + +Watch for: the existing `catch` sets `settingsCache = {}` on failure, which +permanently caches the empty result for the page session. Confirm whether a +failed settings load should stay cached (currently it does) or be retryable. +Do not change that behavior silently while fixing the race. + +Verify after fixing that the links appear on a hard reload with a cold cache +and throttled network, which is the condition that reproduces it. + +DONE. Fixed in `siteSettings.js`: the in-flight promise is now cached alongside +the result, and a failed load is cached for 30s rather than the page session. +Covered by `frontend/src/utils/siteSettings.spec.js` (6 tests). Open question 5 +resolved in favour of the short TTL. + +Measured side benefit: the old code assigned `settingsCache` only AFTER the +response resolved, so each sequential `await` re-entered `loadSettings()` before +the cache existed. `applyBranding()` awaits five settings in a row at bootstrap, +`getServicenowUrls()` four, `getEntraDeviceUrls()` two - all of which were +separate GETs on a cold load. Now one. + +## Phase 7 - PCForm "Failed to load data" (NOT the same bug) + +Separate from Phase 6 and NOT caused by it - `PCForm.vue` does not import +`siteSettings.js`. Surfaced on the F3KKSGH4 edit page. + +Partially addressed, root cause still UNKNOWN. + +Evidence: the form renders with reference data present (printer list populated, +dropdowns built) but record fields empty. That means the eight-way +`Promise.all` of reference endpoints SUCCEEDED and the failure is in the +edit-only block, `PCForm.vue` lines 589-635. The only unprotected call there is +`computersApi.get(route.params.id)`; `ensureModelLoaded()` already swallows its +own errors, so it is not the source. + +Two candidates remain, with different fixes and different layers: +- the `computersApi.get` request fails (backend/route problem), or +- a TypeError while mapping the response into `form.value` (frontend shape + mismatch, plausibly from recent schema work). + +Both land in the same catch and both printed the same generic string, which is +why this could not be diagnosed from the UI. + +Done so far: the catch now uses `apiError(err, ...)` so the real server message +surfaces instead of a generic one, and the stale comment claiming a call +"handles its own failure" was removed - no such call remains in that list. + +NEXT: reload the edit page and read the now-specific error. An axios status +code points at the backend; a TypeError names the property and points at the +mapping block. Do not guess between them. + +Also worth doing regardless: a single failed reference list currently blanks +the whole form via `Promise.all` rejection. Degrading one dropdown instead +would be the behaviour the removed comment falsely claimed. + +--- + +## Simulation suite - required check (from a real dev-environment failure) + +The suite must assert that CODE AND SCHEMA ARE NEVER SERVED OUT OF STEP, and +must do it by bringing the image UP, not by building a schema from the models. + +Observed 2026-09-05 in dev, after rolling containers for the Phase 3/4 UI work: +every query against `endpointstates` returned 500 with + + (1054, "Unknown column 'endpointstates.diskhealthjson' in 'field list'") + +The Phase 2 migrations (endpoints 0014, 0015) were written and committed WITH +their models, correctly. They had simply never been applied to that database. +`flask plugin upgrade-all` fixed it in one command. + +Why nothing caught it: + +- The pytest suite builds its schema from the models, so model and schema are + by construction in step. This class of fault is INVISIBLE to it. A green test + run says nothing about whether a deployed database has the columns. +- Nothing in the startup path checks. `create_app()` and `wsgi.py` do no + migration-state validation, so the container reports HEALTHY while every + query against the affected table 500s. The health check passes because it + does not touch the table. +- `flask db preflight` exists but is manual and pre-migration; it is not a + guard on serving traffic. + +The failure is also SILENT IN THE UI in a way that misdirects: a panel whose +endpoint 500s is dropped from `populatedGroups`, so its whole TAB disappears +from PC Details. The visible symptom was a missing "Service/Process Monitors" +tab, which reads as a UI regression in the tab-vocabulary code and sends you +to entirely the wrong file. Nothing on the page mentions the database. + +So the suite needs, at minimum: + +1. A start-up case that runs the real image against a database deliberately + left one migration behind, and asserts the mismatch is reported LOUDLY - + not a healthy container serving 500s. +2. Coverage that the documented order (`flask db upgrade`, then + `flask plugin upgrade-all`, THEN serve) is what the container actually does. +3. A UI-level assertion that a 500ing panel endpoint does not silently remove a + tab with no explanation on screen. + +Worth considering separately from the suite: a startup guard that refuses to +serve, or at least logs at ERROR, when a model references a column the database +does not have. Cheap here, expensive on a fleet rollout. + +## Open questions - answer before the relevant phase + +1. ~~LAPS~~ - RESOLVED: not a feature request. The user was illustrating why the + outbound Intune/Entra links matter to usability. No LAPS work scoped. The + security concerns noted earlier stand if it is ever revisited. +2. ~~Pending file-rename operations~~ - RESOLVED: collect. Folded into Phase 2. +3. ~~Phase 5 pinning / cascade symptom~~ - RESOLVED: not worrying about the + cascade unless it surfaces. Pinned cards dropped in favour of the header's + empty right-hand block; Phase 5 deferred. +4. **`Collectors` tab** - only relevant if Phase 5 proceeds. New tab for one + moved card, or does IP History belong in an existing tab? +5. **Failed settings load caching** - Phase 6. Should a failed + `settingsApi.list()` stay cached for the page session (current behavior) or + be retryable? Decide while fixing the race, do not change it silently. + +## Sequencing + +1 -> 2 -> 3 -> 4. Phase 6 is independent and can be slotted in anytime; it is +small, self-contained, and fixes a user-visible bug, so it is a good candidate +to do early. Phase 5 is deferred and unscheduled. + +Phases 1 and 2 are additive and independently deployable. Phase 3 is a +contained tab move. Phase 4 needs 2. + +Agent/tray version bumps: Phase 1 touches the probe (tray behavior) and Phase 2 +touches the service collector. Both need csproj changelog entries. Note the +existing coupling hazard - `Program.cs` has no unknown-arg guard, so an older +agent receiving a newer tray's verb starts a full second host rather than +no-oping. Fix that guard before any deployment where tray and agent could skew. + +## Phase 8 - fetchAllPages returns a bare array (LAST, and it affects the above) + +Deliberately last: it is a cross-cutting API-surface change, and every earlier +phase should land against the current shape rather than a moving one. + +`frontend/src/api/index.js:131` - `fetchAllPages()` returns the ROWS: + + return items + +Every other method on every API object returns the axios RESPONSE. So callers +must remember, per call site, which of two shapes they are holding: + + const rows = await vendorsApi.listAll() // array + const response = await vendorsApi.list() // { data: { data: [...] } } + +Reading `.data.data` off the array yields `Cannot read properties of undefined +(reading 'data')`. Inside a `Promise.all` that rejects the whole batch, so one +wrong unwrap blanks an entire form - and if it throws before the record fetch, +NO request is made at all, which makes it look like a backend or schema fault. +That is exactly how the PC edit page presented, and why it was misdiagnosed +twice before the log showed `/api/computers/729` was never requested. + +Known history - this class of bug has now hit at least four times: +- `MachineForm.vue` `pcsRes` (fixed previously, comment left in place) +- `PCForm.vue` `vendorRes` (fixed now) +- `MachineForm.vue` `vendorRes` and `buRes` (fixed now - the warning comment was + already sitting three lines away and the adjacent defects were still missed) +- `MeasuringToolForm.vue` `buResponse` (fixed now) + +`ConfigurationPoliciesList.vue:187` defends with +`Array.isArray(value) ? value : ...`, which is evidence the shape is not +trustworthy at the call site. + +Why this matters to the phases above: Phase 1/2 UI work adds call sites that +consume paged reference data, and Phase 5 (if revived) moves cards between +components. Any new `listAll()` caller is another chance to hit this. Landing +the phases first and this last means one migration pass, not a moving target. + +Options, in order of preference: +1. Make `fetchAllPages` return `{ data: { data: items } }` so every API method + has ONE shape. Mechanical but touches every existing `listAll()` caller in + the same commit, or they break in the opposite direction. Highest value, + highest blast radius - needs the full grep and a single atomic change. +2. Rename to something the shape is obvious from (`fetchAllRows`, + `listAllRows`). Cheap, no behavior change, keeps the trap but makes it + visible at the call site. Good interim step if 1 is too large right now. +3. Leave it and rely on review. Rejected - four occurrences say review does not + catch it, and one of them happened directly beneath a comment warning about + it. + +Whichever is chosen, do the full-codebase grep for `listAll(` first; there are +callers in core views, print views, and at least seven plugin frontends. + +--- + +# Where we left off + +Written mid-session when the test tooling stopped working. Read this first. + +## Phase 1 is code complete but NOT fully validated + +What landed, all uncommitted in the working tree: + +**`agent/ShopDbEndpointAgent.Core/UserContextProbe.cs`** - added `Get-SmbMapping` +alongside the existing `Win32_NetworkConnection` capture, emitted as a separate +`smbmapping` key with its own `smbmappingerror`. The two sources are NOT +reconciled in the probe, deliberately - see the Phase 1 notes above. Still +unelevated, still in the operator session. + +**`plugins/endpoints/api/routes.py`** - added `_drive_letter`, +`_DRIVE_STATUS_RANK`, `_worse_drive_status`, `_mapped_drives`, and pointed +`panel_usercontext()` at `_mapped_drives()`. Merges the two views by drive +letter and shows the WORSE status. Carries a long WARNING comment naming the +three load-bearing behaviours; that comment is doing real work, leave it. + +**`plugins/endpoints/commands.py`** - commented-out `remap_drive` seam with the +user-session/elevation reasoning. Comment only, no behavior. + +## Validation state - the important part + +Last KNOWN-GOOD full run: `8 failed, 2489 passed, 11 skipped`. All 8 failures +are pre-existing and unrelated: +- 3x `test_endpoints_plugin.py` timestamp tests - these fail ONLY on Windows. + `_in_zone()` calls `time.tzset()`, which does not exist on Windows (Unix-only + in CPython), so it raises AttributeError before the assertions ever run. + `_parse_dt` itself is correct. These pass on CI. Consider a + `skipif(not hasattr(time, 'tzset'))` so the suite is honest on Windows. +- 5x docs / security-headers / plugin-guard, all pre-existing. + +TWO CHANGES LANDED AFTER THAT RUN AND ARE UNVERIFIED: +1. `_mapped_drives()` now renders `(Unknown)` for a drive neither source gave a + status for. Previously it rendered blank, which reads as healthy - the exact + silent-green failure the warning comment warns about. Small but real, + behavioral, and untested. +2. The `commands.py` seam. Comment only, zero risk. + +`get_errors` on both changed files is clean, but that only proves they parse. + +## Tooling failure that stopped validation + +Every `run_command_in_terminal` call began failing with "A task was canceled", +including a two-file pytest run that should finish in under a second, and a +background launch that never wrote its `Tee-Object` output file. Two full-suite +runs had succeeded earlier in the same session, so this is not slowness - the +execution path went down partway through. + +Test Explorer discovers all 2508 tests but returns zero results for every +filter tried (`TypeName`, `MethodName`, `FullyQualifiedName`). Likely needs a VS +restart. + +## Next actions, in order + +1. Restart VS, then run `python -m pytest tests/test_plugins/test_endpoints_plugin.py -q`. + Expect 3 failed (the tzset ones) / 23 passed. Anything else is ours. +2. Consider adding tests for `_mapped_drives()` - currently zero coverage on + logic that decides whether a broken drive renders as broken. The cases that + matter: sources disagree (worse wins), drive in only one source (kept), no + status at all (Unknown, not blank), `Get-SmbMapping` absent entirely + (degrades to netuse alone). +3. Only then start Phase 2. Do not begin new collection work on an unverified + base - the untested `(Unknown)` delta would get buried under it. + +## Deferred, agreed explicitly + +Agent and tray republish happens at the END of all phases, not per-phase. The +server change is safe to deploy ahead of the agent: until the new probe ships, +`smbmapping` is simply absent from the payload and `_mapped_drives()` falls +through to `netuse` alone, rendering exactly as it does today. No ordering +constraint between them. + diff --git a/frontend/src/components/panelGroups.spec.js b/frontend/src/components/panelGroups.spec.js index ed30d778..3a4e439c 100644 --- a/frontend/src/components/panelGroups.spec.js +++ b/frontend/src/components/panelGroups.spec.js @@ -13,10 +13,7 @@ import { fileURLToPath } from 'node:url' // constant in the .vue file with no export to assert against. function read(relative) { - // .href, not the URL object: the jsdom test environment supplies its own - // URL global, and node's fileURLToPath rejects a foreign instance with - // "must be ... an instance of URL. Received an instance of URL". - return readFileSync(fileURLToPath(new URL(relative, import.meta.url).href), 'utf8') + return readFileSync(fileURLToPath(new URL(relative, import.meta.url)), 'utf8') } function renderedGroups() { diff --git a/frontend/src/components/pinnedPanels.spec.js b/frontend/src/components/pinnedPanels.spec.js index f21528fe..bf1ee53c 100644 --- a/frontend/src/components/pinnedPanels.spec.js +++ b/frontend/src/components/pinnedPanels.spec.js @@ -13,10 +13,7 @@ import { fileURLToPath } from 'node:url' // than a layout bug. function read(relative) { - // .href, not the URL object: the jsdom test environment supplies its own - // URL global, and node's fileURLToPath rejects a foreign instance with - // "must be ... an instance of URL. Received an instance of URL". - return readFileSync(fileURLToPath(new URL(relative, import.meta.url).href), 'utf8') + return readFileSync(fileURLToPath(new URL(relative, import.meta.url)), 'utf8') } const panelsSource = () => read('./PluginAssetPanels.vue') diff --git a/frontend/src/views/settings/ShopdbBackupsList.vue b/frontend/src/views/settings/ShopdbBackupsList.vue index 0c2c8d44..705b98f4 100644 --- a/frontend/src/views/settings/ShopdbBackupsList.vue +++ b/frontend/src/views/settings/ShopdbBackupsList.vue @@ -4,18 +4,9 @@

Backups

-
- Site Backups is not configured on this deployment. - - The feature needs the backup overlay (a separate container writing sets - onto a shared volume) and a readable backups root. A Windows/IIS install - does not run it, so nothing here would have a store behind it. - -
- -
+
@@ -311,9 +302,6 @@ const sets = ref([]) const loading = ref(true) const isBusy = ref(false) const lastGoodAt = ref(null) -// Assume configured until the list says otherwise, so a slow first load -// does not flash the not-configured notice on a working install. -const isConfigured = ref(true) const restoreTarget = ref(null) // Live restore progress. restoreStatus mirrors the api /status.restore payload; @@ -417,10 +405,6 @@ async function load() { const response = await api.get('/shopdbbackups') sets.value = response.data.data || [] lastGoodAt.value = response.data.meta?.lastgoodat || null - // A deployment without the backup overlay has no store behind any of - // this. Say so instead of showing 'No verified backup yet', which reads - // as a backup that failed rather than a feature that is not installed. - isConfigured.value = response.data.meta?.configured !== false } catch (error) { toast.error('Could not load backups.') } finally { @@ -663,17 +647,6 @@ onUnmounted(() => { .banner-none { border-color: #c53030; } .banner-none .banner-icon { color: #e53e3e; } -.not-configured { - display: flex; - flex-direction: column; - gap: 0.35rem; - padding: 1rem 1.25rem; - margin-bottom: 1rem; - border: 1px solid var(--border-color, #d8dbe0); - border-radius: 8px; - background: var(--surface-muted, #f6f7f9); -} - .explain { margin-bottom: 1.25rem; border: 1px solid var(--border); diff --git a/plugins/computers/frontend/views/PCTypesList.vue b/plugins/computers/frontend/views/PCTypesList.vue index 9a6d95a5..b21e6ccc 100644 --- a/plugins/computers/frontend/views/PCTypesList.vue +++ b/plugins/computers/frontend/views/PCTypesList.vue @@ -107,10 +107,9 @@ onMounted(() => loadData()) async function loadData() { loading.value = true try { - // listAll resolves to a plain array (fetchAllPages), not an axios - // response. Reading .data.data threw and left the page empty. - const response = await computersApi.types.listAll({ active: false }) - pcTypes.value = response || [] + // listAll() pages past the server's 100-row cap and returns the ROWS, not + // an axios response - there is no .data to unwrap. + pcTypes.value = await computersApi.types.listAll({ active: false }) } catch (err) { console.error('Error loading PC types:', err) } finally { diff --git a/plugins/endpoints/api/routes.py b/plugins/endpoints/api/routes.py index d2e49458..17bac052 100644 --- a/plugins/endpoints/api/routes.py +++ b/plugins/endpoints/api/routes.py @@ -1828,6 +1828,25 @@ def revoke_agentkey(): # How long the agent claim poll is held open server-side before returning empty. # Kept below the agent HttpClient timeout and typical proxy idle timeouts so a # quiet fleet re-polls cleanly instead of tripping a read timeout. +# +# 25s is the value this endpoint is designed around; it was temporarily dropped +# to 5s under the blocking gthread worker pool, where a held connection pinned +# one of a worker's 8 OS threads. Measured then: 14 agents at hold=25 left a +# single worker with 8 of 9 threads parked in timed sleep and UI requests hashed +# to that worker waited 12-24s. Under gevent a held connection is a greenlet and +# the per-worker ceiling is --worker-connections (2000), not 8 threads, so an +# idle hold no longer costs anything a UI request needs. +# +# Restored to 25 because hold length sets the fleet's REQUEST RATE, not its +# concurrency: the agent re-polls immediately on a 204, so hold=5 is 5x the +# requests of hold=25, and TLS handshake CPU was the measured burst bottleneck +# (devnotes/gevent-sizing-simulation.md). 25 also restores the margin against +# the agent's MinIdlePollDuration=5s floor, which CommandWorker.IsIdlePollClose +# uses to tell a normal hold-window close apart from a transient failure; at +# hold=5 that margin was ~20ms. +# +# Constraint: gunicorn --timeout must stay above this or a healthy long-poll is +# killed mid-hold. docker-compose.yml sets 60. _CLAIM_HOLD_SECONDS = 25 _CLAIM_POLL_INTERVAL = 1.0 diff --git a/plugins/endpoints/api/siteagent.py b/plugins/endpoints/api/siteagent.py index 56554537..c5d7fd28 100644 --- a/plugins/endpoints/api/siteagent.py +++ b/plugins/endpoints/api/siteagent.py @@ -118,6 +118,7 @@ _BASE_APPSETTINGS = { 'apikey': 'REPLACE_WITH_COLLECTOR_KEY', 'cadenceseconds': 900, 'jitterseconds': 60, + 'startupjitterseconds': 30, 'adoptonfirstrun': True, 'utilitiespath': '', 'logpath': '', diff --git a/plugins/printers/api/asset_routes.py b/plugins/printers/api/asset_routes.py index e626b95e..e2bed15a 100644 --- a/plugins/printers/api/asset_routes.py +++ b/plugins/printers/api/asset_routes.py @@ -1123,12 +1123,18 @@ def set_asset_printer_assignment(asset_id: int): ', '.join(str(assetid) for assetid in sorted(notprinters)))) typeids = _relationship_typeids(_USES_PRINTER, _DEFAULT_PRINTER) - if not typeids[_USES_PRINTER] or not typeids[_DEFAULT_PRINTER]: + missingtypes = [name for name in (_USES_PRINTER, _DEFAULT_PRINTER) + if not typeids[name]] + if missingtypes: # Seed data, not a migration. An un-seeded database cannot hold an - # assignment, and saying so beats writing rows nothing can read. + # assignment, and saying so beats writing rows nothing can read. Name + # the types that are actually missing: a partially seeded table is the + # common case, and "types are not seeded" sends someone looking for an + # empty table they will not find. return error_response( ErrorCodes.INTERNAL_ERROR, - 'Relationship types are not seeded - run: flask seed reference-data', + 'Relationship type(s) not seeded: {0} - run: flask seed ' + 'reference-data'.format(', '.join(missingtypes)), http_code=500) _reconcile_edges(asset_id, typeids[_USES_PRINTER][0], @@ -1784,10 +1790,13 @@ def seed_assignment_from_observed(asset_id: int): details={'hostname': hostname, 'skipped': candidate['skipped']}) typeids = _relationship_typeids(_USES_PRINTER, _DEFAULT_PRINTER, _CONTROLS) - if not typeids[_USES_PRINTER] or not typeids[_DEFAULT_PRINTER]: + missingtypes = [name for name in (_USES_PRINTER, _DEFAULT_PRINTER) + if not typeids[name]] + if missingtypes: return error_response( ErrorCodes.INTERNAL_ERROR, - 'Relationship types are not seeded - run: flask seed reference-data', + 'Relationship type(s) not seeded: {0} - run: flask seed ' + 'reference-data'.format(', '.join(missingtypes)), http_code=500) warnings = [] diff --git a/scripts/backup/Dockerfile.backup b/scripts/backup/Dockerfile.backup index 8c5215b0..a207fdde 100644 --- a/scripts/backup/Dockerfile.backup +++ b/scripts/backup/Dockerfile.backup @@ -14,23 +14,8 @@ FROM mysql:8.4 # bzip2/python3/alembic or a base image that carries them) so the release image # needs no outbound calls and no embedded cert. The cert is gitignored (*.pem) # and must be staged at scripts/backup/zscaler-root-ca.pem before building. -# -# Copied via the directory rather than by name, because the cert is OPTIONAL: it -# is gitignored, so a plain `COPY scripts/backup/zscaler-root-ca.pem` fails the -# build outright and nobody outside a TLS-intercepting network - another site, -# or our own CI - could build this image at all. A trailing glob does not fix -# that either: BuildKit fails a COPY whose pattern matches nothing, which is -# precisely the case being handled. Copying a directory that always exists and -# testing for the file inside it is the form that works both ways. -COPY scripts/backup/ /tmp/buildctx/ -RUN if [ -f /tmp/buildctx/zscaler-root-ca.pem ]; then \ - echo "staging corporate root CA"; \ - cp /tmp/buildctx/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/; \ - update-ca-trust; \ - else \ - echo "no corporate root CA staged - using the public trust store"; \ - fi \ - && rm -rf /tmp/buildctx +COPY scripts/backup/zscaler-root-ca.pem /etc/pki/ca-trust/source/anchors/zscaler-root-ca.pem +RUN update-ca-trust # restic from the distro is often stale; pull the pinned release binary so the # snapshot format and retention flags are predictable across sites. diff --git a/scripts/backup/_probe_manifest.py b/scripts/backup/_probe_manifest.py new file mode 100644 index 00000000..e69de29b diff --git a/scripts/backup/backup.sh b/scripts/backup/backup.sh index 6fa6843e..110f427b 100755 --- a/scripts/backup/backup.sh +++ b/scripts/backup/backup.sh @@ -47,12 +47,6 @@ mkdir -p "${SETDIR}" status_write backup database "${SETNAME}" "Dumping database" # 1. Database: whole-DB logical dump, consistent and non-blocking on InnoDB. -# Dumped WITHOUT --databases on purpose. That flag bakes CREATE DATABASE and USE -# into the file, which pins the dump to the schema name it was taken -# from: restoring onto a site whose MYSQL_DATABASE differs loaded the rows into -# the dump's schema, left the live one untouched, and reported success. -# --no-tablespaces because dumping tablespace metadata needs the PROCESS -# privilege, which the app user does not have on MySQL 8. log "dumping database ${MYSQL_DATABASE} from ${MYSQL_HOST}" mysqldump \ --host="${MYSQL_HOST}" \ @@ -61,8 +55,7 @@ mysqldump \ --single-transaction \ --routines \ --triggers \ - --no-tablespaces \ - "${MYSQL_DATABASE}" \ + --databases "${MYSQL_DATABASE}" \ | gzip -c > "${SETDIR}/db.sql.gz" log "database dump written ($(du -h "${SETDIR}/db.sql.gz" | cut -f1))" @@ -80,48 +73,7 @@ DATAREV="$(mysql \ --skip-column-names --batch \ -e "SELECT version_num FROM alembic_version LIMIT 1;" \ "${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')" -if [ -z "${DATAREV}" ]; then - # Recorded loudly rather than shrugged off: a set with no data revision - # cannot be gated on restore, and the gate treats an empty stamp as "no - # evidence" and refuses it. Better a visible warning now than a set that - # looks restorable and is not. - log "WARN could not read alembic_version - this set will NOT be restorable" - history_append backup warning "${SETNAME}" \ - "Could not read alembic_version; set has no data revision and cannot be restored" -fi - -# The same question for every PLUGIN chain. Each plugin owns an -# alembic_version_ table, and those move independently of the core one: -# code rolled back to an older plugin revision can sit at an identical core -# head while the dump carries plugin tables it has never heard of. Stamping only -# the core revision let that combination pass the restore gate, load, and then -# fail during the plugin upgrade with the data already replaced. Read straight -# from the dumped DB, same as the core revision above. -log "reading applied plugin revisions (alembic_version_*)" -DATAPLUGINS="$(mysql \ - --host="${MYSQL_HOST}" \ - --user="${MYSQL_USER}" \ - --password="${MYSQL_PASSWORD}" \ - --skip-column-names --batch \ - -e "SELECT table_name FROM information_schema.tables - WHERE table_schema = '${MYSQL_DATABASE}' - AND table_name LIKE 'alembic\\_version\\_%';" 2>/dev/null \ - | tr -d '\r' \ - | while read -r table; do - [ -n "${table}" ] || continue - plugin="${table#alembic_version_}" - rev="$(mysql \ - --host="${MYSQL_HOST}" \ - --user="${MYSQL_USER}" \ - --password="${MYSQL_PASSWORD}" \ - --skip-column-names --batch \ - -e "SELECT version_num FROM \`${table}\` LIMIT 1;" \ - "${MYSQL_DATABASE}" 2>/dev/null | tr -d '\r')" - [ -n "${rev}" ] && printf '%s\t%s\n' "${plugin}" "${rev}" - done \ - | python3 -c 'import json,sys; print(json.dumps(dict(l.rstrip("\n").split("\t",1) for l in sys.stdin if "\t" in l)))')" -[ -n "${DATAPLUGINS}" ] || DATAPLUGINS='{}' -log "plugin revisions: ${DATAPLUGINS}" +[ -n "${DATAREV}" ] || log "WARN could not read alembic_version - datarev will be empty" log "writing manifest" python3 /opt/backup/manifest.py \ @@ -130,7 +82,6 @@ python3 /opt/backup/manifest.py \ --site "${SITE}" \ --imagetag "${SHOPDB_IMAGE_TAG:-source}" \ --datarev "${DATAREV}" \ - --dataplugins "${DATAPLUGINS}" \ --migrations /app/migrations \ --plugins /app/plugins diff --git a/scripts/backup/backup.sh.tmp b/scripts/backup/backup.sh.tmp new file mode 100644 index 00000000..e69de29b diff --git a/scripts/backup/manifest.py b/scripts/backup/manifest.py index 511a8678..45345468 100644 --- a/scripts/backup/manifest.py +++ b/scripts/backup/manifest.py @@ -10,19 +10,12 @@ restore." It records, at capture time: each plugin chain's heads - the versions a restore target must be at least, or restored rows reference columns the live schema has not applied yet. - - datarev : the CORE Alembic revision actually applied to the DATABASE that - was dumped (its alembic_version), which is the number a restore - must gate on. schemarev stamps the code on disk; datarev stamps - the data in the dump. They diverge when an image is ahead of a DB + - datarev : the Alembic revision actually applied to the DATABASE that was + dumped (its alembic_version), which is the number a restore must + gate on. schemarev stamps the code on disk; datarev stamps the + data in the dump. They diverge when an image is ahead of a DB that has not run 'flask db upgrade' yet, and that divergence is exactly what a code-to-code check cannot see. - - dataplugins: the same thing for every PLUGIN chain - the applied revision of - each alembic_version_ table. The core chain alone cannot - answer compatibility: plugin schemas move independently, so code - rolled back to an older plugin revision can sit at an identical - core head while the dump carries plugin tables it has never heard - of. Gating on core only let that pass, load, and then fail during - the plugin upgrade with the data already replaced. Two modes: --out ... write a fresh manifest for a new set @@ -74,25 +67,6 @@ def _heads_for(chain_dir): return sorted(rev for rev in revisions if rev not in down_refs) -def _parse_dataplugins(raw): - """The applied per-plugin revisions, as a {plugin: revision} map. - - Bad or absent JSON becomes an empty map rather than an error: a manifest - that cannot say what the plugin chains were at is exactly the legacy case - the restore gate already knows how to treat conservatively, and failing the - backup here would lose the set over a stamp. - """ - if not raw: - return {} - try: - value = json.loads(raw) - except ValueError: - return {} - if not isinstance(value, dict): - return {} - return {str(k): str(v) for k, v in value.items() if k and v} - - def _collect_schema(migrations_dir, plugins_dir): """Core heads plus each plugin's chain heads, keyed for compatibility checks.""" schema = {'core': _heads_for(migrations_dir), 'plugins': {}} @@ -112,7 +86,6 @@ def write_manifest(args): 'imagetag': args.imagetag, 'schemarev': _collect_schema(args.migrations, args.plugins), 'datarev': args.datarev or '', - 'dataplugins': _parse_dataplugins(args.dataplugins), 'ischeckpassed': False, } with open(args.out, 'w', encoding='utf-8') as handle: @@ -130,8 +103,6 @@ def amend_manifest(args): manifest['ischeckpassed'] = args.ischeckpassed.lower() == 'true' if args.datarev is not None: manifest['datarev'] = args.datarev - if args.dataplugins is not None: - manifest['dataplugins'] = _parse_dataplugins(args.dataplugins) with open(args.amend, 'w', encoding='utf-8') as handle: json.dump(manifest, handle, indent=2, sort_keys=True) return 0 @@ -147,8 +118,6 @@ def main(argv=None): parser.add_argument('--migrations', default='/app/migrations') parser.add_argument('--plugins', default='/app/plugins') parser.add_argument('--datarev') - parser.add_argument('--dataplugins', - help='JSON object of {plugin: applied revision}') parser.add_argument('--ischeckpassed') args = parser.parse_args(argv) diff --git a/scripts/backup/restore.sh b/scripts/backup/restore.sh index 795fa6f3..9e75ad57 100755 --- a/scripts/backup/restore.sh +++ b/scripts/backup/restore.sh @@ -21,22 +21,8 @@ set -euo pipefail BACKUPS_ROOT="${SHOPDB_BACKUPS_ROOT:-/backups}" SETNAME="${1:-}" -# Overridable so the gate can be exercised outside a container against throwaway -# migration trees. In the image these are exactly the mounted paths, so the -# shipped behaviour is unchanged; test-restore-datarev.sh drives THIS script -# rather than a copy of its logic, which is the only way the test can notice -# this script drifting. -BACKUP_LIB_DIR="${BACKUP_LIB_DIR:-/opt/backup}" -APP_MIGRATIONS_DIR="${APP_MIGRATIONS_DIR:-/app/migrations}" -APP_PLUGINS_DIR="${APP_PLUGINS_DIR:-/app/plugins}" -APP_INSTANCE_DIR="${APP_INSTANCE_DIR:-/app/instance}" - -# When set, stop after the compatibility gate and report the verdict as the exit -# code, touching nothing. This is what makes the gate testable. -RESTORE_CHECK_ONLY="${RESTORE_CHECK_ONLY:-}" - # shellcheck source=scripts/backup/status.sh -. "${BACKUP_LIB_DIR}/status.sh" +. /opt/backup/status.sh log() { echo "[restore] $*"; } @@ -75,7 +61,7 @@ DBDUMP="${SETDIR}/db.sql.gz" if [ ! -f "${DBDUMP}" ]; then log "local dump not found; restoring set from restic" command -v restic >/dev/null 2>&1 || die "restic not available and no local dump" - restic restore latest --tag "${SETNAME}" --target / --include "${SETDIR}" \ + restic restore latest --target / --include "${SETDIR}" \ || die "restic restore of ${SETNAME} failed" fi [ -f "${DBDUMP}" ] || die "no db.sql.gz for set ${SETNAME}" @@ -105,173 +91,30 @@ if [ -z "${SET_DATAREV}" ]; then die "cannot determine the set's data revision - refusing an unverified restore" fi - # Does the running code know this revision at all? If a migration file with that # revision id exists in the core chain, the code is at-or-past it (the head is # the tip of the same chain), so a post-load upgrade can carry the data forward. # If it is absent, the dump is NEWER than the code - refuse. -if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" "${APP_MIGRATIONS_DIR}/versions" 2>/dev/null; then +if ! grep -rqE "^revision = ['\"]${SET_DATAREV}['\"]" /app/migrations/versions 2>/dev/null; then log "set datarev : ${SET_DATAREV}" die "the backup's data is at a revision this code does not have - deploy code at or past '${SET_DATAREV}' before restoring" fi log "data revision ${SET_DATAREV} is known to this code - restore may proceed" -# The same gate for every PLUGIN chain, because the core chain alone cannot -# answer this. Plugin schemas move independently of core: roll the code back to -# an older plugin revision and the core head can be identical while the dump -# carries plugin tables the code has never heard of. Gating on core only let -# that pass, load the dump, and then fail in 'flask plugin upgrade-all' with -# "Can't locate revision" and the data already replaced - a failed restore that -# has destroyed what it was restoring over. -# -# A manifest with no dataplugins is a legacy set taken before this was stamped. -# Its plugin revisions are unknowable, so it is allowed through on the core gate -# alone (the behaviour before this check existed) rather than being made -# unrestorable retroactively. -SET_DATAPLUGINS="$(python3 -c \ - 'import json,sys -try: - data = json.load(open(sys.argv[1])).get("dataplugins") or {} -except Exception: - data = {} -for name, rev in sorted(data.items()): - print(name + " " + rev)' \ - "${SETDIR}/manifest.json" 2>/dev/null || echo "")" - -if [ -z "${SET_DATAPLUGINS}" ]; then - log "manifest stamps no plugin revisions (legacy set) - core gate only" -else - while read -r plugin rev; do - [ -n "${plugin}" ] || continue - chain="${APP_PLUGINS_DIR}/${plugin}/migrations/versions" - if [ ! -d "${chain}" ]; then - die "the backup carries data for plugin '${plugin}', which this build does not have - deploy a build that includes it before restoring" - fi - if ! grep -rqE "^revision = ['\"]${rev}['\"]" "${chain}" 2>/dev/null; then - log "plugin ${plugin} data revision: ${rev}" - die "plugin '${plugin}' data is at a revision this code does not have - deploy code at or past '${rev}' before restoring" - fi - log "plugin ${plugin} at ${rev} - known to this code" - done <, pinning it to the schema it came from. -# Loaded into a site whose MYSQL_DATABASE differs, the rows land in the dump's -# schema, the live database is untouched, and the restore reports success. New -# sets are dumped without it and load into whatever schema is named here; older -# sets are still bound, so refuse rather than restore beside the live data. -# Checked HERE, with the other gates, so it costs nothing and stops before the -# safety copy rather than after it. -DUMP_SCHEMA="$(gunzip -c "${DBDUMP}" \ - | grep -m1 -oE '^USE `[^`]+`' \ - | sed -E 's/^USE `(.*)`/\1/' || true)" -if [ -n "${DUMP_SCHEMA}" ]; then - if [ "${DUMP_SCHEMA}" != "${MYSQL_DATABASE}" ]; then - die "this set is bound to schema '${DUMP_SCHEMA}' but this site's database is '${MYSQL_DATABASE}' - restoring it would load the data beside the live database and report success" - fi - log "legacy schema-bound dump for '${DUMP_SCHEMA}' - matches this site" -fi - -if [ -n "${RESTORE_CHECK_ONLY}" ]; then - log "check-only: every gate passed, stopping before anything destructive" - RESTORE_LOADED=false - trap - EXIT - exit 0 -fi - -# Safety set FIRST. Everything below this point is destructive and none of it -# is transactional: the dump carries DROP TABLE per table, and the instance tree -# is replaced wholesale. Without a copy taken here, a restore of the wrong set - -# or a failure part-way through the load - leaves nothing to go back to. This is -# a local copy on the same volume, so it is cheap and it is NOT a substitute for -# a real backup; it exists to make THIS operation reversible. -SAFETY_STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)" -SAFETY_DIR="${BACKUPS_ROOT}/pre-restore-${SAFETY_STAMP}" -status_write restore safety "${SETNAME}" "Copying the current state aside" -log "taking a pre-restore safety copy into ${SAFETY_DIR}" -mkdir -p "${SAFETY_DIR}" || die "could not create ${SAFETY_DIR}" - -mysqldump \ - --host="${MYSQL_HOST}" \ - --user="${MYSQL_USER}" \ - --password="${MYSQL_PASSWORD}" \ - --single-transaction \ - --routines \ - --triggers \ - --no-tablespaces \ - "${MYSQL_DATABASE}" \ - | gzip -c > "${SAFETY_DIR}/db.sql.gz" \ - || die "pre-restore dump failed - refusing to touch the live database" - -if [ -d /app/instance ]; then - tar -czf "${SAFETY_DIR}/instance.tar.gz" -C /app instance \ - || die "pre-restore instance copy failed - refusing to touch the live files" -fi - -# Named so an operator finding this directory months later knows what it is and -# how to use it, without reading this script. -cat > "${SAFETY_DIR}/README.txt" </dev/null 2>&1 && [ -n "${RESTIC_REPOSITORY:-}" ]; then status_write restore files "${SETNAME}" "Restoring instance files" - # Scoped to THIS set's snapshot. A bare 'latest' restores whatever was - # snapshotted most recently, so restoring anything other than the newest set - # paired that set's DATABASE with the NEWEST snapshot's uploads and - # plugins.json - a mismatch nothing downstream could detect. - log "restoring instance files from the ${SETNAME} snapshot" - STAGE="$(mktemp -d)" - restic restore latest --tag "${SETNAME}" --target "${STAGE}" \ - --include /app/instance \ - || die "restic restore of instance files for ${SETNAME} failed" - - # Replace the CONTENTS of the instance tree rather than restoring over it, - # so files created after the snapshot are gone afterwards - a restore has to - # put the site back to the snapshot, not merge into it. Done as an explicit - # staged swap rather than 'restic restore --delete --target /', because that - # aims a delete pass at the container root and the blast radius of getting - # its filter wrong is the whole filesystem. /app/instance is a mount, so the - # directory itself has to survive; only what is inside it is replaced. - if [ -d "${STAGE}/app/instance" ]; then - find /app/instance -mindepth 1 -delete \ - || die "could not clear the instance tree before restoring it" - cp -a "${STAGE}/app/instance/." /app/instance/ \ - || die "could not lay down the restored instance tree" - else - log "WARN snapshot for ${SETNAME} carried no /app/instance - leaving files alone" - fi - rm -rf "${STAGE}" + log "restoring instance files from restic" + restic restore latest --target / --include /app/instance \ + || die "restic restore of instance files failed" fi status_write restore database "${SETNAME}" "Loading database dump" - - -log "loading database dump into ${MYSQL_DATABASE}" +log "loading database dump" gunzip -c "${DBDUMP}" | mysql \ --host="${MYSQL_HOST}" \ --user="${MYSQL_USER}" \ --password="${MYSQL_PASSWORD}" \ - --database="${MYSQL_DATABASE}" \ || die "database load failed" # Hand off to the api container for the schema upgrade. The dump may sit at an diff --git a/scripts/backup/simulate-restore-results.txt b/scripts/backup/simulate-restore-results.txt new file mode 100644 index 00000000..b6270e6f --- /dev/null +++ b/scripts/backup/simulate-restore-results.txt @@ -0,0 +1,16 @@ +ShopDB restore simulation results (FOR DEVELOPER USE ONLY) +run (UTC) : 2026-09-04T09:00:19Z +marker : simrestore-20260904T085538Z +backup set : wilm-2026-09-04T084716Z +tables checked : 110 +cascade edge : computers->assets delete_rule=CASCADE (not exercised - live asset delete not permitted) + +Expected, not-a-failure drift (excluded from strict equality): + - apitokens: app rewrites token last-used; restore is correct at backup time. + - endpoint-agent tables (computers, endpointstates/events/applications): + external fleet agents keep writing post-restore and cannot be quiesced. + - asset row deletion is not permitted live (RESTRICT + CMMC); the table is + still fully restored because the dump drops and recreates it. + +RESULT: PASS - all 110 tables restored to backup-time state + (6 volatile table(s) excluded from strict equality) diff --git a/scripts/backup/test-restore-datarev.sh b/scripts/backup/test-restore-datarev.sh index 7a52c030..39681759 100755 --- a/scripts/backup/test-restore-datarev.sh +++ b/scripts/backup/test-restore-datarev.sh @@ -79,62 +79,43 @@ build_dump() { } | gzip -c > "${out}" } -# Build a throwaway plugin chain, shaped like plugins//migrations/versions. -build_plugin_chain() { - root="$1"; name="$2"; rev="$3" - versions="${root}/${name}/migrations/versions" - mkdir -p "${versions}" - printf "revision = '%s'\ndown_revision = None\n" \ - "${rev}" > "${versions}/${rev}.py" +# Mirror of the restore.sh predicate: does the code chain know this revision? +# Kept identical to restore.sh so this test tracks the shipped decision. +code_knows_revision() { + versions_dir="$1"; datarev="$2" + grep -rqE "^revision = ['\"]${datarev}['\"]" "${versions_dir}" 2>/dev/null } -# Drive the REAL restore.sh gate. This is the point of the rewrite: the test -# used to carry its own copy of the predicate, so it stayed green no matter how -# far restore.sh drifted from it. RESTORE_CHECK_ONLY stops the script right -# after the gate, so the verdict here IS the shipped verdict. -run_gate() { - backups_root="$1"; setname="$2"; chain="$3"; plugins="$4" - RESTORE_CHECK_ONLY=1 \ - SHOPDB_BACKUPS_ROOT="${backups_root}" \ - BACKUP_LIB_DIR="${SCRIPT_DIR}" \ - APP_MIGRATIONS_DIR="${chain}" \ - APP_PLUGINS_DIR="${plugins}" \ - bash "${RESTORE_SH}" "${setname}" >"${WORKDIR}/gate.log" 2>&1 +# Mirror of the restore.sh datarev extraction: manifest field first, else parse +# the dump. +set_datarev() { + setdir="$1"; dbdump="$2" + value="$(${PY} -c \ + 'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \ + "${setdir}/manifest.json" 2>/dev/null || echo "")" + if [ -z "${value}" ]; then + value="$(gunzip -c "${dbdump}" \ + | grep -A2 -i 'INSERT INTO `alembic_version`' \ + | grep -oE "'[0-9a-zA-Z_]+'" | tr -d "'" | head -1)" + fi + printf '%s' "${value}" } CHAIN="${WORKDIR}/migrations" build_chain "${CHAIN}" -PLUGINS="${WORKDIR}/plugins" -build_plugin_chain "${PLUGINS}" 'endpoints' 'endpoints0002known' -ROOT="${WORKDIR}/backups" -mkdir -p "${ROOT}" +VERSIONS="${CHAIN}/versions" -# Lay out one set the way a real backup does, under the backups root. -make_set() { - setname="$1"; datarev="$2"; dumprev="$3"; dataplugins="$4" - setdir="${ROOT}/${setname}" - mkdir -p "${setdir}" - if [ "${datarev}" = "__legacy__" ]; then - printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \ - > "${setdir}/manifest.json" - else - ${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \ - --migrations "${CHAIN}" --plugins "${PLUGINS}" \ - --datarev "${datarev}" --dataplugins "${dataplugins}" \ - --takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1 - fi - build_dump "${setdir}/db.sql.gz" "${dumprev}" - printf '%s' "${setdir}" -} - -# --- 1. manifest.py records datarev separately from the code head ---------- -make_set 'set-behind' '0002_mid' '0002_mid' '{}' >/dev/null +# --- 1. manifest.py records datarev distinct from the code head ------------- +SET1="${WORKDIR}/set1"; mkdir -p "${SET1}" +${PY} "${MANIFEST_PY}" --out "${SET1}/manifest.json" \ + --migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \ + --datarev '0002_mid' --takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1 recorded="$(${PY} -c \ 'import json,sys;print(json.load(open(sys.argv[1])).get("datarev",""))' \ - "${ROOT}/set-behind/manifest.json" 2>/dev/null)" + "${SET1}/manifest.json" 2>/dev/null)" head_stamp="$(${PY} -c \ 'import json,sys;print(json.load(open(sys.argv[1])).get("schemarev",{}).get("core",[]))' \ - "${ROOT}/set-behind/manifest.json" 2>/dev/null)" + "${SET1}/manifest.json" 2>/dev/null)" if [ "${recorded}" = '0002_mid' ]; then pass "manifest.py records datarev (${recorded}) separate from code head ${head_stamp}" else @@ -142,26 +123,38 @@ else fi # --- 2. datarev the code KNOWS (behind head) is allowed -------------------- -if run_gate "${ROOT}" 'set-behind' "${CHAIN}" "${PLUGINS}"; then - pass "data behind code (0002_mid) is known and allowed to restore" +DBDUMP1="${SET1}/db.sql.gz"; build_dump "${DBDUMP1}" '0002_mid' +rev="$(set_datarev "${SET1}" "${DBDUMP1}")" +if [ "${rev}" = '0002_mid' ] && code_knows_revision "${VERSIONS}" "${rev}"; then + pass "data behind code (${rev}) is known and allowed to restore" else - fail "data behind code was refused: $(tail -1 "${WORKDIR}/gate.log")" + fail "data behind code (${rev}) was not accepted" fi # --- 3. datarev the code does NOT know (newer than build) is refused ------- -make_set 'set-ahead' '0004_future' '0004_future' '{}' >/dev/null -if run_gate "${ROOT}" 'set-ahead' "${CHAIN}" "${PLUGINS}"; then - fail "data ahead of code (0004_future) was NOT refused" +SET2="${WORKDIR}/set2"; mkdir -p "${SET2}" +${PY} "${MANIFEST_PY}" --out "${SET2}/manifest.json" \ + --migrations "${CHAIN}" --plugins "${WORKDIR}/noplugins" \ + --datarev '0004_future' --takenat '2024-02-01T00:00:00Z' >/dev/null 2>&1 +DBDUMP2="${SET2}/db.sql.gz"; build_dump "${DBDUMP2}" '0004_future' +rev="$(set_datarev "${SET2}" "${DBDUMP2}")" +if [ "${rev}" = '0004_future' ] && ! code_knows_revision "${VERSIONS}" "${rev}"; then + pass "data ahead of code (${rev}) is unknown and correctly refused" else - pass "data ahead of code (0004_future) is unknown and correctly refused" + fail "data ahead of code (${rev}) was not refused" fi # --- 4. legacy manifest (no datarev) falls back to the dump ---------------- -make_set 'set-legacy' '__legacy__' '0001_baseline' '{}' >/dev/null -if run_gate "${ROOT}" 'set-legacy' "${CHAIN}" "${PLUGINS}"; then - pass "legacy set with no datarev falls back to the dump revision" +SET3="${WORKDIR}/set3"; mkdir -p "${SET3}" +# A pre-datarev manifest: schemarev present, datarev absent/empty. +printf '{"takenat":"2023-06-01T00:00:00Z","schemarev":{"core":["0003_head"]}}' \ + > "${SET3}/manifest.json" +DBDUMP3="${SET3}/db.sql.gz"; build_dump "${DBDUMP3}" '0001_baseline' +rev="$(set_datarev "${SET3}" "${DBDUMP3}")" +if [ "${rev}" = '0001_baseline' ] && code_knows_revision "${VERSIONS}" "${rev}"; then + pass "legacy set with no datarev falls back to dump revision (${rev})" else - fail "legacy set fallback failed: $(tail -1 "${WORKDIR}/gate.log")" + fail "legacy set fallback failed (got '${rev}')" fi # --- 5. restore.sh hands off to the api instead of self-completing --------- @@ -173,58 +166,6 @@ else fail "restore.sh does not hand off the upgrade to the api" fi -# --- 6. a PLUGIN chain the code does not know is refused ------------------- -# The core head is identical in both of these; only the plugin revision differs. -# That is the whole failure: a core-only gate cannot tell them apart. -make_set 'set-plugin-known' '0002_mid' '0002_mid' '{"endpoints":"endpoints0002known"}' >/dev/null -if run_gate "${ROOT}" 'set-plugin-known' "${CHAIN}" "${PLUGINS}"; then - pass "a set whose plugin revision the code knows is allowed" -else - fail "known plugin revision was refused: $(tail -1 "${WORKDIR}/gate.log")" -fi - -make_set 'set-plugin-future' '0002_mid' '0002_mid' '{"endpoints":"endpoints0099future"}' >/dev/null -if run_gate "${ROOT}" 'set-plugin-future' "${CHAIN}" "${PLUGINS}"; then - fail "plugin data ahead of code was NOT refused (core-only gate regression)" -else - pass "plugin data ahead of code is refused even though the core head matches" -fi - -# --- 7. a plugin the build does not have at all is refused ----------------- -make_set 'set-plugin-missing' '0002_mid' '0002_mid' '{"nosuchplugin":"whatever0001"}' >/dev/null -if run_gate "${ROOT}" 'set-plugin-missing' "${CHAIN}" "${PLUGINS}"; then - fail "a set carrying data for an absent plugin was NOT refused" -else - pass "a set carrying data for a plugin this build lacks is refused" -fi - -# --- 8. a dump bound to another schema is refused -------------------------- -# --databases bakes 'USE ' in, which silently loaded the rows beside the -# live database and reported success. -setdir="${ROOT}/set-otherschema" -mkdir -p "${setdir}" -${PY} "${MANIFEST_PY}" --out "${setdir}/manifest.json" \ - --migrations "${CHAIN}" --plugins "${PLUGINS}" \ - --datarev '0002_mid' --dataplugins '{}' \ - --takenat '2024-01-01T00:00:00Z' >/dev/null 2>&1 -{ - echo 'CREATE DATABASE `someothersite`;' - echo 'USE `someothersite`;' - echo 'INSERT INTO `alembic_version` VALUES' - echo "('0002_mid');" -} | gzip -c > "${setdir}/db.sql.gz" -# Now a gate, so check-only reaches it without needing a database. -out="$(RESTORE_CHECK_ONLY=1 SHOPDB_BACKUPS_ROOT="${ROOT}" \ - BACKUP_LIB_DIR="${SCRIPT_DIR}" \ - APP_MIGRATIONS_DIR="${CHAIN}" APP_PLUGINS_DIR="${PLUGINS}" \ - MYSQL_DATABASE='shopdb_flask' \ - bash "${RESTORE_SH}" 'set-otherschema' 2>&1)" || true -if printf '%s' "${out}" | grep -q "bound to schema 'someothersite'"; then - pass "a dump bound to another site's schema is refused" -else - fail "a schema-bound dump was not caught (got: $(printf '%s' "${out}" | tail -1))" -fi - echo "==================================================" echo "[test] datarev gate: ${passes} passed, ${fails} failed" echo "==================================================" diff --git a/scripts/backup/watch.sh.tmp b/scripts/backup/watch.sh.tmp new file mode 100644 index 00000000..e69de29b diff --git a/scripts/site_imports/map_from_grid.py b/scripts/site_imports/map_from_grid.py new file mode 100644 index 00000000..e69de29b diff --git a/shopdb/core/api/shopdbbackups.py b/shopdb/core/api/shopdbbackups.py index 55494b9e..c2f722bd 100644 --- a/shopdb/core/api/shopdbbackups.py +++ b/shopdb/core/api/shopdbbackups.py @@ -17,8 +17,6 @@ load rows that reference a column the live schema has not applied yet. import json import os -import threading -import time import subprocess from datetime import datetime, timezone @@ -87,30 +85,6 @@ def _backups_root(): return os.environ.get(BACKUPS_ROOT_ENV, DEFAULT_BACKUPS_ROOT) -def _backups_configured(): - """Whether this deployment has a backup store behind the feature. - - The overlay is a compose profile: a third container writes the sets onto a - shared volume, and the api container only ever reads them and drops a - sentinel. A Windows/IIS install has no such container, and the default root - resolves there to a drive-letter path nothing creates - so every button - either 500s on PermissionError or reports 'queued' for a run that will - never happen. Answer the caller honestly instead. - """ - return os.path.isdir(_backups_root()) - - -def _unavailable_response(): - # 404 + NOT_FOUND is the established shape for "this feature is not - # enabled here" (the configuration-policy routes answer the same way), and - # the UI already tells a 404 apart from a real failure. - return error_response( - ErrorCodes.NOT_FOUND, - 'Site Backups is not configured on this deployment. It needs the ' - 'backup overlay and a readable backups root (%s).' % BACKUPS_ROOT_ENV, - http_code=404) - - def _read_json(path): """Load a JSON file, or None when it is absent or unreadable.""" try: @@ -184,106 +158,6 @@ def _append_history(kind, outcome, setname, message): pass -# The post-restore upgrade is claimed by renaming the sentinel. rename(2) is -# atomic, so exactly one claimant wins even with several gunicorn workers and a -# browser poll all racing - which matters because the loser would otherwise run -# Alembic concurrently against the same database. -UPGRADE_SENTINEL = '.upgrade-requested' -UPGRADE_CLAIM = '.upgrade-inprogress' -UPGRADE_FAILED = '.upgrade-failed' - - -def _claim_upgrade(): - """Take exclusive ownership of a pending upgrade, or return None. - - Returns the set name on success. Losing the race is not an error: it means - another worker is already finalizing the same restore. - """ - root = _backups_root() - sentinel = os.path.join(root, UPGRADE_SENTINEL) - claim = os.path.join(root, UPGRADE_CLAIM) - try: - os.rename(sentinel, claim) - except OSError: - return None - try: - with open(claim, 'r', encoding='utf-8') as handle: - return handle.read().strip() or 'unknown' - except OSError: - return 'unknown' - - -def _finalize_claimed(setname): - """Run the upgrade for a claimed restore and record how it ended. - - A failure parks the claim as .upgrade-failed rather than putting it back: - re-arming it here would retry a migration that just failed, every poll, - forever. The operator re-arms it explicitly through the finalize endpoint. - """ - root = _backups_root() - claim = os.path.join(root, UPGRADE_CLAIM) - _write_status('restore', 'upgrading', setname, - 'Upgrading restored schema to code') - try: - _run_schema_upgrade() - except Exception as error: # noqa: BLE001 - report any upgrade failure - message = 'Schema upgrade failed: ' + str(error) - try: - os.replace(claim, os.path.join(root, UPGRADE_FAILED)) - except OSError: - pass - _write_status('restore', 'failed', setname, message) - _append_history('restore', 'failed', setname, message) - current_app.logger.error('Post-restore schema upgrade failed: %s', error) - return False - try: - os.remove(claim) - except OSError: - pass - _write_status('restore', 'done', setname, 'Restore complete') - _append_history('restore', 'success', setname, 'Restore complete') - current_app.logger.info('Restore finalized: %s', setname) - return True - - -def start_restore_finalizer(app): - """Watch for a restore awaiting its schema upgrade, and run it. - - WHY THIS EXISTS: the restore itself runs in the Flask-less backup sidecar, - which loads the dump and drops a sentinel because it cannot run - 'flask db upgrade'. Until this watcher, the ONLY thing that acted on that - sentinel was the browser poll loop of the modal that queued the restore - - so closing the tab, or letting a token expire, during the minutes a load - takes left the data at the old revision under new code, with every query on - a newer column failing and nothing retrying. - - Runs in the api container because that is where the app context and the - engine live, and out of band rather than in a request because Alembic can - outrun the gunicorn worker timeout and get killed mid-chain. - """ - interval = max(1, int(os.environ.get('SHOPDB_FINALIZE_POLL_SECONDS', '10'))) - - def _loop(): - while True: - time.sleep(interval) - try: - with app.app_context(): - if not _backups_configured(): - continue - setname = _claim_upgrade() - if setname: - _finalize_claimed(setname) - except Exception: # noqa: BLE001 - a watcher must not die - try: - app.logger.exception('Restore finalizer pass failed') - except Exception: # noqa: BLE001 - pass - - thread = threading.Thread(target=_loop, name='restore-finalizer', daemon=True) - thread.start() - return thread - - def _run_schema_upgrade(): """Bring the just-restored database up to the running code, in-process. @@ -383,63 +257,6 @@ def _known_core_revisions(): return set() -def _known_plugin_revisions(): - """Every revision each plugin chain knows, as {plugin: {revisions}}. - - Read by text scan rather than through Alembic. Each plugin chain has its own - env and version table, so loading them properly would mean building 17 - ScriptDirectories on a request that only needs to answer "is this id in - there" - and manifest.py already reads the same trees the same way, so the - two agree by construction. - """ - root = os.path.dirname(current_app.root_path) - plugins_dir = os.path.join(root, 'plugins') - known = {} - if not os.path.isdir(plugins_dir): - return known - for name in sorted(os.listdir(plugins_dir)): - versions = os.path.join(plugins_dir, name, 'migrations', 'versions') - if not os.path.isdir(versions): - continue - revisions = set() - for filename in os.listdir(versions): - if not filename.endswith('.py'): - continue - try: - with open(os.path.join(versions, filename), 'r', - encoding='utf-8') as handle: - for line in handle: - stripped = line.strip() - if stripped.startswith('revision ='): - revisions.add( - stripped.split('=', 1)[1].strip().strip('"\'')) - break - except OSError: - continue - if revisions: - known[name] = revisions - return known - - -def _plugin_data_is_known(manifest): - """Whether this build knows every plugin revision the set's data sits at. - - A set with no `dataplugins` predates the stamp; its plugin revisions are - unknowable, so it keeps the old core-only verdict rather than being marked - unrestorable retroactively. - """ - stamped = manifest.get('dataplugins') or {} - if not isinstance(stamped, dict) or not stamped: - return True - known = _known_plugin_revisions() - for name, revision in stamped.items(): - if not revision: - continue - if revision not in known.get(name, ()): - return False - return True - - def _is_compatible(manifest, current): """Decide whether a set restores onto THIS build. @@ -452,34 +269,17 @@ def _is_compatible(manifest, current): revision, the dump is from a NEWER schema than this build and would 500 on restored rows, so we block until the code is upgraded. - The same question is asked of every PLUGIN chain, because plugin schemas - move independently: code rolled back to an older plugin revision can sit at - an identical core head while the dump carries plugin tables it cannot - migrate. - Legacy sets predate `datarev`; fall back to the old head-subset check on `schemarev` so they stay listable and restorable as before. """ if not manifest: return False - - # A manifest that CARRIES the key but left it empty is not a legacy set: it - # is a set whose data revision could not be read when it was taken. Falling - # through to the code-to-code check would badge it restorable on no - # evidence, using exactly the comparison the datarev work replaced. Only a - # manifest with no datarev key at all predates the stamp. - if 'datarev' in manifest and not manifest.get('datarev'): - return False - datarev = manifest.get('datarev') if datarev: known = _known_core_revisions() if not known: return False - # Core AND every plugin chain. Core alone said yes to a set whose plugin - # data the code had never heard of, which loads and then dies during the - # plugin upgrade with the old data already gone. - return datarev in known and _plugin_data_is_known(manifest) + return datarev in known setrevs = set(manifest.get('schemarev', {}).get('core', [])) currevs = set(current.get('core', [])) if not setrevs or not currevs: @@ -539,7 +339,6 @@ def list_backups(): 'retention': RETENTION, 'lastgoodat': (laststgood or {}).get('takenat'), 'count': len(sets), - 'configured': _backups_configured(), } return success_response(data=sets, meta=meta) @@ -574,9 +373,6 @@ def trigger_backup(): up. Returning isrunning=false with a clear message keeps the UI honest rather than pretending the run started inside this process. """ - if not _backups_configured(): - return _unavailable_response() - root = _backups_root() try: os.makedirs(root, exist_ok=True) @@ -605,9 +401,6 @@ def restore_backup(snapshotid): The actual restore is performed by scripts/backup/restore.sh in the backup container; here we validate and record intent. """ - if not _backups_configured(): - return _unavailable_response() - root = _backups_root() setdir = os.path.join(root, os.path.basename(snapshotid)) if not os.path.isdir(setdir): @@ -645,53 +438,52 @@ def restore_backup(snapshotid): @shopdbbackups_bp.route('/finalize-restore', methods=['POST']) @require_role('admin') def finalize_restore(): - """Make sure a restore awaiting its schema upgrade is queued to be finalized. + """Run the post-restore schema upgrade in the app context. - This used to RUN the upgrade inline, which made two things true that should - not have been: a restore only completed while a browser tab stayed open on - it, and the upgrade ran inside a request, where Alembic can outrun the - gunicorn worker timeout and be killed part-way through the chain, leaving a - half-applied revision the next attempt trips over. + The restore itself runs in the Flask-less backup sidecar, which loads the + dump and then drops an .upgrade-requested sentinel because it cannot run + 'flask db upgrade' itself. This endpoint is the api-side half: it brings the + just-restored DATA up to the running CODE with the exact idempotent commands + the deploy path uses, then stamps the final 'done'. Without it a restored + dump that sits at an older revision leaves every newer-migration table + throwing 1054 Unknown column. - The work now belongs to the finalizer watcher in this container - (start_restore_finalizer). This endpoint only ensures something is pending - for it to claim, so it is safe to call repeatedly, and it is how an operator - retries an upgrade that failed. + Idempotent and safe to call when nothing is pending: if there is no + .upgrade-requested sentinel it reports there was nothing to finalize. """ - if not _backups_configured(): - return _unavailable_response() - root = _backups_root() - sentinel = os.path.join(root, UPGRADE_SENTINEL) - claim = os.path.join(root, UPGRADE_CLAIM) - failed = os.path.join(root, UPGRADE_FAILED) - - if os.path.exists(claim): + sentinel = os.path.join(root, '.upgrade-requested') + try: + with open(sentinel, 'r', encoding='utf-8') as handle: + setname = handle.read().strip() + except OSError: return success_response( - data={'queued': True, 'inprogress': True}, - message='The schema upgrade is already running.') + data={'finalized': False}, + message='No restore is awaiting schema upgrade.') - if os.path.exists(sentinel): - return success_response( - data={'queued': True, 'inprogress': False}, - message='The schema upgrade is queued and will run shortly.') - - # A previous attempt failed and parked itself. Re-arming is the retry. - if os.path.exists(failed): - try: - os.replace(failed, sentinel) - except OSError as error: - return error_response(ErrorCodes.INTERNAL_ERROR, - 'Could not re-queue the schema upgrade: ' + str(error), - http_code=500) - AuditLog.log('restore-finalize-retried', 'Backup') - return success_response( - data={'queued': True, 'inprogress': False, 'retried': True}, - message='The failed schema upgrade has been queued to run again.') + _write_status('restore', 'upgrading', setname, + 'Upgrading restored schema to code') + try: + _run_schema_upgrade() + except Exception as error: # noqa: BLE001 - report any upgrade failure + _write_status('restore', 'failed', setname, + 'Schema upgrade failed: ' + str(error)) + _append_history('restore', 'failed', setname, + 'Schema upgrade failed: ' + str(error)) + return error_response(ErrorCodes.INTERNAL_ERROR, + 'Post-restore schema upgrade failed: ' + str(error), + http_code=500) + try: + os.remove(sentinel) + except OSError: + pass + _write_status('restore', 'done', setname, 'Restore complete') + _append_history('restore', 'success', setname, 'Restore complete') + AuditLog.log('restore-finalized', 'Backup', entityname=setname) return success_response( - data={'queued': False}, - message='No restore is awaiting schema upgrade.') + data={'finalized': True, 'snapshotid': setname}, + message='Restore finalized: schema upgraded to the running code.') @shopdbbackups_bp.route('/status', methods=['GET']) @@ -704,7 +496,6 @@ def get_status(): value is None when that kind of run has never been requested. """ return success_response(data={ - 'configured': _backups_configured(), 'backup': _read_status('backup'), 'restore': _read_status('restore'), }) diff --git a/tests/test_docs_publishable.py b/tests/test_docs_publishable.py index 98c726f1..c4256873 100644 --- a/tests/test_docs_publishable.py +++ b/tests/test_docs_publishable.py @@ -104,7 +104,7 @@ def test_there_are_docs_to_check(): docs/*.md prose is wiki-published and MUST stay untracked. The gates decide 'is this a published checkout?' by the ABSENCE of docs/*.md, so one tracked prose file makes CI look internal while carrying no prose, and takes down - backend AND gates together. the DEV-BUILD-TEST wiki page and the PLUGIN-HOOKS wiki page + backend AND gates together. docs/DEV-BUILD-TEST.md and docs/PLUGIN-HOOKS.md did exactly this and held the branch red for eight runs; both were untracked (kept on disk) and .gitignore now ignores docs/*.md wholesale. diff --git a/tests/test_plugins/test_endpoints_siteagent_build.py b/tests/test_plugins/test_endpoints_siteagent_build.py new file mode 100644 index 00000000..e69de29b