ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add one-click agent self-upgrade from the endpoint detail page

Browse Source 
An operator could see that an endpoint was behind the published build but
had no way to act on it. update_app resolves against Application rows, and
the agent lives in the version store, so there was nothing to push.

Bridge the two: the staged build is packaged and registered as a reserved
Application row, and a managed PowerShell script is seeded to install it.
The endpoint downloads that package over the existing collector-authenticated
app-package route, so no parallel delivery path is introduced.

The new route prepares the delivery and then DELEGATES to
enqueue_endpoint_command as a digest-pinned run_script. It deliberately does
not implement its own enqueue: a second path would duplicate, and eventually
drift from, the endpoint lock, the Configuration Policy gate, stale-command
reaping and the at-most-one-in-flight rule. A fleet-wide agent swap is the
last action that should skip a quarantine check. Tests pin that delegation
rather than only the happy path.

Because it lands as run_script, the caller needs endpoints.runscript on top
of endpoints.agentupdate. That is intended: this delivers arbitrary SYSTEM
PowerShell by the same mechanism as any other script push. The audit event
is written only on an actual 201, so a refused push does not record an
upgrade that never happened.

Replacing the agent is self-modifying the control plane - the process
receiving the command is the one being swapped. The script therefore
re-launches itself detached and exits immediately, because the agent runs
run_script as a direct child and holds its stdout/stderr pipes while
waiting on it. Expect the command to report queued rather than success:
the process that would report completion is stopped mid-run. Judge the
result by the version the endpoint reports afterward.

Tray relaunch does not use the installer's `explorer.exe <path>` trick.
That works in install-shopdbagent.ps1 only because the installer runs as
the user - note its own `-ne "SYSTEM"` guard. This script always runs as
SYSTEM in session 0, where the hand-off reaches no shell and exits 0, so
Start-Process reported success for a relaunch that never happened. It also
cannot descend into the session with a token-duplicating launch:
UserContextSpool documents that WTSQueryUserToken is blocked by endpoint
protection on these hosts. Instead a one-shot scheduled task targets the
pinned ShopFloor account, matching how the user-context spool already
resolves that user, and the tray must run as that account or the
mapped-drive probe reports the wrong (empty) drive table. The relaunch is
verified by polling for the process rather than assumed, and stays
best-effort: the Run key still starts the tray at next logon, and a late
tray is not worth rolling back a good agent upgrade.

panel_status now carries agentbehind/agentversion/agentcurrent so the UI
gates the action on a flag rather than parsing "(behind X)" out of a label.
The button is confirmed by typing UPDATE, naming both versions and the
service interruption, matching how run_script and system_restart are gated.

Verified on live endpoints: upgraded 1.13.0.0 and 1.15.0.0 machines to
1.16.0.0. run_script delivery, digest fetch and system context all predate
1.13.0.0, so no minimum agent version is declared for this path.
This commit is contained in:
benjamin.baker
2026-09-09 22:56:21 -04:00
parent d32293ccb2
commit 8be1c2379f
8 changed files with 957 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
frontend/src/api/index.js
View File

@@ -228,6 +228,18 @@ export const endpointsApi = {
}
return api.post(`/endpoints/${assetId}/commands`, body)
},
// Pushes the published agent build to one endpoint. Separate from
// enqueueCommand because the server prepares the delivery (packages the
// build, seeds the upgrade script) before queueing it as a pinned
// run_script; the caller has no script id to pass and must not invent one.
enqueueAgentUpgrade(assetId, opts = {}) {
const body = {}
if (opts.policyoverride) {
body.policyoverride = true
body.overridereason = opts.overridereason
}
return api.post(`/endpoints/${assetId}/agentupgrade`, body)
},
listCommands(assetId) {
return api.get(`/endpoints/${assetId}/commands`)
},

73
plugins/computers/frontend/views/PCDetail.vue
View File

@@ -218,7 +218,7 @@
:assetid="computer.assetid"
:accessmethods="computer.accessmethods || []"
>
<template #actions v-if="canRestart || canHoldPolicy">
<template #actions v-if="canRestart || canHoldPolicy || (canAgentUpdate && agentBehind)">
<!-- Admin only, and FIRST in the band. Placing a hold switches OFF
automatic policy enforcement, so it sits with the destructive
controls rather than with the read-only refresh. Red rather
@@ -233,6 +233,18 @@
<button v-if="canRestart" @click="confirmRestart"
class="btn btn-warning btn-sm"
:disabled="restarting">Restart Computer</button>
<!-- Only shown when this endpoint is actually behind the
published build, so it is a call to action rather than a
permanent button. Warning-styled: it interrupts the agent
service, but unlike Restart it does not take the machine
down, and it rolls back on failure. -->
<button v-if="canAgentUpdate && agentBehind"
@click="confirmAgentUpgrade"
class="btn btn-warning btn-sm"
:title="`Endpoint is on ${agentversion}, published build is ${agentcurrent}`"
:disabled="upgradingAgent">
{{ upgradingAgent ? 'Queueing...' : `Update Agent to ${agentcurrent}` }}
</button>
</template>
</AssetTriageBand>
</div>
@@ -1246,6 +1258,13 @@ const refreshing = ref(false)
// enforced again server-side; hiding the button is convenience, not the guard.
const canCommand = computed(() => authStore.hasPermission('endpoints.command'))
const canRestart = computed(() => authStore.hasPermission('endpoints.restart'))
// Replacing the agent is self-modifying the control plane, so it carries its
// own grant. The server also requires endpoints.runscript, because the upgrade
// lands as a script push - reflected here so the button does not offer an
// action the server will refuse.
const canAgentUpdate = computed(
() => authStore.hasPermission('endpoints.agentupdate')
&& authStore.hasPermission('endpoints.runscript'))
// Break-glass is admin-only and re-checked server-side against the CALLER, so a
// scoped token cannot inherit it from the admin who minted it. This flag only
@@ -1881,6 +1900,58 @@ const logCollectorVersionNotice = computed(() => {
+ `${reported}. Update the agent on this endpoint to use it.`
})
// True only when both versions parse AND the endpoint is strictly older than
// the published build. Silent when nothing is published or either version is a
// placeholder like '0.0.0-sample' - offering an upgrade off an unreadable
// version would be a false alarm, and the server applies the same rule.
const agentBehind = computed(() => {
if (!agentversion.value || !agentcurrent.value) return false
if (!parseAgentVersion(agentversion.value)) return false
if (!parseAgentVersion(agentcurrent.value)) return false
return !agentMeetsMinimum(agentversion.value, agentcurrent.value)
})
const upgradingAgent = ref(false)
// Typed confirmation, matching confirmRunScript and the destructive Corporate
// Tools: this stops the SYSTEM service that IS the command channel and swaps it
// underneath itself. An OK button is too easy to hit for something that, if it
// goes wrong, takes remote management of the machine with it. The prompt names
// BOTH versions so the operator can see exactly what is being replaced.
async function confirmAgentUpgrade() {
if (!computer.value?.assetid) return
const label = computer.value?.assetnumber || computer.value?.name || 'this PC'
const typed = prompt(`Update the ShopDB agent on ${label}?\n\n`
+ `${agentversion.value} will be replaced with ${agentcurrent.value}.\n`
+ 'The agent service stops and restarts, so this endpoint will briefly '
+ 'stop reporting. If the upgrade fails the agent rolls back to the '
+ 'current build.\n\nType UPDATE to confirm.')
if (typed !== 'UPDATE') return
upgradingAgent.value = true
try {
const response = await endpointsApi.enqueueAgentUpgrade(computer.value.assetid)
// "Queued", not "updated": the agent collects this on its next check-in and
// the swap happens afterwards. Claiming success here would assert an
// outcome nothing has observed.
toast.success(`Agent update queued for ${label}`)
const commandId = response?.data?.data?.endpointcommandid
if (commandId) {
await waitForCommand(commandId)
// Re-read liveness so the version and the behind flag reflect what the
// endpoint now reports, rather than leaving a stale "behind" badge on a
// machine that just upgraded.
await loadLiveness(computer.value.assetid)
await assetPanels.value?.refresh()
}
} catch (error) {
toast.error(apiError(error, `Failed to queue the agent update for ${label}`))
} finally {
upgradingAgent.value = false
}
}
// History for this endpoint. Metadata only - bodies are fetched one at a time,
// because 20 rows of multi-hundred-KB logs would make opening the tab far more
// expensive than reading the one log that is actually wanted.

9
plugins/endpoints/api/agentregistry.py
View File

@@ -38,6 +38,15 @@ def _package_path(appid):
return os.path.join(directory, str(appid), AGENT_PACKAGE_FILENAME)
def get_agent_application():
"""The reserved agent Application row, or None when nothing is registered.
Read-only lookup for callers that need the appid to build a delivery URL
without causing a row to spring into existence as a side effect.
"""
return Application.query.filter_by(appname=AGENT_APPNAME).first()
def _get_or_create_app():
"""Return the reserved agent Application row, creating it when absent.

136
plugins/endpoints/api/agentupgradescript.py Normal file
View File

@@ -0,0 +1,136 @@
"""Seed the agent self-upgrade script into the ShopDB script library.
run_script only delivers bodies that live in the script library, pinned by
SHA-256. The upgrade template therefore has to exist as a real Script row with
a stored body before any upgrade can be pushed.
The template is source-controlled at
plugins/endpoints/api/templates/upgrade-shopdbagent.ps1 and seeded from there
rather than typed into the library by hand, so the reviewed script and the one
endpoints actually run are the same artifact. Editing the row's body in the UI
is pointless - the next seed overwrites it.
It lives under the plugin, not under agent/, because the Docker image copies
shopdb/, plugins/, migrations/ and scripts/ but NOT agent/. A template read
from agent/ resolves fine on a developer checkout and then fails in the
deployed container, which is the worst place to discover it.
One placeholder is rendered at seed time: the agent application's appid, which
only exists once agentregistry has created the row. Rendering happens BEFORE
the digest is computed, so the stored body and its sha256 always agree; a
substitution done after hashing would fail the agent's verification.
"""
import hashlib
import os
from shopdb.api import Script, db, script_body_location
from . import agentregistry
# Reserved script name. Unique in the library, which makes seeding idempotent
# and keeps a hand-made script from colliding with the managed one.
UPGRADE_SCRIPT_NAME = 'ShopDB Endpoint Agent Self-Upgrade'
_UPGRADE_DESCRIPTION = (
'Managed: downloads the current published agent bundle and replaces the '
'installed agent in place, then restarts the service. Seeded from source '
'control - edits here are overwritten.'
)
# Substituted with the agent application's appid so the script can fetch its
# payload from the existing collector-authenticated app-package route.
_APPID_TOKEN = '{{AGENT_APPID}}'
_TEMPLATE_FILENAME = 'upgrade-shopdbagent.ps1'
def template_path():
return os.path.join(os.path.dirname(os.path.abspath(__file__)),
'templates', _TEMPLATE_FILENAME)
def _script_dir():
"""Where script bodies live. Resolved through the contract surface helper
so this cannot drift from the directory the download route serves from."""
directory, _prefix = script_body_location()
return directory
def render_template(appid):
"""Return the upgrade script body with the appid substituted.
Raises if the placeholder is missing: that means the template was edited
in a way that breaks the download, and seeding a script that would fetch
from a literal '{{AGENT_APPID}}' path is worse than refusing.
"""
with open(template_path(), 'r', encoding='utf-8') as handle:
body = handle.read()
if _APPID_TOKEN not in body:
raise ValueError(
'upgrade template is missing the {} placeholder; refusing to seed '
'a script that cannot resolve its package'.format(_APPID_TOKEN))
return body.replace(_APPID_TOKEN, str(appid))
def seed_upgrade_script():
"""Create or refresh the managed upgrade script row and its body.
Returns a dict describing the seeded script, or None when the agent
application row does not exist yet (nothing to point the download at).
Idempotent: unchanged input rewrites nothing and leaves the digest alone.
"""
app = agentregistry.get_agent_application()
if app is None:
return None
body = render_template(app.appid).encode('utf-8')
digest = hashlib.sha256(body).hexdigest()
script = Script.query.filter_by(scriptname=UPGRADE_SCRIPT_NAME).first()
created = script is None
if created:
script = Script(
scriptname=UPGRADE_SCRIPT_NAME,
scriptdescription=_UPGRADE_DESCRIPTION,
# SYSTEM: the script stops a service, writes into Program Files,
# and edits HKLM. None of that is available in a user context.
runcontext='system',
# Failures always report; success is opt-in. Capture it here
# because an upgrade that "succeeded" but rolled back internally
# is only diagnosable from the transcript.
captureoutput=True,
)
db.session.add(script)
db.session.flush()
# Body filename follows the library convention (script-<id>.ps1) so the
# collector download route serves it like any other script.
filename = 'script-{}.ps1'.format(script.scriptid)
path = os.path.join(_script_dir(), filename)
unchanged = (script.filename == filename and script.sha256 == digest
and os.path.isfile(path))
if not unchanged:
os.makedirs(_script_dir(), exist_ok=True)
# Temp-then-replace so an interrupted write cannot leave a truncated
# body at the path the row's digest already claims to describe.
temppath = path + '.tmp'
with open(temppath, 'wb') as handle:
handle.write(body)
os.replace(temppath, path)
script.filename = filename
script.sha256 = digest
script.scriptdescription = _UPGRADE_DESCRIPTION
db.session.commit()
return {
'scriptid': script.scriptid,
'scriptname': script.scriptname,
'created': created,
'updated': not unchanged,
'sha256': digest,
'bytes': len(body),
'appid': app.appid,
}

105
plugins/endpoints/api/routes.py
View File

@@ -569,10 +569,11 @@ def panel_status(asset_id: int):
# "behind" from a placeholder like '0.0.0-sample' would be a false alarm.
agentvalue = state.agentversion
published = agentstore.current_version('agent')
if (agentvalue and published
and parse_agent_version(agentvalue)
and parse_agent_version(published)
and not agent_meets_minimum(agentvalue, published)):
agentbehind = (agentvalue and published
and parse_agent_version(agentvalue)
and parse_agent_version(published)
and not agent_meets_minimum(agentvalue, published))
if agentbehind:
agentvalue = '{} (behind {})'.format(agentvalue, published)
fields = [
@@ -609,7 +610,17 @@ def panel_status(asset_id: int):
'{} monitor(s) expected running but not reported: {}'.format(
len(drift), names)))
return success_response({'fields': fields})
# Machine-readable companion to the "(behind X)" text above. The UI needs to
# decide whether to offer the upgrade action, and parsing that decision back
# out of a display string would tie the button to the wording of a label.
# Emitted as a sibling of fields so nothing about the existing rendering
# changes; an older UI simply ignores it.
return success_response({
'fields': fields,
'agentbehind': bool(agentbehind),
'agentversion': state.agentversion,
'agentcurrent': published,
})
@endpoints_bp.route('/asset/<int:asset_id>/hardware', methods=['GET'])
@@ -2975,6 +2986,90 @@ def enqueue_endpoint_command(asset_id: int):
message='Command queued', http_code=201)
@endpoints_bp.route('/<int:asset_id>/agentupgrade', methods=['POST'])
@require_permission('endpoints.agentupdate')
def enqueue_agent_upgrade(asset_id: int):
"""Push the published agent build to one endpoint.
Replacing the agent is self-modifying the control plane: the process that
receives this command is the one being swapped. It therefore sits behind
endpoints.agentupdate rather than endpoints.command or endpoints.deploy -
see the permission's rationale in plugin.py.
This does NOT enqueue directly. It prepares the delivery (packages the
current build, seeds/refreshes the upgrade script) and then delegates to
enqueue_endpoint_command as a run_script push. Writing a second enqueue
path here would duplicate - and eventually drift from - the endpoint lock
check, the Configuration Policy gate, stale-command reaping and the
at-most-one-in-flight rule. Those are exactly the protections a
fleet-wide agent swap must not be the one action to skip.
Because it lands as run_script, the caller needs endpoints.runscript too
(enforced by the delegate as the elevated permission for that action).
That is deliberate: this delivers arbitrary SYSTEM PowerShell by the same
mechanism as any other script push.
"""
from . import agentregistry, agentupgradescript
# Refresh the package and the script body from the CURRENT staged build
# before queueing, so an upgrade cannot ship a stale payload if the row was
# written before the latest upload. Both operations are idempotent.
try:
registration = agentregistry.register_current_agent_package()
except Exception as exc: # noqa: BLE001 - surfaced, not swallowed
db.session.rollback()
return error_response(ErrorCodes.VALIDATION_ERROR,
f'Could not package the agent build: {exc}')
if registration is None:
return error_response(
ErrorCodes.VALIDATION_ERROR,
'No agent build is published. Upload an agent binary before '
'pushing an upgrade.')
try:
seeded = agentupgradescript.seed_upgrade_script()
except Exception as exc: # noqa: BLE001 - surfaced, not swallowed
db.session.rollback()
return error_response(ErrorCodes.VALIDATION_ERROR,
f'Could not prepare the upgrade script: {exc}')
if seeded is None:
return error_response(ErrorCodes.VALIDATION_ERROR,
'The agent upgrade script could not be prepared.')
# Rewrite the request body to the equivalent run_script push and hand it to
# the real enqueue route. The policyoverride/overridereason fields are
# forwarded untouched so break-glass keeps behaving exactly as it does for
# a normal script push, including its admin-role and reason requirements.
payload = request.get_json(silent=True) or {}
forwarded = {
'action': 'run_script',
'target': 'script:{}'.format(seeded['scriptid']),
}
if payload.get('policyoverride'):
forwarded['policyoverride'] = True
forwarded['overridereason'] = payload.get('overridereason') or ''
# Swap the parsed JSON for the delegate. Flask caches the parsed body on
# the request, so setting the cache is what makes get_json() return this.
request._cached_json = (forwarded, forwarded)
response = enqueue_endpoint_command(asset_id)
# Audit only a push that actually queued. The delegate returns a non-201
# for a lock, a policy denial or an in-flight command, and recording those
# as upgrades would put events in the log for pushes that never happened.
status = getattr(response, 'status_code', None)
if status == 201:
AuditLog.log(
'created', 'AgentUpgrade', entityid=asset_id,
entityname='agent {}'.format(registration['agentversion']),
details='agent upgrade queued for asset {} (version {}, script {})'
.format(asset_id, registration['agentversion'],
seeded['scriptid']))
db.session.commit()
return response
@endpoints_bp.route('/monitors/resolved', methods=['GET'])
def resolved_monitors_for_agent():
"""Agent watch list: the monitors this endpoint should report on.

332
plugins/endpoints/api/templates/upgrade-shopdbagent.ps1 Normal file
View File

@@ -0,0 +1,332 @@
# upgrade-shopdbagent.ps1
# ShopDB Endpoint Agent SELF-UPGRADE template. Delivered to an endpoint by the
# run_script command and executed by the agent as SYSTEM.
#
# WHY THIS TAKES NO PARAMETERS
# run_script executes "powershell.exe -File <script>" with no argument list, so
# there is no channel to pass BaseUrl/CollectorKey in. That is not a problem
# here: an upgrade by definition targets an endpoint that is ALREADY installed
# and therefore already holds its own config. This script READS that config and
# never rewrites it - the values stay exactly as provisioned, so an upgrade can
# not silently repoint an endpoint at a different server or key. A fresh
# install is still install-shopdbagent.ps1's job.
#
# WHY IT RE-LAUNCHES ITSELF DETACHED
# The agent process is what runs this script, and the upgrade must stop that
# same agent to release the lock on shopdbagent.exe (single-file self-contained
# exes are held open while running). If the script stayed a child of the agent,
# stopping the service would kill the script mid-copy and leave the install
# directory half-written. The first pass therefore copies itself to a temp
# location, relaunches detached, and exits immediately so the agent sees a
# clean, fast result. All real work happens in the detached second pass.
#
# REPORTING
# Success is NOT reported through the command result - by the time the upgrade
# succeeds, the agent that would have reported it has been restarted. The new
# version is confirmed on the next check-in. This script's own transcript is
# the local record; see $LogFile below.
[CmdletBinding()]
param(
# Set only when the script relaunches itself. Operators never pass this.
[switch] $Detached
)
$ErrorActionPreference = 'Stop'
$ServiceName = 'ShopDbEndpointAgent'
$InstallDir = 'C:\Program Files\ShopDB\EndpointAgent'
$TrayExeName = 'shopdb-endpoint-tray.exe'
$AgentExeName = 'shopdbagent.exe'
$StateDir = Join-Path $env:ProgramData 'ShopDB\EndpointAgent'
$WorkDir = Join-Path $StateDir 'upgrade'
$LogDir = Join-Path $StateDir 'logs'
$LogFile = Join-Path $LogDir ('upgrade-{0}.log' -f (Get-Date -Format 'yyyyMMdd-HHmmss'))
# Server-relative path of the managed agent bundle. This reuses the EXISTING
# agent-facing app-package route rather than adding a parallel download: it
# already authenticates with the collector key and already resolves an
# application's installpath, which is precisely what upload registration
# stamped for the agent bundle. The appid is substituted by the server when
# this template is rendered, because it is assigned when the row is created.
$PackagePath = '/api/endpoints/apppackage/app/{{AGENT_APPID}}'
function Write-Log {
param([string] $Message, [string] $Level = 'INFO')
$line = '{0} [{1}] {2}' -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Level, $Message
Write-Host $line
try { Add-Content -Path $LogFile -Value $line -ErrorAction Stop } catch { }
}
# Read a value from the 64-bit HKLM view explicitly. If this runs under 32-bit
# PowerShell the HKLM: PSDrive silently redirects to WOW6432Node, where the
# 64-bit agent's config is not, and the upgrade would look unconfigured.
function Get-Hklm64Value {
param([string] $SubKey, [string] $Name)
$base = [Microsoft.Win32.RegistryKey]::OpenBaseKey(
[Microsoft.Win32.RegistryHive]::LocalMachine,
[Microsoft.Win32.RegistryView]::Registry64)
try {
$key = $base.OpenSubKey($SubKey)
if (-not $key) { return $null }
try { return $key.GetValue($Name) } finally { $key.Dispose() }
}
finally { $base.Dispose() }
}
# Config resolution mirrors the agent's own order of preference, so the upgrade
# reaches the same server the agent is actually talking to. Registry first;
# appsettings.json is the fallback for locked-down hosts (CyberArk/EPM) that
# gate the HKLM hive against reads.
function Get-AgentConfig {
$baseUrl = Get-Hklm64Value -SubKey 'SOFTWARE\GE\ShopDB' -Name 'BaseUrl'
$apiKey = Get-Hklm64Value -SubKey 'SOFTWARE\GE\ShopDB' -Name 'CollectorKey'
if ([string]::IsNullOrWhiteSpace($baseUrl) -or [string]::IsNullOrWhiteSpace($apiKey)) {
$settings = Join-Path $InstallDir 'appsettings.json'
if (Test-Path $settings) {
try {
$json = Get-Content $settings -Raw | ConvertFrom-Json
if ([string]::IsNullOrWhiteSpace($baseUrl)) { $baseUrl = $json.baseurl }
if ([string]::IsNullOrWhiteSpace($apiKey)) { $apiKey = $json.apikey }
Write-Log 'read config from appsettings.json (registry unavailable or incomplete)'
}
catch {
Write-Log "could not parse appsettings.json: $($_.Exception.Message)" 'WARN'
}
}
}
if ([string]::IsNullOrWhiteSpace($baseUrl) -or [string]::IsNullOrWhiteSpace($apiKey)) {
throw 'No BaseUrl/CollectorKey found in the registry or appsettings.json. This endpoint does not look installed; run install-shopdbagent.ps1 instead.'
}
return @{ BaseUrl = $baseUrl.TrimEnd('/'); ApiKey = $apiKey }
}
# ---------------------------------------------------------------------------
# First pass: relaunch detached and get out of the way. See header.
# ---------------------------------------------------------------------------
if (-not $Detached) {
New-Item -ItemType Directory -Force -Path $WorkDir, $LogDir | Out-Null
$selfCopy = Join-Path $WorkDir 'upgrade-shopdbagent.ps1'
Copy-Item -Path $PSCommandPath -Destination $selfCopy -Force
# Copied out of the agent's own temp dir on purpose: the delivered script
# may be cleaned up by the agent once the command returns, which would
# delete this file out from under the detached run.
Start-Process -FilePath 'powershell.exe' `
-ArgumentList '-NoProfile', '-ExecutionPolicy', 'Bypass', `
'-File', "`"$selfCopy`"", '-Detached' `
-WindowStyle Hidden | Out-Null
Write-Host 'Upgrade handed off to a detached process; this agent will restart shortly.'
exit 0
}
# ---------------------------------------------------------------------------
# Second pass (detached): the actual upgrade.
# ---------------------------------------------------------------------------
New-Item -ItemType Directory -Force -Path $WorkDir, $LogDir | Out-Null
Write-Log '== ShopDB Endpoint Agent self-upgrade =='
try {
$config = Get-AgentConfig
Write-Log "server: $($config.BaseUrl)"
# 1. Download BEFORE stopping anything. A failed download must leave the
# running agent untouched rather than stranding the endpoint with a
# stopped service and no replacement binaries.
$zipPath = Join-Path $WorkDir 'agent-package.zip'
if (Test-Path $zipPath) { Remove-Item $zipPath -Force }
$url = $config.BaseUrl + $PackagePath
Write-Log "downloading package from $url ..."
# TLS 1.2 explicitly: Windows PowerShell 5.1 still defaults to older
# protocols the server will refuse.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri $url -Headers @{ 'X-API-Key' = $config.ApiKey } `
-OutFile $zipPath -UseBasicParsing -TimeoutSec 600
$size = (Get-Item $zipPath).Length
Write-Log "downloaded $size bytes"
if ($size -lt 1MB) {
throw "Package is only $size bytes; refusing to install a payload this small."
}
# 2. Unpack to a staging dir and validate the payload BEFORE touching the
# install. Expanding straight over Program Files would leave a partially
# replaced install if the archive turned out to be short a binary.
$stage = Join-Path $WorkDir 'stage'
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
New-Item -ItemType Directory -Force -Path $stage | Out-Null
Expand-Archive -Path $zipPath -DestinationPath $stage -Force
$newAgent = Join-Path $stage $AgentExeName
if (-not (Test-Path $newAgent)) {
throw "$AgentExeName not found in the downloaded package; aborting before any change."
}
$newTray = Join-Path $stage $TrayExeName
$haveTray = Test-Path $newTray
Write-Log ("payload validated (agent{0})" -f $(if ($haveTray) { ' + tray' } else { ', no tray' }))
# 3. Keep the current binaries so a failed swap can be undone. The config
# files are deliberately NOT part of this - they are never modified.
$backup = Join-Path $WorkDir 'backup'
if (Test-Path $backup) { Remove-Item $backup -Recurse -Force }
New-Item -ItemType Directory -Force -Path $backup | Out-Null
foreach ($name in @($AgentExeName, $TrayExeName)) {
$existing = Join-Path $InstallDir $name
if (Test-Path $existing) { Copy-Item $existing -Destination $backup -Force }
}
Write-Log 'backed up current binaries'
# 4. Stop the service and kill lingering processes. Until this completes
# the exes are locked and the copy would fail with a file-in-use error.
$svc = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
if ($svc -and $svc.Status -ne 'Stopped') {
Write-Log "stopping service $ServiceName ..."
try { Stop-Service -Name $ServiceName -Force -ErrorAction Stop }
catch { Write-Log "could not stop service: $($_.Exception.Message)" 'WARN' }
}
$trayBase = [System.IO.Path]::GetFileNameWithoutExtension($TrayExeName)
foreach ($name in @('shopdbagent', $trayBase)) {
foreach ($p in (Get-Process -Name $name -ErrorAction SilentlyContinue)) {
Write-Log "killing $($p.ProcessName) (pid $($p.Id))"
try { Stop-Process -Id $p.Id -Force -ErrorAction Stop } catch { }
}
}
Start-Sleep -Milliseconds 750
# 5. Swap the binaries. On failure, restore the backup and restart, so a
# broken upgrade leaves a RUNNING old agent rather than a dead endpoint
# that can no longer be reached to fix remotely.
try {
Copy-Item -Path $newAgent -Destination $InstallDir -Force
if ($haveTray) { Copy-Item -Path $newTray -Destination $InstallDir -Force }
Write-Log 'binaries replaced'
}
catch {
Write-Log "copy failed: $($_.Exception.Message); rolling back" 'ERROR'
Get-ChildItem $backup -File | ForEach-Object {
Copy-Item $_.FullName -Destination $InstallDir -Force
}
Start-Service -Name $ServiceName -ErrorAction SilentlyContinue
throw
}
# 6. Clear Mark-of-the-Web. Files extracted from a downloaded archive carry
# a Zone.Identifier stream; SmartScreen blocks the unsigned exe on first
# run, which would leave the service unable to start.
Get-ChildItem -Path $InstallDir -File | ForEach-Object {
try { Unblock-File -Path $_.FullName -ErrorAction Stop } catch { }
}
# 7. Restart and verify it stays up. A service that starts and immediately
# faults is a failed upgrade, so confirm rather than assume.
Write-Log "starting service $ServiceName ..."
Start-Service -Name $ServiceName
Start-Sleep -Seconds 5
$svc = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
if (-not $svc -or $svc.Status -ne 'Running') {
Write-Log 'service did not stay running; rolling back' 'ERROR'
Get-ChildItem $backup -File | ForEach-Object {
Copy-Item $_.FullName -Destination $InstallDir -Force
}
Start-Service -Name $ServiceName -ErrorAction SilentlyContinue
throw 'Upgraded service failed to start; rolled back to the previous build.'
}
$installed = (Get-Item (Join-Path $InstallDir $AgentExeName)).VersionInfo.FileVersion
Write-Log "upgrade complete; agent version now $installed"
# 8. Relaunch the tray for the interactive user.
#
# This CANNOT use the installer's `explorer.exe <path>` trick. That works in
# install-shopdbagent.ps1 only because the installer runs AS THE USER - note
# its own guard, `if ($interactiveUser -ne "SYSTEM")`. This script always
# runs as SYSTEM in session 0, where `explorer.exe <path>` hands the request
# to a shell that is not reachable from here and then exits 0. Start-Process
# would report success and we would log a relaunch that never happened,
# which is exactly the bug this replaces.
#
# It also must NOT descend into the session with a token-duplicating launch.
# UserContextSpool documents that WTSQueryUserToken is a privileged operation
# that endpoint protection blocks on these hardened hosts - the service tried
# that direction before and it produced nothing on a live endpoint.
#
# So the tray is started the same way its own data path is pinned: against
# the known ShopFloor account. Task Scheduler performs the session/token
# transition, and an Interactive principal with no RunLevel starts the tray
# at the user's normal integrity - preserving the de-elevation the installer
# wanted, because an elevated unsigned tray trips the CyberArk/EPM broker.
#
# Best-effort by design: the agent is already upgraded and running, and the
# HKLM Run key the installer wrote still starts the tray at next logon. A
# late tray is not worth failing - or rolling back - a good agent upgrade.
if ($haveTray) {
$trayPath = Join-Path $InstallDir $TrayExeName
# Same pinned account the spool uses (FileLoggerProvider.ShopFloorUserName):
# every shop-floor endpoint signs in as this local account, and the tray
# must run as it or the mapped-drive probe reports the wrong account's
# (empty) drive table. Only relaunch if that profile is actually present.
$trayUser = 'ShopFloor'
$trayUserProfile = Join-Path $env:SystemDrive (Join-Path 'Users' $trayUser)
if (Test-Path $trayUserProfile) {
$taskName = 'ShopDB tray upgrade relaunch'
try {
# Clear a leftover from an interrupted earlier upgrade so we do
# not stack duplicates.
if (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) {
Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue
}
$action = New-ScheduledTaskAction -Execute $trayPath
$principal = New-ScheduledTaskPrincipal -UserId $trayUser -LogonType Interactive
$settings = New-ScheduledTaskSettingsSet `
-AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
-ExecutionTimeLimit ([TimeSpan]::Zero)
Register-ScheduledTask -TaskName $taskName -Action $action `
-Principal $principal -Settings $settings -Force | Out-Null
Start-ScheduledTask -TaskName $taskName
# Verify rather than assume. The old code logged success off a
# Start-Process that could not fail, so a missing tray looked
# like a clean upgrade. Poll for the real process instead.
$trayBaseName = [System.IO.Path]::GetFileNameWithoutExtension($TrayExeName)
$running = $null
for ($i = 0; $i -lt 10; $i++) {
Start-Sleep -Milliseconds 500
$running = Get-Process -Name $trayBaseName -ErrorAction SilentlyContinue
if ($running) { break }
}
if ($running) { Write-Log "tray relaunched as $trayUser" }
else {
Write-Log ('tray did not appear after relaunch; it will start at next ' +
'logon via the HKLM Run key') 'WARN'
}
# One-shot hand-off, not a persistent artifact: leaving it would
# surface in the uninstaller's leftover sweep and in task audits.
Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue
}
catch {
Write-Log ("could not relaunch tray: $($_.Exception.Message); " +
'it will start at next logon') 'WARN'
}
}
else {
Write-Log ("no $trayUser profile at $trayUserProfile; " +
'leaving the tray to start at next logon') 'WARN'
}
}
Remove-Item $stage -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item $zipPath -Force -ErrorAction SilentlyContinue
exit 0
}
catch {
Write-Log "UPGRADE FAILED: $($_.Exception.Message)" 'ERROR'
exit 1
}

15
shopdb/api/__init__.py
View File

@@ -118,6 +118,20 @@ def application_package_location():
return _apppackage_dir(), APP_PACKAGE_URL_PREFIX
def script_body_location():
"""Where script bodies live, and the URL prefix that serves them.
Returns (directory, urlprefix). Same reasoning as
application_package_location: a plugin that seeds a script into the
library must write the body where the download route reads it from, and
reconstructing that path by hand lets the two drift apart silently.
Imported lazily to avoid the shopdb.core.api -> shopdb.api cycle.
"""
from shopdb.core.api.scripts import _script_dir, SCRIPT_URL_PREFIX
return _script_dir(), SCRIPT_URL_PREFIX
# Serving an uploaded file safely. A plugin that serves user-supplied bytes must
# not have to remember the headers that keep an SVG or an HTML file from running
# as script in this origin - see shopdb/utils/uploads.py.
@@ -408,6 +422,7 @@ __all__ = [
'revoke_service_tokens',
'collector_authorized',
'application_package_location',
'script_body_location',
'SupportTeam',
# Serving uploads
'send_upload',

281
tests/test_plugins/test_endpoints_agent_upgrade_route.py Normal file
View File

@@ -0,0 +1,281 @@
"""The agent self-upgrade enqueue route.
Replacing the agent is self-modifying the control plane: the process that
receives the command is the one being swapped. The route therefore sits behind
endpoints.agentupdate, and - more importantly - it does NOT implement its own
enqueue. It prepares the delivery and delegates to enqueue_endpoint_command so
the endpoint lock, the Configuration Policy gate, stale-command reaping and the
at-most-one-in-flight rule all still apply.
These tests pin that delegation. A parallel enqueue path that quietly skipped a
lock or the in-flight rule would be invisible in normal use and would show up
as a fleet-wide agent swap landing on a quarantined machine.
The store is isolated per test: create_app('testing') resolves instance_path to
the real repo instance/, so without redirection these would stage binaries into
the developer's actual agent store.
"""
import os
import tempfile
import pytest
from werkzeug.security import generate_password_hash
from plugins.endpoints.api import agentstore, agentregistry, agentupgradescript
UPGRADE_URL = '/api/endpoints/{}/agentupgrade'
_PE_BYTES = b'MZ' + b'\x00' * 4096
@pytest.fixture(autouse=True)
def isolate_store(tmp_path, monkeypatch):
"""Redirect the agent store, package output and script bodies."""
siteagent = tmp_path / 'siteagent'
packages = tmp_path / 'applicationpackages'
scripts = tmp_path / 'scripts'
for directory in (siteagent, packages, scripts):
directory.mkdir()
monkeypatch.setattr(agentstore, '_siteagent_dir', lambda: str(siteagent))
monkeypatch.setattr(agentregistry, 'application_package_location',
lambda: (str(packages), '/api/applications/package/'))
monkeypatch.setattr(agentupgradescript, 'script_body_location',
lambda: (str(scripts), '/api/scripts/body/'))
return tmp_path
def _publish_agent(version='1.16.0.0'):
"""Stage a build so there is something to upgrade to."""
agentstore.store_version('agent', version, _PE_BYTES, make_current=True)
def _endpoint(db, hostname, agentversion='1.15.0.0'):
from shopdb.api import Asset, AssetType
from plugins.computers.models import Computer
from plugins.endpoints.models import EndpointState
assettype = AssetType.query.filter_by(assettype='computer').first()
if assettype is None:
assettype = AssetType(assettype='computer', pluginname='computers',
tablename='computers')
db.session.add(assettype)
db.session.flush()
asset = Asset(assetnumber=hostname, name=hostname,
assettypeid=assettype.assettypeid, isactive=True)
db.session.add(asset)
db.session.flush()
db.session.add(Computer(assetid=asset.assetid, hostname=hostname))
db.session.add(EndpointState(assetid=asset.assetid, hostname=hostname,
agentversion=agentversion))
db.session.commit()
return asset.assetid
def _login(db, client, username, permissions=None, admin=False):
"""A user holding exactly the named permissions (or the admin role)."""
from shopdb.core.models import User, Role, Permission
Permission.seed()
db.session.flush()
rolename = 'admin' if admin else username + 'role'
role = Role.query.filter_by(rolename=rolename).first()
if role is None:
role = Role(rolename=rolename)
db.session.add(role)
db.session.flush()
if not admin:
for name in (permissions or []):
perm = Permission.query.filter_by(name=name).first()
if perm is not None and perm not in role.permissions:
role.permissions.append(perm)
db.session.commit()
user = User(username=username, email=username + '@test.local',
passwordhash=generate_password_hash('testpass'))
user.roles.append(role)
db.session.add(user)
db.session.commit()
response = client.post('/api/auth/login',
json={'username': username, 'password': 'testpass'})
token = response.get_json()['data']['access_token']
return {'Authorization': f'Bearer {token}'}
def test_an_upgrade_queues_as_a_pinned_run_script(client, db, app):
"""The push must ride the normal script channel, digest-pinned.
Landing as run_script is what makes the body verifiable on the endpoint;
a bespoke action would need its own pinning and would not get it.
"""
from plugins.endpoints.models import EndpointCommand
_publish_agent()
assetid = _endpoint(db, 'UPGRADE01')
headers = _login(db, client, 'upgradeadmin', admin=True)
response = client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
assert response.status_code == 201
command = EndpointCommand.query.filter_by(assetid=assetid).first()
assert command is not None
assert command.action == 'run_script'
seeded = agentupgradescript.seed_upgrade_script()
assert command.target == 'script:{}'.format(seeded['scriptid'])
# Pinned to the exact body approved at enqueue, so a later edit cannot
# swap what the endpoint runs.
assert command.pinnedsha256 == seeded['sha256']
def test_the_upgrade_requires_the_agentupdate_permission(client, db, app):
"""endpoints.command alone must not be able to swap the SYSTEM agent."""
_publish_agent()
assetid = _endpoint(db, 'UPGRADE02')
headers = _login(db, client, 'plainoperator',
permissions=['endpoints.view', 'endpoints.command'])
response = client.post(UPGRADE_URL.format(assetid), json={},
headers=headers)
assert response.status_code == 403
def test_an_upgrade_is_refused_when_no_build_is_published(client, db, app):
"""Nothing staged means there is no payload; refuse before queueing.
Queueing anyway would send an endpoint to fetch a package that does not
exist and fail opaquely minutes later.
"""
from plugins.endpoints.models import EndpointCommand
assetid = _endpoint(db, 'UPGRADE03')
headers = _login(db, client, 'nobuildadmin', admin=True)
response = client.post(UPGRADE_URL.format(assetid), json={},
headers=headers)
assert response.status_code == 400
assert 'No agent build is published' in \
response.get_json()['data']['error']['message']
assert EndpointCommand.query.filter_by(assetid=assetid).count() == 0
def test_an_endpoint_lock_still_blocks_the_upgrade(client, db, app):
"""The delegate's lock check must not be bypassed by this route.
A quarantined machine is reserved by a person; a fleet agent swap is
exactly the kind of action that must respect that.
"""
from plugins.endpoints.models import EndpointCommand
_publish_agent()
assetid = _endpoint(db, 'UPGRADE04')
# Lock as one user, then act as a different non-admin holder of the grant.
owner = _login(db, client, 'lockowner', admin=True)
locked = client.post(f'/api/endpoints/{assetid}/lock',
json={'reason': 'bench repair'}, headers=owner)
assert locked.status_code in (200, 201)
headers = _login(db, client, 'upgradeoperator',
permissions=['endpoints.view', 'endpoints.command',
'endpoints.agentupdate',
'endpoints.runscript'])
response = client.post(UPGRADE_URL.format(assetid), json={},
headers=headers)
assert response.status_code == 403
assert 'locked' in \
response.get_json()['data']['error']['message'].lower()
assert EndpointCommand.query.filter_by(assetid=assetid).count() == 0
def test_a_second_upgrade_cannot_stack_on_an_in_flight_command(client, db, app):
"""at-most-one-in-flight must still hold through this route."""
from plugins.endpoints.models import EndpointCommand
_publish_agent()
assetid = _endpoint(db, 'UPGRADE05')
headers = _login(db, client, 'stackadmin', admin=True)
first = client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
assert first.status_code == 201
second = client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
assert second.status_code == 409
assert EndpointCommand.query.filter_by(assetid=assetid).count() == 1
def test_the_upgrade_refreshes_the_package_before_queueing(client, db, app):
"""A newer staged build must be what gets pushed.
If the route queued against a previously registered row, an operator who
uploaded a new agent and immediately pushed would ship the old one.
"""
from shopdb.api import Application
_publish_agent('1.16.0.0')
assetid = _endpoint(db, 'UPGRADE06')
headers = _login(db, client, 'refreshadmin', admin=True)
client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
row = Application.query.filter_by(
appname=agentregistry.AGENT_APPNAME).first()
firstsha = row.installersha256
# Stage a different build and push to a second endpoint.
agentstore.store_version('agent', '1.17.0.0', b'MZ' + b'\x01' * 4096,
make_current=True)
second = _endpoint(db, 'UPGRADE07')
response = client.post(UPGRADE_URL.format(second), json={}, headers=headers)
assert response.status_code == 201
db.session.expire_all()
row = Application.query.filter_by(
appname=agentregistry.AGENT_APPNAME).first()
assert row.installersha256 != firstsha
def test_a_failed_push_is_not_audited_as_an_upgrade(client, db, app):
"""A refused push must not leave an AgentUpgrade event behind."""
from shopdb.api import AuditLog
_publish_agent()
assetid = _endpoint(db, 'UPGRADE08')
headers = _login(db, client, 'auditadmin', admin=True)
client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
before = AuditLog.query.filter_by(entitytype='AgentUpgrade').count()
# Second push is refused by the in-flight rule.
blocked = client.post(UPGRADE_URL.format(assetid), json={}, headers=headers)
assert blocked.status_code == 409
after = AuditLog.query.filter_by(entitytype='AgentUpgrade').count()
assert after == before
def test_the_seeded_script_body_is_written_and_pinned(client, db, app,
isolate_store):
"""The agent verifies the body by digest, so both must exist and agree."""
import hashlib
_publish_agent()
assetid = _endpoint(db, 'UPGRADE09')
headers = _login(db, client, 'bodyadmin', admin=True)
response = client.post(UPGRADE_URL.format(assetid), json={},
headers=headers)
assert response.status_code == 201
seeded = agentupgradescript.seed_upgrade_script()
path = os.path.join(str(isolate_store / 'scripts'),
'script-{}.ps1'.format(seeded['scriptid']))
assert os.path.isfile(path)
with open(path, 'rb') as handle:
body = handle.read()
assert hashlib.sha256(body).hexdigest() == seeded['sha256']
# The delivery path must be fully resolved, not a literal placeholder.
assert b'{{AGENT_APPID}}' not in body