ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make rollback, first upgrades and pin changes work as tested on Windows

Browse Source 
Found in an end-to-end test on a Windows 11 VM against a self-signed server.

- Rollback: a 1.20 agent re-enrolls within a second of an un-enroll, so an
  older agent pushed afterwards arrived on an enrolled PC and was refused.
  Revoke now takes {"blockreenroll": true, "reason": ...}; a blocked PC is
  never issued a credential and keeps working on the shared key until
  POST .../assets/<assetid>/allow-reenroll. The Enrollment page offers the
  block when un-enrolling and shows blocked PCs with an allow button.
- First upgrade of old agents: agents before 1.20 check the server
  certificate when downloading the upgrade, so a store wave failed on PCs that
  do not trust a self-signed server while the Migration tab said ready. A
  failed download now marks the PC blocked with "server certificate not
  trusted by this PC". New Install-ShopDBServerCertificate.ps1 (GE-Enforce,
  SYSTEM) adds a non-CA server certificate to the trusted roots after checking
  its thumbprint; the server certificate page offers a .cer download and the
  SHA-1 thumbprint.
- The generated installer writes ServerPins (with an optional next pin for a
  planned rotation), can leave the shared key out so the PC waits for
  approval, never touches an existing credential, and skips the trusted-root
  import for a CA certificate.
- The agent re-reads server pins about every 60 seconds and recycles pooled
  connections within 2 minutes, so a pin change needs no restart.
- An alert can only be acknowledged by a PC it targets.
- A wave no longer counts a PC as failed after a later wave upgraded it.
- The scripts report a pin mismatch only when the pin really is missing, and
  otherwise log the real error.
- Runbook: trust step before the first wave, rollback order, pin re-read,
  verifying the pin against the live URL; agent contract updated.
This commit is contained in:
cproudlock
2026-09-15 13:37:37 -04:00
parent 8eee6f19d8
commit 70d5531b4d
20 changed files with 1713 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
plugins/endpoints/api/enrollment.py
View File

@@ -7,7 +7,8 @@ Agent-facing:
Admin (endpoints.fleet to read, admin role to change):
GET /enrollment credential state per computer + requests
POST /enrollment/requests/<id>/approve | reject
POST /enrollment/assets/<assetid>/revoke
POST /enrollment/assets/<assetid>/revoke body {"blockreenroll", "reason"}
POST /enrollment/assets/<assetid>/allow-reenroll lift a re-enrollment block
PUT /enrollment/settings
GET /enrollment/migration readiness of every computer for agent 1.20
POST /enrollment/migration/wave push the agent upgrade to a batch of PCs
@@ -58,6 +59,7 @@ REASON_ALREADY_ENROLLED = 'already enrolled'
REASON_AWAITING_APPROVAL = 'awaiting approval'
REASON_NONCE_CHANGED = 'nonce changed'
REASON_REJECTED = 'rejected'
REASON_REENROLL_BLOCKED = 'reenroll blocked'
MIGRATION_STATES = ('enrolled', 'pending', 'upgrading', 'ready', 'blocked',
'notreporting')
@@ -71,6 +73,18 @@ BLOCKER_NO_PACKAGE = 'no agent package in store'
BLOCKER_TOO_OLD = 'agent too old for remote upgrade'
BLOCKER_POLICY = 'run_script not permitted by policy'
BLOCKER_AWAITING_ENROLLMENT = 'awaiting enrollment'
BLOCKER_REENROLL_BLOCKED = 'reenroll blocked'
BLOCKER_CERT_UNTRUSTED = 'server certificate not trusted by this PC'
# How agents up to 1.19 word a download refused over the server certificate.
# Only a PC's own recorded failure shows it; the server cannot read a trust store.
_TLS_TRUST_FAILURES = (
'ssl connection could not be established',
'untrustedroot',
'remotecertificatechainerrors',
'the remote certificate is invalid',
'could not establish trust relationship',
)
_OFFLINE_AFTER = timedelta(hours=24)
_WAVE_MAXIMUM = 100
@@ -224,6 +238,8 @@ def enroll_endpoint():
'installation', http_code=403)
if agentauth.active_credential_for(row.matchedassetid) is not None:
return _pending(row, REASON_ALREADY_ENROLLED, row.matchedassetid)
if agentauth.reenroll_blocked(row.matchedassetid):
return _pending(row, REASON_REENROLL_BLOCKED, row.matchedassetid)
if row.matchedassetid is None or db.session.get(Asset, row.matchedassetid) is None:
return _pending(row, REASON_UNKNOWN_HOSTNAME)
return _issue(row, row.matchedassetid,
@@ -274,6 +290,8 @@ def enroll_endpoint():
return _pending(row, REASON_SERIAL_MISMATCH, assetid)
if agentauth.active_credential_for(assetid) is not None:
return _pending(row, REASON_ALREADY_ENROLLED, assetid)
if agentauth.reenroll_blocked(assetid):
return _pending(row, REASON_REENROLL_BLOCKED, assetid)
if sharedkeyauthenticated and agentauth.auto_enroll_enabled():
return _issue(row, assetid, EndpointCredential.ENROLLED_AUTO)
return _pending(row, REASON_AWAITING_APPROVAL, assetid)
@@ -347,8 +365,12 @@ def get_enrollment():
rows = []
for assetid, hostname, serialnumber in _computers():
credential = credentials.get(assetid)
blocked = bool(credential is not None and credential.reenrollblocked)
if credential is not None and credential.revokedat is None:
state = 'enrolled'
elif blocked:
# Its agent keeps asking; those requests are expected, not pending work.
state = 'revoked'
elif _is_pending(assetid, hostname, index):
state = 'pending'
elif credential is not None:
@@ -364,6 +386,9 @@ def get_enrollment():
'issuedat': _utciso(credential.issuedat) if credential else None,
'lastusedat': _utciso(credential.lastusedat) if credential else None,
'enrolledvia': credential.enrolledvia if credential else None,
'reenrollblocked': blocked,
'reenrollblockedreason': (credential.reenrollblockedreason
if blocked else None),
})
counts = Counter(row['state'] for row in rows)
@@ -430,6 +455,10 @@ def approve_enroll_request(requestid):
return error_response(ErrorCodes.CONFLICT,
'That computer is already enrolled; revoke its '
'credential first', http_code=409)
if agentauth.reenroll_blocked(assetid):
return error_response(ErrorCodes.CONFLICT,
'That computer is blocked from enrolling again; '
'allow re-enrollment first', http_code=409)
row.status = EndpointEnrollRequest.STATUS_APPROVED
row.matchedassetid = assetid
@@ -467,23 +496,83 @@ def reject_enroll_request(requestid):
@endpoints_bp.route('/enrollment/assets/<int:assetid>/revoke', methods=['POST'])
@require_role('admin')
def revoke_endpoint_credential(assetid):
"""Un-enroll a PC. Its agent gets CREDENTIAL_INVALID and enrolls again."""
"""Un-enroll a PC. Its agent gets CREDENTIAL_INVALID and enrolls again.
Body (optional) {"blockreenroll": true, "reason": "rollback"} also keeps the
PC from enrolling again until POST .../allow-reenroll. That is the step
before rolling a PC back to an agent that only knows the shared key: without
it the current agent re-enrolls within seconds and the older agent is then
refused. A PC whose credential is already revoked can still be blocked.
"""
payload = request.get_json(silent=True) or {}
if not isinstance(payload, dict):
return error_response(ErrorCodes.VALIDATION_ERROR,
'Body must be a JSON object')
blockreenroll = payload.get('blockreenroll', False)
if not isinstance(blockreenroll, bool):
return error_response(ErrorCodes.VALIDATION_ERROR,
'blockreenroll must be true or false')
reason = None
if blockreenroll:
reason, problem = _text_field(payload, 'reason', 64, False)
if problem is not None:
return error_response(ErrorCodes.VALIDATION_ERROR, problem)
reason = reason or None
credential = agentauth.active_credential_for(assetid)
if credential is None and blockreenroll:
credential = EndpointCredential.query.filter_by(assetid=assetid).first()
if credential is None:
return error_response(ErrorCodes.NOT_FOUND,
'This computer has no active credential',
http_code=404)
credential.revokedat = _naive_utcnow()
credential.revokedby = _username()
AuditLog.log('deleted', 'EndpointCredential', entityid=assetid,
entityname=credential.credentialprefix,
details='endpoint credential {} revoked'.format(
credential.credentialprefix))
wasactive = credential.revokedat is None
if wasactive:
credential.revokedat = _naive_utcnow()
credential.revokedby = _username()
details = []
if wasactive:
details.append('endpoint credential {} revoked'.format(
credential.credentialprefix))
if blockreenroll:
credential.reenrollblocked = True
credential.reenrollblockedreason = reason
details.append('re-enrollment blocked ({})'.format(reason or 'no reason'))
AuditLog.log('deleted' if wasactive else 'updated', 'EndpointCredential',
entityid=assetid, entityname=credential.credentialprefix,
details='; '.join(details))
db.session.commit()
return success_response(credential.to_dict(),
message='Endpoint credential revoked')
@endpoints_bp.route('/enrollment/assets/<int:assetid>/allow-reenroll',
methods=['POST'])
@require_role('admin')
def allow_endpoint_reenroll(assetid):
"""Lift a re-enrollment block. The PC enrolls on its next attempt."""
credential = EndpointCredential.query.filter_by(assetid=assetid).first()
if credential is None:
return error_response(ErrorCodes.NOT_FOUND,
'This computer has no credential record',
http_code=404)
if not credential.reenrollblocked:
return success_response(credential.to_dict(),
message='Re-enrollment was not blocked')
previous = credential.reenrollblockedreason
credential.reenrollblocked = False
credential.reenrollblockedreason = None
AuditLog.log('updated', 'EndpointCredential', entityid=assetid,
entityname=credential.credentialprefix,
changes={'reenrollblocked': {'old': True, 'new': False}},
details='re-enrollment allowed again (was blocked: {})'.format(
previous or 'no reason'))
db.session.commit()
return success_response(credential.to_dict(),
message='Re-enrollment allowed')
@endpoints_bp.route('/enrollment/settings', methods=['PUT'])
@require_role('admin')
def update_enrollment_settings():
@@ -546,6 +635,28 @@ def _store_has_package():
return not _version_below(current, CREDENTIAL_AGENT_VERSION)
def _latest_commands(assetids, target=None):
"""The newest run_script command per asset (optionally for one target)."""
if not assetids:
return {}
newest = (db.session.query(db.func.max(EndpointCommand.endpointcommandid))
.filter(EndpointCommand.assetid.in_(assetids),
EndpointCommand.action == 'run_script'))
if target is not None:
newest = newest.filter(EndpointCommand.target == target)
newest = newest.group_by(EndpointCommand.assetid)
return {command.assetid: command for command in EndpointCommand.query
.filter(EndpointCommand.endpointcommandid.in_(newest)).all()}
def _is_tls_trust_failure(command):
if command is None or command.status != EndpointCommand.STATUS_FAILED:
return False
text = '{} {}'.format(command.resultmessage or '',
command.resultjson or '').lower()
return any(marker in text for marker in _TLS_TRUST_FAILURES)
def compute_readiness(assetids=None):
"""Migration state for every active computer (or just assetids).
@@ -562,8 +673,11 @@ def compute_readiness(assetids=None):
return []
ids = [row[0] for row in computers]
active = {c.assetid for c in EndpointCredential.query
.filter(EndpointCredential.revokedat.is_(None)).all()}
credentialrows = EndpointCredential.query.filter(
EndpointCredential.assetid.in_(ids)).all()
active = {c.assetid for c in credentialrows if c.revokedat is None}
reenrollblocked = {c.assetid for c in credentialrows
if c.reenrollblocked and c.revokedat is not None}
index = _pending_index(_pending_requests())
states = {s.assetid: s for s in
EndpointState.query.filter(EndpointState.assetid.in_(ids)).all()}
@@ -579,6 +693,8 @@ def compute_readiness(assetids=None):
(EndpointCommand.STATUS_PENDING,
EndpointCommand.STATUS_RUNNING)))}
lastscripts = _latest_commands(ids)
haspackage = _store_has_package()
minimum = minimum_agent_version_for('run_script')
offlinecutoff = _naive_utcnow() - _OFFLINE_AFTER
@@ -591,6 +707,9 @@ def compute_readiness(assetids=None):
blockers = []
if assetid in active:
status = 'enrolled'
elif assetid in reenrollblocked:
status = 'blocked'
blockers.append(BLOCKER_REENROLL_BLOCKED)
elif _is_pending(assetid, hostname, index):
status = 'pending'
elif assetid in upgrading:
@@ -610,6 +729,8 @@ def compute_readiness(assetids=None):
blockers.append(BLOCKER_NO_PACKAGE)
if minimum and not agent_meets_minimum(agentversion, minimum):
blockers.append(BLOCKER_TOO_OLD)
if _is_tls_trust_failure(lastscripts.get(assetid)):
blockers.append(BLOCKER_CERT_UNTRUSTED)
if not _version_below(agentversion, CREDENTIAL_AGENT_VERSION):
blockers.append(BLOCKER_AWAITING_ENROLLMENT)
if not blockers and 'run_script' not in ALLOWED_ACTIONS_FOR(
@@ -628,6 +749,13 @@ def compute_readiness(assetids=None):
def _wave_progress(wave, scriptid):
"""Live progress of one wave.
failed counts a member only while it is still stuck: its newest agent
upgrade (from any wave or a single push) failed or expired, it is not
enrolled, and it does not report an agent at or above the credential
version. A PC a later wave upgraded stops counting here.
"""
members = wave.assetidlist
enrolled = set()
failed = set()
@@ -637,21 +765,23 @@ def _wave_progress(wave, scriptid):
EndpointCredential.assetid.in_(members),
EndpointCredential.revokedat.is_(None)).all()}
if scriptid is not None:
commands = (EndpointCommand.query
.filter(EndpointCommand.assetid.in_(members),
EndpointCommand.action == 'run_script',
EndpointCommand.target == f'script:{scriptid}',
EndpointCommand.createdat >= wave.createdat)
.order_by(EndpointCommand.createdat.asc())
.all())
latest = {}
for command in commands:
queued.add(command.assetid)
latest[command.assetid] = command
failed = {assetid for assetid, command in latest.items()
target = f'script:{scriptid}'
queued = {row[0] for row in db.session.query(EndpointCommand.assetid)
.filter(EndpointCommand.assetid.in_(members),
EndpointCommand.action == 'run_script',
EndpointCommand.target == target,
EndpointCommand.createdat >= wave.createdat)
.distinct()}
versions = dict(db.session.query(EndpointState.assetid,
EndpointState.agentversion)
.filter(EndpointState.assetid.in_(members)).all())
failed = {assetid for assetid, command
in _latest_commands(members, target).items()
if assetid not in enrolled
and command.status in (EndpointCommand.STATUS_FAILED,
EndpointCommand.STATUS_EXPIRED)}
EndpointCommand.STATUS_EXPIRED)
and _version_below(versions.get(assetid),
CREDENTIAL_AGENT_VERSION)}
return {
'waveid': wave.waveid,
'createdat': _utciso(wave.createdat),