f0377726a1
Per-PC credentials (replaces the 09-14 client-certificate attempt): - POST /api/endpoints/enroll issues a credential to a PC that matches exactly one computer by hostname and BIOS serial and has none yet. Anything else (unknown or duplicate hostname, serial mismatch, already enrolled) waits as a request an admin approves or rejects. The agent's persisted nonce binds an approved request to the PC that asked. - One auth helper serves every agent route. With X-Endpoint-Credential the asset comes from the credential and a request naming another PC gets 403 ASSET_MISMATCH. The shared key still works for PCs that have not enrolled, is refused (401 SHARED_KEY_REFUSED) for any PC that has, and can be switched off. A revoked credential gets 401 CREDENTIAL_INVALID so the agent re-enrolls. - The core collector accepts the credential for the computers, printers and endpoints collectors through two new plugin hooks exposed on shopdb.api (contract 0.33.0), and refuses the shared key for enrolled PCs. - Admin API: enrollment list, approve, reject, revoke (un-enroll), settings. Migration API: per-PC readiness with blockers, and upgrade waves that package the agent once and queue the existing agent upgrade per PC. - Migration 0023 creates endpointcredentials, endpointenrollrequests and endpointmigrationwaves, drops the certificate columns from assets where test databases have them, and renames app_fingerprint to appfingerprint. 0021 and 0022 keep their ids but no longer alter the core assets table. - Removed: client_cert.py, certificate renewal, the unauthenticated /test/client-cert route, nginx client certificate forwarding and the agent certificate script. The client-cert header could be forged on IIS, the key was readable by users, and the shared key remained a fallback. - GET /settings/server-cert returns the SPKI SHA-256 pin for GE-Enforce. Review fixes: - report_command_result used g without importing it; every result was a 500. - collect_log refuses setup answer files, appsettings, agent directories and key files at enqueue. - Fleet alerts: alerttimeout and alertfleet agree with the migration, the scope is validated and a malformed scope no longer reaches the whole fleet, and the ack race no longer raises NameError. - A refused break-glass override leaves no hold and no audit row. - Agent upgrade checks locks and permissions before repackaging, and writes the package atomically. - Server certificate upload validates the pair before replacing files; generation no longer builds an openssl config from user input; audit rows are actually written. - Health evaluation runs in a savepoint so a caught error cannot lose the check-in. - Required-app policies are found by appid, survive renames, and deleting an app deactivates policies that still name it. The UI and API now say such a policy only takes effect inside a business policy. - Backfill handles duplicate fingerprints and rolls back a failed batch. - Plugin settings routes skip a plugin whose hook raises. - The setup token is no longer printed to the log. - The naming gate checks top-level files again and handles odd file names. - Removed root scratch scripts, one of which carried a token. - Rewrote the fleet alert scheduling and app delta tests on real fixtures.
481 lines
17 KiB
Python
481 lines
17 KiB
Python
"""Fleet alert delivery to the endpoint agent, and terminal acknowledgement.
|
|
|
|
A fleet alert is a notification flagged alertfleet. The endpoint agent polls
|
|
GET /api/endpoints/alerts?hostname=... on the endpoints collector key - the same
|
|
credential and kill switch as the rest of the agent channel, because a shop
|
|
floor bay has no user session - and the tray surfaces what comes back. When the
|
|
operator dismisses one, the agent posts it to the acknowledge route.
|
|
|
|
Two contracts matter here and both are load-bearing for the durable agent-side
|
|
surface:
|
|
|
|
1. The poll response is AUTHORITATIVE for that host. It is every alert that
|
|
should be live on the machine right now, so the agent can retire a pending
|
|
alert that has disappeared from it. That is the only way a server-side
|
|
close or cancel reaches an endpoint already displaying the alert.
|
|
|
|
2. Acknowledgement is TERMINAL for an asset. Once acked, the alert never comes
|
|
back for that machine, and a later server-side close is a no-op. This is
|
|
what lets the agent retry a queued acknowledgement safely.
|
|
|
|
The cross-machine guard gets its own test because it regressed into an
|
|
AttributeError: the handler read .assetid off _resolve_asset_for_hostname,
|
|
which returns the id itself, so the check raised on every call and one bay could
|
|
not be proven unable to acknowledge another bay's alert.
|
|
"""
|
|
|
|
from datetime import timedelta
|
|
|
|
from shopdb.core.models import Asset, AssetType
|
|
|
|
|
|
ALERTS_URL = '/api/endpoints/alerts'
|
|
ACK_URL = '/api/endpoints/alerts/{}/acknowledge'
|
|
COLLECTOR_KEY = 'testcollectorkey'
|
|
|
|
|
|
def _collector_headers(app):
|
|
app.config['COLLECTOR_API_KEY'] = COLLECTOR_KEY
|
|
return {'X-API-Key': COLLECTOR_KEY}
|
|
|
|
|
|
def _computer(db, hostname):
|
|
"""Create a computer Asset + Computer; return the asset."""
|
|
from plugins.computers.models import Computer, ComputerType
|
|
|
|
atype = AssetType.query.filter_by(assettype='computer').first()
|
|
if not atype:
|
|
atype = AssetType(assettype='computer')
|
|
db.session.add(atype)
|
|
db.session.flush()
|
|
|
|
ctype = ComputerType.query.filter_by(computertype='CMM PC').first()
|
|
if not ctype:
|
|
ctype = ComputerType(computertype='CMM PC')
|
|
db.session.add(ctype)
|
|
db.session.flush()
|
|
|
|
asset = Asset(assetnumber=hostname, assettypeid=atype.assettypeid,
|
|
isactive=True)
|
|
db.session.add(asset)
|
|
db.session.flush()
|
|
comp = Computer(assetid=asset.assetid, hostname=hostname,
|
|
computertypeid=ctype.computertypeid)
|
|
db.session.add(comp)
|
|
db.session.commit()
|
|
return asset
|
|
|
|
|
|
def _alert_type(db):
|
|
from plugins.notifications.models import NotificationType
|
|
|
|
ntype = NotificationType.query.filter_by(typename='Fleet Alert').first()
|
|
if not ntype:
|
|
ntype = NotificationType(typename='Fleet Alert', typecolor='#d93f3f',
|
|
isactive=True)
|
|
db.session.add(ntype)
|
|
db.session.commit()
|
|
return ntype
|
|
|
|
|
|
def _fleet_alert(db, message='Line 3 is down', isactive=True, endtime=None,
|
|
timeout=60):
|
|
from plugins.notifications.models import Notification
|
|
|
|
ntype = _alert_type(db)
|
|
note = Notification(notificationtypeid=ntype.notificationtypeid,
|
|
notification=message, alertfleet=True,
|
|
alerttimeout=timeout, isactive=isactive,
|
|
endtime=endtime)
|
|
db.session.add(note)
|
|
db.session.commit()
|
|
return note
|
|
|
|
|
|
def _naive_utcnow():
|
|
from datetime import datetime, timezone
|
|
return datetime.now(timezone.utc).replace(tzinfo=None)
|
|
|
|
|
|
def test_agent_polls_its_active_fleet_alerts(client, db, app):
|
|
"""An active fleet alert reaches the endpoint that asks for it."""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db, message='Line 3 is down')
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01',
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 200, resp.get_data(as_text=True)
|
|
alerts = resp.get_json()['data']
|
|
assert [a['notificationId'] for a in alerts] == [alert.notificationid]
|
|
assert alerts[0]['message'] == 'Line 3 is down'
|
|
assert alerts[0]['timeout'] == 60
|
|
# The tray renders the type colour, so it has to come down with the alert
|
|
# rather than being guessed client side.
|
|
assert alerts[0]['typeColor'] == '#d93f3f'
|
|
|
|
|
|
def test_poll_requires_the_collector_key(client, db, app):
|
|
"""The alert feed is agent-facing, not anonymous."""
|
|
_computer(db, 'BAY-01')
|
|
_fleet_alert(db)
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01')
|
|
|
|
assert resp.status_code in (401, 403), resp.get_data(as_text=True)
|
|
|
|
|
|
def test_unknown_hostname_is_not_found(client, db, app):
|
|
"""A hostname that resolves to no asset is a 404, not an empty list."""
|
|
resp = client.get(f'{ALERTS_URL}?hostname=NOSUCHBAY',
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 404
|
|
|
|
|
|
def test_deactivated_alert_drops_out_of_the_feed(client, db, app):
|
|
"""Server-side close: the alert stops being offered to the endpoint.
|
|
|
|
This is how a cancel reaches a machine. The feed is authoritative, so an
|
|
alert missing from it is the agent's instruction to retire the pending
|
|
toast it is holding.
|
|
"""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
|
|
alert.isactive = False
|
|
db.session.commit()
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01',
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 200
|
|
assert resp.get_json()['data'] == []
|
|
|
|
|
|
def test_expired_alert_drops_out_of_the_feed(client, db, app):
|
|
"""An alert past its endtime is withdrawn the same way a closed one is."""
|
|
_computer(db, 'BAY-01')
|
|
_fleet_alert(db, endtime=_naive_utcnow() - timedelta(minutes=5))
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01',
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 200
|
|
assert resp.get_json()['data'] == []
|
|
|
|
|
|
def test_acknowledgement_removes_the_alert_for_that_asset(client, db, app):
|
|
"""Acknowledgement is terminal: the alert never returns for this machine."""
|
|
asset = _computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
headers = _collector_headers(app)
|
|
|
|
ack = client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid},
|
|
headers=headers)
|
|
assert ack.status_code == 201, ack.get_data(as_text=True)
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01', headers=headers)
|
|
assert resp.get_json()['data'] == []
|
|
|
|
|
|
def test_acknowledgement_is_idempotent(client, db, app):
|
|
"""The agent retries a queued acknowledgement, so a repeat must succeed.
|
|
|
|
Acknowledgement is terminal and the agent holds it on disk until the server
|
|
confirms, which means the same ack can legitimately arrive twice after a
|
|
network outage. The second one is a success, not a duplicate-key error.
|
|
"""
|
|
from plugins.notifications.models import NotificationAcknowledgement
|
|
|
|
asset = _computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
headers = _collector_headers(app)
|
|
payload = {'hostname': 'BAY-01', 'assetid': asset.assetid}
|
|
|
|
first = client.post(ACK_URL.format(alert.notificationid), json=payload,
|
|
headers=headers)
|
|
second = client.post(ACK_URL.format(alert.notificationid), json=payload,
|
|
headers=headers)
|
|
|
|
assert first.status_code == 201
|
|
assert second.status_code == 200, second.get_data(as_text=True)
|
|
assert second.get_json()['data']['acknowledged'] is True
|
|
|
|
rows = (NotificationAcknowledgement.query
|
|
.filter_by(notificationid=alert.notificationid,
|
|
assetid=asset.assetid).all())
|
|
assert len(rows) == 1, 'a retried acknowledgement wrote a second row'
|
|
|
|
|
|
def test_closing_an_acknowledged_alert_changes_nothing(client, db, app):
|
|
"""A server-side close after acknowledgement is a no-op.
|
|
|
|
Acknowledgement already retired the alert for this asset, so there is
|
|
nothing left to withdraw and the alert must not reappear.
|
|
"""
|
|
asset = _computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
headers = _collector_headers(app)
|
|
|
|
client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid},
|
|
headers=headers)
|
|
|
|
alert.isactive = False
|
|
db.session.commit()
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01', headers=headers)
|
|
assert resp.status_code == 200
|
|
assert resp.get_json()['data'] == []
|
|
|
|
|
|
def test_one_bay_cannot_acknowledge_for_another(client, db, app):
|
|
"""The hostname must match the assetid it claims.
|
|
|
|
This guard regressed into an AttributeError and never actually ran, so it
|
|
is pinned: the collector key is shared across the fleet, and without this
|
|
check any endpoint could clear an alert on another machine's behalf.
|
|
"""
|
|
_computer(db, 'BAY-01')
|
|
other = _computer(db, 'BAY-02')
|
|
alert = _fleet_alert(db)
|
|
|
|
resp = client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': other.assetid},
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 400, resp.get_data(as_text=True)
|
|
|
|
# And BAY-02's alert is untouched by the attempt.
|
|
feed = client.get(f'{ALERTS_URL}?hostname=BAY-02',
|
|
headers=_collector_headers(app))
|
|
assert [a['notificationId'] for a in feed.get_json()['data']] == [
|
|
alert.notificationid]
|
|
|
|
|
|
def test_acknowledging_one_bay_leaves_the_others_alerted(client, db, app):
|
|
"""A fleet alert is per-asset: one operator does not clear it for everyone."""
|
|
asset = _computer(db, 'BAY-01')
|
|
_computer(db, 'BAY-02')
|
|
alert = _fleet_alert(db)
|
|
headers = _collector_headers(app)
|
|
|
|
client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid},
|
|
headers=headers)
|
|
|
|
still = client.get(f'{ALERTS_URL}?hostname=BAY-02', headers=headers)
|
|
assert [a['notificationId'] for a in still.get_json()['data']] == [
|
|
alert.notificationid]
|
|
|
|
|
|
def test_acknowledge_requires_the_collector_key(client, db, app):
|
|
"""Acknowledgement is agent-facing and must not answer an anonymous caller."""
|
|
asset = _computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
|
|
resp = client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid})
|
|
|
|
assert resp.status_code in (401, 403), resp.get_data(as_text=True)
|
|
|
|
|
|
def test_acknowledging_an_unknown_notification_is_not_found(client, db, app):
|
|
"""A stale alert id from a rebuilt server is a 404, not a 500."""
|
|
asset = _computer(db, 'BAY-01')
|
|
|
|
resp = client.post(ACK_URL.format(999999),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid},
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 404
|
|
|
|
|
|
def test_non_fleet_notification_is_never_delivered(client, db, app):
|
|
"""Ordinary board notifications are not agent alerts and must not leak."""
|
|
from plugins.notifications.models import Notification
|
|
|
|
_computer(db, 'BAY-01')
|
|
ntype = _alert_type(db)
|
|
db.session.add(Notification(notificationtypeid=ntype.notificationtypeid,
|
|
notification='Board only', alertfleet=False,
|
|
isactive=True))
|
|
db.session.commit()
|
|
|
|
resp = client.get(f'{ALERTS_URL}?hostname=BAY-01',
|
|
headers=_collector_headers(app))
|
|
|
|
assert resp.status_code == 200
|
|
assert resp.get_json()['data'] == []
|
|
|
|
|
|
# --- targeting -------------------------------------------------------------
|
|
#
|
|
# alertscope narrows a fleet alert to part of the floor. It was previously
|
|
# stored, serialised and editable in the authoring form but never read here, so
|
|
# every alert went to every endpoint no matter what was entered - a control that
|
|
# looked like it worked and did not. These pin the behaviour it should have had.
|
|
|
|
|
|
def _ids(client, app, hostname):
|
|
resp = client.get(f'{ALERTS_URL}?hostname={hostname}',
|
|
headers=_collector_headers(app))
|
|
assert resp.status_code == 200, resp.get_data(as_text=True)
|
|
return [a['notificationId'] for a in resp.get_json()['data']]
|
|
|
|
|
|
def test_no_scope_still_means_the_whole_fleet(client, db, app):
|
|
"""The default is unchanged: an unscoped alert reaches everyone."""
|
|
_computer(db, 'BAY-01')
|
|
_computer(db, 'BAY-02')
|
|
alert = _fleet_alert(db)
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
assert _ids(client, app, 'BAY-02') == [alert.notificationid]
|
|
|
|
|
|
def test_an_empty_scope_means_the_whole_fleet(client, db, app):
|
|
"""An empty dict is not a match-nothing."""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
|
|
|
|
def test_hostname_scope_targets_one_endpoint(client, db, app):
|
|
"""The narrowest case, and the one the test card drives."""
|
|
_computer(db, 'BAY-01')
|
|
_computer(db, 'BAY-02')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'hostname': 'BAY-01'}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
assert _ids(client, app, 'BAY-02') == []
|
|
|
|
|
|
def test_hostname_scope_is_case_insensitive(client, db, app):
|
|
"""Operators type these by hand; case must not decide delivery."""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'hostname': 'bay-01'}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
|
|
|
|
def test_a_scope_may_list_several_values(client, db, app):
|
|
"""Any value in a list matches, so a scope can name a handful of bays."""
|
|
_computer(db, 'BAY-01')
|
|
_computer(db, 'BAY-02')
|
|
_computer(db, 'BAY-03')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'hostname': ['BAY-01', 'BAY-03']}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
assert _ids(client, app, 'BAY-02') == []
|
|
assert _ids(client, app, 'BAY-03') == [alert.notificationid]
|
|
|
|
|
|
def test_computertype_scope_targets_a_class_of_machine(client, db, app):
|
|
"""Targeting by what the machine is, not which one it is."""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'computertype': 'CMM PC'}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
|
|
alert.alertscope = {'computertype': 'Kiosk'}
|
|
db.session.commit()
|
|
assert _ids(client, app, 'BAY-01') == []
|
|
|
|
|
|
def test_several_scope_keys_must_all_match(client, db, app):
|
|
"""Keys narrow together - it is an AND, not an OR."""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
|
|
alert.alertscope = {'hostname': 'BAY-01', 'computertype': 'CMM PC'}
|
|
db.session.commit()
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
|
|
# Right host, wrong type: the alert must not land.
|
|
alert.alertscope = {'hostname': 'BAY-01', 'computertype': 'Kiosk'}
|
|
db.session.commit()
|
|
assert _ids(client, app, 'BAY-01') == []
|
|
|
|
|
|
def test_an_unknown_scope_key_is_ignored_not_fatal(client, db, app):
|
|
"""A typo must not silently mute an alert everywhere.
|
|
|
|
Scopes are authored by hand. If {"hostnam": "BAY-01"} matched nothing, the
|
|
alert would vanish fleet-wide and look like a delivery failure. Sending it
|
|
wider than intended is the safe direction for something whose job is to warn
|
|
people, so unrecognised keys are ignored.
|
|
"""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'hostnam': 'BAY-01'}
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
|
|
|
|
def test_a_non_dict_scope_fails_closed(client, db, app):
|
|
"""A scope that is present but not an object is corrupt, not fleet-wide.
|
|
|
|
Create/update refuse such a scope, so a row carrying one did not come from
|
|
an operator's intent to reach everyone. Broadcasting it could put a
|
|
one-bay alert on every screen, so it reaches nobody.
|
|
"""
|
|
_computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = 'BAY-01'
|
|
db.session.commit()
|
|
assert _ids(client, app, 'BAY-01') == []
|
|
|
|
alert.alertscope = ['BAY-01']
|
|
db.session.commit()
|
|
assert _ids(client, app, 'BAY-01') == []
|
|
|
|
|
|
def test_a_json_string_scope_is_parsed(client, db, app):
|
|
"""Legacy rows written through the TEXT column hold the object as text."""
|
|
_computer(db, 'BAY-01')
|
|
_computer(db, 'BAY-02')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = '{"hostname": "BAY-01"}'
|
|
db.session.commit()
|
|
|
|
assert _ids(client, app, 'BAY-01') == [alert.notificationid]
|
|
assert _ids(client, app, 'BAY-02') == []
|
|
|
|
|
|
def test_scope_keys_match_the_notifications_validator():
|
|
"""Create/update validation and the matcher must agree on the key set."""
|
|
from plugins.endpoints.api.routes import _ALERT_SCOPE_KEYS
|
|
from plugins.notifications.models.notification import ALERT_SCOPE_KEYS
|
|
|
|
assert set(_ALERT_SCOPE_KEYS) == set(ALERT_SCOPE_KEYS)
|
|
|
|
|
|
def test_scoping_does_not_leak_into_acknowledgement(client, db, app):
|
|
"""A targeted alert is acknowledged the same way any other one is."""
|
|
asset = _computer(db, 'BAY-01')
|
|
alert = _fleet_alert(db)
|
|
alert.alertscope = {'hostname': 'BAY-01'}
|
|
db.session.commit()
|
|
|
|
ack = client.post(ACK_URL.format(alert.notificationid),
|
|
json={'hostname': 'BAY-01', 'assetid': asset.assetid},
|
|
headers=_collector_headers(app))
|
|
|
|
assert ack.status_code == 201
|
|
assert _ids(client, app, 'BAY-01') == []
|