784964c907
Ports the corporate-GitHub wilm branch onto main, keeping the two branches
separate. Thirteen upstream commits arrive with their authorship intact; the
review fixes and the portability work below are on top.
What arrives:
endpoints plugin - a ShopDB agent reporting endpoint state, telemetry,
services and interfaces, with a command channel the agent
long-polls. Ingest stays on the generic collector route
(ADR-006); the agent key is a UI-managed single-scope
token whose revocation is an instant kill switch.
computers policies - Configuration and Business Policies (RMM-style), scope
trees validated at write time, actions checked against
the endpoints command allowlist. Feature-gated: every
route 404s while the setting is off, so enabling it can
never half-expose the surface.
asset CSV import - templates, a collector script carrying the site path, and
a validate-then-commit import. A plugin can own a virtual
column plus a post-write row hook, which is how a PC's
machine number becomes a controls link without a
hand-built relationships sheet.
agent/ - the Windows agent and its installer.
Contract 0.21.0 -> 0.24.0, additive throughout: register_import_hook (0.22),
mint/list/revoke_service_token (0.23), collector_authorized (0.24). The stability
table had no rows for any of them and now does.
MySQL 5.6 portability. The branch stored three policy columns as sa.JSON, the
only JSON columns in the product, and 5.6 has no JSON type: the migration failed
with a bare syntax error near the word JSON. They are TEXT now, through a
JSONText type decorator, so all seventeen call sites still read and write dicts.
The convention elsewhere is already TEXT (guides.blocksjson,
geenforce.manifestjson).
Review fixes carried on top:
- six policy GETs were jwt_required(optional=True), which authenticates a
caller without checking anything; they require computers.view now, and six
redundant jwt_required decorators are gone
- the agent installer took a default BaseUrl and CollectorKey, so a mistyped
run silently pointed an endpoint at the wrong instance; both are mandatory
and a non-https BaseUrl throws unless -AllowInsecureHttp is passed
- a committing CSV import wrote no audit row
Documentation: the 34 new routes are described in docs/api-inventory.json, which
is hand-maintained on purpose, and the generated spec now covers 331 paths and
477 operations across 22 surfaces. DEVELOPMENT-SETUP's 1071 troubleshooting row
said "use MySQL 8 for dev" and now gives the actual fix, including the trap that
5.6 downgrades ROW_FORMAT=DYNAMIC to Compact with a warning instead of an error.
Full suite 2123 passed, vitest 260 passed, all seventeen plugin chains migrate
clean on MySQL 5.6 and the naming, docs, parity and read-surface gates are green.
112 lines
4.3 KiB
Python
112 lines
4.3 KiB
Python
"""Import-mode helpers: let admins replay legacy history through the API.
|
|
|
|
A migration script importing the classic ASP shopdb
|
|
needs two things the normal API withholds:
|
|
|
|
1. Preserve each row's original createddate/modifieddate instead of stamping
|
|
"now" on insert/update.
|
|
2. Backdate event history such as USB checkouts to when they really happened.
|
|
|
|
Both are gated. The caller must BOTH be an admin AND send the request header
|
|
`X-Import-Mode: true`. Outside import mode every helper here is a no-op, so
|
|
wiring a call into a normal create/update path never changes behavior for
|
|
regular users. See the IMPORT-API wiki page for the operator manual.
|
|
"""
|
|
|
|
from datetime import datetime, timezone
|
|
|
|
from flask import request
|
|
from flask_jwt_extended import verify_jwt_in_request, current_user, get_jwt
|
|
|
|
|
|
# Request header a migration client sets to opt a request into import mode.
|
|
IMPORT_MODE_HEADER = 'X-Import-Mode'
|
|
|
|
# strptime formats accepted for import timestamps, tried in order. Covers the
|
|
# ISO 'T' form and the legacy MySQL 'space' form, with and without fractions,
|
|
# plus a bare date.
|
|
_IMPORT_DATETIME_FORMATS = (
|
|
'%Y-%m-%dT%H:%M:%S.%f',
|
|
'%Y-%m-%dT%H:%M:%S',
|
|
'%Y-%m-%dT%H:%M',
|
|
'%Y-%m-%d %H:%M:%S.%f',
|
|
'%Y-%m-%d %H:%M:%S',
|
|
'%Y-%m-%d %H:%M',
|
|
'%Y-%m-%d',
|
|
)
|
|
|
|
|
|
def import_mode_active():
|
|
"""True when the request is an admin-authenticated import request.
|
|
|
|
Needs header `X-Import-Mode: true` AND an admin caller. Safe to call from
|
|
any request context: verify_jwt_in_request is optional here so a missing or
|
|
bad token yields False instead of raising, and it is idempotent when the
|
|
route already ran @jwt_required."""
|
|
header = (request.headers.get(IMPORT_MODE_HEADER) or '').strip().lower()
|
|
if header != 'true':
|
|
return False
|
|
verify_jwt_in_request(optional=True)
|
|
user = current_user
|
|
if user is None or not user.hasrole('admin'):
|
|
return False
|
|
# A scoped PAT never gets import mode: import mode is an admin-role
|
|
# capability and a scoped token suspends the admin bypass. get_jwt is safe
|
|
# here - an admin user means a valid JWT was decoded above.
|
|
if get_jwt().get('patscopes') is not None:
|
|
return False
|
|
return True
|
|
|
|
|
|
def parse_import_datetime(value):
|
|
"""Parse an import timestamp into naive UTC, or None.
|
|
|
|
Accepts ISO '2020-01-05T12:00:00', legacy 'YYYY-MM-DD HH:MM:SS', a bare
|
|
date, or an already-parsed datetime. A trailing 'Z' is treated as UTC.
|
|
Timezone-aware input is converted to UTC then stripped to naive, matching
|
|
the repo convention of storing naive-UTC in DB DateTime columns."""
|
|
if value is None:
|
|
return None
|
|
if isinstance(value, datetime):
|
|
parsed = value
|
|
else:
|
|
text = str(value).strip()
|
|
if not text:
|
|
return None
|
|
if text[-1] in ('Z', 'z'):
|
|
text = text[:-1]
|
|
parsed = None
|
|
for fmt in _IMPORT_DATETIME_FORMATS:
|
|
try:
|
|
parsed = datetime.strptime(text, fmt)
|
|
break
|
|
except ValueError:
|
|
continue
|
|
if parsed is None:
|
|
# last resort: let fromisoformat try (handles offsets like +00:00)
|
|
try:
|
|
parsed = datetime.fromisoformat(text)
|
|
except ValueError:
|
|
return None
|
|
if parsed.tzinfo is not None:
|
|
parsed = parsed.astimezone(timezone.utc).replace(tzinfo=None)
|
|
return parsed
|
|
|
|
|
|
def apply_import_timestamps(instance, data):
|
|
"""Stamp createddate/modifieddate on a row from the payload, in import mode.
|
|
|
|
Call this in a create/update path after building the instance and before
|
|
commit. It is a no-op unless ALL hold: import mode is active (admin +
|
|
header), the payload carries the field, and the model actually has the
|
|
column. Explicitly setting modifieddate also suppresses the column's
|
|
onupdate=now default on updates, so legacy history survives edits too."""
|
|
if not data or not import_mode_active():
|
|
return
|
|
created = parse_import_datetime(data.get('createddate'))
|
|
if created is not None and hasattr(instance, 'createddate'):
|
|
instance.createddate = created
|
|
modified = parse_import_datetime(data.get('modifieddate'))
|
|
if modified is not None and hasattr(instance, 'modifieddate'):
|
|
instance.modifieddate = modified
|