Commit Graph

74 Commits

Author SHA1 Message Date
benjamin.baker
debe1fce69 it is risen - add restore progress and backup/restore history
The watcher scripts now write live status and append-only run history onto the shared volume; the api exposes them via /shopdbbackups/status and /history. The restore modal shows an indeterminate spinner with phase labels, and two cards list backup and restore run records. Timestamps render in UTC to match the Zulu-stamped set ids, the history directory is excluded from the set list, and a stale-status guard stops a previous run's done phase from collapsing the spinner on the next restore.
2026-09-03 20:43:42 -04:00
benjamin.baker
7e4469c44e Wire restore into the backup watcher and fix backup timestamps
The api queued .restore-requested but nothing consumed it, so restores silently did nothing. The watcher now services the restore sentinel by running restore.sh, and its instance_data mount is read-write so restore can lay files back down (the manual one-shot stays read-only). Backup stamps use a colon-free UTC form that Date could not parse, so the UI showed raw strings; normalize them before formatting.
2026-09-03 20:26:01 -04:00
benjamin.baker
19071ec6c6 Add opt-in backup overlay with third-container restic store
Backups are per-site and not part of the base stack, so they ship as a separate compose overlay with an isolated restic REST server the app can only reach through the backup utility (no shell or db creds). The sidecar image trusts the corporate proxy root CA at build time so its Oracle yum, github, and PyPI fetches succeed behind TLS interception; shell scripts are pinned to LF via .gitattributes so the container entrypoints do not die on a CRLF shebang.
2026-09-03 20:06:45 -04:00
benjamin.baker
d5d823fd8b Merge remote-tracking branch 'origin/wj' into wilm 2026-09-03 17:35:48 -04:00
ff9ae693a6 Remediate the 2026-09-03 feature-test findings 2026-09-03 17:14:53 -04:00
benjamin.baker
ab949f8bec Merge remote-tracking branch 'origin/wj' into wilm
Some checks failed
CI / backend (push) Has been cancelled
CI / naming (push) Has been cancelled
CI / gates (push) Has been cancelled
CI / frontend (push) Has been cancelled
CI / lean-build (push) Has been cancelled
2026-09-03 13:46:17 -04:00
ddf41c3e65 Make a partly-undone import retryable, and refuse a plan it cannot deliver 2026-09-03 09:16:01 -04:00
b0bd73f62d Import the wilm agent store, WINC adoption work and batches CLI, with four fixes 2026-09-03 08:16:38 -04:00
benjamin.baker
17825ec8f1 Add WINC batches CLI and a CLI how-to README
The web import panel can list applied/undone batches, but the CLI had no equivalent, so an operator working from the shell had no way to find a batchid to feed into the undo command. Add a read-only batches command that reuses the same importbatches query, and a short README documenting the four WINC commands and their switches.
2026-09-03 08:07:01 -04:00
benjamin.baker
2998e35ddf Add reversible Maximo/WINC machine-adoption import
Machine data arrives as a Maximo asset export (validated against the WINC
site) that must update existing machines in place, not create canonical
per-table rows, so it does not fit the ShopDB-native/Azure/Thor CSV adapter
path. Bulk edits to live machines also need an escape hatch, so the whole
commit is journaled and can be undone as one batch.

Add a dedicated dry-run analyzer, approval packet, resolver/suggest helpers,
and a reversible writer/undo backed by a new importbatches/importjournalentries
schema (with migration). Expose preview/commit/batches/undo settings routes and
a matching frontend source card and panel, and surface the batch/journal
contract through shopdb.api for plugins.
2026-09-03 05:04:22 -04:00
benjamin.baker
27ccf8962e exclude gitignored devnotes from the naming/style gate
The gate walks the working tree with grep, which does not honor .gitignore, so it scanned devnotes/ (local scratch notes, already gitignored and never committed) and failed on pasted logs full of em-dashes and arrows. Exclude devnotes/ the same way venv, node_modules, and dist are already excluded.
2026-09-02 18:39:23 -04:00
benjamin.baker
ebece2d3bd Merge origin/main into align-wilm-main (align wilm with Cameron's latest)
Bring main's endpoint hardening, operator-report validation, launch-path
policy, and theme-variable CSS into wilm. Channel model reconciled to this
repo's shipping (stable=release tag, edge=main trunk) while keeping wilm's
remote-BuildKit publish step and GHCR pull-based launcher.
2026-09-02 17:41:03 -04:00
bb733f1332 Merge the wilm work, act on the five-agent review, verify the launch-path policy on Windows 2026-09-02 16:15:53 -04:00
benjamin.baker
d0e9f80287 distribute prebuilt channel images via GHCR instead of building on pilots
Pilots behind the inspecting proxy cannot reliably build the frontend from source; npm's many small fetches get reset mid-install. Build the image once in CI on the clean runners and have pilots pull the finished channel image. One resumable image pull survives the proxy where hundreds of npm requests do not, and a pilot now needs only Docker.
2026-09-02 09:34:40 -04:00
benjamin.baker
3cbba2df14 re-clone when git reports dubious ownership on the working copy
a clone first created from an elevated prompt is owned by Administrators; a
later normal-user run then fails every git op with 'dubious ownership'
(exit 128), which the launcher misreported as an auth error. detect that
case and remove the clone for a clean re-clone.
2026-09-02 08:54:37 -04:00
benjamin.baker
aff3bcb8a0 keep launcher cleanup from aborting on docker stderr warnings
docker/compose emit warnings on stderr (e.g. nothing to remove), which
tripped ErrorActionPreference=Stop and killed the run before the clone.
soften the preference around the best-effort cleanup steps.
2026-09-02 08:18:18 -04:00
benjamin.baker
30544988ce reset interrupted runs before retry in the dev launcher
proxy timeouts can kill a build mid-flight, leaving orphaned containers on
the channel ports, dangling build cache, and half-written clones. clean
those on every start, scoped to the channel and keeping the DB volume, so a
re-run starts from a clean slate.
2026-09-02 08:16:13 -04:00
benjamin.baker
58040164c2 auto-detect and stage corp TLS-inspection root CA before docker build
GE/Zscaler proxy inspection breaks npm ci silently, leaving node_modules
incomplete and vite missing at build time. Detect the root CA from the
host's chain to the npm registry and stage it as corp-root-ca.pem so the
Dockerfile trusts it; harmless on clean networks.
2026-09-02 07:58:31 -04:00
benjamin.baker
30b089bc8a replace web iex launcher with a save-and-run script
The iex (irm ...) one-liner could never work against a private repo: raw.githubusercontent.com returns 404 without a token, and a git operation's Credential Manager auth does not apply to a plain irm. The prior token-passing workaround (get.ps1, 717387f) asked pilots to paste a PAT, which they will not do and may not have.

Start-ShopDBDev.ps1 is a downloaded FILE, not a web pipe, so only its git clone/fetch need auth - and those go through Git Credential Manager using the login pilots already have from cloning this repo or signing in via Visual Studio / VS Code. No token, no paste. It takes -Channel stable|edge and hands off to dev-bootstrap.ps1. DEV-BUILD-TEST.md is rewritten for the save-and-run flow with a GCM-seeding fallback.
2026-09-02 07:43:06 -04:00
benjamin.baker
717387f665 authenticate the web launcher against the private repo
raw.githubusercontent.com returns 404 for a private repo without a token, so the anonymous iex one-liner could never fetch get.ps1, and even once fetched its git clone of the same private repo would fail. get.ps1 now reads a GitHub PAT from SHOPDB_TOKEN (or GITHUB_TOKEN) and uses it for the clone/fetch; the raw fetch is authenticated by irm -Headers in the one-liner. The token is kept off disk: origin is reset to the clean URL after clone, and the update path fetches by explicit URL rather than a tokenized origin.
2026-09-02 07:33:30 -04:00
benjamin.baker
68c33ee896 add channel-aware distributable installers and web launcher
Follows the config-driven-badge decision through to packaging: a single channel-parameterized builder emits two distinct installers (stable from main, edge from dev-be) rather than two drifting scripts. Each package ships the image bundle, the airgap+channel compose templates, a pre-filled .env, and a one-command site installer.

Secrets ship as CHANGE_ME_* sentinels and Install-ShopDB.ps1 refuses to start until every one is rotated, so a distributable can never come up on shipped credentials. docker-compose.airgap.yml now threads SHOPDB_ENV_LABEL through the shared app-env anchor, otherwise a distributed edge stack would run the badge-capable image but never light the badge.

get.ps1 is the web launcher that makes the iex one-liners work: dev-bootstrap.ps1 operates on a local clone, so get.ps1 clones/updates the channel branch and hands off. docs/DEV-BUILD-TEST.md is the short reviewer handout with the two one-liners for Cameron to try.
2026-09-02 05:41:25 -04:00
benjamin.baker
0524ce7751 add config-driven channel label and dev-workstation bootstrap
A bleeding-edge container running next to a stable one on the same dev box is easy to confuse, and a branch-only badge would force perpetual rebase/cherry-pick drift between main and dev-be. Instead the badge is config-driven: the settings endpoint injects a synthetic env_label from SHOPDB_ENV_LABEL, the sidebar renders a red DEV tag only when it is non-empty, and stable simply leaves it unset. Same code on both channels, no divergence.

dev-bootstrap.ps1 makes a fresh Windows box turnkey: it checks git/Docker (winget-installs with a prompt), verifies the engine is actually running, then builds and starts an isolated per-channel stack so stable and edge coexist on distinct ports, projects, and volumes. docker-compose.channel.yml is a thin overlay so the base compose file stays channel-agnostic.
2026-09-02 05:27:01 -04:00
benjamin.baker
ab4b9d7ec9 add operator issue reports for endpoint agent
Operators had no way to flag a problem PC from the tray. This adds a report path: the tray captures a reason, free-text note, and optional screenshot and the agent uploads it on the collector token to a new operatorreports table linked to the asset. Ships the endpoints 0009 migration so the table exists before the code queries it; the running DB had drifted behind the code and returned HTTP 500 on upload.
2026-09-01 19:39:32 -04:00
benjamin.baker
a016753bb6 Update shared styles, panels, naming hook, and ignore rules
Supporting changes for the monitor/drift work: shared panel styling for the new warning state, a naming-hook tweak, and gitignore entries for editor state (.vs/) and scratch capture files (pt*.txt) so they stay out of version control.
2026-09-01 15:46:18 -04:00
benjamin.baker
1a1bf0640f Merge origin/main (West Jeff) into wilm
Bring in the natural-key import fix, Metrics->Monitors rename, PC detail tabbing and panel groups, guest search grants, and the 2026-08-30 security fixes. Reconcile endpoint packaging/installer, migrations, contract version, and tests; keep the delivery uninstaller and 6-step installer alongside main's permission-based RBAC and pin-drift protection.
2026-08-31 16:59:07 -04:00
benjamin.baker
022affc5d0 Ship uninstaller in agent bundle and harden tray launch
The site-agent bundle now includes uninstall-shopdbagent.ps1 so operators can cleanly remove an install without hunting for the repo script. The packaging route reads it from scripts/, which was not mounted into the api container, so add a read-only ./scripts:/app/scripts mount (recreate the container to apply).

install.ps1 now registers the tray to autostart for every user via the HKLM Run value ShopDbEndpointTray, adds an All-Users 'ShopDB Agent' Start Menu shortcut for manual relaunch, and launches the tray once post-install. The tray enforces a single instance via a Global named mutex so repeated launches no longer stack duplicate icons. The uninstaller removes the Start Menu shortcut alongside the existing Run key and credential cleanup.
2026-08-31 16:09:05 -04:00
6680c3c5de Merge the wilm delivery work into main
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.

Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.

Contract 0.27.0. Full suite 2231, frontend 314.
2026-08-30 19:41:15 -04:00
benjamin.baker
d6cd0c91b3 Merge origin/main into wilm: dashboard/fleet console, vuln upgrades, colours 2026-08-30 14:41:56 -04:00
benjamin.baker
c3875b0eec Add a standalone endpoint agent uninstaller
Reverses everything the installer and site-agent generator create: stops and deletes the service, kills agent/tray processes, removes the tray Run key and the HKLM\SOFTWARE\GE\ShopDB credential hive in both registry views, drops the Defender path exclusion, and deletes the install directory and ProgramData tree. Every step is best-effort and idempotent so it is safe to re-run and safe on a partially installed host.
2026-08-30 14:41:33 -04:00
benjamin.baker
b9f6815161 Green up
Tracking docs/openapi.json (so the app can serve the spec at runtime) makes docs/ exist on the published GitHub mirror, which defeated the skipif(not DOCS.is_dir()) guard in the docs publish/version tests: they then ran against a prose-less checkout and failed on CI. Key the skip off docs/*.md prose instead, matching the canonical published-checkout definition in scripts/check-gates-ran.sh.

Also: the OpenAPI generator emitted a trailing-slash path (/api/docs/), which a strict proxy like IIS treats as a redirecting resource a subpath-mounted client cannot follow; normalise trailing slashes in _norm_path and correct the committed spec. And allow docx (python-docx) as an undeclared import: it is used only by scripts/docs/generate_docx.py, a developer tool that never runs on a deployed site.
2026-08-30 11:59:01 -04:00
8666e0271c Reciprocal controls links, dead chip links, and a resumable legacy import
The machine page now shows the PC that controls it. The endpoint was correct all
along and returned both directions; AssetTriageBand discarded every peer whose
type was not 'machine' and was mounted only on the PC page. The counterpart type,
label and relationship types are now props.

Fixed while adopting it more widely: every band chip linked to /assets/<id>, and
there is no such route. The resolver that AssetRelationships.vue kept privately
is now frontend/src/utils/assetRoute.js, used by both - and its type lookup is
case-insensitive, because seed data capitalises the type names while this site's
database does not.

The legacy import is now safe to resume. The 409-resolve asked for
per_page=10000 against a server that clamps to 100, so it silently searched the
first 100 rows and dropped links past that on every re-run. The crosswalk is now
bound to the database it describes, so one carried across a rebuild is refused
rather than attaching rows to the wrong assets. Source database names are
resolved from the environment, so the loader can be aimed without editing it.

Full suite 2178 passed, frontend 280 passed.
2026-08-29 19:09:06 -04:00
a8f8580738 Upgrade the eight shipped dependencies carrying advisories
Clears every shipped finding the SBOM scan reported: pyjwt 2.13.0, idna 3.19,
axios 1.20.0, postcss 8.5.26, dompurify 3.4.14, nanoid 3.3.18, follow-redirects
1.16.0, form-data 4.0.6. All in-major bumps inside ranges the manifests already
allowed, so this is a lockfile refresh rather than a migration.

The axios and dompurify floors were raised to match, since the old ones still
nominally admitted the vulnerable range.

The PyJWT allowlist entry was removed rather than kept - it stopped matching once
the upgrade landed.

Verified: full suite 2161 passed, frontend 273 passed, build clean, npm audit
--omit=dev reports 0 vulnerabilities.
2026-08-29 17:42:58 -04:00
7ddfe4d3ef Recommend the smallest upgrade that clears an advisory, not the highest
The rollup took the highest fix version across a component's advisories, which
is wrong whenever an advisory is patched on more than one release branch. nanoid
is fixed on both 3.3.x and 5.x, and it is transitive under a postcss range of
^3.3.11 that no 5.x release can satisfy - so the recommendation was unreachable
rather than merely large.

Per advisory the target is now the lowest fix above the version in use; per
component, the highest of those. Fixes on older branches are skipped rather than
offered as a downgrade.
2026-08-29 17:24:52 -04:00
d49b421914 Add vulnerability scanning on top of the SBOM
scripts/scan_sbom.py checks the generated SBOM against OSV.dev by purl. The SBOM
shipped to every server and nothing read it, so 'are we exposed to this advisory'
was still a person going and looking.

Shipped and build-only findings are both reported, but only shipped components
can fail a build - the highest-severity match in the tree today is against the
test runner, and a gate that fails a release over that gets bypassed and then
catches nothing. Severity is taken from the advisory rather than derived from a
CVSS vector, and an advisory with no recorded word is reported UNKNOWN and never
gates.

scripts/vulnerability-allowlist.json records reviewed advisories with the reason,
the accepter, the date, and what would invalidate the reasoning.

Documented in docs/DEPENDENCY-SECURITY.md.
2026-08-29 17:19:27 -04:00
80bb7b5ba1 Merge wilm: endpoints plugin, agent, policies, CSV import, Entra fields
Thirteen commits from benjamin.baker plus the review fixes. Everything he built
is here: the endpoints plugin and its command queue, the .NET endpoint agent,
the computers policies subsystem, the site-asset CSV import with its adapters
and collector, Entra device identity, and the catalog hand-merge.

His branch predates four commits on main (section ticking, the generator share
links, Secure Send copy message, and the smoke-test workflow), so those come
through untouched - nothing of ours is removed by this merge.

wilm stays as its own branch; this does not close it.
2026-08-28 15:56:58 -04:00
benjamin.baker
af133372b4 Gate endpoint UI refresh on fresh telemetry and land endpoint agent
A command reaching 'succeeded' only proves the agent ran the action, not that it posted a new telemetry snapshot. Reloading panels on command success alone re-rendered stale service/process rows, so a stopped process could still show running right after Refresh Status.

Add a lightweight telemetrymeta probe exposing lastreportedat and gate both the header Refresh Status action and inline row Start/Stop on that stamp advancing past a pre-action baseline. Polling is bounded so an offline or idle agent never hangs the button, and messaging distinguishes a confirmed refresh from a queued one.

Bring the .NET endpoint agent source (service, tray, installer) into the repo so Cameron can merge and test, with an agent/.gitignore keeping bin/obj/publish out. Includes the DncMain launch-path fix, tray single-file host env scrub, and best-effort file logger.
2026-08-28 15:07:20 -04:00
98badcbc26 publish the runtime spec; gate guard 2026-08-28 14:04:10 -04:00
benjamin.baker
01734e5449 harden CSV import reconciliation and preserve saved model selections
Default fuzzy reconcile threshold to 0 and refuse merges when digit runs differ so numbered labels like Bay 311 never bind to Bay 310. Keep edit forms from clearing saved model fields when category filtering hides untyped rows. Includes in-progress computers policy and endpoints command feature work.
2026-08-28 09:29:09 -04:00
benjamin.baker
3deff4f716 Merge remote-tracking branch 'origin/main' into wilm
# Conflicts:
#	frontend/src/views/settings/SiteSettings.vue
2026-08-27 05:45:52 -04:00
benjamin.baker
8b1ba3c310 add endpoints plugin with fleet list view and monitoring drafts
Server-side RMM telemetry plugin: current-snapshot endpointstates plus append-only curated endpointevents, resolved to existing Computer assets by hostname. Adds ADR-010 asset panels (status, services, recent events), dashboard quiet and stability cards, and a minimal Fleet list view that links into existing PC detail pages. Registers the activity sidebar icon and a dev-only sample seeder. Ingest arrives on the generic collector route; the future C# agent design record lives in docs/DESIGN-DECISIONS-agent.md.
2026-08-27 05:41:06 -04:00
0738d71525 Guides image + draft gating, audit retention, ADR-015 gate, backup credential redaction 2026-08-26 15:08:53 -04:00
benjamin.baker
801dfd54ab Merge remote-tracking branch 'origin/main' into wilm
# Conflicts:
#	shopdb/data/catalog.json
2026-08-26 12:53:03 -04:00
benjamin.baker
e2207f2e7c Merge remote-tracking branch 'origin/main' into wilm 2026-08-25 16:48:23 -04:00
benjamin.baker
bdef21da50 add Entra device identity to computers and Azure/Thor import sources
PC records now carry Entra/Azure device identity so imported machines keep a durable link back to their directory object instead of matching on hostname alone. Adds azuredeviceid, azureobjectid, and azuredevicecategory to the Computer model with matching migrations, surfaces them read-only in PCDetail as an Entra Device Info card, and lets PCForm edit the device id directly. The computers plugin registers a machineassetnumber import hook that reconciles PC-to-machine links during bulk import and warns on unresolved or conflicting relationships.

Adds importsources adapters that normalize Azure device-group and Thor PC inventory exports into canonical ShopDB CSVs via hostname alignment, exposes register_import_hook and RowWarning on the shopdb.api contract surface (contract 0.21.0), and refreshes the Import Site Assets page with clearer source instructions. Includes reconcile_locations maintenance script and coverage for the new import paths.
2026-08-25 16:46:00 -04:00
4a5d2122db Repair backup revisions that recorded a mapped drive 2026-08-25 15:00:57 -04:00
4fe4e91ae4 Maintenance scripts load .env, so they reach the site's real database 2026-08-24 14:32:38 -04:00
benjamin.baker
4a1b97362b Add site asset CSV import with fuzzy reconciliation
Sites seed inventory from the Asset Collector script or spreadsheet exports, but lookup names drift between sources (Dell vs Dell Inc.), which would otherwise create duplicate vendors/models/types on every import. The importer now normalizes and fuzzy-matches near-miss references onto existing lookup rows within an operator-set similarity threshold, auto-applying the match while reporting it for review so nothing merges silently. Operating systems and asset rows are never reconciled. Adds the dedicated Import Site Assets page with script/template download and a multi-file dry-run/commit workflow.
2026-08-24 14:29:02 -04:00
d267813999 Printer assignment end to end, and a container that keeps a site's files
Assign printers to a MACHINE and let whichever PC controls it install them, with
the observed half kept strictly apart from the assigned half: a collector report
can never write an assignment row. Adoption is explicit. Adds a singular
relationship type so defaultprinter can mean exactly one, client scripts that
converge a bay's queues and correct a drifted one in place, and a wave gate so a
fleet-wide driver rollout cannot stampede its own share.

On the container side: the compose stack never persisted the instance directory,
so a rebuild discarded plugins.json and every upload. It is a volume now, the
image ships instance/ owned by the app user so the mountpoint is writable, and
the database moves to MySQL 8.4 LTS, which the Windows runbook has recommended
since 8.0 went end of life. The image also could not build at all: the frontend
stage never copied the plugin staging script it runs.

Adds a topology migration guide covering IIS to Docker and back.
2026-08-20 07:46:35 -04:00
09e9d5f047 Carry the level everywhere a position is drawn, and gate it per occurrence 2026-08-18 09:37:29 -04:00
4c3d5746e8 Fix the levels viewer: markers were drawn on whichever plan was showing 2026-08-17 15:01:09 -04:00