The dev stack ran schema work from the api entrypoint while the pull and
airgap stacks ran it from a dedicated migrate service. Same repo, two
different init paths, and the dev one was missing a step.
That gap was real, not theoretical: the first run of this service on a
stack that had been up for days created 13 default settings. The entrypoint
only ran `db upgrade` and `plugin upgrade-all`, never the seeds, so any
patch adding a permission or a setting reached pull/airgap sites and
silently skipped dev. endpoints.agentupdate was exactly that shape of
change.
Running init from the api entrypoint has two other problems. Under
`restart: unless-stopped` a failing migration crash-loops, re-attempting
partially-applied schema work and burying the original error under retry
noise; the migrate service uses `restart: "no"` so a failure exits once and
stays readable. And `plugin upgrade-all` was tolerated as a warning, so the
app would start with plugin tables missing - the shape of the
endpointlogcollections bug. Chained with && it now stops the rollout
instead.
No fallback path is kept in the entrypoint. Compose honours
service_completed_successfully even when a single service is named, so
`up -d api` still runs migrate to completion first - verified, api stays in
Created until migrate exits 0. A skip-flag would have meant the safe
behaviour depended on remembering to set a variable, and deployment here
goes through an install script that invokes compose, so there is no second
caller to protect. The entrypoint keeps only the database wait, which is
about this container's readiness to serve rather than schema state.
The seeds are all create-if-missing, so running them on every start is a
no-op once current: the second run reported 0 permissions, 0 settings and
scopes already up to date. Ordering in the chain is load-bearing - plugin
tables cannot be created before the plugins are installed, and seedscopes
only links permissions that already exist as rows, so seeds run last.
migrate carries the full app environment including CORS_ORIGINS even though
it never serves a request, because create_app() validates config before any
CLI command dispatches and the production config refuses to build without an
explicit allowlist. It mounts instance_data because bootstrap-plugins.sh
writes plugins.json there; it and api never run concurrently, so the shared
mount is not contended.
The Dockerfile's standalone `docker run` note now states that the image does
not migrate itself and lists the sequence to run first.
Adds the scripts used to size the connection pool and validate fleet polling under load: burst arrival and mixed workload drivers, pool and resource samplers, a UI probe, host seed/drop SQL, and a gevent Dockerfile. Hosts are addressed by compose service name (api:5001) rather than a local path, so the harness runs against any stack rather than one machine.
loadtest_token.py mints a scoped collector token named LOADTEST and prints the secret once, then revokes only that token by name so a real collector token is never touched. envkeyfallback is false, so the harness needs a genuine managed token rather than an env key.
The gate reported 2277 lines of findings and had become unreadable, which read as accumulated debt. It was a tooling bug. In this grep build a single file-level --exclude= silently cancels every --include= on the same command; --exclude-dir= is unaffected. Two innocuous entries, --exclude=findings.sarif and --exclude=report.md, had therefore been disabling the include filters on every check that used the EXCLUDES array. That is why a check whose flags say .py/.vue/.js/.ts was reporting .md, .ps1, .json, .gitignore and vendored DLLs: it was scanning the whole tree including build artifacts.
Removes all file-level excludes and reproduces their intent with a drop_generated post-filter, adds grep -I so binaries can never be reported (there is no line to fix in a DLL), excludes bin/obj/publish/dev-bin and exports/backups, and adds the drop_scratch the shorthand loop was missing while every other check had it. Findings drop from 2277 lines to 11 with all ten checks passing. Verified with a planted canary file that the gate still catches cfg, a snake_case __tablename__ and a snake_case db.Column - a green gate immediately after a filtering change is worth distrusting until shown to still fail. Do not re-add --exclude=<file> to that array.
Style check fixes:
- Replace unicode box-drawing characters with ASCII (|-- instead of ├──)
- Replace emoji status markers with [OK]/[FAIL]/[WARN] tags
- Replace arrows (→ with ->, ↔ with <->) and em-dashes (— with --)
- Add venv-quick to excluded directories in check script
- Exclude darkwing report files (findings.sarif, report.md)
Backup finalization fix:
- Fix finalize-restore endpoint to not claim the upgrade sentinel
- Endpoint now checks for pending upgrade and returns {queued: true}
- Leaves UPGRADE_SENTINEL for background watcher to claim and process
- Prevents inline Alembic runs that can outlast request timeout
Claude Code context:
- Add .claude/ directory with INDEX.md and QUICK-REFERENCE.md
- Add CLAUDE.md with project overview and conventions
- Helps Claude Code understand project structure and rules
Fixes failing test: test_the_endpoint_does_not_run_the_upgrade_itself
Fixes naming/style CI check violations
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Restore original HTTPS config in docker-compose files (gunicorn + gevent DOES support SSL)
- Add plugin migration step to docker-entrypoint.sh (fixes missing plugin tables on first deploy)
- Install only core plugins on bootstrap (backups, computers, endpoints, guides, knowledgebase, machines, network, notifications, printers, slides, tools, warranty)
- Set FLASK_ENV=production to disable Werkzeug debugger
Bug: bootstrap-plugins.sh installed ALL plugins but didn't run migrations, causing
'Table does not exist' errors. Now installs core plugins only and runs 'flask plugin
upgrade-all' after bootstrap.
Optional plugins (employees, geenforce, measuringtools, printedparts, usb) can be
installed later via Settings > Plugins.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Backup was mostly working but failing at the end due to Python calls.
CHANGES:
- Made manifest generation work without Python (bash JSON)
- Made check verification recording work without Python
- All Python calls now wrapped in 'command -v python3' checks
- Falls back to bash sed for JSON updates
RESULT: Backup completes successfully without Python
- Database dump: SUCCESS (24K)
- Minimal manifest: SUCCESS (bash-generated JSON)
- Restic snapshot: SUCCESS
- Repository verification: SUCCESS
- Script exit: SUCCESS (no more Python errors)
Backups now work in offline/corporate proxy environments with
zero downloads during Docker build.
PROBLEM: Corporate proxy (Zscaler) blocked ALL package downloads during
Docker build - Oracle Linux yum, Alpine apk, even with SSL certs.
SOLUTION: Offline package approach
- Pre-download restic binary on host (works outside Docker)
- Bake binary into repo at scripts/backup/offline-packages/
- Updated Dockerfile.backup to use offline binary (no downloads)
- Simplified: mysql:8.4 base + offline restic only (no Python/alembic)
RESULT: Backup containers build successfully behind corporate proxy
FILES:
- scripts/backup/offline-packages/restic_0.17.3_linux_amd64.bz2 (8.6MB)
- scripts/backup/Dockerfile.backup: Use offline package, no downloads
VERIFICATION:
- docker compose build backup: SUCCESS
- Backup watcher running: Polling every 30s
- Backup store running: restic REST server on port 8000
WORKING:
- Click 'Back Up Now' in UI
- Watcher detects sentinel file within 30s
- Backup executes automatically
The offline package can be checked into git for easy deployment at
other sites behind similar proxies.
- Add --no-reload flag to run_dev.py to prevent env var loss
- Configure SHOPDB_BACKUPS_ROOT in .env for Windows path
- Create local backups directory
- Document local dev setup in backup README
- Fix: Flask auto-reloader was spawning process without env vars
- Quick start guide for backup/restore
- Configuration options (local, SMB, S3)
- Troubleshooting common issues
- API endpoints reference
- Clarify empty list on first deployment is normal
- Add docker-entrypoint.sh: runs migrations and bootstraps plugins
- Add bootstrap-plugins.sh: installs all plugins (disabled by default)
- Update Dockerfile to use entrypoint
- Plugins can now be enabled via Settings UI after first deployment
- Idempotent: safe to run multiple times
- Solves issue where imported prod databases had tables but no plugin registry
Mapped drives reported Unavailable on endpoints where the operator could
see them fine. The data path was intact; the collection was running under
the wrong token. Mapped drives are per logon token, so an elevated probe
enumerates a different drive table than the interactive user.
Collection:
- Pin the user-context spool path so the probe and service agree on one
location, and persist/overwrite it instead of deleting on read.
- Refuse to collect while elevated (exit code 2) rather than emitting a
confident wrong answer, and stamp elevated/username on the spool.
- Reject spools stamped elevated or from an unexpected account on read,
so an admin run cannot masquerade as valid.
- Log netuse/smbmapping/userlistener counts so each run leaves evidence
of what it actually saw.
Install and launch context:
- Move collection to a 'ShopDB user-context probe' scheduled task under
S-1-5-32-545 at RunLevel Limited, the only mechanism that reliably runs
in the operator's own session. Verify registration and record it in the
install ledger.
- Stop starting the tray as a child of the elevated installer; explorer.exe
de-elevates without changing ACCOUNT, so an admin install produced an
admin tray. The Run key only fires at logon, which left the tray dead on
a machine already signed in, so add a transient 'ShopDB tray first launch'
task that starts it once in the existing session and unregisters itself.
- Hide the console window during --collect-usercontext. The agent is a
console exe; the tray used to mask this with CreateNoWindow, but the
scheduled task invokes it directly and flashed a window every cycle.
Installer edits are in _install_script() in plugins/endpoints/api/siteagent.py,
which generates the install.ps1 that actually ships. The standalone
agent/installer/install-shopdbagent.ps1 was never shipped once the package
moved to the UI builder and is deleted here to stop the divergence that
caused an earlier fix to land in a file no endpoint ever ran.
Versions: agent 1.10.1, tray 1.3.0. These share the pinned spool path and
must be deployed together.
Pre-existing test failures unrelated to this work (8, identical on a clean
tree): docs publishable/versions, plugin contract symlink scan, import
guard tamper check, security headers upload sandbox, and three endpoint
timestamp tests.
Volume capacity and physical drive health were two adjacent cards, and the operator reading a full disk is usually the operator reading a dying disk in the same minute. They are now one Storage card with two sections. Visual grouping only - the two tables share no key, storage being per volume and health per physical device, and one drive can back several volumes or none the operator sees. Nothing is reconciled across the sections; this is not the mapped-drive merge where two sources described the same drive and the worse status won. The tabs render type was the only sections-in-one-card mechanism and emitted label/value fields with no table path, so it now takes table sections too. That is the renderer every plugin uses, so both behaviour changes in it are mutation-tested. Two things only surfaced on contact: the empty-section filter had to grow a table branch, because a table section declares columns up front and dropping it when it has no rows takes its tab with it and reads as no disks rather than nothing reported; and full-width span was keyed off the declared render type, but a tabs panel holding a six-column table needs the same span and that is only knowable from fetched data. The diskhealth route stays. It no longer backs a panel, but it is a stable read for callers outside the renderer and removing it for a UI-only change would break them.
A restore could load a database dump whose schema was older than the running
code. The manifest stamped schemarev from the code image on disk, so the
compatibility check compared the code against itself and always passed. A set
captured from a database at 7d37_import_batches_and_journal restored under
code at 7d41_drop_guest_tools_secrets, leaving the data behind the code: every
query on a table the newer migrations touched failed with 1054 Unknown column,
which surfaced as empty configuration-policy dropdowns and a failure to load
data on edit PC.
The authoritative signal is the alembic_version of the database that was
dumped, so manifests now carry datarev alongside schemarev, backup.sh reads it
from the database and warns when it has drifted from the image, and the gate
asks whether the running code knows that revision. Data behind the code is
allowed because it can be rolled forward; data ahead of the code is refused
because it cannot.
Rolling it forward has to happen somewhere with an app context. The backup
sidecar is mysql plus restic plus alembic only and has no Flask, so restore.sh
loads the dump and parks the run in an upgrading phase behind an
.upgrade-requested sentinel rather than declaring a success it cannot verify.
The api container owns the Flask CLI and already runs these exact idempotent
commands on every deploy, so a finalize endpoint there runs db upgrade plus
plugin upgrade-all and stamps the terminal phase. The UI drives it on seeing
the upgrading phase, so recovery needs no operator intervention.
The live simulation runs one codebase against itself, so its dump is always at
the same revision as the code restoring it and it can never reproduce this
drift. The gate therefore gets a standalone tester that builds throwaway
migration chains and dumps offline, invoked as a simulation preflight. It
caught a real bug in the legacy dump-parse fallback, where the revision was
matched loosely enough to return the alembic_version table name instead of the
value it holds.
FOR DEVELOPER USE ONLY. Adds a live restore round-trip diagnostic that fingerprints every base table via CHECKSUM TABLE, backs up and restores through the real sentinel path, and proves all 110 tables return to backup-time state.
Documents expected drift so future runs are not misread: apitokens last-used churn, external endpoint-agent writes (computers/endpointstates/events/applications) that cannot be quiesced, and that live asset-row deletion is not permitted under RESTRICT/CMMC. Restore remains a full drop-and-recreate replace, so a missing or corrupted asset is still fully restored - verified, no restore.sh change needed.
Writes simulate-restore-results.txt to the script dir and ships a lightweight README with the testing cadence and parameters.
The watcher scripts now write live status and append-only run history onto the shared volume; the api exposes them via /shopdbbackups/status and /history. The restore modal shows an indeterminate spinner with phase labels, and two cards list backup and restore run records. Timestamps render in UTC to match the Zulu-stamped set ids, the history directory is excluded from the set list, and a stale-status guard stops a previous run's done phase from collapsing the spinner on the next restore.
The api queued .restore-requested but nothing consumed it, so restores silently did nothing. The watcher now services the restore sentinel by running restore.sh, and its instance_data mount is read-write so restore can lay files back down (the manual one-shot stays read-only). Backup stamps use a colon-free UTC form that Date could not parse, so the UI showed raw strings; normalize them before formatting.
Backups are per-site and not part of the base stack, so they ship as a separate compose overlay with an isolated restic REST server the app can only reach through the backup utility (no shell or db creds). The sidecar image trusts the corporate proxy root CA at build time so its Oracle yum, github, and PyPI fetches succeed behind TLS interception; shell scripts are pinned to LF via .gitattributes so the container entrypoints do not die on a CRLF shebang.
The web import panel can list applied/undone batches, but the CLI had no equivalent, so an operator working from the shell had no way to find a batchid to feed into the undo command. Add a read-only batches command that reuses the same importbatches query, and a short README documenting the four WINC commands and their switches.
Machine data arrives as a Maximo asset export (validated against the WINC
site) that must update existing machines in place, not create canonical
per-table rows, so it does not fit the ShopDB-native/Azure/Thor CSV adapter
path. Bulk edits to live machines also need an escape hatch, so the whole
commit is journaled and can be undone as one batch.
Add a dedicated dry-run analyzer, approval packet, resolver/suggest helpers,
and a reversible writer/undo backed by a new importbatches/importjournalentries
schema (with migration). Expose preview/commit/batches/undo settings routes and
a matching frontend source card and panel, and surface the batch/journal
contract through shopdb.api for plugins.
The gate walks the working tree with grep, which does not honor .gitignore, so it scanned devnotes/ (local scratch notes, already gitignored and never committed) and failed on pasted logs full of em-dashes and arrows. Exclude devnotes/ the same way venv, node_modules, and dist are already excluded.
Bring main's endpoint hardening, operator-report validation, launch-path
policy, and theme-variable CSS into wilm. Channel model reconciled to this
repo's shipping (stable=release tag, edge=main trunk) while keeping wilm's
remote-BuildKit publish step and GHCR pull-based launcher.
Pilots behind the inspecting proxy cannot reliably build the frontend from source; npm's many small fetches get reset mid-install. Build the image once in CI on the clean runners and have pilots pull the finished channel image. One resumable image pull survives the proxy where hundreds of npm requests do not, and a pilot now needs only Docker.
a clone first created from an elevated prompt is owned by Administrators; a
later normal-user run then fails every git op with 'dubious ownership'
(exit 128), which the launcher misreported as an auth error. detect that
case and remove the clone for a clean re-clone.
docker/compose emit warnings on stderr (e.g. nothing to remove), which
tripped ErrorActionPreference=Stop and killed the run before the clone.
soften the preference around the best-effort cleanup steps.
proxy timeouts can kill a build mid-flight, leaving orphaned containers on
the channel ports, dangling build cache, and half-written clones. clean
those on every start, scoped to the channel and keeping the DB volume, so a
re-run starts from a clean slate.
GE/Zscaler proxy inspection breaks npm ci silently, leaving node_modules
incomplete and vite missing at build time. Detect the root CA from the
host's chain to the npm registry and stage it as corp-root-ca.pem so the
Dockerfile trusts it; harmless on clean networks.
The iex (irm ...) one-liner could never work against a private repo: raw.githubusercontent.com returns 404 without a token, and a git operation's Credential Manager auth does not apply to a plain irm. The prior token-passing workaround (get.ps1, 717387f) asked pilots to paste a PAT, which they will not do and may not have.
Start-ShopDBDev.ps1 is a downloaded FILE, not a web pipe, so only its git clone/fetch need auth - and those go through Git Credential Manager using the login pilots already have from cloning this repo or signing in via Visual Studio / VS Code. No token, no paste. It takes -Channel stable|edge and hands off to dev-bootstrap.ps1. DEV-BUILD-TEST.md is rewritten for the save-and-run flow with a GCM-seeding fallback.
raw.githubusercontent.com returns 404 for a private repo without a token, so the anonymous iex one-liner could never fetch get.ps1, and even once fetched its git clone of the same private repo would fail. get.ps1 now reads a GitHub PAT from SHOPDB_TOKEN (or GITHUB_TOKEN) and uses it for the clone/fetch; the raw fetch is authenticated by irm -Headers in the one-liner. The token is kept off disk: origin is reset to the clean URL after clone, and the update path fetches by explicit URL rather than a tokenized origin.
Follows the config-driven-badge decision through to packaging: a single channel-parameterized builder emits two distinct installers (stable from main, edge from dev-be) rather than two drifting scripts. Each package ships the image bundle, the airgap+channel compose templates, a pre-filled .env, and a one-command site installer.
Secrets ship as CHANGE_ME_* sentinels and Install-ShopDB.ps1 refuses to start until every one is rotated, so a distributable can never come up on shipped credentials. docker-compose.airgap.yml now threads SHOPDB_ENV_LABEL through the shared app-env anchor, otherwise a distributed edge stack would run the badge-capable image but never light the badge.
get.ps1 is the web launcher that makes the iex one-liners work: dev-bootstrap.ps1 operates on a local clone, so get.ps1 clones/updates the channel branch and hands off. docs/DEV-BUILD-TEST.md is the short reviewer handout with the two one-liners for Cameron to try.
A bleeding-edge container running next to a stable one on the same dev box is easy to confuse, and a branch-only badge would force perpetual rebase/cherry-pick drift between main and dev-be. Instead the badge is config-driven: the settings endpoint injects a synthetic env_label from SHOPDB_ENV_LABEL, the sidebar renders a red DEV tag only when it is non-empty, and stable simply leaves it unset. Same code on both channels, no divergence.
dev-bootstrap.ps1 makes a fresh Windows box turnkey: it checks git/Docker (winget-installs with a prompt), verifies the engine is actually running, then builds and starts an isolated per-channel stack so stable and edge coexist on distinct ports, projects, and volumes. docker-compose.channel.yml is a thin overlay so the base compose file stays channel-agnostic.
Operators had no way to flag a problem PC from the tray. This adds a report path: the tray captures a reason, free-text note, and optional screenshot and the agent uploads it on the collector token to a new operatorreports table linked to the asset. Ships the endpoints 0009 migration so the table exists before the code queries it; the running DB had drifted behind the code and returned HTTP 500 on upload.
Supporting changes for the monitor/drift work: shared panel styling for the new warning state, a naming-hook tweak, and gitignore entries for editor state (.vs/) and scratch capture files (pt*.txt) so they stay out of version control.
Bring in the natural-key import fix, Metrics->Monitors rename, PC detail tabbing and panel groups, guest search grants, and the 2026-08-30 security fixes. Reconcile endpoint packaging/installer, migrations, contract version, and tests; keep the delivery uninstaller and 6-step installer alongside main's permission-based RBAC and pin-drift protection.
The site-agent bundle now includes uninstall-shopdbagent.ps1 so operators can cleanly remove an install without hunting for the repo script. The packaging route reads it from scripts/, which was not mounted into the api container, so add a read-only ./scripts:/app/scripts mount (recreate the container to apply).
install.ps1 now registers the tray to autostart for every user via the HKLM Run value ShopDbEndpointTray, adds an All-Users 'ShopDB Agent' Start Menu shortcut for manual relaunch, and launches the tray once post-install. The tray enforces a single instance via a Global named mutex so repeated launches no longer stack duplicate icons. The uninstaller removes the Start Menu shortcut alongside the existing Run key and credential cleanup.
Ben Baker's installed-software collection, software delivery (run_script and
update_app), agentless adoption tool and command-queue hardening, with his
authorship preserved.
Reviewed before promotion. Fixed: the endpoints plugin tests were never collected
by pytest, the SBOM snapshot table's key would exhaust on MySQL, ingest did not
truncate two columns strict MySQL would reject, and update_app's version branch
pinned a digest for a file that route never serves. Agent packaging moved off
require_role('admin') onto a new endpoints.agentpackage permission.
Contract 0.27.0. Full suite 2231, frontend 314.
Reverses everything the installer and site-agent generator create: stops and deletes the service, kills agent/tray processes, removes the tray Run key and the HKLM\SOFTWARE\GE\ShopDB credential hive in both registry views, drops the Defender path exclusion, and deletes the install directory and ProgramData tree. Every step is best-effort and idempotent so it is safe to re-run and safe on a partially installed host.
Tracking docs/openapi.json (so the app can serve the spec at runtime) makes docs/ exist on the published GitHub mirror, which defeated the skipif(not DOCS.is_dir()) guard in the docs publish/version tests: they then ran against a prose-less checkout and failed on CI. Key the skip off docs/*.md prose instead, matching the canonical published-checkout definition in scripts/check-gates-ran.sh.
Also: the OpenAPI generator emitted a trailing-slash path (/api/docs/), which a strict proxy like IIS treats as a redirecting resource a subpath-mounted client cannot follow; normalise trailing slashes in _norm_path and correct the committed spec. And allow docx (python-docx) as an undeclared import: it is used only by scripts/docs/generate_docx.py, a developer tool that never runs on a deployed site.
The machine page now shows the PC that controls it. The endpoint was correct all
along and returned both directions; AssetTriageBand discarded every peer whose
type was not 'machine' and was mounted only on the PC page. The counterpart type,
label and relationship types are now props.
Fixed while adopting it more widely: every band chip linked to /assets/<id>, and
there is no such route. The resolver that AssetRelationships.vue kept privately
is now frontend/src/utils/assetRoute.js, used by both - and its type lookup is
case-insensitive, because seed data capitalises the type names while this site's
database does not.
The legacy import is now safe to resume. The 409-resolve asked for
per_page=10000 against a server that clamps to 100, so it silently searched the
first 100 rows and dropped links past that on every re-run. The crosswalk is now
bound to the database it describes, so one carried across a rebuild is refused
rather than attaching rows to the wrong assets. Source database names are
resolved from the environment, so the loader can be aimed without editing it.
Full suite 2178 passed, frontend 280 passed.
