Files
shopdb-flask/docs/wiki-drafts/Architectural-Findings-2026-09-11.md
cproudlock f9d6b57f0f Move the documentation to the line that ships, and correct it against the code
The docs lived on one lineage and the fleet code on another. That is why six
pages described a shared fleet key as how an agent authenticates, why
COLLECTOR-AUTH had to warn that the per-PC credential code was not in the
repository it sat in, and why ADR-019 recorded a decision whose implementation
was somewhere else. Documentation that cannot see the code it documents goes
stale by construction, and no amount of editing fixes that.

So the corpus moves here: 84 files, 56 pages, 20 ADRs, the proposals, INDEX.md,
llms.txt and both gates. Filenames are unchanged, because a wiki page name is
derived from the filename and a rename breaks every link anybody saved.

THE POINT OF THE MOVE IS THE CORRECTION IT ENABLES. Every caveat that said "not
in this tree" was false the moment the files landed beside the code, so each was
rewritten from what is actually here:

- COLLECTOR-AUTH gained the real enroll request body, the two bounds on the
  unauthenticated route (per-IP rate limit, pending cap with stale rows dropped
  before a real PC is refused), and the rejected pending reason that binds to
  the endpoint identity rather than the nonce.
- ADR-019 now describes an implementation that is present, including its tests,
  and says plainly that what remains is the FLEET migration, not the code.
- proposals/device-identity claimed phase 1 was unbuilt. plugins/computers
  declares deviceid on the collector schema and prefers it over the older
  fallbacks. Corrected in the page, its summary and the index.

WINDOWS/IIS IS NOW DOCUMENTED AS THE LEGACY PATH, per the product decision, and
carefully: it is what production runs today. Every Windows page opens by saying
it is supported, not for a new site, and where a new site should go instead. No
procedure was cut - an operator on an IIS box at 2am needs these pages to work
exactly as well tomorrow. MIGRATE-TOPOLOGY stops being a footnote and becomes
the off-ramp, which is why it nearly doubled: what was there did not match how
the Docker stack really works here.

The Docker material that already lived on this line was kept and given
ownership rather than buried under the ported Windows-centric prose: DEPLOY now
describes the four services, the migrate service that runs the upgrade chain
before api starts, TLS terminated by nginx, and the plugin count corrected from
thirteen to seventeen - a number both lineages had wrong.

ROADMAP is rewritten from evidence rather than memory: what has shipped, read
out of the changelog, the ADRs and the manifests, and ahead of it exactly two
items - finalize a distributed release and polish the setup wizard.

PROJECT-MAP is generated and was stale; the GENERATOR was what was wrong.
--check passes now.

API-REFERENCE is retired to a pointer. /api/docs serves 573 operations with
derived auth, query parameters and harvested examples, and a hand-written page
competing with that is a liability. The file stays so its wiki page and its
inbound links survive.

AND THE MOVE TURNED ON A SCRUB THAT HAD NEVER RUN HERE. test_docs_publishable
keys on docs/*.md, which this line did not have, so it skipped - its own comment
predicted what would surface. Porting the corpus activated it and it found 48
lines that must not reach a published wiki: branch names abbreviating the site,
assistant-tooling paths, the site name in two deployment pages, and an employee
ID in a Windows path that the scrub does not even look for.
2026-09-19 09:10:15 -04:00

421 lines
21 KiB
Markdown

<!-- docmeta
purpose: explanation
audience: developer
status: historical
summary: The 2026-09-11 architecture findings with a 2026-09-18 verdict on each; most were refuted.
loadwith: none
-->
# Architectural Findings - September 11, 2026
> **Read with the date on it.** This is a point-in-time audit, kept as the
> record of what was found and what it turned out to be, not as current
> guidance. One thing it says repeatedly is no longer true: it was written when
> this line carried no ADRs, and the full ADR set now lives beside it in
> [docs/adr](../adr/README.md). Where this page says a decision record is
> missing or lives elsewhere, look there first.
**Scope:** Full codebase analysis via Gortex semantic code intelligence
**Original severity claim:** 22 findings (6 critical, 6 high, 10 medium)
**Status:** RE-CHECKED against the deployment branch at 35d4fcd41 on 2026-09-18. Most of
the original findings do not survive a read of the code. This page keeps them,
with the verdict and the evidence, so nobody acts on a claim that was never
true.
## How to read this page (added 2026-09-18)
The September 11 analysis was written from semantic queries, not from reading
the files it names. Eight of its claims cite code that does not exist on this
branch: a `machines` plugin that imports `computers`, a migration that spells a
column `last_reported_date`, a ServiceNow plugin, `MACHINE_TYPES = ['Lathe',
'Mill', 'CNC']`, a hardcoded `['Bldg 300', 'Bldg 400']`, Vuex, an absent CSP
header, an unhandled `requests.post`. Each of those was grepped for on
2026-09-18 and returns nothing.
Every finding below now carries one of four verdicts:
- **REFUTED** - the cited code does not exist, or the opposite is in the tree.
- **STALE** - it was true once; the tree has since fixed it.
- **HOLDS (narrowed)** - something real is there, but smaller or different than
the original text claims.
- **UNVERIFIED** - cannot be settled from this repository.
Effort and priority numbers in the original text were estimates with no stated
basis. They are left where they sit for findings that still hold and struck
where the finding does not.
## Critical Findings (P0)
### 1. Circular Dependencies Between Plugins
**Verdict: REFUTED as stated. HOLDS (narrowed) as a deferred-import coupling.**
**Original claim:** `computers` imports `machines` and `machines` imports
`computers`; import errors at runtime if plugins load in the wrong order; 25+
violations.
**What the code shows (2026-09-18):**
- `grep -rn "^from plugins\." plugins/ --include=*.py` returns **nothing**.
There is not one module-level cross-plugin import in the tree, so there is no
load-order failure mode and no import cycle at import time.
- `grep -rn "from plugins.computers" plugins/machines/` returns **nothing**.
The `machines` plugin does not reference `computers` at all, so the cycle the
finding is built on does not exist.
- 52 cross-plugin imports do exist, and every one of them is inside a function
body, resolved on call. `plugins/computers/plugin.py:414` and `:1166` reach
`plugins.machines.models` that way; `plugins/endpoints/api/routes.py:235` and
fourteen more reach `plugins.computers.models`.
**What is actually there:** a mutual reference pair between `computers` and
`endpoints`, both deferred:
- `plugins/computers/api/policy_routes.py:44` imports
`plugins.endpoints.commands.elevated_permission_for`, unguarded.
- `plugins/endpoints/commands.py:477` imports
`plugins.computers.services.policyresolver.effective_policy_for`, wrapped in
`try/except` that falls back to the base allowlist (`commands.py:475-481`).
So one side of that pair degrades when its partner is absent and the other
raises. The same asymmetry appears at
`plugins/endpoints/api/agentauth.py:118`, where `resolve_hostname` imports
`plugins.computers.models.Computer` with no guard: an endpoints-without-
computers site fails there rather than returning "unknown hostname".
**What to do instead of the original remediation:** nothing about load order.
The open question is narrower and worth one pass: decide whether a deferred
cross-plugin import must be guarded, and guard the unguarded ones. A linter
rule banning `plugins.*` imports outright would break the working policy and
enrollment paths listed above, which are the product's actual design.
### 2. Undeclared Import Dependencies
**Verdict: REFUTED.**
**Original claim:** 18 plugins import `shopdb.core.models`, 12 import
`shopdb.core.utils`, 7 import `shopdb.core.services`.
**What the code shows:**
`grep -rn "from shopdb\.\(core\|utils\|extensions\|services\)" plugins/
--include=*.py` returns **0 hits**. Not one plugin deep-imports core. The only
`shopdb.*` import in `plugins/` outside `shopdb.api` is
`shopdb.plugins.alembic_template` from migration `env.py` files, which is the
shared Alembic runner ADR-008 sanctions.
ADR-001's rule that `shopdb.api` is the only core surface a plugin may import is
being kept. The thing that is NOT enforced is raw SQL against core tables: see
finding 3a below.
### 2a. Raw SQL against core tables from a plugin (added 2026-09-18)
**Verdict: HOLDS (new, narrow).**
`plugins/computers/plugin.py:2080-2091` runs
`db.session.execute(db.text(...))` joining core `assets`, `assetrelationships`
and `assettypes` inside the `flask computers retype` CLI command. `Asset`,
`AssetType`, `AssetRelationship` and `RelationshipType` are all exported from
`shopdb.api` (`shopdb/api/__init__.py:47-48`, `:424`, `:438-439`), so no
missing export excuses it, and `plugins/endpoints/api/routes.py:3981-3986` is
the same pattern already corrected for `BusinessUnit`. The contract note at
`shopdb/__init__.py:135-142` names the reason: a raw query is coupling to the
core schema that no contract bump can catch.
Blast radius: one admin CLI command, no serving route. Import rule not
literally broken. Worth fixing because it is the kind of thing that gets copied.
### 3. Schema Drift: Models vs Migrations
**Verdict: REFUTED as stated. HOLDS (narrowed) to one column.**
**Original claim:** `Computer.lastreporteddate` in the model but
`last_reported_date` in the migration; `Printer.driver` with no migration;
`AssetRelationship.label` in a migration but not the model; 14 tables affected.
**What the code shows:**
- `grep -rn "last_reported_date" --include=*.py .` returns **nothing**. The
model spells it `lastreporteddate` (`plugins/computers/models/computer.py:85`)
and so does every migration.
- `grep -rPn "sa\.Column\(\s*'[a-z]+_[a-z_]+'" plugins/*/migrations/versions/*.py
migrations/versions/*.py` returns **nothing**. There is no snake_case column
name in any migration in the tree.
- There is no `shopdb/models/` directory to hold the claimed
`AssetRelationship.label`; core models live under `shopdb/core/`.
**The one real instance:** `plugins/notifications/models/notification.py:158`
declares `alertscope = db.Column(db.JSON, nullable=True)` while the migration
that creates it, `plugins/notifications/migrations/versions/0006_add_fleet_alerts.py:45`,
creates `sa.Column('alertscope', sa.Text, nullable=True)` and its own docstring
at line 8 calls it "the alertscope TEXT field (JSON string)". Commit f484be8a3
fixed the two neighbouring columns to agree with 0006 and left this one. Nothing
breaks on MySQL 8 - the SQLAlchemy JSON type serialises into TEXT and reads back
- and consumers already normalise both shapes
(`plugins/endpoints/api/routes.py:4011-4020`). The cost is that the model is not
a true description of the deployed column, which is what autogenerate reads.
**Related gap that is real:** the naming gate cannot see migrations at all.
`scripts/check-naming-and-style.sh:236` greps
`^\s+[a-z]+_[a-z_]+\s*=\s*db\.Column`, which only matches a model attribute
assignment; migrations write `sa.Column('name', ...)` with the name as a
positional string. That is how the shipped build of
`plugins/endpoints/migrations/versions/0021_client_certificates.py` created
`client_cert_fingerprint`, `cert_issued_at` and `cert_expires_at` and passed the
gate (`git show 5cd533239:plugins/endpoints/migrations/versions/0021_client_certificates.py`,
lines 20-28). The tree is conformant today, because 0021 and 0022 are now
no-ops and 0023 renamed the survivor; the gate simply would not notice the next
one.
### 4. Single Points of Failure
**Verdict: UNVERIFIED from this repository, and partly STALE.**
Replication, load balancing and orchestration are properties of the reference site
deployment, not of this tree, so nothing here can confirm or deny them. Two
parts of the original text are wrong about what the repo ships:
- Restart policy is configured, not absent: `docker-compose.yml:24`, `:104` and
`:120` set `restart: unless-stopped`, and three services carry `healthcheck:`
blocks (`:38`, `:112`, `:219`).
- Backups are no longer "manual mysqldump weekly, stored on the same server".
See finding 4a.
The underlying point - one database, one app host, air gap limits the options -
is a statement about the site, and belongs in a deployment document rather than
a code findings page. The 200+ hour estimate has no basis in anything measured.
### 4a. Backups (added 2026-09-18, correcting the original text)
**Verdict: STALE. The repo ships a backup system.**
`scripts/backup/` contains `backup.sh`, `restore.sh`, `status.sh`, `watch.sh`,
`simulate-restore.sh` and `test-restore-datarev.sh`. Retention is enforced by
restic (`scripts/backup/backup.sh:134-141`: `--keep-daily 5 --keep-weekly 4
--keep-monthly 6 --prune`), destinations include SMB and S3 as well as local
(`scripts/backup/README.md`), each set carries `manifest.json` with schema
revisions for compatibility checking, and restore runs from the Settings >
Backups UI with a schema compatibility check. A restore-simulation harness is in
the tree.
### 5. Plugin Migration Chain Validation Gaps
**Verdict: STALE as written, and the credited fix is the wrong artifact.**
Validation is in the test suite, in this repository, and has been for longer
than the `/migration-check` skill: `tests/test_plugin_migrations.py` holds nine
tests covering owners-versus-models, chain shape, no-op anchors and idempotency,
and pins each chain's head in `EXPECTED_HEAD_REVISION`
(`tests/test_plugin_migrations.py:125`). `tests/test_plugins/test_endpoints_credentials_migration.py`
runs the 0020-0023 chain twice from both a legacy and a fresh schema, plus
downgrade and re-upgrade. The endpoints chain today is 24 revisions with one
root and one head and no dangling `down_revision`.
The branch example in the original text (`003_add_policies.py` with
`down_revision=001`) is invented; no such file exists under
`plugins/computers/migrations/versions/`, which holds 6 revisions.
**The real gap, which the original text missed:** no test asserts that a plugin
chain touches only the tables it owns. `plugins/endpoints/migrations/versions/0023_endpoint_credentials.py`
declares `plugin_name = 'endpoints'` (line 33) and then runs
`op.drop_index(..., table_name='assets')` at line 73 and `op.drop_column('assets',
column)` at line 76 against the core `assets` table, cleaning up after the
shipped 0021 which had added those columns from the same plugin chain. `assets`
is not, and cannot be, in `PLUGIN_TABLE_OWNERS['endpoints']`
(`shopdb/plugins/alembic_template.py:57-63`), so the metadata ownership filter
never saw it. Every step is guarded on the live schema and the forward direction
is clean, so nothing is broken; nothing stops the next one either.
### 6. Implicit Database Connection Sharing
**Verdict: REFUTED as a blanket claim.**
Savepoints are in use where concurrency actually bites:
`plugins/endpoints/plugin.py:1318`, `:1387` and `:1692` all wrap work in
`with db.session.begin_nested():`, and the application-delta path reconciles a
unique-constraint collision inside that savepoint rather than sinking the
check-in. The original text's example - a printers rollback silently discarding
a computers write in the same request - is not backed by any cited call site,
and none was found.
A per-plugin transaction policy is still not written down anywhere. That is a
documentation item for PLUGIN-DEVELOPMENT.md, not a critical data-integrity
finding.
## High Priority Findings (P1)
### 7. Frontend Plugin Import Violations
**Verdict: STALE. Enforced and passing.**
`scripts/check-naming-and-style.sh` runs "Checking for cross-plugin / escaping
imports in plugin frontends" and the whole script exits 0 on this branch. The
established pattern is a core component consumed through the `@/` alias: 97
plugin frontend files already import that way, and the kiosk allowlist editor
added in this range follows it
(`frontend/src/components/NetworkAllowlistPicker.vue`, consumed at
`plugins/usb/frontend/views/USBSettings.vue:108` and
`plugins/printedparts/frontend/views/PrintedPartsSettings.vue:164`).
### 8. Hardcoded Site Literals in Plugins
**Verdict: REFUTED as stated. HOLDS (narrowed) as a gate coverage gap.**
`grep -rn "MACHINE_TYPES" . --include=*.py` and
`grep -rn "Bldg 300\|Bldg 400" . --include=*.py` both return **nothing**. The
ADR-015 block in `scripts/check-naming-and-style.sh:392-462` runs and passes,
and a control pattern over the same directories returns 1995 lines, so the
pipeline is not vacuous.
**The narrowed finding:** the ADR-015 scan is scoped at
`scripts/check-naming-and-style.sh:436-440` to `plugins/ shopdb/ scripts/
deploy/ frontend/src/ tools/` with `--include` on
`.py/.vue/.js/.ps1/.psm1/.sh/.iss/.json/.html`. `agent/` is not in that list and
`.cs` is not an included extension, so roughly 2,500 lines of C# added in this
range - `AgentCredentials.cs`, `AgentHttpFactory.cs`, `EnrollmentService.cs`,
`AgentPipeServer.cs`, `ProtectedFileSystem.cs`, `StrictModePreamble.cs`, plus
`agent/appsettings.json` - ship to a sister site unscanned. Re-running the
assembled site patterns by hand over `agent/` (excluding bin/obj) finds one hit,
prose in `agent/ShopDbEndpointAgent/ShopDbEndpointAgent.csproj:108`, which
ADR-015 permits. The tree is clean; the gate would not notice if it stopped
being clean. Note this is a gap in ADR-015's own stated scope, not a breach of
it: the 2026-08-14 amendment names those six directories.
### 9. Missing Error Handling on External Calls
**Verdict: REFUTED.**
The plugins the finding names do not exist. `ls plugins/` gives: applications,
backups, computers, employees, endpoints, geenforce, guides, knowledgebase,
machines, measuringtools, network, notifications, printedparts, printers,
slides, tools, usb, warranty. There is no ServiceNow plugin, no LDAP sync
plugin and no email plugin; mail is core (`shopdb/utils/mailer.py`).
Every outbound HTTP call in the tree carries a timeout and an exception handler:
- `plugins/warranty/services/providers.py:129` and `:157` - `timeout=HTTP_TIMEOUT`,
`except requests.RequestException` raising `WarrantyLookupError`.
- `plugins/printers/services/zabbix_service.py:95` and `:126` -
`timeout=self.REACHABLE_TIMEOUT` / `self.API_TIMEOUT`, caught.
- `shopdb/utils/mailer.py:238` - `timeout=10`, caught, returns `(False, error)`
and never raises.
### 10-12. Additional High Findings
The original text deferred these to a memory file rather than citing code:
"inconsistent permission checking across plugins", "plugin state leakage via
module-level globals", "missing database connection pool tuning".
The third is REFUTED. The pool is sized on purpose and env-driven:
`shopdb/config.py:20-22` (`DB_POOL_SIZE` 10, `DB_MAX_OVERFLOW`, `DB_POOL_TIMEOUT`
5) and `:48-72`, with the reasoning for the non-default `pool_timeout` written
in place, and `docker-compose.yml:172-193` documents the
`workers x (pool_size + max_overflow)` arithmetic against MySQL's
`max_connections`.
The other two were not re-checked, because neither cites a file. They should be
restated with a file:line or dropped.
## Medium Priority Findings (P2)
### 13. No Rate Limiting on API Endpoints
**Verdict: HOLDS (narrowed). Login is limited; two kiosk surfaces are not.**
Login rate limiting exists and is not "planned Q1 2027": `shopdb/core/api/auth.py:35`
`_login_ratelimited()`, keyed per host at `:52`, applied at `:80`, with
`tests/test_core/test_auth_ratelimit.py` over it. Enrollment has its own limiter
(`plugins/endpoints/api/ratelimit.py`, 300 per 60 seconds per IP applied at
`plugins/endpoints/api/enrollment.py:306`).
What has none: the USB kiosk routes. `grep` for `limiter` or `ratelimit` across
`plugins/usb/` returns nothing, while `plugins/usb/api/selfhosted.py:346-371`
resolves a posted badge to an employee name and returns it. That is an
unauthenticated, unthrottled badge-to-name lookup for any caller inside the
configured kiosk CIDRs.
Both limiter counters live in per-process `SimpleCache`
(`plugins/endpoints/api/ratelimit.py:10-11`), so at the shipped four gunicorn
workers the effective budget is four times the configured maximum.
### 14. Frontend State Management Inconsistencies
**Verdict: REFUTED.**
`grep -rn "vuex" frontend/src/` returns **nothing**, and `frontend/package.json:28`
declares `"pinia": "^2.1.0"`. There is no Vuex-versus-Pinia split to
standardise.
### 15. No API Versioning
**Verdict: HOLDS.**
`grep -rn "api/v1" shopdb/ --include=*.py` returns nothing; every route is
mounted under `/api/`. This is a v1.0 decision, not a defect, and ADR-002 already
carries the versioning story for the plugin contract. The 60-hour estimate has
no stated basis.
### 16-22. Additional Medium Findings
Re-checked on 2026-09-18:
- **No content security policy headers: REFUTED as stated.** `shopdb/__init__.py:340`
sets `Content-Security-Policy: frame-ancestors 'self'` on every response, plus
`X-Content-Type-Options`, `X-Frame-Options` and `Referrer-Policy` at `:334-337`;
uploads carry a stricter `default-src 'none'; sandbox`
(`shopdb/utils/uploads.py:13`), pinned by `tests/test_core/test_security_headers.py`.
The accurate, narrower statement is that there is no `script-src` policy, and
the code says so in place and says why.
- **Missing health check endpoints: REFUTED.** `shopdb/core/api/dashboard.py:318`
is `GET /api/dashboard/health`.
- **Database connection pool too small: REFUTED.** See 10-12 above.
- **No request ID tracing: HOLDS.** No request-id plumbing was found in
`shopdb/__init__.py`.
- **No structured logging format: not re-checked.**
- **Plugin discovery relies on a filesystem scan: HOLDS, by design.** ADR-003
decides filesystem discovery under `plugins/`. "Slow" was never measured.
- **Frontend bundle size 2.1MB: UNVERIFIED.** No build was run for this page.
## Remediation Priority (revised 2026-09-18)
| Finding | Verdict | What is actually left to do |
|---------|---------|------------------------------|
| Circular plugin dependencies | REFUTED / narrowed | Decide whether deferred cross-plugin imports must be guarded; guard `agentauth.py:118` and `policy_routes.py:44` |
| Undeclared core imports | REFUTED | Nothing |
| Schema drift | REFUTED / narrowed | Align `notification.py:158` with migration 0006; teach the naming gate to read `sa.Column('...')` |
| Plugin chain touches core tables | HOLDS (new) | Add a test that a plugin chain writes only owned tables |
| Migration chain validation | STALE | Nothing; `tests/test_plugin_migrations.py` is the gate |
| Implicit DB session sharing | REFUTED | Document the transaction pattern in PLUGIN-DEVELOPMENT.md |
| Frontend plugin imports | STALE | Nothing; gate passes |
| Site literals in plugins | REFUTED / narrowed | Extend the ADR-015 scan to `agent/` and `.cs` |
| Missing external-call error handling | REFUTED | Nothing |
| Rate limiting | HOLDS (narrowed) | Throttle the USB kiosk badge lookup; decide whether per-process counters are acceptable at 4 workers |
| Vuex/Pinia split | REFUTED | Nothing |
| API versioning | HOLDS | v1.0 decision |
| Single points of failure | UNVERIFIED | Belongs in a deployment document |
## Testing Performed (2026-09-18 re-check)
```bash
# Every claim above was checked with a grep or a read on the deployment branch at 35d4fcd41.
grep -rn "^from plugins\." plugins/ --include=*.py # 0 hits
grep -rn "from shopdb\.\(core\|utils\|extensions\|services\)" \
plugins/ --include=*.py # 0 hits
grep -rPn "sa\.Column\(\s*'[a-z]+_[a-z_]+'" \
plugins/*/migrations/versions/*.py migrations/versions/*.py # 0 hits
grep -rn "MACHINE_TYPES\|Bldg 300\|Bldg 400" . --include=*.py # 0 hits
grep -rn "vuex" frontend/src/ # 0 hits
grep -rn "api/v1" shopdb/ --include=*.py # 0 hits
bash scripts/check-naming-and-style.sh # exit 0
```
No test suite was run for this re-check; every verdict above is static.
## References
- ADR-001: Asset as Platform Contract (plugin isolation model)
- ADR-003: Plugin Distribution (filesystem discovery)
- ADR-008: Plugin Migration Ownership (linear chains, PLUGIN_TABLE_OWNERS)
- ADR-013: Plugin Catalog and Lean Builds (frontend isolation)
- ADR-015: Site-Specific Configuration (no literals in code)
- ADR-Conformance-2026-09-18.md: the ADR-by-ADR result of the 2026-09-18 review,
including the fact that the ADR set is not in this repository