ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into wilm

Browse Source 
# Conflicts:
#	shopdb/data/catalog.json
This commit is contained in:
benjamin.baker
2026-08-26 12:53:03 -04:00
parent 7cd3aadcd1 271d558f8b
commit 801dfd54ab
21 changed files with 1675 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CHANGELOG.md
View File

@@ -24,7 +24,7 @@ ADR-007 and ADR-002.
`rename` fixes a stripped name with no twin, which neither of the others can
reach. All three report and change nothing unless `--apply` is passed, and
`dedupe` is idempotent. References are discovered from the live schema, not
listed in code: there are 15 foreign keys into these three tables, and
listed in code: there are 14 foreign keys into these three tables, and
`machines.controllervendorid` is not one anybody would enumerate by hand.
- **Secure Send: hand somebody a password without leaving it in a chat
@@ -90,6 +90,16 @@ ADR-007 and ADR-002.
### Fixed
- **The shipped catalog offered two ways to spell the same vendor, and one row
that should never have left a test.** `HP Inc.` and `Zebra Technologies` were
empty rows duplicating `HP` and `Zebra`, which carry every model between
them; neither exists in this site's database, so they would have arrived as
duplicates at a new site and nowhere else. `UnscopedLiveVendor` was a test
artifact: `export_catalog.py` reads the live development database, so a row
left behind by a lab run was published as reference data. The canonical name
is whichever spelling holds the entries, which is why it is `Dell Inc.` but
`HP` and `Zebra`. Vendors 51 to 48.
- **The shipped seed catalog contained duplicate vendors, models and operating
systems.** Four space-stripped twins, each with a correctly spaced original
already present, so a new site inherited the same company listed twice.

29
plugins/backups/api/routes.py
View File

@@ -156,9 +156,34 @@ def asset_info(assetid):
if revision is None:
return success_response({'fields': [], 'sectioncount': 0})
try:
projection = revision.payload
except ValueError:
# payload deliberately raises on unreadable payloadjson so a corrupt row
# is named rather than mistaken for an opaque one. That is right for a
# download; for an at-a-glance card it would take the whole asset page
# down over one bad row, so the card degrades to empty instead.
current_app.logger.warning(
'backuprevision %s has unreadable payloadjson; info card empty',
revision.backuprevisionid)
return success_response({'fields': [], 'sectioncount': 0})
data = kind.buildinfo(
revision.payload, assetid,
partmarkertypes=current_app.config.get('BACKUPS_PARTMARKER_TYPES'))
projection, assetid,
partmarkertypes=current_app.config.get('BACKUPS_PARTMARKER_TYPES'),
# A kind that diffs against its own previous revision needs to know
# which revision it is looking at. Passed generically rather than
# queried inside the kind, so the endpoint stays the single place that
# decides "latest".
revisionid=revision.backuprevisionid,
# The card names the PC a snapshot came from, and chains its diff on the
# same value. Passed from the column rather than read out of the stored
# manifest, which carries its own copy that can disagree.
sourcehostname=revision.sourcehostname,
# Where the snapshot actually is. Read off the revision rather than
# composed from the configured share root, so a snapshot written before
# the layout changed still points at where it was really put.
sharepath=revision.sharepath)
data['backuprevisionid'] = revision.backuprevisionid
data['collectedat'] = _utciso(revision.collectedat)
return success_response(data)

11
plugins/backups/frontend/routes.js
View File

@@ -12,6 +12,17 @@
* JWT + a backups.* permission; an anonymous visit would only render errors.
*/
export default [
{
// The plugin's five settings had no page at all until this one: the core
// settings catalog is a hardcoded list that a plugin cannot extend, so
// backups_shareroot and friends were reachable only through the API or a
// shell on the server. Paired with get_settings_cards in plugin.py, which
// puts the card in the rail - both vanish when the plugin is disabled.
path: 'settings/backups',
name: 'backups-settings',
component: () => import('./views/BackupsSettings.vue'),
meta: { requiresAuth: true, requiresAdmin: true, plugin: 'backups' }
},
{
path: 'backups/asset/:assetid',
name: 'backups-asset-history',

43
plugins/backups/frontend/views/BackupHistory.vue
View File

@@ -66,6 +66,16 @@
UNC path so the tech can open it directly. -->
<template v-if="rev.storagebackend === 'share'">
<code class="bh-path" :title="rev.sharepath">{{ rev.sharepath }}</code>
<!--
A copy button rather than a file:// link. Browsers block
navigation from an https page to file://, so a link would look
clickable and silently do nothing - worse than no link. Copy,
then paste into Explorer's address bar.
-->
<button type="button" class="bh-copy" @click="copyPath(rev.sharepath)"
:title="'Copy path for Explorer'">
{{ copiedPath === rev.sharepath ? 'Copied' : 'Copy path' }}
</button>
</template>
<template v-else>
<button v-for="fmt in rev.formats" :key="fmt.id"
@@ -115,6 +125,29 @@ import { useRoute, useRouter } from 'vue-router'
import api, { settingsApi } from '@/api'
import { formatInZone, DEFAULT_TZ } from '@/utils/datetime'
const copiedPath = ref('')
async function copyPath(path) {
if (!path) { return }
try {
await navigator.clipboard.writeText(path)
copiedPath.value = path
setTimeout(() => { copiedPath.value = '' }, 2000)
} catch (err) {
// Clipboard needs a secure context and a user gesture. Both hold here, but
// an older browser or a http:// instance will refuse - select the text so
// the tech can copy it by hand instead of getting nothing.
const node = document.querySelector('.bh-path')
if (node && window.getSelection) {
const range = document.createRange()
range.selectNodeContents(node)
const selection = window.getSelection()
selection.removeAllRanges()
selection.addRange(range)
}
}
}
const route = useRoute()
const router = useRouter()
const assetid = route.params.assetid
@@ -274,6 +307,16 @@ onMounted(async () => {
font-size: 0.82rem;
}
.bh-path { word-break: break-all; color: var(--text-light); }
/* Deliberately quiet: the path is the content, this is just the way to get it
into Explorer. Sits inline after the path rather than in the actions column,
which belongs to the revision itself. */
.bh-copy {
font-size: 0.72rem; margin-left: 0.4rem; padding: 0.12rem 0.4rem;
border: 1px solid var(--border); background: transparent;
color: var(--text-light); border-radius: 3px; cursor: pointer;
white-space: nowrap; vertical-align: middle;
}
.bh-copy:hover { border-color: var(--primary); color: var(--primary); }
.bh-actions-col { width: 22rem; }
.bh-btn {

157
plugins/backups/frontend/views/BackupsSettings.vue Normal file
View File

@@ -0,0 +1,157 @@
<template>
<div>
<div class="page-header">
<h2>Backups</h2>
</div>
<div class="section-card">
<div class="setting-group">
<h3>Collection</h3>
<p class="setting-description">
How often a PC attempts to capture the config it backs up. GE-Enforce runs every
cycle, so this is what stops a bay collecting several times a day. A capture that
finds nothing changed writes no revision - it only records that the check happened.
</p>
<div class="setting-row">
<label>
<span>Interval (hours)</span>
<input type="number" min="1" v-model="values.backups_intervalhours"
@blur="save('backups_intervalhours')" :disabled="saving">
<small class="input-hint">Default 24. Read by the fleet from /api/settings/public.</small>
</label>
</div>
<div class="setting-row">
<label>
<span>Share root (UNC)</span>
<input type="text" v-model="values.backups_shareroot"
placeholder="\\fileserver\shopfloor\backup"
@blur="save('backups_shareroot')" :disabled="saving">
<small class="input-hint">
Where opaque backups (part-marker snapshots) are written by the collecting PC.
Blank until the site sets it - no default is right at more than one site.
Advisory: each revision records the path it was actually written to.
</small>
</label>
</div>
</div>
</div>
<div class="section-card">
<div class="setting-group">
<h3>Health</h3>
<p class="setting-description">
A backup is judged on when it was last CONFIRMED, not when it last changed. An
unchanged config writes no revision, so a machine stable for months is healthy -
the one to worry about is the machine whose backup stopped running.
</p>
<div class="setting-row">
<label>
<span>Stale after (days)</span>
<input type="number" min="0" v-model="values.backups_staledays"
@blur="save('backups_staledays')" :disabled="saving">
<small class="input-hint">
Default 3. Drives the fleet Backup badge and the dashboard card. 0 disables both.
</small>
</label>
</div>
</div>
</div>
<div class="section-card">
<div class="setting-group">
<h3>Retention</h3>
<p class="setting-description">
Both off by default, and dedup already keeps growth low - a revision appears only
when a config really changed. Retention exists for a value that flaps. The newest
and oldest revision of a chain are never pruned.
</p>
<p class="setting-description">
<strong>Note:</strong> pruning removes database revisions only. Files written to the
share are not deleted, so enabling this on a share-backed kind leaves snapshot
directories with no revision pointing at them.
</p>
<div class="setting-row">
<label>
<span>Keep at most (revisions per chain)</span>
<input type="number" min="0" v-model="values.backups_retentioncount"
@blur="save('backups_retentioncount')" :disabled="saving">
<small class="input-hint">0 = keep everything.</small>
</label>
</div>
<div class="setting-row">
<label>
<span>Also drop older than (days)</span>
<input type="number" min="0" v-model="values.backups_retentiondays"
@blur="save('backups_retentiondays')" :disabled="saving">
<small class="input-hint">0 = no age limit.</small>
</label>
</div>
</div>
</div>
<div v-if="error" class="error-message">{{ error }}</div>
<div v-if="success" class="settings-success">{{ success }}</div>
</div>
</template>
<script setup>
// The backups plugin ships five settings and, until now, no page for any of
// them - they were reachable only through the API or a flask shell on the
// server. The core settings catalog is hardcoded and cannot be extended by a
// plugin, so this page is contributed the ADR-010 way: a settings card from
// get_settings_cards plus a route here, both of which disappear when the
// plugin is disabled.
import { ref, reactive, onMounted } from 'vue'
import { settingsApi } from '@/api'
import { apiError } from '@/utils/apiError'
const KEYS = [
'backups_intervalhours',
'backups_shareroot',
'backups_staledays',
'backups_retentioncount',
'backups_retentiondays',
]
const values = reactive({})
const saving = ref(false)
const error = ref('')
const success = ref('')
async function load() {
try {
const response = await settingsApi.list()
const rows = response.data?.data || response.data || []
// The list endpoint returns every setting; pick out ours rather than
// fetching five times.
for (const key of KEYS) {
const row = rows.find(item => item.key === key)
values[key] = row ? row.value : ''
}
} catch (err) {
error.value = apiError(err, 'Could not load backup settings')
}
}
async function save(key) {
saving.value = true
error.value = ''
success.value = ''
try {
await settingsApi.update(key, values[key])
success.value = 'Saved'
setTimeout(() => { success.value = '' }, 2000)
} catch (err) {
error.value = apiError(err, `Could not save ${key}`)
} finally {
saving.value = false
}
}
onMounted(load)
</script>

18
plugins/backups/models/backup.py
View File

@@ -67,7 +67,10 @@ class BackupRevision(db.Model):
# (assetid, backupkind).
contenthash = db.Column(db.String(64), nullable=False, index=True)
# Populated for storagebackend='shopdb' only. MEDIUMTEXT holding serialized
# Populated for storagebackend='shopdb', and optionally for 'share' kinds
# that send a manifest describing what their snapshot contains (the bytes
# still live on the share; this is the index, not the content).
# MEDIUMTEXT holding serialized
# JSON rather than a native JSON column, matching geenforce.manifestjson.
# Text also means the bytes come back exactly as written, so the canonical
# key ordering the codec produces survives the round trip - a native JSON
@@ -105,7 +108,18 @@ class BackupRevision(db.Model):
@property
def payload(self):
"""Decoded payloadjson, or None for opaque (share-backed) kinds."""
"""Decoded payloadjson, or None when the row stores nothing.
A 'shopdb' kind always has one: the parsed projection ShopDB is
authoritative for. A 'share' kind has one only when its client sends a
manifest describing the snapshot's contents, and None otherwise - which
includes every share revision written before manifests existed.
Note the manifest is NOT covered by contenthash: the hash attests the
bytes on the share, so two posts with identical content but different
manifests dedup to one row. apply_collector_payload backfills a missing
manifest on a no-op for that reason.
"""
if not self.payloadjson:
return None
try:

90
plugins/backups/plugin.py
View File

@@ -133,6 +133,31 @@ class BackupsPlugin(BasePlugin):
panels.append(panel)
return panels
def get_settings_cards(self) -> List[Dict]:
"""ADR-010: contribute the Backups settings card.
The core settings catalog (settingsNav.js) is a hardcoded list a plugin
cannot extend, so this plugin shipped five settings and no page for any
of them - backups_shareroot and the collection interval were reachable
only through the API or a shell on the server, which is not a
configurable setting in any useful sense.
Filed under Machines because that is where the assets these back up
live, and it puts the card next to Machine Types rather than in a
platform group nobody visits.
"""
return [
{
'group': 'Machines',
'to': '/settings/backups',
'icon': 'package',
'title': 'Backups',
'description': 'Collection interval, share root for part-marker '
'snapshots, staleness threshold and revision retention',
'position': 21,
},
]
def get_settings_defaults(self) -> List[Dict]:
"""Schedule + retention, read by the collector and the prune job.
@@ -263,6 +288,15 @@ class BackupsPlugin(BasePlugin):
'type': 'string',
'description': "Full UNC path. Required for 'share' kinds.",
},
'manifest': {
'type': 'object',
'description': "Optional description of a 'share' kind's "
"contents - what is in the snapshot, not "
"the bytes themselves. Lets an asset panel "
"list a snapshot and diff it against the "
"previous one without the app server "
"mounting the share.",
},
'sourcefilename': {'type': 'string'},
'sourcehostname': {
'type': 'string',
@@ -325,6 +359,42 @@ class BackupsPlugin(BasePlugin):
"contenthash and sharepath are required for kind '{}'".format(
kindkey))
# A share kind may also describe itself. The bytes stay on the
# share - that part is unchanged - but a manifest of what the
# snapshot CONTAINS is small, structured, and the only way an asset
# panel can list a part marker's patterns or say which one
# disappeared last month. The alternative is the app server
# mounting SFLD, which the read routes deliberately refuse to do.
#
# It is advisory: dedup still keys on contenthash, so a client that
# sends no manifest behaves exactly as before.
manifest = payload.get('manifest')
if manifest is not None:
# Validated at the door, not at render time. A shallow check
# here would let a malformed manifest become the LATEST
# revision, and every view of the asset panel would then fail
# until the content next changed - a client bug turning into a
# broken page that outlives it.
if not isinstance(manifest, dict):
raise ValueError('manifest must be an object')
files = manifest.get('files')
if files is not None:
if not isinstance(files, list):
raise ValueError('manifest files must be a list')
for entry in files:
if not isinstance(entry, dict):
raise ValueError(
'each manifest file must be an object')
if not isinstance(entry.get('name'), str) or not entry['name']:
raise ValueError(
'each manifest file needs a name')
size = entry.get('bytes')
if size is not None and not isinstance(size, int):
raise ValueError(
'manifest file {!r} has a non-integer size'.format(
entry['name']))
projection = manifest
# Dedup is against the LATEST row of THIS SOURCE's chain, not the
# asset's. Several PCs legitimately share one machine number here - the
# part markers on 0613, 0615 and WJPRT do, and their configs genuinely
@@ -356,6 +426,26 @@ class BackupsPlugin(BasePlugin):
# backup that died six months ago. Cheap: one column on a row that
# already exists, no new history.
latest.lastseenat = datetime.utcnow()
# BACKFILL, on a no-op. The contenthash attests the BYTES, not the
# manifest, so a client that starts describing itself posts the same
# hash as before and would otherwise no-op forever - leaving the
# index unstored on exactly the markers that are healthy and
# unchanging, which is most of them. A revision that already has a
# manifest is left alone; this only fills a hole.
#
# Honest by construction: the hash proves the content is identical,
# so indexing it now describes the same snapshot. lastseenat
# semantics are untouched.
if projection is not None and latest.payload is None:
latest.payload = projection
# Same reasoning for the pointer: a snapshot that moved on the share
# (a changed layout, a renamed asset directory) reports a new
# sharepath with unchanged bytes, and the stored path would
# otherwise go on pointing at a directory nobody writes to.
if sharepath and latest.sharepath != sharepath:
latest.sharepath = sharepath
db.session.flush()
return {
'action': 'noop',

184
plugins/backups/services/partmarkerinfo.py Normal file
View File

@@ -0,0 +1,184 @@
"""Marker Contents card.
What a part marker holds right now, on the marker's own asset page, plus what
changed since the previous snapshot.
WHY THIS EXISTS AT ALL
A 'share' kind keeps its bytes on SFLD and only a UNC pointer in ShopDB, so
the obvious way to list a snapshot's contents - read the directory - is
exactly what the read routes refuse to do: mounting the share in the app
server turns a permissions slip into an unexplained empty panel. Instead the
collector sends a MANIFEST alongside the hash, and this builds the card from
that. Small, structured, and it works from a server that cannot see the
share at all.
WHAT IT ANSWERS
"What patterns are on this marker" - the question a tech arrives with, and
today the only way to answer it is to walk to the bay and read the panel.
"What changed" - the added/removed diff against the previous snapshot. This
is the reason the snapshot layout exists: a pattern deleted from a
controller is invisible everywhere else until someone needs it. Here it is
dated, and the snapshot that still holds it is one row above in the history.
The manifest is written by Backup-PartMarker.ps1 and carries, per file: name,
bytes, sha256, and the marking system it was authored for.
"""
def _previousmanifest(assetid, currentid):
"""The manifest of the snapshot before this one, or None.
Scoped to the same source host, matching how dedup chains revisions: two
PCs reporting the same marker keep separate histories, and diffing across
them would report every file as added and removed at once.
The host comes from the REVISION's own column, not from a field inside the
manifest. Those can disagree - the column is what the collector recorded and
what retention chains on - and using the manifest's copy would silently diff
against the wrong chain, or against nothing at all.
"""
from shopdb.api import db
from ..models import BackupRevision
from .retention import samesource
current = db.session.get(BackupRevision, currentid)
if current is None:
return None, None
sourcehostname = current.sourcehostname
row = (db.session.query(BackupRevision)
.filter(BackupRevision.assetid == assetid,
BackupRevision.backupkind == 'partmarker',
BackupRevision.backuprevisionid < currentid,
samesource(sourcehostname))
.order_by(BackupRevision.backuprevisionid.desc())
.first())
if row is None:
return None, None
when = row.collectedat or row.createdat
try:
return row.payload, when
except ValueError:
# A corrupt neighbour must not take the card down - the current
# snapshot is still worth showing without its diff.
return None, None
def _hashes(manifest):
"""name -> sha256, for spotting a file edited in place."""
return {entry.get('name'): entry.get('sha256')
for entry in (manifest or {}).get('files', [])
if entry.get('name') and entry.get('sha256')}
def _names(manifest):
return {entry.get('name') for entry in (manifest or {}).get('files', [])
if entry.get('name')}
def _summarise(files):
"""Counts by extension, so 34 patterns and 12 settings read as two facts."""
groups = {}
for entry in files:
name = entry.get('name') or ''
ext = name.rsplit('.', 1)[-1].upper() if '.' in name else 'OTHER'
groups[ext] = groups.get(ext, 0) + 1
labels = {'PAT': 'Patterns', 'DAT': 'Settings files', 'TFF': 'Fonts'}
return [(labels.get(ext, ext), count) for ext, count in sorted(groups.items())]
def build(projection, assetid, revisionid=None, sourcehostname=None,
sharepath=None, **options):
"""Build the card from a stored partmarker manifest.
Returns the keyvalue shape. An older revision predating manifests has no
projection at all, which renders as no card rather than an error - the
revision itself is still listed in the history panel.
"""
# "No manifest" and "a manifest listing nothing" are different facts. The
# first is an older revision predating manifests, which has no card. The
# second is a controller that has been wiped or replaced - precisely the
# disaster the Removed line exists to report - so it must render, with the
# diff intact.
if not projection:
return {'fields': [], 'sectioncount': 0}
manifest = projection
files = manifest.get('files') or []
fields = []
for label, count in _summarise(files):
fields.append({'label': label, 'value': str(count)})
total = sum(int(entry.get('bytes') or 0) for entry in files)
fields.append({'label': 'Total size', 'value': '{:,} bytes'.format(total)})
if manifest.get('family'):
fields.append({'label': 'Controller', 'value': manifest['family']})
# The heads patterns were authored for. Worth showing because they can
# DIFFER from the controller holding them - a TMC420 in the reference site
# holds both a TMP3200/470 and a TMP4210/420 pattern - and pushing a pattern
# back onto a system with a different head is a mistake worth catching
# before the restore, not after.
systems = manifest.get('systems') or []
if systems:
fields.append({'label': 'Authored for', 'value': ', '.join(systems)})
# The snapshot's own location, in full, as plain text. NOT a link: browsers
# refuse to navigate from an https page to file://, so a clickable path
# would look actionable and silently do nothing, which is worse than no
# link at all. The history page pairs the same string with a copy button;
# here it is what a tech reads or selects before pasting it into Explorer.
if sharepath:
fields.append({'label': 'Snapshot', 'value': sharepath, 'mono': True})
# From the revision's own column where the caller passes it: the manifest
# carries its own copy, and the two can disagree. The column is what the
# collector recorded and what the diff above is chained on, so naming a
# different PC here would contradict the comparison shown beside it.
capturedfrom = sourcehostname or manifest.get('sourcehost')
if capturedfrom:
fields.append({'label': 'Captured from', 'value': capturedfrom})
# The diff. Only meaningful with a predecessor: on a marker's first snapshot
# every file would otherwise read as "added", which says nothing.
if revisionid:
previous, when = _previousmanifest(assetid, revisionid)
if previous is not None:
before, now = _names(previous), _names(manifest)
added = sorted(now - before)
removed = sorted(before - now)
# Same name, different bytes. A revision exists only because the
# content hash moved, so if the file SET is identical then something
# was edited in place - and saying "no files added or removed" would
# be a flat contradiction of why this row exists. The per-file
# sha256 is in the manifest, so the edited files can be named.
beforehashes = _hashes(previous)
changed = sorted(name for name, digest in _hashes(manifest).items()
if name in beforehashes and beforehashes[name] != digest)
# Date the window. Retention prunes middle revisions, so "previous"
# can be months older than the row before it in the history, and an
# undated diff would quietly attribute an old removal to this
# capture.
label = 'Since {}'.format(when.strftime('%Y-%m-%d')) if when else 'Since previous'
if added:
fields.append({'label': 'Added', 'value': ', '.join(added)})
if removed:
# The one the whole design exists for. Named, not counted -
# "1 removed" would send someone to the share to find out which.
fields.append({'label': 'Removed', 'value': ', '.join(removed)})
if changed:
fields.append({'label': 'Changed', 'value': ', '.join(changed)})
if not (added or removed or changed):
fields.append({'label': label, 'value': 'No files added, removed or changed'})
else:
fields.append({'label': 'Compared with', 'value': label.replace('Since ', '')})
return {'fields': fields, 'sectioncount': 1}

60
plugins/backups/services/registry.py
View File

@@ -279,11 +279,61 @@ class PartMarkerKind(BackupKind):
key = 'partmarker'
displayname = 'Part Marker Configuration'
storagebackend = 'share'
# NO asset panel. The marker's DNC settings are a tab on the single DNC Info
# card, and a separate card that is empty on 145 of 147 machines earns
# nobody anything. The kind is still fully live - it stores, dedupes and
# serves revisions, which are listed on the backup history page.
assettypes = []
# Marker assets are core type 'machine', like the other 144 machines - but
# only a marker ever has a partmarker revision, and the list renderer hides
# a panel with no rows and no empty text. So declaring 'machine' shows the
# panel on markers and nowhere else, with no special-casing.
#
# This was deliberately EMPTY while backups were filed against the operation
# number, when a partmarker card would have appeared on every machine and
# said nothing. Since a marker PC's backup is filed against its own marker
# asset (see markerforsource), the panel has a home of its own.
assettypes = ['machine']
def sharedir(self, machinetype, identifier, shareroot=None):
"""<root>\\partmarker\\<marker asset>\\ - the SITE's layout, which
inverts the base class's <root>\\<machinetype>\\<identifier>\\<kind>.
Overridden rather than left to the default so what ShopDB displays and
validates matches what the collector actually writes. Still advisory:
the authoritative path is the sharepath on each revision, which is what
keeps older snapshots resolvable after a PC swap renames the asset.
Snapshots live in timestamped subdirectories BELOW this - one per
capture, so a pattern deleted from the controller stays recoverable
from the snapshot that still holds it.
"""
root = shareroot or DEFAULTSHAREROOT
if not root:
return ''
return '{}\\{}\\{}'.format(root, self.key, identifier or 'unknown')
def infopanel(self):
"""Contents of the latest snapshot, on the marker's own asset page.
A 'share' kind stores no bytes in ShopDB, so this is built from the
manifest the collector sends alongside the hash. Without it the app
server would have to mount SFLD to say what a snapshot contains, which
the read routes refuse to do on purpose.
"""
return {
'id': 'backups-partmarker-contents',
'title': 'Marker Contents',
'assettypes': ['machine'],
'endpoint': '/api/backups/asset/{assetid}/info?kind=partmarker',
'render': 'keyvalue',
# Above the history panel: "what is on this marker" is the question
# a tech arrives with; the history is for the rarer restore.
'position': 39,
}
def buildinfo(self, projection, assetid, **options):
from . import partmarkerinfo
return partmarkerinfo.build(
projection, assetid,
revisionid=options.get('revisionid'),
sourcehostname=options.get('sourcehostname'),
sharepath=options.get('sharepath'))
def resolveassetid(self, payload):
from shopdb.api import db, Asset

56
plugins/geenforce/api/routes.py
View File

@@ -1079,10 +1079,64 @@ def _attach_backup_state(facts, hostnames):
rows.sort(key=lambda revision: (revision.lastseenat is not None,
revision.lastseenat or _EPOCH),
reverse=True)
# Reduce to the FRESHEST revision per (host, kind), then judge a host by its
# STALEST kind.
#
# Until part markers, a PC reported exactly one kind and "newest wins" was
# the same thing as "the truth". A marker bay reports two - ntlars for the
# machine's DNC settings and partmarker for the controller's patterns - and
# newest-wins then means the daily ntlars post keeps the badge green while
# the marker backup has been failing for a month. The failure would show
# only on that marker's own asset page, which is not where anyone looks to
# find out that something stopped.
#
# So: a host is healthy when EVERYTHING it backs up is current, and the
# badge names the kind that is not. Freshest-per-kind first, because
# lastseenat only moves on the newest revision of a chain - an older
# revision of the same kind keeps its original value and would otherwise
# read as stale for ever.
#
# KNOWN CONSEQUENCE: a device retired from a PC leaves its last revision
# behind, so that host stays stale until the revision is pruned or removed.
# That is the honest failure direction - it complains about a backup that
# genuinely is not running any more, rather than staying quiet about one.
freshest = {}
for revision in rows:
key = (revision.sourcehostname or '').lower()
if key not in facts:
continue
kindkey = (key, revision.backupkind)
if kindkey not in freshest:
# rows are already newest-confirmed-first, so the first is freshest.
freshest[kindkey] = revision
# Among kinds that can be JUDGED, the stalest decides. A kind whose newest
# revision has never been confirmed (lastseenat NULL - a backfilled row, or
# one written before the column existed) is not stale; it is unmeasurable,
# and it must not win.
#
# Treating NULL as "worst" looked right and was wrong: a host with a healthy
# ntlars backup and one never-confirmed row showed NO BADGE, because the
# verdict for a NULL confirmation is tri-state None. A working backup then
# renders as nothing at all, which is the one thing the badge must never do.
# This is the same rule staleness.py already applies: a chain becomes
# measurable the first time its PC posts, and says nothing until then.
stalest = {}
for (key, _kind), revision in freshest.items():
if revision.lastseenat is None:
continue
current = stalest.get(key)
if current is None or revision.lastseenat < current.lastseenat:
stalest[key] = revision
# Every kind unmeasurable: fall back to any revision so the host still
# reports which kind it has, with a None verdict and no badge.
for (key, _kind), revision in freshest.items():
stalest.setdefault(key, revision)
for key, revision in stalest.items():
entry = facts.get(key)
# First row per host wins.
if entry and entry['backuplastseen'] is None:
entry['backupkind'] = revision.backupkind
entry['backuplastseen'] = (

13
plugins/guides/manifest.json Normal file
View File

@@ -0,0 +1,13 @@
{
"name": "guides",
"display_name": "Guides",
"version": "1.0.0",
"description": "Step-by-step guides written and read inside ShopDB, with images, working checklists and printable output",
"author": "ShopDB Team",
"dependencies": [],
"core_version": ">=0.1.0,<1.0.0",
"api_prefix": "/api/guides",
"provides": {
"features": ["guides"]
}
}

3
plugins/guides/models/__init__.py Normal file
View File

@@ -0,0 +1,3 @@
from .guide import Guide, GuideImage, BLOCKTYPES
__all__ = ['Guide', 'GuideImage', 'BLOCKTYPES']

188
plugins/guides/models/guide.py Normal file
View File

@@ -0,0 +1,188 @@
"""Guides: step-by-step procedures written and read inside ShopDB.
WHY THIS IS NOT THE KNOWLEDGE BASE PLUGIN
The knowledgebase plugin is a LINK REGISTRY - a short description, a URL and
some keywords, pointing at a document that lives somewhere else. That is a
different job. This holds the document itself.
WHY BLOCKS RATHER THAN A BLOB OF HTML
A guide is stored as an ordered list of typed blocks, each with a stable id,
serialized as JSON. Free-form HTML would be easier to write and would defeat
three of the four things this feature exists for:
checklists a tickable step has to be identifiable to be remembered, and
an <input> buried in pasted markup has no identity
dark mode pasted content carries hardcoded colours, which then fight the
theme - black text on a dark background, and nobody can say
why
printing print has to know where a step begins and ends to avoid
splitting one across a page
Blocks also mean a new block type later is additive: old guides keep
rendering, because the renderer skips what it does not know.
INLINE TEXT IS A RESTRICTED MARKDOWN SUBSET - bold, italic, inline code, links,
line breaks, and nothing else. No fonts, sizes or colours: emphasis is semantic
and the theme decides how it looks. That single rule is what keeps a guide
written next year looking like one written today, and what makes dark mode work
without an author thinking about it.
"""
import json
from datetime import datetime
from shopdb.api import db
from shopdb.core.models.base import BaseModel
def _utcnow():
return datetime.utcnow()
# Block types the renderer knows. Stored as data, not as markup, so the same
# guide can render to screen, to dark mode and to paper differently.
BLOCKTYPES = (
'heading',
'paragraph',
'step', # a numbered step, with its own body
'checklist', # tickable items - the reason progress is a feature
'bullets',
'numbers',
'callout', # info / warning / success, per the KB house style
'code',
'image',
'table',
)
class Guide(BaseModel):
"""One guide. Blocks live in a JSON column rather than a child table.
A guide is edited and saved WHOLE - nobody reorders one paragraph in
isolation - so a child table would buy ordering machinery, a second
migration and N+1 reads, in exchange for nothing this feature needs. The
same reasoning the geenforce manifests and backup payloads already follow.
"""
__tablename__ = 'guides'
guideid = db.Column(db.Integer, primary_key=True)
# Human title, and the slug the URL uses. The slug is stored rather than
# derived on read so an edited title cannot silently break a link somebody
# pasted into a work order.
title = db.Column(db.String(300), nullable=False)
slug = db.Column(db.String(300), nullable=False, unique=True, index=True)
# One line, shown in the list and in search results.
summary = db.Column(db.String(1000))
# Comma-separated, matching how knowledgebase.keywords already works, so one
# search can cover both and there is one tag vocabulary rather than two.
tags = db.Column(db.String(500))
# 'draft' or 'published'. A draft is visible to people who can edit, and
# invisible to everyone else: a half-written procedure on a shop floor is
# worse than no procedure.
status = db.Column(db.String(20), nullable=False, default='draft', index=True)
# MEDIUMTEXT: serialized blocks. Text rather than a native JSON column so the
# bytes come back exactly as written and a diff between versions is stable -
# the same reason the backups plugin stores its projection as text.
blocksjson = db.Column(db.Text(length=16777215), nullable=False, default='[]')
createdby = db.Column(db.Integer, db.ForeignKey('users.userid'))
updatedby = db.Column(db.Integer, db.ForeignKey('users.userid'))
createdat = db.Column(db.DateTime, nullable=False, default=_utcnow)
updatedat = db.Column(db.DateTime, nullable=False, default=_utcnow,
onupdate=_utcnow)
publishedat = db.Column(db.DateTime)
# Bumped on every save. Cheap, and it gives a printed copy something
# honest to put in its footer: "version 7, printed 26 Aug 2026".
revision = db.Column(db.Integer, nullable=False, default=1)
@property
def blocks(self):
"""The decoded block list, or [] on a corrupt row.
Deliberately forgiving, unlike the backups payload getter: a guide that
will not render is a page a person cannot read, and an empty guide with
its title and tags intact is a better failure than an error page. The
corruption is still visible - the editor shows no blocks.
"""
if not self.blocksjson:
return []
try:
data = json.loads(self.blocksjson)
except (ValueError, TypeError):
return []
return data if isinstance(data, list) else []
@blocks.setter
def blocks(self, value):
self.blocksjson = json.dumps(value or [], separators=(',', ':'))
def taglist(self):
"""Tags as a clean list, empties dropped."""
return [t.strip() for t in (self.tags or '').split(',') if t.strip()]
def to_dict(self, includeblocks=False):
data = {
'guideid': self.guideid,
'title': self.title,
'slug': self.slug,
'summary': self.summary,
'tags': self.taglist(),
'status': self.status,
'revision': self.revision,
'createdat': self.createdat.isoformat() + 'Z' if self.createdat else None,
'updatedat': self.updatedat.isoformat() + 'Z' if self.updatedat else None,
'publishedat': (self.publishedat.isoformat() + 'Z'
if self.publishedat else None),
}
if includeblocks:
data['blocks'] = self.blocks
return data
class GuideImage(BaseModel):
"""An image pasted or dropped into a guide.
Bytes live in the database, not on disk. Guides are small, the images are
screenshots rather than photographs, and a row travels with a database
backup - whereas a file under instance/ has been lost to a container rebuild
on this very deployment before. If a site ever puts thousands of these in,
the storage decision is one column and one route to revisit.
"""
__tablename__ = 'guideimages'
guideimageid = db.Column(db.Integer, primary_key=True)
# Nullable: an image is uploaded while a NEW guide is still unsaved, so it
# exists before the guide it belongs to does. The editor attaches it on
# save; anything still unattached is orphaned and prunable.
guideid = db.Column(db.Integer, db.ForeignKey('guides.guideid',
ondelete='CASCADE'), index=True)
filename = db.Column(db.String(255))
mimetype = db.Column(db.String(100), nullable=False)
bytesize = db.Column(db.Integer, nullable=False)
data = db.Column(db.LargeBinary(length=(2 ** 24) - 1), nullable=False)
uploadedby = db.Column(db.Integer, db.ForeignKey('users.userid'))
createdat = db.Column(db.DateTime, nullable=False, default=_utcnow)
def to_dict(self):
return {
'guideimageid': self.guideimageid,
'guideid': self.guideid,
'filename': self.filename,
'mimetype': self.mimetype,
'bytesize': self.bytesize,
'url': '/api/guides/images/{}'.format(self.guideimageid),
'createdat': self.createdat.isoformat() + 'Z' if self.createdat else None,
}

0
plugins/guides/services/__init__.py Normal file
View File

131
plugins/guides/services/blocks.py Normal file
View File

@@ -0,0 +1,131 @@
"""Block validation.
The editor is the only thing that writes blocks today, so this is not defending
against a hostile author - it is defending against a BAD SAVE. A block list that
is wrong in a way nobody notices at save time becomes a guide that will not
render weeks later, when the person who wrote it has moved on and the only
evidence is a blank page.
So: validate at the door, name the block that is wrong, and refuse the save.
The alternative - storing whatever arrives and coping at render time - was tried
in the backups plugin with a manifest and produced exactly the failure this
avoids: a client bug turning into a broken page that outlived it.
WHAT IS NOT VALIDATED HERE: the inline text. It is a restricted Markdown subset
rendered through a sanitiser in the browser, which is where a renderer belongs.
Storing text verbatim keeps what the author typed recoverable even if the
rendering rules change later.
"""
import json
from ..models import BLOCKTYPES
# A guide is a procedure, not a document repository. This cap is generous for
# anything a person will read on a shop floor and small enough that a runaway
# paste cannot fill a MEDIUMTEXT column.
MAXBLOCKSBYTES = 2 * 1024 * 1024
MAXBLOCKS = 500
MAXTEXT = 20000
class BlockError(ValueError):
"""Raised with a message naming the offending block, for the API to relay."""
def _fail(index, message):
raise BlockError('block {}: {}'.format(index + 1, message))
def validate(blocks):
"""Check a block list, returning it unchanged, or raise BlockError.
Kept pure - no database, no request - so the rules are testable directly and
the same function can be reused by an importer later.
"""
if not isinstance(blocks, list):
raise BlockError('blocks must be a list')
if len(blocks) > MAXBLOCKS:
raise BlockError('a guide is limited to {} blocks'.format(MAXBLOCKS))
seen = set()
for index, block in enumerate(blocks):
if not isinstance(block, dict):
_fail(index, 'must be an object')
blockid = block.get('id')
if not isinstance(blockid, str) or not blockid.strip():
# Ids are what make a checklist item rememberable between visits.
# A block without one silently loses its tick the next time the
# guide is opened, which reads as "the checkboxes do not work".
_fail(index, 'needs a stable id')
if blockid in seen:
_fail(index, 'duplicate id {!r} - ticks would follow the wrong item'
.format(blockid))
seen.add(blockid)
blocktype = block.get('type')
if blocktype not in BLOCKTYPES:
_fail(index, 'unknown type {!r}. Known types: {}'.format(
blocktype, ', '.join(BLOCKTYPES)))
text = block.get('text')
if text is not None:
if not isinstance(text, str):
_fail(index, 'text must be a string')
if len(text) > MAXTEXT:
_fail(index, 'text is longer than {} characters'.format(MAXTEXT))
items = block.get('items')
if items is not None:
if not isinstance(items, list):
_fail(index, 'items must be a list')
for item in items:
# Checklist items carry their own id for the same reason blocks
# do: that id is the thing a tick is stored against.
if isinstance(item, dict):
if not isinstance(item.get('id'), str) or not item['id'].strip():
_fail(index, 'every item needs an id')
if not isinstance(item.get('text', ''), str):
_fail(index, 'item text must be a string')
elif not isinstance(item, str):
_fail(index, 'items must be strings or objects')
encoded = json.dumps(blocks, separators=(',', ':'))
if len(encoded.encode('utf-8')) > MAXBLOCKSBYTES:
raise BlockError('this guide is larger than {} MB'.format(
MAXBLOCKSBYTES // (1024 * 1024)))
return blocks
def plaintext(blocks):
"""Flatten a guide to searchable text.
Search should find a guide by something written halfway down it, not only by
its title - that is how people actually look for a procedure ("the one about
the drive letter"). Cheap because it runs on save, not on every query.
"""
parts = []
for block in blocks or []:
if not isinstance(block, dict):
continue
if isinstance(block.get('text'), str):
parts.append(block['text'])
for item in block.get('items') or []:
if isinstance(item, str):
parts.append(item)
elif isinstance(item, dict) and isinstance(item.get('text'), str):
parts.append(item['text'])
for row in block.get('rows') or []:
if isinstance(row, list):
parts.extend(cell for cell in row if isinstance(cell, str))
return '\n'.join(parts)
def checklistitemcount(blocks):
"""How many tickable items a guide has, for the list and the progress bar."""
total = 0
for block in blocks or []:
if isinstance(block, dict) and block.get('type') == 'checklist':
total += len(block.get('items') or [])
return total

123
scripts/fix_drive_letter_sharepaths.py Normal file
View File

@@ -0,0 +1,123 @@
"""Rewrite backup revisions whose sharepath is a mapped drive, not a UNC path.
WHY THESE EXIST
The part-marker client writes its snapshot through the drive letter GE-Enforce
mapped (W:), because that is the connection it authenticated - reaching the same
server by another name costs a second authentication, or fails outright with
error 1219. For a while it also RECORDED that path.
A drive letter is not an answer to "where is this backup". W: exists only inside
the SYSTEM logon session, only while the enforcement cycle runs, and only on the
bay that wrote it. A tech opening the marker's page in ShopDB got a path that
resolves to nothing anywhere. The client now records the UNC; this repairs the
rows written before it did.
WHAT IT CHANGES: backuprevisions.sharepath only. No revision is created,
deleted, re-filed or re-hashed, and no file on the share is touched - the bytes
are already where the UNC will point.
Usage (on the target instance, in the app dir):
venv\\Scripts\\python scripts\\fix_drive_letter_sharepaths.py \\
--drive W: --unc "\\\\fileserver\\share\\path"
... --commit # apply; dry run by default
The --unc value is what that drive was mapped to - the SHARE root, without the
directory the snapshots live in, since that part is already in the stored path.
Check it against GE-Enforce.ps1's shopfloorShareRoot, or `net use` on a bay
mid-cycle. Take a database backup before --commit.
"""
import argparse
import os
import re
import sys
from pathlib import Path
# Run from anywhere: put the repo root (parent of scripts/) on the path.
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from dotenv import load_dotenv
# Load .env before anything reads config. Without it DATABASE_URL stays unset
# and config falls back to a hardcoded root@localhost URL: on a real deployment
# that fails with a bare 1045 Access denied naming a user the site does not use,
# and on a box where those credentials happen to work it is worse, because the
# script would WRITE to the wrong database.
load_dotenv(Path(__file__).resolve().parent.parent / '.env')
from shopdb import create_app
from shopdb.api import db
from plugins.backups.models import BackupRevision
DRIVEPATH = re.compile(r'^([A-Za-z]):(\\.*)$')
def rewrite(sharepath, drive, unc):
"""The UNC form of one stored path, or None if it needs no change.
Matches on the drive letter alone. A path that is already UNC, or sits on
some other drive, is left exactly as it is - this repairs a known mistake
rather than normalising everything it can reach.
"""
if not sharepath:
return None
match = DRIVEPATH.match(sharepath)
if not match:
return None
if match.group(1).upper() != drive.rstrip(':').upper():
return None
return unc.rstrip('\\') + match.group(2)
def main():
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--drive', default='W:',
help='Drive letter to rewrite (default W:)')
parser.add_argument('--unc', required=True,
help=r'Share root that drive pointed at, e.g. \\server\share')
parser.add_argument('--kind', default=None,
help='Limit to one backup kind (default: every kind)')
parser.add_argument('--commit', action='store_true',
help='Apply the changes. Dry run by default.')
args = parser.parse_args()
if not args.unc.startswith('\\\\'):
raise SystemExit('--unc must be a UNC path starting with two backslashes')
app = create_app()
with app.app_context():
query = db.session.query(BackupRevision)
if args.kind:
query = query.filter(BackupRevision.backupkind == args.kind)
changed = 0
for revision in query.order_by(BackupRevision.backuprevisionid).all():
newpath = rewrite(revision.sharepath, args.drive, args.unc)
if newpath is None:
continue
changed += 1
print('revision {} {}\n{}-> {}'.format(
revision.backuprevisionid, revision.sharepath,
' ' * (len(str(revision.backuprevisionid)) + 11), newpath))
if args.commit:
revision.sharepath = newpath
if not changed:
print('Nothing to fix: no revision stores a {} path.'.format(args.drive))
return
if args.commit:
db.session.commit()
print('\n{} revision(s) updated.'.format(changed))
else:
print('\n{} revision(s) WOULD be updated. Re-run with --commit to apply.'
.format(changed))
if __name__ == '__main__':
main()

4
shopdb/data/catalog.json
View File

@@ -659,7 +659,7 @@
"vendor": "Toshulin"
},
{
"vendor": "VMware"
"vendor": "VM-Ware"
},
{
"vendor": "WJDT"
@@ -1428,7 +1428,7 @@
},
{
"modelnumber": "Virtual Server",
"vendor": "VMware",
"vendor": "VM-Ware",
"modeltype": "Server"
},
{

58
tests/test_fix_drive_letter_sharepaths.py Normal file
View File

@@ -0,0 +1,58 @@
"""The path rewrite, exercised without a database.
rewrite() is pure on purpose: the risky part of this script is deciding WHICH
paths to touch, and that judgement should be testable without standing up an
app context.
"""
import importlib.util
from pathlib import Path
import pytest
SCRIPT = Path(__file__).resolve().parent.parent / 'scripts' / 'fix_drive_letter_sharepaths.py'
def _rewrite(*args, **kwargs):
# Imported lazily and by path: scripts/ is not a package, and importing the
# module runs load_dotenv + create_app imports at module scope.
spec = importlib.util.spec_from_file_location('fixsharepaths', SCRIPT)
module = importlib.util.module_from_spec(spec)
try:
spec.loader.exec_module(module)
except Exception as exc: # pragma: no cover - env-dependent
pytest.skip('script imports need an app environment: {}'.format(exc))
return module.rewrite(*args, **kwargs)
# Generic on purpose: a real site hostname in a test is still a site
# hostname in the published repo (ADR-015), and the rewrite does not care
# which server it is.
UNC = r'\\fileserver\share\path'
def test_a_mapped_drive_path_becomes_its_unc_equivalent():
got = _rewrite(r'W:\backup\partmarker\WJPRT\20260825-144500', 'W:', UNC)
assert got == UNC + r'\backup\partmarker\WJPRT\20260825-144500'
def test_a_path_that_is_already_unc_is_left_alone():
"""The script repairs a known mistake; it does not normalise what it finds."""
assert _rewrite(UNC + r'\backup\partmarker\WJPRT\x', 'W:', UNC) is None
def test_another_drive_is_left_alone():
assert _rewrite(r'S:\DT\Shopfloor\backup\x', 'W:', UNC) is None
def test_an_empty_or_missing_path_is_left_alone():
assert _rewrite('', 'W:', UNC) is None
assert _rewrite(None, 'W:', UNC) is None
def test_the_drive_letter_match_is_case_insensitive():
assert _rewrite(r'w:\backup\x', 'W:', UNC) == UNC + r'\backup\x'
def test_a_trailing_backslash_on_the_unc_does_not_double_up():
assert _rewrite(r'W:\backup\x', 'W:', UNC + '\\') == UNC + r'\backup\x'

416
tests/test_plugins/test_backups.py
View File

@@ -207,10 +207,21 @@ def test_partmarker_kind_is_share_backed_and_not_renderable():
def test_partmarker_sharedir_is_under_the_configured_backups_root():
"""The marker kind keys its tree on the MARKER, not the machine type.
It overrides the base layout (<root>\\<machinetype>\\<identifier>\\<kind>)
because the site files marker snapshots as <root>\\partmarker\\<asset>\\,
with one timestamped snapshot directory per capture below that. The
override exists so what ShopDB displays matches what the collector writes.
"""
root = r'\\fileserver\shopfloor\backups'
path = registry.getkind('partmarker').sharedir('lathe', '3204', shareroot=root)
assert path.startswith(root)
assert path.endswith(r'lathe\3204\partmarker')
assert path.endswith(r'partmarker\3204')
# The machine type is deliberately NOT in the path: a marker is identified
# by its own asset, and the type of the machine it serves is irrelevant to
# where its patterns live.
assert 'lathe' not in path
def test_sharedir_is_empty_when_no_root_is_configured():
@@ -608,8 +619,19 @@ def test_infopanel_is_kind_owned_so_a_successor_ships_its_own_card():
panel = ntlarskind.infopanel()
assert panel['id'] == 'backups-dncinfo'
assert 'kind=ntlars' in panel['endpoint']
# A kind with nothing to summarise contributes no panel at all.
assert registry.getkind('partmarker').infopanel() is None
# The part marker kind now ships a card of its own, built from the manifest
# the collector sends beside the hash. It could not before: a share kind
# stored no description of its contents, and reading the share from the app
# server is refused on purpose.
markerpanel = registry.getkind('partmarker').infopanel()
assert markerpanel['id'] == 'backups-partmarker-contents'
assert 'kind=partmarker' in markerpanel['endpoint']
# Marker assets are core type 'machine' like the other 144, and the panel
# is kept off those by having no rows and no empty text rather than by any
# special-casing.
assert markerpanel['assettypes'] == ['machine']
assert 'empty' not in markerpanel
def test_base_kind_buildinfo_is_an_empty_card():
@@ -1006,3 +1028,391 @@ def test_retention_reads_the_key_the_settings_page_writes(app, db):
# The helper cannot see it, which is the whole defect: it looks under
# plugin.backups.backups_retentioncount, which nothing writes.
assert int(BackupsPlugin().get_setting('backups_retentioncount', 0) or 0) == 0
# ---------------------------------------------------------------------------
# Share kinds that describe themselves
#
# A 'share' kind keeps its bytes on SFLD, so ShopDB knew a snapshot existed and
# where, but nothing about what was in it. The collector now accepts an optional
# manifest - the index, not the content - which is what lets a marker's asset
# page list its patterns and name the one that disappeared. Reading the share
# from the app server is refused on purpose, so this is the only route.
# ---------------------------------------------------------------------------
def _markermanifest(names, sourcehost='FBYTNCX3', family='TMC470'):
return {
'capturedat': '2026-08-25T12:00:00Z',
'machine': '3204',
'sourcehost': sourcehost,
'family': family,
'systems': ['TMP1700/470'],
'files': [{'name': n, 'bytes': 266, 'sha256': 'ab' * 32,
'system': 'TMP1700/470'} for n in names],
}
def test_a_share_kind_may_send_a_manifest_and_it_is_stored(bk_app, bk_plugin):
from plugins.backups.models import BackupRevision
with bk_app.app_context():
result = bk_plugin.apply_collector_payload({
'machinenumber': '3204',
'backupkind': 'partmarker',
'contenthash': 'a' * 64,
'sharepath': r'\\host\share\partmarker\3204\20260825-120000',
'manifest': _markermanifest(['TESTPAT.PAT', 'NET.DAT']),
})
assert result['action'] == 'created'
revision = _db.session.get(BackupRevision, result['backuprevisionid'])
# Bytes still live on the share; only the index is in the database.
assert revision.storagebackend == 'share'
assert revision.sharepath.endswith('20260825-120000')
assert [f['name'] for f in revision.payload['files']] == \
['TESTPAT.PAT', 'NET.DAT']
def test_a_share_kind_without_a_manifest_still_works(bk_app, bk_plugin):
"""The manifest is advisory. A client that sends none behaves as before."""
from plugins.backups.models import BackupRevision
with bk_app.app_context():
result = bk_plugin.apply_collector_payload({
'machinenumber': '3204',
'backupkind': 'partmarker',
'contenthash': 'b' * 64,
'sharepath': r'\\host\share\partmarker\3204\20260825-130000',
})
assert result['action'] == 'created'
assert _db.session.get(
BackupRevision, result['backuprevisionid']).payload is None
def test_a_manifest_that_is_not_an_object_is_rejected(bk_app, bk_plugin):
with bk_app.app_context():
with pytest.raises(ValueError):
bk_plugin.apply_collector_payload({
'machinenumber': '3204',
'backupkind': 'partmarker',
'contenthash': 'c' * 64,
'sharepath': r'\\host\share\x',
'manifest': 'not an object',
})
def test_marker_card_counts_by_kind_and_names_what_was_removed(bk_app, bk_plugin):
"""The removed line is the reason the snapshot layout exists.
A pattern deleted from a controller is invisible everywhere else until
somebody needs it. It is NAMED rather than counted - "1 removed" would send
a tech to the share to find out which one.
"""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
first = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'd' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['A.PAT', 'B.PAT', 'NET.DAT']),
})
_db.session.commit()
second = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'e' * 64, 'sharepath': r'\\host\s\2',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['A.PAT', 'C.PAT', 'NET.DAT']),
})
_db.session.commit()
assert first['action'] == 'created' and second['action'] == 'created'
revision = _db.session.get(BackupRevision, second['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
fields = {f['label']: f['value'] for f in card['fields']}
assert fields['Patterns'] == '2'
assert fields['Settings files'] == '1'
assert fields['Controller'] == 'TMC470'
assert fields['Added'] == 'C.PAT'
assert fields['Removed'] == 'B.PAT'
def test_the_first_snapshot_has_nothing_to_diff_against(bk_app, bk_plugin):
"""Every file would otherwise read as 'added', which says nothing."""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
result = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'f' * 64, 'sharepath': r'\\host\s\1',
'manifest': _markermanifest(['A.PAT']),
})
_db.session.commit()
revision = _db.session.get(BackupRevision, result['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
labels = [f['label'] for f in card['fields']]
assert 'Added' not in labels and 'Removed' not in labels
def test_a_revision_with_no_manifest_renders_no_card():
"""Older revisions predate manifests; they must not error the panel."""
from plugins.backups.services import partmarkerinfo
assert partmarkerinfo.build(None, 1)['sectioncount'] == 0
assert partmarkerinfo.build({}, 1)['fields'] == []
def test_a_noop_backfills_a_manifest_the_row_never_had(bk_app, bk_plugin):
"""The hash attests the BYTES, not the manifest.
A client that starts describing itself posts the same contenthash as
before, so without this the manifest would never be stored on exactly the
markers that are healthy and unchanging - which is most of them. The panel
would then be visible only on markers that had recently changed.
"""
from plugins.backups.models import BackupRevision
with bk_app.app_context():
bare = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '1' * 64, 'sharepath': r'\\host\s\old',
'sourcehostname': 'FBYTNCX3',
})
_db.session.commit()
assert _db.session.get(BackupRevision, bare['backuprevisionid']).payload is None
described = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '1' * 64, 'sharepath': r'\\host\s\new',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['A.PAT']),
})
_db.session.commit()
# Still a no-op: nothing changed on the controller, so no new history.
assert described['action'] == 'noop'
assert described['backuprevisionid'] == bare['backuprevisionid']
revision = _db.session.get(BackupRevision, bare['backuprevisionid'])
assert [f['name'] for f in revision.payload['files']] == ['A.PAT']
# And the pointer follows a snapshot that moved on the share.
assert revision.sharepath == r'\\host\s\new'
def test_a_noop_does_not_overwrite_a_manifest_that_is_already_there(bk_app, bk_plugin):
from plugins.backups.models import BackupRevision
with bk_app.app_context():
first = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '2' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['ORIGINAL.PAT']),
})
_db.session.commit()
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '2' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['REPLACEMENT.PAT']),
})
_db.session.commit()
revision = _db.session.get(BackupRevision, first['backuprevisionid'])
assert [f['name'] for f in revision.payload['files']] == ['ORIGINAL.PAT']
def test_a_file_edited_in_place_is_named_as_changed(bk_app, bk_plugin):
"""Same file set, different bytes.
A revision exists only because the hash moved, so reporting "no files added
or removed" here would contradict the reason the row is there at all.
"""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
before = _markermanifest(['A.PAT', 'B.PAT'])
after = _markermanifest(['A.PAT', 'B.PAT'])
after['files'][1]['sha256'] = 'cd' * 32
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '3' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3', 'manifest': before})
_db.session.commit()
second = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '4' * 64, 'sharepath': r'\\host\s\2',
'sourcehostname': 'FBYTNCX3', 'manifest': after})
_db.session.commit()
revision = _db.session.get(BackupRevision, second['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
fields = {f['label']: f['value'] for f in card['fields']}
assert fields['Changed'] == 'B.PAT'
assert 'Added' not in fields and 'Removed' not in fields
def test_two_pcs_reporting_one_marker_do_not_cross_diff(bk_app, bk_plugin):
"""Dedup chains per source host, and the diff must follow the same chain.
Diffing across chains would report every file as added and removed at once
the moment a second PC reported the same marker.
"""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '5' * 64, 'sharepath': r'\\host\s\a',
'sourcehostname': 'PC-A',
'manifest': _markermanifest(['ONLY-ON-A.PAT'])})
_db.session.commit()
onb = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '6' * 64, 'sharepath': r'\\host\s\b',
'sourcehostname': 'PC-B',
'manifest': _markermanifest(['ONLY-ON-B.PAT'])})
_db.session.commit()
revision = _db.session.get(BackupRevision, onb['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
fields = {f['label']: f['value'] for f in card['fields']}
# PC-B's first snapshot has no predecessor OF ITS OWN, so no diff at all
# rather than a diff against PC-A's unrelated history.
assert 'Added' not in fields and 'Removed' not in fields
def test_a_predecessor_without_a_manifest_yields_no_diff(bk_app, bk_plugin):
"""Mixed rollout: the previous snapshot predates manifests."""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '7' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3'})
_db.session.commit()
second = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '8' * 64, 'sharepath': r'\\host\s\2',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['A.PAT'])})
_db.session.commit()
revision = _db.session.get(BackupRevision, second['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
labels = [f['label'] for f in card['fields']]
assert 'Added' not in labels and 'Removed' not in labels
assert 'Patterns' in labels # the card itself still renders
def test_a_wiped_controller_still_renders_and_reports_the_loss(bk_app, bk_plugin):
"""files: [] is a fact, not an absence.
A controller wiped or swapped is the disaster the Removed line exists for,
so it must not render as no panel at all.
"""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': '9' * 64, 'sharepath': r'\\host\s\1',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['GONE.PAT'])})
_db.session.commit()
empty = _markermanifest([])
wiped = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'a1' * 32, 'sharepath': r'\\host\s\2',
'sourcehostname': 'FBYTNCX3', 'manifest': empty})
_db.session.commit()
revision = _db.session.get(BackupRevision, wiped['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid)
fields = {f['label']: f['value'] for f in card['fields']}
assert card['sectioncount'] == 1
assert fields['Removed'] == 'GONE.PAT'
def test_a_malformed_manifest_is_refused_before_it_can_break_a_panel(bk_app, bk_plugin):
"""A client bug must not become a page that fails until content changes."""
bad = [
{'files': ['A.PAT']}, # not objects
{'files': [{'bytes': 10}]}, # nameless
{'files': [{'name': 'A.PAT', 'bytes': '1,024'}]}, # size not an int
{'files': 'A.PAT'}, # not a list
]
with bk_app.app_context():
for manifest in bad:
with pytest.raises(ValueError):
bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'b2' * 32, 'sharepath': r'\\host\s\x',
'manifest': manifest})
def test_buildinfo_kwargs_do_not_disturb_the_live_ntlars_card(bk_app, bk_plugin):
"""The endpoint passes revisionid and sourcehostname to EVERY kind.
NTLARS is live in production; a new keyword reaching its buildinfo must be
absorbed by **options rather than raising.
"""
with bk_app.app_context():
result = bk_plugin.apply_collector_payload(_payload())
_db.session.commit()
from plugins.backups.models import BackupRevision
revision = _db.session.get(BackupRevision, result['backuprevisionid'])
# The assertion is that this does not raise TypeError on the new
# keywords, not that this particular fixture produces sections.
card = registry.getkind('ntlars').buildinfo(
revision.payload, revision.assetid,
partmarkertypes=None, revisionid=revision.backuprevisionid,
sourcehostname=revision.sourcehostname)
assert 'sectioncount' in card
def test_the_card_shows_the_snapshot_path_it_was_actually_written_to(bk_app, bk_plugin):
"""Plain text, and from the REVISION - not composed from the setting.
A snapshot written before the share layout changed still lives where it was
put, so the stored path is the only honest answer. It is not a link: a
browser refuses to navigate from https to file://, so a clickable path
would look actionable and do nothing.
"""
from plugins.backups.services import partmarkerinfo
from plugins.backups.models import BackupRevision
with bk_app.app_context():
result = bk_plugin.apply_collector_payload({
'machinenumber': '3204', 'backupkind': 'partmarker',
'contenthash': 'c3' * 32,
'sharepath': r'\\host\share\partmarker\3204\20260825-144500',
'sourcehostname': 'FBYTNCX3',
'manifest': _markermanifest(['A.PAT'])})
_db.session.commit()
revision = _db.session.get(BackupRevision, result['backuprevisionid'])
card = partmarkerinfo.build(revision.payload, revision.assetid,
revisionid=revision.backuprevisionid,
sharepath=revision.sharepath)
snapshot = [f for f in card['fields'] if f['label'] == 'Snapshot']
assert snapshot and snapshot[0]['value'] == revision.sharepath
assert snapshot[0]['mono'] is True
def test_the_card_omits_the_snapshot_row_when_no_path_is_passed(bk_app, bk_plugin):
from plugins.backups.services import partmarkerinfo
card = partmarkerinfo.build(_markermanifest(['A.PAT']), 1)
assert 'Snapshot' not in [f['label'] for f in card['fields']]

22
tests/test_plugins/test_geenforce_publish_gate.py
View File

@@ -155,9 +155,20 @@ def test_hosts_below_libversion_reports_the_floor(db):
# -- backup state on the reports table ---------------------------------------
def test_backup_state_prefers_the_newest_confirmed_revision(db):
"""A host with several revisions shows the most recently CONFIRMED one, and
a revision that has never been confirmed (lastseenat NULL) never wins.
def test_backup_state_reports_the_stalest_measurable_kind(db):
"""A host is judged by the WORST backup it has, not the best.
This used to prefer the most recently confirmed revision. That was right
while a PC reported one kind, and became wrong when part markers arrived: a
marker bay reports ntlars AND partmarker, so a daily ntlars post kept the
badge green while the marker backup had been failing for a month. A host is
healthy only when everything it backs up is current, and the badge names the
kind that is not.
A revision that has never been confirmed (lastseenat NULL) still never wins.
It is not stale, it is unmeasurable - and letting it win hid a working
backup entirely, because the verdict for a NULL confirmation is tri-state
None, which renders as no badge at all.
Ordering happens in Python on purpose. Doing it in SQL needs
ORDER BY ... NULLS LAST, which SQLite accepts and MySQL rejects outright -
@@ -191,7 +202,10 @@ def test_backup_state_prefers_the_newest_confirmed_revision(db):
facts = {'wjpc01': {'backupkind': None, 'backuplastseen': None}}
_attach_backup_state(facts, {'WJPC01'})
assert facts['wjpc01']['backupkind'] == 'udc'
# ntlars is 9 days old, udc was confirmed just now: the host is judged by
# ntlars, and the badge names it so somebody knows WHICH backup stopped.
assert facts['wjpc01']['backupkind'] == 'ntlars'
# 'files' has never been confirmed, so it neither wins nor blanks the badge.
assert facts['wjpc01']['backuplastseen'] is not None

77
tests/test_plugins/test_geenforce_reporting.py
View File

@@ -486,3 +486,80 @@ def test_the_capture_date_is_reported_separately_from_the_check(client, db, app,
assert row['backupcollectedat'] is not None
assert row['backupcollectedat'] < row['backuplastseen'], \
'captured before it was last confirmed'
def _extra_revision(db, hostname, kind, ageda, contenthash=None):
"""A second backup kind for a host that already exists.
_backup_revision creates the asset and the Computer row, so it cannot be
called twice for one host.
"""
from datetime import timedelta
from plugins.backups.models import BackupRevision
from plugins.geenforce.service import _utcnow
from plugins.computers.models import Computer
computer = Computer.query.filter_by(hostname=hostname).first()
db.session.add(BackupRevision(
assetid=computer.assetid, backupkind=kind, sourcehostname=hostname,
contenthash=(contenthash or (kind[0] * 64))[:64],
collectedat=_utcnow() - timedelta(days=ageda + 30),
lastseenat=_utcnow() - timedelta(days=ageda)))
db.session.commit()
def test_a_stale_kind_is_not_hidden_by_a_healthy_one(client, db, app, auth_headers):
"""A marker bay reports TWO kinds, and the badge must not report the better.
ntlars posts daily for the machine's DNC settings; partmarker posts the
controller's patterns. Judging a host by its newest revision meant a
marker backup that had been failing for a week still showed green, because
ntlars kept the badge fed. The failure was then visible only on the
marker's own asset page, which is not where anyone looks to discover that
something stopped.
"""
_seed_and_publish(app)
secret = _token(client, auth_headers, ['geenforce.report'])
_backup_revision(db, 'WJBAK10', 'ntlars', ageda=1)
_extra_revision(db, 'WJBAK10', 'partmarker', ageda=9)
client.post('/api/geenforce/report', json={
'hostname': 'WJBAK10', 'scopename': 'gea-shopfloor-cmm', 'counts': {}},
headers={'X-API-Key': secret})
row = _row_for(client, auth_headers, 'WJBAK10')
assert row['backupok'] is False
# And it names the kind that is actually broken, not the healthy one.
assert row['backupkind'] == 'partmarker'
def test_a_host_is_good_only_when_every_kind_is_current(client, db, app, auth_headers):
_seed_and_publish(app)
secret = _token(client, auth_headers, ['geenforce.report'])
_backup_revision(db, 'WJBAK11', 'ntlars', ageda=1)
_extra_revision(db, 'WJBAK11', 'partmarker', ageda=1)
client.post('/api/geenforce/report', json={
'hostname': 'WJBAK11', 'scopename': 'gea-shopfloor-cmm', 'counts': {}},
headers={'X-API-Key': secret})
row = _row_for(client, auth_headers, 'WJBAK11')
assert row['backupok'] is True
def test_an_older_revision_of_a_kind_does_not_make_it_look_stale(client, db, app, auth_headers):
"""lastseenat only ever moves on the NEWEST revision of a chain.
Older revisions keep the value they had when they were superseded, so
judging on every row rather than the freshest per kind would report a
healthy, frequently changing config as permanently stale.
"""
_seed_and_publish(app)
secret = _token(client, auth_headers, ['geenforce.report'])
_backup_revision(db, 'WJBAK12', 'ntlars', ageda=1)
_extra_revision(db, 'WJBAK12', 'ntlars', ageda=40, contenthash='f' * 64)
client.post('/api/geenforce/report', json={
'hostname': 'WJBAK12', 'scopename': 'gea-shopfloor-cmm', 'counts': {}},
headers={'X-API-Key': secret})
row = _row_for(client, auth_headers, 'WJBAK12')
assert row['backupok'] is True
assert row['backupkind'] == 'ntlars'