ge-aerospace/shopdb-flask
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Import the wilm agent store, WINC adoption work and batches CLI, with four fixes

Browse Source
This commit is contained in:
Cameron Proudlock
2026-09-03 08:16:38 -04:00
parent 4b21aa3887
commit b0bd73f62d
51 changed files with 5107 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
scripts/site_imports/winc/README.md Normal file
View File

@@ -0,0 +1,67 @@
# WINC machine-adoption import (CLI)
Adopts machines from a Maximo asset export (validated against the WINC / West
Manufacturing site). It only updates machines that already exist; it never
creates canonical per-table rows. Every commit is journaled so it can be undone
as one batch.
Run all commands from the repo root. Each reads `.env` and connects to the
configured database.
## 1. Preview (read-only)
```
python -m scripts.site_imports.winc.dry_run --export EXPORT.csv
```
| switch | what it does |
| --- | --- |
| `--export PATH` | the Maximo ASSET export CSV to analyze (required) |
| `--rows` | print the per-row plan, not just the summary |
| `--csv PATH` | write the full plan to a CSV so runs can be diffed |
Nothing is written. Use this to see what would change and which values are
unresolved before committing.
## 2. Commit (writes, reversibly)
```
python -m scripts.site_imports.winc.writer --export EXPORT.csv --commit --acknowledge businessunit
```
| switch | what it does |
| --- | --- |
| `--export PATH` | the same export CSV (required) |
| `--commit` | actually write; without it this is a dry run |
| `--acknowledge GROUP` | approve a needs-create group (repeatable), e.g. `businessunit` |
| `--createdby NAME` | who approved the import; stamped on the batch |
On success it prints the new `batchid` and the exact undo command.
## 3. List batches
```
python -m scripts.site_imports.winc.batches
```
| switch | what it does |
| --- | --- |
| `--limit N` | show at most N batches, newest first (default 50) |
| `--json` | emit JSON instead of a table |
Read-only. Use it to find the `batchid` to undo.
## 4. Undo a batch
```
python -m scripts.site_imports.winc.undo --batch BATCHID --commit
```
| switch | what it does |
| --- | --- |
| `--batch BATCHID` | the batch to reverse (required) |
| `--commit` | actually reverse; without it this is a no-op |
Reversal is conservative: a cell a human changed since the import is left as-is
and reported, and a created business unit is deleted only when nothing outside
the batch still references it.

0
scripts/site_imports/winc/__init__.py Normal file
View File

67
scripts/site_imports/winc/aliases.py Normal file
View File

@@ -0,0 +1,67 @@
"""Alias and normalization tables for the WINC machine adoption dry-run.
This file is DATA, not logic. It is the single place a human edits when the
analyzer reports an unresolved value. Keep it boring: plain dicts, one entry per
line, lowercase keys. The analyzer lowercases and strips before it looks a value
up here, so keys here must be lowercase and already stripped.
Two independent problems get solved here:
1. eq1 -> business unit. The May export's eq1 column is a free-text value-stream
label (LG PARTS, SM SPOOLS, Large Spoo, HPTCR ...). It is dirty: truncated at
ten characters, inconsistently cased, and it does NOT match anything in the
live businessunits table (that table is a list of offices auto-created by an
earlier printer import). So every eq1 value must be mapped by hand to the
canonical business-unit name we want on the asset. A canonical name that does
not yet exist in the database is reported by the analyzer as "would need to
be created" - the analyzer never invents one and never writes one.
2. location code overrides. Most WM-<bay> codes resolve by simply dropping the
WM- prefix and matching a location by name (WM-J3 -> J3). A handful do not,
because the bay code carries a slash, a note in parentheses, or a room name
the location table spells differently. Those exceptions - and only those -
live here, mapping the RAW export location string to the exact locationname
already in the database.
"""
# eq1 (lowercased, stripped) -> canonical business unit name.
# The value on the right is what we intend to store; the analyzer checks whether
# it already exists in businessunits and flags the ones that do not.
EQ1_TO_BUSINESSUNIT = {
'lg parts': 'Large Parts',
'lg part': 'Large Parts',
'large part': 'Large Parts',
'sm spools': 'Small Spools',
'sm spool': 'Small Spools',
'large spoo': 'Large Spools',
'lg spools': 'Large Spools',
'lg spool': 'Large Spools',
'hptcr': 'HPT Rotor',
'bliskcr': 'Blisk Rotor',
'blisk': 'Blisk Rotor',
'leap x': 'LEAP-X',
}
# Raw export location string (lowercased, stripped) -> exact locationname in DB.
# Only for codes that do NOT resolve by dropping the WM- prefix. Leave a value
# as None to say "known to be unmappable, do not touch location for this row".
#
# The entries below the two None rows were promoted from the dry-run's
# high-confidence suggestions and verified to exist in the locations table. A
# slash-joined bay code (T6/U6) names two adjacent bays, but an asset carries a
# single locationid, so each maps to the FIRST (primary) bay - the deterministic
# choice a human confirmed. LD7W and LD6W are intentionally absent: they had no
# existing match and need a location created before they can be aliased.
LOCATION_OVERRIDES = {
'wm': None,
'wm-toolroom': None,
'wm-t6/u6': 'T6',
'wm-t9/10': 'T9',
'wm-u9/10': 'U9',
'wm-t4/u4': 'T4',
'wm-ld9w(ge90)': 'LD9W',
'wm-m6.3': 'M6',
'wm-l1-4': 'L1',
'wm-ld7w': None,
'wm-ld6w': None,
}

91
scripts/site_imports/winc/batches.py Normal file
View File

@@ -0,0 +1,91 @@
"""WINC machine adoption - BATCHES (list applied/undone import batches).
A read-only companion to writer.py and undo.py: it prints the WINC batches
newest-first so an operator can find a batchid to feed `--batch` into the undo
CLI. Like the other WINC commands it is WINC-named only for discoverability; the
underlying importbatches table is import-agnostic, so this simply filters to the
winc-machine-adoption kind.
"""
import argparse
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__))))))
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path(__file__).resolve().parents[3] / '.env')
from sqlalchemy import text
from scripts.site_imports.winc.dry_run import build_app_connection
_KIND = 'winc-machine-adoption'
def _iso(value):
# PyMySQL hands back datetime objects; SQLite hands back the stored ISO
# string. Normalise both to an ISO string so the output is stable.
if value is None:
return '-'
if hasattr(value, 'isoformat'):
return value.isoformat()
return str(value)
def list_batches(connection, limit):
rows = connection.execute(text(
'SELECT batchid, kind, rowcount, createdat, createdby, status, '
'undoneat FROM importbatches WHERE kind = :kind '
'ORDER BY createdat DESC LIMIT :limit'),
{'kind': _KIND, 'limit': limit}).fetchall()
return [
{'batchid': r[0], 'kind': r[1], 'rowcount': r[2],
'createdat': _iso(r[3]), 'createdby': r[4], 'status': r[5],
'undoneat': _iso(r[6])}
for r in rows
]
def _print_table(batches):
if not batches:
print('No WINC import batches found.')
return
header = '%-36s %-8s %5s %-20s %-16s %s' % (
'batchid', 'status', 'rows', 'createdat', 'createdby', 'undoneat')
print(header)
print('-' * len(header))
for batch in batches:
print('%-36s %-8s %5s %-20s %-16s %s' % (
batch['batchid'], batch['status'], batch['rowcount'],
batch['createdat'], batch['createdby'] or '-', batch['undoneat']))
def main(argv=None):
parser = argparse.ArgumentParser(
description='List applied/undone WINC import batches, newest first.')
parser.add_argument('--limit', type=int, default=50,
help='Maximum number of batches to show (default 50).')
parser.add_argument('--json', action='store_true',
help='Emit the batches as JSON instead of a table.')
args = parser.parse_args(argv)
if args.limit < 1:
parser.error('--limit must be a positive integer')
_app, connection = build_app_connection()
batches = list_batches(connection, args.limit)
if args.json:
import json
print(json.dumps(batches, indent=2))
else:
_print_table(batches)
return 0
if __name__ == '__main__':
sys.exit(main())

385
scripts/site_imports/winc/dry_run.py Normal file
View File

@@ -0,0 +1,385 @@
"""Dry-run analyzer for the WINC (West Manufacturing) machine adoption.
READ ONLY. This script opens the database, reads the May asset export, decides
what an import WOULD change, and prints a report. It never writes. Nothing here
commits, updates, or inserts - that is a separate tool, written only once this
report reads clean and a human has signed off on it.
What it does, in order:
1. Load every existing machine's asset number and build the crosswalk key
(the workstation number, e.g. 0083A) the export will be matched on.
2. Read the export with a real CSV parser (quoted commas, BOM) and, for each
row that carries a workstation number, decide:
- does it match an existing machine? (in scope vs skipped)
- what name would we set? (export description)
- what location would we set? (WM-<bay> -> locations, via
aliases.LOCATION_OVERRIDES for
the exceptions)
- what business unit would we set? (eq1 -> aliases.EQ1_TO_
BUSINESSUNIT -> businessunits)
3. Classify each field resolution as RESOLVED, BLANK (nothing to set, leave
as-is), NEEDS-CREATE (target value is valid but not yet in the database),
or UNRESOLVED (no alias entry - a human must add one to aliases.py).
4. Print a summary and, with --rows, a per-row table. With --csv PATH it also
writes the full plan so the numbers can be diffed between runs.
Run it against the real database read-only:
python -m scripts.site_imports.winc.dry_run --export "exports/ASSET_current May26.csv"
python -m scripts.site_imports.winc.dry_run --export EXPORT.csv --rows
python -m scripts.site_imports.winc.dry_run --export EXPORT.csv --csv plan.csv
The design goal is reliability over cleverness: one match rule, one location
rule, one business-unit rule, every exception in aliases.py, and a report that
tells a human exactly what is unresolved so the alias table can be completed
before anyone runs the writer.
"""
import argparse
import csv
import os
import re
import sys
from collections import OrderedDict
from pathlib import Path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__))))))
from dotenv import load_dotenv
# Same reasoning as scripts/export_catalog.py: a plain `python -m` run gets
# neither wsgi.py's nor the flask CLI's .env load, so DATABASE_URL would be
# unset and config would silently fall back to a hardcoded localhost URL. Load
# it here before anything reads config.
load_dotenv(Path(__file__).resolve().parents[3] / '.env')
from scripts.site_imports.approval import build_packet, format_packet
from scripts.site_imports.suggest import suggest
from scripts.site_imports.resolver import (
BLANK,
NEEDS_CREATE,
RESOLVED,
Resolution,
UNRESOLVED,
load_lookup,
norm as _norm,
resolve_alias,
resolve_name,
)
from scripts.site_imports.winc.aliases import (
EQ1_TO_BUSINESSUNIT,
LOCATION_OVERRIDES,
)
# WM- is the only bay prefix in the WINC export; a location code resolves by
# dropping it and matching a location by name. Everything else is an override.
_WM_PREFIX = re.compile(r'^WM-?', re.IGNORECASE)
# A workstation number is a LEADING run of digits with an optional single letter
# suffix. The export writes it clean (W0083A); the machines table stores the
# same number but often with a trailing descriptor (0083 Debur, 0083A Inspect).
# So the key is the leading token on both sides, matched unanchored - anchoring
# with $ would drop every existing machine whose name carries a descriptor and
# silently shrink the in-scope set. The leading W is the single biggest gotcha
# in this data, so it is stripped here in exactly one place.
_WSNUM = re.compile(r'^(\d{3,5}[A-Z]?)')
def workstation_key(assetnum):
"""The crosswalk key (e.g. '0083A') for an asset number, or None.
Strips a leading W and keeps the leading run of digits plus an optional
single trailing letter, ignoring any descriptor that follows. Returns None
for anything that is not a workstation number (blank rows, facilities codes,
named test machines) so non-machine rows fall out of scope on their own
rather than being force-fit.
"""
text = (assetnum or '').strip().upper()
if text.startswith('W'):
text = text[1:]
match = _WSNUM.match(text)
return match.group(1) if match else None
def load_existing_machine_keys(connection):
"""Map workstation key -> list of existing machine asset numbers.
A list, not a single value, because the machines table has a few numbers
that collide once reduced to their workstation key (0083 exists as
'0083 Debur' and '0083 Inspection'). The analyzer reports those as ambiguous
rather than guessing which one the export row means.
"""
from sqlalchemy import text
keys = OrderedDict()
rows = connection.execute(text(
"SELECT a.assetnumber "
"FROM assets a JOIN machines m ON m.assetid = a.assetid"))
for (assetnumber,) in rows:
key = workstation_key(assetnumber)
if key:
keys.setdefault(key, []).append(assetnumber)
return keys
def load_locations(connection, display=None):
"""locationname (normalized) -> locationid, from the live locations table."""
return load_lookup(connection, 'locations', 'locationid', 'locationname',
display=display)
def load_businessunits(connection):
"""businessunit (normalized) -> businessunitid, from the live table."""
return load_lookup(connection, 'businessunits', 'businessunitid',
'businessunit')
def resolve_location(raw_location, locations_by_name):
"""Resolution for a raw WM bay code.
Rule: an override in aliases wins; otherwise drop the WM- prefix and match a
location by name. A blank export cell means "leave the asset's location as
is" and is reported as BLANK, not UNRESOLVED, because there is nothing to
set and nothing for a human to fix. Overrides are checked first because they
carry the exceptions the plain prefix rule cannot handle.
"""
raw = (raw_location or '').strip()
if not raw:
return Resolution(BLANK, None, None)
if _norm(raw) in LOCATION_OVERRIDES:
return resolve_alias(raw, LOCATION_OVERRIDES, locations_by_name)
name = _WM_PREFIX.sub('', raw).strip()
return resolve_name(name, locations_by_name)
def resolve_businessunit(raw_eq1, businessunits_by_name):
"""Resolution for an eq1 value stream.
A blank eq1 is BLANK (leave the asset alone). A non-blank value must have an
alias entry; without one it is UNRESOLVED so a human adds it to aliases.py. A
mapped canonical name that does not yet exist in businessunits is
NEEDS-CREATE - flagged, never created here.
"""
return resolve_alias(raw_eq1, EQ1_TO_BUSINESSUNIT, businessunits_by_name)
def read_export(path):
"""Rows of the export as dicts. utf-8-sig handles the BOM; the csv module
handles the quoted commas embedded in some descriptions."""
with open(path, newline='', encoding='utf-8-sig') as handle:
return list(csv.DictReader(handle))
def parse_export_text(text):
"""Rows of an export given its raw CSV text (same shape as read_export).
Split out so an API route can hand the analyzer an uploaded file's bytes
without writing them to disk, while the CLI keeps reading from a path.
"""
import io
return list(csv.DictReader(io.StringIO(text)))
def analyze(connection, export_path):
"""Build the per-row plan from a file path. Returns (plan, summary, lookups)."""
return analyze_rows(connection, read_export(export_path))
def analyze_rows(connection, export_rows):
"""Build the per-row plan. Returns (plan_rows, summary_dict, lookups).
Takes already-parsed export rows so the same analysis serves both the CLI
(which reads a path) and the API route (which reads uploaded bytes).
"""
existing = load_existing_machine_keys(connection)
location_names = {}
locations = load_locations(connection, display=location_names)
businessunits = load_businessunits(connection)
plan = []
summary = OrderedDict([
('export_rows', len(export_rows)),
('non_workstation_rows', 0),
('in_scope', 0),
('skipped_no_existing_machine', 0),
('ambiguous_match', 0),
('location_' + RESOLVED, 0),
('location_' + BLANK, 0),
('location_' + NEEDS_CREATE, 0),
('location_' + UNRESOLVED, 0),
('businessunit_' + RESOLVED, 0),
('businessunit_' + BLANK, 0),
('businessunit_' + NEEDS_CREATE, 0),
('businessunit_' + UNRESOLVED, 0),
])
for row in export_rows:
assetnum = (row.get('assetnum') or '').strip()
key = workstation_key(assetnum)
if key is None:
summary['non_workstation_rows'] += 1
continue
matches = existing.get(key, [])
if not matches:
summary['skipped_no_existing_machine'] += 1
continue
summary['in_scope'] += 1
ambiguous = len(matches) > 1
if ambiguous:
summary['ambiguous_match'] += 1
loc = resolve_location(row.get('location'), locations)
bu = resolve_businessunit(row.get('eq1'), businessunits)
summary['location_' + loc.status] += 1
summary['businessunit_' + bu.status] += 1
plan.append(OrderedDict([
('assetnum', assetnum),
('workstation_key', key),
('matched_asset', '|'.join(matches)),
('ambiguous', 'yes' if ambiguous else ''),
('name', (row.get('description') or '').strip()),
('raw_location', (row.get('location') or '').strip()),
('location_status', loc.status),
('locationid', loc.id if loc.id is not None else ''),
('location_name', loc.name or ''),
('raw_eq1', (row.get('eq1') or '').strip()),
('businessunit_status', bu.status),
('businessunitid', bu.id if bu.id is not None else ''),
('businessunit_name', bu.name or ''),
]))
return plan, summary, {'locations': locations,
'location_names': location_names,
'businessunits': businessunits}
def print_summary(summary):
print('=' * 60)
print('WINC machine adoption - DRY RUN (no changes written)')
print('=' * 60)
for label, value in summary.items():
print(' %-32s %6d' % (label.replace('_', ' '), value))
def print_unresolved(plan, lookups=None):
loc = OrderedDict()
bu = OrderedDict()
for row in plan:
if row['location_status'] == UNRESOLVED:
loc[row['raw_location']] = loc.get(row['raw_location'], 0) + 1
if row['businessunit_status'] == UNRESOLVED:
bu[row['raw_eq1']] = bu.get(row['raw_eq1'], 0) + 1
needs_loc = OrderedDict()
needs_bu = OrderedDict()
for row in plan:
if row['location_status'] == NEEDS_CREATE:
needs_loc[row['location_name']] = needs_loc.get(row['location_name'], 0) + 1
if row['businessunit_status'] == NEEDS_CREATE:
needs_bu[row['businessunit_name']] = needs_bu.get(row['businessunit_name'], 0) + 1
if loc:
print('\nUNRESOLVED locations (add to aliases.LOCATION_OVERRIDES):')
location_lookup = (lookups or {}).get('locations') or {}
location_names = (lookups or {}).get('location_names') or {}
for value, count in sorted(loc.items(), key=lambda kv: -kv[1]):
print(' %5d %s' % (count, value))
stripped = _WM_PREFIX.sub('', value).strip()
for hint in suggest(value, stripped, location_lookup,
display=location_names):
print(' -> suggest "%s" (%s, confidence %s)'
% (hint.candidate, hint.reason, hint.confidence))
if bu:
print('\nUNRESOLVED eq1 (add to aliases.EQ1_TO_BUSINESSUNIT):')
for value, count in sorted(bu.items(), key=lambda kv: -kv[1]):
print(' %5d %s' % (count, value))
if needs_loc:
print('\nNEEDS-CREATE locations (mapped, but not yet in locations):')
for value, count in sorted(needs_loc.items(), key=lambda kv: -kv[1]):
print(' %5d %s' % (count, value))
if needs_bu:
print('\nNEEDS-CREATE business units (mapped, but not yet in businessunits):')
for value, count in sorted(needs_bu.items(), key=lambda kv: -kv[1]):
print(' %5d %s' % (count, value))
def print_rows(plan):
print('\nPER-ROW PLAN:')
header = ('assetnum', 'matched_asset', 'name', 'raw_location',
'location_status', 'raw_eq1', 'businessunit_status')
print(' ' + ' | '.join(header))
for row in plan:
print(' ' + ' | '.join(str(row[h]) for h in header))
def write_csv(plan, path):
if not plan:
return
with open(path, 'w', newline='', encoding='utf-8') as handle:
writer = csv.DictWriter(handle, fieldnames=list(plan[0].keys()))
writer.writeheader()
writer.writerows(plan)
def build_app_connection():
"""Yield a live SQLAlchemy connection inside an app context.
Returns (app, connection). Caller is responsible for closing nothing - the
connection lives for the app context and the process exits right after.
"""
from shopdb import create_app
from shopdb.api import db
app = create_app()
context = app.app_context()
context.push()
return app, db.session.connection()
def main(argv=None):
parser = argparse.ArgumentParser(
description='Dry-run analyzer for WINC machine adoption. Read only.')
parser.add_argument('--export', required=True,
help='Path to the ASSET export CSV.')
parser.add_argument('--rows', action='store_true',
help='Print the full per-row plan.')
parser.add_argument('--csv', dest='csv_out',
help='Write the full per-row plan to this CSV path.')
parser.add_argument('--packet', action='store_true',
help='Print the approval packet (hash, needs-create '
'and blocking summary) a writer would enforce.')
args = parser.parse_args(argv)
if not os.path.exists(args.export):
parser.error('export file not found: %s' % args.export)
_app, connection = build_app_connection()
plan, summary, lookups = analyze(connection, args.export)
print_summary(summary)
print_unresolved(plan, lookups)
if args.rows:
print_rows(plan)
if args.csv_out:
write_csv(plan, args.csv_out)
print('\nWrote per-row plan: %s (%d rows)' % (args.csv_out, len(plan)))
if args.packet:
packet = build_packet(
'winc-machine-adoption', plan, summary,
[('businessunit', 'businessunit_status', 'raw_eq1',
'businessunit_name'),
('location', 'location_status', 'raw_location', 'location_name')])
print('\n' + format_packet(packet))
return 0
if __name__ == '__main__':
sys.exit(main())

55
scripts/site_imports/winc/undo.py Normal file
View File

@@ -0,0 +1,55 @@
"""WINC machine adoption - UNDO (reverse an applied import batch).
A thin CLI over the shared, import-agnostic journal.undo_batch. It is WINC-named
only for discoverability; the actual work is generic, so the same reversal serves
any importer that wrote through the journal. Reverting is conservative: a cell a
human changed since the import is left as-is and reported, and a created business
unit is deleted only when nothing outside the batch still references it.
"""
import argparse
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__))))))
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path(__file__).resolve().parents[3] / '.env')
from scripts.site_imports import journal
from scripts.site_imports.winc.dry_run import build_app_connection
from scripts.site_imports.winc.writer import _UNDO_REFERENCES
def main(argv=None):
parser = argparse.ArgumentParser(
description='Undo an applied WINC import batch (reversible journal).')
parser.add_argument('--batch', required=True,
help='The batch id to reverse.')
parser.add_argument('--commit', action='store_true',
help='Actually reverse. Without it, this is a no-op.')
args = parser.parse_args(argv)
if not args.commit:
print('DRY: pass --commit to reverse batch %s' % args.batch)
return 0
_app, connection = build_app_connection()
from shopdb.api import db
try:
result = journal.undo_batch(connection, args.batch,
references=_UNDO_REFERENCES)
db.session.commit()
except Exception:
db.session.rollback()
raise
print(journal.format_undo_result(result))
return 0
if __name__ == '__main__':
sys.exit(main())

278
scripts/site_imports/winc/writer.py Normal file
View File

@@ -0,0 +1,278 @@
"""WINC machine adoption - WRITER (applies the approved plan, reversibly).
This is the write half of the WINC import; scripts/site_imports/winc/dry_run.py
is the read-only half. The writer never invents its own view of the data: it
rebuilds the exact same plan the dry-run produced, verifies that plan against the
approval packet the human signed off on, and only then writes - recording every
change to the shared import journal so the whole batch can be undone.
The contract, in order (the one documented in approval.py, made concrete here):
1. rebuild the plan from the same export + live DB,
2. build a packet from it,
3. compare planhash to the approved packet - abort on mismatch (drift),
4. refuse if the packet is blocking (any unresolved value),
5. require explicit acknowledgment for every needs-create group,
6. write inside ONE transaction, stamping the batch id onto the journal:
a. create each acknowledged needs-create business unit (journal insert),
b. update each in-scope asset's name / location / business unit
(journal update, capturing the pre-image first).
Everything commits together or not at all. Ambiguous rows (one workstation key
matching several machines) are never written - the dry-run flags them and the
writer skips them, because guessing which machine an export row means is exactly
the kind of cleverness this import refuses.
"""
import argparse
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(
os.path.dirname(os.path.abspath(__file__))))))
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path(__file__).resolve().parents[3] / '.env')
from scripts.site_imports import journal
from scripts.site_imports.approval import build_packet, packet_matches_plan
from scripts.site_imports.resolver import NEEDS_CREATE, RESOLVED, norm
from scripts.site_imports.winc.dry_run import (
analyze,
build_app_connection,
)
# The packet groups the WINC dry-run declares, reused verbatim so the writer
# hashes and reads the plan exactly the way approval did.
_PACKET_GROUPS = [
('businessunit', 'businessunit_status', 'raw_eq1', 'businessunit_name'),
('location', 'location_status', 'raw_location', 'location_name'),
]
# References into a created row's id, so undo can tell whether an inserted
# business unit is still pointed at by something outside this batch before it
# hard-deletes it. Shared with the undo CLI and the web undo path.
_UNDO_REFERENCES = {
'businessunits': [('assets', 'businessunitid')],
}
# References into a created row's id, so undo can tell whether an inserted
# business unit is still pointed at by something outside this batch before it
# hard-deletes it.
# Max width of assets.name; the export description can exceed it (a few rows
# carry a whole safety notice in the field), so a set name is clamped rather
# than letting the whole transaction fail on one over-long row. The pre-image is
# journalled, so the clamp is reversible like any other change.
_ASSET_NAME_MAXLEN = 100
class WriterError(Exception):
"""The plan cannot be written as approved (drift, blocking, or missing
acknowledgment). Carries a human-readable reason; never a partial write."""
def _load_asset_ids(connection):
"""assetnumber -> assetid for every asset, so a plan row (which carries the
matched asset *number*) can be turned into the id the update targets."""
from sqlalchemy import text
out = {}
for assetid, assetnumber in connection.execute(text(
'SELECT assetid, assetnumber FROM assets')):
out[assetnumber] = assetid
return out
def _create_businessunit(connection, batchid, sequence, name):
"""Insert one business unit and journal it. Returns (new_id, next_sequence).
Only ever called for an acknowledged needs-create name, so the journal's
insert entry describes strictly a row this batch created."""
from sqlalchemy import text
now = journal._utcnow()
result = connection.execute(text(
'INSERT INTO businessunits '
'(businessunit, createddate, modifieddate, isactive) '
'VALUES (:name, :now, :now, 1)'),
{'name': name, 'now': now})
new_id = result.lastrowid
journal.record_insert(connection, batchid, sequence, 'businessunits',
new_id)
return new_id, sequence + 1
def _update_cell(connection, batchid, sequence, table, targetid, column,
oldvalue, newvalue):
"""Apply one cell update and journal its pre/post image. Returns the next
sequence. Skips (and does not journal) a no-op where the value is unchanged."""
from sqlalchemy import text
if journal._as_text(oldvalue) == journal._as_text(newvalue):
return sequence
connection.execute(text(
'UPDATE %s SET %s = :newvalue WHERE %s = :targetid'
% (table, column, _asset_pk(table))),
{'newvalue': newvalue, 'targetid': targetid})
journal.record_update(connection, batchid, sequence, table, targetid,
column, oldvalue, newvalue)
return sequence + 1
def _asset_pk(table):
# The writer only updates the assets table; keep the mapping explicit rather
# than reflecting, so an unexpected table name fails loudly.
return {'assets': 'assetid'}[table]
def apply_plan(connection, plan, approved_packet, acknowledge=None,
createdby=None):
"""Apply an approved WINC plan, reversibly. Returns the batch id.
connection a live SQLAlchemy connection in an open transaction.
plan the per-row plan from dry_run.analyze (rebuilt fresh).
approved_packet the packet a human signed off on.
acknowledge set of needs-create group names the human confirmed
(e.g. {'businessunit'}); required for every group the
packet lists under needscreate.
createdby who acknowledged, stamped on the batch.
Raises WriterError before touching anything if the plan drifted, is
blocking, or a needs-create group was not acknowledged.
"""
acknowledge = set(acknowledge or ())
# (2)-(3) rebuild the packet and prove nothing drifted since approval.
current_packet = build_packet('winc-machine-adoption', plan,
approved_packet.get('counts', {}),
_PACKET_GROUPS)
if not packet_matches_plan(approved_packet, plan):
raise WriterError(
'plan has changed since approval (hash mismatch) - re-run the '
'dry-run and re-approve before importing')
# (4) never write over an unresolved value.
if current_packet.get('blocking'):
raise WriterError(
'plan is blocking: unresolved values remain - add the missing '
'aliases and re-run the dry-run')
# (5) every needs-create group must be explicitly acknowledged.
missing_ack = set(current_packet.get('needscreate', {})) - acknowledge
if missing_ack:
raise WriterError(
'needs-create not acknowledged for: %s - confirm these will be '
'created before importing' % ', '.join(sorted(missing_ack)))
batchid = approved_packet['batchid']
journal.open_batch(connection, approved_packet, createdby=createdby)
sequence = 1
# (6a) create acknowledged needs-create business units once each, first, so
# the asset updates below can point at them.
created_bu = {}
for row in plan:
if row['businessunit_status'] != NEEDS_CREATE:
continue
name = row['businessunit_name']
key = norm(name)
if key not in created_bu:
created_bu[key], sequence = _create_businessunit(
connection, batchid, sequence, name)
# (6b) update each in-scope, unambiguous asset.
asset_ids = _load_asset_ids(connection)
from sqlalchemy import text
for row in plan:
if row['ambiguous']:
continue
assetnumber = row['matched_asset']
assetid = asset_ids.get(assetnumber)
if assetid is None:
continue
current = connection.execute(text(
'SELECT name, locationid, businessunitid FROM assets '
'WHERE assetid = :assetid'), {'assetid': assetid}).fetchone()
cur_name, cur_location, cur_bu = current
if row['name']:
new_name = row['name'][:_ASSET_NAME_MAXLEN]
sequence = _update_cell(connection, batchid, sequence, 'assets',
assetid, 'name', cur_name, new_name)
if row['location_status'] == RESOLVED and row['locationid'] != '':
sequence = _update_cell(connection, batchid, sequence, 'assets',
assetid, 'locationid', cur_location,
row['locationid'])
new_bu = None
if row['businessunit_status'] == RESOLVED and row['businessunitid'] != '':
new_bu = row['businessunitid']
elif row['businessunit_status'] == NEEDS_CREATE:
new_bu = created_bu[norm(row['businessunit_name'])]
if new_bu is not None:
sequence = _update_cell(connection, batchid, sequence, 'assets',
assetid, 'businessunitid', cur_bu, new_bu)
return batchid
def main(argv=None):
parser = argparse.ArgumentParser(
description='WINC machine adoption writer. Applies an approved plan '
'reversibly. Requires --commit to write.')
parser.add_argument('--export', required=True,
help='Path to the ASSET export CSV.')
parser.add_argument('--acknowledge', action='append', default=[],
metavar='GROUP',
help='Acknowledge a needs-create group (repeatable), '
'e.g. --acknowledge businessunit.')
parser.add_argument('--createdby', default=None,
help='Who is approving this import (stamped on batch).')
parser.add_argument('--commit', action='store_true',
help='Actually write. Without it, this is a no-op that '
'reports what would happen.')
args = parser.parse_args(argv)
if not os.path.exists(args.export):
parser.error('export file not found: %s' % args.export)
_app, connection = build_app_connection()
plan, summary, _lookups = analyze(connection, args.export)
packet = build_packet('winc-machine-adoption', plan, summary,
_PACKET_GROUPS)
if not args.commit:
print('DRY: rebuilt plan of %d rows, hash %s' % (
packet['rowcount'], packet['planhash']))
print('DRY: pass --commit with --acknowledge for each needs-create '
'group to write. Needs-create groups: %s'
% (', '.join(packet.get('needscreate', {})) or 'none'))
return 0
from shopdb.api import db
try:
batchid = apply_plan(connection, plan, packet,
acknowledge=args.acknowledge,
createdby=args.createdby)
db.session.commit()
except WriterError as error:
db.session.rollback()
print('REFUSED: %s' % error)
return 2
except Exception:
db.session.rollback()
raise
print('COMMITTED batch %s' % batchid)
print(' to undo: python -m scripts.site_imports.winc.undo --batch %s'
% batchid)
return 0
if __name__ == '__main__':
sys.exit(main())