shopdb-flask/scripts/site_imports/wjf/PILOT-DEPLOY.md

Production pilot runbook (West Jefferson)

Goal: stand up a real shopdb-flask instance loaded with WJ's classic-ASP data, run it in parallel with the classic app for a validation window, then cut over. This runbook adds the legacy-data import + verification + cutover on top of the generic stand-up in DEPLOY.md. Read that first; this only calls out the pilot-specific steps.

Related: IMPORT-ADOPTION.md (import model), IMPORT-API.md (the contract), BACKUP-RESTORE.md, scripts/site_imports/wjf/ (the loader).

---

0. Pre-flight checklist

• Host provisioned (Docker + compose, or a VM with Python 3 + MySQL 8).
• Three current classic dumps in hand: shopdb (main), cmmc_usb, wjf_employees. Take fresh dumps at import time - the classic app is live.
• Target MySQL 8, utf8mb4 (charset is contract, ADR-004). Old MySQL <5.7 needs innodb_large_prefix=ON + Barracuda.
• Decide the pilot URL (e.g. shopdb-pilot.example.net) - separate from the classic app; do not reuse its hostname yet.
• Confirm the import decisions still hold (see the loader README / the import plan): assetnumber fallback + skip-dups, metrology routing, cmmc-only USB, warranties = Dell, occurrences parked.

1. Stand up the pilot instance

Follow DEPLOY.md steps 1-6 against a NEW empty database (name it clearly, e.g. shopdb_flask_pilot):

flask db upgrade
flask plugin upgrade-all          # applies every plugin's chain
flask seed permissions
flask seed settings
flask seed reference-data          # seeds communicationtypes (IP) + the rest

Enable every bundled plugin the site tracks - including usb, which ships disabled. A plugin's routes only register when it is enabled at app start, and the importer needs them:

# apply-profile installs AND enables, in dependency order, and runs each
# plugin's own migration chain. `plugin enable` alone refuses a plugin that is
# not installed yet, so on a fresh database the loop below exited 1 on every
# iteration and enabled nothing (ADR-013 records the same defect).
flask plugin apply-profile deploy/site-profile-universal.json
flask plugin upgrade-all

Do not run the setup wizard yet - the import fills the data the wizard would otherwise ask you to seed.

2. Load the classic data

The loader (scripts/site_imports/wjf/) reads the classic dumps and drives the import API. It is site glue, not product code.

1. Load the three dumps into scratch source DBs the loader can read (strip the CREATE DATABASE/USE lines so they land under scratch names, no clobber):

   for pair in "shopdb_src:shopdb_dump.sql" "cmmc_usb_src:cmmc_usb_dump.sql" \
               "wjf_employees_src:wjf_employees_dump.sql"; do
     db="${pair%%:*}"; f="${pair##*:}"
     mysql -h HOST -u root -p -e "CREATE DATABASE $db CHARACTER SET utf8mb4;"
     sed -E '/^CREATE DATABASE/d; /^USE `/d' "$f" | mysql -h HOST -u root -p "$db"
   done
   

2. Point the loader at the PILOT database and run all stages:

   DATABASE_URL='mysql+pymysql://USER:PW@HOST:3306/shopdb_flask_pilot?charset=utf8mb4' \
     venv/bin/python -m scripts.site_imports.wjf.run
   

   The 17 stages run in order (reference -> employees -> catalog -> assets hub -> locations -> printers -> metrology -> communications -> applications -> warranties -> notifications -> knowledgebase -> relationships -> defaultprinters -> subnets -> usb -> verify).

   Pointing it at a different source. The three source databases are named logically in the loader and resolved from the environment, so a restore under another name needs no code edit:

   SOURCE_DB_MAIN=prodscratch \
   SOURCE_DB_USB=cmmc_usb_src \
   SOURCE_DB_EMPLOYEES=wjf_employees_src \
   DATABASE_URL=... venv/bin/python -m scripts.site_imports.wjf.run
   

   Also SOURCE_DB_HOST, SOURCE_DB_PORT, SOURCE_DB_USER, SOURCE_DB_PASSWORD. The run prints every source it redirected, so "it imported nothing" can be told apart from "it read the wrong database".

   Resume is per target database. A crashed run resumes from the crosswalk in idmap.json, which maps this site's legacy ids to assetids in one specific database. The loader stamps that database's identity into the file and refuses a crosswalk built against a different one, because reusing it attaches warranties and printer assignments to whatever asset now holds that id - wrong data, no error. If you rebuild the target, delete idmap.json or pass --idmap a fresh path.

   The exit code means something: non-zero if any endpoint errored, so a run that logged 200 failures cannot read as a success in a deploy log.

3. Reclassify servers into network devices. The classic DB stored servers as PCs, so the import lands them as computer assets. Re-point them in place:

   DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py            # dry run, prints matches
   DATABASE_URL=... venv/bin/python scripts/reclassify_servers_to_network.py --commit   # apply
   # match on an exact computer type instead of the SVR- name prefix:
   ... --type "Server" --commit
   

   The assetid does not change: communications, relationships, map position, and audit history carry over. Only the extension row is swapped (computers -> networkdevices) and the asset type flipped; reclassified devices get the Server networkdevicetype. Run the dry run, eyeball the list, then commit. Re-running is safe (already-moved assets no longer match).

Expected magnitude (from the WJ dumps used in development - your fresh dumps will differ slightly):

entity	count
assets [*]	~983 (computer ~663, machine ~76, network ~58, measuring-tool ~136, printer ~50)
locations	~24
employees	~415
installs	~850
primary IPs	~461
warranties	~464
notifications	~261
knowledge base	~341
relationships	~93
subnets	~37
USB devices / events	~18 / ~232

[*] Counts taken AFTER scripts/reclassify_servers_to_network.py --commit. Servers imported as computers are re-pointed to network devices, so the computer count drops and network rises by the same amount versus a raw import.

PLACEHOLDER - re-measure before publishing. The computer/network split shown in the assets row above still reflects a RAW import (pre-reclassify). Re-run the counts on the current prodscratch AFTER the reclassify step above and drop in the actual numbers; do not carry these development figures forward as if they already account for the reclassify.

The verify stage prints a source-vs-target row-count audit; the gaps are the documented skips (inactive rows, duplicate machinenumbers, LocationOnly, the 9999 placeholder).

3. Verify the import

• Read the verify stage output - source vs target counts line up modulo the documented skips.
• Create the admin: flask seed admin --username ... --email ... (password printed once). Mark setup done so the app does not force the wizard: set setup_complete=true in settings (or click through the wizard, skipping the seed steps).
• UI spot-check (log in): Computers list paginates the full fleet; the Shop Floor Map plots assets, color-coded by type (positions came from mapleft/maptop); open a PC detail (installs), a printer (IP + share), an application (installed-on list), a KB article; check the employee directory; check a couple of asset relationships.
• Branding: upload the site logo + floor-plan blueprint under Settings, set facility name (Settings drive these per CONFIG.md).
• Photos are deferred - employees show initials until a photo batch is run.

4. Parallel-run window

• Keep the classic app authoritative during the window. The pilot is read-mostly for validation; do not dual-write.
• Have a few real users (IT + a floor lead) work the pilot and log gaps.
• Re-import is cheap: fix a loader mapping, drop + rebuild the pilot DB, delete idmap.json, re-run. Nothing you do to the pilot touches classic. The crosswalk is bound to the database it was built against and the loader refuses a stale one, but deleting it is the habit to have - a refusal costs you a restart, and before the guard existed a stale file silently attached rows to the wrong assets.
• Point the collector (GE-Enforce fleet ingest) at the pilot in parallel to confirm live PC check-ins land (see COLLECTOR-INTEGRATION.md), using a scoped service token.

5. Cutover

When the window is clean:

1. Freeze classic writes (announce a short read-only window).
2. Take final fresh dumps; re-run the loader into a clean pilot DB so the cutover data is current. A clean DB means a fresh idmap.json too.
3. Verify counts + a fast UI spot-check.
4. Repoint the production hostname/DNS (or the reverse proxy) at the pilot.
5. Retire the classic app to read-only standby (do not delete - keep it as the rollback for the agreed period).

6. Rollback

• Pre-cutover: trivially point back at classic (it never stopped being authoritative).
• Post-cutover, within the standby window: repoint DNS/proxy back at classic; investigate; re-cut when fixed. Because the loader is deterministic and the classic DB is untouched, a re-run reproduces the flask DB exactly.

7. Post-cutover

• Backups on a schedule (BACKUP-RESTORE.md) - mysqldump + the instance/ dir (uploaded logos, floor plans, tokens).
• Run the employee-photo batch.
• GE-Enforce: publish manifests + cut the fleet over to the flask endpoints when ready (GE-ENFORCE-DEPLOY.md) - independent of this pilot.
• Schedule the deferred data (occurrences, full communications fidelity) only if a real need appears.