Backfill an asset's type from its model, by exact name only

Correcting an earlier judgement. I said the model's type could not be used to
fill an asset's type, because modeltypes is the catalog-wide list covering every
kind of asset - it holds "Access Point", "Camera" and "Desktop PC" - and only
about two thirds of its names exist as machine types.

That is true across the whole catalog and misleading in practice. Restricted to
the models an asset class actually uses, the picture is different: all 262
machines in the development database map exactly, because the non-machine
entries are never used by machines. The blanks on the machines list are rows
whose type the database could already have supplied.

So the backfill now fills the type as well, under a rule that cannot mistype
anything: exact name match or nothing. A model type with no identically named
entry in the asset's own type table is reported with a count and left untouched,
so somebody can decide rather than have a guess written into their data. The
same shape covers computers, printers and network devices, each against its own
type table.

Verified against the development database by nulling one machine's type inside a
transaction: it was detected as fillable, the proposal read "LocationOnly" ->
"LocationOnly", the update restored exactly the original id, and the rollback
left the row unchanged.

Still a dry run unless given --commit, and a table missing the model column is
skipped, so it runs against a server whose network migration is not yet applied.

scripts/backfill_vendor_from_model.py

@@ -10,11 +10,18 @@ ONLY fills rows where the asset's vendor is NULL and the model names one. It
 never overwrites a vendor somebody chose, and it never guesses: if the model has
 no vendor either, the row is left alone.
 
-DELIBERATELY NOT DONE HERE: the asset's TYPE. modeltypes is the catalog-wide
-list covering every kind of asset - it holds "Access Point", "Camera" and
-"Desktop PC" alongside the machine entries - so it is a different taxonomy from
-machinetypes, and mapping one onto the other by name would mistype whatever does
-not match. Only about two thirds of the names overlap.
+It also fills the asset's TYPE from the model's type, but ONLY where the two
+names are identical. modeltypes is the catalog-wide list covering every kind of
+asset - it holds "Access Point", "Camera" and "Desktop PC" alongside the machine
+entries - so it is a different taxonomy from machinetypes and the two cannot be
+equated in general. Across the whole catalog only about two thirds of the names
+overlap. Restricted to the models an asset class actually uses the picture is
+different: every one of the 262 machines in the development database maps
+exactly, because the non-machine entries are never used by machines.
+
+So the rule is exact name match or nothing. A model type with no same-named
+entry in the asset's own type table is REPORTED and left alone, never guessed
+at, because the failure mode is a machine labelled "Desktop PC".
 
 Dry run by default; nothing is written without --commit.
 
@@ -38,6 +45,16 @@ TARGETS = [
     ('networkdevices', 'networkdeviceid', 'hostname'),
 ]
 
+# The asset's own type column, and the table it points at. Each asset class
+# keeps its own type list; the catalog's modeltypes is a different taxonomy, so
+# these are joined by NAME and only when the name matches exactly.
+TYPE_TARGETS = {
+    'machines':       ('machinetypeid',       'machinetypes',       'machinetype'),
+    'computers':      ('computertypeid',      'computertypes',      'computertype'),
+    'printers':       ('printertypeid',       'printertypes',       'printertype'),
+    'networkdevices': ('networkdevicetypeid', 'networkdevicetypes', 'networkdevicetype'),
+}
+
 
 def resolve(connection, table, idcol, labelcol):
     """Rows that would change, newest table columns tolerated."""
@@ -62,6 +79,46 @@ def resolve(connection, table, idcol, labelcol):
     return label, rows
 
 
+def resolve_types(connection, table, idcol, labelcol):
+    """Rows whose type could be taken from the model, plus the ones that cannot.
+
+    Returns (fillable, unmatched). unmatched rows have a model type that has no
+    identically named entry in this asset's own type table, and are never
+    touched - they are reported so somebody can decide.
+    """
+    from sqlalchemy import inspect, text
+    spec = TYPE_TARGETS.get(table)
+    if not spec:
+        return [], []
+    typecol, typetable, typenamecol = spec
+    columns = {c['name'] for c in inspect(connection).get_columns(table)}
+    if 'modelnumberid' not in columns or typecol not in columns:
+        return [], []
+    label = labelcol if labelcol in columns else idcol
+
+    fillable = connection.execute(text(f"""
+        SELECT a.{idcol} AS assetid, a.{label} AS label,
+               mt.modeltype AS modeltype, tt.{typenamecol} AS typename
+        FROM {table} a
+        JOIN models mo     ON a.modelnumberid = mo.modelnumberid
+        JOIN modeltypes mt ON mo.modeltypeid = mt.modeltypeid
+        JOIN {typetable} tt ON tt.{typenamecol} = mt.modeltype
+        WHERE a.{typecol} IS NULL
+        ORDER BY a.{idcol}
+    """)).fetchall()
+
+    unmatched = connection.execute(text(f"""
+        SELECT mt.modeltype AS modeltype, COUNT(*) AS n
+        FROM {table} a
+        JOIN models mo     ON a.modelnumberid = mo.modelnumberid
+        JOIN modeltypes mt ON mo.modeltypeid = mt.modeltypeid
+        LEFT JOIN {typetable} tt ON tt.{typenamecol} = mt.modeltype
+        WHERE a.{typecol} IS NULL AND tt.{typenamecol} IS NULL
+        GROUP BY mt.modeltype ORDER BY n DESC
+    """)).fetchall()
+    return fillable, unmatched
+
+
 def main():
     parser = argparse.ArgumentParser(description=__doc__,
                                      formatter_class=argparse.RawDescriptionHelpFormatter)
@@ -108,6 +165,31 @@ def main():
                       AND m.vendorid IS NOT NULL
                 """))
 
+            # Type, by exact name only.
+            typed, unmatched = resolve_types(connection, table, idcol, labelcol)
+            spec = TYPE_TARGETS.get(table)
+            print(f'{table}: {len(typed)} row(s) would get a type')
+            for row in typed[:args.limit]:
+                print(f'    {row.assetid:>7}  {str(row.label)[:28]:<28} '
+                      f'model type {row.modeltype[:22]:<22} -> {row.typename}')
+            if len(typed) > args.limit:
+                print(f'    ... and {len(typed) - args.limit} more')
+            for row in unmatched:
+                print(f'    LEFT ALONE: {row.n} row(s) with model type '
+                      f'"{row.modeltype}" - no matching {spec[1]} entry')
+            total += len(typed)
+
+            if args.commit and typed:
+                typecol, typetable, typenamecol = spec
+                connection.execute(text(f"""
+                    UPDATE {table} a
+                    JOIN models mo      ON a.modelnumberid = mo.modelnumberid
+                    JOIN modeltypes mt  ON mo.modeltypeid = mt.modeltypeid
+                    JOIN {typetable} tt ON tt.{typenamecol} = mt.modeltype
+                    SET a.{typecol} = tt.{typecol}
+                    WHERE a.{typecol} IS NULL
+                """))
+
         if args.commit:
             db.session.commit()
             print(f'\nCommitted. {total} row(s) updated.')